I have an awesome idea for King of the Hill, but what if you can't do tests yourself due to time and system constraints. Would providing a test program be enough, or do you have to do the tests yourself?
If you post it in the sandbox explicitly saying that, someone might volunteer to test if they're interested enough. You can take them up on the offer if you wish, but remember that if you post it, it's still your responsibility to award a winner. If something happens and the volunteer can't do it, you need a backup plan.
I certainly wouldn't expect each user to test themselves. While having a program to test with is fine, they can just report whatever they want. In fact, according to my tests, I'm undefeated in the BattleBots Tournament right now. Of course, I may or may not have tested against all entries, but my statment remains true. I'm not saying all (or even most) users would outright lie, but even innocent mistakes could be detrimental to the post in general. Besides, people do crazy things to win at contests, especially when you don't win much of actual value from them. Go figure, humans are weird.
Bottom line: Unless the testing is done by you or someone you trust, you really have no way of knowing for sure who won. Bickering will ensue, hordes of goblins will descend from the mountainside, and chaos shall reign eternal.
I think if you don't have one central 'authority' doing the scoring runs, then all the contestants are going to have to install all the compilation/execution environments for all their opponents on their machine to check that their bot has a chance. Not everyone is prepared to do that so you might alienate some people who might otherwise compete.
I agree with Gareth that ultimately someone has to take on the burden of installing all the necessary build environments and running the tests, or else in the absence of someone it becomes everyone.
The idea of providing some incentive to someone to volunteer to host a KotH contest is nice in theory, but I wonder in practice how many willing volunteers would not simply busy themselves with hosting KotH contests they dreamed up themselves?
There is another way: create KotH contests structured around existing test beds. The Internet is full of game servers that accept bots for play. I know that Go and Backgammon are definitely options as I've played against bots on KGS and FIBS, and there are bound to be many others. There's obviously a trust aspect any time you enable a user to participate in their own testing, as Geobits said, and I am sure you'd get someone running a top-rated bot program not of their own making now and then. But with the proper limitations (e.g. Best Go bot in 200 tokens of atomic code golf) it could be made pretty obvious that someone was cheating.
The upside here is that most of these servers come with built-in rating systems, so the contest management becomes something as simple as tracking and reporting the bots' elo ratings during the contest.
Having given this fine suggestion, I do join Geobits in suggesting that you give some thought to dmckee's comment. The overhead to manage one of these contests is probably not as great as you imagine. Even if you are paranoid about foreign code on your system, virtual machines plus package managers like apt & homebrew make setting up an environment not too big a headache, and you can easily blitz it all when you're done.
I think I just talked myself into hosting one myself soon. :)