2
\$\begingroup\$

I have an awesome idea for King of the Hill, but what if you can't do tests yourself due to time and system constraints. Would providing a test program be enough, or do you have to do the tests yourself?

\$\endgroup\$
  • 1
    \$\begingroup\$ What makes you think that your resources won't cut it? If you start getting into a situation where running the tourney would take more than overnight on a basic consumer machine you're talking about a big job. Modern computers are ridiculously powerful. \$\endgroup\$ – dmckee Mar 12 '14 at 3:30
  • 1
    \$\begingroup\$ Time constraints? Writing a good question is going to take longer than testing the results. \$\endgroup\$ – Peter Taylor Mar 12 '14 at 8:44
1
\$\begingroup\$

If you post it in the sandbox explicitly saying that, someone might volunteer to test if they're interested enough. You can take them up on the offer if you wish, but remember that if you post it, it's still your responsibility to award a winner. If something happens and the volunteer can't do it, you need a backup plan.

I certainly wouldn't expect each user to test themselves. While having a program to test with is fine, they can just report whatever they want. In fact, according to my tests, I'm undefeated in the BattleBots Tournament right now. Of course, I may or may not have tested against all entries, but my statment remains true. I'm not saying all (or even most) users would outright lie, but even innocent mistakes could be detrimental to the post in general. Besides, people do crazy things to win at contests, especially when you don't win much of actual value from them. Go figure, humans are weird.

Bottom line: Unless the testing is done by you or someone you trust, you really have no way of knowing for sure who won. Bickering will ensue, hordes of goblins will descend from the mountainside, and chaos shall reign eternal.

\$\endgroup\$
  • \$\begingroup\$ Not to mention that in one previous king-of-the-hill there was some weird irreproducibility issue. \$\endgroup\$ – Peter Taylor Mar 12 '14 at 8:43
  • \$\begingroup\$ This brings up another point...How does an asker reward a volunteer tester? \$\endgroup\$ – David Wilkins Mar 12 '14 at 14:07
  • \$\begingroup\$ @DavidWilkins My definition of volunteer is "to freely offer to do something". I think if you're only doing it for a possible reward, you're not really volunteering. \$\endgroup\$ – Geobits Mar 12 '14 at 14:10
  • \$\begingroup\$ @Geobits Poor choice of words on my part...Recognize would be more what I mean....Or are the volunteers destined to be eternal unsung heroes of PPCG? \$\endgroup\$ – David Wilkins Mar 12 '14 at 14:16
  • \$\begingroup\$ @DavidWilkins A bounty on a good answer of the volunteer is possible. Personally, if I were going to do the testing, it would be because the challenge sounds interesting and I want to compete, not because I want recognition for doing the testing. Honestly, though, I think dmckee's comment on the question is a good one, and probably a better answer than this. \$\endgroup\$ – Geobits Mar 12 '14 at 14:28
1
\$\begingroup\$

I think if you don't have one central 'authority' doing the scoring runs, then all the contestants are going to have to install all the compilation/execution environments for all their opponents on their machine to check that their bot has a chance. Not everyone is prepared to do that so you might alienate some people who might otherwise compete.

\$\endgroup\$
0
\$\begingroup\$

I agree with Gareth that ultimately someone has to take on the burden of installing all the necessary build environments and running the tests, or else in the absence of someone it becomes everyone.

The idea of providing some incentive to someone to volunteer to host a KotH contest is nice in theory, but I wonder in practice how many willing volunteers would not simply busy themselves with hosting KotH contests they dreamed up themselves?

There is another way: create KotH contests structured around existing test beds. The Internet is full of game servers that accept bots for play. I know that Go and Backgammon are definitely options as I've played against bots on KGS and FIBS, and there are bound to be many others. There's obviously a trust aspect any time you enable a user to participate in their own testing, as Geobits said, and I am sure you'd get someone running a top-rated bot program not of their own making now and then. But with the proper limitations (e.g. Best Go bot in 200 tokens of atomic code golf) it could be made pretty obvious that someone was cheating.

The upside here is that most of these servers come with built-in rating systems, so the contest management becomes something as simple as tracking and reporting the bots' elo ratings during the contest.

Having given this fine suggestion, I do join Geobits in suggesting that you give some thought to dmckee's comment. The overhead to manage one of these contests is probably not as great as you imagine. Even if you are paranoid about foreign code on your system, virtual machines plus package managers like apt & homebrew make setting up an environment not too big a headache, and you can easily blitz it all when you're done.

I think I just talked myself into hosting one myself soon. :)

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .