Are authors of challenges required to post test programs along with their challenges?

A comment on this question prompted me to ask. The author of that comment appears to think that the lack either an API or a test program meant that the question deserved to be flagged as unclear.

If the consensus is:

  • "Yes." then something needs to be updated. Do we have a "How to ask" section that is site specific? Sorry for my noobness.
  • "No." then comments like that example I linked to should be responded to.
  • "Only if it is a KotH challenge." then the KotH tag needs to be properly updated. After the tag is updated, it should not apply to challenges that already existed before the change was made.

In other words, whatever the concensus, I don't want this to turn into one of those posts that people link to in two years as a weird community ruling. There should be some action in all cases except for "No." that make the consensus apparent to a new user.


3 Answers 3


I've neither flagged nor downvoted that question but my opinion is that, yes, a question should have the scoring/testing program available from the outset. It gives anyone writing an answer something to test with, and avoids someone misunderstanding the spec and wasting effort on an answer that doesn't work.

  • \$\begingroup\$ Your answer is specific to King of the Hill, whereas the question was more general than that. If KotH is a specific exception to the rule, you should be explicit about that. \$\endgroup\$
    – Rainbolt
    Commented Apr 25, 2014 at 16:18

tl;dr: Challenge authors are responsible for insuring that entry authors can know if they have or have not done the right thing; and there is flexibility in how that is accomplished.

All challenges

Potential authors of entries should be able to check for themselves if their program is or is not doing the right thing.

In principle a really complete spec is sufficient, but we all know that

  1. it is easy to miss a corner case when writing specifications
  2. specifications in words are sometimes subject to misunderstanding or even deliberate misconstrual
  3. challenge authors can have hidden assumptions that are not specified--often derived from limited experience, think about problems that are very hard in imperative programming but easy in logic programming and the like

Sometimes these ambiguities are good because they allow room for exploration in writing solutions, but often they just leave the challenge open to uninteresting abuse and yet another pass at the same stale old jokes.

I prefer to see a reference implementation, a test scaffold, or a reasonably complete set of test cases. I am not, however, picky about which of these you provide; I simply assume that corner cases not tested are undefined and that each program may treat them or fail in it's own way.


For a challenge, the usually questions about what the spec means are exacerbated by the need to conform to some kind of interface to the scoring system. In the first of these challenges I simply provided the (earliest working version of) the scoring program, so that authors could check the performance of their entries directly. As a side effect several bugs were discovered in the scoring program, brought to my attention and fix. There was also an alternate scorer developed. Another thing you can do is provide a background field of entries whose implementations model the use of the API.

Make the specification clear for challenges includes everything needed for other tags plus making the interface unambiguous.

How to deal with challenges that don't comply

If someone posts a challenge you think you'd like to try and you are stymied by the lack of a good spec that is a reasonable reason to downvote. By all means, be a good neighbor and ask for clarification and reverse your vote if it is received.

We will only get consistently good challenges if we make it more rewarding to write a good one than a bad one. While it is true that some people find enough reward in the act alone, that set does not seem to encompass all the posters on CodeGolf SE, and the rest are going to need to be mentored and chivvied along with some gentle advice and downvotes.


Challenges should not be required to post test programs and test cases for the following reasons:

  • A well defined spec should contain all necessary information.
  • Authors of submissions are capable of writing their own test cases and test drivers when test cases and test drivers are not given.

More specifically, authors of King of the Hill challenges should be encouraged to post their tried-and-true, bug-free control programs. If competing with an AI, the logic for the AI must obviously be provided in some form or fashion. The author's reasons for not doing so may include the following.

  • It may not be possible to build the test program until after it is known what languages the submissions will use.
  • The source code for the test program may not be available.
  • The author may choose to withhold the test program until some number of days after the challenge starts.
  • Test programs may contradict the written spec. Abuses of those contradictions can lead to arguments.

If the test program is required for the spec to be clear, then the spec obviously has a problem.

  • \$\begingroup\$ Some of those reasons are really bad. If the author can't write a test program which correctly follows the spec, should they be asking other people to implement the spec? And what would be the point of withholding the test program initially? To force everyone to write their own and then throw it away? \$\endgroup\$ Commented Apr 26, 2014 at 8:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .