tl;dr: Challenge authors are responsible for insuring that entry authors can know if they have or have not done the right thing; and there is flexibility in how that is accomplished.
All challenges
Potential authors of entries should be able to check for themselves if their program is or is not doing the right thing.
In principle a really complete spec is sufficient, but we all know that
- it is easy to miss a corner case when writing specifications
- specifications in words are sometimes subject to misunderstanding or even deliberate misconstrual
- challenge authors can have hidden assumptions that are not specified--often derived from limited experience, think about problems that are very hard in imperative programming but easy in logic programming and the like
Sometimes these ambiguities are good because they allow room for exploration in writing solutions, but often they just leave the challenge open to uninteresting abuse and yet another pass at the same stale old jokes.
I prefer to see a reference implementation, a test scaffold, or a reasonably complete set of test cases. I am not, however, picky about which of these you provide; I simply assume that corner cases not tested are undefined and that each program may treat them or fail in it's own way.
KoTH
For a king-of-the-hill challenge, the usually questions about what the spec means are exacerbated by the need to conform to some kind of interface to the scoring system. In the first of these challenges I simply provided the (earliest working version of) the scoring program, so that authors could check the performance of their entries directly. As a side effect several bugs were discovered in the scoring program, brought to my attention and fix. There was also an alternate scorer developed. Another thing you can do is provide a background field of entries whose implementations model the use of the API.
Make the specification clear for king-of-the-hill challenges includes everything needed for other tags plus making the interface unambiguous.
How to deal with challenges that don't comply
If someone posts a challenge you think you'd like to try and you are stymied by the lack of a good spec that is a reasonable reason to downvote. By all means, be a good neighbor and ask for clarification and reverse your vote if it is received.
We will only get consistently good challenges if we make it more rewarding to write a good one than a bad one. While it is true that some people find enough reward in the act alone, that set does not seem to encompass all the posters on CodeGolf SE, and the rest are going to need to be mentored and chivvied along with some gentle advice and downvotes.
