There's a lot to remember to include when writing a challenge for this site. What are some things that are easy to forget?
1\$\begingroup\$ Good answers from this one should be clubbed and put into the sandbox (also?) (as a summary?). Sandbox is the place where most people knowing meta would be posting their submissions too. Would be nice to have a summary there. \$\endgroup\$– OptimizerJul 20, 2015 at 18:26
\$\begingroup\$ @Optimizer, That's a great suggestion. \$\endgroup\$– Nick B.Jul 20, 2015 at 18:29
\$\begingroup\$ Related \$\endgroup\$– Peter TaylorJul 21, 2015 at 21:24
Here's a list of details and considerations a good challenge writer should be aware of. I try to mentally go over these myself whenever I write a challenge.
This list assumes the principal parts of the challenge - namely the problem statement, the scoring criterion, any examples, an introduction - are already present and fleshed out. Rather, it focuses on oft forgettable, loophole-closing particulars.
What, if any, similar challenges already exist?
- Has this challenge been done before?
- Is my challenge similar to another one? If another similar challenge has been done, consider changing the challenge creatively to be more distinct.
If there is input, how can it be taken?
- Through stdin?
- Through command line arguments?
- If a file is part of the input, should the path to the file be taken, or can the raw file data be input? Can the name and location of the file be hardcoded?
- Can minor modifications be made to the specified input format? e.g. commas instead of spaces, parentheses instead of brackets, can trailing spaces or newlines be assumed?
- Is input guaranteed to come in the specified format? That is, have you accurately described what counts as input and what doesn't - including corner cases like empty strings or numbers larger than 32 bits?
If there is output (there almost always is), how should it be given?
- Printed to stdout? (usual default)
- Present in some variable by the end of the program?
- If a language does not have an "stdout" what should be done?
- If a file is part of the output, should it really be properly created and saved or does printing the raw file data suffice? If the former, can the name and location of the file be hardcoded?
- Can minor modifications be made to the specified output format? (I usually mention that an optional trailing newline is allowed because many print statements include one by default.)
Is a full program required? A function? A code snippet?
- If a function is allowed, can it take the input like a full program or must the input be passed in via an argument?
- If a function is allowed, can the output be returned instead of printed?
- Do functions need to be named or are anonymous functions (lambdas) alright? (allowed by default)
What characters or character encodings are allowed in the program?
- Allowing any characters (all of Unicode) is the norm but sometimes restricting things to ASCII or printable ASCII makes more sense.
- If you restrict things to printable ASCII, you may want to also allow tabs and newlines.
- In some cases, the distinction between different types of newlines (
\n) can matter.
- Consider what characters should be allowed in the input and output as well.
- Remember that some languages like Piet don't technically use characters. (But don't exclude them for that without reason.)
If you're making a code-golf challenge, or otherwise need to size up code, consider well whether counting bytes or characters is best. Bytes is the norm and usually the best choice, but characters are sometimes appropriate.
Does your scoring criterion have a tiebreaker? Ties do happen, especially in code-golf. A tie going to the earliest submission (with edits considered) is a good default but there are other options that can work in the right situations, e.g. "tie goes to the answer with the fewest [most?] distinct characters".
In ascii-art and similar challenges, it's important to consider what kind of whitespace (usually spaces and newlines) may be present in the input and output.
- Will the input be a rectangle padded with whitespace?
- Can the output have leading/trailing whitespace? How much?
Are external libraries or specialized built-in functions allowed? Generic libraries such as ones that aid in basic image processing are usually acceptable, but sometimes it's important to exclude them. (Just note that some languages like Mathematica have built-ins for practically everything anyway.)
What programming languages can submissions be written in? All languages are allowed by default and it's best to leave it that way unless you are writing a king-of-the-hill or tips question, or perhaps doing something with Stack Snippets.
Should the answers run within some time limit? If so, what are the specs of the computer the time limit applies to? If the programs really will be run and timed, whose computer will be used? (Yours?)
Can programs be non-deterministic, i.e. can they use (pseudo)random numbers or be dependent on the time of day they are run? This may be an important consideration for king--of-the-hill submissions.
If your challenge involves extremely large strings or integers (like more that 64 bits can hold) or lots of memory (more than is on the computer), are constructs like BigInteger required or is it only important that the program would "theoretically" work, given enough resources?
If erroring programs are part of your challenge, be sure to define what you mean by an error.
- Do compile-time and runtime errors count?
- What if the output was produced and then the program errored?
- Do compiler warnings count as errors? (Are there restrictions on the compiler settings to begin with?)
Can all or part of the problem to solve be done with the help of the web? Getting the output externally is a standard loophole, but then again, some challenges implicitly require answers to access the web. It's best to clarify when appropriate.
In quine challenges, can the program be given itself as input or read itself with special commands? Usually the answer is 'no' but that's not an absolute rule.
Are comments (single or multi-line) in the code allowed? Usually the answer is 'yes' but sometimes restricting comments is reasonable.
Are programs that run forever allowed? (assuming they correctly produce output) Usually the answer is 'no' but in rare cases allowing this is viable.
List is roughly in order of most important to least. "You" refers to the challenge author.
Not all of these points need to be mentioned in every challenge (that would be overkill). But they are all things to consider as openings to potential loopholes, and should be mentioned when appropriate.
Here are the meta posts on our "official" defaults for some various list points:
- Input & Output
- Program, Function, or Snippet
- Function Literals
- Loopholes that are forbidden by default
I personally don't know much about our "official" defaults so anyone is welcome to add links to this list.
2\$\begingroup\$ Mathematica has a built-in for everything... Except for string handling, which most questions require. \$\endgroup\$ Jan 4, 2016 at 22:51
If your question requires the use of floating point, you will almost certainly need to constrain the input and allow answers to produce whatever output they want for inputs outside the specified range. You should do sufficient numerical analysis to ensure that it's possible to meet your specification for accuracy in the output over the entire range of valid input.
If you ask for numbers to be rounded to integers, you should specify which subset of the following rounding modes is acceptable:
- Floor (round all numbers towards -infinity) (e.g.
-1.2 -> -2,
1.2 -> 1)
- Ceiling (round all numbers towards +infinity) (e.g.
-1.2 -> -1,
1.2 -> 2)
- Round all numbers towards zero (e.g.
-1.2 -> -1,
1.2 -> 1)
- Round to nearest, half-down (e.g.
0.5 -> 0,
1.5 -> 1)
- Round to nearest, half-up (e.g.
0.5 -> 1,
1.5 -> 2)
- Round to nearest, half to even (e.g.
0.5 -> 0,
1.5 -> 2)
- Round to nearest, half to odd (e.g.
0.5 -> 1,
1.5 -> 1)
Consider edge cases
Are there important edge cases in your question that need to be covered? Often these are situations that are not immediately obvious and can be overlooked by a challenge solution if it is not explicitly pointed out. Either disallow these cases from occurring in your input, or explicitly mention them and how they should be handled (i.e. a test case)
This can be difficult to do, especially since it's easy for you to also overlook these edge cases! So it's always a good idea to sandbox your posts to see if others come across these edge cases.
Examples from actual questions
- Where will the ball land? Challenge author covers edge cases of column 0 (off the board), and the ball becoming trapped and never reaching the end.
- How to randomize letters in a word Challenge author realises and edits in the edge cases of words with 1 or 2 letters.
When you are using a custom scoring criterion, make sure you specify whether smallest or largest score wins.
Testable date questions
If you're thinking about asking a question which uses today's date, don't.
Instead, make it take "today's" date as an input. This way you can actually write test cases which cover all the tricky situations which arise with dates. It's not reasonable to expect people to play around with their system clock just to test their (or someone else's) code.
Make a reference implementation
There are times that specifying the challenge is difficult. Making a reference implementation for your challenge solves the specification problem.