Tips for writing good code golf questions

Question

I think the code-golf tag wiki doesn't have enough on writing a good code golf question. I'd like to collect advice on these, and perhaps the most well-liked can be incorporated.

What makes a code-golf question good? What are some pitfalls to avoid?

Answer 1

Golf your own question

Actually write code that solves your question. Nope, don't just say you can do it. Write the code to the spec, and use it to generate or check your test cases. That way, if there's something unclear or wrong in the spec, you'll notice.

Also, make a best effort to golf your code. Have a submission you could actually post (but don't post it immediately). Compare different methods. That way, if there's an exploit, you're likely to catch it. Maybe you find out that you can't beat hardcoding the answer. Maybe you find a library function just does it for you. Or, if you thought there were multiple methods, but one turns out far better, you can change your question or tune the parameters to be more interesting.

Answer 2

Check your scoring method

If you choose to score in something other than just characters or bytes, make sure the formula actually does what you want it to do.

For example, you might have a classification problem that allows programs to output a confidence score between 0 and 1, with part of the scoring being <confidence> * <-1 if incorrect, 1 if correct> for each test case. It sounds nice, but the optimal strategy would be to only output one of two confidence scores — 0 or 1.

If you want to award bonuses for additional tasks, think about how that might translate for different languages, and whether or not it'd be worth aiming for the bonus. You may want to consider awarding a bonus of -X% of bytes as opposed to -X bytes.

Answer 3

Beware Brute Force

Say you have a question to find an example of something or an optimal something. Are you OK with code that just iterates over all the possibilities? It's easy to forget about algorithms that are massive overkill in computation but take little code to implement. Often, such code is short and beats more interesting domain-specific solutions.

If this is a possibility, you should specify whether it's legal to try, say, all 2^64 machine integers or all 256^10 strings of ten ASCII characters. If you wish, you can impose a time limit on test cases to rule out such solutions, or specify that solutions must be asymptotically polynomial time in some parameters.

Answer 4

Cover your ground with test cases

You should include test cases in your question so that someone can check if their program is correct (with high probability) by whether it gets all the test cases right. These test cases should give inputs and outputs.

(Of course, this is moot for challenges with no input and a fixed output, like kolmogorov-complexity.)

Your test cases should cover the conceptual span of possible inputs, so that if a program gets your test cases right, it is likely to behave correctly on all inputs. Does your spec require negative numbers to be handled? Test them! Is the empty string a valid input? Include it! Can the list length go up to 255? Include a length-255 list!

Particularly important are edge cases that could be pitfalls for slightly-wrong algorithms. If there's a heuristic algorithm that gets most inputs right, but not all, emphasize these "interesting" inputs in the test cases. Conversely, look at your test cases and see if there's an unintentional pattern so you can include additional ones that break it.

However, if your challenge is such that there really only one "type" of input, and anything that gets one interesting input right surely gets all of them right, then just one of two test cases is fine.

Answer 5

Specify input and output restrictions

Sometimes, in a code golf question, you will want answers to take input in a specific form, and return it in a specific form. Other times, the requirements are more along the lines of "Whatever is most convenient for your language, but no preprocessing." Either way, you should say which one it is.

Answer 6

Seek Inspiration in Everyday Life

I have done this with very successful results. For example this question was inspired by looking at a multiple-choice test and seeing when I answered the same letter multiple times in a row.

The more challenges one conceives, the greater the chances that one of them will be successful. So always be on the lookout for a new challenge.