Things to avoid when writing challenges

Question

If there is something that you find annoying / counterproductive / unfair / detrimental / no longer funny in challenges (question posts), describe it in an answer here, and propose a recommended response. Voting will indicate whether that response has the backing of the community.

For example, you might suggest one of the following:

1. Comment linking here
2. Edit question
3. Close question
4. Delete question
5. Other (please specify)

Consider quoting from (rather than linking to) example challenges, as the fact that you are posting here makes it likely that the challenge may be edited making the link confusing for future readers.

---

For aspects of solutions (answer posts) to be avoided, see instead Loopholes that are forbidden by default

---

Return to FAQ index

Answer 1

Excessively long back stories

This is a site for people who like programming challenges. They already want to see your idea for a challenge. You don't need a back story to make your challenge appeal to them. If you decide to include some context for the task, try to keep it short.

Personally, I'm more likely to be put off by a needlessly long challenge wording. If it's long because it's complex, I'll read it. If it's long because of telling a story to make a simple challenge more interesting, it actually becomes less interesting to me. Simple challenges are good. Let's keep them simple.

Answer 2

Tacked-on fixes

If you clarify or modify your challenge after posting, don't just tack on a note to the end or to a list of bullet points. Edit in the change.

It's confusing when the body of the challenge says one thing but a later part says another. Of course, it would be ideal not to change the challenge after posting, but if you have to edit, do so cleanly.

For example, if you said to "take two integers as input", but, when asked, clarify that you had positive integers in mind, don't just add a note at the end that says

• You may assume the input integers will be positive.

Instead, change the line to "take two positive integers as input".

If you have to make a substantial change, or there's already answers, it might be worth making a note about your change at the end to make people aware of it.

Answer 3

Exceptional edge cases

Don't require special behavior when the input is the empty list, the number 0, a negative length, etc. This makes for nasty surprises for solvers who worked on a solution only to find it's invalid on a technicality.

When possible, the edge case should follow the same rules as any other input without special instructions. However, even when the right answer is mathematically clear, it's often better to just specify the input will be non-empty, positive, etc, and avoid the issue altogether. If you do insist on including the edge case, do include a test case for it.

One place this comes up is the string representation of 0, which is conventionally '0' but mathematically is the empty string. Once in a while I find a nice recursive function that beats the boring built-in string conversion, but it fails the 0 test case, and it costs too many ugly bytes to patch. Specifying nonzero input would avoid this.

Answer 4

Adding special cases for the sake of completeness

This is a generalization of "complicated" number types, and is similar in scope to adding unnecessary fluff.

What I mean by this is that many challenges will try to make the problem well-defined for all possible inputs, when the challenge would have been much better if the set of inputs had been constrained more.

The challenge that prompted me to write this was multiply a string and a number. The basic idea is to repeat each character in a string a given amount of times. However, it also specifies what to do when the input number is negative, and the behavior is not a simple extension of the behavior for nonnegative inputs.

In my opinion, this special condition makes the challenge worse. I come to PPCG to solve interesting programming problems, and conditionally reversing a string based on whether a number is negative is not an interesting programming problem. It is unnecessary boilerplate that stops me from wanting to solve the challenge.

Furthermore, there are languages in which solving the problem is hard enough as it is. In the challenge linked above, one user decided to try and solve the problem in Retina. Retina was not designed for numeric handling, and the solution only worked for positive inputs. Thus, the edge condition stopped the answer from being competitive.

There are also languages such as Underload where input scope would be complete. In Underload, the most natural representation of numbers is whole numbers, and requiring that negative numbers be handled makes the challenge feel much more clunky.

In general, always choose the smallest set of inputs for which the problem is well-defined and interesting, and only require submissions to handle those. Anything else is edge cases, and handling edge cases is not fun.

Answer 5

Assuming you've addressed sandbox feedback

When your challenge is in the sandbox and someone suggests a change or clarification, make sure your edit actually addresses their point. Too often the same complaints are brought up on the live challenge, and the poster is confused thinking they already dealt with them.

First, consider if the comment still holds true for your edited version. Maybe the rule is clearer than before, but still unclear. Maybe you've addressed some points but not others.

After editing, leave the challenge in the sandbox for more feedback. Maybe the change introduced new problems. Maybe the commenter will clarify their suggestions or offer further revisions. You can ping them in the comments to ask them to take another look.

Answer 6

Multipart challenges with unrelated sub-tasks

In general, a challenge on Code Golf should focus on a single core task. Challenges with multiple sub-tasks are discouraged, especially when the sub-tasks have zero or very little interaction or common aspects.

If the answer to at least one of the following questions is "yes", your challenge has a high chance to be poorly received on Code Golf.

1. If the sub-tasks are posted as individual challenges, are they interesting on their own, or are some of them already on Code Golf?
   • If multiple individual sub-tasks are already on Code Golf, your challenge has a high risk of getting closed as duplicate. An extreme example.
2. If you collect golfed answers to individual sub-tasks and simply merge into one, is it likely to be competitive?

One type of newbies' challenges is "replicate this small but multi-function utility", which suffers from being multipart by definition. (Note that they often also suffer from other kinds of quality problems, such as rigid I/O, input validation/error handling, and sometimes lack of clarity due to random numbers or arithmetic expressions.)

More in-depth discussion can be found in this post by Martin Ender.

Answer 7

Using the sandbox to "defend" your challenge

The sandbox is meant to be an aid to improve the challenge. But once posted to the main site, the quality of the challenge is entirely the poster's responsibility. Regardless of the sandbox, the challenge on the main site should be solid and well specified.

If you post to the main site and the challenge is found to have some flaw, having posted it in the sandbox first does not entitle you to anything. The fact is that the challenge has a flaw, and it should be corrected if possible, or else expect downvotes or close-votes. It's pointless to try to "defend" the challenge on the grounds that it spent several days in the sandbox and no one complained.

---

So why does it sometimes happen that people find the flaw on the main site and not in the sandbox? I see two main reasons:

• Fewer people see the challenge in the sandbox than on the main site;
• People who do see the challenge in the sandbox typically read it trying to find potential problems, but on the main site people try to solve it, which brings out problems more easily.

Possible solutions include:

• Try to call more attention to the challenge while in the sandbox, for example in The Nineteenth Byte chatroom.
• Ask someone to try to solve the challenge before posting.

---

Related: Assuming you've addressed sandbox feedback

Answer 8

Parsing expressions

In challenges dealing with algebraic expressions, don't make the golfer parse input strings like x^3-2x+1 or e^2^x, or to output in this form. This is cumbersome I/O that takes a lot of bytes and effort in languages that aren't like Mathematica, distracting from the actual mathematical task. For instance, a challenge about differentiating polynomials should allow them to be represented as lists of coefficients.

If you do insist on allowing a broad class of nested expressions like 2^(x+3), consider allowing them in a pre-parsed tree like ["Exp", 2, ["Plus", Var("x"), 3]]. But, make sure you give a precise, comprehensive description of exactly what operations and values can be included. Try to include as few operations as is necessary for the challenge to work.

Beware that not all expressions are algebraically "nice", and make sure your test cases cover the gamut of possible expressions. In particular, consider how to deal with things like x/0 or 0^0 or (-2)^(1/2) or (x*x)**0.5/x.

Make sure your problem is actually solvable. Note that there provably doesn't exist any algorithm to decide whether an expression in a real variable x is always zero for every x, if certain operations and constants are allowed. So, your challenge might be outright impossible.

Answer 9

Requiring multiple types as return value

This is very similar to Explicitly disallowing or disadvantaging arbitrary (classes of) languages though it's more subtle (and not necessarily explicit):

A lot of languages don't allow returning multiple types therefore it's not a good idea to require different types as a return value because it rules out those languages for no reason (of course there might be challenges exempt from this).

Note: While not requiring a different type, the same should hold for exit codes and errors - in some languages it is not possible to generate errors or certain exit codes.

Examples

• Given some input X if it satisfies property Y return a list of integers, else return a falsy value
• Given some input X, if it satisfies property Y return the integers f(X), else return a string saying zzz
• Given the input X if it achieves Y return some Z else exit with a non-zero error cod
• Given X if it Ys then return some Z else error

Answer 10

Problems without solutions

When writing a challenge, make sure it is actually solvable within the framework of recreational programming. While a task like "Identify objects in a picture" is certainly a difficult programming challenge, it's not something people can solve in an afternoon: The entirety of the computer science field has been working on cracking that nut for decades and still hasn't found a satisfactory solution.

Therefore, when posting a challenge, you should have at least a clear idea of an algorithm that can perform the task you want, and that you could implement in a reasonable amount of time. Ideally, you'd even have written and tested it. (Such an implementation wouldn't need to be golfed, or pretty, or even fast: The points is that you can prove by example that it's possible to solve the problem.)

Answer 11

Think twice if you're writing a challenge around an algorithm you found on the Internet

Simply put, it will never work as you expect, (though it could work as "a challenge") unless you pick your algorithm and the problem very carefully.

• If you make it a plain code-golf without any restrictions, answers are free to choose brute force (which works universally since we don't require "it should run in practice" kind of thing, and is guaranteed to be golfier in all languages).
• If you make it a restricted-time, a heuristic search might suffice, and you're possibly penalizing slow languages from faster ones. restricted-complexity might work, but it wouldn't if your algorithm's time complexity (or the problem's) is not very well-defined in terms of the input (could be only empirically fast, or depend on additional parameters like LLL).
• If you make it a fastest-code or fastest-algorithm, you need to choose your problem where the intended algorithm would work well, and study additional algorithms that might also work (comparable complexity or empirical performance). In addition, fastest-algorithm might not work for the same reasons as restricted-complexity.

Answer 12

Real-valued output without further specification, or with a bad one

For integer outputs, you can check correctness with exact equality. For exact fractions, many challenges require to output a (numerator, denominator) pair which has worked out pretty well in practice.

But for real number outputs, the situation is totally different. All we have is floating-point numbers, which is a mere approximation of real numbers. Since it is an approximation, the challenge needs a requirement on accuracy as a validity criterion. And writing a good requirement on accuracy is HARD. Here are some guidelines to consider when writing such a challenge.

• Use relative error. "Should output within a relative error of \$e\$" means that, if the true value is \$x\$ and the output is \$y\$, \$\frac{|x-y|}{x} \le e\$. The value of \$e\$ is commonly set to \$10^{-6}\$, but can be adjusted as necessary. (More on this below.)

  Or even better, use "absolute or relative error" (use absolute error when the expected result is less than 1 in magnitude, and relative error otherwise) when the result can be small as well. Some (theoretically correct) algorithms may fail to handle sufficiently small numbers and just give 0, which always has a relative error of 1 by definition.

  • "Should output within 1 of the correct answer" is saying about absolute error. It is bad if the result can be arbitrarily large, as it requires more and more correct digits as the expected result increases, and eventually it will go outside the limits of usual floating-point numbers (usually 64-bit for most languages).

• Consider various ways to compute the given function. One could use numerical integration (e.g. Riemann sums), iterative convergence (e.g. Newton's method), or maybe a simple approximation formula exists. These should be taken into account in various aspects, such as the following.

  • Decide which methods to disallow. Maybe an approximation formula looks too cheap? Then you might need to tighten the error margin \$e\$ (mentioned above) and provide test cases where the formula doesn't produce close enough values.
  • Does the task contain some extreme cases (the output is 0, +/- infinity, undefined or otherwise discontinuous)? The best option in that case is "the inputs close to such extreme points won't be given".

• Ensuring the error limit for all possible inputs is extremely hard, and most of us simply don't know how to prove it. So it is better to give multiple test cases and specify that an answer "should output within error for the given test cases".

Circumference of an ellipse (Disclaimer: one of my challenges) is a reference of a well-specified real-valued task.

Answer 13

Consider if your challenge really needs random output

Challenges that require random output require extra clarifications that challenges with deterministic output do not, and additionally exclude languages with no source of non-determinism.

Often times randomness only acts as a cumbersome output format, adding the randomness boilerplate from whatever language (or preventing submissions in that language).

At the same time there are few benefits that it provides to a challenge.

If there are a finite number of valid outputs for every input (which seems to be the majority of random challenges) you can change a challenge from

Generate a random X

to

Output all Xs

Of course there are other ways to remove unnecessary randomness, for example if challenges of the form:

Generated a random X that satisfies property Y

often (but not always) break down into two challenges

Generate a random X

and

Given an X determine if it satisfies property Y

And you will often find a more worthwhile challenge in one of these two (or both).

---

Of course there are challenges that are best with random outputs. The clearest case of this are challenges that ask to generate output with a specific distribution, where meeting the specific distribution is the challenge.

Some examples of such challenges

• Shortest code to produce non-deterministic output

Answer 14

Requiring time limits

This may differ depending on your opinion, as quite a lot of challenges have time restrictions, but, for me, this is a form of requiring a minimum score.

If a challenge is code-golf, unless the specific goal of the challenge is to do the task quickly, adding a time limit simply prevents people from posting good answers and adds nothing positive to the challenge. Older challenges are more guilty of this, but occasionally a new challenge will be posted requiring answers to finish within X minutes.

A lot of challenges that contain a time limit often do so in an attempt to make the challenge more interesting. Banning specific builtins, classes of languages or other arbitrary things has been repeatedly shown to not make a challenge more interesting (see the other answers in this thread), so requiring answers to meet a specific time limit is unlikely to as well.

Furthermore, timing is non-perfect. What takes 5 seconds on one computer may take 50 on another. That's why we require specifications to be provided for fastest-code, otherwise it's no longer objective. TryItOnline has a time limit of one minute, so could possibly be used for such a time restriction. However Dennis - the host of TIO - has said repeatedly that TIO isn't reliable enough for fastest-code, and so probably isn't accurate enough for time limits.

The worst culprits in my experience are chameleon challenges, which detail an inefficient algorithm/method to complete the task, then all but outlaw that method by requiring answers to meet a time limit (see a lot of the older combinatorics challenges for example)

Obviously, this is a matter of opinion; some people prefer to see solutions finish, others don't mind. However, when considering adding in a time limit, ask yourself if it actually improves the challenge or just prevents otherwise valid answers from being posted.

---

As an aside, this isn’t to say that you should never include a time limit. Challenges where the sole aim is to “Do this thing quickly and golfy” are the main areas where a time limit works. Challenges where the aim is “Do this thing golfy. Oh and make sure you choose an algorithm that isn’t too slow” are a poor idea. Instead, look into either fastest-code, or consider using an asymptotic complexity limit, restricted-complexity, which is much more objective.

Answer 15

Exclamation marks in titles!

Quite a lot of questions have exclamation marks at the end of their titles! To me this just seems like a vain attempt to attract more attention with the hope of getting more upvotes! I don't think I've ever seen a challenge with an exclamation mark in the title where the exclamation mark enhances the challenge in any way!

Answer 16

Pure programming puzzles

Despite our name, programming puzzles aren't generally welcome here. I'm talking about challenges where you're given a task with some impossible-seeming restrictions and must use some clever trick or language feature to solve it.

Programming puzzles are closed most of ther time for not having an objective winning criterion: something that lets you compare submissions to decide a winner. A criterion for a valid solution is not enough -- others need to be able to do better and beat the solution. You could say the first valid solution wins, but some people find it unsatisfying that someone else can win before they even see the challenge. It's better when more people can participate in and enjoy a challenge.

If it makes sense for your challenge, you can say the shortest solution wins (code golf) or have some other metric. This is best for puzzles that can be solved by more than one method, so the search for better methods can be interesting. Beware that once someone posts a solution to the puzzle, others can see it and copy the key idea with a better-scoring implementation.

Answer 17

Challenges whose validity can only be judged based on an external source

Based on this discussion

Avoid writing challenges like "scrape this website" or "use this API" or reproduce some website. Things on the internet change or go down all the time. If this happens all answers will be invalidated.

Instead, try to fix the relevant parts in your code. "Send a API request matching this JSON schema to this URL". Even if the URL no longer exists or has changed, you can still validate if submissions send something matching the schema.

For something like scrapping you can do the same. Include a text like "Assume the response will be formatted like the following" and then include a HTML sample.

Of course, if you are straying from a real site anyways you could consider simplifying these resources. If they don't have to match a real site you could remove parts that don't add much fun to the actual golfing.

Answer 18

cops-and-robbers with easy fallback to general find-a-solution

Actually lot of these question do have one, the first few I see:

• It almost works (Cops' Thread) (t=input();f(t);print())
• The Math is Right, But the Answer is Wrong - Robbers (f(input())?1e999-1e999:0)
• Guess my password (cops' thread) (f(input()))
• Peel away the layers - Cops (code[x+='0'], code[x+='1'], checker)

Answer 19

Entirely nonsensical rationales for short code length.

This was fun (and was documented as a local meme) when the rationales vaguely made sense, for example, "We want the program that does this to be as small as possible so it can fit into the margins" (from The Margin is too Narrow).

However, this has gotten out of hand. Recently more challenges have included rationales that make absolutely no sense, in or out of context. For example, "Because you can only stack about 5 goats before they start to fall, your code will need to be as short as possible" (from Tokenize a Stack-Based language, since edited out).

In an effort to keep challenge specs concise and readable, I propose that we do the following when we see something clearly off-topic for the challenge in the spec:

• Edit the question
• Link here in the edit description (it will appear in the revision history)
• Send attack drones