# Things to avoid when writing challenges

If there is something that you find annoying / counterproductive / unfair / detrimental / no longer funny in challenges (question posts), describe it in an answer here, and propose a recommended response. Voting will indicate whether that response has the backing of the community.

For example, you might suggest one of the following:

2. Edit question
3. Close question
4. Delete question

Consider quoting from (rather than linking to) example challenges, as the fact that you are posting here makes it likely that the challenge may be edited making the link confusing for future readers.

For aspects of solutions (answer posts) to be avoided, see instead Loopholes that are forbidden by default

• I think that this question is directly targeting the "rationales for short code lengths" posing as a broader (but mostly useless) question. I'd prefer if a different meta post were made specifically talking about the rationales, and allow the community to present both sides. – Nathan Merrill Jan 11 '16 at 14:41
• @NathanMerrill You're welcome to post a response here. Just give your reasoning and recommend no action. It should work just fine on this post. – Alex A. Jan 11 '16 at 18:32
• This may be a good thing to feature, or put in some other high traffic location so it can be treated as a reference. – J Atkin Jan 17 '16 at 1:39

# Cumbersome I/O formats

Generally, allow flexible input and output formats. People want to write code to do your task of, say, composing permutations, not reading/writing numbers in a particular semicolon-separated format, or from a file, or with input validation. For code golf, it's annoying when most of your byte count is boilerplate, especially when other languages read the format natively.

Allowing I/O in the language's native format (list, string, tuple, etc) is usually a better option. Yes, it makes comparisons between languages less exact, but that's worth it.

In particular, don't use long strings for outputs. Instead of "Why, yes, good sir or madam, this list is in fact a palindrome", let programs output 1/0 or True/False or general Truthy/Falsey values.

Of course, if your task is about specific formats, that's fine. ASCII art challenges usually require a precise output. A challenge to process certain file headers can expect a rigid input format.

• I'm tempted to make a language that outputs true as Why, yes, good sir or madam, this is in fact the case! – Nic Hartley Apr 20 '16 at 21:20
• In particular, as a bare minimum, no question should be more restrictive than the defaults codegolf.meta.stackexchange.com/questions/2447/… (That is to say, no question should say things like "full program taking input over STDIN only, no single function answers") – Lyndon White Dec 12 '17 at 8:10
• @LyndonWhite There still are exceptions to that, of course. – MilkyWay90 May 23 at 23:01

# Rules inferred from test cases

Test cases are examples for people to check their understanding of the spec and the correctness of their code. They should not replace an explanation of how the output must relate to the input. (There are rare exceptions where it's straightforward, like a scaling ASCII art pattern.)

Don't leave readers guessing as to what the rules must be from examples. In particular, it can be unclear if the precise input/output format of the test cases is required. Also, additional requirements in test cases are liable to be missed by someone who starts coding before looking at them.

• unless the entire challenge is one of those kolmos with: for each of these as input output the other thing – Destructible Lemon Jun 14 '17 at 1:25

## Explicitly disallowing or disadvantaging arbitrary (classes of) languages

This has become much rarer recently, but the occasional challenge by a new user still includes it, so here as an answer to point them to.

Disallowing arbitrary languages (or classes of languages, primarily things like "no golfing languages allowed") is not in the spirit of this community, as has been discussed many times on meta. If you don't like golfing languages beating others, a) don't pose a code golf challenge or b) look at the challenge as a separate competition in each language.

This answer also encompasses penalties that only apply to certain languages.

Suggested course of action: (for voters)

• Downvote to discourage this behaviour and show that it is not welcome in this community.
• Leave a friendly comment that this is not how we roll and link them here.
• Don't edit it out without the author's permission - I don't think we have a consensus that such restrictions are disallowed or off-topic. They're just a bad idea.

A milder form of this is challenges where the task depends on the language name (e.g. "print the language name"). Those give a byte-count penalty to languages with long names. If this is necessary for the challenge to make sense/be fun, a better idea might be to subtract the language's name from the byte count and require that it appears verbatim in the source code (without the latter requirement, you'd be giving a penalty to languages with short names instead).

This answer does not encompass challenges which are impossible in some (or many) languages due to technical limitations (audio output, file system manipulation etc.). That said, challenges should strive to be as inclusive as possible without damaging the core of the challenge (do submissions really have to play the audio directly via the speakers, or could they also write an audio file to STDOUT?).

When the OP needs to run all submissions on their own machine (e.g. for , or certain s), it's also fine to require that all languages have a freely available compiler or interpreter. I wouldn't call that "arbitrary" restriction. (Whereas, I would call it arbitrary if someone imposed the same restriction on a standard code golf where the OP isn't planning to test every answer on their own machine anyway.)

• I would make an exception for requiring languages that have a freely available compiler/interpreter (so that the OP can test the answers) – aditsu Mar 24 '16 at 16:37
• @aditsu Especially true for king-of-the-hill puzzles – Not that Charles Mar 24 '16 at 20:34
• @aditsu I think such a limitation isn't an "arbitrary" class of languages, but I'll clarify that. ;) – Martin Ender Mar 31 '16 at 12:38
• Well, I plan to test every answer even for standard code golf. – aditsu Mar 31 '16 at 23:41
• @aditsu I think in that case it's viable to let someone else test it who has the language. For code golf it's not necessary that all entries are tested on the same machine. Otherwise, we'd essentially be banning proprietary languages completely. – Martin Ender Apr 1 '16 at 6:04
• I agree with this for the most part but disallowing golfing languages is not arbitrary. They are practically cheating and skew votes in their favor on challenges so they aren't "arbitrary" and I can understand very well why someone would want to disallow them – Downgoat Jul 4 '16 at 16:42
• @Downgoat I fail to see how they are cheating, but this comment thread is probably the wrong place for this discussion. – Martin Ender Jul 4 '16 at 16:43
• I was initially resistant to golfing languages, but then I realized they're simply a different way to enjoy the puzzle. If you create a puzzle and intend for it to be solved in Java, C# or something and some eso-lang fiend writes a program shorter than the language name to solve it, that doesn't make your puzzle any less fun! Also, if it is so easily solved, perhaps it wasn't quite as interesting as originally thought? – corsiKa Aug 23 '16 at 22:00
• Specifically, what made me change my mind was a game called Kittens. I like Kittens and it's fun to play. But I have a wife and two kids, so I don't get much time to play. So I play it with the game paused and manually pass the time using Javascript - this way I can simulate hours at a time (although I greatly enjoy the micromanagement aspect, so it's usually minutes at a time.) The original creator thinks this is cheating - I think it's simply a different way to play. I think the same thing applies here - it's just a different way to play. – corsiKa Aug 23 '16 at 22:02
• Something to note: for the example of tasks that depend on the programming language's name requiring that the language's name appears verbatim in the source code is a disadvantage to languages that do not have string literals (such as brainfuck or Brain-Flak) or impossible in others whose source code is not text (like Piet). Another way of handling is allowing the programs to take their language name from stdin. – 0 ' Mar 10 '17 at 18:38
• @0 ' When you do that, the challenge is no longer dependent on the language's name, so that's fine of course. – Martin Ender Mar 15 '17 at 12:34
• If you dislike golfing languages, the best thing to do is to ask for something they can't do. Most golfing languages have a tiny set of built-ins and standard libraries, while languages like Java and Mathematica have a broad range of tools. If your challenge requires something real languages have a std lib for but golfing languages don't, I doubt, despite their beautiful conciseness, that golfing languages can write a lib in fewer characters than you can import one. – Jared K Jul 27 '18 at 19:26

# Making assumptions about language features

Unless you're writing a language-specific challenge, avoid terms specific to some class of languages, because these might not make sense for other languages and prevent them from participating. Languages can very different from what you're used to: functional, minimalist, graphical, untyped, strongly typed, and weird in many ways.

• "write a function that"
• "return a pointer to"
• "if the input is invalid, throw an exception"
• "at compile time"
• "you may not use the + operator."
• "if they are of different types"
• "modify the input object"

Usually, this can be fixed by using broader terms, but is sometimes is a sign of a problem with the challenge itself. Consider defining the requirements in terms of input and output without reference to the program internals.

• The "write a function that" should be replaced by "write a program or function that" in most challenges. – wizzwizz4 Jan 16 '16 at 9:45
• @user81655 I feel like saying yes, but I can't think of any examples. However, there are programming languages such as ES6 where it is golfier to create a function than a program. – wizzwizz4 Jan 16 '16 at 10:50
• Or Java 8...the boilerplate – CalculatorFeline Feb 28 '16 at 22:04

## Do X without Y

This isn't always bad, but it's been a particular trap for beginners, so be careful.

In the past, there were popular questions about doing a simple task but with the obvious method banned:

These have fallen out of favor because they inspired many copycats that basically came down to the same tricks. In general, if your challenge is too simple to be interesting, it won't be saved by banning things. It's better to make the challenge meatier instead.

Also, it's hard to define "without Y" for a wide class of languages, leading to nitpicking like "I'm not using a+b, just sum(a,b)".

"Do X without Y" can be good questions, but I've found the odds stacked sorely against them, especially for new posters, so be careful. Look at past questions and check that yours requires novel ideas, and beware that site standards have gotten much more stringent over time.

# Changing the challenge in the comments

If something was unspecified in your challenge, or if you've decided to make a change to the challenge (Such as to close a loophole), don't just leave a comment. These are often not visible unless you click on the correct button, and will lead to people getting confused.

Instead, edit the challenge to incorporate the change/clarification.

Appropriate solution:

Edit the change/clarification into the question.

## Bonuses in code golf

Bonuses in code golf say things like "-30% of your byte count if your code can handle any number of strings, not just two." From what I've seen, they range from slightly improving the challenge to seriously harming it. I haven't seen a bonus that totally made a question, but I've seen ones that ruined it. If you do include a bonus, be confident that it makes the question better.

Too often bonuses come from avoiding a hard decision about your challenge. Cutting is hard, so you preserve a weaker or extraneous part of the challenge. Or, you can't decide the scope, so you try to have it both ways. The weak parts might be underspecified, poorly received in the Sandbox, tangential to the rest of the task, or make the spec too cluttered. Don't sweep them into a bonus -- delete them without mercy! Conversely, if you do include a bonus, make sure it as at least as strong as the rest of the challenge.

Bonuses are only interesting when they pose a trade-off that could go either way. It's hard to balance, and often a bonus turns out completely unviable or completely mandatory for the language I'm golfing in. A flat bonus (-20 bytes) means much more in a golfing language than Java, and a multiplicative bonus (-30%) depends on the language's degree of boilerplate and its well-suitedness the bonus relative to the main challenge. Sometimes there's no number that works for everything.

When the bonuses are well-balanced, you can have the opposite problem that a golfer needs to write a bunch of different variants of the program to see what's worth in, which becomes exponentially more cumbersome with more and more bonuses.

• Case in point. Having solved the core problem, I then had to golf three output formats to see which would give the lowest score. That was nowhere near as interesting as the core problem. – Peter Taylor Jan 14 '16 at 21:18
• I definitely prefer this harsher version. Bonuses nearly always make me feel like I'm reading the answers to two or more separate challenges jumbled together on the same page. – trichoplax Jan 15 '16 at 15:57
• exponentially more cumbersome Exactly! If a challenge has n independent bonuses, there are 2ⁿ different ways to combine them. One bonus is bad enough IMHO, three or more are a disaster. – Dennis Jan 19 '16 at 17:17

# Putting test cases in a hard to use format

Supplying test cases is always a good idea, but people are more likely to use them (and thus have better tested code) if they can easily be copied and pasted into a test suite.

So format your test cases as simply and as consistently as possible, preferably in a single code block. If you have lots of test cases and you want to section them out by type, put empty lines between sections instead of using several separate code blocks.

Be sure to include any examples used during explanation as test cases. People may otherwise forget important examples while focusing on test cases.

# Do This:

Input -> Output
cat -> kitten
dog -> puppy
seal -> pup
bear -> cub
cow -> calf
bird -> chick
echidna -> puggle


Replace -> with something else if the - and > might get confused with the input and output. Don't use → (right arrow) or other Non-ASCII since not all languages/compilers may allow it.

# DON'T Do This:

+---------+--------+
|  Input  | Output |
+---------+--------+
| cat     | kitten |
| dog     | puppy  |
| seal    | pup    |
| bear    | cub    |
| cow     | calf   |
| bird    | chick  |
| echidna | puggle |
+---------+--------+


Yes, it looks nicer, but it's more hassle to get the pure data.

# DON'T Do This:

The inputs cat, dog seal, bear, cow, bird, and echidna have the respective outputs kitten, puppy, pup, cub, calf, chick, and puggle.

# DON'T Do This:

The input

cat


has output

kitten


The input

dog


has output

puppy


Likewise, the input

seal


has output

pup


and the input

bear


has output

cub


and the input

cow


has output

calf


and the input

bird


has output

chick


and, finally, the input

echidna


has output

puggle


Yes, I have been guilty of this. No, I am not going to fix all my old challenges.

• The ASCII art table isn't too bad I think. If you just copy out the lines that actually contain test cases the format is about as easy to transform as the -> format. – Martin Ender Jan 14 '16 at 9:26
• @MartinBüttner But there are two extra | dividers and an unknown number of spaces to deal with. Not everyone here knows regex and may rely on strip's and split's and the like, which is more cumbersome with the table. – Calvin's Hobbies Jan 15 '16 at 1:06
• @MartinBüttner Related chat-challenge/poll: chat.stackexchange.com/transcript/message/26850192#26850192 – Calvin's Hobbies Jan 15 '16 at 1:15
• in fact, I even prefer the -> format with the arrows aligned. I think that's the best compromise between human readability and being easy to process into something your code can deal with. – Martin Ender Jan 15 '16 at 6:47
• <kbd>.Header 1. </kbd><kbd>.Header 2. </kbd><kbd>.Header 3.</kbd><br/> <kbd>.Cell 1.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</kbd><kbd>.Cell 2.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</kbd><kbd>.Cell 3.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</kbd><br/> <kbd>.Cell 4.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</kbd><kbd>.Cell 5.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</kbd><kbd>.Cell 6.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</kbd><br/> Modified from here. Pain to type, looks pretty, easy to copy/paste into Calc or something. Can change dots to quotes or whatever. – MichaelS Feb 1 '16 at 23:11
• Of late I've given example inputs like this. bad? ok? good? better? – Digital Trauma Apr 27 '16 at 23:12
• Only if there were tables.. – lol Apr 4 '17 at 14:21
• +1 for monotremes. – Andrew Grimm Jul 2 '17 at 1:32

## Chameleon challenges

Chameleon challenges look like they're about one thing but are really about another. In doing the challenge, most of the effort is spent on something peripheral. Fix this by being honest about your challenge in the title and description, or by simplifying or removing the non-core parts.

Some causes of chameleon challenges:

• Requiring a cumbersome input/output format that takes more effort to process than the challenge itself
• Requiring that malformed inputs are detected and specially handled.
• In code golf, having long fixed output strings that benefit from compression.
• Including bonuses that are so large that any competitive answer must do them.
• Scoring a challenge on a small set of test cases where it's more effective to hardcode the outputs or overfit them than to actually do the task.
• Scoring by fraction right on a test battery where it's not hard to get 100% right, making the tiebreak the real score.
• Scoring by a combination of two factors, like length and effectiveness, in a way that one contribution is much larger than the other.
• I'd like to note that we actually have (and support) a large number of chameleon challenges on PPCG: There are many challenges where the task is the "difficult" part, and attaching the code-golf is simply to make it fit our standards. Ideally we'd prefer these challenges to have a different scoring method, but that doesn't always work out. – Nathan Merrill Jul 7 '17 at 16:35

## Requiring minimum scores

In short, don't post a code golf that says "Your code has to be shorter than 100 bytes" (the same applies to any other winning criterion). The usual motivation is that the author found some code somewhere and challenges people to beat the score of that code. That's a fine motivation for a challenge, but actually requiring all answers to be better than that benchmark is incredibly detrimental to the challenge. It doesn't increase competition between answers that are better than the requirement. But it does disallow answers which are worse than the limit but might contain interesting ideas which the better answers could also benefit from, and it also rules out some languages entirely which are simply incapable of beating the given score. There's also the possibility that you've set the limit too low and made the challenge impossible. Additionally, most often answers aren't originally posted with the best possible score, but are later improved because other people give suggestions. This would be impossible if the first attempt wasn't allowed to be posted.

Most importantly though, all that a minimum score is saying is "if you're not good enough, you can't participate", which I think is plainly rude.

• +1, but if an author really wanted a score threshold, I would be OK with answers that do not meet the score threshold being allowed but being marked as not competitive. – Mego Apr 8 '16 at 5:26
• On the other hand, a minimum score may be a reasonable threshold for defining what qualifies as a serious contender. For example, on my latest challenge, I would consider submissions that do not have positive scores to not be serious contenders, because they failed the basic premise of the challenge (optimize the output length). – Mego Apr 11 '16 at 1:22
• "the same applies to any other winning criterion" - what about program should terminate in one minute for input less than smth? – Qwertiy Apr 11 '16 at 9:23
• @Qwertiy That depends on the challenge. If it's a code golf, adding that as a validity criterion is fine, because it's not related to the scoring and only ensures efficient solutions. If it's a code-challenge/fastest-code along the lines of "scored by largest input you can handle in one minute", then that's also fine because time is not a variable that goes into scoring. But if you have a fastest code that says "fastest code wins, but you must be faster than one minute" you're back to saying "if you're not good enough you can't participate". – Martin Ender Apr 11 '16 at 9:28
• @MartinBüttner, but if some code executes 24 hours eating 100% of processor, are you really going to wait until it finishes? I mean, there should be some reasonable limit on time. By the way, about limit on length: the answer length is limited by 30K chars, so you can't post the anser longer than that. – Qwertiy Apr 11 '16 at 9:31
• @Qwertiy Probably not, because it's going to be beaten by a faster submission anyway, so it's exact score is irrelevant. If I designed the challenge such that all answers will take more than 24 hours, that's really my own fault, and requiring them to be faster than 1 minute isn't actually going to make them faster. Also with such an extreme difference, there's a good chance that the slow answer is intentionally slow which makes it invalid as not trying to compete (as per the help centre). – Martin Ender Apr 11 '16 at 9:35
• @Mego You're right that in your case having a negative score clearly makes it not a serious contender, but that's obvious for everyone to see and doesn't need an explicit rule. I think it's quite rare that there's such a clear cut-off as in your case, and in all other cases I'd rather have the community decide which answers are serious contenders and which aren't, instead of letting the OP guess an arbitrary threshold. – Martin Ender Apr 11 '16 at 9:36
• @Qwertiy Re, answer length limit. You sure can (and we've had answers whose length was given in scientific notation), provided you can describe how the code is generated. Just think of Unary answers. There's no point (and it's impossible) to actually include the code in the answer, but all you need to know is how many 0s the program has. – Martin Ender Apr 11 '16 at 9:37
• Can I say "lowest code length+running time(sec) win, at most 1000 char, at most 10 min"? – l4m2 Apr 3 '18 at 16:21

## Don't allow / ask for different things in languages with different capabilities

Don't write specs like "Do X. If your language doesn't support X, you can do Y instead". Examples are:

• Do some math in floating point. If your language doesn't support floating point, you can use integer
• Read input from a file. If your language doesn't support file I/O, you can read from stdin

X and Y, can be quite different tasks, although they might look similar. Either allow the alternatives Y for all languages or for none.

• Full support on this one. A compromise that I've found can be effective is "Do X. If your language cannot do X, you can do Y instead, but the submission will be non-competing". – Mego Sep 4 '16 at 23:52
• I strongly agree with this answer. I'm not a fan of non-competing submissions. If it doesn't fit the challenge, it should be posted elsewhere. – trichoplax Sep 18 '16 at 16:45
• If a language cannot do X, I don't see any reason to try to include that language. We've got plenty of other challenges that language can compete in. – Nathan Merrill Oct 3 '16 at 15:58
• @trichoplax I don't mind non-competing submissions if they're non-competing because the language came after the challenge, and the language wasn't obviously designed to solve the challenge well. – Mego Dec 10 '16 at 1:11
• @Mego Yes I agree about languages post dating the challenge. I just don't like solutions that solve a different challenge (or challenges that invite this). – trichoplax Dec 10 '16 at 14:55

# Arbitrarily overriding the defaults

The default code formats, input/output methods, and other defaults are a product of community thought and discussion. Don't change them just because you feel like it or disagree with them. They have good reasons for being as they are, some of which are only apparent for specific languages that would otherwise be unusable or cumbersome or require ad-hoc rulings.

Only override the defaults when they don't serve their purpose because of some specific feature of your challenge, and even then think if you can make the challenge accommodate instead.

• I'd like to note that, from my experience, this doesn't typically apply to KOTH challenges. – Nic Hartley Apr 30 '16 at 5:41

Avoid recommended features in your spec. If it's not mandatory, answers won't do it. The goal is to write the shortest/fastest/winningest code that still satisfies the requirements, however minimally, so any features that can be cut or simplified will be.

Of course, popularity contests are an exception.

• Even worse: "Ideally, your code will...; -10% bonus for that feature". – Trang Oul Jan 14 '16 at 13:08
• @TrangOul that's a separate thing to avoid, stated higher up. – John Dvorak Oct 3 '16 at 16:06

## Requiring the use of unnecessarily "complicated" number types

A large proportion of our challenges deal with numbers, or lists of numbers. An important consideration is always what sort of numbers are valid input (positive integers, non-negative integers, all integers, floating-point numbers, complex integers, complex floating-point numbers...).

In the interest of making the challenge accessible to (and enjoyable in) the widest range of languages, try making the number types as simple as possible without harming the challenge.

If your challenge is about sorting a list of numbers, positive integers suffice. Using floating point numbers doesn't add anything interesting to the challenge but makes it incredibly tough or practically impossible to solve the challenge in languages that only support integer data, even though they might be able to solve the actual task of sorting quite easily.

In some cases, the actual values are completely arbitrary and only exist as place holders in the structure of an array. In this case you can even consider restricting the input to single-digit integers, unless you want to specifically make string-processing based approaches less feasible.

Of course, some challenges only make sense for floating point numbers or even complex floating-point, and that's completely fine. But when the core of the challenge is not about handling those data types, they only distract from the interesting parts.

As a rule of thumb I propose that people ask "if I change this from integer to float or vice-versa, would this affect any viable approaches in mainstream languages?" If that's the case, then requiring floating point might make sense (if those are the approaches you're looking for). But if not, limiting the challenge to integers won't do it any harm for the common languages, and will make the challenge more interesting to people who prefer to use esolangs for a little extra challenge.

## Non-observable program requirements

The validity of a program should depend on things that can be observed when the program is treated as a black box. Examples are data written to standard output or error streams, drawing on the screen, file operations, memory usage, and runtime. Non-observable behaviors include "string operations" and "implementing the Fu-Barr algorithm". Additionally, "using Y" would fall into the non-observable category in most instances of "Do X without Y"

Non-observable requirements tend to be vague, subjective, or based on false assumptions about the properties of programming languages. For example, requiring that answers use a specific algorithm is highly open to interpretation. What modifications are allowed? Can some steps be replaced with different operations that produce the same result?

### Exceptions

A commonly used non-observable requirement is to avoid using a library function that solves the entire challenge. I believe these rules are OK. Like other non-observable requirements, one can find borderline cases of whether a built-in function solves all of the task or not, but the benefits of higher-quality answers outweigh the costs.

• Would you consider the usual definition of truthiness to be unobservable? – xnor Sep 4 '16 at 21:23
• I agree with everything except the exception. I think this answer is a good reason to avoid excluding library functions. – trichoplax Sep 4 '16 at 21:34
• @xnor Yes, that's a good example of making false assumptions about the programming language. It falsely assumes that every language has "if" statements. – feersum Sep 5 '16 at 1:59
• Note that if you want to force the use of a particular algorithm, you can normally do it observably by requesting that program prints out the intermediate steps of the algorithm. – user62131 Mar 28 '17 at 22:16
• What a strange consensus. That a programming language actually be a programming language as per "A purported programming language should be accepted as such if and only if it is capable of addition of natural numbers and primality testing of natural numbers," seems to be a non-observable requirement. What about restricted-source? One could even argue that code length a la Starry Night is a non-observable requirement. – primo Jul 7 '18 at 15:38
• @primo I don't understand. Length of a source code and what characters it includes are things which can quite easily be measured. – feersum Jul 8 '18 at 4:50
• if you're willing to examine the code, i.e. not treat it as a "black box", then yes. – primo Jul 8 '18 at 8:43

# Saying you should produce one or several outputs randomly without further specification

## One random output

Say someone writes a challenge about generating a labyrinth contained in a rectangle of specified size, and they say that you should create a randomly arranged labyrinth with the specified size.

What does random mean here? As it has been specified, you may very well create two possible labyrinths that comform to the size specs and pick randomly between the two. Or even create just one and claim it is random.

As it stands, the challenge is not well specified. Just saying random means nothing. The probability distribution needs to be specified. Some correct specifications would be (these are excluding alternatives):

1. All possible labyrinths of the given size should be produced with the same probability;

2. All possible labyrinths of the given size should have a nonzero probability of occurrence;

3. Your code should be able to produce at least 10 labyrinths with nonzero probability

Option 1 corresponds to a uniform distribution, which is "as random as it gets" (but may be too demanding for certain challenges). Option 2 is more flexible on the answerer. With option 3, you should be careful that it is always possible to generate at least 10 valid outputs for your specific challenge.

## Several random outputs

The problem is aggravated when there are several random pieces (numbers, characters, etc) that need to be produced. Consider for example a challenge that takes a string as input and asks to introduce a random printable ASCII character between each character of the string.

First off, the same considerations as above apply (Can I choose from just "a", "b", "c" as my random characters? Do I need to pick among those characters with the same probability?). But in this case there's the additional problem of statistical relationship between the random outputs. Can I randomly pick a character and insert that same character between all positions of the original string? Or do the inserted characters need to be independently picked?

To clarify this, the possible relationship between the randomly generated pieces needs to be specified in the challenge. In more proper terms, the joint probability distribution of the random pieces needs to be specified. Sorry if it sounds a little complicated, but this is necessary.

If you are not sure and want to keep things simple, after having specified the distribution of each piece (as discussed above) you can say that the random pieces should be statistically independent. This means that each piece should be randomly generated without regard of the random choices that were made/will be made for the other pieces.

## Bottomline

The point is to properly specify "randomness". Don't rely on "common sense" or assume that answerers will adhere to the "spirit" of the challenge. Your that's against the spirit of the challenge may very well be someone else's clever trick that saved me a few bytes.

• Related, but not exactly overlapping. – Geobits Dec 15 '16 at 13:52
• I've now added a definition for uniform on that meta post too. – trichoplax Dec 22 '16 at 1:51
• @trichoplax Good point, and well explained there – Luis Mendo Dec 22 '16 at 14:35
• "Your that's against the spirit of the challenge may very well be someone else's clever trick that saved me a few bytes." This is a strong argument for clear, objective rules and specifications everywhere. – Esolanging Fruit Feb 13 '18 at 4:02

## Allowing standard loopholes in general

Don't say "Standard loopholes are allowed". They ban silly things like making up a language where the solution is a single character or just printing the string "the answer". Even if you find these amusing, we got tired of them long ago and that's why they're forbidden.

Allowing specific loopholes, such as fetching the output from an external source can and should be overriden if it makes the challenge better. For example, if the challenge is about getting information from a website, then it should definitely be allowed. This answer addresses allowing standard loopholes in general, as they apply to most challenges.

• I'd argue that it's worth removing the allowance from the challenge completely, even if the author wants it to stay. We as a community have overwhelmingly voted to disallow the standard loopholes across the site, so one user should not be able to override that strong consensus. – Mego Jul 27 '16 at 5:09
• Broadly allowing standard loopholes is a bad idea, but we frequently have good challenges that allow a single loophole. For example, Fetching the desired output from an external source has been "overriden" a couple of times, such as for Automate the OEIS – Nathan Merrill Oct 3 '16 at 16:43

What's the main point of the challenge? Try to keep the challenge to just that. Don't go adding frills to the challenge just because it "looks too easy", or inserting strange modifications to a perfectly okay challenge just to "make things interesting". Keep it lean and straight to the point.

In addition to the related posts above, here are more examples of what not to do (when there's no good alternative approaches):

• Combining two or more unrelated core challenges into one — consider splitting the challenge up into separate challenges or dropping unnecessary parts. Examples include:

• Instead of simply summing a list of numbers n at the end, requiring answers to sum over some unnecessary transformation (e.g.n^2, n+1, nth Fibonacci number) instead.

• Instead of simply applying a function f(x) to a list of inputs, adding an unnecessary filtering step before the function is applied (e.g. median of only the prime numbers in the input).

• Unnecessary extra applications, e.g. instead of calculating a single function f(x):

• Making answers calculate f(x) for every x in a list.
• Making answers take an additional parameter n and applying f to the input x, n times.
• Making answers calculate x-f(x) until the result is 1, then counting how many steps that took.
• The last two bullet points are debatable, especially when f is simple, since there may be (more or less obvious) shortcuts. Case in point. – Zgarb Jun 17 '16 at 14:16
• @Zgarb The emphasis was meant to be on "when there's no good alternative approaches" :) (I should probably add that the only reason they're there is because there's been such examples in the past) – Sp3000 Jun 17 '16 at 14:41
• I think it's a "don't just combine two challenges"? (assuming some basic challenges were there for long time) – l4m2 Apr 21 '18 at 10:55

## Using shortest code as a tie-breaking winning criterion in code-challenge questions

In challenges such as this one, there is nothing stopping users from copying another answer's solution, implementing it in a terser language, and posting it as a separate answer. The user therefore would receive the same score for their submission and win the tie-breaker, despite the original user doing the lion's share of the work and being more deserving of winning.

Using earliest submission as the tie-breaking criterion avoids this problem entirely.

• cough, cough Yeah, or as part of the primary winning criterion in the first place hack, cough – cat Mar 13 '16 at 2:36
• @tac Isn't length of code the primary criterion in all code-golfs? – Adám Mar 14 '16 at 12:08
• @Nᴮᶻ We're talking about code-challenge here – cat Mar 14 '16 at 12:17
• @tac Are we? – Adám Mar 14 '16 at 12:22
• @Nᴮᶻ It literally says code challenge in bold at the top of the answer. – Mego Mar 14 '16 at 17:44
• @​Mego I know, but @tac was probably referring to this. – Adám Mar 14 '16 at 18:52
• @Nᴮᶻ That is a code challenge, not code golf, so it's relevant. – Mego Mar 14 '16 at 19:41
• But, but... My point was exactly that my challenge is a code-golf. Community just wouldn't let me tag it that way. I think it is strange to complain that there is length scoring one something I wanted to be tagged code golf, even though the community changed the tagging. In other words, my tagging and scoring system were consistent! – Adám Mar 14 '16 at 19:50
• @Nᴮᶻ No, it wasn't consistent. This isn't the proper place to have this discussion. If you are still confused as to why your challenge was not code golf, consult the tag wiki, and ask any questions in chat or in a separate meta question. – Mego Mar 14 '16 at 19:52
• I agree with this. Any tie breaker which is also an additional challenge distracts from the main objective and causes the same additional work as bonuses. – trichoplax May 23 '16 at 0:05
• I'd like to offer this challenge as a counterexample where the tie-breaker felt entirely natural. I think it helped that the main score was sort of length-based and very tied to the specific language used. – Ørjan Johansen Apr 16 '18 at 2:12

## Most kinds of generalised quines

Quines continue to fascinate this community... but we already have a plain quine challenge so we're "forced" to innovate by making quine-related challenges that aren't pure quines. For the purpose of this post, I'm defining a generalised quine as a program P which prints F(P), where F is some function of a string. (E.g. "print the reverse of your source code" would have a string reversal function as F.)

Just to be clear, this answer does not apply to other extensions of quines, like putting extra restrictions on them, as in a radiation-hardened quine, or extending them to mutual quines etc. This is purely about "print this transformation of your source code" challenges.

The problem with many of these is that there is no other solution than a) take your language's standard quine to get your source code as a string, b) tack on an implementation of F. If that's the case, the quine part doesn't add anything interesting - there's no better technique than copying "half" your solution from an established quine. In that case there's two possibilities: F itself is boring or has been done before, in which case the challenge doesn't add anything interesting at all, or F itself is actually an interesting string processing task, in which case it should be posted as such, without wrapping it in a quine framework.

That doesn't mean all generalised quines are bad. Some do admit interesting shortcuts than don't require the full standard quine to be reproduced. Before posting a generalised quine challenge, you should ask yourself whether that's possible or not.

## Patching out approaches

On your challenge, someone posts a solution that's cheap and not at all what you intended. You have the option to change the rules, but do so very carefully as there's many pitfalls.

Don't:

• Criticize the answerer. They were just trying their best towards the winning criteria as stated.
• Simply encourage avoiding this approach. If it's not against the rules, it's fair game.
• Require using a certain method. A lot of the fun is in finding novel, unexpected approaches.
• Attack the symptoms. If the exploit suggests a broader problem with the challenge, fix that problem. Also, it's often hard to specify exactly what is off-limits.
• Make rash changes. These might introduce other problems. It's hard to think clearly under time pressure. If you're unsure, ask that the challenge be closed, post it in the Sandbox to get feedback, and reopen it when it's in good shape.
• Post changes solely in the comments.

# The prime numbers

We have 226 questions about prime numbers at the time of writing. Almost every single one of these involves some adaptation of the “canonical prime checking code” or the “canonical prime generator”. Primes are notorious for not really conforming to any mathematical symmetry in a lot of ways, so very often, there is no way around a “brute force” approach: you just do exactly what the challenge tells you to do with your bit of primality logic; and that bit of logic is usually the “ungolfable” part.

I beg you — I plead you — consider any other integer sequence for once. Spice things up! Whatever challenge you were going to write called “Onerous primes” or “Highfalutin primes” or “Supercalifragilistic primes”: ask it about the odious numbers or the lucky numbers or the Ulam numbers instead. Or heck, ask it about the squares!

There isn’t anything wrong with the prime numbers, but there isn’t anything particularly nice about them either. In fact, primality is quite an “ugly” condition, in the sense that there are no formulas for the next prime after $p$, or the number of primes below $n$, etc. This is what makes them interesting to study from a mathematical point of view, but they’re a nuisance to compute things about, because you can’t apply very many tricks, and this might actually limit the set of approaches golfers could reasonably take to solve your challenge with.

Please, give this some thought before you post the 227th, or 228th, or 229th question asking us to find “the n-th prime number such that (blah blah blah).”

• Shouldn't there at least be a prime number of such challenges? – Anush Jul 6 '18 at 7:48
• I disagree that there aren't many tricks to use. As someone who has done quite a bit of Project Euler I can tell you there are a lot of interesting shortcuts (many involving totient function, Möbius inversion, prime-counting algorithm) – qwr Aug 18 '18 at 16:31
• There are many shortcuts in speed, but rarely ever in code size. – Lynn Aug 18 '18 at 19:35

## Popularity Contests

Popularity contests, while allowed, are heavily disfavored by site culture. You're likely better off reworking your challenge to use an objective winning criterion that isn't votes, such as code golf or fastest code. That is, unless you're an experienced user who knows when to disregard such advice.

Of the last 15 pop-cons as of writing, 11 have been closed. If you're a newish user, chances are yours will be too. I'm not claiming bias, but that one needs a very good feel of the community to navigate the minefield of pop-cons, and that takes time to develop.

Most pop cons are closed as "too broad" for having "too many possible answers". Saying "do [simple task] in a creative way" won't cut it -- pop-cons needs an exact, specific task that is hard to solve, and whose solutions can be further evaluated by human criteria. Even with this, I'd advise new-ish users to avoid pop-cons altogether, since it's easy to mess up and might be closed or downvoted anyway.

• It should be noted that part of the reason they are received poorly is because of the "fastest gun in the west" effect. – mbomb007 Feb 28 '16 at 5:21
• To emphasize the point here for future viewers: Pop-cons should not be unconditionally avoided. It's just that new users will likely have better luck writing other types of challenges (namely code-golf). – Calvin's Hobbies Apr 29 '16 at 20:06
• -1 popularity contests are not inherently bad, but rather the challenge in and of itself. Many of the top questions on this sight are popularity contests. SImply the question spec should be tight, and allow variation and objective human judging – Rohan Jhunjhunwala Jul 23 '16 at 19:52
• They're allowed, but will be closed, convert them to golf... Or at least that's been my experience. – AJFaraday Aug 23 '18 at 9:40

Links pointing to relevant information that aid in understanding the challenge are good. Links that point to information that is common knowledge or irrelevant to the challenge are not. They distract the reader, and can lead to dozens of tabs being opened at once.

If a challenge is posted with useless links, edit them out. It doesn't actively harm the challenge in any way.

• +1 Thank you for the informative duck link. It really helped shine a light on what you're saying here, and the answer would be less convincing without it. – Geobits Jun 9 '16 at 15:08
• This sounds sensible to me. Perhaps also link here so it's not seen as just an arbitrary decision. – trichoplax Jun 10 '16 at 15:23
• I'm guessing the downvote is not to say "We should have more pointless links" but just to say "The line between pointful and pointless is subjective". Although there is always going to be disagreement on what counts as relevant, I still think avoiding pointless links is a good thing to keep in mind. – trichoplax Jun 10 '16 at 15:40
• Can’t belive I just got rickrolled... – Andreï Kostyrka Sep 5 '16 at 4:28
• @AndreïKostyrka That's why I've personally opened the link before. It shows up as already viewed >:D – moonheart08 Apr 20 '18 at 13:19

# Formulating the challenge as something and then including a twist that completely changes the task

Don't hide information from the reader. Don't enounce the challenge as something that later on it will turn out not to be.

To ellaborate, consider the following formulation:

Do < task A, usually easy, described in one or several paragraphs >

But wait! / Not so fast! / There is a twist...

You can only use ... < clarification that significantly changes the task into B >

This is misleading. The initial description says something, then the challenge turns out to be something else.

A typical use of this is to present the task as deceivingly simple (A), and then clarify that it is actually much harder (B). The writer's intention may be to make the challenge more "fun" or to add some "spice". But it's better to be honest from the start and include all the relevant information at the beginning, not leaving important information for later. The "twist" may be fun for the writer; but the reader may feel they have been deceived. What was initially told is not really true.

I'm not saying that clarifications are not allowed. As the challenge text progresses, the initial description can (should) be refined. But don't include a significant change in the middle of the text. If a "clarification" contradicts or significantly alters the challenge, it should be put right at the outset (it's not a clarification, it's one of the main ideas).

The order of ideas in the text should be chosen to minimize surprise to the reader. Start with the main ideas, then include refinements that modify or qualify any previous statements as little as possible.

• I disagree. I do this quite often as part of my writing style. – Stephen Leppik Apr 12 '18 at 16:29
• +1. This is the first rule of technical writing. While misdirection is acceptable for creative writing, it's not acceptable for technical writing. We aim for our challenges to be as easy to understand as possible. – Mego Apr 12 '18 at 17:39
• Whether or not twists are acceptable is largely determined by how it is explained. If the twist is explained early on, or if explaining the twist depends on everything above it, then that's acceptable. You want to get to the core (the twist) of the challenge as fast as possible. Don't waste reader's time. – Nathan Merrill Apr 13 '18 at 16:56
• @NathanMerrill I agree with you, except on terminology. If the twist is explained early on... then it's not a twist. By twist I mean something that contradicts or significantly changes the previous information (i.e. as per a definition like this) – Luis Mendo Apr 13 '18 at 17:13
• I agree. However, given your short example above, it appears like your twist appears as the second sentence, so I want to make it clear that that's perfectly acceptable. – Nathan Merrill Apr 13 '18 at 17:22
• @NathanMerrill The first item in the example (as well as the third) was meant to represent more than a sentence; usually one or several paragraphs. I've clarified that – Luis Mendo Apr 13 '18 at 18:23
• This can be avoided by giving a description of the pre-"twist" challenge but not phrasing it as if this is actually the problem to be solved. example (one of my challenges) – Esolanging Fruit Apr 16 '18 at 20:33
• @EsolangingFruit Yes, that's a good way to word it, because it doesn't try to fool the reader – Luis Mendo Apr 16 '18 at 20:39

# Narrow references

Beware of challenge topics that tickle your fancy, but others won't appreciate.

• An in-joke among friends
• A reference to a movie you like
• A task from actual code you wrote
• A small part of an existing algorithm without context

These can be sources of inspiration, but make sure the challenge itself is interesting to do. Imagine someone without the background: would they appreciate the topic once you explain it? Often it's best to just pose the task, then unobtrusively explain the background separately.

• Might be an idea to s/movie/creative work/ – user77406 Mar 1 at 5:41
• So I can't create a question about that one time that guy at the office emailed me that his family was "craving pumpkins this Saturday"? – Redwolf Programs May 5 at 2:13

## Using old challenges as a model

Just because an old challenge did something doesn't mean yours should too. Many old challenges wouldn't pass muster nowadays. Don't be surprised if your challenge is downvoted or closed even though a comparable old one is at +87.

Site standards have risen a lot over the years. Challenges are expected to be objective, precisely specified, and comprehensively covered by test cases. Moreover, some formerly-popular types of challenges have fallen out of community favor: pop cons about creativity, pure programming puzzles, code golfs with bonuses, Do X without Y, etc.

For a rough guide, as of this writing (Oct 2016), I'd say 2013 and before is a different world, and things from 2014-2015 are iffy as references.

Also, beware in general of "but this challenge did it!" regardless of the time. It may not have been a good idea for that challenge, and even if it was, it may not be a good idea for yours.

## Tacked-on fixes

If you clarify or modify your challenge after posting, don't just tack on a note to the end or to a list of bullet points. Edit in the change.

It's confusing when the body of the challenge says one thing but a later part says another. Of course, it would be ideal not to change the challenge after posting, but if you have to edit, do so cleanly.

For example, if you said to "take two integers as input", but, when asked, clarify that you had positive integers in mind, don't just add a note at the end that says

• You may assume the input integers will be positive.

Instead, change the line to "take two positive integers as input".

If you have to make a substantial change, or there's already answers, it might be worth making a note about your change at the end to make people aware of it.

# Bonuses in code-golf

This might be controversial - Personally I'm not sure which way to go on this one.

In most cases, code-golf scoring is simple - shortest answer in bytes wins.

However some questions offer bonuses to reduce scores if certain optional requirements are met. This necessarily complicates scoring to the possible detriment of the question. Often this ends up with either negative-score answers or non-integer score answers. Of course comparisons of such scores are not rocket science, but they do leave a greater margin of error, as well as lower answer readability due to the extra cruft of score calculations.

It can be argued that a well-written challenge can stand alone without the need for bonuses.

On the other hand, bonuses can make an question more accessible, both by allowing a larger range of participant ability, and possibly allowing answers in a more diverse set of languages.

Also bonuses add an extra dimension to the golfing process, which can make a problem more interesting to work on.

• I think the complication to the scoring is not even the bad part of bonuses. The problem is that you often have to solve the problem with all possible subsets of bonuses implemented to find out which one actually gives you the lowest score. – Martin Ender Jan 14 '16 at 17:41
• @MartinBüttner Yes - that's the extra dimension to the golfing process I'm talking about - a good or bad thing, depending on your perspective – Digital Trauma Jan 14 '16 at 17:58
• Yeah, I figured, but I wanted to add that it's not necessarily an advantage. ;) – Martin Ender Jan 14 '16 at 17:59
• If linking to this, it's worth also considering the stronger version, which also currently has more community support – trichoplax Jun 30 at 9:48

# Input Validation

This is a subset of adding special cases for completeness that seems to come up pretty often. From a given set of possible inputs, a solution shouldn't have to sort out inputs that don't apply to the challenge.

For example, imagine a challenge that asks to find the index of a value in a sequence. For values that are not in the sequence, or values that don't apply (strings or floats in an integer sequence):

### Don't:

• Require printing "Invalid input!" or something similar
• Ask for a specific return value, such as -1, 0 or the empty string.

### Do:

• Say that all inputs are assumed to be valid.
• Allow undefined behaviour
• (Less preferably) the return value should be any value not normally returned by valid inputs. This includes errors, undefined values, etc.

It's not as fun to have to program in a bunch of edge cases testing whether an element is actually in the array or a number is non-negative. Plus having to allocate precious bytes for a string literal "Invalid input" in the middle of your code is unsatisfying and unnecessary.

If your challenge becomes trivial without input validation, then consider making it a instead.

• Also note that having to hardcode a string like "Invalid input" gives even more advantage to golfing languages with compressed strings. – Esolanging Fruit Jul 6 '18 at 16:11
• The check for allowed input can be golfed too... I not see where is the problem... One has always make in count each routine can fail for bad input in every place – RosLuP Mar 13 at 19:26
• The issue is that it turns one challenge into two tasks. If a particular format is interesting to validate, then maybe it deserves to have its own challenge so a clever solution for using the input isn't held back by having to validate it and a clever solution for validating it isn't held back by having to use it. Imagine if solutons here had to ensure that each coin is a quarter, dime, nickel or penny--there'd barely be a use for the ninth-character hack at all! – Unrelated String Mar 20 at 2:16