In light of the controversy sparked by this question, I think we need to create a general "rule of thumb" policy on how objective a needs to be in order to be on-topic.

The main issue, as I see it, is that people are not in agreement over whether an objective specification should be able to determine the validity of an answer independently of the voters. I think we need to figure out some sort of compromise on what exactly makes a specification objective.

  • 3
    \$\begingroup\$ A discussion room for this is available here. (This way we don't clog up the comments.) \$\endgroup\$ Commented Jan 12, 2016 at 19:54

5 Answers 5


The current state is good for nobody

The rules as they apply to pop cons are bizarre. Whether they should be loosened or tightened, either would be better than this strange in-between.

All challenge questions on this site should have:

  • A clear specification of what constitutes a correct submission, so that it is possible to indisputably decide whether an entry is valid or not. Test cases are highly encouraged.

  • An objective primary winning criterion, so that it is possible to indisputably decide which entry should win.

Pop cons get by the second point on a technicality, that votes are an "objective primary winning criterion", when votes are as subjective and arbitrary as things come. This looks like the result of an uneasy past compromise where some people wanted to showcase programming skill and creativity without a competitive format, so they made up the "most votes wins" to satisfy the rule, even though nobody thinks in terms of a winner. (I'm speculating here, I wasn't actually around). This takes too much doublethink. Either the rule should change, or pop cons should change.

Given that votes count for winning, the first rule (indisputably decide valid entry) is really awkward for pop cons. So you can be totally fuzzy about who wins, but the baseline must be indisputable? That blunts the whole point of pop cons, that they pose open-ended tasks.

It's rare for a task that requires human judgment to allow a precise baseline cutoff. Even if some things sure look like the Mona Lisa and other don't, how do you define that line? It does work for tasks that are basically challenges but with a bonus of "who do it most stylishly", but these are barely pop cons. Saying "pop cons are allowed, they just need have objective validity criteria" strikes me like a law that says "owning a gun is allowed, you just need our [hard-to-obtain] safety certification".

So where does that leave us? Either we change the rules to allow for pop cons without needing mental gymnastics, or we stop allowing pop cons.

  • 6
    \$\begingroup\$ I don't remember who posted the first popcon, but I do remember gnibbler, who was then a pro tem mod, saying that he chose not to close it as off-topic, even though it clearly was by the norms of the site at the time, because he was worried that the site seemed to be dying and maybe a new type of challenge would breathe new life into it. \$\endgroup\$ Commented Jan 13, 2016 at 8:29
  • 1
    \$\begingroup\$ @PeterTaylor The first challenge marked as a popularity-contest was (Re)Implementing Tetris (although it wasn't the first "most votes wins" challenge - most closed as off-topic regardless of whether they were interesting or not - and the tag had been proposed years earlier by CJY). I don't ask a lot of questions, and in fact I asked this one specifically to introduce the tag. The question was well-received, and the tag didn't really spark any controversy at all until code-trolling emerged. \$\endgroup\$
    – primo
    Commented Jan 23, 2016 at 7:52

Let's not argue over semantics

Trying to interpret SE policy and the meaning of objectivity here feels like reading tea leaves. These rules were written for a different situation: questions that are actually requests for help, and answers that give information.

Challenges run as competitions with rules and votes are so different that if we stretch terms to fit (question = challenge, answer = code submission, objective = baseline validity criterion), the intended meaning has been lost. While we should try to respect SE's kindly letting us have a weird site and run by their rules, we should look for the goals of the rules in sustaining a good site, not their wording.

Rather, let's look at recent pop-cons and see if they're good or bad for the site, or what separates good and bad ones. Do voters react to them well? Do they attract quality answers, and do the best answers rise to the top? Do they push clever coding? Can their purpose not be served by other challenge types?

  • 2
    \$\begingroup\$ Actually, the "objectivity" close reason was one made by us. \$\endgroup\$ Commented Jan 12, 2016 at 20:26
  • \$\begingroup\$ Perhaps that whole last paragraph can be a good indicator. \$\endgroup\$ Commented Jan 13, 2016 at 0:02
  • 1
    \$\begingroup\$ +1, though I would extend your last paragraph slightly and say let's look at recent questions tagged with both popularity-contest and graphical-output, since this is where the particular controversy over "art contests" lies. I would say that on the whole many of these are good for the site, because many have encouraged the development of new, highly non-trivial algorithms. (That's what I was attempting to do with my question, and I'm sad that we may not get to see if it would have worked.) \$\endgroup\$
    – N. Virgo
    Commented Jan 13, 2016 at 3:03
  • \$\begingroup\$ You should know better than to dissuade a bunch of programmers from arguing over semantics. Arguably (hue hue), that's most of what we do on Meta. \$\endgroup\$
    – user45941
    Commented Jan 13, 2016 at 3:56
  • 4
    \$\begingroup\$ The rules in the help center are specific to PPCG and can be edited by moderators. If the rules are flawed or do not reflect community consensus in their current form, they should be modified, not ignored. \$\endgroup\$
    – Dennis
    Commented Jan 13, 2016 at 4:16
  • 1
    \$\begingroup\$ I'm surprised this is getting mostly upvoted despite being factually incorrect, as has been pointed out. I guess people are for the spirit of it regardless? I think I just have to leave it as is. \$\endgroup\$
    – xnor
    Commented Jan 13, 2016 at 19:13

Which challenges are on topic here?

First and foremost, the same rules apply to all challenges, whether they are popularity contests or not. These rules are outlined in our help center. They were phrased differently in the past, but their essence was the same ever since the help center was created in 2013.

Programming Puzzles & Code Golf is for programming contests and challenges. We welcome questions from beginners and experts alike.

All challenge questions on this site should have:

  1. A clear specification of what constitutes a correct submission, so that it is possible to indisputably decide whether an entry is valid or not. Test cases are highly encouraged.
  2. An objective primary winning criterion, so that it is possible to indisputably decide which entry should win.

Now, let's take the following, theoretical challenge:

Write a program that prints an unpublished poem to STDOUT. Shortest code in byte wins.

Clearly, this challenge has an objective winning criterion (second sentence). Is this challenge on topic? Of course not! Where did it go wrong?

  • There is no objective validity criterion.

    What exactly counts as a poem? If you google the term, you will find definitions full of weasel words, and you'll soon come to the conclusion that indisputably deciding whether any arbitrary collection of words is or not a poem will be impossible.

  • It isn't a programming contest.

    The lion share of the challenge is coming up with the poem. Printing it should be straightforward in most programming languages.

What does that have to do with anything?

Unfortunately, many popularity contests are strikingly similar to the above example. Now, I don't think is because popularity contests are inherently worse than other challenge types, but because off topic challenges are much more likely to be labeled popularity contest.

Yes, popularity contests are judged by the community's votes, and yes, best vote tally is an objective winning criterion. However, that does not excuse popularity contests from complying with the remaining rules.

I'll use Paint the Mona Lisa in 1 KiB of code as an example.

  • There is no objective validity criterion.

    As far as validity criteria go,

    Your task is to reproduce the Mona Lisa in 1024 bytes or less.

    is as vague as it gets. Sure, it's easy to form an individual opinion for each submission, and for very good or very awful submissions, all involved people may agree that they do or do not "reproduce the Mona Lisa", but the same people may have different opinions anything in between.

    In particular, the reference implementation produces this output, which doesn't look at all like the Mona Lisa in my opinion.

    Mona Lisa?

  • It's not a real programming contest.

    There's nothing in the challenge's rules that forbids using an image manipulation program to compress the original image to ~1 KiB and find a golfy way to print it.

    I did exactly that with GIMP, recompressed the image with LZMA and obtained a 1,110 byte Bubblegum solution that prints this image.

    Mona Lisa!

    Creating this theoretical submission required little to no programming skills.

    Fixing this shortcoming could be rather easier, for example, by changing the contest to accept an image as input and producing a compressed version, not longer than 1,024 bytes. This eliminates the possibility of handcrafting an image, and turns an image manipulation contest into a programming contest.

Beyond being on topic

There are easy ways around the validity criterion problem. For example, one could just specify

Print any image. The image that looks the most like the Mona Lisa (decided by vote tally) wins.

creating an objective validity criterion (output is an image). Such a contest should still be closed as too broad, since there are too many possible answers. At the very least, the challenge should have specified a desired resolution and a list of acceptable image file formats. Ideally, it should be more restrictive than that. For example, American Gothic in the palette of Mona Lisa: Rearrange the pixels contains a strict set of rules and even a validation snippet.

Furthermore, good guidelines for voting can also help to narrow down the challenge. Let us examine the Mona Lisa challenge once more:

Voters are expected to judge answers according to the following criteria:

  • Resemblance of the output image to da Vinci's masterpiece, in terms of colour, composition, details etc.
  • The extent to which it captures the famous "enigmatic smile"
  • Novelty and cleverness of the techniques used (answers should clearly describe this)
  • Elegance and quality of the code in general

The first suggestion says a lot without actually saying anything. With the size restriction, answers will have to choose between similar colors and details, and there's no indication which one should be preferred.

The second point highlights the smile. Does that mean the rest of the image is less important?

I'm not sure how what counts as novel or clever in this scenario.

Finally, I think judging the coding style is an absurd idea if the point is producing the best possible image in a limited amount of bytes.

Further suggestions for posting challenges

  • X has been asked before an was well received, so Y will work out as well is usually a bad starting point.

    Many older challenges that met the quality standards in their time would be instantly closed if posted today.

  • Use the sandbox and give it time to work.

    A good rule of thumb is too leave challenge proposals at least one week in the sandbox, to give it as much exposure as possible. Most users do not visit the sandbox every day.

  • Consider several scoring options.

    Popularity contests are ideal for challenges which should be judged by humans, but only for those. Calvin's Hobbies' answer contains an interesting proposal for judging the Mona Lisa challenge, which has two advantages over a popularity contest:

    • While existing image compression utilities and algorithms are designed to please the human eye, finding an existing one that would score well in a code challenge should be significantly more difficult.

      This means participants have to write their own image codecs.

    • It makes the winning criterion objective, which automatically narrows down the challenge.

      All participants immediately now what their submissions should aim for.

Apologies to @Nathaniel for picking his challenge apart, but is the challenge that started the whole discussion.

  • \$\begingroup\$ @Nathaniel Let us continue this conversation in chat. \$\endgroup\$
    – Dennis
    Commented Jan 13, 2016 at 4:51

Objectivity gets mixed up with other aspects

Several different factors affect whether a popularity contest is welcomed here, not just objectivity. Trying to discuss these factors all at once causes confusion, and the debate becomes cluttered. Let's separate out the different potential problems so that objectivity can be analysed in isolation, as that is the part that seems to cause most division of the community.

The separate problems

Too broad

A challenge needs to be specific enough to make it interesting. Where the line is drawn is highly subjective and it is unrealistic to try to set a fixed definition. As a rough guide for popularity contests, if two answers are difficult to compare, the challenge is probably too broad. Voters should be able to judge "Which solves the challenge best?" rather than simply "Which is your favourite?"

Not enough programming

A challenge needs to be a programming challenge to be on topic here. If programming is not required in order to produce a competitive answer then the challenge fails to be a programming challenge.

Too subjective

I see two ways of defining this problem:

  1. The spec does not define algorithmically how to order the answers.

  2. Two different voters will disagree on the order of the answers.

For a code golf, (1) is usually the useful definition. We expect the order of answers to be unambiguous (to the point that a Stack Snippet can determine the order and display a leaderboard).

For a popularity contest, (1) is not suitable. If there were an unambiguous order there would be no need for human judgement, and the challenge would not be a popularity contest. When applying (2), there is no clear line between challenges in which everyone votes differently and challenges in which everyone votes the same. In all challenges voting will be somewhere in between. The more it seems that voting will be arbitrary, the more likely that the challenge is too subjective.

Subjectivity spectrum

It's tempting to try to compose a list of types of popularity contest, and clearly divide those that are on topic from those that are not. For example, imagine using the following list, from most subjective to least subjective:

  • How unexpected/impressive is the output?
  • How aesthetically pleasing is the output?
  • How elegant is the programming approach?

  • How well does the output achieve an objective?

Although it seems reassuring to have a clear dividing line, in practise this is unlikely to help beyond separating out the most obviously off topic challenges. The community-dividing difficult judgements will tend to be below the line - "How well does the output achieve an objective". The difficult decision will be in whether the objective is defined sufficiently objectively.

Confusing validity with ordering

Two unrelated questions seem to have been frequently conflated:

  • What makes an answer objectively valid?
  • What makes an answer objectively better than another?

The second question cannot be answered objectively, otherwise it wouldn't be a popularity contest. The first question should be answered as much as possible - the challenge should give a clear and strict definition of what constitutes an answer. The second question can only be addressed with guidelines on what voters should look for. By definition these will be subjective guidelines.

Confusing "valid answer" with "sufficient effort"

Validity can be objectively defined (even for a popularity contest). However, "sufficient effort" cannot be objectively defined, even for a code golf challenge. A challenge is required to define objectively what constitutes a valid answer. A challenge is not required to define what counts as "sufficient effort".

Generic guidelines are insufficient

Popularity contests are widely varied and each should be assessed individually. Generic posts can offer guidance and rough limits but in many cases there will also be the need for a meta discussion. This will give a much better view of the community's feelings and reasons than just going through the close-reopen cycle a few times.

  • \$\begingroup\$ "A challenge needs to be a programming challenge to be on topic here"— I dispute this. There are several well-received questions that really don't involve any programming at all. \$\endgroup\$ Commented Mar 8, 2018 at 23:27

Despite their more subjective nature, popularity contents are not except from the site's rule of objectivity. All popularity contests still need a concrete, objective way to determine whether a submission is valid and deserving of votes.

"But wait," you say, "couldn't I just make the criterion 'output must be an image'? That's objectively verifiable." In this contrived example, that criterion is indeed objective in some respect, but that's far too broad for this site and any challenge claiming such a criterion should be closed as such.

We occasionally get popularity contests that are more art-focused than programming-focused. As a site with programming right in the name, all challenges here should be centered around programming. Consider Draw an Image as a Voronoi Map as an example. This challenge requires making pretty pictures, but there's more to it than that. Specifically, (quoting directly from the challenge)

Your task is to approximate a given image with such a Voronoi map.

This is a popularity contest, so the answer with the most net votes wins. Voters are encouraged to judge answers by

  • how well the original images and their colours are approximated.
  • how well the algorithm works on different kinds of images.
  • how well the algorithm works for small N.
  • whether the algorithm adaptively clusters points in regions of the image that require more detail.

The challenge to this is constructing the code/algorithm to implement this. The above details give the popularity contest its objective criteria and thus makes it on topic.

  • 3
    \$\begingroup\$ "Output must be an image" is too broad. "Output must be an image that looks like the Mona Lisa" is more narrow. \$\endgroup\$
    – Rainbolt
    Commented Jan 12, 2016 at 19:31
  • \$\begingroup\$ @Rainbolt "Looks like the Mona Lisa" is still quite subjective. \$\endgroup\$
    – Alex A.
    Commented Jan 12, 2016 at 19:32
  • 2
    \$\begingroup\$ I didn't say it wasn't subjective. I said it was more narrow. \$\endgroup\$
    – Rainbolt
    Commented Jan 12, 2016 at 19:33
  • 2
    \$\begingroup\$ I suppose you are implying that only objective rules count when determining broadness, but that would be wrong. The entire spec counts when determining how broad the entire spec is. \$\endgroup\$
    – Rainbolt
    Commented Jan 12, 2016 at 19:33
  • \$\begingroup\$ @Rainbolt That's not what I'm implying. I'm just saying that objective rules are required regardless; broadness is its own issue. \$\endgroup\$
    – Alex A.
    Commented Jan 12, 2016 at 19:38
  • 3
    \$\begingroup\$ I'm failing to see how the above Voronoi map question's specifications as quote-boxed are sufficient, but the Mona Lisa question's specifications are not. What am I missing? \$\endgroup\$ Commented Jan 12, 2016 at 19:38
  • 3
    \$\begingroup\$ @AlexA. Right. Broadness is its own issue, but you framed this specific question as Too Broad using only a tiny portion of the spec. And you replied to my comment about how narrow the spec is with "it's still quite subjective". I'm not the one trying to blur the two issues. I'm trying to separate them. \$\endgroup\$
    – Rainbolt
    Commented Jan 12, 2016 at 19:41
  • 1
    \$\begingroup\$ @AlexA. The Voronoi challenge has requirements like "how well the image is approximated," "how well it works for different images," "how well it works for small N" ... which are also not objective validity criteria. If Alice could think that it works well, and Bob could think that it doesn't work well, then it's not objective. So ... again ... what am I missing? \$\endgroup\$ Commented Jan 12, 2016 at 19:45
  • 1
    \$\begingroup\$ @TimmyD Well, Voronoi is about programming. As Dennis pointed out in a comment on Mona Lisa, programming isn't even required for that challenge. \$\endgroup\$
    – Alex A.
    Commented Jan 12, 2016 at 19:48
  • 1
    \$\begingroup\$ @AlexA. OK, then the Mona Lisa challenge is too trivial (i.e., the naïve approach is the "best"), which is a different issue than what's being discussed here. This meta is for understanding what makes one set of criteria "objective" and another set "subjective." \$\endgroup\$ Commented Jan 12, 2016 at 19:53
  • 1
    \$\begingroup\$ It's awkward to have a competition where the winning criterion is subjective, yet there must be an objective baseline for participation. The point of pop-cons is that they ask you to do a task that is fuzzy and stated in human terms. I think that this halfway point is unsatisfying to both sides. \$\endgroup\$
    – xnor
    Commented Jan 12, 2016 at 20:01
  • 3
    \$\begingroup\$ Without wanting to delve into the semantics of what is meant by objective, I feel that art based popcon's greatly contribute to PPCG. codegolf.stackexchange.com/questions/35569/… is lovely and codegolf.stackexchange.com/questions/11141/… is something that a lot of people have talked about online as an example of amazing things on PPCG. I concede that if every other question was like this we would have a problem, but they aren't, so we don't. \$\endgroup\$
    – user9206
    Commented Jan 12, 2016 at 20:04
  • 2
    \$\begingroup\$ I do not like or understand the idea of "validity criteria" being the important thing here. OK, so every challenge must pass some bar in order to be valid. For my challenge, the bar is set by the length of the code, and I'd appreciate some discussion of why that by itself is not sufficient, when for codegolf.stackexchange.com/questions/35569/… it apparently is. But the bigger issue is that by the definition of popularity-contest, entries are still judged on votes once they pass that bar. [...] \$\endgroup\$
    – N. Virgo
    Commented Jan 13, 2016 at 3:21
  • 2
    \$\begingroup\$ [...] If I added some extra arbitrary restrictions on the output, that would have made the challenge more complicated, but it wouldn't have fundamentally altered its nature. Would that have been enough to avoid it being closed? In practice yes, it probably would, but it makes no logical sense to me. \$\endgroup\$
    – N. Virgo
    Commented Jan 13, 2016 at 3:21
  • 2
    \$\begingroup\$ Also, for this to work as a criterion for whether a question is on topic, you will need to specify some kind of objective criterion for judging whether the "validity" bar is indeed set in an appropriate place. Otherwise other people will fall into the same trap as I did, of putting work into writing a challenge that seems to be in keeping with the rules, only to have it closed. Without an explanation of why, it just doesn't make sense to be told this is "too broad" while these questions are not. \$\endgroup\$
    – N. Virgo
    Commented Jan 13, 2016 at 3:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .