In light of the controversy sparked by this question, I think we need to create a general "rule of thumb" policy on how objective a popularity-contest needs to be in order to be on-topic.
The main issue, as I see it, is that people are not in agreement over whether an objective specification should be able to determine the validity of an answer independently of the voters. I think we need to figure out some sort of compromise on what exactly makes a popularity-contest specification objective.
The rules as they apply to pop cons are bizarre. Whether they should be loosened or tightened, either would be better than this strange in-between.
All challenge questions on this site should have:
A clear specification of what constitutes a correct submission, so that it is possible to indisputably decide whether an entry is valid or not. Test cases are highly encouraged.
An objective primary winning criterion, so that it is possible to indisputably decide which entry should win.
Pop cons get by the second point on a technicality, that votes are an "objective primary winning criterion", when votes are as subjective and arbitrary as things come. This looks like the result of an uneasy past compromise where some people wanted to showcase programming skill and creativity without a competitive format, so they made up the "most votes wins" to satisfy the rule, even though nobody thinks in terms of a winner. (I'm speculating here, I wasn't actually around). This takes too much doublethink. Either the rule should change, or pop cons should change.
Given that votes count for winning, the first rule (indisputably decide valid entry) is really awkward for pop cons. So you can be totally fuzzy about who wins, but the baseline must be indisputable? That blunts the whole point of pop cons, that they pose open-ended tasks.
It's rare for a task that requires human judgment to allow a precise baseline cutoff. Even if some things sure look like the Mona Lisa and other don't, how do you define that line? It does work for tasks that are basically challenges but with a bonus of "who do it most stylishly", but these are barely pop cons. Saying "pop cons are allowed, they just need have objective validity criteria" strikes me like a law that says "owning a gun is allowed, you just need our [hard-to-obtain] safety certification".
So where does that leave us? Either we change the rules to allow for pop cons without needing mental gymnastics, or we stop allowing pop cons.
Trying to interpret SE policy and the meaning of objectivity here feels like reading tea leaves. These rules were written for a different situation: questions that are actually requests for help, and answers that give information.
Challenges run as competitions with rules and votes are so different that if we stretch terms to fit (question = challenge, answer = code submission, objective = baseline validity criterion), the intended meaning has been lost. While we should try to respect SE's kindly letting us have a weird site and run by their rules, we should look for the goals of the rules in sustaining a good site, not their wording.
Rather, let's look at recent pop-cons and see if they're good or bad for the site, or what separates good and bad ones. Do voters react to them well? Do they attract quality answers, and do the best answers rise to the top? Do they push clever coding? Can their purpose not be served by other challenge types?
First and foremost, the same rules apply to all challenges, whether they are popularity contests or not. These rules are outlined in our help center. They were phrased differently in the past, but their essence was the same ever since the help center was created in 2013.
Programming Puzzles & Code Golf is for programming contests and challenges. We welcome questions from beginners and experts alike.
All challenge questions on this site should have:
- A clear specification of what constitutes a correct submission, so that it is possible to indisputably decide whether an entry is valid or not. Test cases are highly encouraged.
- An objective primary winning criterion, so that it is possible to indisputably decide which entry should win.
Now, let's take the following, theoretical challenge:
Write a program that prints an unpublished poem to STDOUT. Shortest code in byte wins.
Clearly, this challenge has an objective winning criterion (second sentence). Is this challenge on topic? Of course not! Where did it go wrong?
There is no objective validity criterion.
What exactly counts as a poem? If you google the term, you will find definitions full of weasel words, and you'll soon come to the conclusion that indisputably deciding whether any arbitrary collection of words is or not a poem will be impossible.
It isn't a programming contest.
The lion share of the challenge is coming up with the poem. Printing it should be straightforward in most programming languages.
Unfortunately, many popularity contests are strikingly similar to the above example. Now, I don't think is because popularity contests are inherently worse than other challenge types, but because off topic challenges are much more likely to be labeled popularity contest.
Yes, popularity contests are judged by the community's votes, and yes, best vote tally is an objective winning criterion. However, that does not excuse popularity contests from complying with the remaining rules.
I'll use Paint the Mona Lisa in 1 KiB of code‡ as an example.
There is no objective validity criterion.
As far as validity criteria go,
Your task is to reproduce the Mona Lisa in 1024 bytes or less.
is as vague as it gets. Sure, it's easy to form an individual opinion for each submission, and for very good or very awful submissions, all involved people may agree that they do or do not "reproduce the Mona Lisa", but the same people may have different opinions anything in between.
In particular, the reference implementation produces this output, which doesn't look at all like the Mona Lisa in my opinion.
It's not a real programming contest.
There's nothing in the challenge's rules that forbids using an image manipulation program to compress the original image to ~1 KiB and find a golfy way to print it.
I did exactly that with GIMP, recompressed the image with LZMA and obtained a 1,110 byte Bubblegum solution that prints this image.
Creating this theoretical submission required little to no programming skills.
Fixing this shortcoming could be rather easier, for example, by changing the contest to accept an image as input and producing a compressed version, not longer than 1,024 bytes. This eliminates the possibility of handcrafting an image, and turns an image manipulation contest into a programming contest.
There are easy ways around the validity criterion problem. For example, one could just specify
Print any image. The image that looks the most like the Mona Lisa (decided by vote tally) wins.
creating an objective validity criterion (output is an image). Such a contest should still be closed as too broad, since there are too many possible answers. At the very least, the challenge should have specified a desired resolution and a list of acceptable image file formats. Ideally, it should be more restrictive than that. For example, American Gothic in the palette of Mona Lisa: Rearrange the pixels contains a strict set of rules and even a validation snippet.
Furthermore, good guidelines for voting can also help to narrow down the challenge. Let us examine the Mona Lisa challenge once more:
Voters are expected to judge answers according to the following criteria:
- Resemblance of the output image to da Vinci's masterpiece, in terms of colour, composition, details etc.
- The extent to which it captures the famous "enigmatic smile"
- Novelty and cleverness of the techniques used (answers should clearly describe this)
- Elegance and quality of the code in general
The first suggestion says a lot without actually saying anything. With the size restriction, answers will have to choose between similar colors and details, and there's no indication which one should be preferred.
The second point highlights the smile. Does that mean the rest of the image is less important?
I'm not sure how what counts as novel or clever in this scenario.
Finally, I think judging the coding style is an absurd idea if the point is producing the best possible image in a limited amount of bytes.
X has been asked before an was well received, so Y will work out as well is usually a bad starting point.
Many older challenges that met the quality standards in their time would be instantly closed if posted today.
Use the sandbox and give it time to work.
A good rule of thumb is too leave challenge proposals at least one week in the sandbox, to give it as much exposure as possible. Most users do not visit the sandbox every day.
Consider several scoring options.
Popularity contests are ideal for challenges which should be judged by humans, but only for those. Calvin's Hobbies' answer contains an interesting proposal for judging the Mona Lisa challenge, which has two advantages over a popularity contest:
While existing image compression utilities and algorithms are designed to please the human eye, finding an existing one that would score well in a code challenge should be significantly more difficult.
This means participants have to write their own image codecs.
It makes the winning criterion objective, which automatically narrows down the challenge.
All participants immediately now what their submissions should aim for.
‡ Apologies to @Nathaniel for picking his challenge apart, but is the challenge that started the whole discussion.
Several different factors affect whether a popularity contest is welcomed here, not just objectivity. Trying to discuss these factors all at once causes confusion, and the debate becomes cluttered. Let's separate out the different potential problems so that objectivity can be analysed in isolation, as that is the part that seems to cause most division of the community.
A challenge needs to be specific enough to make it interesting. Where the line is drawn is highly subjective and it is unrealistic to try to set a fixed definition. As a rough guide for popularity contests, if two answers are difficult to compare, the challenge is probably too broad. Voters should be able to judge "Which solves the challenge best?" rather than simply "Which is your favourite?"
A challenge needs to be a programming challenge to be on topic here. If programming is not required in order to produce a competitive answer then the challenge fails to be a programming challenge.
I see two ways of defining this problem:
The spec does not define algorithmically how to order the answers.
Two different voters will disagree on the order of the answers.
For a code golf, (1) is usually the useful definition. We expect the order of answers to be unambiguous (to the point that a Stack Snippet can determine the order and display a leaderboard).
For a popularity contest, (1) is not suitable. If there were an unambiguous order there would be no need for human judgement, and the challenge would not be a popularity contest. When applying (2), there is no clear line between challenges in which everyone votes differently and challenges in which everyone votes the same. In all challenges voting will be somewhere in between. The more it seems that voting will be arbitrary, the more likely that the challenge is too subjective.
It's tempting to try to compose a list of types of popularity contest, and clearly divide those that are on topic from those that are not. For example, imagine using the following list, from most subjective to least subjective:
Although it seems reassuring to have a clear dividing line, in practise this is unlikely to help beyond separating out the most obviously off topic challenges. The community-dividing difficult judgements will tend to be below the line - "How well does the output achieve an objective". The difficult decision will be in whether the objective is defined sufficiently objectively.
Two unrelated questions seem to have been frequently conflated:
The second question cannot be answered objectively, otherwise it wouldn't be a popularity contest. The first question should be answered as much as possible - the challenge should give a clear and strict definition of what constitutes an answer. The second question can only be addressed with guidelines on what voters should look for. By definition these will be subjective guidelines.
Validity can be objectively defined (even for a popularity contest). However, "sufficient effort" cannot be objectively defined, even for a code golf challenge. A challenge is required to define objectively what constitutes a valid answer. A challenge is not required to define what counts as "sufficient effort".
Popularity contests are widely varied and each should be assessed individually. Generic posts can offer guidance and rough limits but in many cases there will also be the need for a specific-question meta discussion. This will give a much better view of the community's feelings and reasons than just going through the close-reopen cycle a few times.
Despite their more subjective nature, popularity contents are not except from the site's rule of objectivity. All popularity contests still need a concrete, objective way to determine whether a submission is valid and deserving of votes.
"But wait," you say, "couldn't I just make the criterion 'output must be an image'? That's objectively verifiable." In this contrived example, that criterion is indeed objective in some respect, but that's far too broad for this site and any challenge claiming such a criterion should be closed as such.
We occasionally get popularity contests that are more art-focused than programming-focused. As a site with programming right in the name, all challenges here should be centered around programming. Consider Draw an Image as a Voronoi Map as an example. This challenge requires making pretty pictures, but there's more to it than that. Specifically, (quoting directly from the challenge)
Your task is to approximate a given image with such a Voronoi map.
This is a popularity contest, so the answer with the most net votes wins. Voters are encouraged to judge answers by
- how well the original images and their colours are approximated.
- how well the algorithm works on different kinds of images.
- how well the algorithm works for small N.
- whether the algorithm adaptively clusters points in regions of the image that require more detail.
The challenge to this is constructing the code/algorithm to implement this. The above details give the popularity contest its objective criteria and thus makes it on topic.
