I want to open a discussion about this recent highly-popular question that has been closed and reopened a few times: Animation of Code Golf.

The turbulence of that question and similar past questions is clearly related to the fact that there is disagreement on whether "art contests" can be programming challenges. (See The line between art and programming.)

But Martin Büttner's first comment on the question, and other past comments I've seen, made me think that the main schism is between those who think programming contests should be judged on implementation only, and those who think they should be judged on implementation and output.

This part of the comment brought this to mind:

... In this challenge very few people would look at the implementation and only vote by prettiness which is in no way correlated to a clever implementation. Imagine one solution which throws a 3rd-party syntax highlighting at the GIF and another which uses a very clever edit distance algorithm. Guess which one gets the votes?

The point made is surely true. People will vote for the thing with flashy output, often regardless of implementation. Such might be the nature of many of our closed "art contests". But I want to ask: Is judging on output necessarily bad? (People do it anyway of course.)

When I think "programming contest" I think of lots of people competing in some sort task that requires programming, and has definite winner(s) and losers. Whether the goal is to create beautiful code or beautiful output (or both) is up to the contest author. As long as programming is required and there's a winner, it is a "programming contest".

We rarely get (and quickly remove) questions that don't involve programming, and we always enforce objective winning criteria, so the issue lies in what parts of the program are being judged.

One could quickly argue that "programming contest" obviously implies that the programming itself is the part being judged, but regardless of those semantics, is there really harm in letting "art contests" and the like (especially popular ones) remain open?

This is a site based on entertainment and user interaction after all. I see no harm in allowing users to compete in contests where the output is explicitly part of the judging. If you're worried your coding efforts will be overlooked in such questions then you needn't answer them. And it's not like we're overflowing with questions, we may as well keep what we can.

(P.S. I'm not totally on the pro-art-contest side, I'm playing devils advocate a little here. Mainly my goal is to open up the discussion.)

(P.P.S. I mean no offense to Martin Büttner, your comment just spurred me to voice my say :) )

  • 3
    \$\begingroup\$ "And it's not like we're overflowing with questions, we may as well keep what we can." -- I believe I'm playing the devil's advocate here, but lack of quantity doesn't mean we should keep bad questions around. \$\endgroup\$ Commented Sep 16, 2014 at 9:55
  • 1
    \$\begingroup\$ @JanDvorak Agreed, but 20 votes does not indicate a bad question. \$\endgroup\$ Commented Sep 16, 2014 at 9:56
  • 4
    \$\begingroup\$ I don't mind judging on "implementation and output". But this question in particular (at least in its original form) would likely have been judged solely on output. Similarly, I don't quite agree with your definition of a programming contest as "As long as programming is required and there's a winner". Imagine a graphical-output contest, where there is only one possible algorithm to produce the results, and the actual graphics are determined by parameters. Programming is involved, but everyone has to implement the same thing and the winner is just the person who finds the nicest parameters. \$\endgroup\$ Commented Sep 16, 2014 at 10:02
  • 6
    \$\begingroup\$ Also, I don't think the popularity of a question should decide whether it's on topic or not, or we'd still have code trolling around. I don't want to repeat my entire argument from back then, but if you don't mind browsing through this wall of text, there's a section "But they're popular!" addressing this: meta.codegolf.stackexchange.com/a/1517/8478. Basically, many of the most popular questions on SO are also closed/locked. SE has a quality standard to maintain, so doing the popular thing might not always be the best way to do things. \$\endgroup\$ Commented Sep 16, 2014 at 10:05
  • \$\begingroup\$ (I'll probably turn those two comments into an answer after lunch.) \$\endgroup\$ Commented Sep 16, 2014 at 10:07
  • 2
    \$\begingroup\$ @MartinBüttner I know that bowing to whim of sensationalism and popularity is not an ideal answer, and makes me uneasy sometimes. But it just seems silly to me that 20+ people can't enjoy creating something together on what I deem (perhaps incorrectly) ought to be an informal and entertainment based environment. \$\endgroup\$ Commented Sep 16, 2014 at 10:15
  • 1
    \$\begingroup\$ This discussion inspired me to make a proposal for popularity contests to specify what criteria make a good answer: meta.codegolf.stackexchange.com/q/2212/20260 \$\endgroup\$
    – xnor
    Commented Sep 18, 2014 at 21:39

3 Answers 3


Output is necessary for judging

I think one of the most important things to learn as a programmer is that we as programmers are not qualified to judge programs. With experience we can learn to spot which of two algorithms will be better without actually running them, but even then there will be times when we are wrong. There will be times when the results surprise us.

Whether professionally or recreationally, the output of a program is a vital part of assessing the program itself.

This doesn't help in where to draw the line, but my point is that wherever we choose to draw that line when voting, the line certainly will not be at the far end where output is discounted altogether.

For a code golf, the output is often required to be identical for all answers, so that the judging is purely on character-efficiency, but the output is still essential to distinguish valid answers from invalid ones.

Choosing voters

Regardless of what we decide here in meta, we can't make people vote in the way that we consider best. People will still be subjective and I don't see that as a bad thing. What we can influence is what type of subjective voters we have, by deciding what type of people to attract to the site.

If we encourage questions that attract people who love puzzles and algorithms and new insights, then we'll get more votes on questions and answers that will attract still more. If we encourage (or even tolerate) questions that lean too far away from being about programming then we'll attract people who will put most of their votes into such questions and answers and flood the site with users who may take the site in a different direction due to the overwhelming number of votes.

So I want to see popularity contests and I want to see judging by output, but I still want such questions to be carefully scrutinised and closed if they are not sufficiently programming based.

Is the implementation the source of the output?

I'm perfectly happy with a question where judging is based purely on the output, provided that the implementation is the reason for that output being appealing or impressive. If the output is incidental to the implementation then I would want the question to be closed.

For example, the animated diff question may end up with many similar answers that are all implemented in different ways. Since the requirement is the output, the way in which it was implemented is not relevant to judging, not even indirectly. As such I would vote to close the question if it was reopened without changes. If there were some restrictions so that achieving the desired output required ingenuity then it would be a puzzle, rather than just programming. I like programming, but it seems that to fit this site a question needs to be more than just setting a programming task - the task needs to be part of a puzzle. Code golf is a specific type of puzzle - can you complete the programming task in fewer characters than your rivals.

Judging questions will always be subjective

I see the comments of the animated diff question contain a lot of argument about whether or not the question is objective, along with pointing out that this alone is not enough. That fact is, and will remain, that the definition of what is on topic for this site is not objective. When voting to close or reopen we use the guidelines but there will always be questions that fall near the border where we disagree about whether the site benefits from their presence, and the guidelines for what is on topic may change over time. There is no perfect definition of a question that fits the site. We should also vote with more in mind than just the guidelines. That's the advantage of having humans voting. We can spot potential problems not foreseen in the guidelines.

For the specific question raised, about animated diffs, I would expect such animations to be of interest to a large number of people in the PPCG community, but I don't see it as a good fit for the site. I would much rather see it modified and reopened than left closed, but if it is reopened with insufficient improvement then I will be voting to close.


I think githubphagocyte did a good job with the generics, but let me also comment on the specific question at hand (Animation of Code Golf). I'd like to point to this quote from the OP about the example given:

Personally, I think this a pretty poor job. It doesn't do syntax highlighting, it doesn't try and move code chunks that get rearranged, it doesn't feel like someone is live editing it, and so on. Lots of room for improvement!

He says it's missing (at least) these three things that would make it "better" in his opinion. Are those things even in the spec? Well, syntax highlighting kinda is, but it's just mentioned as optional. Nothing about moving chunks of code at a time or having a "live feel", which I'm not sure I could begin to put into words.

It's clear that entries are expected to be judged based on things not mentioned in the spec at all, and you should vote for the ones that add the most unmentioned bells and whistles.

I voted to close.


Main point

I'm going to summarize and hopefully clarify what I posted in the comments of Animation of Code Golf. I've already spent way too much time defending this question and I'm not even particularly a fan of it, so I'll try to keep this short.

I had several problems with the second close vote. The question was closed for being off-topic. There are guidelines in place that seek to define what is on-topic. Those guidelines are extremely simple. A question is not off-topic if it has

  1. A clear specification of what constitutes a correct submission, so that it is possible to indisputably decide whether an entry is valid or not. Test cases are highly encouraged.
  2. An objective primary winning criterion, so that it is possible to indisputably decide which entry should win.

I think #2 was clearly satisfied by the question. Whether or not you like it, "most votes wins" is an objective primary winning criterion. Then the debate here should chiefly be over whether or not the question provided a clear specification of what constituted a correct submission. The question stated plainly "make a program that produces the prettiest animated diffs" and " output will be an animated GIF". That is a broad specification, but I don't find it to be unclear.

For these reasons I voted to re-open.


I respect that most of the people here, including (and especially) those who close voted, are all trying to make this site better. That is a good thing. But personally, I think this site would benefit from tolerating a wider range of questions. I'm not saying that we should never close vote. I am saying that I think it happens more often than it needs to and occasionally for the wrong reasons.

I would urge the following to everyone: If you're going to close vote a question, at least leave a comment. If your intention is really to improve the site, then explain why you've close voted and offer suggestions on how the user can improve the question. It's insulting to the hard work of the person who posted the question otherwise.

  • 4
    \$\begingroup\$ It seems that you're slightly misunderstanding the point of closing a question. When a question gets closed it's called "on hold" for a reason. The point is that the question definitely had a nice core idea, but some (fixable) problems with the standards we usually have for challenges. So we close the question in order to salvage it before answers come in. Once answers are there, it's a lot more problematic to amend the specification, because it's likely to disqualify existing answers. Hence: close fast, keep it closed until it's fixed, then reopen. \$\endgroup\$ Commented Sep 17, 2014 at 7:32
  • 4
    \$\begingroup\$ Furthermore, I'm pretty sure several people who close voted left a comment. The others likely just agreed with the comments that were already there. So I don't see how anything of that's insulting, unless the OP can't handle constructive criticism. Finally, yes the debate should be over point #1 you listed above, and it was! Quincunx, Howard and I all said that the problem is that it's not clear what a valid submission must provide. Maybe the chosen close vote reason wasn't the most appropriate, but then "Too Broad" still would have applied and the question should have been closed. \$\endgroup\$ Commented Sep 17, 2014 at 7:35
  • \$\begingroup\$ As it stands, we don't have a close vote reason for "missing an objective validity criterion", and many people choose "missing an objective winning criterion" instead. The former is just as much a problem though, and if the question is that broad it's more likely that the OP didn't actually want to leave it that broad, and then answers get comments like "hm, that's not what I had in mind" - which would then just prove that there is no objective validity criterion. In some cases, the close vote reasons aren't as spot on, but that's why people did leave comments. \$\endgroup\$ Commented Sep 17, 2014 at 7:37
  • 3
    \$\begingroup\$ @MartinBüttner FWIW I wanted it as broad as originally specified. I do not want people to generate the same GIF. After all these edits - which I don't like but made to appease you - the problem is now so tight that new people are commenting that its too tight. My growing dismay is that there is a small cabal on codegolf.SE who want to regulate some definition of what constitutes 'recreational programming'. I know those involved here feel empowered and doing it for our the good of the site and other ways to justify the power trip, but really its just a kind of social poison. \$\endgroup\$
    – Will
    Commented Sep 17, 2014 at 9:22
  • 7
    \$\begingroup\$ @Will SE has certain quality standards and we're just trying to maintain them. If SE didn't, we'd still be doing code golf on SO, "because it's fun and people like it". If we allow every question that's popular, SE will just turn into Reddit. Not every form of recreational programming is appropriate on PPCG, and no one ever said that would be the case. You can't really expect not to hit any resistance if you post an intentionally broad challenge on a site that has a built-in close vote reason for questions that are too broad. \$\endgroup\$ Commented Sep 17, 2014 at 9:49
  • 2
    \$\begingroup\$ Furthermore, there was a single person with a single comment saying that he found one restriction too narrow, to make a "useful" program (and not "people are commenting"). PPCG challenges shouldn't usually be aimed at creating something useful (again, because "useful" is rarely objective). If you want to create some nice diff visualisation tools for PPCG without any guidelines whatsoever, the right place for that is the chat and not a challenge. But for challenges, standards have been established over the past 4 years for a reason. \$\endgroup\$ Commented Sep 17, 2014 at 9:52
  • 5
    \$\begingroup\$ When you state "A question is not off-topic if it has..." you're committing a logical fallacy. A question is not on-topic if it lacks either of those items, but that doesn't mean that every question which has a specification and a winning condition is on-topic. In particular, the question has to be a programming contest rather than an art contest. \$\endgroup\$ Commented Sep 17, 2014 at 10:17
  • 2
    \$\begingroup\$ @Will i'm with you. This site seems to mainly cater for a core group of traditionalists, with moderation often confusing "quality" with "difference of opinion". What I don't understand is why people need to impose their own views of what makes a good code challenge onto other users. If you don't like a type of question, you are not forced to answer it, or even look at it (that's what "tag preferences" are for). imo, the questions that attract 3 character golfscript answers are the least interesting. \$\endgroup\$
    – rdans
    Commented Sep 18, 2014 at 20:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .