Voting by implementation vs. output

Question

I want to open a discussion about this recent highly-popular question that has been closed and reopened a few times: Animation of Code Golf.

The turbulence of that question and similar past questions is clearly related to the fact that there is disagreement on whether "art contests" can be programming challenges. (See The line between art and programming.)

But Martin Büttner's first comment on the question, and other past comments I've seen, made me think that the main schism is between those who think programming contests should be judged on implementation only, and those who think they should be judged on implementation and output.

This part of the comment brought this to mind:

... In this challenge very few people would look at the implementation and only vote by prettiness which is in no way correlated to a clever implementation. Imagine one solution which throws a 3rd-party syntax highlighting at the GIF and another which uses a very clever edit distance algorithm. Guess which one gets the votes?

The point made is surely true. People will vote for the thing with flashy output, often regardless of implementation. Such might be the nature of many of our closed "art contests". But I want to ask: Is judging on output necessarily bad? (People do it anyway of course.)

When I think "programming contest" I think of lots of people competing in some sort task that requires programming, and has definite winner(s) and losers. Whether the goal is to create beautiful code or beautiful output (or both) is up to the contest author. As long as programming is required and there's a winner, it is a "programming contest".

We rarely get (and quickly remove) questions that don't involve programming, and we always enforce objective winning criteria, so the issue lies in what parts of the program are being judged.

One could quickly argue that "programming contest" obviously implies that the programming itself is the part being judged, but regardless of those semantics, is there really harm in letting "art contests" and the like (especially popular ones) remain open?

This is a site based on entertainment and user interaction after all. I see no harm in allowing users to compete in contests where the output is explicitly part of the judging. If you're worried your coding efforts will be overlooked in such questions then you needn't answer them. And it's not like we're overflowing with questions, we may as well keep what we can.

(P.S. I'm not totally on the pro-art-contest side, I'm playing devils advocate a little here. Mainly my goal is to open up the discussion.)

(P.P.S. I mean no offense to Martin Büttner, your comment just spurred me to voice my say :) )

Answer 1

Output is necessary for judging

I think one of the most important things to learn as a programmer is that we as programmers are not qualified to judge programs. With experience we can learn to spot which of two algorithms will be better without actually running them, but even then there will be times when we are wrong. There will be times when the results surprise us.

Whether professionally or recreationally, the output of a program is a vital part of assessing the program itself.

This doesn't help in where to draw the line, but my point is that wherever we choose to draw that line when voting, the line certainly will not be at the far end where output is discounted altogether.

For a code golf, the output is often required to be identical for all answers, so that the judging is purely on character-efficiency, but the output is still essential to distinguish valid answers from invalid ones.

Choosing voters

Regardless of what we decide here in meta, we can't make people vote in the way that we consider best. People will still be subjective and I don't see that as a bad thing. What we can influence is what type of subjective voters we have, by deciding what type of people to attract to the site.

If we encourage questions that attract people who love puzzles and algorithms and new insights, then we'll get more votes on questions and answers that will attract still more. If we encourage (or even tolerate) questions that lean too far away from being about programming then we'll attract people who will put most of their votes into such questions and answers and flood the site with users who may take the site in a different direction due to the overwhelming number of votes.

So I want to see popularity contests and I want to see judging by output, but I still want such questions to be carefully scrutinised and closed if they are not sufficiently programming based.

Is the implementation the source of the output?

I'm perfectly happy with a question where judging is based purely on the output, provided that the implementation is the reason for that output being appealing or impressive. If the output is incidental to the implementation then I would want the question to be closed.

For example, the animated diff question may end up with many similar answers that are all implemented in different ways. Since the requirement is the output, the way in which it was implemented is not relevant to judging, not even indirectly. As such I would vote to close the question if it was reopened without changes. If there were some restrictions so that achieving the desired output required ingenuity then it would be a puzzle, rather than just programming. I like programming, but it seems that to fit this site a question needs to be more than just setting a programming task - the task needs to be part of a puzzle. Code golf is a specific type of puzzle - can you complete the programming task in fewer characters than your rivals.

Judging questions will always be subjective

I see the comments of the animated diff question contain a lot of argument about whether or not the question is objective, along with pointing out that this alone is not enough. That fact is, and will remain, that the definition of what is on topic for this site is not objective. When voting to close or reopen we use the guidelines but there will always be questions that fall near the border where we disagree about whether the site benefits from their presence, and the guidelines for what is on topic may change over time. There is no perfect definition of a question that fits the site. We should also vote with more in mind than just the guidelines. That's the advantage of having humans voting. We can spot potential problems not foreseen in the guidelines.

For the specific question raised, about animated diffs, I would expect such animations to be of interest to a large number of people in the PPCG community, but I don't see it as a good fit for the site. I would much rather see it modified and reopened than left closed, but if it is reopened with insufficient improvement then I will be voting to close.

Answer 2

I think githubphagocyte did a good job with the generics, but let me also comment on the specific question at hand (Animation of Code Golf). I'd like to point to this quote from the OP about the example given:

Personally, I think this a pretty poor job. It doesn't do syntax highlighting, it doesn't try and move code chunks that get rearranged, it doesn't feel like someone is live editing it, and so on. Lots of room for improvement!

He says it's missing (at least) these three things that would make it "better" in his opinion. Are those things even in the spec? Well, syntax highlighting kinda is, but it's just mentioned as optional. Nothing about moving chunks of code at a time or having a "live feel", which I'm not sure I could begin to put into words.

It's clear that entries are expected to be judged based on things not mentioned in the spec at all, and you should vote for the ones that add the most unmentioned bells and whistles.

I voted to close.

Answer 3

Main point

I'm going to summarize and hopefully clarify what I posted in the comments of Animation of Code Golf. I've already spent way too much time defending this question and I'm not even particularly a fan of it, so I'll try to keep this short.

I had several problems with the second close vote. The question was closed for being off-topic. There are guidelines in place that seek to define what is on-topic. Those guidelines are extremely simple. A question is not off-topic if it has

1. A clear specification of what constitutes a correct submission, so that it is possible to indisputably decide whether an entry is valid or not. Test cases are highly encouraged.
2. An objective primary winning criterion, so that it is possible to indisputably decide which entry should win.

I think #2 was clearly satisfied by the question. Whether or not you like it, "most votes wins" is an objective primary winning criterion. Then the debate here should chiefly be over whether or not the question provided a clear specification of what constituted a correct submission. The question stated plainly "make a program that produces the prettiest animated diffs" and " output will be an animated GIF". That is a broad specification, but I don't find it to be unclear.

For these reasons I voted to re-open.

Ranting

I respect that most of the people here, including (and especially) those who close voted, are all trying to make this site better. That is a good thing. But personally, I think this site would benefit from tolerating a wider range of questions. I'm not saying that we should never close vote. I am saying that I think it happens more often than it needs to and occasionally for the wrong reasons.

I would urge the following to everyone: If you're going to close vote a question, at least leave a comment. If your intention is really to improve the site, then explain why you've close voted and offer suggestions on how the user can improve the question. It's insulting to the hard work of the person who posted the question otherwise.