What makes winning criteria "objective"?

Question

The FAQ says (in part)

All questions on this site, whether a programming puzzle or a code golf, should have [... an] objective primary winning criterion, so that it is possible to indisputably decide which entry should win.

That was the criterion I used when closing "Obfuscation of $ dmesg | tail | base64 > ~/outfile" just the other day.

However, there have been complaints that the site is mostly code-golf in part because of the "objective" criteria. Now I see SHiNKiROU's effort Fake error message that comes with a serious and fairly comprehensive effort to put a firm grading scale under it.

Other maybe subjective things floating around include

• Peter Of The Corn's recent Write Buggy Code
• Several "speed" contest, which seem slighlty troubling to me because I immediately think "Speed measured by whom on what hardware? And what if I write in a language they don't have?"
• FUZxxl's Implement a sorting algorithm with no change which seems to have no particular metric. (Which I not only left unclosed, but submitted and entry to. Oh! Fickle me!)
• The very well received What can you do in a 4k data URI?
• And others.

The question

What meaning shall we use for "objective"? And why?

Please help me know what I'm supposed to be doing here. And help the site get going too.

---

See also A Modest Proposal: the [popularity-contest] tag.

Answer 1

I guess one thing that immediately comes to mind:

Could two different humans being judge the contest by the specified criteria and get close to the same results?

If not, then the criteria is probably too subjective..

Answer 2

I do not propose a solution to this problem of how to handle subjective questions, but I would like to add another dimension to the discussion.

As much as we like getting that coveted green arrow signifying our prowess at some task in some language, I don't think it should be a necessary component of every task posted.

Programming challenges and puzzles are, by their nature, supposed to be fun and entertaining to develop and share. Does every task require an accepted answer?

I realize we are working within the SE Q&A framework here, which uses percentage-of-questions-answered as a metric for determining the health of a SE, so maybe we are bound to accept an answer on all questions.

Even still, my main point is that the majority of the pleasure and satisfaction we derive from these puzzles is in the brainstorming, implementation, tweaking of the solution, and then the sharing and discussion (and upvotes!) that occur after. A green checkmark makes all that better, to be sure, but is it necessary for every task?

Edit: I'm not advocating leaving every question open, much to the contrary some questions should be closed for vagueness, ill-defined goals, poorly written, etc. The dmesg obfuscation task comes to mind here.

Answer 3

There's a simple test:

Can I measure how my answer fares against the existing answers before I submit it?

Answer 4

Subjetivity is a hard subject, and it is a spectrum rather than a clear cut line between what is objective or not. This can certainly make it harder to decide what is appropriate for a programming puzzle here and what is not.

Common sense is usually what arbitrates sort of thing. On the one hand, there are cheeky answers that technically fulfill the criteria of the question but are obviously not intended, but on the other hand, there are questions (like the one you used as an example) which have very unclear and subjective criteria. If an answer doesn't meet all of the criteria, allowances will often be made and nobody will object if the language is inherently difficult to do certain things in. All sorts little things just have no hard rules that will apply in all cases.

In light of that, I think people should use their common sense in deciding if they should close a question or not. Think as a person trying to answer the question: are you confident that you understand the requirements of the problem that you could make an attempt at creating the program, and, if you were successful, that the asker will accept it as a legitimate answer? If you saw another answer, would you be able to assess it in comparison to your own and know which one would be accepted? If so, I think the problem has few problems with regard to subjective criteria. If not, if it is a little problem, you can comment and ask for clarification, or if it is a rather critical problem, you can vote to close.

If all of you people who, perhpas unlike me, have common sense, think that my Buggy Code question is unclear or too subjective to be able to ascertain which answers are "better", by all means, close it.

Answer 5

only about the Speed contests:

With proper preparation, a speed contest can contain a measurement like this:

Write a XYfooSort function, which sorts some Collection (Array, Buffer, List, ...) in place, using a swap(a,b) function. Count the number of swap-calls, and don't replace them. Print the number of calls in the end.

This way, solutions in different languages can be objectively be compared.

---

Different approach, which would allow to measure the real speed, would be, that the second solution has to compare itself with first, the 3rd can choose to compare with one of them, so that you would get a tree of solutions, with factors, how to translate the numbers. Of course this must not be reliable for different C-compilers on different plattforms, but for an rough estimate it might be a useful approach.

To detect cheaters, you would have to name your system, if your measurements seem unbelievable to others.

Example:

  User Lang Time 
   a    a    4
   b    a    6
   b    b    5 
   c    a    3
   c    c    6
   d    c    6
   d    d    5
   e    d   10
   e    e    8

User a starts with Lang a in 4s. User bs b-solution is slower, but his a measurement is even more, so it might be his processor - a/b = 6/5, now b is the fastest solution. On Cs machine, the a-code is even faster, but his c-Code is poor: c/b=12/5. D compares his d-Code to c, which is faster, but not enough: d/c=5/6, therefore d/b=5*12/6*5=60/30=2/1. Es code is again faster, 4/5e=d, but 8/5 of b, so b would be the winner here.

Answer 6

The simplest grading method is to find the post with most upvotes.

One factor that affects subjectivity is the language design, for example, the "what is considered a change in variable" problem arises when you are dealing with Haskell and Prolog. Also, all RPN languages can be disqualified for manipulating the stack, which changes variable.

For the "most creative solution", common sense only weed out the obvious "dull" answers.

I thunk the most subjective questions will involve visual art.