17
\$\begingroup\$

Yesterday (probably the day before yesterday in some time zones), I decided to run an experiment on the "Fastest Gun In The West" phenomenon, in which early answers tend to get disproportionately rewarded compared to late answers.

My experiment involved this question, to which I posted two answers. For the first answer, in Underload, I intentionally aimed to post it as fast as possible (I had the answer ready about 10 minutes after I saw the question (I saw it about 10 minutes after it was posted)).

The second answer, in 7, was posted about 2½ hours after the first, and I had been working on it for most of that time.

I think the second answer is more or less objectively better than the first, in pretty much every dimension we care about:

  • The answers both belong to the same language family (concatenative low-level tarpit esolangs). So it's unlikely that the choice of language had a huge influence on the voting. Additionally, we can look at how well my answers in Underload and 7 normally do; Underload normally does fairly well, but 7 normally does better. So the first answer doesn't have an advantage in terms of language choice.

    Additionally, I created both of the languages in question, so there's no potential bias in terms of language creator / people-golfing-in-their-own-language; the two answers are symmetrical this way.

  • The first answer took almost no thought to write. It was a case of "what's a language where I can quickly write a quine, cat program and string printer?". The latter two are builtins in Underload, and it is trivial to modify one of the two standard Underload quines, (aS(:^)S):^ (which I have memorised, as does probably every Underload golfer), to contain a payload (or in this case, some extra useless characters). I made it even easier by picking an interpreter which (non-standardly) treats unknown characters as no-ops. Generally speaking, the recommendations are to not upvote trivial builtin-based solutions (and we should probably recommend not upvoting "trivial modification to a standard program" solutions, too).

    Contrasting with this, the second answer was much more difficult to write. Even printing a string literal has a range of subtleties in 7 (e.g. I spent a while pondering about whether there was a way to shorten it by somehow making use of the 6), and that was the easiest of the three programs to deal with. The quine took a while, because I'd never previously needed to fit a 6 into a quine before (or indeed, created a quine that wasn't just two identical halves that printed each other or where one printed the other twice), and so I spent some time working out a quine template that could make it work. The cat program was a nightmare; 7 doesn't have a builtin for this. Reusable code in 7 is sufficiently context-dependent that it's hard to define what a function even is, meaning that I would need to read standard input explicitly, and 7 can only read a character at a time. So then I needed to construct a loop, in a low-level esolang that's very unintuitive and hard to reason about (probably worse than Brain-Flak). This involved inventing entire new programming techniques that I don't think have been used in 7 before (such as polyglotting the loop body with a literal constant integer).

    Yet, despite all this, I managed to fit the cat and quine programs in using only two extra characters (a 6 for the cat program and an additional 7 for the quine program), costing only ¾ of a byte (which ends up rounding down to 0 because the original program length was slightly over a byte boundary).

    So I consider the 7 answer to be much better-crafted, both in terms of effort (a lot more work is required to write something like that), and in terms of the golfing skill demonstrated by the solution, and it seems likely that people who upvote based on how impressive a program is would prefer the 7 program.

  • The Underload answer is vaguely cheaty: it trivialises the aspect of the problem simply by defining functions that output to standard output (meaning that we can do what we like to the function's return value with the extra characters we're forced to use), so it's basically just exploiting the I/O conventions. (The type of string that the program takes as input isn't even good for anything other than eval or printing.) I needed to use an extra colon in it in order to compensate for the exclamation mark in the string (which would otherwise make the quine take an argument), but apart from that, the entire point behind the question gets ignored. In this sense, I consider it quite similar to answering a question in Lenguage (or indeed a "do X without letters" question in 7), which shouldn't be considered that interesting or funny at this point. Additionally, the answer is interpreter-specific and won't work on most Underload interpreters, due to requiring a specific handling of invalid commands.

    The 7 answer, meanwhile, complies with the spec comprehensively and in a way that fits the spirit of the question (we're reusing the characters from the string literal to write the other two programs, which is what the OP seems to have had in mind based on Sandbox comments, and indeed the quine uses up the entire supply of available 3s). There are no I/O exploits: we're writing full programs with input from standard input and output to standard output. So this is the sort of answer that, I suspect, fits the question particularly well.

  • When initially posted, the Underload answer didn't have an explanation. I added an explanation after it received 1 upvote (I didn't delay this intentionally; just tried to post as fast as possible and then started working on the explanation). The explanation is incredibly simple and straightforward and generally only helpful for people who don't already know Underload (because the programs are so simple and/or standard).

    For the 7 answer, I posted the explanation at the same time as the answer. The programs are much more complex, so there's a lot more to explain, with all three programs needing extensive explanations of what they do and descriptions of what's going on internally (in fact, with two of the programs, I needed to explain not only the program itself, but also the behaviour of various strings that were constructed and evaled during the program's execution). So there's much more that you can learn from the explanation than you can with the Underload program (and the explanation inherently needs to be much longer).

  • When initially posted, the Underload solution was incorrect. I was the first person to notice, and it had already received 5 upvotes at that point. (I've since fixed the solution, but at the cost of two bytes.) The problem is that I'd gotten confused about function/full program distinctions and I/O conventions, and accidentally written code that only works as a snippet (which is not allowed on CGCC). The fix was trivial (just add () around each solution to make the sequence of instructions into a function literal), but it's at least mildly surprising to me that nobody noticed this. I probably would have gotten this right first time if I hadn't been rushing to make sure I got the first answer.

    As far as I'm aware, the 7 answer has been correct ever since it was posted. (An initial version had a mistake – reordering the programs to reduce byte count of the encoding, I initially forgot that 7 "virtually" pads out programs with 7 characters in order to round them up to a full byte, and thus writing a program that ends with 7 would fail to obey the requirement because it wouldn't be part of the string that was seen by the interpreter. But I caught that while writing the explanation and fixed it before pressing the post button, rearranging the programs so that they all ended with some other character. If I'd been trying to FGITW with that answer, it would have been posted with the mistake in and probably nobody would have caught it for ages.)

What I expected would happen from this experiment is that the Underload answer (having been posted first and attracting a rush of early upvotes – it was at +5 when I posted the 7 answer) would likely stay at the top for a while (it's hard to dislodge the first-posted answer), but that the 7 answer would attract upvotes faster and eventually end up overtaking it, and I was interested to see how long that would take.

What actually happened was somewhat concerning. The 7 answer started slowly catching up to the Underload answer for a while, eventually reaching +7 for Underload and +6 for 7. Then someone upvoted the Underload answer but not the 7 answer, making it +8 and +6. And then, a couple of other people's answers (first Ruby, and then JavaScript) got upvoted and managed to come in between. (Those are good answers, but a little harder to directly compare to the Underload than the 7 answer is, so it's hard to determine which of the answers "should" objectively have the top spot, but it doesn't really matter for the experiment; it's clear that regardless of which of the other three programs is better, it shouldn't be the Underload.) But the further down the page an answer is, the more difficulty it has in receiving upvotes. Because the 7, Ruby and JavaScript answers were all swapping round for the second and third spots, whereas the Underload answer was in the top spot all that time, the Underload answer has now pulled away to +10, whilst the other three answers are languishing at +8 (Ruby)/+7 (JavaScript)/+6 (7); people who are just upvoting two or three answers are going to miss good answers further down the page.

I think this phenomenon is doing actual damage to the site. Suppose that I were aiming primarily for reputation, as opposed to just golfing for fun (and/or to do experiments into how Stack Exchange behaves); note that the site tries to strongly push people into gaining reputation and/or badges. Posting answers like the Underload answer is a good way to get reputation, even though those answers don't add to the site much (indeed, I obtained a bronze badge from it even despite the community wiki mark on the post). Posting answers like the 7 answer is not a good way to get reputation, even though there was only a 2½ hour time difference! So the site is pushing people into posting answers that fulfil the question and have nothing inherently wrong with them (apart from, apparently, sloppy mistakes due to trying to post too quickly), and pushing people away from posting answers that put a lot of thought in and aim to solve the question particularly well.

(Note that one reason I community-wiki all my posts is to avoid being tempted to post answers like the Underload answer, rather than answers like the 7 answer. When there's no reputation at stake, there's no incentive to post lower-quality answers in order to gain it. I often post answers like the Underload answer anyway; but, with no reputation at stake, it makes sense to delay before posting it so that it doesn't accidentally get pinned to the top of the thread and end up distorting the voting long into the question's future.)

I'm not sure if there are any real conclusions to be drawn from all this, but I thought it was nonetheless worth posting the evidence and starting a discussion.

(EDIT: I just noticed that the question in question is on HNQ (I saw it on the site sidebar). This probably isn't too surprising, all things considered, but wanted to make note of the situation.)

UPDATE: It's now one week since the question was posted, and the scores are 11 for the Underload answer, 9 for the Ruby answer, and 8 each for the 7 and JavaScript answers. So the top four answers have had one upvote each since this post was posted. I'm actually a little surprised at the difference in the voting rate between very new posts and slightly older posts (presumably it fell off the front page?), but if late votes are rare, that gives a clear explanation of why posts that get an early lead tend to stay there.

UPDATE 2: It's now one month since the question was posted. The scores are 12 for the Underload answer, 9 for the Ruby and 7 answers, and 8 for the JavaScript answer. So only two of the top answers have been upvoted since the last update (although one of them was in third place at the time).

This makes me suspect that the underlying problem is that people aren't giving as much attention to answers on old questions as Stack Exchange's voting mechanism expects: the vast majority of votes on a question's answers altogether appear to come shortly after the question is posted. It may be that the reason why an early lead never disappears is that the number of votes an early answer collects before the better answers are posted may be higher than the total number of voters who even looked at the later answers, rather than anything to do with the choices people make as to what to look for. (Or in other words, "just upvote the good answers" may not be a workable solution; a late answer could be upvoted by 100% of the people who see it and have enough reputation, and still never catch the early answers.)

\$\endgroup\$
7
  • 1
    \$\begingroup\$ I think this is a really interesting experiment to have run, and that it somewhat confirms a lot of people's suspicions about the problem with FGITW (even if I'm not 100% sold on your conclusions/assumptions about the answers and their scores). Unfortunately, I'm not quite sure if there's any discussion to be had beyond "Yeah, this is a problem". As part of the SE network, we're prone to some of their flaws (pushing for rep/badges/etc.) and that'll never change while we're on the network. And, it's often newer users from HNQ that cause this effect, not the users who are likely to see this post \$\endgroup\$ – caird coinheringaahing Mar 15 at 19:25
  • 3
    \$\begingroup\$ Finally, as harmful as it can be, people are entirely free to vote how they want; SE just recommends voting in certain ways. If a user with the 101 rep from the association bonus stops by the site from HNQ, sees this "magic language" that can do this with only a few characters more than Hello, Permutations, clearly it deserves an upvote right? And, if someone wants to think and vote this way, they are free to do so. People who think this way aren't going to stop by Meta to see posts on how they should vote, they're just going to leave an upvote and move on \$\endgroup\$ – caird coinheringaahing Mar 15 at 19:29
  • 1
    \$\begingroup\$ I'm aware that it's probably unfixable, but I think it was still worthwhile gathering the data (even if it's only one data point). Seeing a number of different programs take the #2 slot over time is heartening, at least. FWIW, I don't think this is a CGCC-specific problem; even on other Stack Exchange sites, an early (but useful) answer will nearly always end up beating a later but better answer. \$\endgroup\$ – ais523 Mar 15 at 22:42
  • 3
    \$\begingroup\$ One more thing to consider is how intimidating the 7 answer looks. I have personally skipped it the first time I saw it because knowing how complex programming in 7 is and looking at how long the answer was, I knew it would have taken me a long time to read and understand all of it. Maybe this is part of the reason why that answer didn't receive as many upvotes as the Underload one. \$\endgroup\$ – Leo Mar 15 at 23:49
  • 2
    \$\begingroup\$ @Leo: That's an interesting point; maybe answers that took less effort to write are 'better' in some sense because they're more straightforward / easier to understand, and thus are more informative for people who are trying to learn more about golf than the higher-effort answers that confuse even the people who wrote them. I guess there's an interesting argument there about who/what we should be optimising the site for. In particular, is "spreading knowledge" seen as an important goal for CGCC, and is it more or less important than competing with each other? \$\endgroup\$ – ais523 Mar 15 at 23:53
  • 5
    \$\begingroup\$ I think a potential solution could be to try make SE have CGCC default to sorting by active rather than votes (I think that's what most experienced users here do anyway), which would also help with bandwagon voting \$\endgroup\$ – Redwolf Programs Mar 16 at 21:34
  • \$\begingroup\$ What's strange is that someone still upvoted my answer on "Minceraft" even though I (accidentally) used 1e5 instead of the correct 1e4. I also happened to be the first answerer. \$\endgroup\$ – Recursive Co. Apr 25 at 11:17
9
\$\begingroup\$

These are some very interesting and, unfortunately, not-too-surprising results. I do think the FGITW phenomenon is quite disappointing and discouraging for what I'd say are higher quality and higher effort answers; my proudest answers came months or even years after the question, not the quick 5-byte answers I wrote in a few seconds from memory of Jelly's features.

There unfortunately isn't much we can do about it. Especially for HNQ questions, since most people sort by vote (because except PPCG and Puzzling, basically every other site uses votes to determine quality, and thus higher voted = better), high voted questions get attention and get upvoted again. This is pretty much uniquely unhealthy on these two non-Q&A sites.

We could always bring back the contest for underappreciated / under-upvoted answers. I organized this a couple years back because I was noticing the same issue wherein voting seemed almost inversely proportional to effort - low-effort meme challenges were getting hundreds of upvotes, compared to higher effort and much more thought-provoking challenges only getting a handful, and quick 5-byte (or even 0-byte) answers getting to the top of the scoreboard early and never losing their #1 spot (no offense to any of the challenge or answer writers who benefited from this of course - it's a phenomenon of the system and the way people vote, and I believe that people are acting in good will and not intentionally gaming the system for reputation or anything like that).

Of course, we could encourage people to sort by active/new or scroll more and look for higher quality answers to upvote. In fact, I feel like we already have. But especially with HNQ attention, people have the right to vote how they want and for most people that's just popping into a question page and upvoting the top three answers and leaving.

Unfortunately, last time I tried, it ended after just a couple of runs because there just weren't many people interested in it - offering bounties, finding and nominating answers, etc. But, if there's enough support for a similar contest to find and bounty underrated answers that took effort but never got the hype or attention, I'd definitely be open to organizing it again.

\$\endgroup\$
2
  • 9
    \$\begingroup\$ I want to highlight the recommendation to sort by active. I've had a better experience on CGCC since I made the switch. With new posts at the top automatically, it's easy to see new answers and golfs, and voting for them comes naturally. \$\endgroup\$ – xnor Mar 16 at 3:16
  • 6
    \$\begingroup\$ Underappreciated Answers contest sounds interesting. But given that the existing form takes almost the same amount of community effort as the annual Best Of, we could simplify the process somewhat, e.g. just announce an endless bounty to accept candidates, and anyone who thinks a candidate deserves more attention can give out a bounty on it. \$\endgroup\$ – Bubbler Mar 16 at 8:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .