An experiment into the FGITW phenomenon

Question

Yesterday (probably the day before yesterday in some time zones), I decided to run an experiment on the "Fastest Gun In The West" phenomenon, in which early answers tend to get disproportionately rewarded compared to late answers.

My experiment involved this question, to which I posted two answers. For the first answer, in Underload, I intentionally aimed to post it as fast as possible (I had the answer ready about 10 minutes after I saw the question (I saw it about 10 minutes after it was posted)).

The second answer, in 7, was posted about 2½ hours after the first, and I had been working on it for most of that time.

I think the second answer is more or less objectively better than the first, in pretty much every dimension we care about:

• The answers both belong to the same language family (concatenative low-level tarpit esolangs). So it's unlikely that the choice of language had a huge influence on the voting. Additionally, we can look at how well my answers in Underload and 7 normally do; Underload normally does fairly well, but 7 normally does better. So the first answer doesn't have an advantage in terms of language choice.

  Additionally, I created both of the languages in question, so there's no potential bias in terms of language creator / people-golfing-in-their-own-language; the two answers are symmetrical this way.

• The first answer took almost no thought to write. It was a case of "what's a language where I can quickly write a quine, cat program and string printer?". The latter two are builtins in Underload, and it is trivial to modify one of the two standard Underload quines, (aS(:^)S):^ (which I have memorised, as does probably every Underload golfer), to contain a payload (or in this case, some extra useless characters). I made it even easier by picking an interpreter which (non-standardly) treats unknown characters as no-ops. Generally speaking, the recommendations are to not upvote trivial builtin-based solutions (and we should probably recommend not upvoting "trivial modification to a standard program" solutions, too).

  Contrasting with this, the second answer was much more difficult to write. Even printing a string literal has a range of subtleties in 7 (e.g. I spent a while pondering about whether there was a way to shorten it by somehow making use of the 6), and that was the easiest of the three programs to deal with. The quine took a while, because I'd never previously needed to fit a 6 into a quine before (or indeed, created a quine that wasn't just two identical halves that printed each other or where one printed the other twice), and so I spent some time working out a quine template that could make it work. The cat program was a nightmare; 7 doesn't have a builtin for this. Reusable code in 7 is sufficiently context-dependent that it's hard to define what a function even is, meaning that I would need to read standard input explicitly, and 7 can only read a character at a time. So then I needed to construct a loop, in a low-level esolang that's very unintuitive and hard to reason about (probably worse than Brain-Flak). This involved inventing entire new programming techniques that I don't think have been used in 7 before (such as polyglotting the loop body with a literal constant integer).

  Yet, despite all this, I managed to fit the cat and quine programs in using only two extra characters (a 6 for the cat program and an additional 7 for the quine program), costing only ¾ of a byte (which ends up rounding down to 0 because the original program length was slightly over a byte boundary).

  So I consider the 7 answer to be much better-crafted, both in terms of effort (a lot more work is required to write something like that), and in terms of the golfing skill demonstrated by the solution, and it seems likely that people who upvote based on how impressive a program is would prefer the 7 program.

• The Underload answer is vaguely cheaty: it trivialises the source-layout aspect of the problem simply by defining functions that output to standard output (meaning that we can do what we like to the function's return value with the extra characters we're forced to use), so it's basically just exploiting the I/O conventions. (The type of string that the program takes as input isn't even good for anything other than eval or printing.) I needed to use an extra colon in it in order to compensate for the exclamation mark in the string (which would otherwise make the quine take an argument), but apart from that, the entire point behind the question gets ignored. In this sense, I consider it quite similar to answering a restricted-source question in Lenguage (or indeed a "do X without letters" question in 7), which shouldn't be considered that interesting or funny at this point. Additionally, the answer is interpreter-specific and won't work on most Underload interpreters, due to requiring a specific handling of invalid commands.

  The 7 answer, meanwhile, complies with the spec comprehensively and in a way that fits the spirit of the question (we're reusing the characters from the string literal to write the other two programs, which is what the OP seems to have had in mind based on Sandbox comments, and indeed the quine uses up the entire supply of available 3s). There are no I/O exploits: we're writing full programs with input from standard input and output to standard output. So this is the sort of answer that, I suspect, fits the question particularly well.

• When initially posted, the Underload answer didn't have an explanation. I added an explanation after it received 1 upvote (I didn't delay this intentionally; just tried to post as fast as possible and then started working on the explanation). The explanation is incredibly simple and straightforward and generally only helpful for people who don't already know Underload (because the programs are so simple and/or standard).

  For the 7 answer, I posted the explanation at the same time as the answer. The programs are much more complex, so there's a lot more to explain, with all three programs needing extensive explanations of what they do and descriptions of what's going on internally (in fact, with two of the programs, I needed to explain not only the program itself, but also the behaviour of various strings that were constructed and evaled during the program's execution). So there's much more that you can learn from the explanation than you can with the Underload program (and the explanation inherently needs to be much longer).

• When initially posted, the Underload solution was incorrect. I was the first person to notice, and it had already received 5 upvotes at that point. (I've since fixed the solution, but at the cost of two bytes.) The problem is that I'd gotten confused about function/full program distinctions and I/O conventions, and accidentally written code that only works as a snippet (which is not allowed on CGCC). The fix was trivial (just add () around each solution to make the sequence of instructions into a function literal), but it's at least mildly surprising to me that nobody noticed this. I probably would have gotten this right first time if I hadn't been rushing to make sure I got the first answer.

  As far as I'm aware, the 7 answer has been correct ever since it was posted. (An initial version had a mistake – reordering the programs to reduce byte count of the encoding, I initially forgot that 7 "virtually" pads out programs with 7 characters in order to round them up to a full byte, and thus writing a program that ends with 7 would fail to obey the source-layout requirement because it wouldn't be part of the string that was seen by the interpreter. But I caught that while writing the explanation and fixed it before pressing the post button, rearranging the programs so that they all ended with some other character. If I'd been trying to FGITW with that answer, it would have been posted with the mistake in and probably nobody would have caught it for ages.)

What I expected would happen from this experiment is that the Underload answer (having been posted first and attracting a rush of early upvotes – it was at +5 when I posted the 7 answer) would likely stay at the top for a while (it's hard to dislodge the first-posted answer), but that the 7 answer would attract upvotes faster and eventually end up overtaking it, and I was interested to see how long that would take.

What actually happened was somewhat concerning. The 7 answer started slowly catching up to the Underload answer for a while, eventually reaching +7 for Underload and +6 for 7. Then someone upvoted the Underload answer but not the 7 answer, making it +8 and +6. And then, a couple of other people's answers (first Ruby, and then JavaScript) got upvoted and managed to come in between. (Those are good answers, but a little harder to directly compare to the Underload than the 7 answer is, so it's hard to determine which of the answers "should" objectively have the top spot, but it doesn't really matter for the experiment; it's clear that regardless of which of the other three programs is better, it shouldn't be the Underload.) But the further down the page an answer is, the more difficulty it has in receiving upvotes. Because the 7, Ruby and JavaScript answers were all swapping round for the second and third spots, whereas the Underload answer was in the top spot all that time, the Underload answer has now pulled away to +10, whilst the other three answers are languishing at +8 (Ruby)/+7 (JavaScript)/+6 (7); people who are just upvoting two or three answers are going to miss good answers further down the page.

I think this phenomenon is doing actual damage to the site. Suppose that I were aiming primarily for reputation, as opposed to just golfing for fun (and/or to do experiments into how Stack Exchange behaves); note that the site tries to strongly push people into gaining reputation and/or badges. Posting answers like the Underload answer is a good way to get reputation, even though those answers don't add to the site much (indeed, I obtained a bronze badge from it even despite the community wiki mark on the post). Posting answers like the 7 answer is not a good way to get reputation, even though there was only a 2½ hour time difference! So the site is pushing people into posting answers that fulfil the question and have nothing inherently wrong with them (apart from, apparently, sloppy mistakes due to trying to post too quickly), and pushing people away from posting answers that put a lot of thought in and aim to solve the question particularly well.

(Note that one reason I community-wiki all my posts is to avoid being tempted to post answers like the Underload answer, rather than answers like the 7 answer. When there's no reputation at stake, there's no incentive to post lower-quality answers in order to gain it. I often post answers like the Underload answer anyway; but, with no reputation at stake, it makes sense to delay before posting it so that it doesn't accidentally get pinned to the top of the thread and end up distorting the voting long into the question's future.)

I'm not sure if there are any real conclusions to be drawn from all this, but I thought it was nonetheless worth posting the evidence and starting a discussion.

(EDIT: I just noticed that the question in question is on HNQ (I saw it on the site sidebar). This probably isn't too surprising, all things considered, but wanted to make note of the situation.)

UPDATE: It's now one week since the question was posted, and the scores are 11 for the Underload answer, 9 for the Ruby answer, and 8 each for the 7 and JavaScript answers. So the top four answers have had one upvote each since this post was posted. I'm actually a little surprised at the difference in the voting rate between very new posts and slightly older posts (presumably it fell off the front page?), but if late votes are rare, that gives a clear explanation of why posts that get an early lead tend to stay there.

UPDATE 2: It's now one month since the question was posted. The scores are 12 for the Underload answer, 9 for the Ruby and 7 answers, and 8 for the JavaScript answer. So only two of the top answers have been upvoted since the last update (although one of them was in third place at the time).

This makes me suspect that the underlying problem is that people aren't giving as much attention to answers on old questions as Stack Exchange's voting mechanism expects: the vast majority of votes on a question's answers altogether appear to come shortly after the question is posted. It may be that the reason why an early lead never disappears is that the number of votes an early answer collects before the better answers are posted may be higher than the total number of voters who even looked at the later answers, rather than anything to do with the choices people make as to what to look for. (Or in other words, "just upvote the good answers" may not be a workable solution; a late answer could be upvoted by 100% of the people who see it and have enough reputation, and still never catch the early answers.)

UPDATE 3: It's now over a year since the question was posted. (I was planning to update this exactly a year afterwards, but forgot – in fact, it's been over a year since update 2.) The Underload answer is now on 14 points, with the 7 answer in second place with 11. The Ruby and JavaScript answers are on 9 and 8, with a Python answer on 7.

This most likely indicates that two people found the question and voted on it since the last update, and upvoted both of my answers. We don't have enough evidence to know which answer they found better – just that they liked both. This actually goes some way towards pointing to an explanation as to why better answers tend not to overtake worse answers: most people who are voting look at some number of answers, probably reading them in sort order (votes, active, oldest, or whatever), and upvote all the "good" answers. This means that if people are sorting by votes, the only way for a better, lower-scored answer to overtake a worse, higher-scored answer is if someone considers the higher-scored answer to not be worth an upvote (and if that answer is good but not as good, it is still likely to be considered worth an upvote).

If this reasoning is correct, the problem could be solved to some extent by persuading everyone to sort by active or newest. There are two problems with that. One is that these sort orders are not the default, so the number of people arriving at the page with those sort orders is going to be very low. (HNQ traffic, in particular, is unlikely to have non-default sort orders because much of it will be from users who didn't have an account here.) The other is that the sort of people who sort by active are often fairly diligent, and likely to read a long way down into the list, meaning that both answers end up getting upvoted anyway.

UPDATE 4: It's now a little over a year since update 3 was posted. The Underload and 7 answers are both on 13 points each (which is interesting, because this means the Underload score has a lower answer than previously: I wonder whether that was an unupvote or a downvote?), with the Ruby and JavaScript answers on 9 points.

I'm wondering whether traffic to the question in question is now predominantly from this meta post, rather than from the main site itself: that might help to explain the equalisation of scores. In any case, with the scores equalising, the experiment is now effectively over because the equalisation of scores means that the FGITW effect has finally stopped affecting the order in which the answers are ranked (unless someone chooses to sort by "oldest").

It is notable that, despite it being over 2 years since the question was first posted, the vast majority of votes on the Underload answer came on the first day: the Underload answer was +10 on the first day and is now +13. Likewise, the Ruby and JavaScript answers were +8 and +7 and are now both at +9. The 7 answer is an outlier on this metric, being at +6 after one day and +13 today, which is one thing that makes me suspicious that this meta post may be biasing the voting. (However, another possibility is that it's much higher up the page than the other high-scoring posts when sorting by "active", which many of the more dedicated CGCC users use when reviewing old posts, due to my edits golfing off another byte in January last year – these may have interfered with the experiment, but OTOH I didn't want the experiment to interfere with the main site.)

---

Full disclosure: I improved the 7 code in January 2022 (a little less than a year after the question was posted), golfing off 2 characters / 1 byte. So maybe the Underload code was "better" in that it was better-golfed. On the other hand, maybe this is an argument that even though 10 minutes isn't enough time to produce a good answer, 3 hours isn't enough time either, and crafting a really good answer to a question can take days or months of thought.

Answer 1

These are some very interesting and, unfortunately, not-too-surprising results. I do think the FGITW phenomenon is quite disappointing and discouraging for what I'd say are higher quality and higher effort answers; my proudest answers came months or even years after the question, not the quick 5-byte answers I wrote in a few seconds from memory of Jelly's features.

There unfortunately isn't much we can do about it. Especially for HNQ questions, since most people sort by vote (because except PPCG and Puzzling, basically every other site uses votes to determine quality, and thus higher voted = better), high voted questions get attention and get upvoted again. This is pretty much uniquely unhealthy on these two non-Q&A sites.

We could always bring back the contest for underappreciated / under-upvoted answers. I organized this a couple years back because I was noticing the same issue wherein voting seemed almost inversely proportional to effort - low-effort meme challenges were getting hundreds of upvotes, compared to higher effort and much more thought-provoking challenges only getting a handful, and quick 5-byte (or even 0-byte) answers getting to the top of the scoreboard early and never losing their #1 spot (no offense to any of the challenge or answer writers who benefited from this of course - it's a phenomenon of the system and the way people vote, and I believe that people are acting in good will and not intentionally gaming the system for reputation or anything like that).

Of course, we could encourage people to sort by active/new or scroll more and look for higher quality answers to upvote. In fact, I feel like we already have. But especially with HNQ attention, people have the right to vote how they want and for most people that's just popping into a question page and upvoting the top three answers and leaving.

Unfortunately, last time I tried, it ended after just a couple of runs because there just weren't many people interested in it - offering bounties, finding and nominating answers, etc. But, if there's enough support for a similar contest to find and bounty underrated answers that took effort but never got the hype or attention, I'd definitely be open to organizing it again.