Could PPCG benefit from a new sorting mode for answers?

Question

The default sorting for answers (by votes) seems somewhat inappropriate here on PPCG. In any case it tends to put early answers over late answers, because once a few are there, few people scroll to the bottom to see newer answers and even fewer people sort by activity.

If you consider that better answers will usually take longer to come up with and implement, that means the answers that deserve the votes are actually at a disadvantage.

This is problematic for different reasons on different challenge types:

Popularity Contest

Sorting by votes is still useful here, because that actually shows you the ranking of the answers by the winning criterion. However, for that very reason (that votes determine the winner) a bias towards early answers can't be in the interest of the community.

Other Challenges

Here, sorting by votes doesn't even make really make sense in the first place. The reason this is the default sorting on other SE sites is that it's a decent heuristic for which answer is the best. But we don't need that, because the quality of an answer can be determined objectively by the winning criterion. Of course, sorting by winning criterion is not an option, but sorting by votes doesn't seem useful either. Hence, some other sorting could at least help later and better answers get the votes they deserve.

So, I'd say we should think about different, more appropriate, sorting modes and also consider whether the default sorting should be changed. If we can come to a community consensus here, we might be able to bring this to the attention of SE staff.

The questions to discuss are: are the existing sorting modes sufficient or do we need new ones (which ones)? Should a different (new or existing) sorting mode be the default?

Answer 1

Measuring the Strength of the FGITW Effect

Although this answer isn't a suggestion for a solution, it should provide a better understanding of the problem. A proper understanding is necessary to create a solution.

The FGITW effect ("fastest gun in the west" effect) is the phenomenon in which the earliest answers receive the most votes and always stay at the top of the answer list. The feedback loop is this:

early answer -> some votes -> top of the answer list -> more visibility -> even more votes

Although I think everyone recognizes the existence of FGITW, people might not have an idea as to its power. How can we measure the FGITW effect in an objective way? With the help of user Eric Tressler, some data was collected and measured.

We decided to look at the RPSLS tournament as an example of FGITW. This challenge was posted about 3.5 weeks ago. Here is the data (collected by Eric):

// Score, date posted rank, #votes for Rock Paper Scissors Lizard Spock
n = 65
64  28 18
62  55 1
62  65 2
61  64 1
60  29 5
58  27 3
57  32 2
53  61 1
51  42 1
51  16 20
49  31 2
49  49 1
47  34 3
46  56 1
45  24 4
44  48 1
43  40 4
42  20 4
42  53 1
41  12 2
41  13 3
40  23 3
40  46 2
39  62 1
38  38 4
38  33 2
38  43 2
37  50 1
35  15 8
35  58 1
35  54 1
34  3  4
33  10 3
32  4  8
32  44 1
32  14 5
32  17 2
31  19 8
31  45 2
31  51 1
30  1  3
30  60 1
29  22 4
28  25 3
27  57 1
25  30 2
24  2  7
24  39 3

The three variables are:

• Score (s), showing how well the post performed in the contest
• Date rank (d), with lower numbers being the earliest answers
• Votes (v), which is just the net number of votes (up - down)

We wanted to find the relationships between these variables. We used Spearman's Rank Correlation Coefficient (Rho) to measure the correlation between the variables.

Dates-Scores:

The above dates-scores graph shows how performance was related to submission time. You can see how the very best entries (high 50s and 60s) came later in the competition, while the earliest submissions generally performed below-average (the average score was 34.6615).

We calculated rho = 0.243443, so there was a moderately-weak correlation between newer posts and higher scores. This is as expected, since the later bots could be tailored to the competition. Also, the best posts would probably take longer to write.

Given a sample size of 65, the standard error for rho is approximately 0.0791, so these results are significant.

Dates-Votes:

This is the interesting part - the FGITW. Here is the graph, which speaks for itself.

This is the FGITW in full force, you can clearly see how the most up-voted answers are typically early answers. The most-voted answers were all early in the competition. After a certain point (the latest 18), no answer has more than 2 upvotes. Although we would expect older posts to have some more votes, simply by virtue of being older, it is clear that the FGITW Effect is strong.

We calculated rho = -0.766193, which is a moderately strong correlation between earlier posts and more votes. This is objective evidence as to the strength of the FGITW, and I believe that the age of the post is one of the most important factors in determining the number of votes an answer will receive.

Scores-Votes:

While the relationship between dates and votes should show the strength of the effect, the relationship between scores and votes should show why this is a problem.

The above graph shows the relationship between performance and the number of votes. With the exception of a few outliers, this third graph looks rather similar to the second graph. The submissions with lower scores actually had more votes than most of the submissions with higher scores.

We calculated rho = -0.250174, which is a moderately weak correlation between "higher score" and "fewer votes."

A negative correlation is the opposite of what we should want. Submissions with better performance should receive more votes. I suspect that this is due to the FGITW as well. As we saw above, earlier posts receive more votes, while later posts perform better. It turns out that this actually causes a negative correlation between score (performance) and votes.

Answer 2

Random Sorting

I propose that a purely random default sorting would solve the above problems. It would simply remove the bias towards any kind of answer and would distribute the votes more fairly. Of course, the existing sorting by activity will also remove the bias towards early answers, but the fact that it's deterministic is still problematic.

Especially in cases where there are so many answers that few people read all of them, some answers are just almost never seen because they are too far down the list (depending on which sorting is active). If the sorting was random by default, people still wouldn't scroll through all of them, but at least each answer would get the same number of views on average.

I would even go so far to suggest that this default sorting is not remembered between visiting different questions (currently, if you switch to sorting by activity, all questions will be sorted by activity until you switch back).

Note: random sorting has been suggested before a few years ago back on SO but without much response. I don't even think it's that useful on other SE sites, but PPCG could really use it.

Pros

• Trivial to implement.
• All answers are equally likely to be seen. So there is no bias towards any answers, hence it's as fair as it gets.

Cons

• All answers are equally likely to be seen. Even the bad ones.

(However, this would lead to really bad answers getting more downvotes and hence being more likely to be deleted, resulting in less overall clutter. So maybe this is another pro after all.)

Answer 3

Randall Munroe - a statistical estimation ordering

When reddit had a similar problem with new comments getting lost at the bottom, Randall Munroe brought in a new sorting algorithm. Obviously don't just vote for this because you like Randall Munroe - it's an interesting algorithm well worth reading about before you decide.

It only requires the information that stack exchange already uses to sort by votes anyway (upvotes and downvotes), but deals elegantly with precisely the problem we're discussing. The example at the end demonstrates this nicely.

The algorithm has the best features of sorting by votes, sorting randomly, and sorting by time, without actually using any randomness or time, just upvotes and downvotes.

In short: The Randall Munroe algorithm works well on reddit because it uses both upvotes and downvotes to measure how statistically likely a post is to get to the top, and orders by the expected future position rather than just its current votes. So it doesn't need time because the downvotes give a percentage measure rather than just a plain count of upvotes The problem with applying that to stack exchange is that downvotes are a rare thing here - most answers have zero downvotes, so we might need to take a different approach that bases the statistical measure on upvotes, downvotes, and time. These are all already available to the sorting code of stack exchange so it is feasible to implement.

Answer 4

Reddit-style sorting

I don't remember exactly how it works, but Reddit's sorting algorithm (or HN's, or ...) takes both time and votes into account to present "hot" content, i.e. content climbing faster than typical or with higher than typical votecount. In practice high-voted content tends to be the norm, with new or rising content spliced in occasionally.

Our situation is a bit different: we don't want sufficiently old answers to suddenly drop quickly the same way as Reddit does, and we have an additional "active" parameter separate from the time since initial answer. Still, Reddit's (or a similar service's) algorithm could serve as a base for a good sorting algorithm. Reddit-style sorting might also be of interest in other SE subsites as well (moreso than random sorting, at least), so there's more reason for SE to provide it.

---

I think it'd be unfair to high-voted entries with completely random sorting, and even though we don't need the vote-based sorting for questions with something other than popularity-contest as the winning criterion it still makes sense to promote high-voted answers since they presumably follow the winning criterion well (or otherwise entertain the voters).

Answer 5

Not enough information given

I can't agree to change the default sort at all unless more evidence is given that the basic premise is true.

PhiNotPi gives some data showing that the FGITW effect happens, and I don't doubt that. However, the only correlation it really shows is between early answers and votes. For one question. With tons of answers.

To be clear, 80 answers is just not typical. The site as a whole averages a bit under 10 answers per question. Certain challenges might garner 30+, but there aren't very many, and they just drag the average up. I don't think the default sort for all questions should be changed for the sake of those few.

...few people scroll to the bottom to see newer answers and even fewer people sort by activity.

This could be true. It also might not be. A simpler explanation to explain FGITW could be that early answers get more page views, so have a better chance of getting an upvote. Even if every user scrolls to see every answer on every question, the early ones still get seen more.

Changing the sort method won't change that, except (most likely) on questions that go 30+ answers and paginate. I think it likely that on most questions, people read at least the majority of answers, because there aren't very many anyway. In addition, since the vast majority of challenges are code-golf, there are typically <10 short answers.

Answer 6

Vote Order + Semi-Random Permutation

A combination of votes and a randomizer might be best.

One method of accomplishing this is to take the placement of each answer (1, 2, 3, etc.) and add a random real number 0-5 to that number. Then, sorting by these new numbers gives the new order. The goal is to create a semi-random permutation of the order such that no answer is too far from its original location.

Pros:

• The most upvoted answers are still near the top, with just enough randomization that the not-quite-as-upvoted answers also get the change to be near the top.
• Answers with no votes are still lower down, but aren't always dead last.
• The "catchy" answers are still visible to random visitors, which helps create interest in this site.
• Simpler than other methods.
• Changing the 0-5 range to something bigger (like 0-10) is an easy way to increase randomness. As the range becomes larger, it is closer to the random sorting. If is is smaller, it is closer to vote order.

Cons:

• Newer answers still have an advantage if the choice of range is small, as highly-voted answers will remain visible.
• If there are a ton of 0-score answers, most of those answers will end up being swapped around with each other.

Answer 7

Why sorting by votes is a good idea

PPCG benefits a lot from people stumbeling upon popular questions, when they open thsi question, we definitely want to showcase the best and most interesting answers. In many categories, there is a very clear difference between good and interesting answers. This answer explains this better than I ever could..

Why sorting by votes is a terrible idea

Well, basically, for all the reasons you mentioned, late answers don't get the attention they deserve and the gap between early answers and late answers only gets bigger over time (especially when answers exceed one page).

Problems with random sorting

While it is more fair and we could still let new users (or people who aren't logged in) keep their ranking by votes. It's still valuable to have answers which are obviously good near the top and terrible answers (or very unremarkable ones) buried. Random sorting won't do this and whenever I visit a question with a lot of answers (the sort of question where this ranking would be desirable) I'll be presented with mostly 'meh' answers.

An alternative Ranking system

We could opt for a ranking systems that still largely bases on votes but favors newer answers, similar to how hot network questions work A possible formula for this could look something like: votes on this anwer/votes on all answers + a*votes on this anwer/votes on all answer since this answer was posted where a is a carefully selected weight. This would give late answers a fighting chance when they are first posted. If they gather some upvotes then, they'll stay up and even float up, if being at the top doesn't gather them upvotes though, they'll drop down again.

A random addend could also be added to this score to bring some life into things.

Answer 8

Rate of votes compared to placement on page?

This is sort of a long-shot thinking-out-loud idea, but I feel that a "hotness" ranking should be determined by two things:

• The rate of votes that the answer is experiencing
• The answer's history of placement on the page

My logic is that the rate of votes is determined by two primary factors:

• The quality of that answer (which may be very roughly approximated by votes/viewer)
• The visibility of that answer (which determines the rate of new viewers)

Basically, we want to calculate votes/viewer = rate of votes / rate of new viewers.

I think the ranking system should try to determine which answers have the highest votes/viewer ratio. The ranking algorithm would look at the rate of votes and try to determine how much of that rate is due to answer quality and how much is due to visibility.

What we would need to do is take an empirical measurement of how an answer's placement on the page affects the rate of votes it is experiencing. I expect the visibility to decrease with lower placemement, with large decreases once the answers are on second/third pages.

Perhaps here is one way of calculating it. This is by no means the best way, just a possible realization of the main idea of this post. The idea of taking answer placement into account can be incorporated into several of the other suggestions.

A = answer, N = placement position number, votes(A) = all votes, score(A) = up - down

expectedrate(N) = average rate of votes experienced by all answers in Nth placement position
timespent(A,N) = the total time spent by answer A in placement position N
expectedvotes(A) = summation( timespent(A,N) * expectedrate(N) )
result(A) = score(A) * votes(A) / expectedvotes(A)

Basically, the answer's ranking is affected by how that answer's number of votes compares to the expected number of votes. The expected number of votes takes into account the how long the answer spent in each position and the rate of votes experienced by each position.

Pros:

• High-quality answers with very low visibility will have that taken into account and rise to the top very quickly
• Highly-visible answers with the most upvotes will be penalized if the only reason for their upvotes is the visibility and not because of above average quality.

Cons:

• Might not work when the pool of voters is very small and starts to diminish.
• The popularity of the question probably determines how influential answer placement is.
• Hard to calculate or measure.

Answer 9

Sort by votes minus amount of time since post

Here's a fancy userscript for your testing pleasures (doesn't work for multiple pages):

function rank(x) {
  var score = parseInt($('.vote-count-post', x).text());
  var milliseconds = new Date($('.post-signature:last-child .user-action-time span.relativetime', x).attr('title')) * 1,
    hours = milliseconds / 1000 / 60 / 60;
  return score - hours; // or some other factor
}

var ranked = $('.answer').sort(function(a, b) { return rank(a) < rank(b) ? 1 : -1; });
$('#answers').empty().append(ranked);

This will rank a +20 answer posted 10 hours ago the same as a +15 answer posted 5 hours ago. I.e., it will subtract the amount of hours since post from the score. (Technically, hours since the epoch, but they work the same in practice.)

Of course, the amount that time has to do with the sorting could be changed, but 1 hour == 1 upvote seems like a good factor, from brief testing.

Advantages:

• Very simple.
• Can be tweaked easily.
• You see new answers when you come back in a day or two.
• Answers with a ridiculous amount of upvotes still stay at the top.
• Answers posted, say, a few months later will have a very large advantage.

Disadvantages:

• You see new answers when you come back in a day or two.
• Answers with a ridiculous amount of upvotes still stay at the top.
• Answers posted, say, a few months later will have a very large advantage.
• Not very fair for users who post on times like the weekend, whose posts sink down and don't have much of a chance at rising back up.
• Can still be affected by bias.

Answer 10

highest = vote count for highest voted answer on the same question

hours = number of hours since answering

votes = number of votes on your answer

sorting: max(0, highest - votes - hours) + votes

By taking into account the highest voted answer, it ensures that all answers appear at the top, at least when first posted. "max" makes sure that the time since posting only benefits the post for a suitable period of time, during which users have the chance to vote on it. And in the end it still comes back to votes, but at least all answers would have had a chance to be upvoted.

note, when I say votes above, of course I mean upvotes - downvotes. Also, hours could be replaced with "number of hours since answering * n", to tweak it.

Answer 11

Slightly weighted random sorting

This has in some form or another been suggested, but the weight was a little too inflexibly strong in all of them IMO. How about this algorithm (in Python pseudocode)?

# an arbitrary constant signifying the weight given to either the votes
# (greater than 1), the time (0 < n < 1), or neither (1).
ratio_weight = 1

# an arbitrary constant signifying the weight given to the statistics of
# each entry (greater than 1), the random element (0 < n < 1), or
# neither (1).
stats_weight = 1

def order_list(entry_list):
  total_votes = reduce(entry_list, lambda x, y: x + y)

  # this is the main weighting of the sorting. `entry` is a tuple in the
  # form of (answer_entry, random_index_to_use).
  def get_sorting_index(entry):
    e = entry[0]
    return entry[1] + stats_weight * (1 -
      (ratio_weight * (e.votes/e.time)) / total_votes)

  # get a list of random indices to work with
  random_indices = list(range(len(entry_list)))
  random.shuffle(random_indices)

  # get a zipped list with the random indices
  entry_list = zip(entry_list, random_indices)

  # sort using the weighted, randomized entry indices
  entry_list.sort(key=lambda entry: get_sorting_index(entry))
  return [x[0] for x in entry_list] # strip random sorting index

Please comment if you think something doesn't look quite right (either bug or proposal).

Answer 12

Merge Active + Votes + Random Rankings using a Condorcet method

I think each of these has its benefits, which can be merged.

Active sorting means that newer answers are more visible and given a chance to be viewed. This helps to counteract the bias towards older answers. It allows a sort of kick-start to new answers.

Votes sorting means that new users will see primarily see quality posts as opposed to downvoted posts. We don't want to place too much faith in here, but we can't simply ignore it.

Random sorting helps to scramble up the order. It is blind to quality, which can be both a pro and a con.

This algorithm would take the three rankings and use them as the votes in a Condercet election. There would be X copies of the active ranking, Y copies of the votes ranking, and Z copies of the random ranking (each copy is the same ordering). The X, Y, and Z are the weights. There are several Condercet-compliant methods to merge these, but something like Ranked Pairs could work decently.

Pros:

• The weightings are easy flexible, offering a continuum of options all the way to complete randomness.
• With the correct weights, the new posts with few votes are on the same footing as the old posts with many votes.
• This system favors the newer posts with many votes and is against the older posts with no votes.

Cons:

• Finding the correct weightings would be an empirical thing.
• The active ranking can be kinda chaotic sometimes when older posts are edited. It can give bad posts a second chance at greatness after being fixed, but could motivate people to create pointless edits. We could replace Active with Newest if this is a concern with little impact on the algorithm.