Adaptive Sampling
Ideally an approach would be specified in the question that can be applied regardless of the number of answers or the strategies used. This would describe what will be done and how it will be determined whether a given sample size is sufficient. This should allow the sample size to be varied based on the results, rather than trying to infer an ideal size from the number and type of answers.
If a perfect result cannot be obtained, this can be acknowledged by showing joint nth place rather than a potentially incorrect leaderboard order.
Example approach (for all against all games)
For an all against all game, run N games and record the result. Run another N games and compare the result with the first. If they differ, run another 2N games and compare the result with the first 2N. Continue doubling until the results match.
Using this approach to obtain a strictly ordered leaderboard (no joint nth place) is likely to require a very large number of games, especially if there are a large number of players and where there are two or more similarly matched players. To keep the required number of games manageable, the strictness can be compromised in some way.
If only the winner is important, then the process can simply be repeated until 1st place is consistent, and any differences below that can be converted into joint places. Alternatively the process can be continued until any joint places only include 2 players (so there are not 3 or more players joint nth place).
As the time required will increase as new answers come in, the question poster may choose a hybrid approach, where there is a maximum number of games. This way early leaderboards will be strictly ordered, and later leaderboards will have an upper limit on the time required, but may have joint places.
A faster equivalent
With the approach described, doubling the number of games each time the first half doesn't match the second half, the worst case is that 2M+1 games are required for an accurate ordering, in which case this approach will require 2M+1 games. If instead only 2 extra games are played each time, then the worst case will require 2M+2 games (roughly half as many) and be equivalent in the best case.
So the approach is to play 2N games, and if the first N do not match the last N, play an additional 2 games. Now if the first N+1 do not match the last N+1, repeat as required. The process can be terminated early with joint places as before.
Example approach (for pairwise games)
If players are paired up and compete against each other one on one, then N games can be run for each possible pairing, and then compared with another N games each. Only those that do not match need to be repeated, so that the majority of samples will be of closely matched pairs and further time need not be wasted on pairs where there is a clear winner.
Again, after a time limit is reached joint places can be accepted (or joint places can be accepted below 1st place, with more time being allowed if 1st place is still ambiguous).
Start with a reasonably large N
Note that starting with N (the initial number of games) too small will give a high probability of early termination with inaccurate results. For example, starting with N=1, there may be a significant probability of the 2 runs of 1 game having the same outcome, even if that outcome is not the same as the long term average. Starting with a larger N makes it much less likely that both runs will match and be inaccurate.
