Elo ratings, treating pairs of submissions as games.
I'm using the Elo system here.
Essentially, the solution is find the best solutions in each language to each challenge, and then treat each pair of such solutions as a match between those two languages, with the winner being the shorter submission.
Languages with less than 10 entries are discarded, and languages have a starting rating of 1000.
One drawback of this system is that ELO favors more recent entries. To compensate, I used a relatively low K value, which makes updates slower, and so old answer count for more.
The data is from this query, courtesy of Alex A. To run the code, put the two code blocks below in files with the names given, put the query csv file in ratings.csv, and run parse.py
New results, with bug fix
In the old version of the code, there were two bugs:
I have fixed both bugs. In doing so, the code now considers differently named versions of a language (e.g. Python 2.7 and Python 2) to be different languages. I don't think this is a major problem - allowing languages with numbers in the name is much more important.
Also, since many more answers are correctly parsed, more languages meet the threshold to be displayed.
September 2021 Results:
Husk: 1902
Jelly: 1880
Canvas: 1864
SOGL V0.12: 1802
Neim: 1799
Stax: 1790
Pyke: 1757
Japt: 1751
05AB1E: 1750
MathGolf: 1736
MATL: 1667
Brachylog: 1652
Ohm v2: 1632
Actually: 1626
Ohm: 1616
Gaia: 1614
Pyth: 1602
gs2: 1584
APL: 1580
Japt v2.0a0: 1553
Convex: 1551
Pushy: 1545
Pyt: 1538
Jolf: 1536
SOGL: 1527
Charcoal: 1526
Noodel: 1522
Pip: 1489
V: 1481
K: 1448
Gogh: 1442
Burlesque: 1433
J: 1432
CJam: 1428
Keg: 1389
K4: 1389
Dyalog APL: 1388
Jellyfish: 1376
Alice: 1372
𝔼𝕊𝕄𝕚𝕟: 1371
Bubblegum: 1363
Gema: 1359
QuadR: 1348
RProgN 2: 1340
Bash + coreutils: 1309
Raku: 1308
Sledgehammer: 1301
braingasm: 1300
cQuents: 1292
Stacked: 1289
Fuzzy Octo Guacamole: 1288
Clip: 1277
x86: 1276
Wonder: 1272
Microscript II: 1265
Vitsy: 1264
TI: 1263
GolfScript: 1263
Minkolang 0.15: 1260
Attache: 1260
Deorst: 1256
APL+WIN: 1243
Snails: 1233
Ahead: 1228
q/kdb+: 1227
Julia 0.4: 1220
Braingolf: 1215
Wumpus: 1210
Implicit: 1209
Backhand: 1203
Q: 1177
Befunge 93: 1174
SED: 1169
Retina 0.8.2: 1168
Hexagony: 1157
Cubix: 1156
Z80Golf: 1152
Wolfram Language: 1145
Ruby: 1141
Retina: 1140
Vim: 1139
Octave: 1138
GNU Sed: 1118
Julia 0.6: 1116
Pari/GP: 1113
Perl 6: 1110
QBIC: 1106
Dc: 1106
MY: 1101
Julia 0.5: 1099
Aceto: 1099
DUP: 1093
PowerShell: 1092
Proton: 1076
Add++: 1073
MATLAB / Octave: 1072
Mathematica: 1066
Cheddar: 1059
Befunge: 1059
BBC BASIC: 1059
Sage: 1054
Coconut: 1054
Prelude: 1052
Bash + Core utilities: 1050
x86 machine code: 1049
Scala: 1049
PostScript: 1048
Perl 6: 1048
sh: 1047
Labyrinth: 1046
zsh: 1043
Julia 1.0: 1035
SmileBASIC: 1034
Windows PowerShell: 1032
8th: 1027
Reticular: 1021
Perl: 1019
Röda: 1003
Funky: 1003
Bash: 998
Groovy: 995
Cubically: 994
Ruby 1.9: 993
Noether: 989
jq: 983
Bash + GNU utilities: 980
bash + Unix utilities: 978
Triangular: 978
Runic Enchantments: 975
Fourier: 974
Haskell: 973
JavaScript: 972
Perl 5: 965
LOGO: 960
MAWP: 956
Excel: 955
Fission: 954
Aheui: 943
R: 939
FEU: 935
bc: 929
shortC: 916
Rebol: 913
Julia: 911
Tcl/Tk: 909
FALSE: 904
Pure bash: 900
Shell: 887
Flurry: 886
Node.js: 883
WinDbg: 878
Python 3.8: 877
Python 3: 875
Python: 874
HTML+JavaScript: 874
Matlab: 872
Excel VBA: 872
JavaScript ES6: 869
Regex: 866
Marbelous: 864
Python 3.6: 863
Python 3.5: 863
Momema: 850
Yabasic: 846
PHP: 841
Python 2: 839
Chip: 838
CoffeeScript: 836
ES6: 829
Underload: 822
Hoon: 821
Erlang: 819
PowerShell v2+: 809
Red: 799
Google Sheets: 794
Crystal: 793
AppleScript: 783
Kotlin: 781
QBasic: 776
PowerShell v3+: 774
OCaml: 769
Rust: 765
VBA: 755
Java 8: 748
Forth: 746
uBASIC: 744
Processing: 737
Python 2.7: 735
Awk: 732
Symbolic Python: 729
Scheme: 729
Elixir: 721
Turing Machine Code: 719
Icon: 719
Factor: 712
Clojure: 712
Clean: 709
Decimal: 701
Delphi: 699
T: 698
Rexx: 698
SimpleTemplate: 695
Swift 3: 694
SAS: 694
Java: 693
C#: 693
PostgreSQL: 687
C#6: 686
Dart: 683
StandardML: 674
C: 672
Piet: 671
Java 10: 670
VBScript: 668
JS: 667
Operation Flashpoint scripting language: 659
SQL: 651
Java 11: 650
GameMaker Language: 649
Swift: 647
Lithp: 647
Common Lisp: 644
D: 643
Prolog: 638
TSQL: 634
Swift 4: 626
brainfuck: 623
Lua: 623
Batch: 615
tinylisp: 607
Windows Batch: 602
Oracle SQL: 599
LaTeX: 597
Racket: 579
tcl: 578
F#: 570
MBASIC: 551
Visual Basic .NET: 546
Pascal: 544
Pepe: 540
SNOBOL4: 529
Alchemist: 512
Brain: 508
S.I.L.O.S: 485
C++14: 483
LOLCODE: 482
ForceLang: 481
COW: 474
Oracle SQL 11.2: 470
Shakespeare Programming Language: 440
Java 7: 422
Brainetry: 420
C++: 417
Go: 393
Whitespace: 384
ArnoldC: 374
Taxi: 318
Code:
elo.py:
STARTING = 1000
SCALE = 400
K = 10
PAD = 50
import itertools
def concat(x):
return list(itertools.chain.from_iterable(x))
class Ratings:
def __init__(self, challenges):
langs = concat(challenges)
self.ratings = {lang: STARTING for lang in langs}
for challenge in challenges:
self.update_challenge(challenge)
def update_game(self, first, second, score):
f_rat = self.ratings[first]
s_rat = self.ratings[second]
f_exp = 1/(1 + 10 ** ((s_rat - f_rat) / 400))
s_exp = 1/(1 + 10 ** ((f_rat - s_rat) / 400))
f_upd = K * (score - f_exp)
s_upd = K * ((1 - score) - s_exp)
self.ratings[first] += f_upd
self.ratings[second] += s_upd
def update_challenge(self, scores):
for lang1 in sorted(scores):
for lang2 in sorted(scores):
if lang1 > lang2:
s1 = scores[lang1]
s2 = scores[lang2]
result = 1 if s1 < s2 else 0.5 if s1 == s2 else 0
self.update_game(lang1, lang2, result)
def __str__(self):
return '\n'.join(key + ':' +
' ' * (max(len(lang) + 1 for lang in self.ratings) - len(key)) + str(val)
for key, val in sorted(((lang, int(self.ratings[lang]))
for lang in self.ratings),
key=lambda x:x[::-1], reverse=True))
parse.py:
import csv
import re
import elo
CUTOFF = 10
lines = []
scores = {}
challenges = []
lang_counts = {}
canon_to_actual = {}
with open('ratings.csv') as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
lines.append((row['Body'], row['ParentId']))
count = 0
for line, challenge in lines:
line = re.sub("<s(trike)?>.*</s(trike)?>", "", line)
line = re.sub("<a[^>]*>", "", line)
line = re.sub("</a>", "", line)
match = re.match("<.*>(.*)</.*>", line)
if match:
line = match.group(1)
lang = re.match("([^-,\(:]*)", line)
if lang:
lang = lang.group(1).strip()
canon = lang.lower().replace(' ', '')
canon_to_actual[canon] = lang
score = re.search("(\d+)\s*\w*\)?$", line)
if score:
score = int(score.group(1))
c = scores.get(challenge, [])
c.append((canon, score))
scores[challenge] = c
lang_counts[canon] = lang_counts.get(canon, 0) + 1
for c in sorted(scores, key=int):
results = scores[c]
c_map = {}
for canon, score in results:
if lang_counts[canon] > CUTOFF and re.match("\w", canon):
lang = canon_to_actual[canon]
old_score = c_map.get(lang, score)
c_map[lang] = min(score, old_score)
challenges.append(c_map)
print(elo.Ratings(challenges))
Here are the original ratings, generated with the buggy code, for posterity.
2016 Results
Jelly: 1906
Pyth: 1711
GS: 1674
Japt: 1653
MATL: 1607
Dyalog APL: 1580
J: 1579
CJam: 1578
Jolf: 1559
Seriously: 1535
TeaScript: 1499
GolfScript: 1497
Pip: 1478
Burlesque: 1460
APL: 1443
O: 1427
Vitsy: 1424
𝔼𝕊𝕄𝕚𝕟: 1384
Vim: 1371
q: 1355
K: 1322
TI: 1301
sh: 1294
Octave: 1289
PARI/GP: 1272
dc: 1269
GNU sed: 1229
Retina: 1213
x: 1209
Bash + coreutils: 1207
Bash + GNU utilities: 1201
bc: 1189
golflua: 1187
Sed: 1182
HTML: 1168
Julia: 1165
QBasic: 1156
Sage: 1146
Ruby: 1124
AWK: 1101
ZSH: 1086
Mathematica: 1083
Gema: 1079
Bash: 1079
jq: 1060
Haskell: 1051
MATLAB: 1042
PowerShell v: 1028
Perl: 1028
BBC BASIC: 1022
Piet: 1019
Score: 1018
Shell script: 1017
Mumps: 1015
Languages: 1013
ECMAScript: 1012
Regex: 1000
LiveScript: 993
Tcl: 989
CoffeeScript: 988
Groovy: 986
Postscript: 984
Windows PowerShell: 981
Powershell: 966
R: 964
Befunge: 964
gawk: 951
AutoHotkey: 934
Javascript ES: 931
Smalltalk: 928
STATA: 926
Matlab/Octave: 926
REBOL: 923
SAS: 917
Clojure: 913
ES: 909
PHP: 908
Dart: 908
Python: 894
JavaScript: 894
Node.js: 893
F#: 880
Marbelous: 875
Excel VBA: 858
AutoIt: 853
Shell: 850
Common Lisp: 845
Brainfuck: 843
Lua: 835
Processing: 832
Scheme: 824
CSS: 823
Pure bash: 821
SWI: 786
Game Maker Language: 785
FORTH: 778
SpecBAS: 776
Scala: 772
Cobra: 769
HTML + JavaScript: 758
Whitespace: 750
Applescript: 739
T: 724
Batch: 720
Fortran: 715
Prolog: 711
Turing Machine Code: 708
C: 706
SQL: 704
VBA: 703
Factor: 700
Kotlin: 690
JS: 686
Pascal: 685
Erlang: 674
Emacs Lisp: 673
OCaml: 668
Racket: 658
VBScript: 654
VB.NET: 643
Swift: 636
D: 621
Windows Batch: 612
Ceylon: 591
C++: 587
Java: 559
Oracle SQL: 548
Delphi: 539
Go: 537
C#: 535
Rust: 528
"Hello, World!"(with or without the trailing") is a valid program in dozens of languages. Or ones where a particular language has a clear built-in that leapfrogs it ahead of where it would "normally" sit. Challenges like that would skew the numbers incorrectly. Not that I'm against the idea, it just needs a lot of rigor. \$\endgroup\$