Elo ratings, treating pairs of submissions as games.
I'm using the Elo system here.
Essentially, the solution is find the best solutions in each language to each challenge, and then treat each pair of such solutions as a match between those two languages, with the winner being the shorter submission.
Languages with less than 10 entries are discarded, and languages have a starting rating of 1000.
One drawback of this system is that ELO favors more recent entries. To compensate, I used a relatively low K
value, which makes updates slower, and so old answer count for more.
The data is from this query, courtesy of Alex A. To run the code, put the two code blocks below in files with the names given, put the query csv file in ratings.csv, and run parse.py
New results, with bug fix
In the old version of the code, there were two bugs:
I have fixed both bugs. In doing so, the code now considers differently named versions of a language (e.g. Python 2.7 and Python 2) to be different languages. I don't think this is a major problem - allowing languages with numbers in the name is much more important.
Also, since many more answers are correctly parsed, more languages meet the threshold to be displayed.
January 2022 Results:
Jelly: 2064
05AB1E: 1998
SOGL V0.12: 1958
Gaia: 1949
Neim: 1922
MATL: 1920
Husk: 1912
Stax: 1904
gs2: 1899
Ohm v2: 1899
Pyke: 1893
Vyxal: 1882
Japt: 1845
MathGolf: 1842
Canvas: 1836
Japt v2.0a0: 1834
Pyt: 1792
Ohm: 1792
Brachylog: 1788
Actually: 1720
Pyth: 1713
Convex: 1688
MMIX: 1675
Arn: 1666
Jolf: 1661
QuadR: 1638
Seriously: 1608
Pip: 1602
RProgN 2: 1591
APL: 1590
CJam: 1550
Charcoal: 1544
Burlesque: 1538
k: 1530
Pushy: 1512
Keg: 1507
J: 1479
TeaScript: 1477
cQuents: 1467
Dyalog APL: 1452
x86 Machine Code: 1427
Golfscript: 1425
𝔼𝕊𝕄𝕚𝕟: 1411
Alice: 1405
Attache: 1391
x86: 1389
Bubblegum: 1374
K4: 1355
stacked: 1336
Vitsy: 1327
BRASCA: 1323
q/kdb+: 1297
Braingolf: 1296
Z80Golf: 1290
Labyrinth: 1281
Jellyfish: 1281
APL+WIN: 1277
Ahead: 1267
Aceto: 1262
O: 1257
Add++: 1234
Ruby: 1231
Raku: 1231
Octave: 1223
Vim: 1201
MY: 1189
Runic Enchantments: 1187
Ly: 1180
Wolfram Language: 1167
rs: 1155
LOGO: 1150
Hexagony: 1146
sed: 1141
Julia 0.6: 1135
Retina: 1134
Perl: 1128
DC: 1125
Q: 1123
MUMPS: 1120
Julia: 1120
V: 1116
Cubix: 1113
Proton: 1111
Julia 1.0: 1108
bash+coreutils: 1104
QBIC: 1091
shortC: 1090
Cheddar: 1078
Perl 6: 1077
convey: 1072
Zsh: 1062
Retina 0.8.2: 1061
Octave / Matlab: 1061
Matlab/Octave: 1061
MAWP: 1046
TI: 1030
Regex: 1015
Groovy: 1015
PARI/GP: 1014
Röda: 1013
JavaScript: 996
Fourier: 986
GNU sed: 983
Bash: 981
Factor: 975
Coconut: 971
Matlab: 968
PowerShell: 954
bash + Unix utilities: 953
Mathematica: 948
PowerShell Core: 943
jq 1.5: 938
R: 931
Google Sheets: 930
Perl 5: 929
Postscript: 920
Python 2: 917
Funky: 917
CoffeeScript: 917
Python 3.8: 892
jq: 887
CHIP: 881
befunge: 876
ES6: 871
SmileBASIC: 863
Mouse: 853
Python: 851
AWK: 851
Elixir: 847
Haskell: 844
Scala: 838
AHK: 835
Python 3.5: 831
Processing: 825
Python 3.6: 814
Cubically: 813
HTML: 806
Python 3: 802
PowerShell v2+: 791
Excel: 781
Clojure: 779
PHP: 775
PowerShell v3+: 762
Desmos: 754
JavaScript ES6: 749
Standard ML: 745
Pure bash: 741
Excel VBA: 740
Java: 734
Red: 727
Brainfuck: 725
C: 717
Icon: 699
QBasic: 697
Swift 4: 696
Scratch: 694
Forth: 693
C#: 692
Ink: 688
Hoon: 683
Clean: 679
REXX: 675
Kotlin: 667
Emacs Lisp: 650
8th: 630
T: 628
Java 8: 626
Game Maker Language: 625
SpecBAS: 619
Prolog: 611
Hassium: 603
Lua: 599
Dart: 599
Java 11: 597
naz: 592
Yabasic: 586
Rust: 585
Common Lisp: 571
SimpleTemplate: 562
Tcl: 561
Nim: 561
Swift: 554
Racket: 554
F#: 554
Python 2.7: 547
VBA: 545
Turing Machine Code: 541
TSQL: 540
C# 6: 536
Java 10: 533
tinylisp: 531
Batch: 530
Java 7: 529
Pepe: 525
MarioLANG: 522
D: 501
C++: 491
SQL: 489
C++11: 477
C++14: 469
Windows Batch: 468
VBScript: 454
Fortran: 452
SWI: 445
Swift 3: 438
Brain: 428
Oracle SQL: 425
SNOBOL4: 373
Oracle SQL 11.2: 369
S.I.L.O.S: 362
ForceLang: 356
Pascal: 346
Go: 323
Whitespace: 301
Rockstar: 259
Deadfish~: 255
Poetic: 144
Taxi: 92
ArnoldC: 87
Shakespeare Programming Language: 56
Code:
elo.py:
STARTING = 1000
SCALE = 400
K = 10
PAD = 50
import itertools
def concat(x):
return list(itertools.chain.from_iterable(x))
class Ratings:
def __init__(self, challenges):
langs = concat(challenges)
self.ratings = {lang: STARTING for lang in langs}
for challenge in challenges:
self.update_challenge(challenge)
def update_game(self, first, second, score):
f_rat = self.ratings[first]
s_rat = self.ratings[second]
f_exp = 1/(1 + 10 ** ((s_rat - f_rat) / 400))
s_exp = 1/(1 + 10 ** ((f_rat - s_rat) / 400))
f_upd = K * (score - f_exp)
s_upd = K * ((1 - score) - s_exp)
self.ratings[first] += f_upd
self.ratings[second] += s_upd
def update_challenge(self, scores):
for lang1 in sorted(scores):
for lang2 in sorted(scores):
if lang1 > lang2:
s1 = scores[lang1]
s2 = scores[lang2]
result = 1 if s1 < s2 else 0.5 if s1 == s2 else 0
self.update_game(lang1, lang2, result)
def __str__(self):
return '\n'.join(key + ':' +
' ' * (max(len(lang) + 1 for lang in self.ratings) - len(key)) + str(val)
for key, val in sorted(((lang, int(self.ratings[lang]))
for lang in self.ratings),
key=lambda x:x[::-1], reverse=True))
parse.py:
import csv
import re
import elo
CUTOFF = 10
lines = []
scores = {}
challenges = []
lang_counts = {}
canon_to_actual = {}
with open('ratings.csv') as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
lines.append((row['Body'], row['ParentId']))
count = 0
for line, challenge in lines:
line = re.sub("<s(trike)?>.*</s(trike)?>", "", line)
line = re.sub("<a[^>]*>", "", line)
line = re.sub("</a>", "", line)
match = re.match("<.*>(.*)</.*>", line)
if match:
line = match.group(1)
lang = re.match("([^-,\(:]*)", line)
if lang:
lang = lang.group(1).strip()
canon = lang.lower().replace(' ', '')
canon_to_actual[canon] = lang
score = re.search("(\d+)\s*\w*\)?$", line)
if score:
score = int(score.group(1))
c = scores.get(challenge, [])
c.append((canon, score))
scores[challenge] = c
lang_counts[canon] = lang_counts.get(canon, 0) + 1
for c in sorted(scores, key=int):
results = scores[c]
c_map = {}
for canon, score in results:
if lang_counts[canon] > CUTOFF and re.match("\w", canon):
lang = canon_to_actual[canon]
old_score = c_map.get(lang, score)
c_map[lang] = min(score, old_score)
challenges.append(c_map)
print(elo.Ratings(challenges))
Here are the original ratings, generated with the buggy code, for posterity.
2016 Results
Jelly: 1906
Pyth: 1711
GS: 1674
Japt: 1653
MATL: 1607
Dyalog APL: 1580
J: 1579
CJam: 1578
Jolf: 1559
Seriously: 1535
TeaScript: 1499
GolfScript: 1497
Pip: 1478
Burlesque: 1460
APL: 1443
O: 1427
Vitsy: 1424
𝔼𝕊𝕄𝕚𝕟: 1384
Vim: 1371
q: 1355
K: 1322
TI: 1301
sh: 1294
Octave: 1289
PARI/GP: 1272
dc: 1269
GNU sed: 1229
Retina: 1213
x: 1209
Bash + coreutils: 1207
Bash + GNU utilities: 1201
bc: 1189
golflua: 1187
Sed: 1182
HTML: 1168
Julia: 1165
QBasic: 1156
Sage: 1146
Ruby: 1124
AWK: 1101
ZSH: 1086
Mathematica: 1083
Gema: 1079
Bash: 1079
jq: 1060
Haskell: 1051
MATLAB: 1042
PowerShell v: 1028
Perl: 1028
BBC BASIC: 1022
Piet: 1019
Score: 1018
Shell script: 1017
Mumps: 1015
Languages: 1013
ECMAScript: 1012
Regex: 1000
LiveScript: 993
Tcl: 989
CoffeeScript: 988
Groovy: 986
Postscript: 984
Windows PowerShell: 981
Powershell: 966
R: 964
Befunge: 964
gawk: 951
AutoHotkey: 934
Javascript ES: 931
Smalltalk: 928
STATA: 926
Matlab/Octave: 926
REBOL: 923
SAS: 917
Clojure: 913
ES: 909
PHP: 908
Dart: 908
Python: 894
JavaScript: 894
Node.js: 893
F#: 880
Marbelous: 875
Excel VBA: 858
AutoIt: 853
Shell: 850
Common Lisp: 845
Brainfuck: 843
Lua: 835
Processing: 832
Scheme: 824
CSS: 823
Pure bash: 821
SWI: 786
Game Maker Language: 785
FORTH: 778
SpecBAS: 776
Scala: 772
Cobra: 769
HTML + JavaScript: 758
Whitespace: 750
Applescript: 739
T: 724
Batch: 720
Fortran: 715
Prolog: 711
Turing Machine Code: 708
C: 706
SQL: 704
VBA: 703
Factor: 700
Kotlin: 690
JS: 686
Pascal: 685
Erlang: 674
Emacs Lisp: 673
OCaml: 668
Racket: 658
VBScript: 654
VB.NET: 643
Swift: 636
D: 621
Windows Batch: 612
Ceylon: 591
C++: 587
Java: 559
Oracle SQL: 548
Delphi: 539
Go: 537
C#: 535
Rust: 528
"Hello, World!"
(with or without the trailing"
) is a valid program in dozens of languages. Or ones where a particular language has a clear built-in that leapfrogs it ahead of where it would "normally" sit. Challenges like that would skew the numbers incorrectly. Not that I'm against the idea, it just needs a lot of rigor. \$\endgroup\$