Elo ratings, treating pairs of submissions as games.
I'm using the Elo system here.
Essentially, the solution is find the best solutions in each language to each challenge, and then treat each pair of such solutions as a match between those two languages, with the winner being the shorter submission.
Languages with less than 10 entries are discarded, and languages have a starting rating of 1000.
One drawback of this system is that ELO favors more recent entries. To compensate, I used a relatively low K
value, which makes updates slower, and so old answer count for more.
The data is from this query, courtesy of Alex A. To run the code, put the two code blocks below in files with the names given, put the query csv file in ratings.csv, and run parse.py
New results, with bug fix
In the old version of the code, there were two bugs:
I have fixed both bugs. In doing so, the code now considers differently named versions of a language (e.g. Python 2.7 and Python 2) to be different languages. I don't think this is a major problem - allowing languages with numbers in the name is much more important.
Also, since many more answers are correctly parsed, more languages meet the threshold to be displayed.
March 2021 Results:
Husk: 1899
Jelly: 1877
SOGL V0.12: 1819
Canvas: 1816
Neim: 1812
Ohm v2: 1802
Stax: 1789
Gaia: 1747
Pyke: 1742
GS2: 1741
05AB1E: 1741
MathGolf: 1716
Japt: 1709
MATL: 1704
Pyt: 1653
Pyth: 1640
Brachylog: 1640
Actually: 1632
Jolf: 1570
Ohm: 1562
Japt v2.0a0: 1545
APL: 1533
Pip: 1532
Convex: 1529
SOGL: 1528
RProgN 2: 1509
Pushy: 1506
PlatyPar: 1484
Seriously: 1481
Implicit: 1453
k: 1435
TeaScript: 1433
CJam: 1433
Burlesque: 1425
Recursiva: 1422
gema: 1414
V: 1411
cQuents: 1409
Braingasm: 1409
Keg: 1396
Charcoal: 1396
Dyalog APL: 1392
Alice: 1386
J: 1368
Vitsy: 1347
Sledgehammer: 1337
𝔼𝕊𝕄𝕚𝕟: 1332
Retina: 1321
Candy: 1314
Bubblegum: 1295
oK: 1288
O: 1286
x86: 1276
Deorst: 1274
TI: 1271
Minkolang 0.14: 1270
Stacked: 1264
K4: 1251
APL+WIN: 1251
Perl 6: 1249
Microscript II: 1243
MY: 1235
GolfScript: 1231
Cubix: 1220
Ahead: 1220
q/kdb+: 1211
Backhand: 1211
Klein: 1202
Attache: 1195
Hexagony: 1189
q: 1188
PARI/GP: 1185
Z80Golf: 1183
Ruby: 1183
Raku: 1183
Labyrinth: 1180
Sh: 1169
sed: 1164
ARBLE: 1160
Braingolf: 1149
Noether: 1148
Minkolang 0.15: 1148
8086 Machine code: 1148
Mathematica: 1146
Befunge93: 1134
x86 machine code: 1126
Vim: 1123
dc: 1112
Octave: 1110
Add++: 1109
Wolfram Language: 1104
Triangular: 1097
MUMPS: 1088
Aceto: 1082
Röda: 1078
BBC BASIC: 1078
GNU sed: 1075
perl: 1068
Logo: 1063
Runic Enchantments: 1062
Julia 0.6: 1059
Befunge: 1057
QBIC: 1054
zsh: 1048
Retina 0.8.2: 1040
RProgN: 1036
Bash + Core utilities: 1032
bc: 1030
PowerShell: 1023
Bash+coreutils: 1019
Befunge 98: 1015
Arcyóu: 1008
Ruby 1.9: 1006
MATLAB / Octave: 1003
DUP: 998
R: 996
Cheddar: 995
ShapeScript: 994
Ly: 988
Perl 5: 985
Fourier: 980
jq: 978
Windows PowerShell: 975
Tcl/Tk: 975
Flobnar: 975
JavaScript: 973
Julia 1.0: 963
MATLAB: 959
Python 2: 957
Bash + GNU utilities: 953
6502 machine code: 953
Bash: 952
Chip: 951
Vim Script: 949
Python 3.5: 949
HTML: 941
AHK: 937
Proton: 931
Julia: 926
PowerShell v3+: 921
Applescript: 917
Google Sheets: 915
Jq 1.5: 912
Regex: 907
Haskell: 907
Cubically: 900
SmileBASIC: 898
Octave/Matlab: 897
PowerShell v2+: 896
Javascript ES6: 896
shortC: 891
Mouse: 886
Node.js: 885
Python 3.8: 880
JS: 876
Coffeescript: 871
Python: 865
Casio: 865
Python 3: 864
Funky: 864
IBM/Lotus Notes Formula: 863
Rebol: 855
Groovy: 853
Marbelous: 850
ES6: 847
Pure bash: 828
Excel: 827
Python 3.6: 822
PostScript: 820
43 bytes: 818
Casio BASIC: 816
Red: 811
Java: 808
Erlang: 808
Java 8: 802
Excel VBA: 790
C99: 784
Crystal: 775
PHP: 766
4: 763
HTML + JavaScript: 762
Cobra: 755
C: 751
Processing: 748
Swift 4: 741
Scala: 740
VBA: 737
Kotlin: 737
Brainfuck: 729
Clojure: 722
C#: 722
QBasic: 721
AWK: 721
naz: 711
Hassium: 711
Decimal: 708
Icon: 703
Delphi: 700
Turing Machine Code: 697
OCaml: 697
Factor: 696
C++14: 693
Standard ML: 690
Symbolic Python: 688
Fortran: 687
Forth: 684
Emacs Lisp: 684
VBSCRIPT: 680
REXX: 679
TSQL: 677
tcl: 662
Clean: 660
Prolog: 652
Java 10: 650
Dart: 635
Yabasic: 628
Scheme: 624
SQL: 622
SpecBAS: 614
Python 2.7: 612
Ceylon: 612
Oracle SQL: 609
VB.net: 607
T: 607
Common Lisp: 594
SAS: 593
LaTeX: 593
Java 11: 591
GameMaker Language: 590
Brain: 589
Rust: 576
Elixir: 574
D: 574
lua: 558
F#: 549
Windows Batch: 542
Racket: 539
Pepe: 536
Haxe: 530
Batch: 520
SNOBOL4: 519
Swift: 512
Small Basic: 507
Java 7: 495
MBASIC: 481
Oracle SQL 11.2: 465
Pascal: 417
LOLCODE: 413
S.I.L.O.S: 388
C++: 382
Go: 377
Whitespace: 341
ArnoldC: 331
Taxi: 193
Shakespeare Programming Language: 190
Code:
elo.py:
STARTING = 1000
SCALE = 400
K = 10
PAD = 50
import itertools
def concat(x):
return list(itertools.chain.from_iterable(x))
class Ratings:
def __init__(self, challenges):
langs = concat(challenges)
self.ratings = {lang: STARTING for lang in langs}
for challenge in challenges:
self.update_challenge(challenge)
def update_game(self, first, second, score):
f_rat = self.ratings[first]
s_rat = self.ratings[second]
f_exp = 1/(1 + 10 ** ((s_rat - f_rat) / 400))
s_exp = 1/(1 + 10 ** ((f_rat - s_rat) / 400))
f_upd = K * (score - f_exp)
s_upd = K * ((1 - score) - s_exp)
self.ratings[first] += f_upd
self.ratings[second] += s_upd
def update_challenge(self, scores):
for lang1 in sorted(scores):
for lang2 in sorted(scores):
if lang1 > lang2:
s1 = scores[lang1]
s2 = scores[lang2]
result = 1 if s1 < s2 else 0.5 if s1 == s2 else 0
self.update_game(lang1, lang2, result)
def __str__(self):
return '\n'.join(key + ':' +
' ' * (max(len(lang) + 1 for lang in self.ratings) - len(key)) + str(val)
for key, val in sorted(((lang, int(self.ratings[lang]))
for lang in self.ratings),
key=lambda x:x[::-1], reverse=True))
parse.py:
import csv
import re
import elo
CUTOFF = 10
lines = []
scores = {}
challenges = []
lang_counts = {}
canon_to_actual = {}
with open('ratings.csv') as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
lines.append((row['Body'], row['ParentId']))
count = 0
for line, challenge in lines:
line = re.sub("<s(trike)?>.*</s(trike)?>", "", line)
line = re.sub("<a[^>]*>", "", line)
line = re.sub("</a>", "", line)
match = re.match("<.*>(.*)</.*>", line)
if match:
line = match.group(1)
lang = re.match("([^-,\(:]*)", line)
if lang:
lang = lang.group(1).strip()
canon = lang.lower().replace(' ', '')
canon_to_actual[canon] = lang
score = re.search("(\d+)\s*\w*\)?$", line)
if score:
score = int(score.group(1))
c = scores.get(challenge, [])
c.append((canon, score))
scores[challenge] = c
lang_counts[canon] = lang_counts.get(canon, 0) + 1
for c in sorted(scores, key=int):
results = scores[c]
c_map = {}
for canon, score in results:
if lang_counts[canon] > CUTOFF and re.match("\w", canon):
lang = canon_to_actual[canon]
old_score = c_map.get(lang, score)
c_map[lang] = min(score, old_score)
challenges.append(c_map)
print(elo.Ratings(challenges))
Here are the original ratings, generated with the buggy code, for posterity.
2016 Results
Jelly: 1906
Pyth: 1711
GS: 1674
Japt: 1653
MATL: 1607
Dyalog APL: 1580
J: 1579
CJam: 1578
Jolf: 1559
Seriously: 1535
TeaScript: 1499
GolfScript: 1497
Pip: 1478
Burlesque: 1460
APL: 1443
O: 1427
Vitsy: 1424
𝔼𝕊𝕄𝕚𝕟: 1384
Vim: 1371
q: 1355
K: 1322
TI: 1301
sh: 1294
Octave: 1289
PARI/GP: 1272
dc: 1269
GNU sed: 1229
Retina: 1213
x: 1209
Bash + coreutils: 1207
Bash + GNU utilities: 1201
bc: 1189
golflua: 1187
Sed: 1182
HTML: 1168
Julia: 1165
QBasic: 1156
Sage: 1146
Ruby: 1124
AWK: 1101
ZSH: 1086
Mathematica: 1083
Gema: 1079
Bash: 1079
jq: 1060
Haskell: 1051
MATLAB: 1042
PowerShell v: 1028
Perl: 1028
BBC BASIC: 1022
Piet: 1019
Score: 1018
Shell script: 1017
Mumps: 1015
Languages: 1013
ECMAScript: 1012
Regex: 1000
LiveScript: 993
Tcl: 989
CoffeeScript: 988
Groovy: 986
Postscript: 984
Windows PowerShell: 981
Powershell: 966
R: 964
Befunge: 964
gawk: 951
AutoHotkey: 934
Javascript ES: 931
Smalltalk: 928
STATA: 926
Matlab/Octave: 926
REBOL: 923
SAS: 917
Clojure: 913
ES: 909
PHP: 908
Dart: 908
Python: 894
JavaScript: 894
Node.js: 893
F#: 880
Marbelous: 875
Excel VBA: 858
AutoIt: 853
Shell: 850
Common Lisp: 845
Brainfuck: 843
Lua: 835
Processing: 832
Scheme: 824
CSS: 823
Pure bash: 821
SWI: 786
Game Maker Language: 785
FORTH: 778
SpecBAS: 776
Scala: 772
Cobra: 769
HTML + JavaScript: 758
Whitespace: 750
Applescript: 739
T: 724
Batch: 720
Fortran: 715
Prolog: 711
Turing Machine Code: 708
C: 706
SQL: 704
VBA: 703
Factor: 700
Kotlin: 690
JS: 686
Pascal: 685
Erlang: 674
Emacs Lisp: 673
OCaml: 668
Racket: 658
VBScript: 654
VB.NET: 643
Swift: 636
D: 621
Windows Batch: 612
Ceylon: 591
C++: 587
Java: 559
Oracle SQL: 548
Delphi: 539
Go: 537
C#: 535
Rust: 528
"Hello, World!"
(with or without the trailing"
) is a valid program in dozens of languages. Or ones where a particular language has a clear built-in that leapfrogs it ahead of where it would "normally" sit. Challenges like that would skew the numbers incorrectly. Not that I'm against the idea, it just needs a lot of rigor. \$\endgroup\$ – AdmBorkBork Mar 25 '16 at 17:51