Elo ratings, treating pairs of submissions as games.
I'm using the Elo system here.
Essentially, the solution is find the best solutions in each language to each challenge, and then treat each pair of such solutions as a match between those two languages, with the winner being the shorter submission.
Languages with less than 10 entries are discarded, and languages have a starting rating of 1000.
One drawback of this system is that ELO favors more recent entries. To compensate, I used a relatively low K
value, which makes updates slower, and so old answer count for more.
The data is from this query, courtesy of Alex A. To run the code, put the two code blocks below in files with the names given, put the query csv file in ratings.csv, and run parse.py
New results, with bug fix
In the old version of the code, there were two bugs:
I have fixed both bugs. In doing so, the code now considers differently named versions of a language (e.g. Python 2.7 and Python 2) to be different languages. I don't think this is a major problem - allowing languages with numbers in the name is much more important.
Also, since many more answers are correctly parsed, more languages meet the threshold to be displayed.
August 2023 Results:
Vyxal: 2053
Thunno 2: 2044
Nekomata: 2034
Jelly: 2028
Nibbles: 1997
Husk: 1926
SOGL V0.12: 1916
Neim: 1896
05AB1E: 1893
Stax: 1847
MathGolf: 1837
Pyke: 1831
MATL: 1825
Canvas: 1798
Japt v2.0a0: 1789
Japt: 1784
Pyth: 1780
Ohm: 1735
Gaia: 1706
Ohm v2: 1704
Pyt: 1703
Brachylog: 1702
Actually: 1700
Convex: 1647
Jolf: 1628
Arn: 1622
Seriously: 1575
APL: 1560
Pushy: 1546
Burlesque: 1531
Dyalog APL: 1529
J: 1508
Keg: 1503
CJam: 1497
V: 1483
Pip: 1476
cQuents: 1470
BQN: 1444
TeaScript: 1439
Charcoal: 1414
Haskell + hgl: 1391
x86 Machine Code: 1372
k: 1372
𝔼𝕊𝕄𝕚𝕟: 1366
K4: 1334
Attache: 1332
x86 32: 1326
stacked: 1313
Ly: 1310
Braingolf: 1284
Add++: 1283
x86: 1278
Golfscript: 1273
Labyrinth: 1261
q/kdb+: 1233
Aceto: 1218
APL+WIN: 1208
Jellyfish: 1184
Raku: 1178
PARI/GP: 1170
TI: 1167
Q: 1167
MY: 1166
DC: 1129
Regex: 1127
Julia 0.6: 1110
Runic Enchantments: 1105
Octave: 1100
Proton: 1085
Cubix: 1077
Wolfram Language: 1068
Factor: 1061
Retina: 1059
Matlab/Octave: 1058
Perl: 1053
QBIC: 1050
Fourier: 1046
R: 1044
Bubblegum: 1043
bash+coreutils: 1042
Ruby: 1031
Mathematica: 1028
Julia 1.0: 1026
Vim: 1024
Curry: 1015
Julia: 1013
Octave / Matlab: 1012
MAWP: 1011
JavaScript: 1011
Cheddar: 1010
jq: 983
Arturo: 983
Hexagony: 963
Perl 5: 949
Retina 0.8.2: 947
Röda: 944
befunge: 943
CoffeeScript: 919
Perl 6: 912
Alice: 908
GNU sed: 902
Postscript: 896
jq 1.5: 889
sed: 878
Funky: 870
PowerShell: 862
Groovy: 856
Matlab: 850
Python 3.8: 842
CHIP: 841
Python: 835
Python 3.5: 830
Clojure: 830
Bash: 828
Python 2: 825
Google Sheets: 824
Elixir: 815
Zsh: 814
Haskell: 803
Python 3: 802
Knight: 793
PowerShell Core: 790
Cubically: 790
Desmos: 788
AWK: 786
SmileBASIC: 773
Lexurgy: 767
ES6: 766
QBasic: 763
Python 3.6: 756
Java: 732
PHP: 731
Processing: 724
PowerShell v2+: 712
Excel: 711
Excel VBA: 707
MMIX: 695
Forth: 679
JavaScript ES6: 676
Java 8: 669
Prolog: 667
Rust: 652
Kotlin: 647
Red: 644
Icon: 640
C#: 638
Swift 4: 637
C: 634
REXX: 626
Clean: 623
Batch: 596
T: 581
SpecBAS: 580
Java 11: 580
8th: 580
Scala: 578
Lua: 573
Python 2.7: 566
Windows Batch: 563
naz: 547
Yabasic: 543
Dart: 543
Swift: 530
Brainfuck: 522
Common Lisp: 517
tinylisp: 498
Tcl: 490
Java 10: 486
Java 7: 480
TSQL: 476
F#: 475
VBA: 470
Nim: 458
Pepe: 455
Swift 3: 432
Racket: 427
C++14: 419
SWI: 415
D: 404
SNOBOL4: 382
Fortran: 374
C++: 357
Go: 355
Brain: 352
Oracle SQL 11.2: 349
Scratch: 339
Whitespace: 287
Rockstar: 286
Pascal: 282
Deadfish~: 105
Taxi: 53
Shakespeare Programming Language: 13
Editors note: The cutoff was set to 40
when generating the list, as it was the smallest cutoff (that was a multiple of 5) that didn't give any false positives like languages named 46 bytes
.
Code:
elo.py:
STARTING = 1000
SCALE = 400
K = 10
PAD = 50
import itertools
def concat(x):
return list(itertools.chain.from_iterable(x))
class Ratings:
def __init__(self, challenges):
langs = concat(challenges)
self.ratings = {lang: STARTING for lang in langs}
for challenge in challenges:
self.update_challenge(challenge)
def update_game(self, first, second, score):
f_rat = self.ratings[first]
s_rat = self.ratings[second]
f_exp = 1/(1 + 10 ** ((s_rat - f_rat) / 400))
s_exp = 1/(1 + 10 ** ((f_rat - s_rat) / 400))
f_upd = K * (score - f_exp)
s_upd = K * ((1 - score) - s_exp)
self.ratings[first] += f_upd
self.ratings[second] += s_upd
def update_challenge(self, scores):
for lang1 in sorted(scores):
for lang2 in sorted(scores):
if lang1 > lang2:
s1 = scores[lang1]
s2 = scores[lang2]
result = 1 if s1 < s2 else 0.5 if s1 == s2 else 0
self.update_game(lang1, lang2, result)
def __str__(self):
return '\n'.join(key + ':' +
' ' * (max(len(lang) + 1 for lang in self.ratings) - len(key)) + str(val)
for key, val in sorted(((lang, int(self.ratings[lang]))
for lang in self.ratings),
key=lambda x:x[::-1], reverse=True))
parse.py:
import csv
import re
import elo
CUTOFF = 40 # change as needed
lines = []
scores = {}
challenges = []
lang_counts = {}
canon_to_actual = {}
with open("merged.csv", encoding="utf-8") as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
lines.append((row["FirstLine"], row["ParentId"]))
count = 0
for line, challenge in lines:
line = re.sub("<s(trike)?>.*</s(trike)?>", "", line)
line = re.sub("<a[^>]*>", "", line)
line = re.sub("</a>", "", line)
match = re.match("<.*>(.*)</.*>", line)
if match:
line = match.group(1)
lang = re.match("([^-,\(:]*)", line)
if lang:
lang = lang.group(1).strip()
canon = lang.lower().replace(" ", "")
canon_to_actual[canon] = lang
score = re.findall(r"(\d+(?:\.\d+)?)[^0-9]+bytes?", line)
if score:
score = score[-1]
score = eval(score.lstrip("0") or "0")
c = scores.get(challenge, [])
c.append((canon, score))
scores[challenge] = c
lang_counts[canon] = lang_counts.get(canon, 0) + 1
for c in sorted(scores, key=int):
results = scores[c]
c_map = {}
for canon, score in results:
if lang_counts[canon] > CUTOFF and re.match("\w", canon):
lang = canon_to_actual[canon]
old_score = c_map.get(lang, score)
c_map[lang] = min(score, old_score)
challenges.append(c_map)
print(elo.Ratings(challenges))
Here are the original ratings, generated with the buggy code, for posterity.
2016 Results
Jelly: 1906
Pyth: 1711
GS: 1674
Japt: 1653
MATL: 1607
Dyalog APL: 1580
J: 1579
CJam: 1578
Jolf: 1559
Seriously: 1535
TeaScript: 1499
GolfScript: 1497
Pip: 1478
Burlesque: 1460
APL: 1443
O: 1427
Vitsy: 1424
𝔼𝕊𝕄𝕚𝕟: 1384
Vim: 1371
q: 1355
K: 1322
TI: 1301
sh: 1294
Octave: 1289
PARI/GP: 1272
dc: 1269
GNU sed: 1229
Retina: 1213
x: 1209
Bash + coreutils: 1207
Bash + GNU utilities: 1201
bc: 1189
golflua: 1187
Sed: 1182
HTML: 1168
Julia: 1165
QBasic: 1156
Sage: 1146
Ruby: 1124
AWK: 1101
ZSH: 1086
Mathematica: 1083
Gema: 1079
Bash: 1079
jq: 1060
Haskell: 1051
MATLAB: 1042
PowerShell v: 1028
Perl: 1028
BBC BASIC: 1022
Piet: 1019
Score: 1018
Shell script: 1017
Mumps: 1015
Languages: 1013
ECMAScript: 1012
Regex: 1000
LiveScript: 993
Tcl: 989
CoffeeScript: 988
Groovy: 986
Postscript: 984
Windows PowerShell: 981
Powershell: 966
R: 964
Befunge: 964
gawk: 951
AutoHotkey: 934
Javascript ES: 931
Smalltalk: 928
STATA: 926
Matlab/Octave: 926
REBOL: 923
SAS: 917
Clojure: 913
ES: 909
PHP: 908
Dart: 908
Python: 894
JavaScript: 894
Node.js: 893
F#: 880
Marbelous: 875
Excel VBA: 858
AutoIt: 853
Shell: 850
Common Lisp: 845
Brainfuck: 843
Lua: 835
Processing: 832
Scheme: 824
CSS: 823
Pure bash: 821
SWI: 786
Game Maker Language: 785
FORTH: 778
SpecBAS: 776
Scala: 772
Cobra: 769
HTML + JavaScript: 758
Whitespace: 750
Applescript: 739
T: 724
Batch: 720
Fortran: 715
Prolog: 711
Turing Machine Code: 708
C: 706
SQL: 704
VBA: 703
Factor: 700
Kotlin: 690
JS: 686
Pascal: 685
Erlang: 674
Emacs Lisp: 673
OCaml: 668
Racket: 658
VBScript: 654
VB.NET: 643
Swift: 636
D: 621
Windows Batch: 612
Ceylon: 591
C++: 587
Java: 559
Oracle SQL: 548
Delphi: 539
Go: 537
C#: 535
Rust: 528
"Hello, World!"
(with or without the trailing"
) is a valid program in dozens of languages. Or ones where a particular language has a clear built-in that leapfrogs it ahead of where it would "normally" sit. Challenges like that would skew the numbers incorrectly. Not that I'm against the idea, it just needs a lot of rigor. \$\endgroup\$