Elo ratings, treating pairs of submissions as games.
I'm using the Elo system here.
Essentially, the solution is find the best solutions in each language to each challenge, and then treat each pair of such solutions as a match between those two languages, with the winner being the shorter submission.
Languages with less than 10 entries are discarded, and languages have a starting rating of 1000.
One drawback of this system is that ELO favors more recent entries. To compensate, I used a relatively low K
value, which makes updates slower, and so old answer count for more.
The data is from this query, courtesy of Alex A. To run the code, put the two code blocks below in files with the names given, put the query csv file in ratings.csv, and run parse.py
New results, with bug fix
In the old version of the code, there were two bugs:
I have fixed both bugs. In doing so, the code now considers differently named versions of a language (e.g. Python 2.7 and Python 2) to be different languages. I don't think this is a major problem - allowing languages with numbers in the name is much more important.
Also, since many more answers are correctly parsed, more languages meet the threshold to be displayed.
Feburary 2021 Results:
Jelly: 1921
05AB1E: 1900
Stax: 1853
MATL: 1831
Neim: 1807
GS2: 1781
Husk: 1778
Japt: 1766
Canvas: 1764
Gaia: 1748
SOGL V0.12: 1743
Pyke: 1736
Brachylog: 1703
MathGolf: 1700
Actually: 1697
Ohm v2: 1689
2sable: 1625
Jstx: 1622
Convex: 1610
Sesos: 1604
Pyth: 1602
Jolf: 1602
APL: 1593
Implicit: 1586
RProgN 2: 1574
Ohm: 1566
V: 1536
Noodel: 1483
GolfScript: 1476
k: 1466
CJam: 1466
Burlesque: 1466
Charcoal: 1456
J: 1440
Vitsy: 1428
Dyalog APL: 1425
cQuents: 1415
k4: 1411
x86 machine code: 1409
Keg: 1405
Befunge: 1393
𝔼𝕊𝕄𝕚𝕟: 1368
Carrot: 1361
Pushy: 1353
Japt v2.0a0: 1351
Pip: 1349
TeaScript: 1347
Retina: 1346
Recursiva: 1344
Alice: 1336
TI: 1331
APL+WIN: 1331
K5: 1308
Runic Enchantments: 1300
rs: 1296
Stacked: 1290
q/kdb+: 1286
Perl 6: 1281
Ahead: 1271
Attache: 1267
x86: 1247
Triangular: 1245
Braingolf: 1244
Wolfram Language: 1238
Jellyfish: 1233
Add++: 1224
Rebmu: 1212
Maple: 1210
Befunge 93: 1209
33: 1204
Julia 1.0: 1198
Z80Golf: 1192
O: 1187
Labyrinth: 1167
Ouroboros: 1163
sed: 1160
Beam: 1154
Hexagony: 1147
Cascade: 1135
Bubblegum: 1128
q: 1126
dc: 1118
Julia: 1117
Snails: 1113
Octave: 1113
BBC Basic: 1113
Regex: 1105
jq: 1103
MUMPS: 1102
Octave / MATLAB: 1101
Vim: 1098
Ly: 1097
Cheddar: 1096
Gema: 1094
Rebol: 1089
Retina 0.8.2: 1087
PostScript: 1085
Proton: 1076
PARI/GP: 1069
R: 1066
QBIC: 1063
Matlab/Octave: 1060
MY: 1058
Tcl/Tk: 1056
bc: 1055
Perl: 1051
ink: 1043
Mathematica: 1039
Bash + GNU utilities: 1037
Julia 0.6: 1034
Ruby: 1031
Ruby 1.9: 1029
Sage: 1026
Bash + coreutils: 1026
8086 machine code: 1019
Bean: 1014
Zsh: 1008
GNU Sed: 1007
PowerShell: 1006
Windows PowerShell: 1004
Röda: 1000
PHP 7: 999
perl5.10: 997
Underload: 995
Element: 993
Scala: 992
ShapeScript: 990
PowerShell v3+: 987
Perl 5: 986
MATLAB: 982
Pure bash: 980
Cardinal: 978
Brian & Chuck: 976
Bash + Unix utilities: 976
Logo: 963
Fourier: 959
Erlang: 957
ES6: 951
JavaScript: 950
Cubix: 947
CoffeeScript: 938
SmileBASIC 3: 935
Bash: 923
SmileBASIC: 921
Gawk: 921
Excel: 913
Python 3.8: 909
Awk: 909
scheme: 906
PowerShell v2+: 904
Haskell: 900
AppleScript: 899
Groovy: 896
Excel VBA: 892
Google Sheets: 888
Python 3.6: 884
TXR Lisp: 867
Assembly: 865
ZX Spectrum BASIC: 864
JavaScript ES6: 852
TeX: 851
Red: 848
PHP: 842
Processing: 829
AutoHotkey: 828
HTML + JavaScript: 827
Python 2: 826
Dart: 826
Nim: 821
Python: 815
Java: 809
Python 2.x: 806
Python 3: 803
Python 3.5: 794
Icon: 788
Forth: 785
C99: 785
Chip: 782
SpecBAS: 780
AutoIt: 780
HTML: 776
ECMAScript 6: 774
Factor: 772
Rexx: 770
Node.js: 766
Kotlin: 760
C#: 760
C: 760
JS: 758
MarioLANG: 756
Cubically: 755
Rust: 747
Emacs Lisp: 747
Hoon: 745
AHK: 745
Clojure: 744
SQL: 740
VBScript: 733
Clean: 730
Swift 4: 729
SQF: 728
Cobra: 724
IBM/Lotus Notes Formula: 723
SAS: 722
Racket: 722
Java 8: 722
OCaml: 721
Pepe: 720
Game Maker Language: 712
Yabasic: 708
Brain: 704
Tcl: 701
Piet: 696
Prolog: 694
8th: 687
Turing Machine Code: 685
Batch: 685
Stata: 683
Brainfuck: 683
Small Basic: 680
CSS: 675
C# .NET: 670
Python 2.7: 653
Java 10: 648
Common Lisp: 643
SimpleTemplate: 638
SWI: 634
Elixir: 632
F#: 627
Hassium: 626
Lua: 620
Qbasic: 609
C++14: 608
VBA: 600
TSQL: 598
T: 594
VB.NET: 587
Java 7: 582
Delphi: 576
Go: 575
Ceylon: 574
Swift 3: 570
C++11: 562
Fortran: 558
OIL: 557
Windows Batch: 555
Swift: 543
Oracle SQL: 532
ListSharp: 519
Malbolge: 516
S.I.L.O.S: 514
Oracle SQL 11.2: 512
Seed: 503
D: 502
C++: 502
Whitespace: 499
LOLCODE: 490
Pascal: 471
SNOBOL4: 443
Taxi: 387
Emojicode: 379
Poetic: 327
Pyramid Scheme: 291
Shakespeare Programming Language: 265
Arnold C: 238
Code:
elo.py:
STARTING = 1000
SCALE = 400
K = 10
PAD = 50
import itertools
def concat(x):
return list(itertools.chain.from_iterable(x))
class Ratings:
def __init__(self, challenges):
langs = concat(challenges)
self.ratings = {lang: STARTING for lang in langs}
for challenge in challenges:
self.update_challenge(challenge)
def update_game(self, first, second, score):
f_rat = self.ratings[first]
s_rat = self.ratings[second]
f_exp = 1/(1 + 10 ** ((s_rat - f_rat) / 400))
s_exp = 1/(1 + 10 ** ((f_rat - s_rat) / 400))
f_upd = K * (score - f_exp)
s_upd = K * ((1 - score) - s_exp)
self.ratings[first] += f_upd
self.ratings[second] += s_upd
def update_challenge(self, scores):
for lang1 in sorted(scores):
for lang2 in sorted(scores):
if lang1 > lang2:
s1 = scores[lang1]
s2 = scores[lang2]
result = 1 if s1 < s2 else 0.5 if s1 == s2 else 0
self.update_game(lang1, lang2, result)
def __str__(self):
return '\n'.join(key + ':' +
' ' * (max(len(lang) + 1 for lang in self.ratings) - len(key)) + str(val)
for key, val in sorted(((lang, int(self.ratings[lang]))
for lang in self.ratings),
key=lambda x:x[::-1], reverse=True))
parse.py:
import csv
import re
import elo
CUTOFF = 10
lines = []
scores = {}
challenges = []
lang_counts = {}
canon_to_actual = {}
with open('ratings.csv') as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
lines.append((row['Body'], row['ParentId']))
count = 0
for line, challenge in lines:
line = re.sub("<s(trike)?>.*</s(trike)?>", "", line)
line = re.sub("<a[^>]*>", "", line)
line = re.sub("</a>", "", line)
match = re.match("<.*>(.*)</.*>", line)
if match:
line = match.group(1)
lang = re.match("([^-,\(:]*)", line)
if lang:
lang = lang.group(1).strip()
canon = lang.lower().replace(' ', '')
canon_to_actual[canon] = lang
score = re.search("(\d+)\s*\w*\)?$", line)
if score:
score = int(score.group(1))
c = scores.get(challenge, [])
c.append((canon, score))
scores[challenge] = c
lang_counts[canon] = lang_counts.get(canon, 0) + 1
for c in sorted(scores, key=int):
results = scores[c]
c_map = {}
for canon, score in results:
if lang_counts[canon] > CUTOFF and re.match("\w", canon):
lang = canon_to_actual[canon]
old_score = c_map.get(lang, score)
c_map[lang] = min(score, old_score)
challenges.append(c_map)
print(elo.Ratings(challenges))
Here are the original ratings, generated with the buggy code, for posterity.
2016 Results
Jelly: 1906
Pyth: 1711
GS: 1674
Japt: 1653
MATL: 1607
Dyalog APL: 1580
J: 1579
CJam: 1578
Jolf: 1559
Seriously: 1535
TeaScript: 1499
GolfScript: 1497
Pip: 1478
Burlesque: 1460
APL: 1443
O: 1427
Vitsy: 1424
𝔼𝕊𝕄𝕚𝕟: 1384
Vim: 1371
q: 1355
K: 1322
TI: 1301
sh: 1294
Octave: 1289
PARI/GP: 1272
dc: 1269
GNU sed: 1229
Retina: 1213
x: 1209
Bash + coreutils: 1207
Bash + GNU utilities: 1201
bc: 1189
golflua: 1187
Sed: 1182
HTML: 1168
Julia: 1165
QBasic: 1156
Sage: 1146
Ruby: 1124
AWK: 1101
ZSH: 1086
Mathematica: 1083
Gema: 1079
Bash: 1079
jq: 1060
Haskell: 1051
MATLAB: 1042
PowerShell v: 1028
Perl: 1028
BBC BASIC: 1022
Piet: 1019
Score: 1018
Shell script: 1017
Mumps: 1015
Languages: 1013
ECMAScript: 1012
Regex: 1000
LiveScript: 993
Tcl: 989
CoffeeScript: 988
Groovy: 986
Postscript: 984
Windows PowerShell: 981
Powershell: 966
R: 964
Befunge: 964
gawk: 951
AutoHotkey: 934
Javascript ES: 931
Smalltalk: 928
STATA: 926
Matlab/Octave: 926
REBOL: 923
SAS: 917
Clojure: 913
ES: 909
PHP: 908
Dart: 908
Python: 894
JavaScript: 894
Node.js: 893
F#: 880
Marbelous: 875
Excel VBA: 858
AutoIt: 853
Shell: 850
Common Lisp: 845
Brainfuck: 843
Lua: 835
Processing: 832
Scheme: 824
CSS: 823
Pure bash: 821
SWI: 786
Game Maker Language: 785
FORTH: 778
SpecBAS: 776
Scala: 772
Cobra: 769
HTML + JavaScript: 758
Whitespace: 750
Applescript: 739
T: 724
Batch: 720
Fortran: 715
Prolog: 711
Turing Machine Code: 708
C: 706
SQL: 704
VBA: 703
Factor: 700
Kotlin: 690
JS: 686
Pascal: 685
Erlang: 674
Emacs Lisp: 673
OCaml: 668
Racket: 658
VBScript: 654
VB.NET: 643
Swift: 636
D: 621
Windows Batch: 612
Ceylon: 591
C++: 587
Java: 559
Oracle SQL: 548
Delphi: 539
Go: 537
C#: 535
Rust: 528
"Hello, World!"
(with or without the trailing"
) is a valid program in dozens of languages. Or ones where a particular language has a clear built-in that leapfrogs it ahead of where it would "normally" sit. Challenges like that would skew the numbers incorrectly. Not that I'm against the idea, it just needs a lot of rigor. \$\endgroup\$ – AdmBorkBork Mar 25 '16 at 17:51