Golfing Class of a language

Question

I feel like we may have enough data on this site to meaningfully break programming languages into classes.

What I mean by classes is akin to Go's ranking system:

If a player can win 90% of even games against a 2 kyu player, the AGA believes he or she is 1.33 ranks higher, the EGF believes (s)he is 2.42 ranks higher, and the IGS believes (s)he is more than 3 ranks higher.

We may need to break the competition down into tags (e.g., a kolmogorov-complexity puzzle may have a different ranking system than a grid puzzle (cough, Bubblegum).

The use case for this would be to more reasonably compare your success - If a Python answer beats one in J, the author should be pretty proud. If Pyth beats Java by just a bit, the Java author should also be pretty happy with themselves (or the Pyth author ashamed).

Would anyone be interested in this endeavor?

Answer 1

K-means clustering by code length

Note: This is just a first-pass attempt with a very simple method. There are certainly better ways to approach this but at least this is a start.

My approach here was to get all answers to code golf challenges without bonuses that had (at least somewhat) appropriately formatted post headers, parse out the language name and byte count from the header, perform k-means clustering on the sizes with 6 groups, and for each language select the cluster into which most of the sizes for that language fell. Languages which have been used fewer than 10 times on the site, as well as any answer with a byte count greater than 170, were excluded.

The clusters are listed in ascending order of the values of their cluster centers.

Group 1, mean 12.03099

APL            GS2             Rebmu
Arcyou         Hexagony        rs
AutoIt         J               Sed
Batch          Japt            Self
BC             Jelly           Seriously
Beeswax        Jolf            Stuck
Befunge        K               TeaScript
Brachylog      Labyrinth       TI-BASIC
Brainfuck      Marbelous       Vim
Burlesque      MATL            Vitsy
Candy          Microscript     zsh
Clip           Minkolang
DC             Mouse
DUP            O
Element        Piet
ESMin          Pip
Factor         Platypar
Fission        Prelude
Gema           Pyth
GolfScript     QBASIC

Group 2, mean 31.75217

><>
CJam
Dart
F#
GTB
JQ
Julia
KDB
Mathematica
Matlab
Milky Way
PARI/GP
Perl
Q
Rebol
Regex
Scheme
Shell
Snails

Group 3, mean 54.14769

Awk
Bash
Clojure
D
Foo
Forth
Game Maker Language
Hassium
Pascal
PowerShell
R
Retina
Ruby
Sclipting
Smalltalk
Stata

Group 4, mean 79.65123

BBC Basic        Scala
CoffeeScript     SQL
Common Lisp      Swift
CSS              TCL
Emacs Lisp       VB.NET
Fortran          VBScript
GolfLua
Groovy
Haskell
JavaScript
Lua
Mumps
PHP
Processing
Prolog
Python
Racket
Rust
Sage
SAS

Group 5, mean 109.9524

C
C++
Ceylon
Delphi
Erlang
Gawk
Java
Postscript
VBA

Group 6, mean 147.5168

AppleScript
C#
Cobra
Go
HTML
OCaml
SpecBAS
Whitespace

Thoughts on this grouping

This is roughly what one might expect, with the golfing languages in tier 1 and Java in tier 5. This particular grouping appears to be somewhat of a success in that regard. However, 6 was probably too many groups; 4 or 5 would likely be sufficient. Heirarchical clustering may be another interesting route for analysis since no number of classes is prespecified.

Disclaimer

This should not, under any circumstances, be used to impose any kind of language handicap or restriction. It's merely for the sake of interest and giving folks a group to mentally compare their scores to, should they feel the need.

---

For the curious

The data were collected via this Stack Exchange Data Explorer query. The analysis was done in R and is available via this Gist.

Answer 2

Elo ratings, treating pairs of submissions as games.

I'm using the Elo system here.

Essentially, the solution is find the best solutions in each language to each challenge, and then treat each pair of such solutions as a match between those two languages, with the winner being the shorter submission.

Languages with less than 10 entries are discarded, and languages have a starting rating of 1000.

One drawback of this system is that ELO favors more recent entries. To compensate, I used a relatively low K value, which makes updates slower, and so old answer count for more.

The data is from this query, courtesy of Alex A. To run the code, put the two code blocks below in files with the names given, put the query csv file in ratings.csv, and run parse.py

Results:

Jelly:                1906
Pyth:                 1711
GS:                   1674
Japt:                 1653
MATL:                 1607
Dyalog APL:           1580
J:                    1579
CJam:                 1578
Jolf:                 1559
Seriously:            1535
TeaScript:            1499
GolfScript:           1497
Pip:                  1478
Burlesque:            1460
APL:                  1443
O:                    1427
Vitsy:                1424
𝔼𝕊𝕄𝕚𝕟:                1384
Vim:                  1371
q:                    1355
K:                    1322
TI:                   1301
sh:                   1294
Octave:               1289
PARI/GP:              1272
dc:                   1269
GNU sed:              1229
Retina:               1213
x:                    1209
Bash + coreutils:     1207
Bash + GNU utilities: 1201
bc:                   1189
golflua:              1187
Sed:                  1182
HTML:                 1168
Julia:                1165
QBasic:               1156
Sage:                 1146
Ruby:                 1124
AWK:                  1101
ZSH:                  1086
Mathematica:          1083
Gema:                 1079
Bash:                 1079
jq:                   1060
Haskell:              1051
MATLAB:               1042
PowerShell v:         1028
Perl:                 1028
BBC BASIC:            1022
Piet:                 1019
Score:                1018
Shell script:         1017
Mumps:                1015
Languages:            1013
ECMAScript:           1012
Regex:                1000
LiveScript:           993
Tcl:                  989
CoffeeScript:         988
Groovy:               986
Postscript:           984
Windows PowerShell:   981
Powershell:           966
R:                    964
Befunge:              964
gawk:                 951
AutoHotkey:           934
Javascript ES:        931
Smalltalk:            928
STATA:                926
Matlab/Octave:        926
REBOL:                923
SAS:                  917
Clojure:              913
ES:                   909
PHP:                  908
Dart:                 908
Python:               894
JavaScript:           894
Node.js:              893
F#:                   880
Marbelous:            875
Excel VBA:            858
AutoIt:               853
Shell:                850
Common Lisp:          845
Brainfuck:            843
Lua:                  835
Processing:           832
Scheme:               824
CSS:                  823
Pure bash:            821
SWI:                  786
Game Maker Language:  785
FORTH:                778
SpecBAS:              776
Scala:                772
Cobra:                769
HTML + JavaScript:    758
Whitespace:           750
Applescript:          739
T:                    724
Batch:                720
Fortran:              715
Prolog:               711
Turing Machine Code:  708
C:                    706
SQL:                  704
VBA:                  703
Factor:               700
Kotlin:               690
JS:                   686
Pascal:               685
Erlang:               674
Emacs Lisp:           673
OCaml:                668
Racket:               658
VBScript:             654
VB.NET:               643
Swift:                636
D:                    621
Windows Batch:        612
Ceylon:               591
C++:                  587
Java:                 559
Oracle SQL:           548
Delphi:               539
Go:                   537
C#:                   535
Rust:                 528

Code:

elo.py:

STARTING = 1000
SCALE = 400
K = 10
PAD = 50

import itertools
def concat(x):
    return list(itertools.chain.from_iterable(x))

class Ratings:
    def __init__(self, challenges):
        langs = concat(challenges)
        self.ratings = {lang: STARTING for lang in langs}
        for challenge in challenges:
            self.update_challenge(challenge)

    def update_game(self, first, second, score):
        f_rat = self.ratings[first]
        s_rat = self.ratings[second]

        f_exp = 1/(1 + 10 ** ((s_rat - f_rat) / 400))
        s_exp = 1/(1 + 10 ** ((f_rat - s_rat) / 400))

        f_upd = K * (score - f_exp)
        s_upd = K * ((1 - score) - s_exp)

        self.ratings[first] += f_upd
        self.ratings[second] += s_upd

    def update_challenge(self, scores):
        for lang1 in sorted(scores):
            for lang2 in sorted(scores):
                if lang1 > lang2:
                    s1 = scores[lang1]
                    s2 = scores[lang2]
                    result = 1 if s1 < s2 else 0.5 if s1 == s2 else 0
                    self.update_game(lang1, lang2, result)

    def __str__(self):
        return '\n'.join(key + ':' +
                ' ' * (max(len(lang) + 1 for lang in self.ratings) - len(key)) + str(val)
                         for key, val in sorted(((lang, int(self.ratings[lang]))
                             for lang in self.ratings),
                             key=lambda x:x[::-1], reverse=True))

parse.py:

import csv
import re
import elo

CUTOFF = 10

lines = []
scores = {}
challenges = []
lang_counts = {}
canon_to_actual = {}

with open('ratings.csv') as csvfile:
    reader = csv.DictReader(csvfile)
    for row in reader:
        lines.append((row['Body'], row['ParentId']))

for line, challenge in lines:
    line = re.sub("<s(trike)?>.*</s(trike)?>", "", line)
    match = re.match("<.*>(.*)</.*>", line)
    if match:
        line = match.group(1)
        lang = re.match("([^-,\(\d:]*)", line)
        if lang:
            lang = lang.group(1).strip()
            canon = lang.lower().replace(' ', '')
            canon_to_actual[canon] = lang
            score = re.search("(\d+)\s*\w*\)?$", line)
            if score:
                score = int(score.group(1))
                c = scores.get(challenge, [])
                c.append((canon, score))
                scores[challenge] = c
                lang_counts[canon] = lang_counts.get(canon, 0) + 1

for c in sorted(scores, key=int):
    results = scores[c]
    c_map = {}
    for canon, score in results:
        if lang_counts[canon] > CUTOFF and re.match("\w", canon):
            lang = canon_to_actual[canon]
            old_score = c_map.get(lang, score)
            c_map[lang] = min(score, old_score)
    challenges.append(c_map)

print(elo.Ratings(challenges))