11
\$\begingroup\$

I was bored so I wrote a program in Mathematica to download all posts from the first 10 front pages (155 posts) and parse all entries to gather data from the languages and to make a top-list for code golfing.

The place is determined by the quotient of total bytes by posts, so it can be interpreted as bytes/post.

Here is the list.

Or, if you want to search for your language, have an alphabetical ordered one.

However, more interesting in my opinion is the comparison of the top 15 most popular languages by posts:

enter image description here

Also interesting are some graphs of the data:

enter image description here

enter image description here

enter image description here

The next graph is interesting. We see, that there is a clear gap between the data. I don't really know how this happens. (It reminds me very much of Sloane's Gap)

enter image description here

The code I used is far from perfect but does its job.

LoadCodeGolfPostLinks[index_]:=Module[{linkList,refinedLinkList},
linkList=Import["http://codegolf.stackexchange.com/questions?page="<>ToString[index]<>"&sort=votes","Hyperlinks"];
refinedLinkList=Select[linkList,If[StringLength[#]>44,StringTake[#,44]=="http://codegolf.stackexchange.com/questions/",False]&];
Select[refinedLinkList,(MatchQ[URLParse[#]["Path"],{"","questions",a_String?(IntegerQ@ToExpression[#]&),_String}])&]
]
LoadCodeGolfPostSource[link_]:=Module[{sideSource,sideMax},
sideSource=Import[link,"Source"];
sideMax=Max[{0}~Join~(ToExpression/@StringCases[sideSource,"title=\"go to page "~~a_?(IntegerQ[ToExpression[#]]&)~~"\""->a])];
If[sideMax>1,
sideSource=StringJoin[{sideSource}~Join~Table[Import[link<>"?page="<>ToString[i],"Source"],{i,2,sideMax}]]
];
sideSource
]
LoadEntries[source_]:=StringTrim/@(StringCases[source,"<div class=\"post-text\" itemprop=\"text\">"~~Whitespace~~("<h1>"|"<h2>"|"<h3>")~~e:Repeated[Except["<"|"\n"]]~~("</h1>"|"</h2>"|"</h3>")->e])
ParseEntrie[entrie_]:=Module[{languageNames,byteCount,centrie},
languageNames=(StringCases[entrie,Repeated[Except[" "|"\t"|","]]]);If[Length[languageNames]<1,
Null,centrie=StringDrop[entrie,StringLength[languageNames[[1]]]];
byteCount=ToExpression@(SelectFirst[StringCases[StringDelete[centrie,{"(",")"}]<>" bytes",count:Repeated[DigitCharacter]~~Whitespace...~~("bytes"|"characters"|"chars")->count,IgnoreCase->True],If[SyntaxQ[#],IntegerQ[ToExpression[#]],False]&]/.Missing["NotFound"]->(-1));If[IntegerQ[byteCount]&&byteCount>=0&&byteCount<10000,{StringTrim[StringTrim[languageNames[[1]],{":"}]],byteCount},
Null
]
]
]
MergeLC[lci_]:=Flatten/@(lci//.{"Wolfram"->"Mathematica","Brainf***"->"brainfuck","Brainf*ck"->"brainfuck","C++11"->"C++","C++14"->"C++","Python3.4.0b2"->"Python","AHK"->"AutoHokey","Windows"->"Batch","Bat"->"Batch","Visual"->"VBA",{a___,{"C/C++",b_},c___}:>{a,{"C",b},{"C++",b},c},{a___,{"C++/C",b_},c___}:>{a,{"C",b},{"C++",b},c}})
CountLC[lci_]:=Module[{languages,lMatches},
languages=DeleteDuplicates[lci[[All,1]]];
Function[{l},
lMatches=Select[lci,StringMatchQ[#[[1]],l,IgnoreCase->True]&];
{l,Total[lMatches[[All,2]]],Length[lMatches]}
]/@languages
]

Download code:

Monitor[
step="download links";
frontPages=10;
ll=Flatten[Table[step="download links n:"<>ToString[i]<>"/"<>ToString[frontPages];LoadCodeGolfPostLinks[i],{i,1,frontPages}]];
step="download source";
ss=Table[step="download source n:"<>ToString[i]<>"/"<>ToString[Length[ll]];LoadCodeGolfPostSource[ll[[i]]],{i,1,Length[ll]}];
,step]

Parsing code:

Monitor[
step="load entries";
ee=Flatten[LoadEntries/@ss];
step="parse entries";
lce=DeleteCases[ParseEntrie/@ee,Null];
step="merge language counts";
lc=MergeLC[lce];
step="count languages";
result=CountLC[lc];
step="calculate quotients";
quotientResult={#[[1]],#[[2]],#[[3]],N[#[[2]]/#[[3]]]}&/@result;
step="sort results";
resultData=SortBy[quotientResult,#[[4]]&];
,step]
resultData=DeleteDuplicatesBy[resultData,ToLowerCase[#[[1]]]&];

If you have any questions, you are free to ask them. Perhaps I can answer it.

| |
\$\endgroup\$
  • \$\begingroup\$ For the graph with the gap, how do you determine "place of language"? \$\endgroup\$ – Martin Ender Oct 6 '16 at 15:44
  • \$\begingroup\$ The place is determined by average bytes per post. I edited it in. \$\endgroup\$ – Julien Kluge Oct 6 '16 at 15:46
  • 10
    \$\begingroup\$ If you want to compare languages by size, please restrict it to answers in the [code-golf] tag. Other challenge types (koth, code-challenge) are more often answered in languages such as python, java, and c, which would inflate their scores over golf/eso-langs. They don't need to be inflated even more than they naturally are ;) \$\endgroup\$ – Geobits Oct 6 '16 at 16:02
  • 3
    \$\begingroup\$ I'm pretty sure your Sloane gap comes from the fact that the bottom curve is languages there were used exactly once and the next curve is languages that were used twice. (And the next, thinner line for languages used three times and then the really popular languages are all the noise at the top because they're used so often that it inflated their total byte count). \$\endgroup\$ – Martin Ender Oct 6 '16 at 21:43
  • \$\begingroup\$ Related: meta.codegolf.stackexchange.com/q/5079/25180 \$\endgroup\$ – jimmy23013 Oct 11 '16 at 3:29
  • \$\begingroup\$ Related challenge. \$\endgroup\$ – Stewie Griffin Oct 11 '16 at 11:30
  • \$\begingroup\$ Another related challenge and very nice graphs in this answer \$\endgroup\$ – Sanchises Oct 14 '16 at 10:43
  • 4
    \$\begingroup\$ This information isn't entirely useful if you're using the entire post to count bytes. There are plenty of answers which contain both a golfed and an ungolfed version, plus an explanation paragraph. \$\endgroup\$ – mbomb007 Oct 14 '16 at 18:58
  • 2
    \$\begingroup\$ I'm voting to close this question as off-topic because it isn't actually a question. \$\endgroup\$ – Mego Oct 17 '16 at 22:32
  • \$\begingroup\$ am I reading this right? the posts aren't even parsed to exclude the non-code blocks? \$\endgroup\$ – user60119 Oct 18 '16 at 23:41

Browse other questions tagged .