What does Code Golf's progress look like right now?

Question

I'm Grace Note, a Community Manager at Stack Exchange.

We have a process for how sites go about graduating or sticking around in Beta, but as noted in the comments there, Code Golf is one of two sites that we do not have periodic "Site Evaluations" where we compare questions against the rest of the internet. Instead, we have silent internal reviews every now and then.

I've been asked by various folks, especially after recent attention given to the sister site Code Review (who equally has the same lack of periodic evaluations), how Code Golf is doing.

The internal review back in October I did, there were two points I brought up - the traffic had been pretty stagnant for three years, and the userbase was dedicated but rep distribution was a mess. In fact the site was... pretty mediocre in just about every department. The prospects were very much the opposite of positive.

Things have changed.

---

code-trolling seems to be something to point at - you guys received a ginormous traffic boost from the birth of this tag. Prior to Christmas, the site's average traffic levels were, like, 500-1000 for the whole thousand plus days of lifetime. Then on December 28th, you guys jumped by over a factor of 30, and since then throughout January and February you guys have been maintaining an average closer to 5000 visits per day. By comparison, Code Review is also at that level and they're pretty happily marching on from that as well. This is a nice thing to see and we'll probably want to watch this for some more time to see how well this is sticking, but so far it is really sticking.

It was previously a concern of mine that there's a certain difficulty in generating new questions for the site, because not only must one create some idea of a puzzle, but also must make one that didn't intersect with the three years of questions built up already. The invention of a new puzzle type, and perhaps continued innovation in that department, helps a lot. The traffic and answer activity levels are great - I think question income could do some notches higher but we're still on a great rise from before.

---

On the user side of things, I pointed out this answer as sharing my concerns - there was a major issue with voting. Code Review has also had this issue and they're really powering through it, though they still have some distance to go.

To my surprise, wow you guys made a huge shift here. In these four months, six people crowned over 10,000 reputation, and you've got a very nice collection of users in the higher reputation levels. The rep distribution is also nice, and you have 14% of users between 500 and 1000 reputation, and an additional 14% of users above 1000 reputation. These are mostly still-active users, too, so there's a lot of engaged activity. Your meta activity has also exploded into one of the more active metas in recent weeks.

This is a site that lives off of engagement - whereas other sites deal with producing solutions to problems, this site's competitive nature means that it thrives when it has people actively engaging against each other. And that seems to have started to really kick up. The userbase size is something that could be said to be low, but not in a number that is problematic or anything.

---

The overall scenario is extremely positive. What was once a site that may've languished and sat in Beta status indefinitely, has returned with extreme strength and dedication to their task. All in all, much like I am extremely pleased with how Code Review shaped up when spurred to action, I'm very delighted at what this community has brought on their own to improve and beyond improve their progress.

Answer 1

It was previously a concern of mine that there's a certain difficulty in generating new questions for the site, because not only must one create some idea of a puzzle, but also must make one that didn't intersect with the three years of questions built up already.

That is a genuine problem. On the other hand, the site only needs one or two good questions per day, because most good questions take at least a couple of hours to answer well, and there's three years' worth of questions for newcomers to tackle. (Of course, Sturgeon's Law applies, so getting only one or two questions per day would not be good enough).

The invention of a new puzzle type, and perhaps continued innovation in that department, helps a lot.

Really? There have only been 5 code-trolling questions posted in the past month, of which one is closed (and the issue of whether all of them should be deleted divides the community). From where I sit it seems that the thing which helped a lot was having a question posted on Reddit when lots of people were on holiday.

The long-term effect seems to be that we tend to have one or two questions in the "hot network questions" list, and while they tend not to be the best questions my hypothesis is that they attract enough people to also draw eyeballs onto the next "hot" question. I don't know whether the data available to you is sufficient to estimate the effect of "hot" questions in the ongoing visitor stats.

Answer 2

In my experience here, things are turned a bit upside down from other sites. Because making a good puzzle has turned out to be harder than solving it.

That is because--if your puzzle is going to be any good--you have to provide sample inputs and outputs. Which means most of the time you really should solve your puzzle before you post it. So you think one up, then solve it, then lay out the parameters...it's basically working as a CS professor for free. It is very unlike the other Q&A sites where the questioner doesn't do a whole lot of work relative to the answerers.

The maligned code-trolling tag, I think, is disliked because specifically it doesn't require that expected effort on the part of the asker.

It just seems way more fun to answer than to ask. Given the time to compose a creative question, you start getting disappointed with only a few answers, especially if people seem to be just solving your problem and not golfing it. Also by its very nature, there is no timeline for accepting answers. If you've invested a lot in defining a puzzle then why declare a winner if it closes your puzzle?

So there is an issue here about the fact that grafting puzzle/solution onto the SE model has a bit of a mismatch. I don't think it's correctible, which isn't a condemnation of the site in any way. It just means that it's always going to be kind of weird and policy needs to be laid out.

Thus I think it's still in beta until there's a larger amount of precedent set in policy space. But hopefully the ads will get run to drive a bit more traffic and awareness. This one cracks me up:

Answer 3

Let's go to the data

I've been a lurker, not a participant, for most of the time since I joined. So I think the best thing I can contribute to this discussion is use of my data-wrangling skills. Here are a few interesting queries. And bear with me, this post is beyond TL,DR

New Questions Analysis by Year/Month

This is a simple trace of each month's new question activity since 2011. There is a boom at the beginning, followed by a slow decline, a resurgence in early-to-mid-2012, another slow decline, and a recent megaboom in December, January, and February. This is just a high-level view so there's no attempt to exclude low-quality questions, anomalous questions, or anything like that. We'll get to that. What this does show is that traffic has ramped up enormously. Discussion questions: Has the 10-fold increase in new question views come at the cost of a 10-fold reduction in site quality? Has the 6- to 8-fold increase in new questions meant that 7/8 of those questions are bad?

year month AverageAnswerCount AverageNewQuestionViews NewQuestionCount NewQViewCount 
---- ----- ------------------ ----------------------- ---------------- ------------- 
2011 1     16                 2102                    50               105137        
2011 2     9                  976                     112              109389        
2011 3     10                 1266                    62               78492         
2011 4     9                  1356                    52               70559         
2011 5     6                  843                     37               31213         
2011 6     8                  1290                    35               45184         
2011 7     7                  1019                    29               29561         
2011 8     7                  1121                    31               34770         
2011 9     5                  744                     20               14899         
2011 10    9                  934                     9                8410          
2011 11    6                  823                     23               18932         
2011 12    9                  910                     39               35527         
2012 1     8                  1701                    37               62939         
2012 2     10                 1066                    17               18136         
2012 3     6                  693                     30               20798         
2012 4     9                  1195                    28               33467         
2012 5     9                  2611                    31               80966         
2012 6     7                  898                     36               32361         
2012 7     8                  766                     29               22227         
2012 8     6                  830                     36               29887         
2012 9     8                  919                     39               35844         
2012 10    5                  686                     39               26772         
2012 11    7                  696                     28               19507         
2012 12    7                  853                     23               19631         
2013 1     6                  885                     28               24807         
2013 2     4                  757                     34               25742         
2013 3     6                  502                     29               14572         
2013 4     10                 1084                    34               36868         
2013 5     5                  598                     29               17370         
2013 6     6                  539                     18               9707          
2013 7     8                  597                     24               14331         
2013 8     6                  975                     35               34137         
2013 9     5                  466                     30               13985         
2013 10    6                  486                     34               16537         
2013 11    8                  590                     46               27167         
2013 12    12                 2159                    93               200826        
2014 1     10                 1778                    205              364679        
2014 2     11                 1245                    184              229179        
2014 3     10                 261                     9                2355     

  

Where does this recent boost come from?

Let's look at the same stats, but broken down by tag. This shows tags that are high in one or more of these stats, over the past 3 months. Some notable items: code-golf is alive and well as a tag, commanding 39k, 265k(!!), and 105k new question views in Dec, Jan, and Feb, and dominating the new question submissions. Code-trolling makes a big splash in December, but a much smaller one in January and falls off as a factor in February, so perhaps it is not as big a portion of the recent surge as we thought based on Grace Note's report. It's interesting how much traffic hello-world is responsible for, and busy-beaver, and restricted-source. However, much of restricted-source can be attributed to one question: the 2014 question. Or can it? .......

year month TagName            AverageAnswersPerQuestion AverageNewQuestionViews NewQuestionCount NewQViewCount 
---- ----- ------------------ ------------------------- ----------------------- ---------------- ------------- 
2013 12    algorithm          14                        10694                   1                10694         
2013 12    ascii-art          11                        3513                    5                17566         
2013 12    code-challenge     13                        7418                    13               96439         
2013 12    code-golf          8                         744                     53               39454         
2013 12    code-trolling      38                        11032                   12               132384        
2013 12    math               6                         2852                    4                11411         
2013 12    popularity-contest 25                        6496                    21               136421        
2014 1     ascii-art          16                        3942                    10               39429         
2014 1     busy-beaver        59                        10730                   1                10730         
2014 1     code-challenge     7                         979                     24               23506         
2014 1     code-golf          10                        2038                    130              265058        
2014 1     code-trolling      15                        4398                    5                21992         
2014 1     hello-world        18                        2805                    7                19637         
2014 1     popularity-contest 12                        1723                    43               74121         
2014 1     random             38                        4846                    1                4846          
2014 1     regex-golf         25                        10816                   1                10816         
2014 1     regular-expression 6                         1948                    8                15585         
2014 1     restricted-source  60                        48557                   3                145673        
2014 1     string             8                         1800                    10               18002         
2014 2     ascii-art          7                         1244                    9                11200         
2014 2     bitwise            47                        2437                    1                2437          
2014 2     code-golf          11                        1022                    105              107356        
2014 2     graphical-output   12                        3451                    6                20707         
2014 2     hello-world        35                        11873                   2                23747         
2014 2     pi                 30                        2256                    2                4513          
2014 2     popularity-contest 14                        2028                    58               117676        
2014 2     quine              22                        5277                    1                5277     

But but but...what about Reddit? What about crappy popular questions like 2014?

OK, let's look at the top 5 tags for different stats over the past 3 months. But let's first assume that the big runaway success in each tag is perhaps too popular for its own good. It's an easy challenge that draws easy answers. OK, throw it away. Here's what we get:

TagName            NewQuestionViews 
------------------ ---------------- 
code-golf          278826           
popularity-contest 237419           
code-trolling      68359            
ascii-art          46089            
code-challenge     33297    

Where do the eyeballs go? Code Golf, and Popularity Contest. The latter is, I know, divisive. But I theorize that it reflects that many people who look at the site do not know any of the "good for golfing" languages. Popularity contest lets them participate in something they have a hope of winning. Thought question: Could we make more Atomic Code Golf questions to downplay the need for popularity contests? Could we invent other tags that have objective criteria without implicitly favoring the languages obsessed with brevity?

TagName            NewQuestions 
------------------ ------------ 
code-golf          285          
popularity-contest 124          
code-challenge     55           
string             25           
math               23         

It's notable here that people come to look at ascii-art, but when they make new questions, Math appears more often. Also, code-trolling shows similarly: fun to look at, but doesn't actually get that many questions created for it. Meanwhile, the core tags of code-golf, code-challenge, and popularity-contest show well for both views and new question growth, which seems to show that the site has a healthy core.

TagName            AvgQuestionViews 
------------------ ---------------- 
restricted-source  7724             
hello-world        3238             
code-trolling      3107             
ascii-art          2003             
popularity-contest 1914    

This is interesting. Even with the wildly anomalous 2014 question excluded, restricted source questions - the questions many love to hate - draw views per question that average well above most other types of question. Ascii-art is unsurprising, because people like to look when there are pictures, but restricted source is interesting. Perhaps people just love an arbitrary challenge - "Play a Paganini caprice with an arm tied behind your back!" However, I began to suspect there might be another superstar question skewing the results, so I edited the query to require 3 or more questions per tag and got this instead:

TagName            AvgQuestionViews 
------------------ ---------------- 
hello-world        3238             
code-trolling      3107             
ascii-art          2003             
popularity-contest 1914             
sorting            1137  

However so much we are tired of hello world questions, the tag has an audience.

TagName            AnswersPerQuestion 
------------------ ------------------ 
sorting            23                 
code-trolling      21                 
hello-world        20                 
popularity-contest 14                 
ascii-art          10   

This one is very interesting, too. Code-trolling has an obvious appeal that probably draws lots of answers because if you're not normally the sort to troll you will likely jump at the chance to do so consequence-free and actually gain reputation. But, sorting? Surprisingly good at attracting answers.

In Conclusion

I've thought out loud here about some of what I think the data might mean, but my analysis is by no means conclusive. Please sound off and discuss what we can see here. Please modify the queries, or request modifications from me if you don't know how. I'm interested in making data like this a valuable resource for the site.

I think it is at least safe to say that the numbers show a healthy site that is growing. Not all of that growth may be in the right direction for the site, but there's a visible healthy core that remains devoted to the site's founding purpose and has exploded in size alongside any recent innovations or anomalies.

Doin' alright,
Gettin' good scores
The future's so bright..
We gotta wear plus-fours!