32
\$\begingroup\$

I'm Grace Note, a Community Manager at Stack Exchange.

We have a process for how sites go about graduating or sticking around in Beta, but as noted in the comments there, Code Golf is one of two sites that we do not have periodic "Site Evaluations" where we compare questions against the rest of the internet. Instead, we have silent internal reviews every now and then.

I've been asked by various folks, especially after recent attention given to the sister site Code Review (who equally has the same lack of periodic evaluations), how Code Golf is doing.

The internal review back in October I did, there were two points I brought up - the traffic had been pretty stagnant for three years, and the userbase was dedicated but rep distribution was a mess. In fact the site was... pretty mediocre in just about every department. The prospects were very much the opposite of positive.

Things have changed.


seems to be something to point at - you guys received a ginormous traffic boost from the birth of this tag. Prior to Christmas, the site's average traffic levels were, like, 500-1000 for the whole thousand plus days of lifetime. Then on December 28th, you guys jumped by over a factor of 30, and since then throughout January and February you guys have been maintaining an average closer to 5000 visits per day. By comparison, Code Review is also at that level and they're pretty happily marching on from that as well. This is a nice thing to see and we'll probably want to watch this for some more time to see how well this is sticking, but so far it is really sticking.

It was previously a concern of mine that there's a certain difficulty in generating new questions for the site, because not only must one create some idea of a puzzle, but also must make one that didn't intersect with the three years of questions built up already. The invention of a new puzzle type, and perhaps continued innovation in that department, helps a lot. The traffic and answer activity levels are great - I think question income could do some notches higher but we're still on a great rise from before.


On the user side of things, I pointed out this answer as sharing my concerns - there was a major issue with voting. Code Review has also had this issue and they're really powering through it, though they still have some distance to go.

To my surprise, wow you guys made a huge shift here. In these four months, six people crowned over 10,000 reputation, and you've got a very nice collection of users in the higher reputation levels. The rep distribution is also nice, and you have 14% of users between 500 and 1000 reputation, and an additional 14% of users above 1000 reputation. These are mostly still-active users, too, so there's a lot of engaged activity. Your meta activity has also exploded into one of the more active metas in recent weeks.

This is a site that lives off of engagement - whereas other sites deal with producing solutions to problems, this site's competitive nature means that it thrives when it has people actively engaging against each other. And that seems to have started to really kick up. The userbase size is something that could be said to be low, but not in a number that is problematic or anything.


The overall scenario is extremely positive. What was once a site that may've languished and sat in Beta status indefinitely, has returned with extreme strength and dedication to their task. All in all, much like I am extremely pleased with how Code Review shaped up when spurred to action, I'm very delighted at what this community has brought on their own to improve and beyond improve their progress.

\$\endgroup\$
  • 2
    \$\begingroup\$ Yay, thank you for this :-D now I don't have to pester you on posts and in chat! :-P So anyway, what could we be doing better? I see your two major concerns (voting and new challenges) and also that we're working on fixing them; should we just keep it up? Or is there anything else we should do? \$\endgroup\$ – Doorknob Mar 6 '14 at 16:07
  • \$\begingroup\$ Those were concerns in the past - not so much now (though obviously we don't want to just forget and laze on those matters as if it's senior year). For now, the main thing to do is keep up the good work. \$\endgroup\$ – Grace Note Mar 6 '14 at 16:11
  • 16
    \$\begingroup\$ Good work? Surely you joke? You don't seem to realise that the quality of this site has plummeted since the travesty that is [code-trolling] was unleashed. There are now just as many good questions as there were previously; the difference is that they get hidden amongst the extra trash - the seemingly endless rehashes of 'Hello, World!', quines, and 'print x without using y'. It makes me despair every time I visit. \$\endgroup\$ – Gareth Mar 6 '14 at 17:47
  • 1
    \$\begingroup\$ @Gareth Are rehashes of trivial tasks really a new thing around here? Even in the [code-golf] tag, there are a bunch of quine/hello-world types, and I can't imagine they're all really different enough to warrant separate questions. The main difference I see is that now they come in faster due to high traffic, but rehashes seem to be the inherent nature of a "post a puzzle" site. Just vote to close them as dupes, just as we do on SO when someone posts the millionth "Why don't these strings equal each other?" question. \$\endgroup\$ – Geobits Mar 6 '14 at 19:02
  • 4
    \$\begingroup\$ @Geobits No they're not new, they're just a lot more common since December. We do vote to close, but there are a number of users who instantly vote to reopen anything just to be contrarians. These are probably the people who earned lots of rep posting answers like this or this. \$\endgroup\$ – Gareth Mar 6 '14 at 22:36
  • 1
    \$\begingroup\$ @Gareth That's just due to more traffic, though, which is generally a good thing. Laying the blame on a single tag/post/incident seems naive, and I think any large increase in traffic is going to cause more low-quality posts, simply because you get a lot of new people who aren't familiar with the rules. I like the way Shog9 put it when people complained about "Hot" traffic on MSO: "You can work to fix that - as painful as that process is - or you can bury your head in the sand and blame it on all of those stupid people from elsewhere." \$\endgroup\$ – Geobits Mar 6 '14 at 22:44
  • \$\begingroup\$ FWIW, I see at least a couple posts on the front page right now that should probably(IMO) be closed, yet have zero close votes. I've held back on closing so far, since I'm new here and still trying to get a feel for the community. So far it seems to me that the community just doesn't moderate as much as it could, yet everyone on meta seems to have a gripe about the post quality. \$\endgroup\$ – Geobits Mar 6 '14 at 22:46
  • 1
    \$\begingroup\$ @Geobits We often can't close the rubbish questions because a lot of the 'more traffic' has gained enough reputation off the back of answers to rubbish no-intelligence-necessary questions to reopen them straight away. That's the problem. Please, if you see questions that should be closed, vote to close them. If you're unlucky you may get to see what I mean. \$\endgroup\$ – Gareth Mar 6 '14 at 23:02
  • 5
    \$\begingroup\$ @Geobits And to make matters worse, we've now got everyone upvoting anything and everything in some misguided attempt to boost our chances of graduation. I think that one of the best things that Grace Note and the team could do for us is to guarantee that we will never graduate. Leave us in perpetual beta so that people stop artificially inflating each other's reputation to meet some imaginary targets. \$\endgroup\$ – Gareth Mar 6 '14 at 23:05
  • 4
    \$\begingroup\$ @Gareth That was actually my reaction when I saw that thread, too. If anything, the site needs more downvoting, but then you see things like "upvoted to counter unfair downvote", and you wonder why you bother. The problem with perpetual beta, though, is that it keeps the VTC privilege at 500, where anyone can get it after basically no time. \$\endgroup\$ – Geobits Mar 6 '14 at 23:07
  • \$\begingroup\$ @Geobits, it was unfortunate that the large increase in traffic coincided with a time when, due to the holiday, there were fewer people around to moderate. \$\endgroup\$ – Peter Taylor Mar 6 '14 at 23:54
  • 3
    \$\begingroup\$ @hosch250, can you cite any source to back up "If we don't graduate, we will probably get closed"? I've never expected PCG to leave beta, but it seems to serve its purpose of keeping code-golf off SO without graduating. \$\endgroup\$ – Peter Taylor Mar 7 '14 at 10:02
  • 1
    \$\begingroup\$ We have been looking at the concept of the "perpetual beta" for a number of sites, and while the status of it hasn't quite been set, I'm pretty confident that Code Golf isn't close to the chopping block. \$\endgroup\$ – Grace Note Mar 7 '14 at 11:53
  • 2
    \$\begingroup\$ Sorry to bother you yet again, but I've been thinking: seriously, why hasn't Code Golf (and Code Review by extention) graduated yet? We have no major problems, and we do have much better stats than some sites like WebApps when it graduated. (I mean really, they only had 3 closers and their highest repped user had 5011.) \$\endgroup\$ – Doorknob Mar 20 '14 at 23:33
  • 1
    \$\begingroup\$ Oh, and we even both have better weekly and monthly rep than Super User, one of the most active SE sites! Honestly I see no reason why we haven't graduated already. \$\endgroup\$ – Doorknob Mar 20 '14 at 23:40
18
\$\begingroup\$

It was previously a concern of mine that there's a certain difficulty in generating new questions for the site, because not only must one create some idea of a puzzle, but also must make one that didn't intersect with the three years of questions built up already.

That is a genuine problem. On the other hand, the site only needs one or two good questions per day, because most good questions take at least a couple of hours to answer well, and there's three years' worth of questions for newcomers to tackle. (Of course, Sturgeon's Law applies, so getting only one or two questions per day would not be good enough).

The invention of a new puzzle type, and perhaps continued innovation in that department, helps a lot.

Really? There have only been 5 questions posted in the past month, of which one is closed (and the issue of whether all of them should be deleted divides the community). From where I sit it seems that the thing which helped a lot was having a question posted on Reddit when lots of people were on holiday.

The long-term effect seems to be that we tend to have one or two questions in the "hot network questions" list, and while they tend not to be the best questions my hypothesis is that they attract enough people to also draw eyeballs onto the next "hot" question. I don't know whether the data available to you is sufficient to estimate the effect of "hot" questions in the ongoing visitor stats.

\$\endgroup\$
  • 1
    \$\begingroup\$ I'm hoping to see more of such innovation, sorry if that wasn't clear from my verbiage. There's only 22 in total but they represented a turn of traffic, not just for their own purposes but for questions in general. You've had more questions per day asked of all categories in the last two months than you guys have had for years. Traffic boost was in much part be because of them turning up on Hot Network Questions, but from me I still feel that the pull of actual new question types is something that will be extremely helpful going forward to maintaining growth. \$\endgroup\$ – Grace Note Mar 6 '14 at 16:50
  • 1
    \$\begingroup\$ I see "more questions per day asked of all categories" as supporting my contention that it wasn't the new category but the timing of the Reddit post that was relevant, but I think that we may have to agree to disagree. \$\endgroup\$ – Peter Taylor Mar 6 '14 at 16:55
  • 1
    \$\begingroup\$ I could actually provide the numeral traffics that further support your side - year-to-date, the amount of traffic incoming from Stack Overflow is an order of magnitude above the rest, followed by Ask Ubuntu (...huh?) and then Reddit. I don't want to agree to disagree because I don't think we disagree. \$\endgroup\$ – Grace Note Mar 6 '14 at 17:06
  • 5
    \$\begingroup\$ @GraceNote While it is true that the amount of good/ok questions increased during the last months the number of not so good questions is overwhelming. Previously it was working for a well-designed challenge if it couldn't get an answer for a few days (which was ok, because good answers to great puzzles need some time). Such a question is now buried by an avalanche of questions and answers of the hello world type. That means that new users don't find the interesting puzzles and might be encouraged to also post interesting challenges but see this site as a heap of me-too questions/answers. \$\endgroup\$ – Howard Mar 7 '14 at 9:10
  • \$\begingroup\$ @Howard maybe a more thoughtful use of tagging will help with that matter. If all the "hello world"-type questions were tagged with something like an easy tag and more complex, interesting challenge with a hard tag, they wouldn't be really buried. There seems to be clearly several "levels" of code-golfers on the site so it might be a good thing that those "easy" questions exists so that golfers with lower skills can still participate (and eventually improve their skills and participate in the harder ones). \$\endgroup\$ – plannapus Mar 7 '14 at 10:02
  • \$\begingroup\$ Of course the easiness of a challenge being somewhat subjective would need ideally to be agreed on at the sandbox level. \$\endgroup\$ – plannapus Mar 7 '14 at 10:04
  • 3
    \$\begingroup\$ @plannapus It's more a question of good-bad than of easy-tricky. And I don't think we can establish the tags badly worded and unclearly specified. \$\endgroup\$ – Howard Mar 7 '14 at 10:30
  • \$\begingroup\$ @Howard fair enough. \$\endgroup\$ – plannapus Mar 7 '14 at 10:40
15
\$\begingroup\$

In my experience here, things are turned a bit upside down from other sites. Because making a good puzzle has turned out to be harder than solving it.

That is because--if your puzzle is going to be any good--you have to provide sample inputs and outputs. Which means most of the time you really should solve your puzzle before you post it. So you think one up, then solve it, then lay out the parameters...it's basically working as a CS professor for free. It is very unlike the other Q&A sites where the questioner doesn't do a whole lot of work relative to the answerers.

The maligned code-trolling tag, I think, is disliked because specifically it doesn't require that expected effort on the part of the asker.

It just seems way more fun to answer than to ask. Given the time to compose a creative question, you start getting disappointed with only a few answers, especially if people seem to be just solving your problem and not golfing it. Also by its very nature, there is no timeline for accepting answers. If you've invested a lot in defining a puzzle then why declare a winner if it closes your puzzle?

So there is an issue here about the fact that grafting puzzle/solution onto the SE model has a bit of a mismatch. I don't think it's correctible, which isn't a condemnation of the site in any way. It just means that it's always going to be kind of weird and policy needs to be laid out.

Thus I think it's still in beta until there's a larger amount of precedent set in policy space. But hopefully the ads will get run to drive a bit more traffic and awareness. This one cracks me up:

enter image description here

\$\endgroup\$
8
\$\begingroup\$

Let's go to the data

I've been a lurker, not a participant, for most of the time since I joined. So I think the best thing I can contribute to this discussion is use of my data-wrangling skills. Here are a few interesting queries. And bear with me, this post is beyond TL,DR

New Questions Analysis by Year/Month

This is a simple trace of each month's new question activity since 2011. There is a boom at the beginning, followed by a slow decline, a resurgence in early-to-mid-2012, another slow decline, and a recent megaboom in December, January, and February. This is just a high-level view so there's no attempt to exclude low-quality questions, anomalous questions, or anything like that. We'll get to that. What this does show is that traffic has ramped up enormously. Discussion questions: Has the 10-fold increase in new question views come at the cost of a 10-fold reduction in site quality? Has the 6- to 8-fold increase in new questions meant that 7/8 of those questions are bad?

year month AverageAnswerCount AverageNewQuestionViews NewQuestionCount NewQViewCount 
---- ----- ------------------ ----------------------- ---------------- ------------- 
2011 1     16                 2102                    50               105137        
2011 2     9                  976                     112              109389        
2011 3     10                 1266                    62               78492         
2011 4     9                  1356                    52               70559         
2011 5     6                  843                     37               31213         
2011 6     8                  1290                    35               45184         
2011 7     7                  1019                    29               29561         
2011 8     7                  1121                    31               34770         
2011 9     5                  744                     20               14899         
2011 10    9                  934                     9                8410          
2011 11    6                  823                     23               18932         
2011 12    9                  910                     39               35527         
2012 1     8                  1701                    37               62939         
2012 2     10                 1066                    17               18136         
2012 3     6                  693                     30               20798         
2012 4     9                  1195                    28               33467         
2012 5     9                  2611                    31               80966         
2012 6     7                  898                     36               32361         
2012 7     8                  766                     29               22227         
2012 8     6                  830                     36               29887         
2012 9     8                  919                     39               35844         
2012 10    5                  686                     39               26772         
2012 11    7                  696                     28               19507         
2012 12    7                  853                     23               19631         
2013 1     6                  885                     28               24807         
2013 2     4                  757                     34               25742         
2013 3     6                  502                     29               14572         
2013 4     10                 1084                    34               36868         
2013 5     5                  598                     29               17370         
2013 6     6                  539                     18               9707          
2013 7     8                  597                     24               14331         
2013 8     6                  975                     35               34137         
2013 9     5                  466                     30               13985         
2013 10    6                  486                     34               16537         
2013 11    8                  590                     46               27167         
2013 12    12                 2159                    93               200826        
2014 1     10                 1778                    205              364679        
2014 2     11                 1245                    184              229179        
2014 3     10                 261                     9                2355     

Where does this recent boost come from?

Let's look at the same stats, but broken down by tag. This shows tags that are high in one or more of these stats, over the past 3 months. Some notable items: code-golf is alive and well as a tag, commanding 39k, 265k(!!), and 105k new question views in Dec, Jan, and Feb, and dominating the new question submissions. Code-trolling makes a big splash in December, but a much smaller one in January and falls off as a factor in February, so perhaps it is not as big a portion of the recent surge as we thought based on Grace Note's report. It's interesting how much traffic hello-world is responsible for, and busy-beaver, and restricted-source. However, much of restricted-source can be attributed to one question: the 2014 question. Or can it? .......

year month TagName            AverageAnswersPerQuestion AverageNewQuestionViews NewQuestionCount NewQViewCount 
---- ----- ------------------ ------------------------- ----------------------- ---------------- ------------- 
2013 12    algorithm          14                        10694                   1                10694         
2013 12    ascii-art          11                        3513                    5                17566         
2013 12    code-challenge     13                        7418                    13               96439         
2013 12    code-golf          8                         744                     53               39454         
2013 12    code-trolling      38                        11032                   12               132384        
2013 12    math               6                         2852                    4                11411         
2013 12    popularity-contest 25                        6496                    21               136421        
2014 1     ascii-art          16                        3942                    10               39429         
2014 1     busy-beaver        59                        10730                   1                10730         
2014 1     code-challenge     7                         979                     24               23506         
2014 1     code-golf          10                        2038                    130              265058        
2014 1     code-trolling      15                        4398                    5                21992         
2014 1     hello-world        18                        2805                    7                19637         
2014 1     popularity-contest 12                        1723                    43               74121         
2014 1     random             38                        4846                    1                4846          
2014 1     regex-golf         25                        10816                   1                10816         
2014 1     regular-expression 6                         1948                    8                15585         
2014 1     restricted-source  60                        48557                   3                145673        
2014 1     string             8                         1800                    10               18002         
2014 2     ascii-art          7                         1244                    9                11200         
2014 2     bitwise            47                        2437                    1                2437          
2014 2     code-golf          11                        1022                    105              107356        
2014 2     graphical-output   12                        3451                    6                20707         
2014 2     hello-world        35                        11873                   2                23747         
2014 2     pi                 30                        2256                    2                4513          
2014 2     popularity-contest 14                        2028                    58               117676        
2014 2     quine              22                        5277                    1                5277     

But but but...what about Reddit? What about crappy popular questions like 2014?

OK, let's look at the top 5 tags for different stats over the past 3 months. But let's first assume that the big runaway success in each tag is perhaps too popular for its own good. It's an easy challenge that draws easy answers. OK, throw it away. Here's what we get:

TagName            NewQuestionViews 
------------------ ---------------- 
code-golf          278826           
popularity-contest 237419           
code-trolling      68359            
ascii-art          46089            
code-challenge     33297    

Where do the eyeballs go? Code Golf, and Popularity Contest. The latter is, I know, divisive. But I theorize that it reflects that many people who look at the site do not know any of the "good for golfing" languages. Popularity contest lets them participate in something they have a hope of winning. Thought question: Could we make more Atomic Code Golf questions to downplay the need for popularity contests? Could we invent other tags that have objective criteria without implicitly favoring the languages obsessed with brevity?

TagName            NewQuestions 
------------------ ------------ 
code-golf          285          
popularity-contest 124          
code-challenge     55           
string             25           
math               23         

It's notable here that people come to look at ascii-art, but when they make new questions, Math appears more often. Also, code-trolling shows similarly: fun to look at, but doesn't actually get that many questions created for it. Meanwhile, the core tags of code-golf, code-challenge, and popularity-contest show well for both views and new question growth, which seems to show that the site has a healthy core.

TagName            AvgQuestionViews 
------------------ ---------------- 
restricted-source  7724             
hello-world        3238             
code-trolling      3107             
ascii-art          2003             
popularity-contest 1914    

This is interesting. Even with the wildly anomalous 2014 question excluded, restricted source questions - the questions many love to hate - draw views per question that average well above most other types of question. Ascii-art is unsurprising, because people like to look when there are pictures, but restricted source is interesting. Perhaps people just love an arbitrary challenge - "Play a Paganini caprice with an arm tied behind your back!" However, I began to suspect there might be another superstar question skewing the results, so I edited the query to require 3 or more questions per tag and got this instead:

TagName            AvgQuestionViews 
------------------ ---------------- 
hello-world        3238             
code-trolling      3107             
ascii-art          2003             
popularity-contest 1914             
sorting            1137  

However so much we are tired of hello world questions, the tag has an audience.

TagName            AnswersPerQuestion 
------------------ ------------------ 
sorting            23                 
code-trolling      21                 
hello-world        20                 
popularity-contest 14                 
ascii-art          10   

This one is very interesting, too. Code-trolling has an obvious appeal that probably draws lots of answers because if you're not normally the sort to troll you will likely jump at the chance to do so consequence-free and actually gain reputation. But, sorting? Surprisingly good at attracting answers.

In Conclusion

I've thought out loud here about some of what I think the data might mean, but my analysis is by no means conclusive. Please sound off and discuss what we can see here. Please modify the queries, or request modifications from me if you don't know how. I'm interested in making data like this a valuable resource for the site.

I think it is at least safe to say that the numbers show a healthy site that is growing. Not all of that growth may be in the right direction for the site, but there's a visible healthy core that remains devoted to the site's founding purpose and has exploded in size alongside any recent innovations or anomalies.

Doin' alright,
Gettin' good scores
The future's so bright..
We gotta wear plus-fours!

\$\endgroup\$
  • \$\begingroup\$ In answer to the question "Are 7/8 of questions bad?", yes. That's to be expected. In fact, I reckon that about 7/8 of the questions from January which haven't been deleted are at best a bit naff, so the proportion taking those into account may exceed Sturgeon's 90%. \$\endgroup\$ – Peter Taylor Mar 8 '14 at 9:25
  • \$\begingroup\$ Two versions of exploring the Sturgeon hypothesis: data.stackexchange.com/codegolf/query/173590/… OR data.stackexchange.com/codegolf/query/173591/… \$\endgroup\$ – Jonathan Van Matre Mar 8 '14 at 13:31
  • \$\begingroup\$ To sum those queries up: only 68 out of 336 open questions created in January ever got a single downvote. (55/254 Feb, 34/108 Mar) If we assume a single heroic dissenter is enough to make a question bad, that still leaves the lion's share of the new content as being of sufficient quality that no one considered it bad enough to downvote. Where is the 7/8? \$\endgroup\$ – Jonathan Van Matre Mar 9 '14 at 16:09
  • \$\begingroup\$ 336? I used the Questions page, ordered by date, and found just over 200 undeleted questions from January. I went through them all to do a best-of, and Sturgeon's law is on the nail: about 20 of them were good questions, IMAO. As far as downvotes go, I have three observations: firstly, I don't think many people vote on every question: I upvote good ones, downvote the worst, and leave the merely naff ones alone; secondly, as Geobits commented above, we probably need to downvote more; and thirdly, when I vote to close I don't always downvote, and I don't go back to downvote ones which survive. \$\endgroup\$ – Peter Taylor Mar 9 '14 at 17:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .