Answer-dependent questions

Question

Calvin's Hobbies recently posted Evolution of Hello World. The challenge is unique in several ways, but the most revolutionary is that the definition of "valid answer" depends on other answers.

This inspired several more answer-dependent¹ questions (they're still in the sandbox). However, as discussed a bit in chat, these questions have some problems. For example, the finding the relevant answer(s) can be tricky, so it's not easy to figure out what needs to be done.

What do you think? Should² answer-dependent questions be encouraged?

---

¹ These challenges have sometimes been called domino-coding. However, not all answer-dependent questions involve a chain reaction. One idea mentioned was a tree, where answers can depend on any previous answer.

² I'm not really asking about the original Evolution of Hello World challenge. I think it's interesting and unique enough that it easily outweighs any of the genre's problems. However, I'm much less certain about having dozens more of these questions.

Answer 1

I would really like to make this challenge type work. (Full disclosure: partly because I seem to have come up with with a fairly popular challenge idea based on it. Mostly, though, because I genuinely think it's fun and novel.) But I do see a lot of problems.

• Answers on SE have no inherent order, and forcing one onto them just feels odd. In the case of a linear chain, "Sort by oldest" happens to work reasonably well, but in a non-linear case like a tree or a DAG, it would be a real challenge to actually grasp the current structure of answers.
• For anything more trivial than "Print Hello World!", it will be seriously frustration if I worked on an answer for half and hour or more and someone else posts half a second before me. (In fact, even if someone posts when I just half-finished my submission it would be a waste of my time.) This is not a problem in non-linear cases.
• People make mistakes. Especially when the tasks get harder, it's very likely that some submission doesn't comply with the rules for one reason or another, and it might take a while for people to notice - in the meantime several new questions might have been posted. Now what should we do with that wrong question? Fix it? That might invalidated subsequent answers, depending on the fix and the type of challenge. Delete it? That makes the chain incomprehensible unless you're a high-rep user. Leave it? That would imply we're cool with rule-breaking answers now, which I hope we aren't.
• We're imposing a lot of restrictions which the SE system itself doesn't (and won't) impose. Examples include "Users may not answer twice within X hours.", "Users may not answer twice in a row.", "Do not edit your submission." and, while this one hasn't been stated explicitly yet, "Do not delete your (valid) submission." is actually equally important to keep the thread consistent.
• [There might be other problems. I'll amend this list, if I can think of any, or people point them out.]

I would really see those problems fixed, but I'm not sure we can fix them all. Here are a few suggestions which might be able to help with individual problems:

• I suggested in chat that someone could write a user script which displays a navigation for the challenge. This could just be a list ordered by time of posting for linear challenges, or it could be an expandable/collapsible tree for even a nicely rendered DAG for non-linear challenges. Of course, this requires people to stick exactly to the desired answer format, so the script can parse the structure, and for the non-linear challenges it would probably require everyone to install that user script. We might be able to put this in a Stack Snippet instead, which could be embedded in the challenge itself. This would go a long way towards to making the challenge more manageable for everyone and to keeping leader boards up-to-date (now that I think about it, this would be a great snippet for most challenges).
• As I said above, non-linear challenges would fix the problem of answering in parallel, but it would make the challenge a lot more confusing. Depending on the challenge, you'd have to check every path down the tree to figure out which path you want to "reply" to. Even if the challenge only depends on the latest answer in the chain (i.e. the leaf nodes of a tree or the sinks of a DAG), you'd still have to find all those answers first. Furthermore, there is still the case where I work on a submission for half an hour, and another answer comes in and I realise that I would rather reply to that one, which requires me to change my answer (and before I'm ready the same might happen again).

• I can think of two ways to deal with broken submissions.

  1. Delete them and deal with holes in the chain. It really depends on the challenge if this is feasible or not.
  2. Answers need to be reviewed before people may reply to them. One possibility would be that you may only reply to another submission if that submission has two comments reading "Confirmed" from users which are neither yourself nor the author of that previous submission. The main issue I see with this is that uncommon languages or ones that are hard to test could get the challenge stuck for a while until two other people can actually review the submission.

But most importantly, I don't think there is any way to enforce the restrictions SE was simply not built for. We're trying here to build a challenge based on the linear or tree-like structure of other forums or mailing lists, but SE simply puts no relation at all between answers. If we can make all the other problems work, I'm thinking we might be able to get away with it anyway (I mean, technically, in a king-of-the-hill a good answer will also depend on all previous answers), but we might need to work a bit harder before we get there.

Edit: I am now trying out several of my suggestions in this challenge. In particular, I am requiring verification of answers before new ones are posted, and I've provided a Stack Snippet to help with the bookkeeping. I'll report back how this goes.

Answer 2

Thanks for posting this, I had been meaning to. Note that I'm trying out another one of these.

A few random notes and comments...

My original notion of these questions

An incremental answer question (or whatever we're calling it) has the following properties:

1. Each answer depends on the last by building upon it or using information in it in some way.
2. Answers increase in difficulty to a point where additional answers are impractical or impossible.

That's about it. Note that text replacement or unique languages or an answer time limit is not part of it. Though those features may be necessary for well bounded challenges.

The tree structure was very much an afterthought, and would be very hard to manage. But it might make some really interesting contests.

Problems

The problems Martin mentioned are definitely present but we've done amazingly well policing the Hello World question. There may be a couple technically invalid answers but that doesn't necessarily detract from the rest, it just means that those two answers are stuck being invalid.

One problem I've had is the scoring. I'm not 100% happy with either of my contests scoring mechanisms, since everything from most answers to byte count to last answer could be considered. The vast quantity of answers makes it hard to keep a more precise score equation updated.

Don't get too hyped

Evolution of "Hello World!" was something new and accessible, which is why I believe it did so well. I've noticed diminishing returns before for repeated challenge types so I'm not certain ASCII's 95 Characters...95 Movie Quotes will do so well. I may have just gotten lucky the first time. Even I am not totally confident that this type of challenge is here to stay.

Tag Possibilities

If the challenge type does take hold it will need a tag.

I suggested [domino-coding] and [incremental-answers].

It looks like we have [answer-dependent] here.

Other things that spring to mind are [linked-list-something] or [induction-something].

Answer 3

I'm not sure how relevant this may be, but there recently appeared a spontaneous acronym acrostic. So there's definitely a desire among the community for something more collaborative. These challenges achieve that, so get some wagons for this band.