update: This is resolved now - I did update the challenge, and I'm glad I did so.
I recently posted the challenge Write Moby Dick, approximately, which has proven popular and the answers are flowing in. However, it has one unforeseen issue that I'm tempted to try and fix.
The current best-scoring answers solve the problem in two steps: first strip out the newlines, then run some kind of text prediction algorithm, then use a second, separate algorithm to work out whether to replace a space with a newline or not. This is because the
whale.txt file is wrapped to a line width of 73 characters, which makes the task of predicting newlines quite different from predicting the other characters.
I suspect that most if not all competitive answers will have to solve these two problems separately. This is inelegant, and it creates a barrier to entry for new approaches, since someone might have a great idea for a new text prediction approach, but it's unlikely to be competitive unless they also implement a separate linefeed prediction algorithm.
Because of this, I'm tempted to create a new file,
whale2.txt, that's processed to remove this line wrapping. (It would still contain newlines, but only at the end of paragraphs.) I would provide both this file and the original
whale.txt, and give people the choice of which to use. This way existing answers won't be invalidated, but new answers will not have to face the linefeed issue.
My question is just whether updating the challenge in this way is the right thing to do, or if it's better to just leave the question as it is. I think fixing the issue would improve the challenge, but I want to make sure this kind of change is acceptable in this community. I'd appreciate any comments, opinions, links to previous meta discussions, etc.
Update: I've created the whale2.txt file, and I guess I'll go ahead with this in 5 hours or so unless someone tells me it's frowned upon.