Winning conditions not measurable until question author decides to reveal test data set

This question has an interesting property, in that judging a winner is dependent on unknown data:

Recognize handwritten digits

This is actually not an uncommon way of things working in puzzle-space. In programming competitions, you often are given simple test data and then you submit your program to a server where it is run against some kind of pathological data which you are not given in advance.

But if a winner is to be declared, there is a question of when and how that data will be disclosed. It introduces a factor of timing into it such that no entry can be accepted after the revelation point.

Does such a question with an undisclosed winning condition deserve its own tag? Should there be a rule that once a winner is declared, the test data be published, and no answers accepted after that point?

• If shortness of code is not the condition then it isn't [code-golf]. What's wrong with [code-challenge]? – dmckee --- ex-moderator kitten May 19 '14 at 6:54
• @dmckee Nothing wrong with code-challenge, but that question threw on code-golf. My suggestion was that if the test data is not given up front, and performance against undisclosed test data is your winning criterion, you don't use the code-golf tag (which that question does, ATM) Code golfing is a very particular type of puzzle, and I don't think it makes sense to do it against a problem with incomplete constraints. – HostileFork says dont trust SE May 19 '14 at 6:59
• @dmckee Notice also that, after the winner is disclosed and the test data set revealed, the challenge disappears... for instance, the OCR test in the question would be solvable by a program just printing the string that you read. This is why I wonder if a new tag is needed, to say "after the reveal of the data set, and a winner declared, you can't 'win' after that" – HostileFork says dont trust SE May 19 '14 at 7:16
• @m.buettner The more general question is about a puzzle's dependency on unknown data. You then have the question of when and how that data will be disclosed, and it introduces a factor of timing into it such that no entry can be accepted after the revelation point. I'm not sure what to tag that but time-sensitive is the sort of thing I'm thinking. – HostileFork says dont trust SE May 19 '14 at 11:42
• @m.beuttner Okay, well, then you agree. :-) – HostileFork says dont trust SE May 19 '14 at 12:21
• @Dr.Rebmu haha, yes, that's what I was trying to tell you ^^ – Martin Ender May 19 '14 at 12:42

I am also currently planning a challenge that would be scored by me against an undisclosed set of test data. I was thinking about handling it as follows:

• There will be an openly available set of example data, so that the contestants can estimate their score on their own. Also, the means to easily create new test sets will be made available.

• I will regularly calculate and update the official score with my secret test set. I might decide to only do so after major updates of a solution, so as to discourage people from trying to randomly optimize their solution for the specific test set.

• After some fixed time (say three weeks) without a change in scores (no new answer and no updates to existing solutions) I will announce a winner and accept their answer. Selecting a winner at some point, even while better answers might still be coming, should not be more of a problem than with other challenges.

• The challenge will remain open and answers are still welcome. New clever answers will still gain upvotes.

• After I have announced a winner I will make the test set available. This is to prove I did not cheat and it will allow users to grade answers even in the future when I might no longer follow the challenge too closely.

• Of course, with the test data available, people may exploit that knowledge to gain better scores. However, I would trust on the community to still follow the spirit of the challenge and not give undue praise to uncreative exploits.

I'm the one who posted that question, and I just noticed this meta question now. I'll try to address some of the issues discussed here:

• judging a winner is dependent on unknown data - kind of, but not really - the (60 times) larger data set is fully disclosed, and the test data is randomly selected, so the intention is clearly to solve any case from the (known) large set
• when and how that data will be disclosed - I don't plan to disclose the test data (that is, the particular chosen subset of the fully disclosed data set)
• If shortness of code is not the condition then it isn't [code-golf]. What's wrong with [code-challenge]? – I tagged it as both code-golf and code-challenge, as shortness of code is part of the condition. Peter Taylor then removed code-golf and I'm ok with that.
• the OCR test in the question would be solvable by a program just printing the string that you read - no, it would not. Even if the data set is known, it has 1000 cases and your program has to solve one at a time, independently of each other. Also, if I interpret what you wrote literally, printing out the input is guaranteed to fail since it's not an ASCII digit.
• I think that what he means by the last point is that if you publish your test cases then someone can submit a program which is just a kolmogorov-complexity submission for the correct output string. Easily defeated by reordering the input. Writing a real solution against the 1000 test cases rather than the 60000 candidate test cases might give a slight edge. But I think the genuine issue here is that if you stop participating in the site, new submissions can no longer be scored. – Peter Taylor May 29 '14 at 9:08
• @PeterTaylor The output is one digit, not a string of 1000 digits, so I don't see how that would work anyway. It could only work if the program somehow saved state between subsequent runs, and the cases were always tested in the same order from the first to the last one. "Writing a real solution against the 1000 test cases" might give a slight edge indeed, but a really tiny one. And yes you are right that scoring currently needs to be done by me, but so far they are really close to the estimated scores. Would you suggest changing to test all 60000 cases? – aditsu quit because SE is EVIL May 29 '14 at 10:57
• I think that on balance, testing all 60000 cases would be an improvement to the question. – Peter Taylor May 29 '14 at 11:46
• The downside being that it takes 60 times longer to run. I'll think about it. – aditsu quit because SE is EVIL May 29 '14 at 13:09