Winning conditions not measurable until question author decides to reveal test data set

Question

This question has an interesting property, in that judging a winner is dependent on unknown data:

Recognize handwritten digits

This is actually not an uncommon way of things working in puzzle-space. In programming competitions, you often are given simple test data and then you submit your program to a server where it is run against some kind of pathological data which you are not given in advance.

But if a winner is to be declared, there is a question of when and how that data will be disclosed. It introduces a factor of timing into it such that no entry can be accepted after the revelation point.

Does such a question with an undisclosed winning condition deserve its own tag? Should there be a rule that once a winner is declared, the test data be published, and no answers accepted after that point?

Answer 1

I am also currently planning a challenge that would be scored by me against an undisclosed set of test data. I was thinking about handling it as follows:

• There will be an openly available set of example data, so that the contestants can estimate their score on their own. Also, the means to easily create new test sets will be made available.

• I will regularly calculate and update the official score with my secret test set. I might decide to only do so after major updates of a solution, so as to discourage people from trying to randomly optimize their solution for the specific test set.

• After some fixed time (say three weeks) without a change in scores (no new answer and no updates to existing solutions) I will announce a winner and accept their answer. Selecting a winner at some point, even while better answers might still be coming, should not be more of a problem than with other challenges.

• The challenge will remain open and answers are still welcome. New clever answers will still gain upvotes.

• After I have announced a winner I will make the test set available. This is to prove I did not cheat and it will allow users to grade answers even in the future when I might no longer follow the challenge too closely.

• Of course, with the test data available, people may exploit that knowledge to gain better scores. However, I would trust on the community to still follow the spirit of the challenge and not give undue praise to uncreative exploits.

Answer 2

I'm the one who posted that question, and I just noticed this meta question now. I'll try to address some of the issues discussed here:

• judging a winner is dependent on unknown data - kind of, but not really - the (60 times) larger data set is fully disclosed, and the test data is randomly selected, so the intention is clearly to solve any case from the (known) large set
• when and how that data will be disclosed - I don't plan to disclose the test data (that is, the particular chosen subset of the fully disclosed data set)
• If shortness of code is not the condition then it isn't [code-golf]. What's wrong with [code-challenge]? – I tagged it as both code-golf and code-challenge, as shortness of code is part of the condition. Peter Taylor then removed code-golf and I'm ok with that.
• the OCR test in the question would be solvable by a program just printing the string that you read - no, it would not. Even if the data set is known, it has 1000 cases and your program has to solve one at a time, independently of each other. Also, if I interpret what you wrote literally, printing out the input is guaranteed to fail since it's not an ASCII digit.