Yes, but they need to be made public after X time units
In my opinion for test-battery challenges this can be a sensible thing to do, however there are two important things that a challenge should do in such a case:
- they need to specify a time span (at least once some people are answering) for when they will choose the winner and by that time make the test cases public
- they need to make sure (eg. by providing a cryptographic hash of the test cases/seed to generate them) that the test cases were not chosen after writing the challenge, this is to make it "impossible" for them to adjust it to an answer they like
Addressing issues mentioned above
1) It makes the winning condition objectivity more probabilistic
If the method of generating hidden test cases is not known, this might be true. However king-of-the-hill challenges are the same. Since the conditions are the same for everyone, this won't cause any advantages/disadvantages for other people. And more importantly just because a challenge fulfils our criterions for being valid, it does not mean that it makes a good challenge:
In my opinion a good test-battery challenge will have at least one example and some public test cases which reflect approximately what's being tested. The scoring could then be a mix between hidden and public tests or just the hidden ones.
2) The score reported by the challenge author cannot be independently verified
Once the test cases are public the score can be verified by everyone and providing a hash will make sure that they are indeed the original tests.
3) The challenge dies when the author stops responding
The first point is very similar to what we do on cops-and-robbers challenges where cops are required to post their solution after some time (usually 7 days), though probably a challenge should give more time than 7 days to solve it.
I doubt that an OP of a challenge will just disappear before they chose a winner, we don't disallow cops-and-robbers challenges because the cop could stop responding and we shouldn't disallow test-battery challenges for that reason. Of course it is a possibility but I don't think that this will create problems.
Note: For this answer I don't think it really matters whether there's a known method of generating the hidden test cases. If anything points (1) and (3) won't cause troubles since people can generate tests themselves to get an approximative score and if an author stops responding the community could come up with new tests (which probably will never happen).