I am working on a code challenge. The scoring will be based on a set of benchmark inputs (i.e. it will be the sum of the individual scores).
Now I'm wondering if it's alright to add additional test cases once the challenge has started (in which case I'd redetermine all submissions' scores, obviously).
Why I might want to do this:
- Maybe I just think of an additional interesting/pathological test case that I forgot initially.
- Someone posts a solution which is tailored specifically to the existing test cases (hardcoded or similar), which I'd like to discourage.
- Someone posts a solution which solves the problem optimally and runs in reasonable time on the existing test cases, that sort of breaks the challenge. It's likely that others will manage that, too, resulting in a massive tie of optimal submissions. In such a case, I'd like to add additional larger tests where the optimal algorithms become infeasible.
Why it might be a problem:
- You can't figure out your final score at the time of posting, because you can't know which test cases might be added later on.
Why I think it might be alright:
- There are test cases already, and if submissions are just general algorithms that can deal with any input (like they're supposed to), their scores shouldn't be massively affected by adding new test cases.
- Participants are always free to tweak their submissions if they realise they missed some important optimisations after I add more test cases.
I realise this question is similar to this one, but I think my case is slightly different, because there the benchmark test cases weren't disclosed at all.
What's the community's opinion on this?