5
\$\begingroup\$

I am working on a code challenge. The scoring will be based on a set of benchmark inputs (i.e. it will be the sum of the individual scores).

Now I'm wondering if it's alright to add additional test cases once the challenge has started (in which case I'd redetermine all submissions' scores, obviously).

Why I might want to do this:

  • Maybe I just think of an additional interesting/pathological test case that I forgot initially.
  • Someone posts a solution which is tailored specifically to the existing test cases (hardcoded or similar), which I'd like to discourage.
  • Someone posts a solution which solves the problem optimally and runs in reasonable time on the existing test cases, that sort of breaks the challenge. It's likely that others will manage that, too, resulting in a massive tie of optimal submissions. In such a case, I'd like to add additional larger tests where the optimal algorithms become infeasible.

Why it might be a problem:

  • You can't figure out your final score at the time of posting, because you can't know which test cases might be added later on.

Why I think it might be alright:

  • There are test cases already, and if submissions are just general algorithms that can deal with any input (like they're supposed to), their scores shouldn't be massively affected by adding new test cases.
  • Participants are always free to tweak their submissions if they realise they missed some important optimisations after I add more test cases.

I realise this question is similar to this one, but I think my case is slightly different, because there the benchmark test cases weren't disclosed at all.

What's the community's opinion on this?

\$\endgroup\$
4
\$\begingroup\$

The title of this question is slightly misleading, so as an minor point: it's always good to add test cases to questions which don't have them, or to add ones which cover corner cases which aren't tested by the existing ones.

I think the word benchmark in your question is the real heart of it. One approach which I believe was suggested in the early days of is that everyone who goes to the effort of writing an answer thereby gains to right to add one set of input to the benchmark. That provides an easy way to deal with the first two specific issues you raise.

In terms of the third issue, tie-breaking, it wouldn't be very reasonable to object to adding new cases there. If you're really worried that people might accuse you of bias, you could write a deterministic track generator which takes a size parameter.

\$\endgroup\$
  • \$\begingroup\$ Fixed the question title. I really like the idea of participants submitting additional test cases! And for some reason I didn't think of the third case as simply adding a tie-breaker. I'll have a look if I can write a track generator (although I think it's like that it will be qualitatively identical to existing tracks). \$\endgroup\$ – Martin Ender Jun 8 '14 at 9:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .