5
\$\begingroup\$

Winning criteria determine which submission wins. For example, the following are winning criteria.

The submission with the fastest code on any reasonable desktop machine wins.

Valid criteria determine if a submission is correct. For example, the following are valid criteria.

The submission must run in under 2 minutes on any reasonable desktop machine.

While discussing exactly how subjective we want to allow questions to be, another user and I came up with two possible standards. I added the third for good measure.

  • Winning criteria should be objective. Valid criteria should be given leeway.
    • The example winning criteria is unacceptable because it failed to define the testing environment, but the example valid criteria is acceptable.
  • Winning criteria should be objective. Valid criteria should also be objective.
    • The example winning criteria is unacceptable because it failed to define the testing environment, and the example valid criteria is unacceptable for the same reason.
  • Winning criteria should have leeway. Valid criteria should also have leeway.
    • The example winning criteria is acceptable. The example valid criteria is also acceptable.

Which stance is correct?

\$\endgroup\$
4
  • 2
    \$\begingroup\$ Don't have time to post a full answer, but I'd go with option one. Ideally all specs should be as objective as possible, but I'd allow leeway in validation criteria for practical reasons: Having to test all submissions yourself is impractical, because no one wants to install all those interpreters. You put up with it for KotH and fastest-code, but luckily that's the exception. It would be annoying if all code-golf/code-challenge posts that don't want solutions to be brute force had to test all submissions locally as well. Hence, allow some leeway. \$\endgroup\$
    – Martin Ender
    May 5, 2014 at 17:53
  • \$\begingroup\$ remotely related \$\endgroup\$
    – Martin Ender
    May 5, 2014 at 17:55
  • \$\begingroup\$ Plural: criteria. Singular: criterion. \$\endgroup\$
    – TRiG
    Sep 8, 2014 at 16:18
  • \$\begingroup\$ @TRiG Fixed. Thanks. \$\endgroup\$
    – Rainbolt
    Sep 8, 2014 at 16:49

1 Answer 1

Reset to default
1
\$\begingroup\$

Winning criteria and valid criteria should both be well-defined.

If a challenge contains any kind of performance related criteria, then the conditions for testing should also be well described, regardless of whether the criteria is "winning" or "valid". If the author of the challenge is unable to completely describe the testing environment, then they should take sole responsibility for testing the submissions on their own machine. This should be stated clearly in the challenge.

If testing is being performed automatically, the source code for the test driver should be included. For time related challenges, clock time should be the default unless CPU time (or other) is specified.

\$\endgroup\$
2
  • 5
    \$\begingroup\$ I'm not convinced that listing the specs makes much difference. There are more factors involved in the precise timing of a test than just the processor, memory, and bus. If you want something completely reproducible then the options are either measuring the number of times a certain operation is performed (e.g. the number of array reads) or creating some kind of single-tasking virtual machine which allows you to count clock cycles reproducibly. \$\endgroup\$
    – Peter Taylor
    May 5, 2014 at 19:09
  • \$\begingroup\$ @PeterTaylor You're right - completely describing the testing environment for anything that is being precisely timed is usually not feasible. In those cases, the author should take responsibility for running tests. In fact, I said that in my answer. I edited it to be a little clearer. \$\endgroup\$
    – Rainbolt
    May 5, 2014 at 19:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .