8
\$\begingroup\$

Somewhat related

In my Bernoulli Numbers challenge, I've been seeing a lot of invalid answers that fail to compute certain values to the required accuracy/precision, or that take substantially longer to run than the amount of time between the posting of the challenge and the answer (such as this one - it's been churning for nearly 40 hours at just computing 30-36, and the answer was posted less than 24 hours after the question). Now, whether said requirements were good requirements is another discussion altogether - I'm mainly concerned with the fact that many answers have been posted that have obviously not been tested by the posters.

The entire point of providing test cases is to allow users to test their prospective answers and verify that they produce the expected output before posting them, rather than having the OP1 test them and inform the poster of their answer's invalidity. Needless to say, it is frustrating to go through the effort of compiling test cases and see them not being utilized.

I would like to see a policy where users are required to test their code before submitting it as an answer, to reduce the effort required of the OP in testing all of the answers for validity. Is this a reasonable request? If so, how might we go about requiring proof of testing/validity?

1I use "OP" to mean the user who posted the question/challenge

\$\endgroup\$
13
  • 8
    \$\begingroup\$ 1) I thought this was already the policy, and 2) it's worth noting that you only had the test cases 0-10 initially. I generally only test my code with test cases in the question, and on top of that, I have a bad habit of testing only a few values. I doubt I'm the only user with these tendencies. \$\endgroup\$
    – El'endia Starman
    Commented Dec 3, 2015 at 7:31
  • \$\begingroup\$ @El'endiaStarman 1) As far as I've seen, the policy is to get on to the users who post incorrect, untested code. We don't enforce that users test their code before posting, just nag the ones who don't and are caught with invalid answers. 2) Prior to adding 0-60, I had stated that 0-10 were just for reference. Regardless, that does not exempt users from testing their code for all required inputs. \$\endgroup\$
    – user45941
    Commented Dec 3, 2015 at 7:35
  • \$\begingroup\$ I tested my code with 60 too and it looked fine. 0-10 and 60. There were problematic ones in the 35+ range...but you didn't give us a reference for that. \$\endgroup\$
    – El'endia Starman
    Commented Dec 3, 2015 at 7:45
  • \$\begingroup\$ @El'endiaStarman Reading the linked Wikipedia page would have informed you that, for odd n starting with 3, B_n is always 0. That's simple enough to check for, while I was working on compiling 0-60 for the full test suite. \$\endgroup\$
    – user45941
    Commented Dec 3, 2015 at 7:49
  • \$\begingroup\$ Well, but you also specified six decimal digits of accuracy, which for most results for odd numbers was 0.000000.... I did have an iteration of my code where all of the odd numbers were just set to 0, and there were still inaccuracies for the higher even numbers \$\endgroup\$
    – El'endia Starman
    Commented Dec 3, 2015 at 7:51
  • \$\begingroup\$ @El'endiaStarman I agree that not providing the full test suite immediately was a mistake on my part. However, that is tangent to the main issue I am bringing up here: users are not testing their code before submitting \$\endgroup\$
    – user45941
    Commented Dec 3, 2015 at 7:57
  • \$\begingroup\$ Granted, that is indeed the main point. \$\endgroup\$
    – El'endia Starman
    Commented Dec 3, 2015 at 7:59
  • \$\begingroup\$ How would you verify that a submitter actually tested it? \$\endgroup\$
    – mroman
    Commented Dec 3, 2015 at 13:00
  • 1
    \$\begingroup\$ @Mego: There's a difference between "users are not testing" and "users are not testing enough". I've tested my Haskell solution for small n and for n=60 (the test cases available when I wrote my answer) and it works fine. However, it utterly fails for big odd n, because rounding errors sum up. So, yes, I've tested not enough and my answer is wrong. I've deleted it. PS: I've written a memoizing wrapper so b 60 runs in less than a second, but removed it again for golfing reasons. \$\endgroup\$
    – nimi
    Commented Dec 3, 2015 at 15:37
  • \$\begingroup\$ "I would like to see a policy..." If users do not test their code (or show that it has been tested) anyway, what do you propose doing to the answer? Deleting it outright (or just downvoting it), even if it works? \$\endgroup\$
    – Doorknob Mod
    Commented Dec 3, 2015 at 17:04
  • \$\begingroup\$ @mroman Exactly the point of this post - I haven't come up with a good way yet, so I am reaching out to the community for ideas. \$\endgroup\$
    – user45941
    Commented Dec 3, 2015 at 17:06
  • \$\begingroup\$ @Doorknob I don't have a good answer myself, hence why I am opening a discussion on it. \$\endgroup\$
    – user45941
    Commented Dec 3, 2015 at 17:07
  • \$\begingroup\$ @Mego if you want tested/checked answer I suggest you look at golf.shinh.org which supports about 100 languages and has automatic verification through test-cases. \$\endgroup\$
    – mroman
    Commented Dec 4, 2015 at 20:48

2 Answers 2

Reset to default
13
\$\begingroup\$

Entirely untested submissions should not be posted.

Even if you're sure the code would work if it were tested, it should not be posted unless it has been tested. If it's obvious that an answer has not been tested, be it due to obvious syntax errors or other blaring inaccuracies, it should be downvoted.

That said, there's no way to reliably ensure that a user has sufficiently tested their submission prior to posting. I believe it should generally be taken on good faith that an answer is correct to the best of the user's knowledge, if they have provided at least some documentation.

What constitutes "documentation" of an answer?

In my opinion, it's one or more of the following:

  1. An ungolfed, commented, easier to follow but functionally equivalent version of the scored code
  2. An explanation of the code as written and the algorithm it implements
  3. A link to an online interpreter, preferrably with a test suite

If an answer has none of those, I don't upvote.

Not all languages have free online interpreters. That's fine as long as a user can explain what the code does and it's obvious how you can test it yourself.

In my opinion, pasting example runs of the code in an answer with its output is helpful but isn't sufficient to verify correctness.

When in doubt, test it yourself.

\$\endgroup\$
12
\$\begingroup\$

I am strongly against making a general policy that all answers should be run to completion before they can be considered tested and permitted to be posted. I can think of a few answers on this site which won't run to completion before the Earth is swallowed up by the Sun, and I don't think that's a bad thing. E.g. some of the answers to Shamir's secret sharing were only tested with smaller primes because they use brute force to find modular inverses; Shortest terminating program whose output size exceeds Graham's number obviously doesn't have any fully testable answers.

On the other hand, there's been a trend over the past two or so years* for questions to specify that answers must complete for inputs up to a certain size within a time limit which permits testing. That's not a bad thing either: golfing without any constraints other than correctness and golfing with performance constraints as well are similar but different and can both be fun.

Some questions include stack snippets which verify answers. Perhaps we should try to encourage those for questions like the Bernoulli numbers one where checking that the output is within tolerance is best done by computer rather than by eye.

* Personal impression - I haven't calculated any statistics to support this figure.

\$\endgroup\$
6
  • 2
    \$\begingroup\$ Particularly since this seems to be the previously established policy, I don't disagree at all. But it does result in situations that I think are hard to resolve. How do you know that an answer produces correct results if it doesn't terminate before the sun turns into a red dwarf? Like in this specific example, how do you know that a solution will meet the specified precision requirements if you can't run it to completion? Put to the extreme, it could be beneficial to post very inefficient answers, because nobody could demonstrate that they fail some specified requirements. \$\endgroup\$
    – Reto Koradi
    Commented Dec 4, 2015 at 5:58
  • 1
    \$\begingroup\$ @RetoKoradi I agree with the issue you brought up - posting an answer that would take years (or more!) to run and verify for all inputs would be an issue, especially if it's (eventually) shown to be incorrect/invalid. Though I like the idea of restricting running time to make testing more reasonable, it seems to me to go against the spirit of code golf to add the restriction only for the sake of testing. I'm very undecided on this topic, as you can see :P \$\endgroup\$
    – user45941
    Commented Dec 4, 2015 at 7:17
  • 5
    \$\begingroup\$ @RetoKoradi, the only reason that the precision requirements of this particular question are testable at all is that the input is a single integer and the requirements only apply to 60 cases. There are a lot of questions which deal with floating point where there are on the order of 2^50 cases and it's necessary to do formal error analysis. Floating point and numerical analysis always open a can of worms. \$\endgroup\$
    – Peter Taylor
    Commented Dec 4, 2015 at 7:25
  • \$\begingroup\$ @PeterTaylor So what about those cases where it is possible/necessary to do formal error analysis? Should the impetus be on the answerer to prove that their solution is valid? Or should that responsibility reside with the OP? \$\endgroup\$
    – user45941
    Commented Dec 4, 2015 at 8:38
  • 1
    \$\begingroup\$ @Mego, I don't think that's black and white. We don't want to completely push away people who haven't studied numerical analysis, so there has to be some freedom to push answers in good faith while recognising that everyone makes mistakes. \$\endgroup\$
    – Peter Taylor
    Commented Dec 4, 2015 at 18:21
  • \$\begingroup\$ @PeterTaylor I agree. It's a very tricky problem to navigate. \$\endgroup\$
    – user45941
    Commented Dec 4, 2015 at 21:36

You must log in to answer this question.