4
\$\begingroup\$

The specific question

I recently posted my test suite as a community wiki answer to a couple of questions that I worked on. Two contributions so far are:

The test suites use perl TAP, which is as close to universal as I know how to get ... easily. I made them community wiki to make it clear they are a common resource and encourage other people to contribute to them (I can dream). I had occasionally posted test suites as part of my answer in the past and decided that releasing them to the wild might be useful. Should I keep posting this specific type of test suite using this specific tool?

The general question

Do you think posting a test suite for questions with specific, non-trivial answers as a community wiki is a good idea? Why or why not? (The kind of open ended question I can only get away with in Meta.) If yes, is there a better tool for this than perl TAP?

Not the question

I am not proposing a requirement that every question include a test suite.

\$\endgroup\$
8
  • \$\begingroup\$ I guess my question would be "Why put it in an answer?" Isn't the test suite effective part of the specification? \$\endgroup\$
    – dmckee --- ex-moderator kitten
    Aug 28, 2014 at 2:15
  • \$\begingroup\$ The test suite is helpful, but it's not exactly an answer. Unfortunately a comment isn't able to contain the same amount of content. It could be an edit to the question, but that makes it look like it's specifically made by the OP, not you. A link to something like Github in the comments might work, accompanied by a small addendum to the question. \$\endgroup\$
    – Doorknob
    Aug 28, 2014 at 2:16
  • \$\begingroup\$ Because testing tools are as varied as the languages and environments used, I wanted to avoid cluttering up the question. Especially with Unix-specific test suites. I have run TAP in Windows, but we had to bring out the big hammers to make it fit. \$\endgroup\$
    – Scott Leadley
    Aug 28, 2014 at 2:22
  • \$\begingroup\$ @dmckee Martin Büttner supplied an excellent test set in Family Tree Solver. I didn't improve on it any (yet), I just made it executable. Your comment is quite relevant to Put a list in order which only had 4 test cases specifying correct input and output for a very complex search space. \$\endgroup\$
    – Scott Leadley
    Aug 28, 2014 at 2:33
  • \$\begingroup\$ I'm not asking why a test suite. For any moderately complicated problem they are very much in order, it's just that I would have expected them to be in the question (or if they got really large to be hosted off-site with the link in the question, sub-optimal though that may be). \$\endgroup\$
    – dmckee --- ex-moderator kitten
    Aug 28, 2014 at 2:36
  • \$\begingroup\$ I know that making some "answers" community wiki is pushing the boundaries of the Stack Exchange paradigm. It felt right, though, as a way to encourage a collaborative ancillary to the answers. \$\endgroup\$
    – Scott Leadley
    Aug 28, 2014 at 2:41
  • \$\begingroup\$ Oops, this means I shouldn't have done this, this, and this \$\endgroup\$
    – justhalf
    Sep 5, 2014 at 5:20
  • 1
    \$\begingroup\$ @justhalf What I took away from this was that improving the question was more important than an executable test suite. (Views were <100, votes ~10%.) Your example answers seem semi-off-topic, but answers to koth and popularity-contest questions seem to get a lot of leeway. And you had fun and nobody pitched a fit. \$\endgroup\$
    – Scott Leadley
    Sep 5, 2014 at 5:48

2 Answers 2

Reset to default
8
\$\begingroup\$

Do not post test suites as answers

Meaning: do not post test-suites as answers for themselves without any real answer to the question. If it is an add-on to your answer you may well post it there - or consider adding it to the question.

Basically because it not is an answer but part of the specification. While it adds nothing to the answers (just clutters the list of answers - many will not even notice that there is a test suite among all the postings) it adds a lot to the question itself and should therefore become part of the question.

You may add a sub header (Test suite (provided by USER)) in order to indicate that this part is not from the original question. If OP dislikes the test suite or finds it invalid she can always re-edit the question.

In order to be suitable for many users try not to use a specific testing framework but rather give a list of input/expected output pairs - any puzzler should be able to transform this format into his favourite testing framework/language.

\$\endgroup\$
1
  • 2
    \$\begingroup\$ In other words (mine, not yours), the Stack Exchange way is to improve the question for posterity. If additional test cases significantly improve the question, add them to the question. (Significantly means avoid jostling the OP with trivial additions & changes.) \$\endgroup\$
    – Scott Leadley
    Aug 31, 2014 at 17:03
-1
\$\begingroup\$

Yes, do post test suites as wiki'fied answers for some questions

Executable test suites are an important part of programming practice. For problems with large and/or complex output spaces, they are necessary in order to quickly verify correct program behavior and track down incorrect behavior. Adding a wiki'fied "answer" containing an executable test suite is better than cluttering up the question with testing scripts that may not run in all environments and may not be usable with all languages (JavaScript being a noteworth example).

\$\endgroup\$
1
  • \$\begingroup\$ I'm not sure even I agree with this solution (wiki'fied answers containing test suites) to what I see as a problem (where the output space is complex, a significant number of "answers" don't really solve the problem). But I think the question deserves some debate. \$\endgroup\$
    – Scott Leadley
    Aug 31, 2014 at 23:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .