Definitive policy about answers not meeting the challenge specification

Question

This question has been asked so often, I have no idea what our actual policy is:

• Almost-correct answers
• Should answers that break the rules be deleted?
• Should we flag and delete Not an Answer posts?
• Can we stop flagging poor answers as "not an answer"?
• What should count as "not an answer" here?
• What should the policy be on partial answers?

None of them are closed as duplicates. Most of them overlap but not all. Also, all of those opinions are from 2014 (the one answer from 2015 is essentially a repost of one of the others). We still regularly get Not-An-Answer flags on answers that don't meet the spec, so it would be great if there was a single, up-to-date policy to refer to.

To summarise what results those posts yielded:

• Consensus seems to be mostly that answers which don't meet the spec are "just" factually wrong answers, and Meta.SE says about those:

  Answers that are answers but are factually wrong [...] should get downvoted, but not deleted.

  Or just straight from one of the flag decline reasons we have as mods:

  flags should not be used to indicate technical inaccuracies, or an altogether wrong answer

  Geobits made a good argument in favour of this view that downvoted bad answers act as signposts for future visitors. However, the one problem I have with this policy is that its main justification is always that "that's the standard SE policy". That policy was created for Q&A sites. PPCG is not a Q&A site, so if we adopt standard policy like that, it would be great if there was some additional support for why this even makes sense here.

• trichoplax made an argument in favour of deleting such answers. And I think it's a fairly good one: downvoted wrong answers acting as signposts only works as long as they are actually downvoted. Too often, we have answers that a) accumulate tons of upvotes before anyone notices they are wrong (and then are left in that state by the author and the voters) or b) are deliberately wrong but "funny" or contain nice pictures or whatever and are upvoted for those regardless of the actual code. (Example for users who can see deleted posts.) Those are especially problematic with HNQ challenges, which get lots of votes from users who aren't active here ... and those are exactly the challenges where having downvoted answers as signposts would be useful. This is why I'm not convinced that the standard SE policy is actually the best course of action for PPCG.

• Another thing I found in a comment by PhiNotPi is that maybe we should distinguish between answers where the author knows they are wrong and those where he doesn't. I haven't seen this point addressed in any answers at all yet.

• To top it all off the only recent related discussion has a score of 28 in favour of not posting almost-working solutions at all in the first place.

This is why I'd like to have one more, but comprehensive discussion on this matter to see what the community actually thinks these days, and how you expect us as moderators to deal with these flags if they are raised.

Answer 1

Scope

As I see it, there are five types of invalid answers:

1. Answers that produce incorrect results.

   This is the most common type, and usually an accident.

2. Answers that produce correct results, but break a rule of the challenge, ignore parts of the spec or violate a loophole.

   For example, answers that make use of banned built-ins fall into this category.

3. Answers that produce correct results, comply with the rules of the challenge, but aren't a serious contender for the winning criteria.

   For code golf, e.g., this is limited to answers that do not even attempt to golf the code. Answers that are simply poorly golfed are not invalid.

4. Answers that have one of the following, fatal flaws:

   • They exist only to promote a product or service and do not disclose the author's affiliation.
   • A reasonable person would find their content inappropriate for respectful discourse.

5. Answers that aren't actually answers.

   This includes anything but code that is intended to solve the challenge.

Marking an answer as non-competing does not exempt it from being invalid.

Deletion

Answers of type 4 and 5 aren't actually answers. The policy across the entire network is that they should be deleted on sight, and there's no reason to make an exception here.

I think all other types should be removed as well, with varying levels of urgency. Even without an official policy, this is commonplace across the entire network. While these answers should not be flagged on other SE sites, high rep users delete them all the time.

If the answer is invalid without the poster's knowledge, it shouldn't be deleted without notifying the poster first and giving him some time to fix their mistake.

If the answer is deliberately invalid (most common for type 2), there really is no reason to wait or tell the poster something he already knows.

Self-deletion

If you figured out or have been told that your answer is invalid, I think self-deletion is the best option.

• Deleting the answer quickly will prevent it from accumulating downvotes, which may or may not get removed after the answer has been fixed.

• Deleting the answer yourself means that you can undelete it just as easily.

  In contrast, undeletion will require moderator intervention if the answer has been deleted by others.

• While deleting your answer will invalidate any rep your may have earned from it, you'll get it back if you fix the answer.

Flags

Not an answer flags were creates specifically for type 5 answers, and should be cast when they are encountered.

Likewise, spam and rude or abusive flags were created for type 4 answers, and should be used instead of not an answer flags.

Very low quality flags pretty much apply to all other types, but can only be cast on answers with a non-positive score. They are also cleared automatically when the post is edited, so they may be ineffective if the intention is to get a post removed.

This leaves flagging as in need of moderator intervention.

The policy across the entire network is that flags should not be used to indicate [...] an altogether wrong answer, but I think our site should be (once again) an exception.

First of all, we have a huge moderator-to-activity ratio. Stack Overflow moderators cannot be expected to complete the time-consuming task of deciding whether an answer is valid or not, but our flag volume is rather low and checking PPCG answers is usually easier. A flag that clearly explains why the answer is invalid (i.e, which rule it violates or which test case it fails) does not require an unreasonable amount of moderator time.

Also, unlike other sites, I don't think we need examples of what not to do. There is absolutely no merit in keeping an answer that violates a rule of the challenge; they don't teach anything. If an answer passes all test cases but still produces incorrect results, there is no benefit in keeping the answer after a new test case has been added to the question.

But it is important to wait until the moderator can actually take action. I think deleting an accidentally invalid answer on sight is an unnecessarily unpleasant experience for the poster, and undeleting it requires yet another moderator intervention.

Voting to delete

Trusted users (currently 20k+ rep) can cast deletion votes on answers. In all of the above situations that suggest flagging, trusted users can also vote to delete an answer.

With three votes from trusted users, involving a moderator becomes unnecessary. However, trusted users should flag instead or on top of their delete votes unless all of the following conditions are met:

• The answers has a negative score.

  This is a hard requirement, since even trusted users cannot vote to delete answers with non-negative scores.

• The challenge is still active.

  Trying to gather three delete votes for an answer on a question that barely gets views anymore will take a long time. The moderation tools (10k+ rep) could help with this, but barely anybody seems to use them.

• The answer should be removed entirely, not converted into a comment.

  Only moderators can convert answers into comments.

• The answer is neither spam nor rude or abusive.

  Six "red" flags will not only delete the post, but hide it in the revision history and apply a 100 rep penalty to the poster. Deleting the post via deletion votes would prevent this.

TL;DR

• Immediately flag invalid answers of type 4 as spam or rude or abusive.

• If your own answer turns out to be invalid, delete it yourself.

• Immediately leave a comment on all invalid answers that are not your own.

• Immediately flag invalid answers of type 5 as not an answer.

  If you are a trusted user, the challenge is fairly active and the answer should not get converted into a comment, consider voting to delete instead.

• Give the poster time to fix or self-delete their invalid answer of type 1 to 3.

• If a sufficient amount of time has passed (say 48 hours) or the answer was deliberately invalid, flag the answer as in need of moderator intervention, providing an explanation of why it is invalid.

  If you are a trusted user and the challenge is fairly active, consider voting to delete instead.

• If you fix your self-deleted answer, undelete it.

  If you fix your answer and cannot undelete it yourself, flag it as in need of moderator intervention, requesting its undeletion.

Answer 2

Marking an answer Community Wiki does not excuse it from this policy

Introducing an answer with "I know this doesn't follow the rules but I worked hard on it so I want to post it here" is not an acceptable excuse.

Saying "I've marked it Community Wiki so I won't get any rep from it" does not change the fact that it is taking up space meant for valid answers from people who not only put the hard work in but also followed the rules.

Answer 3

The new policy should apply to answers that predate it

If the community decides to remove invalid answers (and it certainly looks like it right now), that policy should apply to all answers. Reasons for this include:

• We are not changing the rules.

  For a policy that invalidates previously valid answers, retroactive application wouldn't be entirely fair, but that's not the case here. The answers were always invalid. The only thing that is debated now is how to deal with them.

• Old or new, they are low quality.

  While there was no clear consensus that invalid answers should be removed, there always was consensus that they were low quality. Low quality contributions are and always have been subject to removal. In fact,

  The answer is extremely low quality: There is little to no scope for improvement

  is one of the suggested reasons for deletion for trusted users.

• There is little to no rep loss.

  As explained in Reputation and Historical Archives, any answer with a score of +3 or higher that has been visible for 60 days or more will not cause a reputation loss if it is deleted.

Answer 4

Delete answers that don't follow the spec, downvote low quality but otherwise valid answers

We need to make a distinction between unpopular answers and invalid answers. A valid but poorly-golfed (or otherwise low quality, as determined by the winning criteria) answer deserves to stay, but should be downvoted as a signal to the author that they could do better. Invalid answers should be deleted, so that new users don't get the impression that they are valid but unpopular. Since deletion can always be reversed, it gives the author the opportunity to fix the answer and undelete it or flag it for undeletion. When an answer is deleted because it is invalid, a comment should always be left explaining why it was deleted, and possibly what could be done to make the answer valid.

Ideally, a challenge's answers should all be valid (except for any deleted answers), and the votes should reflect ingenuity and impressiveness (or the lack thereof for downvotes).

This leaves open which flag should be used for invalid answers; Not An Answer and Low Quality would both suffice. I'd prefer the latter, since the former is intended for answers that don't make an attempt at solving the challenge (e.g. answers that should be comments).

Answer 5

Keep the answer but mark it incorrect in header

Let the answerer keep the answer as long as they label it incorrect in the header (and so noncompetitive) and explain how it fails. Any downvotes might they take from keeping it are at their own risk.

My reasoning is that invalid answers range from blatantly wrong untested answers to honest mistakes where the poster interpreted an ambiguous spec not as the asker intended. We should let votes decide between the cases. The blatantly wrong answers will get downvoted, pressuring the poster to delete them.

I want to avoid the frustrating situation where someone works hard to golf a problem, posts it, gets upvotes, and then it told they have to delete it for a reason that wasn't really their fault, such as:

• The asker changes the question
• The asker makes an unexpected ruling that invalidates the answer
• The answers passes all test cases, but fails an obscure corner case that one might not expect to be valid
• The spec is ambiguous, and the answered interpreted it a reasonable way without seeing the potential ambiguity
• The answer breaks a condition that is in the challenge, but is so unexpected, buried in the spec or comments, and/or contrary to the rest of the challenge that a typical reader is likely to miss it. (I expect this case to be the most contentious, so I could remove it if people object to it specifically.)