33
\$\begingroup\$

This question has been asked so often, I have no idea what our actual policy is:

None of them are closed as duplicates. Most of them overlap but not all. Also, all of those opinions are from 2014 (the one answer from 2015 is essentially a repost of one of the others). We still regularly get Not-An-Answer flags on answers that don't meet the spec, so it would be great if there was a single, up-to-date policy to refer to.

To summarise what results those posts yielded:

  • Consensus seems to be mostly that answers which don't meet the spec are "just" factually wrong answers, and Meta.SE says about those:

    Answers that are answers but are factually wrong [...] should get downvoted, but not deleted.

    Or just straight from one of the flag decline reasons we have as mods:

    flags should not be used to indicate technical inaccuracies, or an altogether wrong answer

    Geobits made a good argument in favour of this view that downvoted bad answers act as signposts for future visitors. However, the one problem I have with this policy is that its main justification is always that "that's the standard SE policy". That policy was created for Q&A sites. PPCG is not a Q&A site, so if we adopt standard policy like that, it would be great if there was some additional support for why this even makes sense here.

  • trichoplax made an argument in favour of deleting such answers. And I think it's a fairly good one: downvoted wrong answers acting as signposts only works as long as they are actually downvoted. Too often, we have answers that a) accumulate tons of upvotes before anyone notices they are wrong (and then are left in that state by the author and the voters) or b) are deliberately wrong but "funny" or contain nice pictures or whatever and are upvoted for those regardless of the actual code. (Example for users who can see deleted posts.) Those are especially problematic with HNQ challenges, which get lots of votes from users who aren't active here ... and those are exactly the challenges where having downvoted answers as signposts would be useful. This is why I'm not convinced that the standard SE policy is actually the best course of action for PPCG.

  • Another thing I found in a comment by PhiNotPi is that maybe we should distinguish between answers where the author knows they are wrong and those where he doesn't. I haven't seen this point addressed in any answers at all yet.

  • To top it all off the only recent related discussion has a score of 28 in favour of not posting almost-working solutions at all in the first place.

This is why I'd like to have one more, but comprehensive discussion on this matter to see what the community actually thinks these days, and how you expect us as moderators to deal with these flags if they are raised.

\$\endgroup\$
  • 8
    \$\begingroup\$ About point 2: there are answer that are blatantly worng and still get many upvotes. We should have an effective way to manage them \$\endgroup\$ – edc65 Jan 7 '16 at 16:32
41
\$\begingroup\$

Scope

As I see it, there are five types of invalid answers:

  1. Answers that produce incorrect results.

    This is the most common type, and usually an accident.

  2. Answers that produce correct results, but break a rule of the challenge, ignore parts of the spec or violate a loophole.

    For example, answers that make use of banned built-ins fall into this category.

  3. Answers that produce correct results, comply with the rules of the challenge, but aren't a serious contender for the winning criteria.

    For code golf, e.g., this is limited to answers that do not even attempt to golf the code. Answers that are simply poorly golfed are not invalid.

  4. Answers that have one of the following, fatal flaws:

    • They exist only to promote a product or service and do not disclose the author's affiliation.
    • A reasonable person would find their content inappropriate for respectful discourse.
  5. Answers that aren't actually answers.

    This includes anything but code that is intended to solve the challenge.

Marking an answer as non-competing does not exempt it from being invalid.

Deletion

Answers of type 4 and 5 aren't actually answers. The policy across the entire network is that they should be deleted on sight, and there's no reason to make an exception here.

I think all other types should be removed as well, with varying levels of urgency. Even without an official policy, this is commonplace across the entire network. While these answers should not be flagged on other SE sites, high rep users delete them all the time.

If the answer is invalid without the poster's knowledge, it shouldn't be deleted without notifying the poster first and giving him some time to fix his mistake.

If the answer is deliberately invalid (most common for type 2), there really is no reason to wait or tell the poster something he already knows.

Self-deletion

If you figured out or have been told that your answer is invalid, I think self-deletion is the best option.

  • Deleting the answer quickly will prevent it from accumulating downvotes, which may or may not get removed after the answer has been fixed.

  • Deleting the answer yourself means that you can undelete it just as easily.

    In contrast, undeletion will require moderator intervention if the answer has been deleted by others.

  • While deleting your answer will invalidate any rep your may have earned from it, you'll get it back if you fix the answer.

Flags

Not an answer flags were creates specifically for type 5 answers, and should be cast when they are encountered.

Likewise, spam and rude or abusive flags were created for type 4 answers, and should be used instead of not an answer flags.

Very low quality flags pretty much apply to all other types, but can only be cast on answers with a non-positive score. They are also cleared automatically when the post is edited, so they may be ineffective if the intention is to get a post removed.

This leaves flagging as in need of moderator intervention.

The policy across the entire network is that flags should not be used to indicate [...] an altogether wrong answer, but I think our site should be (once again) an exception.

First of all, we have a huge moderator-to-activity ratio. Stack Overflow moderators cannot be expected to complete the time-consuming task of deciding whether an answer is valid or not, but our flag volume is rather low and checking PPCG answers is usually easier. A flag that clearly explains why the answer is invalid (i.e, which rule it violates or which test case it fails) does not require an unreasonable amount of moderator time.

Also, unlike other sites, I don't think we need examples of what not to do. There is absolutely no merit in keeping an answer that violates a rule of the challenge; they don't teach anything. If an answer passes all test cases but still produces incorrect results, there is no benefit in keeping the answer after a new test case has been added to the question.

But it is important to wait until the moderator can actually take action. I think deleting an accidentally invalid answer on sight is an unnecessarily unpleasant experience for the poster, and undeleting it requires yet another moderator intervention.

Voting to delete

Trusted users (currently 20k+ rep) can cast deletion votes on answers. In all of the above situations that suggest flagging, trusted users can also vote to delete an answer.

With three votes from trusted users, involving a moderator becomes unnecessary. However, trusted users should flag instead or on top of their delete votes unless all of the following conditions are met:

  • The answers has a negative score.

    This is a hard requirement, since even trusted users cannot vote to delete answers with non-negative scores.

  • The challenge is still active.

    Trying to gather three delete votes for an answer on a question that barely gets views anymore will take a long time. The moderation tools (10k+ rep) could help with this, but barely anybody seems to use them.

  • The answer should be removed entirely, not converted into a comment.

    Only moderators can convert answers into comments.

  • The answer is neither spam nor rude or abusive.

    Six "red" flags will not only delete the post, but hide it in the revision history and apply a 100 rep penalty to the poster. Deleting the post via deletion votes would prevent this.

TL;DR

  • Immediately flag invalid answers of type 4 as spam or rude or abusive.

  • If your own answer turns out to be invalid, delete it yourself.

  • Immediately leave a comment on all invalid answers that are not your own.

  • Immediately flag invalid answers of type 5 as not an answer.

    If you are a trusted user, the challenge is fairly active and the answer should not get converted into a comment, consider voting to delete instead.

  • Give the poster time to fix or self-delete his invalid answer of type 1 to 3.

  • If a sufficient amount of time has passed (say 48 hours) or the answer was deliberately invalid, flag the answer as in need of moderator intervention, providing an explanation of why it is invalid.

    If you are a trusted user and the challenge is fairly active, consider voting to delete instead.

  • If you fix your self-deleted answer, undelete it.

    If you fix your answer and cannot undelete it yourself, flag it as in need of moderator intervention, requesting its undeletion.

\$\endgroup\$
  • 1
    \$\begingroup\$ What if the interpreter does not yet exist at the time of the answer's posting? \$\endgroup\$ – Esolanging Fruit Apr 25 '17 at 6:21
  • 1
    \$\begingroup\$ To post an answer, there has to be an interpreter. If said interpreter postdates the challenge, the answer is non-competing. If there's no inyerpreter at all, then the language doesn't exist as far as PPCG is concerned, so the answer is invalid. \$\endgroup\$ – Dennis Apr 25 '17 at 6:27
  • \$\begingroup\$ It looks like an edit by @ais523 in March changed category 2, (inadvertently?) removing the exemption for features added after a challenge was written. Should that be changed back? \$\endgroup\$ – user2357112 May 9 '17 at 18:14
  • \$\begingroup\$ @user2357112: I was trying to minimally change the policy to be consistent with the new rule that non-free languages can be submitted as noncompeting in cops-and-robbers. Did I accidentally change something else at the same time? \$\endgroup\$ – user62131 May 9 '17 at 21:46
  • \$\begingroup\$ @ais523: The old version mentioned that a submission could be marked non-competing if it uses language features added after the challenge was posed. The new version doesn't mention anything about new language features (and it's not quite clear about languages entirely invented after the challenge). Also, "interpreter" isn't really fair to compiled implementations. \$\endgroup\$ – user2357112 May 9 '17 at 21:53
  • \$\begingroup\$ @user2357112: I don't think that's a change; a language is defined by the implementation, so using new features = using a new implementation = using a new language. "interpreter" is a mistake, though; it should say "implementation". I'll change that. \$\endgroup\$ – user62131 May 9 '17 at 21:59
  • \$\begingroup\$ As far as the "The challenge is still active." goes, you can just delete vote and post a link and an explanation in chat. It should be deleted pretty soon. \$\endgroup\$ – programmer5000 May 18 '17 at 16:10
  • \$\begingroup\$ Can I post answers produces correct results, complies with the rules, and is good enough to compete without wanting to compete myself? e.g. posting an answer to a code-golf challenge in a very non-esoteric language just for the fun of it, and thus wishing it weren't competing, but golfing it down so that it may be a serious contender as far as it's own language goes? \$\endgroup\$ – Simply Beautiful Art Nov 24 '17 at 13:28
  • 1
    \$\begingroup\$ @SimplyBeautifulArt Serious contenders do not contain superfluous code, but you can use any language you want in code golf competitions, no matter how verbose. \$\endgroup\$ – Dennis Nov 25 '17 at 11:54
  • \$\begingroup\$ Does marking an answer as non-competing exempt it from being non-serious contender? \$\endgroup\$ – Adám Jul 8 at 10:18
  • \$\begingroup\$ If answer produces correct result, but author voluntarily follows extra restrictions in addition to challenge spec (as in this case) which makes code longer than if he didn't follow these extra restrictions, does it violate Rule 2 and Rule 3? \$\endgroup\$ – Daniil Tutubalin Jul 8 at 10:56
  • \$\begingroup\$ @Adám Marking an answer as anything doesn't exempt it from anything. \$\endgroup\$ – Dennis Jul 8 at 20:08
  • \$\begingroup\$ @Dennis Thank you. Can you move "Marking an answer as non-competing does not exempt it from being invalid." to before or after the list so that is clear? Right now, it looks like it only applies to 2. \$\endgroup\$ – Adám Jul 8 at 20:10
  • \$\begingroup\$ @DaniilTutubalin It breaks rule 3. Serious contenders do not contain superfluous code \$\endgroup\$ – Dennis Jul 8 at 20:11
  • \$\begingroup\$ @Dennis that suppresses creativity and supports boreness. \$\endgroup\$ – Daniil Tutubalin Jul 8 at 21:06
19
\$\begingroup\$

Marking an answer Community Wiki does not excuse it from this policy

Introducing an answer with "I know this doesn't follow the rules but I worked hard on it so I want to post it here" is not an acceptable excuse.

Saying "I've marked it Community Wiki so I won't get any rep from it" does not change the fact that it is taking up space meant for valid answers from people who not only put the hard work in but also followed the rules.

\$\endgroup\$
16
\$\begingroup\$

The new policy should apply to answers that predate it

If the community decides to remove invalid answers (and it certainly looks like it right now), that policy should apply to all answers. Reasons for this include:

  • We are not changing the rules.

    For a policy that invalidates previously valid answers, retroactive application wouldn't be entirely fair, but that's not the case here. The answers were always invalid. The only thing that is debated now is how to deal with them.

  • Old or new, they are low quality.

    While there was no clear consensus that invalid answers should be removed, there always was consensus that they were low quality. Low quality contributions are and always have been subject to removal. In fact,

    The answer is extremely low quality: There is little to no scope for improvement

    is one of the suggested reasons for deletion for trusted users.

  • There is little to no rep loss.

    As explained in Reputation and Historical Archives, any answer with a score of +3 or higher that has been visible for 60 days or more will not cause a reputation loss if it is deleted.

\$\endgroup\$
  • 1
    \$\begingroup\$ I disagree. There are answers out there that predate the common loopholes and thous have been valid at the time they have been written, so the rules have changed. \$\endgroup\$ – Johannes Kuhn Jan 25 '17 at 0:49
  • \$\begingroup\$ Also by little you mean something like 54 reputation? \$\endgroup\$ – Johannes Kuhn Jan 25 '17 at 0:53
  • \$\begingroup\$ A +8/-8 answer is a rare edge case, precisly because that answer was already controversial when it was posted. I agree that most standard loopholes should not apply retroactively, but I also think that outsourcing the real answer is one of those that should. \$\endgroup\$ – Dennis Jan 25 '17 at 1:57
8
\$\begingroup\$

Delete answers that don't follow the spec, downvote low quality but otherwise valid answers

We need to make a distinction between unpopular answers and invalid answers. A valid but poorly-golfed (or otherwise low quality, as determined by the winning criteria) answer deserves to stay, but should be downvoted as a signal to the author that they could do better. Invalid answers should be deleted, so that new users don't get the impression that they are valid but unpopular. Since deletion can always be reversed, it gives the author the opportunity to fix the answer and undelete it or flag it for undeletion. When an answer is deleted because it is invalid, a comment should always be left explaining why it was deleted, and possibly what could be done to make the answer valid.

Ideally, a challenge's answers should all be valid (except for any deleted answers), and the votes should reflect ingenuity and impressiveness (or the lack thereof for downvotes).

This leaves open which flag should be used for invalid answers; Not An Answer and Low Quality would both suffice. I'd prefer the latter, since the former is intended for answers that don't make an attempt at solving the challenge (e.g. answers that should be comments).

\$\endgroup\$
  • 1
    \$\begingroup\$ Note that it's not even necessary to flag an answer for undeletion if the author self-deletes it. That is only necessary if the answer is deleted via 3 delete votes or by a moderator. If you find that your own answer doesn't meet the spec, deleting it while it's being fixed and then undeleting is a really uncomplicated process. \$\endgroup\$ – Martin Ender Jan 7 '16 at 15:40
  • \$\begingroup\$ @MartinBüttner Good point, that thought didn't make its way to my fingers as I was typing this. \$\endgroup\$ – Mego Jan 7 '16 at 15:41
  • \$\begingroup\$ +1 - this also prevents answers getting spammed with downvotes when the spec is unclear, as has happened a bit recently. And it's probably worth mentioning that you should leave a comment to the author as well (maybe before voting to delete?). \$\endgroup\$ – grc Jan 7 '16 at 16:12
  • \$\begingroup\$ I think this is probably OK, but I would like to see some sort of grace period (1 week?) before closure. Answerers should be given a chance to correct any errors. \$\endgroup\$ – Digital Trauma Jan 7 '16 at 16:30
  • \$\begingroup\$ Actually, I just read the TL;DR to @Dennis's answer and I think that captures better what I'm thinking of. \$\endgroup\$ – Digital Trauma Jan 7 '16 at 16:32
  • 1
    \$\begingroup\$ @DigitalTrauma Deletion is temporary and easily reversed. \$\endgroup\$ – Mego Jan 7 '16 at 16:32
  • \$\begingroup\$ @Mego true, though I think it could be off-putting to new users \$\endgroup\$ – Digital Trauma Jan 7 '16 at 16:33
  • 1
    \$\begingroup\$ @DigitalTrauma Leaving a comment as to why the answer is invalid and why it was deleted, and explaining that deletion is temporary until the answer is fixed will help mitigate that. \$\endgroup\$ – Mego Jan 7 '16 at 16:34
-4
\$\begingroup\$

Keep the answer but mark it incorrect in header

Let the answerer keep the answer as long as they label it incorrect in the header (and so noncompetitive) and explain how it fails. Any downvotes might they take from keeping it are at their own risk.

My reasoning is that invalid answers range from blatantly wrong untested answers to honest mistakes where the poster interpreted an ambiguous spec not as the asker intended. We should let votes decide between the cases. The blatantly wrong answers will get downvoted, pressuring the poster to delete them.

I want to avoid the frustrating situation where someone works hard to golf a problem, posts it, gets upvotes, and then it told they have to delete it for a reason that wasn't really their fault, such as:

  • The asker changes the question
  • The asker makes an unexpected ruling that invalidates the answer
  • The answers passes all test cases, but fails an obscure corner case that one might not expect to be valid
  • The spec is ambiguous, and the answered interpreted it a reasonable way without seeing the potential ambiguity
  • The answer breaks a condition that is in the challenge, but is so unexpected, buried in the spec or comments, and/or contrary to the rest of the challenge that a typical reader is likely to miss it. (I expect this case to be the most contentious, so I could remove it if people object to it specifically.)
\$\endgroup\$
  • 1
    \$\begingroup\$ I can understand the frustration, but I think this is dangerously blurring the lines of what is acceptable as an answer. If we allow invalid answers with a notice in the header, that still dilutes the high-quality content. We've had some trickier questions (notably those with restricted-complexity or source-layout) that attract a very high number of invalid answers that people put effort into, and I don't want to be wading through those to find the one or two valid ones. \$\endgroup\$ – Martin Ender Jan 7 '16 at 17:11
  • 1
    \$\begingroup\$ (And as the deleted answer I linked in the question shows, even deliberately, blatantly wrong answers sometimes get decent positive scores.) \$\endgroup\$ – Martin Ender Jan 7 '16 at 17:11
  • \$\begingroup\$ @MartinBüttner I see your point that voters can be poor judges of these things. Would you prefer if mods or delete voters made the call? I think the broken window harm of leaving them is greatly lessened if the answers are marked invalid, much like a historical lock. \$\endgroup\$ – xnor Jan 7 '16 at 17:20
  • 3
    \$\begingroup\$ It's not just the broken window effect I'm worried about. Allowing all kinds of invalid answers to remain on the post will be detrimental to the overall quality of our content. (Especially on challenges that attract a large portion of invalid answers.) \$\endgroup\$ – Martin Ender Jan 7 '16 at 17:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .