I may be in the minority, but I say no. You should downvote it or leave a comment explaining why it breaks the rules.
An answer that is not an answer should be deleted (should be a comment, doesn't answer the question etc...).
Answers that are answers but are factually wrong or are actively bad (say promote SQL Injection) should get downvoted, but not deleted.
I'm of the opinion that answers that don't meet spec are factually wrong. They are absolutely an attempt to answer the question, and they serve as an example of what the community sees as "bad".
If you delete answers that don't meet the spec, you're simply inviting more answers that break the same rules, because nobody can tell it's already been tried and shot down. Showing people that we downvote might discourage that.
Of course, there are exceptions. It's simply not a black/white issue. Many of these answers are a simple slip or misreading of the spec. Some are obviously horrible, but in almost all cases, I believe in downvoting.
Lack of downvoting is something I see as a bit of a problem here. There are tons of answers that break rules, don't meet spec, or simply don't work. I downvote them when I come across them, and I was shocked to recently find out that I am the #1 downvoter here among those with over 100 total votes. That just seems insane to me, because I upvote pretty often, too.