Many times I have had answers and found out they were invalid due to a obscure meta post. But people have said things like: "This is the 3rd time people have done this and it is still not funny", "This is still invalid just like the other removed answers". I can't see those answers and I believe ~500+ should be able to see removed answers to help avoid this. BUT they can't see who made this answer.
There's nothing we can do about this
The perk of seeing deleted answers are set on 2000 rep, and is set by the SE team. It's not a local rule to set reputations for these perks.