There’s actually a color-blind rule here that’s fairly simple: An insult can be friendly banter if it can be thrown right back at you.
Source: Slurs: Who Can Say Them, When, and Why | The Weekly Sift
This article is a wonderfully clear explanation of why and how slurs can be asymmetric, as well as how they differ from “curse” words. Also, why taboos are for children but you should still watch what you say. Well worth reading.