How do i get over the fear of getting people upset when they realize i dont give a fuck?

Because whenever I dont give a fuck and dont let people get in my head for a second, is when they try even harder to bring you down the most it seems. Like this crab in a bucket mentality and to be quite frank, the audacity of them trying to get a reaction out of me, annoys me. So i go back to caring.