Advice?

I’m trying to break my pick me ways so please be gentle with the comments.

I have this guy that I used to see many years back, like at least 6 years ago. We’ve stayed friends the entire time but we don’t talk on a normal basis. He just hits me up when he’s bored. (Red flag, I know)

The thing that bothers me the most is I’m such a nice person that when he asks me to come over I always feel bad for saying no because I say no like every single time. And I say no because 99.76% of the time he’s always trying to do something sexual with me. (Red flag #2)

I guess I’m asking for advice for how to not feel guilty for saying no. Not only to this dude but just in general. I know he’s red flags everywhere but I do feel guilty for saying no to him.