Am I in the wrong?

I’ll try to make this somewhat confusing story short. I invited my man to a boat party this weekend with roughly 50 of my friends and this was his first time meeting them all. We were having an amazing time and everyone was getting along great. Rewind to the night before. My man went out and got drunk with friends, so I told my friend there was a chance he was going to oversleep and not come due to being hungover, as we had to meet the next morning at 8:30. He did end up coming. One of my friends texted me, “did _____ come?” and I responded “unfortunately” as a joke. That’s just how her and I banter with each other other. Obviously If I didn’t want him to come, I wouldn’t have invited it. Back to being on the boat, he sees the message and get instantly upset and leaves. I tried explaining so many time that it was a joke, but he didn’t care. Now he’s “unsure if he should be with me” and “needs time to figure out if this relationship is the right thing because he doesn’t know anymore”. Is what I did that big of a deal for the relationship to end over? I’m really beating myself up about it.