Is there something wrong with not talking to people?

I was at work today and when I went on my break, the breakroom was full of people talking and chatting about someone's daughter or something. I just went in and sat there without saying anything and went on my phone to go read some articles.

I thought nothing of it, but then once virtually everyone left except for a man and a woman, the man slammed his hand on the table I was sitting on (in a comedic way) and said: "say something!".

I just told him I had nothing to say and went on with my day. Later on, a woman that I was having a casual conversation with whilst we were working told me that I am very quiet.

Is there something wrong with being quiet? I understand that it's anti-social and to be honest, part of me believes that people don't want to talk to me, but it's also partly because I genuinely don't want to talk.