Why is being physically bigger considered as important and guys often brag about being bigger than others?

Organized crime existed where the heads were not necessarily bigger and faster than ordinary men.

No matter how big a man is, I think a group of man can take him easily.

So why so we still think, being bigger is important? Is this lack of organization in modern life?

Is it simply because we find ourselves alone when someone wants to puts us through threat of violence?