I’m curious on everybody’s thoughts about this one?

I'm asking this question seriously and not trying to play "gotcha" and throw strawmen at you. I'm genuinely curious about this.

In the past, I was guilty of being the guy espousing the very logic in my post title, but I was much younger and politically blind. Over time, as I matured, became more politically aware, and had more women in my life I started to understand their plight a bit more. I started to walk back on seeing everything "women's only" as unfair because I realized there was a lot of things geared toward men or men dominating co-ed spaces and shutting the women out (be it intentionally or not).

An example I remember from younger was the advent of the all-female gym, Curves. I can understand why women don't want to be around men in a place where they're working on their bodies, potentially wearing revealing clothing, etc. because they will feel judged and creeped on. However, my skewed understanding is that if an explicitly male-only gym was created, this would cause some backlash, lambasted in the media, and could potentially get shut down. Now, I could be completely wrong in that assumption because I'm going on sensationalist examples from the media throughout time that might not actually be the case. Maybe for every all-male space that gets dragged in the public, there's 10,000 that operate unabated. Historically, any time I've been in a room full of men who wanted to do "men things", the conversation typically involves into disparaging women and people who aren't white/American, so I don't totally eyeroll at women thinking these spaces are inherently toxic, because my experience as a man in said spaces fits the mold.

My question to you is this, if indeed men wanted to have a public space to be explicitly all-male (as in advertised as such), will that ever be allowed without it getting dragged and shut down? Or is the assumption always going to be that the men doing it are up to no good and need to be removed? Conversely, is this just a wacky media stereotype and women in general don't really care if men have their little clubhouses to themselves as long as they aren't using it as a means to harm others?

(Personally, IDGAF if I have an all-male space to be a part of, so I'm not here to whine about it. The thought occurred to me more if men are just flailing in all directions when they make the "Imagine if men..." arguments, or is there actual validity to it?)