I looked at CDC data from 2006-2008 (before the average of virginity rised a little). Of men ages 25-29, 3.8% have never engaged in any sexual contact with the opposite sex, compared to 3.1% of men ages 30-34, 1.3% of men ages 35-39, and 1.2% of men ages 40-44. This means 92.3% of men who never had any sexual contact in their late 30s will remain that way in their 40s! 34.21% of men who never engaged in any sexual contact by their older 20s will stay that way in their late 30s and 42% of men who never engaged in any sexual contact by ages 30-34 will stay that way in their late 30s. Of the ones who stay this way in their late 30s, only 7.7% will lose it by their early 40s. Among those who had 0 sexual partners (meaning no vaginal intercourse partners), 45% of men who had 0 sex partners in their early 30s still don't in their late 30s, and only 7% of the ones in their late 30s find a sex partner by their early 40s. 36.84% of those with 0 sex partners by their older 20s won't have any partners by their late 30s.

On the other hand, among women ages 25-29 who are virgins (3.4% of them), 55.88% stay that way at age 30-34 (compared to 81.58% of male virgins in their older 20s). 47.37% of women who are virgins in their early 30s stay that way in their later 30s. But guess what? Of female virgins in their older 30s, 44.44% stay that way by their early 40s compared to almost 93% of male virgins in their older 30s. This means only 21% of female virgins in their early 30s stay that way in their early 40s and only 11.76% of female virgins in their older 20s stay that way at 40.

There's slightly more male virgins than female virgins in older 20s (3.8% vs 3.4%), whereas in early 30s, male virgins are more common (3.1% vs 1.9%) and in older 30s, 1.4% of men and 0.9% of women are virgins and 1.3% of men and 0.4% of women are virgins. Even in ages 20-24, there were more male than female virgins (12.6% of women that age and 14.4% of men that age).

And remember, the men who lost their virginity late probably were virgins by choice or waiting until marriage or maybe used hookers. Among involuntary virgins, the odds they stayed that way at 40 might be much higher because virgins by choice managed to still have seduction skills.

https://www.cdc.gov/nchs/data/nhsr/nhsr036.pdf