That question is actually a way into thinking about the whole issue. If you would not admit a Pantomime Dame, my guess is that is because you think they are simply a man pretending to be a woman. Fine. Next question: How about an old-fashioned male - to-female transvestite who cuts a very striking figure in high heels and booming voice? Is that person more than a Pantomime Dame, but just off-stage? If so, what makes the difference? or, What has to happen to qualify that person for a "Women Only" meeting? And what qualifies anyone to make up the rules? A degree in Gender Studies?
Germaine Greer has also said that "just because you lop off your dick it doesn't make you a woman". This is obviously true: men have their dicks lopped off in car crashes, industrial accidents and - most frequently - misadventures with military high explosives. Few if any of them breathe a sigh of relief or think "Now I can be the woman I always wanted to be". Greer is saying that even if you lose your dick as part of a self-mutilation or voluntarily undergone medical procedure, that in itself is not sufficient to make you a woman, not enough to get you into the "Women Only" meeting. That seems correct: you need a supporting story which explains why you did it and how it forms part of the "woman" identity you are claiming. It seems to me quite possible that someone whose dick is intact could have a stronger claim than the dickless person to be regarded as a woman - just as there are quite a lot of people who are more expert in the study of History than some of those who are titled Professors of History.
As it is with stories, so it is with reactions. Just because you may encounter hostile or dismissive reactions does not mean that you are morally superior to those around you. You will still have your own weaknesses, vanities, unkindnesses – things which make everyone uncomfortable with themselves at one time or another, things which we would like to wish away with a “No, that’s not me”. But we can never be entirely who we say we are or what we say we are. That's just one of life's unfairnesses.
At the back of my mind I have this thought. The history of medicine is littered with histories of doctors doing terrible things to people, supposedly to "cure" them of this or that. Some of the medical techniques employed to enable gender transitions have been around a long time: sheikhs had eunuchs in their harems, the Vatican had castrati in its choirs (until very recently), chemical castration was already around to deal with homosexuals like Alan Turing. Now we have a wider range of surgeries and chemistries. But there is a possibility that a hundred years from now, those who by then believe themselves to be progressive and humane may regard at least some of those techniques as barbaric, even when chosen, and falsely offering cures for catastrophic dilemmas which require other modes of approach.
Meanwhile, if you don't make it into the paradise of the Women Only meeting, there are still places you can go where people are too lazy to ask you for your Identity but where, hopefully, at least most people will welcome you.
Footnote: After receiving reassurances from the University that her personal safety would be guaranteed, Greer did in fact deliver the lecture she had been invited to give and which was not about transgender issues