Senior Liberals say Sussan Ley's removal was about polling, not gender. But experts warn it may reinforce doubts about the party's appeal to women.