Baroness Sayeeda Warsi said the party needed an inquiry if it was to claim it was inclusive when she spoke at Byline Festival in East Sussex on Saturday.

Discussing Boris Johnson’s controversial burkas comments, she said: “British Muslim women who wear the face veil or anything that could be considered out of the ordinary for mainstream Britain are easy political battlegrounds.

“We saw that with Boris.

“It shouldn’t be that British Muslim women are a convenient political battleground for old Etonians to be able to increase their poll ratings. It shouldn’t be that British Muslim women are a useful tool to be able to get you back in the press.”

The first Muslim woman Cabinet member said no-one should be able to “have a pop and get away with it”, adding: “If we are genuinely committed to being an inclusive party, there has to be an inquiry.”

A message needed to be sent that “this kind of aggression, this kind of stereotyping or abuse or whatever we want to call it against a community will not be tolerated”, she said.

Mr Johnson has made no response on the subject since his comments in the Daily Telegraph column.

It was not “just” the Conservatives, and mainstream political parties including Labour needed to do some “soul searching”, Baroness Warsi said.

The former criminal lawyer, who said while in politics she was described as the “enemy within”, claimed the party’s headquarters previously decided the “British Muslim community didn’t really matter” when it came to votes.

She also branded the term “British values” a “dishonest concept” and a “false term” because problems like the gender pay gap still exist amid a society that promotes equality.

The four-day independent journalism and free speech festival is taking place in Pippingford Park in the Ashdown Forest near Uckfield.

Sports presenter and former footballer Gary Lineker is due to talk about immigration and Brexit on Sunday, while Cambridge Analytica whistleblower Chris Wylie will appear on Monday.