Lord Sheikh, who founded the Conservative Muslim Forum, said the former foreign secretary was not a “superhuman being” and should lose the Tory whip.

It comes after Theresa May joined calls for Mr Johnson to apologise for the remarks made in a Telegraph column as he made the case against an outright ban on full face veils.

But a source close to the Tory bigwig refused to back down and said the furore was a bid to shut down free speech.

Mr Johnson wrote in his Monday newspaper column that it was "absolutely ridiculous that people should choose to go around looking like letter boxes".

And he said school teachers or university lecturers would be within their rights to refuse to talk to students who arrived at class "looking like a bank robber"

Lord Sheikh, who used to advise David Cameron, told BBC Newsnight: “Take the whip from him. Why not? He's not a super human being, he's a member of the party.

“The party chairman, the Prime Minister has the right to take the whip. It's not out of order and that's the thing I'd like to see.”

Addressing journalists on her visit to Edinburgh yesterday, the Prime Minister said the remarks “clearly caused offence” and joined Tory chair Brandon Lewis in calling for an apology.

“Some of the terms Boris used describing people's appearance obviously have offended,” she said.

“What's important is do we believe people should have the right to practise their religion and, in the case of women and the burka and niqab, to choose how they dress.”

But a source close to Mr Johnson said: “It is ridiculous that these views are being attacked - we must not fall into the trap of shutting down the debate on difficult issues.

“We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists.”

The remarks drew criticism from figures across the political divide, including from senior Labour figures and from key representatives of the Muslim community.

Other Tory figures, including Armed Forces Minister Tobias Ellwood and Culture Secretary Jeremy Wright, also weighed in to condemn their colleague.

Mr Wright told the BBC people should "be careful how we express ourselves and the words we choose" and said branding the burqa "ridiculous" was "the sort of language we should try and avoid using".