A Midland peer yesterday called on Tory leader Michael Howard to sack one of the party's vice-chairmen in a new row over racism.
The black Conservative peer Lord Taylor of Warwick said Mr Howard should fire Andrew Rosindell after he defended Ann Winterton, the MP who made a racist joke about the deaths of the 20 Chinese cocklers in Morecambe Bay.
Lord Taylor also expressed concern at the way the Conservative whip had been "quietly" restored to Mrs Winterton last month after she finally apologised for her remarks.
He said it was clear she had been readmitted to the parliamentary party on a "false premise" after she said in a newspaper interview that she had been playing a "game of tokenism" when she made her apology.
Lord Taylor said her apology had obviously not been genuine and urged Mr Howard to withdraw the whip again.
Appearing on BBC1's Question Time on Thursday night, Mr Rosindell defended Mrs Winter-ton after she was compared to Ron Atkinson who quit after a racist comment about a black player was caught on air.
Mr Rosindell said there was a "huge difference" between the two cases as Mrs Winterton's comments had been intended as a joke made at a private function for which she had now apologised.
While he had made clear that he did not condone her remarks, he said that "a lot of jokes would have to be banned" if the same standards were applied across the board.
"Of course I don't like those remarks, or agree with them in any sense at all, but it was at a private dinner party that it happened. At the end of the day you can't regulate what goes on at a private dinner party," he said.
"Of course, people do say things in private which they would not say in public.
"You could take this to the extent whereby the sensitivities are so great that no one dare say anything at all... Ron Atkinson was not making a joke so there is a totally different situation."
However Lord Taylor, who is the only black Conservative in either House of Parliament, accused Mr Rosindell of "double standards" and said that his position was no longer tenable.
"He is not fit to be vice-chairman of the Conservative Party. He should step down from that position or be sacked. He is sending out the wrong messages," Lord Taylor said.
He said Mr Howard should also withdraw the whip again from Mrs Winterton after an interview earlier this month in a Sunday newspaper in which she said: "It is demanded I celebrate tokenism. And I will play the game of tokenism and apologise".
She was also quoted as saying that Enoch Powell had been "right" on immigration and that the country "can't go on absorbing more and more people who don't appreciate British ways".
Lord Taylor said: "It is quite clear that she has got no remorse whatsoever and was only giving an apology to 'play the game'. The whip was restored to her on a false premise and I think Michael Howard has been misled."
Lord Taylor warned that if the Tory leader failed to act it could undermine the progress the party had been making through the promotion of strong black and Asian candidates.
Last night Mr Rosindell said in a statement: "I am firmly opposed to any expressions of racism and, as I made clear last night (Thursday), I did not like the remarks made by Anne Winterton.
"It was right that she apologised for any offence caused."
However Labour MP Martin Salter said Mr Howard must act against both Mr Rosindell and Mrs Winterton.
"If Ron Atkinson isn't fit to be a pundit, then Ann Winterton isn't fit to be a MP and Andrew Rosindell is quite clearly not fit to be vice-chairman of the Conservative Party," he said. "Racist jokes are never acceptable and Michael Howard should take action."