A group of former Iranian MPs has appealed to a powerful clerical panel to investigate if Supreme Leader Ayatollah Ali Khamenei is fit to rule, BBC reported.
The call was made to the Assembly of Experts, which under Iranian law has the power to remove the supreme leader.
The letter denounces the crackdown on protests after June's disputed election and the trials which followed.
Meanwhile a senior cleric has said a reformist leader should be prosecuted for alleging protesters had been raped.
Ayatollah Ahmad Khatami said defeated election candidate Mehdi Karroubi's remarks boosted Iran's enemies, particularly the US and Israel.
Mr Karroubi has alleged that some protesters - male and female - were raped while detained in prison. He has also said that some were tortured to death.
Officials have denied the rape allegations, but have admitted that abuses have taken place.
During his sermon at Friday prayers in Tehran, Ayatollah Khatami said Mr Karroubi's claims were "full of libel, a total slander against the Islamic system" and he demanded he be prosecuted.
"We expect the Islamic system to show an appropriate response to this," Ayatollah Khatami said.
In earlier remarks reported by the Iranian ILNA news agency, he said: "If someone libels the system by saying that rape takes place in prisons, then he must either prove it or, if he cannot, then the system must press charges and the public prosecutor must act."
Former MPs' letter
The content of the letter from the group of former MPs appeared on several opposition websites. The reports did name any of the group, nor say how many had signed the letter.
Addressed to former Iranian President Ali Akbar Rafsanjani, who heads the Assembly of Experts, it demands "a legal probe on the basis of Article 111 of the constitution, which is a responsibility of the Assembly of Experts".
The article says that if the supreme leader "becomes incapable of fulfilling his constitutional duties" he will be dismissed.
The letter denounced the recent trials of protesters held in Tehran as a "Stalinesque court".
It also said Kahrizak prison near Tehran, where much of the alleged abuse of detainees took place, was worse than the US facilities at Abu Ghraib, Iraq, and Guantanamo Bay in Cuba.
There has so far been no response from the assembly to the letter.
However, correspondents say that even if the call is ignored, it is the most direct challenge to Ayatollah Khamenei so far.
The letter breaks a taboo among Iran's political classes against openly challenging the supreme leader, whose position has long been unquestioned, analysts say.
President Mahmoud Ahmadinejad won June's poll, but opposition leaders and their supporters claimed the election had been rigged. Security forces crushed the mass protests that followed.
Hundreds were arrested and opposition leaders say 69 protesters died - more than double the official figure of about 30 fatalities.