Defence Secretary John Hutton has become the latest minister to quit the government - although he says he will remain loyal to Gordon Brown, BBC reported.
It comes after James Purnell quit as work and pensions secretary with a call for the PM to "stand aside" to prevent Labour defeat at the next election.
Mr Brown is reshuffling his top team as he fights for his political future.
Alan Johnson moves to the Home Office but Chancellor Alistair Darling and other key figures stay in place.
Universities Secretary John Denham is expected to succeed Hazel Blears as communities secretary - after earlier suggestions he might move to health.
Chief Secretary to the Treasury Yvette Cooper is expected to replace Mr Purnell as work and pensions secretary.
Mr Hutton said he thought fellow Blairite minister James Purnell had made "the wrong decision".
"I'm standing down from the cabinet today because I'm leaving frontline politics," Mr Hutton told the BBC.
"I'm not going to be contesting my seat in the next general election and I think it's absolutely right that Gordon, who I'm supporting as our prime minister and party leader, should have a cabinet that's going to take him through the next election and beyond."
He denied that as the fourth cabinet minister to quit in recent days he was "leaving a sinking ship".
BBC Political Editor Nick Robinson said Mr Hutton's decision not to attack Mr Brown - coupled with the loyalty of Alan Johnson, who was tipped by some backbenchers as a possible leadership contender - had shored up his position as prime minister.
...