Attackers from the al Shabaab group charged into the block that houses the office of Mogadishu's mayor around noon and got caught in a firefight with security forces, the ministry and witnesses said.
Security killed six of the militants and cleared the area by about 6 p.m., the ministry said on its Facebook page.
Al Shabaab has stepped up attacks in a show of resilience since President Hassan Sheikh Mohamud's government launched an offensive against the al Qaeda-linked group in August.
"We were in the office and we were deafened by a blast. We ran out. Gunfire followed," Farah Abdullahi, who works in the mayor's office, told Reuters.
Abdikadir Abdirahman, director of Aamin Ambulance Services, gave a higher estimate of the death count, saying it had transported the bodies of eight civilian victims.
At least another 16 people were injured in the attack, he added.
The mayor's office is in the local government headquarters building in a well-guarded area of Mogadishu.