Bolivian police on Wednesday said they recovered the presidential sash and medal that had been stolen Tuesday night, Xinhua reported.
"We received an anonymous call indicating they were left in the atrium of Saint Peter's Church, in black bags, inside of which was the valuable material," Johnny Aguilera, director of the La Paz special police task force against crime, said.
The items were snatched from a car belonging to a military official who was in charge of taking them to the central Bolivian city of Cochabamba, where President Evo Morales was to wear them at a ceremony marking the 123rd anniversary of the founding of the armed forces.
At the time of the theft, the military official was reportedly at a bordello in the city's red light district.
The official has been arrested and turned over to authorities for dereliction of duty, Aguilera told the state Bolivian Information Agency.
Police are looking for the thieves, who have been identified as Peruvian citizens, he added.
"The fact that they left the jewels doesn't mean the act goes unpunished," said Aguilera.
Video from security cameras showed two individuals opening one of the car doors and taking a backpack that contained the valuable items.
Morales was seen at the ceremony without the medal and sash, which bears the colors of the Bolivian flag and is worn during his inauguration and other official events.
The medal dates back to the country's founding in 1825, when it was presented to Latin American independence hero Simon Bolivar.
Both items are in the care of the Central Bank of Bolivia and kept in a vault until needed. The military is solely responsible for transporting them to and from events.