The panel of retired police and military officers told the Senate that Iraqi forces should be replacing US troops by early next year. ( BBC )
But the report also warned Iraqi troops would not be ready to take over fully within the next 18 months.
It also said the Iraqi police force was ineffective and should be scrapped.
The report is the latest in a series to be considered by the US Congress as it debates the Iraq war.
Security surge
The panel's head, Gen James Jones, told the Senate Armed Services Committee: "The force footprint should be adjusted in our view to represent an expeditionary capability and to combat a permanent-force image of today's presence.
"Significant reductions, consolidations and realignments would appear to be possible and prudent... This will make an eventual departure much easier."
Democrats welcomed the call for a reduction of presence as supporting their plans for US redeployment.
But Gen Jones did not back their call for a deadline on US troop withdrawal.
"I think deadlines can work against us. I think a deadline of this magnitude would be against our national interest."
The US has this year deployed thousands more troops to Iraq to carry out a "security surge" in and around Baghdad.
Analysts say the Jones report, released on Thursday, is not necessarily in opposition to the strategy of President George W Bush, who has supported handing over responsibilities to Iraqi troops.
But the report's verdict on the readiness of Iraqi forces to take the lead will once again raise the question of just when US troops can be safely brought home.
The Iraqi military showed signs of progress, Gen Jones said, and was "gaining size and strength, and will increasingly be capable of assuming greater responsibility".
Iraq's national police should be disbanded, the Jones report says
But he said it was "not at a rate sufficient to meet their essential security responsibilities".
He also said the national police force was ineffective and so rife with sectarianism it should be scrapped.
A Pentagon official said the administration did not back such a move.
Iraq's interior ministry told Reuters news agency that the issue of sectarianism was being tackled and that the ministry did not agree with the panel's assessment.
"We admit there were some problems before due to sectarian loyalties but this involved just a few people," ministry spokesman Brig-Gen Abdul-Kareem Khalaf told Reuters.