Tripoli, Libya (CNN) -- More symbols of Moammar Gadhafi's rule over Libya began to crumble Monday as forces of the country's new government took over one of the last cities loyal to him while others bulldozed the walls of his Libya compound.

Forces took over the city of Bani Walid on Monday, National Transitional Council military spokesman Abdelrahman Busin said.

Meanwhile, in Tripoli, fighters began bulldozing the outer walls of Gadhafi's Bab al Aziziya compound. Seizure of the compound by then-rebel forces in August marked the end of the regime.

Fighters participating in the effort said they may begin as soon as Tuesday to bulldoze a structure in the compound that Gadhafi called the House of Resistance, his former residence bombed by the United States in 1986 and frequently used as a site for speeches.

The fall of Bani Walid leaves Gadhafi's hometown of Sirte as the focus of military action by Libya's new government.

National Transitional Council forces will bolster their number in Bani Walid as soon as the battle for Sirte is over, Busin said.

As the battle for control of the last Gadhafi outposts continues, the U.N. human rights office expressed concern about the number of prisoners in Libya and their treatment.

"It could be up to 7,000," said Mona Rishmawi, a senior official with the Office of the High Commissioner for Human Rights in Geneva, Switzerland. "At this stage, there is no police infrastructure, there is no prison authorities. ... Right now, the Justice Ministry is not fully functional."

"There is allegations and evidence of torture" in the prisons, she said, citing lawyers, clients and human rights groups.

State Department spokeswoman Victoria Nuland also expressed concern Monday for treatment of prisoners who are "being detained apparently on their skin color and an assumption that they have supported Gadhafi."

"We urge the (National Transitional Council) to honor its stated commitment to the rule of law and respect for the universal human rights of all people in Libya," Nuland said.

CNN's Raja Razek contributed to this report.