Bulldozers arrived after dark to demolish the outer walls, and have been working through the night.
There is heavy security around the compound, which served as Bin Laden's hideout for more than five years.
Residents say an unannounced curfew has been placed in the area, and residents have been asked not to leave their homes, the BBC's Ilyas Khan in Islamabad reports.
The site is a large compound with high walls built around the actual house.
Officials say the compound was handed over by the military to the civil authorities before the operation started.
They say the demolition was decided soon after the 2 May 2011 raid, but it was put off when the government set up a judicial commission to investigate the operation by US forces.
"Since the commission has almost completed its work and did not need the compound for any further investigation, it was decided it should be razed," an official said.
He said the reason for the demolition was the visitors the place continued to attract, which posed a security threat to the area, located in an important garrison town.
Stealth operation Osama Bin Laden is said to have lived in the compound with his family for several years.
On that day, a team of US special forces flew from Afghanistan to Bin Laden's hiding place during the night in stealth helicopters on a secret operation.
They swept through the buildings within the high-walled enclosure and shot dead a total of five people, including Bin Laden.
Some 40 minutes later they left, taking with them Bin Laden's body and a hoard of computer data devices and other information containing intelligence about al-Qaeda and Bin Laden's activities.
The compound has been a painful reminder for Pakistan, which was embarrassed by the unilateral US operation that killed Bin Laden, correspondents say.