Police said they received a report from workers on Muk Tai Street at around 9:30am.
The scaffolding that came down was estimated to be 20 metres high.
Video footage circulating online appeared to have captured the moment the structure crumbled. Several workers were close to the site at the time, but were left unscathed.
The Fire Services Department said it deployed 54 firefighters and seven ambulance personnel, but construction workers had left the scene before they arrived.
The Labour Department said it had launched an investigation into the incident.