The fire at Po Tat Estate broke out around 3am on Friday in the fifth floor flat.
The victims are aged between two and 40.
Two other people were hospitalised, with one in a critical condition and the other stable.
The blaze was put out around 4am and firefighters are investigating the cause.
Some 150 residents were evacuated from the building, with some saying the hallways were filled with black smoke.
One neighbour expressed sadness over the deaths of the family, saying he heard cries for help in the middle of the night but the corridor was already filled with smoke.