Since I normally drone on about the inherent racism underlying popular images of Western empire, I'll take a different tack and say, let's be generous:
Part of this does come from a misunderstanding of the process of decolonisation. People do a little reading and are stunned by how, in the final years of empire, Britain, France et al seemed to suddenly rush out of their colonies without leaving the infrastructure, educational, governmental and bureaucratic systems necessary to build a lasting state.
The reader sometimes thinks that the lesson here is that decolonisation should have happened, but it should have happened slower. Hence, longer lasting empire.
It's an easy mistake to make, but it breaks down when you consider that any empire interested in building up their colony to that extent wouldn't have colonised them in the first place.
More to the point, before those final years decolonisation was often a process of people delaying and delaying and murdering and delaying for as long as they could.
I don't pretend this explains all of the phenomenon, mind you, but I wanted to change my record.