From what I understand, it is more about how greedy the designer is when they size the turbo. It is possible to put in a (relatively) small turbo and get a nice performance gain without much lag at all (as is the case in my car), but if you really want to squeeze the max from an engine you can strap a massive turbo to it and deal with the lag. This is common in the smaller displacement cars in Europe and Asia. 200HP from 1L is totally possible this way, but you'll need to keep it above 3000RPM at all times.
There are ways around that of course, variable geometry turbos and the various twin turbo systems will get you there.
Dave is right. Chrysler started using variable vane turbos in the 90's which helped with the lag. Also, variable length/volume induction system also helped by allowing the intake manifold to have longer narrower (low volume) runners at low engine speeds with butterfly type valves that would open up at higher engine speeds to allow higher air volumes. Longer/larger/higher volume runners would be better at moving air at higher speeds, producing more power. Narrower runners would have better flow characteristics at low speed to give better low end torque.
A more knowledgeable explanation can be found here: https://www.youtube.com/watch?v=4jruZL5UCSA
Also it comes down to turbo size as well. Smaller turbos have lower inertia (faster response) but less ability to move air (lower power production). That's why twin turbos give you a lot of potential...you can have different size turbos and stage them or just have two smaller ones and and get the responsiveness you want with the airflow you want for peak power. Not sure which is the typical way twin turbo cars to it.