It is two-fold. The last 500 m on Lake Shore can sometimes take 20 minutes.If I recall correctly, the issues are with the traffic outside of the downtown core, specifically on the Gardiner, not the approach to the bus terminal, so I doubt ramps would have made a difference here.
That side view rendering really puts the size of these buildings in perspective. From bridge to bridge, phase 2 is about at wide as it is tall.About half of it. You can see details and a rendering here: https://www.toronto.ca/legdocs/mmis/2022/te/bgrd/backgroundfile-225604.pdf