If anything, the Olympics would bolster the argument that the Gardiner should stay due to the increased traffic in the East end. I don't like it but that's a battle left for another generation.
In terms of venues that can be brought to the Portlands that would serve the community after the Games, an Olympic Stadium could become the new home of Toronto FC and Toronto Argos. By 2024, if the current trend continues, TFC will be up for yet another expansion and a beautiful purpose built soccer stadium would take the team to another level. The Track & Field infrastructure could stay in place where Toronto could bid for future international athletic events.
This was the relatively modest proposal for the TO2008 Olympic Stadium.
An Olympic swimming pavilion could be built on the waterfront with a pool on the main level and ice rinks above as was the original plan for a stacked ice skating building in the Miller days.
The Athletes Village could be built along the Portland's Ship Channel and then converted to condos, a portion of which would be affordable housing. An Olympic rowing centre could be established on the Ship Channel itself, and then converted into a rowing club with the intention of training future Olympic athletes.
Waters edge features like boardwalks and parks along with infrastructure like hydro, water, sewage, data would establish groundwork to build the Portlands into a community, attracting private investment.
We have so much to gain from hosting an Olympic Games. That's not even counting the international exposure and the resulting influx of tourism dollars into our economy and international investment in the development of our city.