This would not be fully non-res to residential, but I would do this to the "Nightmare on Elm Street" - the brutalist apartment on Elm west of Bay:
1. Leave the apartments as-is, except perhaps much more funky windows
2. Convert the above-grade parking garage to townhouses, keeping the concrete exposed
3. Gut and renovate the row homes between the apartment building and Bay Street. They could become townhouse-condos as part of the complex
4. Renovate and perhaps redevelop the Bay St. hotel on the corner of Bay and Elm. Put retail at the bottom
5. Develop a Mozo-like new building directly to the south in the existing parking lot. Underground parking would replace the existing above grade lot in the "Nightmare", and serve the entire complex
By doing this, the fabric of Elm could be pulled across Bay Street, the Uno Prii building could be saved/upgraded, and new housing and commercial could be added filling a major gap on Bay St.
I hear the bus terminal will eventually move elsewhere, so the direct vicinity is ripe for improvement.