This went up in 2009, showing the block and recommendations:
http://www.toronto.ca/legdocs/mmis/2009/pb/bgrd/backgroundfile-22247.pdf
Now these are most definitely just my opinions:
I think 580-582 and 592 Church should remain untouched. Happily, 580-582 are listed. 592 (the yellow brick apartment building), is not listed, but is one complete building. With some
serious TLC, it could probably clean up real good. I think a facedectomy would be an awful fate for it. It should be saved.
67 Gloucester: This Gloucester Mansion apartment building I think should also be perfectly retained in it's entirety. It should not be touched. It's lovely and classic.
596 Church: The other Gloucester Mansion building on the corner at is presently part dump, part pretty. It looks like what was an original house at the back was partly demolished and built right out to the sidewalk at the front. The back portion is actually quite lovely. The front part is an lousy block of a thing.
If there was some way to restore the building back to one integral, good looking structure, I'd be for it. Otherwise, I wouldn't have too many qualms about demolishing this one. It's a bad corner building.
584 Church: This house is currently operating successfully as a spa. Although it doesn't add much to the street life, it's not a pain either. Seeing as it's not a blight or a difficulty, I don't think it needs to be altered.
If someone followed my rules (har!) I don't see how a tower could go here. I would rather see a developer focus their attention either south or north where there are really problematic gaps and buildings on the streetscape.
I've highlighted empty lots - or buildings that have lifeless fronts that don't contribute much to the street - in pink.