Don't limit yourself to the Financial District. Focus on the area bordered by King / Spadina on the West, King / Sherbourne on the East and King / College on the North. I lived in Mozo at King / Sherbourne - 15 min walk to First Canadian Place - and a great neighbourhood - St. Lawrence Market...