That's a great explanation, although I don't think the CN Tower is actually the epicentre. See, the tower is actually itself isolated from the rest of the city by virtue of being on the wrong side of the railway tracks and being so far west. I'd venture that King & Bay is probably the "most downtown", although you could also argue that Union Station also qualifies. City Hall might sound good, but Queen St. is the outer edge of the financial district and pretty far North!
For what it's worth, I live just North of Wellesley between Yonge & Bay, and I think of there being a few distinct "downtown" areas - financial district (Front-Queen), "downtown" (Queen-College), and midtown/uptown/Bloor-Yorkville (College-Bloor). It's all relative though, so there's no point in arguing about it!