As far as I'm concerned, downtown refers to the area bounded by Bathurst, Parliament, the lake and Davenport/Bloor. Yorkville, Annex are part of downtown, Cabbagetown, Parkdale, Rosedale, etc. are central but not downtown. I don't consider it to be limited by simply being a commercial/retail area, residential areas can be downtown too. The fact that there are single family homes mere blocks from office towers does not discount an area from being downtown.
I grew up in Scarborough and there was only "downtown" and then everything else, which was identified by either neighbourhood or intersection - "uptown" and "midtown" didn't exist. Never heard anyone use those other 2 terms in regards to this city until I started reading this or other similar message boards. Up/Midtown seem to be appropriated from descriptions of other cities, not Toronto and seem to be fairly meaningless in this city, since there is no accepted definition of their locations. I would never say to someone I live in "midtown", because they wouldn't have any idea where that is: could be Yonge and St. Clair, Mt. Pleasant and Eglinton or Yonge and Bloor! The fact that there is much disagreement in this topic just shows how they don't work for a city like Toronto so there isn't really any reason for trying to force them to fit.