I think that's a pretty quick knee jerk reaction, the issue isn't about "poor people" having a place to live, there are incredible number of affordable housing developments in Toronto right now, the regent park development alone will have in excess of a dozen current and future TCHC buildings. Also if you have noticed, there has been no opposition to the planned block 32 and 36 developments which contain a significant amount of affordable housing.
The owners in City Place are more unhappy with change to the scale and nature of this particular development. Again if the city had made it clear that this was going to be a 41 storey affordable housing building in block 32 before West One, N1, Montage, Luna, and now Parade was sold and built, there would be no issue. The problem is that the general feeling is that the City and TCHC waited untill the "Paying Customers" put up the capital to build up this neighbourhood and now wants to reap the benefits. If the city was committed to this area and the affordable housing model, they should have built the TCHC Buildings, the library, school and community centres first.
In any case I see any additions to the overall diversty of my neighbourhood to be a strength, but I do understand why a lot of owners would be unhappy with this development.