I don't know that removing these things is going to change any of that, either. Until i see some empirical evidence saying they help or hinder, I'm going to remain indifferent to them. Although I agree keeping all of them in one area is a recipe for trouble.
I agree that certain neighbourhoods take on more than their fair share of these duties. I was thrilled when the Salvation Army didn't open on Parliament. I think cabbagetown does more than it's fair share, and i would love it if there were more shelters in other areas to disperse some of the extreme poverty to other areas of the city. There is definitely a case for your idea of a "critical mass" for this type of thing. I think at a certain point, it becomes more than the some of its parts, and cabbagetown has surpassed that point.
However, the idea that people on parole or probation should not be downtown is ridiculous and wrong. Even people with criminal records have their rights as citizens, and I would much, much, rather live somewhere where this was true than have a place sterilized by a system such as the one you've proposed. Can you imagine the powers you'd have to give the police to enforce such a system? Terrifying. No, Mr. Beez, it is their right to be wherever they want, although not to do whatever they want. That's when the power of the state should be used. What you are proposing smacks of fascism. It's amazing to me what you're willing to do to others so that you may live in your utopia.
Also, it is sad you see all these people as vermin. Some have major mental disorders and are receiving no help from anyone. Can you really see a schizophrenic on the street and feel disdain for them? No pity?
As Mr. Ebeneezer Scrooge said "Are there no prisons? Are there no workhouses?"
Moreover, you'd be surprised how many non-vermin are involved in the drug trade. Maybe your doctor, or a TTC driver, or lawyer, or your children's teacher, wouldn't be able to go to work downtown due to being caught with a bit of hash in their house.