If you want to get technical, "East Asians" (Chinese, Japanese, Korean, etc) comprise 11% of Toronto, and South Asians/Middle Easterns comprise 10% of Toronto, and blacks comprise 8%. Meanwhile in San Francisco, Asians comprise 31%, Latin Americans comprise 20% and blacks comprise 8%. I do NOT see how those figures are any less diverse than Toronto's. I think the only difference is there are larger amounts of visible minorities in San Francisco, and they have hispanics instead of South Asians.
And the differences between someone from South Asia is even more trivial than the differences between someone who is Latin American. For example, New York has puerto ricans, mexicans, dominicans, etc all classified as Hispanics. Are you really going to say Toronto is diverse simply because South Asians can be from Pakistan, India, etc? Like I said, San Francisco and New York seem equally as diverse as Toronto, just replace South Asians with Hispanics.
And in my opinion, the reason there is neighbourhood segregation is because of income disparities, not because of racist/prejudice attitudes. Hispanics and African Americans tend to be associated with issues of poverty. Whereas, this is not the case for many asians. It's not like white people never have to see visible minorities. They interact with different groups on a daily basis.