The ugliest 'building' on earth is probably some kind of Stalinist shack in Eastern Europe that has been commandeered by gypsies. A large number of the earth's structures probably don't even meet safety standards, they would automatically beat anything in Toronto as they may not even do their job of providing shelter. If you add the caveat of architect designed buildings in the first world, then Toronto would probably have some contenders for ugliest buildings. Out of buildings which were specifically designed by an architect to be beautiful, the Ryerson Library is notable for its' ugliness.
Is it the ugliest building in this category? How can anyone say? It makes more sense to just categorize buildings as opposed to trying to individually rank them. On one hand you have five star buildings like the Swiss Re building or the Bank of China tower in HK. Then maybe buildings like Bay Adelaide or the 4SC, buildings which do their job pleasantly and well, but are ultimately unremarkable. It could also include buildings that are remarkable in their appearance, but impractical otherwise. New City Hall may fit into this category, along with things like San Fran's Federal Building.
In the one star category you get buildings like Ryerson's Library. It wasn't original (if ever a concrete box was). The choice of concrete in Toronto's climate was questionable to begin with, I am unaware of a material that weathers worse, maybe cotton candy. It doesn't even use concrete to its' full potential, like Boston City Hall or even Robart's. That the building is justified because a quarter of it isn't meant to be seen is the architectural equivalent of calling a woman beautiful, with the prerequisite of putting a bag on her face. It's hardly unique in this category, probably thousands of buildings around the world could qualify, at least a dozen from around Toronto. They aren't original, they aren't appealing and they are usually built on the tail end of a more impressive building in the attempt to capitalize on other people's work