The south end of the Spadina line isn't really the problem for the most part. King, Front, Bremner, Queens Quay are all locations it's worth having stops at.
The bigger problem, IMO, is the top end - Sussex and Harbord are so close to each other it's almost offensive (if really necessary, they could've split the difference and had one Sussex/Harbord stop midway between the two), and Harbord - Wilcocks is scarcely better. South of Spadina Circle, Nassau, Sullivan, and Richmond need to go, as does the pointless traffic light by Clarence Square, and also the one protecting the entrance into Queen's Quay loop. This would be a start, but the inane safety rules also need to go.
The other day, I had the misfortune of taking the 510 up and I noticed that at EVERY SINGLE track intersection that was also protected by traffic lights (i.e. King, Adelaide, Queen), after stopping at the traffic light, my driver would speed up to about 2 km/h after the lights cleared, then stopped for the switch, then proceeded again. I'm not sure if this guy was just an excessively paranoid nut, or if "respecting the traffic lights" cannot be combined with "stop/check/go" under TTC policy, but whatever it is, it's garbage.