Richmond and Adelaide are fine the way they are. What needs change in terms of traffic rules is Queen St. I am surprised that parking is allow on this busiest commercial street, making the already congest traffic even worse. Cars should be forbidden to enter queen st from Yonge to Buthurst, to make this street a lot more pedestrian friendly. Street cars, however, should be able to run on it. People on Queen West will J-walk no matter what as both sides are full of interesting store, and right now, it is very dangerous. Driving on Queen is a nightmare and shouldn't be allowed in the first place.
As to Richmond and Adelaide, as someone pointed out, the problem is not one way or two way, it is the the use of these streets. There is hardly anything on Adelaide street, and not much on Richmond either save one busy portion. So why people would ever stop there? Most people will either bike or drive along these two just to get somewhere (just like on Bay st). For a pedestrian, I'd walk on King or Queen.
Additionally, Yonge St between Queen and Bloor should be pedestrian only as well, if we really want to be pedestrian friendly. Drivers can use either Bay or Jarvis street easily. Why would anyone choose to drive on downtown Yonge anyway?