The two YUS stations with the biggest drop are St. Patrick and Dundas - both on Dundas St.. The 8,000 lost passengers here, presumably from the Dundas streetcar reconstruction, seem to be part of the 16,000 increase at College and Queen's Park station on Carlton St. (the Carlton reconstruction was in 2004 - so it seems that this isn't part of the effect).
Queen is down 3,000 passengers, but Osgood (on Queen St) is up a bit, and King is up by almost 13,000 passengers, which is the biggest increase of any station except Bloor-Yonge (up just over 13,000 passengers). Perhaps with the notorious unreliable Queen streetcar service, there has been a shift to the more frequent King service?
The latest route ridership results may provide some answers.