Scaramouche has always had a special place for me. Great service that isn't overbearing, food is amazing, view is great, desserts are to die for...too bad their lease expires soon.
It's on Avenue Rd just south of Davenport (on westside) in lower level sharing the same building with Spuntini Ristorante. It's the go to place for celebrities during TIFF. It's as intimate as it gets.
I tried Bymark for the first time recently... I was extremely impressed. Everything was solid, from the drinks to appetisers to main courses to dessert. Best lamb chops I've ever had. I hear their burgers rock too. The atmosphere is modern and cool, not sure I'd describe it as romantic.
North 44 is also a solid choice for food, maybe a little cosier than Bymark.
Southern Accent on Markham Street just south of Honest Eds/Suspect Video. The food is always great (and adventurous!), moderately priced, never had bad service and the atmosphere rocks. Drinks follow afterward at Panarama atop the Manulife Centre as a great way to continue the evening.