My mother was one of those women who learned to play golf. It didn't get her ahead in her work, it just got her on the golf course. Her entire career she has watched younger and less capable white men sail past her in salary and opportunity. Want to guess how many times she's been called aggressive or strident? She's achieved some success, but make no mistake, she's been bashing her head against the glass ceiling her entire life and has had to struggle for every achievement.
Since you seem unwilling to dig into the reasons why people were upset about the golf comments and about his most recent comments, let me spell it out for you: it's systemic, stupid.
John Tory's denial of white privilege, along with saying women never came to negotiate higher salaries and that women should learn to play golf if they want to advance their careers misses the point that people of colour and women are held back by much greater forces that they have little or no control over. Tory's insensitivity to those realities is what makes everyone so upset.