Part of the problem is the main demographic of people obsessing over this do not live along Finch. They are obsessed with end to end travel time, when the line is meant to service people living in between the two ends of the line. As noted, no LRT is beating a 20-min trip on the highway. An express highway service will always be faster than one that stops on local roads - and that's by design.
Of course there's been much discourse about the speed of the line and that presents challenges for those using it daily as well, but the shorter your trip gets, the less your overall journey time is affected by travel speed relative to waiting time (frequency). There are still improvements that could be made on both fronts, without focusing on the competitiveness of Humber College<>Finch West Station trips.