First, my daughters had tons of opportunity to play sports growing up in Toronto, so I don't 'know' that sporting infrastructure for children is a 'joke' at all. Second, you changed your argument dramatically from 'inspiring people to be fit' to saying the infrastructure is worth it. That's very different.
Firstly, the problem is not finding sports leagues to play in. There are tons of those, and that just indicates how popular sports of all types are in the Greater Toronto Area. The problem is that a lot of the infrastructure across the region was old, and needed renovation. A lot of the infrastructure required for elite sports (ie. Olympic ones) did not even exist. There wasn't any major impetus to upgrade any facilities until Pan Am came along, and it didn't look like that was going to change either.
Secondly, maybe I'm phrasing my argument badly. But it boils down to this, I think the event is good because it inspires people to play sport at all levels. To get 5,000 Pan-Am athletes, you need to whittle that down from tens of thousands of contenders. I think the infrastructure cost and investment was probably worth it. We certainly wouldn't build any of what we built for any less than we did. The security costs seem tremendous, but I'm not an expert on that, so it's hard to comment. At the end of the day, though, the infrastructure investment would never have happened without the event itself ... so we might have to look at that cost as simply the cost of infrastructure renewal.
Here's a small example, because it is local for me. The Etobicoke Swim Club, which is located at the Olympium, has a team of 250 competitive swimmers. But they also host programs for thousands of non-competitive swimmers. Most "elite facilities" (such as the Velodrome) operate or will operate in a similar way. The elite competitive training is only a part of the program. They've also hosted a number of provincial and national competitions, which attract tourists to the city. I believe the upgrade was worth about $20m, and I find it highly unlikely that something like that would have happened without Pan Am.
And despite the new construction being built for the games, pretty much all of the new builds also have facilities that are scaled for local fitness programs, as they all contain multi-function rooms, fitness rooms, gymnasiums, etc. The Aquatic Centre, for instance, might have 2 Olympic sized pools and a diving tower, but it also has four gymnasiums, a climbing wall, a running track, racquet courts, and a fitness centre. The Markham Pan-Am Centre after the games will provide tons of gymnasium space, multi-function rooms, a pool, and a huge fitness-centre for the people of Markham in a new area that currently doesn't have much of anything. The Athletes Village after the Games will become condos (which are likely to sell well), but we also got a new YMCA out of the deal.
EDIT:
As for
TIm Hortons Field, the Provincial contribution was actually rather small (about $22m) with the remainder split between the City of Hamilton ($54m) and the Federal Government ($69m). The Ti-Cats signed a 20-year lease for $1.2m a year. I suppose Hamilton sees economic value in supporting the team, so I'm not going to argue it, when local Toronto teams (and others all across NA) have also received public funding.