That goes both ways. It becomes a way for drivers to know if they need to speed up to get themselves through in time.
The old flashing hand system put ambiguity into the equation, forcing drivers to second guess as to whether they had enough time to make it through; often opting to err on the side of caution. The same applies to pedestrians.
"After controlling other factors, the mixed-effect model results further indicate displaying a PCS (Pedestrial Countdown Signal) to drivers increase the approaching speed by approximately 11 km/h." -
An empirical analysis of the effect of pedestrian signal countdown timer on driver behavior at signalized intersections; Wooseok Do, Nicolas Saunier & Luis Miranda-Moreno
In effect, if the countdowns are visible to drivers, they promote rushing through before light changes and thus make intersections less safe.
A better solution would be polarized filters on signals to allow only a very narrow viewing angle, or countdown signals at angles not visible to drivers (say, at the beg button locations). Or, just reverting to old-style flashing hands.