They are slower than driving except in peak traffic. Caltrain san Francisco to san Jose is about 2x driving time, and on neither end does the train get you into real downtown. San Jose is close but still a 15 minute walk before you get to anything interesting. Francisco is in a relatively shady area near a stadium, also 15 mins from market.
If you don't have to park, getting to the airport by transit involves switching from Caltrain to BART at a random suburb so as an example San Jose to SFO is 30 mins by car, 1:30 by train. Note the tracks for the Caltrain and Bart are parallel here.
If you mean to say "trains with high average speeds have high average speeds" I'll agree, but even in countries trying to get faster average speeds it's still relatively new outside of Japan.
There's also cost to consider. You can get a fast train Paris to Berlin and it's 4 hours faster than driving, but costs more than 4x to transport 1 passenger vs a car which can bring 5. The main route is slow, matching the speed of a car. (If you had a family, would you pay 60 euros to drive, 520 euros to train at car speeds, or 932 euros to train faster?)
However the "fast" train is still only reaching normal highway speeds (120-130 kmph) on average...it's just the roads are so bad google is estimating an average car speed of less than 80 kmph which is essentially like...suburb/business area street speed here in this country.