Whether or not you take into account the entire driving record (which you arguably should not since people's driving ability changes over time), another relevant fact is that some people are unusually bad drivers - causing lots and lots of accidents or near-accidents all the time. The few drivers who are actively a menace to everyone else make the rest of us look good by comparison. There's an upper limit on how good a driver you can be and most drivers aren't terribly far from that limit - they know their abilities and drive within them most of the time. But there's no strong lower limit on how bad a driver you can be. So we all grok intuitively that a few people who are drunk, high, angry, near-blind, aggressive or just terribly lacking in judgement are way over there on the other side of the driving-well scale from us; that makes us "above average drivers".
Actually by your argument, there is a strong upper bound. Therefore, by basic statistics reasoning, the mean is no longer a remotely valid measure of "average" - mode is much better in this case. Therefore average still constitutes most people. If you strictly are defining "average" to be mean, then sure, everyone is an above mean driver, however if you take average to be defined in a way statistically relevant, then it means "the measure of population that encompasses most of the population - the typical value of the population". (hint, if it is the typical value, most people can't be better than typical, it is definitionally impossible). Basically, I'm taking the long way of telling you to stop being disingenuous if you want to make an argument.
