Waymo safety study shows not all self-driving cars are created equal
A new joint study between Waymo and Swiss Re indicates their cars are safer than human drivers when operating fully autonomously
The past few weeks haven’t been good for self-driving cars. Early in October a Cruise self-driving car ran over a woman in San Francisco and dragged her along the street. On October 24 the State of California suspended Cruise’s license to operate its fleet of driverless cars. Cruise stopped all operations in the US two days later. In the past few days reports have started emerging that indicate Cruise vehicles have difficulties identifying children. And yesterday it was announced that Cruise is removing all 950 of its driverless fleet off the road for a software update.
Things are so bad that AI expert Gary Marcus asked “Could Cruise be the Theranos of AI?”
Yet overshadowed by all the self-driving-cars-gone wrong headlines around Cruise, a report recently came out about Waymo — another company with driverless vehicles on the road in California, as well as Arizona — that tells a very different story.
The report is the result of a collaboration between Google spin-off Waymo and reinsurance giant Swiss Re. It compares third party insurance claims involving Waymo vehicles and human-driven vehicles in the same operating area over the same time period. And the results indicate self-driving Waymos are significantly safer than their human counterparts.
I’ve been keeping an eye on Waymo for a while now, and must confess that I like the company.1 They’ve been operating in the Phoenix area for some time, and have shown a very cautious, slow-and-steady, and responsible approach to developing and deploying their cars.
For some time now it’s been possible to dial up a driverless Waymo in the Phoenix metro area, much as you would order an Uber, and have a car sans driver take you to your destination. The same service is available in San Francisco, and until a few days ago was offered alongside Cruise’s driverless-vehicles.
My experiences riding in driverless Waymo vehicles have always been positive. For sure there are occasional annoyances and idiosyncrasies — cars taking the long (but safe) route, cars driving themselves as if possessed by a hesitant 80 year old, and cars temporarily flummoxed by unexpected obstacles. But these annoyances have all been fleeting, and certainly not behaviors that impact safety.
Yet despite my good experiences and periodic shock at how appallingly bad some human drivers are, anecdote is no substance for data. And so I was intrigued to read the Swiss Re report.
Here, it’s worth noting that Swiss Re is a very serious — and very prominent — reinsurance company. Their business depends on cold, hard analysis of risk and liability, with no room whatsoever for fudging the figures or making decisions on PR rather than solid data. Likewise their reputation stands on their ability to demonstrate that they are trustworthy.
In other words, I cannot imagine a scenario where Swiss Re sacrifices the very core of its business by putting out a report based on spin rather than solid analysis.
This is important, because the report shows very clearly that, in nearly 4 million miles driven without a person behind the wheel, Waymo vehicles are substantially safer than their human counterparts.
The report compared third party insurance claims associated with Waymo vehicles, with those from human drivers within the same geographical area. This — according to Swiss Re — allows a much more accurate assessment of crashes and associated damage than police reports, which tend to underrepresent incidents.
There are a number of statistics presented in the paper, but the ones that stood out to me are that, across 3,868,506 miles driven by Waymo vehicles with no human driver behind the wheel, there were zero insurance claims involving bodily injury, and only 0.78 claims for property damage per million miles driven.
To put these into context, in the same region and over the same time period there were 1.11 claims for bodily injury per million miles driven by humans, and 3.26 claims for property damage per million miles driven by humans.
In other words, while you’d expect a few injuries associated with people driving nearly 4 million miles, there were none associated with the Waymo vehicles. The risk, it seems, is so low that even with the masses of data available to Swiss Re, they weren’t able to quantify it.
Where there are data that allow risk to be quantified, the Waymo vehicles were 76% less likely to be involved in crashes that led to property damage than human drivers. This is even more impressive when you take into consideration evidence that many crashes involving Waymo cars are associated with poor human drivers rear-ending them.
These figures certainly support my admittedly anecdotal assessment of the technology’s safety. But I must confess that I was concerned by one aspect of the analysis.
The data on human drivers from Swiss Re wasn’t road-specific. In other words, it included crashes on interstates as well as other roads. However, the Waymo driverless service does not currently operate on interstates.
Was there a chance that this disparity between roads driven along somehow biased the analysis?
The report’s authors acknowledge this, and conclude that “[d]ue to variations in collision frequency per million miles between freeways and non-freeways, this may have led to a baseline which may be more conservative than a roadway- matched baseline …” but that “the impact of these differences is expected to be negligible.”
Not convinced, I did some back-of-the-envelope calculations based on data from the National Highway and Transportation Administration and the Federal Highway Administration, just to see whether this difference in roads could have somehow messed the analysis up.
It turns out that humans driving off-interstate are more dangerous than when on interstates — at least as far as crashes go. The data comparison is a little messy as NHTSA data are based on police reported incidents. But for the whole of the US in 2021, the incidence of crashes leading to death, injury, or property damage, were three times higher for non-interstate roads than they were for interstates per mile travelled.
In other words, the human driver baseline used in the report most likely underestimates the number of crashes on the types of roads used by Waymo vehicles. And by inference, the actual reduction in risk associated with the driverless Waymo cars is probably greater than reported.
Of course, there remains the serious question of how safe self-driving cars should be, and whether comparing them to humans is the right metric. Yet unlike the Cruise vehicles, at least we now have clear evidence that the Waymo cars are significantly safer than humans within the specific boundaries of how and where they are being used — and by a substantial degree.
This is great news for anyone — me included — who worries about how we’ve normalized loss of life, limb, and property resulting from inattentive and poorly trained humans being allowed behind the wheel of fast-moving hunks of metal.
And even though there’s still a long way to go to ensure that driverless vehicles are as safe and as beneficial as possible under a wide range of circumstances, it just goes to show that, when it comes to cars that drive themselves, not all technologies — or companies — are created equal.
In the spirit of full disclosure, I should be clear that Waymo once sent me a pair of socks. I don’t think this has clouded my judgement in this piece, but they are nice socks. In all seriousness, I do occasionally receive care packages that Waymo send out to people they have engaged with in some capacity or another — these usually contain items like coffee, travel mugs, or socks. And while I have never worked for or been paid by Waymo, it’s important to note the existence of these.
Cruise have never sent me socks.
I'd like some socks.
Thanks Andrew - a very interesting read. Is there another issue here associated with the data being from collision causation? If you're in a vehicle (lets say a robotaxi) does it matter who is to blame if you're involved and injured? Claims wouldn't be made against the robotaxi company if they're rear-ended but the blame is of little comfort to the person in the back of the robotaxi who suffers from whiplash. A robotaxi that stops abruptly for no apparent reason causing an accident doesn't make it ok even if the law says the vehicle behind is at fault. So perhaps data on collisions - whatever the cause - is also important. (Also - I enjoyed my week's-worth of free Waymo rides. No socks to declare).