Author: Elliot Williams / Source: Hackaday
In 2016, a Tesla Model S T-boned a tractor trailer at full speed, killing its lone passenger instantly. It was running in Autosteer mode at the time, and neither the driver nor the car’s automatic braking system reacted before the crash. The US National Highway Traffic Safety Administration (NHTSA) investigated the incident, requested data from Tesla related to Autosteer safety, and eventually concluded that there wasn’t a safety-related defect in the vehicle’s design (PDF report).
But the NHTSA report went a step further. Based on the data that Tesla provided them, they noted that since the addition of Autosteer to Tesla’s confusingly named “Autopilot” suite of functions, the rate of crashes severe enough to deploy airbags declined by 40%.
That’s a fantastic result.Because it was so spectacular, a private company with a history of investigating automotive safety wanted to have a look at the data. The NHTSA refused because Tesla claimed that the data was a trade secret, so Quality Control Systems (QCS) filed a Freedom of Information Act lawsuit to get the data on which the report was based. Nearly two years later, QCS eventually won.
Looking into the data, QCS concluded that crashes may have actually increased by as much as 60% on the addition of Autosteer, or maybe not at all. Anyway, the data provided the NHTSA was not sufficient, and had bizarre omissions, and the NHTSA has since retracted their safety claim. How did this NHTSA one-eighty happen? Can we learn anything from the report? And how does this all align with Tesla’s claim of better-than-average safety line up? We’ll dig into the numbers below.
But if nothing else, Tesla’s dramatic reversal of fortune should highlight the need for transparency in the safety numbers of self-driving and other advanced car technologies, something we’ve been calling for for years now.
How Many Crashes Per Million Miles?
How is it possible that the NHTSA would conclude that Autosteer reduces crashes by 40% when it looks like it actually increases the rate of crashes by up to 60%? And what’s with this shady “up to” language? The short version of the QCS report could read like this: Tesla gave the NHTSA patchy, incomplete data, and the NHTSA made some assumptions about the missing data that were overwhelmingly favorable to Tesla, and fairly unlikely to reflect reality.
Crucially, none of the data about miles driven with Autosteer engaged have ever made it out of Tesla’s hands. What Tesla gave the NHTSA, and which they claimed represented a trade secret, was the mileage on the odometer before and after Autosteer was installed, the mileage on the car at last check, and whether or not there was an airbag deployment. From this, the most you can conclude is about the accident rate per mile of cars with Autosteer installed, rather than the rate of accidents per mile with Autosteer active. Only Tesla knows what percentage of miles are driven with Autosteer on, and they’re not telling, not even the government.
How You Get a Range of +40% to -60%
So did the NHTSA get data on how many miles were driven with Autosteer installed? Not quite. Tesla gave them data with the “Previous Mileage before Autosteer Install Reported” — an odometer value. They also gave “Next Mileage after Autosteer Install Reported” — another odometer value but there was no guarantee this would match up with the previous readout and the data varies quite a bite here.
For data samples where these two numbers did match, you can assume that’s the mileage when Autosteer was installed. We’ll call these cars “known mileage” — they correspond to Figure 1 in the QCS report. Known-mileage cars make up 13% of cars in the sample, and for these cars the accident rate increased by 60% after Autosteer was installed. There were 32 airbag deployments for 42 million miles driven before Autosteer was installed, and 64 deployments over 52 million miles afterwards. At first blush, this looks like Teslas with Autosteer installed appear to crash…
The post Does Tesla’s Autosteer Make Cars Less Safe? appeared first on FeedBox.