Much of the push toward self-driving cars has been underwritten by the hope that they will save lives by getting involved in fewer crashes with fewer injuries and deaths than human-driven cars. But so far, most comparisons between human drivers and automated vehicles have been at best uneven, and at worst, unfair.

The statistics measuring how many crashes occur are hard to argue with: More than 90 percent of car crashes in the U.S. are thought to involve some form of driver error. Eliminating this error would, in two years, save as many people as the country lost in all of the Vietnam War.

But to me, as a human factors researcher, that’s not enough information to properly evaluate whether automation may actually be better than humans at not crashing. Their respective crash rates can only be determined by also knowing how many non-collisions happen. For human drivers is it one collision per billion chances to crash, or one in a trillion?

Assessing the rate at which things do not happen is extremely difficult. For example, estimating how many times you didn’t bump into someone in the hall today relates to how many people there were in the hallway and how long you were walking there. Also, people forget non-events very quickly, if we even notice them happening. To determine whether automated vehicles are safer than humans, researchers will need to establish a non-collision rate for both humans and these emerging driverless vehicles.

Comparing appropriate statistics

Crash statistics for human-driven cars are compiled from all sorts of driving situations, and on all types of roads. This includes people driving through pouring rain, on dirt roads and climbing steep slopes in the snow. However, much of the data on self-driving cars’ safety comes from Western states of the U.S., often in good weather. Large amounts of the data have been recorded on unidirectional, multi-lane highways, where the most important tasks are staying in the car’s own lane and not getting too close to the vehicle ahead.

Automated cars are rather good at those kinds of tasks – but then again, so are humans. The data on fully automated systems will naturally expand to cover more roads as states allow automated vehicles to operate more widely. But it will take some time before self-driving cars can cover as many miles in a year and in as many circumstances as human drivers presently do.

It is true that self-driving cars don’t get tired, angry, frustrated or drunk. But neither can they yet react to uncertain and ambiguous situations with the same skill or anticipation of an attentive human driver, which suggests that perhaps the two still need to work together. Nor do purely automated vehicles possess the foresight to avoid potential peril: They largely drive from moment to moment, rather than thinking ahead to possible events literally down the road.

To an automated vision system, a bus shelter full of people might appear quite similar to an uninhabited corn field. Indeed, deciding what action to take in an emergency is difficult for humans, but drivers have sacrificed themselves for the greater good of others. An automated system’s limited understanding of the world means it will almost never evaluate a situation the same way a human would. And machines can’t be specifically programmed in advance to handle every imaginable set of events.

New tech brings new concerns

Some people may argue that the promise of simply reducing the number of injuries and deaths is enough to justify expanding the use of driverless cars. I do agree that it would be a great thing if tomorrow were the dawn of a new day when a completely driverless roadway killed or injured no one; although such an arrangement might suck more of the enjoyment from our everyday lives, especially for those who love driving.

But experience from aviation shows that as new automated systems are introduced, there is often an increase in the rate of adverse events. Though temporary, this potential uptick in the crash rate may cause concern for the general public and then politicians, lawmakers and even manufacturers – who might be discouraged from sticking with the new technology.

As a result, comparisons between humans and automated vehicles have to be performed carefully. This is particularly true because human-controlled vehicles are likely to remain on the roads for many years and even decades to come. How will people and driverless cars mix together, and who will be at fault for any collisions between them?

To fairly evaluate driverless cars on how well they fulfill their promise of improved safety, it’s important to ensure the data being presented actually provide a true comparison. Choosing to replace humans with automation has more effects than simply a one-for-one swap. It’s important to make those decisions mindfully.

I have, in the past, received funding from many organizations for my work on ground transportation. As of this day, I am not in receipt fo such funds (although I hope to secure some continuing funding shortly. All such funding is openly expressed upon my vita; available at: www.peterhancock.ucf.edu.