Caution: Why you should be skeptical about assessing autonomous technology based on disengagement reports
A few weeks ago, the California State of Motor Vehicles published the latest series of disengagement reports from those companies testing autonomous vehicle in the state. Overall, the numbers paint a picture of decreasing disengagements – and consequently an improving technology?!
Our experts regularly conduct on-site evaluations of state-of-the-art autonomous technology around the world. Based on these, we are not convinced that disengagement reports are a reliable quality indicator for three main reasons:
First: The reliability of the report
You don’t lie to the governing body that hands out your testing certification, do you? Technically, the reports can paint a skewed picture without lying. Consider the following quote from one of the disengagement reports „Most of our disengagements do not involve a failure of the autonomous technology and do not require an immediate takeover of control by the driver“. Given this statement and the California Code of Regulations Title 13, Article 3.7 which officially governs autonomous vehicle testing, most of the disengagements do not necessarily need to be reported. In praxis, test drivers may disengage the system shortly before the technology “fails” (a term that in itself allows for some interpretations), thus making this particular disengagement a voluntary one which is not required to be reported. Consequently, companies testing autonomous vehicles have quite some flexibility in terms of framing their disengagement reports in one way or another without failing to comply with the reporting requirements.
Second: Miles are not comparable
To illustrate this point, let us compare two fictive companies: Company A tests their vehicles exclusively on a small public highway with little complexity and low traffic. Company B tests in a major city with a lot of traffic and high complexity. Company A may drive 1’000s of miles without disengaging their system, while company B probably does not make a single mile without being challenged by the many elements of complex urban traffic. Consequently, to asses the overall robustness of an autonomous technology, we do not only need the quantity of test miles but also the quality, i.e. the environment(s) the vehicles operated in. This element is rarely reported and if so only on a very superficial level.
Third: Reality does not add up
Finally, and probably most importantly, reality just looks different. Let’s consider an example of one firm testing extensively in the Bay area with a significant amount of testing conducted in urban, thus complex, environments. According to their disengagement reports, they drive significantly more than 1,000 miles per disengagement. A solid performance on paper. However, when seeing their test vehicles on the streets in real-life, they hardly make it one minute in dense traffic without a problem occurring. While it is difficult at times to decide whether a test vehicle is in autonomous mode or disengaged, in many instances it is clear. We observe countless left turns at stop signs where the autonomous system is not performing in heavy traffic, interactions with pedestrians are still a challenges, and lane change attempts often lead to heavy breaking and a subsequent disengagement. Hence, while reporting a number of miles per disengagement significantly higher than 1,000, in reality, we haven’t seen the vehicles in heavy traffic do 10 miles without an issue once.
Overall, combining the three reasons, we must all be cautious with the publicly available disengagement reports. When you really want to understand how the many different projects on autonomous vehicles perform, you require a systematic assessment approach that goes below the surface. We keep on checking… !
