Caution: Why you should be skeptical about assessing autonomous technology based on disengagement reports

A few weeks ago, the Cali­for­nia Sta­te of Motor Vehi­cles published the latest series of dis­en­ga­ge­ment reports from tho­se com­pa­nies tes­ting auto­no­mous vehi­cle in the sta­te. Over­all, the num­bers paint a pic­tu­re of decrea­sing dis­en­ga­ge­ments – and con­se­quent­ly an impro­ving tech­no­lo­gy?!

Our experts regu­lar­ly con­duct on-site eva­lua­ti­ons of sta­te-of-the-art auto­no­mous tech­no­lo­gy around the world. Based on the­se, we are not con­vin­ced that dis­en­ga­ge­ment reports are a relia­ble qua­li­ty indi­ca­tor for three main rea­sons:

First: The relia­bi­li­ty of the report

You don’t lie to the gover­ning body that hands out your tes­ting cer­ti­fi­ca­ti­on, do you? Tech­ni­cal­ly, the reports can paint a ske­wed pic­tu­re without lying. Con­si­der the fol­lo­wing quo­te from one of the dis­en­ga­ge­ment reports „Most of our dis­en­ga­ge­ments do not invol­ve a fail­u­re of the auto­no­mous tech­no­lo­gy and do not requi­re an imme­dia­te take­over of con­trol by the dri­ver“. Given this state­ment and the Cali­for­nia Code of Regu­la­ti­ons Tit­le 13, Arti­cle 3.7  which offi­ci­al­ly governs auto­no­mous vehi­cle tes­ting, most of the dis­en­ga­ge­ments do not necessa­ri­ly need to be repor­ted. In pra­xis, test dri­vers may dis­en­ga­ge the sys­tem short­ly befo­re the tech­no­lo­gy “fails” (a term that in its­elf allows for some inter­pre­ta­ti­ons), thus making this par­ti­cu­lar dis­en­ga­ge­ment a vol­un­ta­ry one which is not requi­red to be repor­ted. Con­se­quent­ly, com­pa­nies tes­ting auto­no­mous vehi­cles have qui­te some fle­xi­bi­li­ty in terms of framing their dis­en­ga­ge­ment reports in one way or ano­t­her without fai­ling to com­ply with the reporting requi­re­ments.

Second: Miles are not com­pa­ra­ble

To illus­tra­te this point, let us com­pa­re two fic­tive com­pa­nies: Com­pa­ny A tests their vehi­cles exclu­si­ve­ly on a small public high­way with litt­le com­ple­xi­ty and low traf­fic. Com­pa­ny B tests in a major city with a lot of traf­fic and high com­ple­xi­ty. Com­pa­ny A may dri­ve 1’000s of miles without dis­en­ga­ging their sys­tem, while com­pa­ny B pro­bab­ly does not make a sin­gle mile without being chal­len­ged by the many ele­ments of com­plex urban traf­fic. Con­se­quent­ly, to asses the over­all robust­ness of an auto­no­mous tech­no­lo­gy, we do not only need the quan­ti­ty of test miles but also the qua­li­ty, i.e. the environment(s) the vehi­cles ope­ra­ted in. This ele­ment is rare­ly repor­ted and if so only on a very super­fi­ci­al level.

Third: Rea­li­ty does not add up

Final­ly, and pro­bab­ly most import­ant­ly, rea­li­ty just looks dif­fe­rent. Let’s con­si­der an examp­le of one firm tes­ting exten­si­ve­ly in the Bay area with a signi­fi­cant amount of tes­ting con­duc­ted in urban, thus com­plex, envi­ron­ments. Accord­ing to their dis­en­ga­ge­ment reports, they dri­ve signi­fi­cant­ly more than 1,000 miles per dis­en­ga­ge­ment. A solid per­for­mance on paper. Howe­ver, when see­ing their test vehi­cles on the stre­ets in real-life, they hard­ly make it one minu­te in den­se traf­fic without a pro­blem occur­ring. While it is dif­fi­cult at times to deci­de whe­ther a test vehi­cle is in auto­no­mous mode or dis­en­ga­ged, in many ins­tan­ces it is clear. We obser­ve count­less left turns at stop signs whe­re the auto­no­mous sys­tem is not per­for­ming in hea­vy traf­fic, inter­ac­tions with pede­stri­ans are still a chal­len­ges, and lane chan­ge attempts often lead to hea­vy brea­king and a sub­se­quent dis­en­ga­ge­ment. Hence, while reporting a num­ber of miles per dis­en­ga­ge­ment signi­fi­cant­ly hig­her than 1,000, in rea­li­ty, we haven’t seen the vehi­cles in hea­vy traf­fic do 10 miles without an issue once.

Over­all, com­bi­ning the three rea­sons, we must all be cau­tious with the publicly avail­ab­le dis­en­ga­ge­ment reports. When you real­ly want to under­stand how the many dif­fe­rent pro­jects on auto­no­mous vehi­cles per­form, you requi­re a sys­te­ma­tic assess­ment approach that goes below the sur­face. We keep on che­cking… !

Contact

Rainer Hoffmann

Seni­or Part­ner

rainer.hoffmann@huz.de

Contact

Andreas Gabriels

Head of
Busi­ness Intel­li­gence

andreas.gabriels@huz.de

Contact

Benjamin Scher

Stra­te­gy &
Inno­va­ti­on Con­sul­tant

benjamin.scher@huz.de