The technology uses sensors and cameras to capture surrounding conditions and records status data for the vehicle such as the steering wheel angle, road conditions, weather, visibility and speed. The AI system learns to recognize patterns with the data.
Image: Taras Makarenko / Pexels

AI recognizes potentially critical traffic situations seven seconds in advanceNew early warning system for self-driving cars

A team of researchers at the Technical University of Munich (TUM) has developed a new early warning system for vehicles that uses artificial intelligence to learn from thousands of real traffic situations. A study of the system was carried out in cooperation with the BMW Group. The results show that, if used in today’s self-driving vehicles, it can warn seven seconds in advance against potentially critical situations that the cars cannot handle alone – with over 85% accuracy.

To make self-driving cars safe in the future, development efforts often rely on sophisticated models aimed at giving cars the ability to analyze the behavior of all traffic participants. But what happens if the models are not yet capable of handling some complex or unforeseen situations?

A team working with Prof. Eckehard Steinbach, who holds the Chair of Media Technology and is a member of the Board of Directors of the Munich School of Robotics and Machine Intelligence (MSRM) at TUM, is taking a new approach. Thanks to artificial intelligence (AI), their system can learn from past situations where self-driving test vehicles were pushed to their limits in real-world road traffic. Those are situations where a human driver takes over – either because the car signals the need for intervention or because the driver decides to intervene for safety reasons.

Pattern recognition through RNN

The technology uses sensors and cameras to capture surrounding conditions and records status data for the vehicle such as the steering wheel angle, road conditions, weather, visibility and speed. The AI system, based on a recurrent neural network (RNN), learns to recognize patterns with the data. If the system spots a pattern in a new driving situation that the control system was unable to handle in the past, the driver will be warned in advance of a possible critical situation.

“To make vehicles more autonomous, many existing methods study what the cars now understand about traffic and then try to improve the models used by them. The big advantage of our technology: we completely ignore what the car thinks. Instead we limit ourselves to the data based on what actually happens and look for patterns,” says Steinbach. “In this way, the AI discovers potentially critical situations that models may not be capable of recognizing, or have yet to discover. Our system therefore offers a safety function that knows when and where the cars have weaknesses.”

Warnings up to seven seconds in advance

The team of researchers tested the technology with the BMW Group and its autonomous development vehicles on public roads and analyzed around 2500 situations where the driver had to intervene. The study showed that the AI is already capable of predicting potentially critical situations with better than 85 percent accuracy – up to seven seconds before they occur.

Collecting data with no extra effort

For the technology to function, large quantities of data are needed. After all, the AI can only recognize and predict experiences at the limits of the system if the situations were seen before. With the large number of development vehicles on the road, the data was practically generated by itself, says Christopher Kuhn, one of the authors of the study: “Every time a potentially critical situation comes up on a test drive, we end up with a new training example.” The central storage of the data makes it possible for every vehicle to learn from all of the data recorded across the entire fleet.


Christopher B. Kuhn, Markus Hofbauer, Goran Petrovic, Eckehard Steinbach: “Introspective Failure Prediction for Autonomous Driving Using Late Fusion of State and Camera Information,” IEEE Transactions on Intelligent Transportation Systems (2020), DOI: 10.1109/TITS.2020.3044813

More information:

Prof. Eckehard Steinbach is a member of the Board of Directors of the Munich School of Robotics and Machine Intelligence (MSRM). The MSRM is an integrative research center of TUM. Its purpose is to research the fundamentals of robotics, perception and artificial intelligence in order to develop technologies for the areas of healthcare, work and mobility.

Technical University of Munich

Corporate Communications Center Christine Lehner

Contacts to this article:

Prof. Dr. Eckehard Steinbach
Technical University of Munich
Chair of Media Technology
Tel: +49 89 289.23503 or +49 89 289 23504

Related articles at

Prof. Wolfgang Kellerer in his lab.

“6G will be human-focused”

From hologram-based communications to remote-controlled surgical procedures – the 6G wireless standard will make many high-tech applications a reality. A major project now being launched at the Technical University of…

The future cluster M Cube is researching the urban mobility of tomorrow.

M Cube future cluster wins federal funding

The Munich Cluster for the Future of Mobility in Metropolitan Regions (M Cube), under the leadership of the Technical University of Munich (TUM), is among the winners of the German government’s Clusters4Future competition.…

Computer scientists have developed software that prevents autonomous cars from causing accidents.

The accident preventers

Before autonomous vehicles participate in road traffic, they must demonstrate conclusively that they do not pose a danger to others. New software developed at the Technical University of Munich (TUM) prevents accidents by…

The Blickfeld founders Mathias Müller, Florian Petit and Rolf Wojtech (f.l.).

Far-sighted innovation for autonomous driving

Blickfeld, a company launched at the Technical University of Munich (TUM), has won Germany's most richly endowed start-up prize: the "Start me up!" award. It recognizes the company's powerful and cost-effective lidar…