New technology for safe self-driving: Using artificial intelligence to predict pedestrian behavior and prevent accidents

A ball rolls into the road, there is a child on the pavement – which would immediately set alarm bells ringing for drivers. The result: they brake because they assume that the child will run into the road. Drivers are similarly able to guess how pedestrians will behave in other potentially dangerous situations. Self-driving vehicles, which will be increasingly dominating our streets, cannot do this. Although current technologies are able to react to critical situations, they do not have the ability to predict behaviour. A new research project is beginning at Paderborn University seeking to fill this gap and enable self-driving vehicles to identify pedestrians’ intentions before they act.

Experimental studies of pedestrians’ decision-making behaviour

The future of road traffic faces a major challenge: how can self-driving vehicles and pedestrians successfully interact – efficiently and safely? This is the question being tackled by Dr.-Ing. Sandra Gausemeier and Dr. rer. medic. Tim Lehmann. Their idea: self-driving vehicles should be able to identify intended actions using a combination of AI (artificial intelligence) methods and motion analysis. This is a promising new approach. Dr. Gausemeier is an expert in driver assistance systems working as part of the ‘Control Engineering and Mechatronics’ research group at the Heinz Nixdorf Institute. This means that his everyday work involves conducting research in the model-based development of mechatronic systems. Dr. Lehmann is a scientist in the Exercise Science and Neuroscience section of the Department of Exercise and Health. He specialises in researching human motor behaviour and the underlying neurocognitive processes. These two researchers have joined forces for the project, with work including experimental studies on humans’ decision-making behaviour. These will later serve as a foundation for predictive algorithms used in self-driving vehicles. 

Bold scientific ideas

The researchers have received Paderborn University’s Research Award for their efforts. The university leadership awards this accolade (and 150,000 euros) in recognition of exceptional research projects that think outside the box and are therefore promoting bold scientific ideas. ‘With its combination of artificial intelligence and neurocognitive analysis, this project is seeking to bring about a paradigm shift in the interactions between people and autonomous systems. As well as being hugely relevant to society, this is also visionary in the best sense’, said Professor Thomas Tröster, Vice-President for Research and Academic Careers at Paderborn University.

More than just collision computations

‘Our aim is to develop an AI-based system that is able to evaluate pedestrians’ intended future actions based on their motor activity, predict their behaviour, create risk profiles and thus proactively avoid critical situations’, Dr. Gausemeier explained. To achieve this, the first ever experimental studies are being conducted on humans’ decision-making behaviour in real urban scenarios. ‘This goes well beyond simulation-based or laboratory-based studies and addresses the complex, hugely dynamic interactions between humans and machines. Autonomous systems would then be able to do more than simply make collision computations, but rather also incorporate pedestrians’ situational awareness and distractions into manoeuvre planning’, Dr Lehmann added.

Pattern recognition to record human motion

The behaviour of other vehicles is determined by factors such as traffic regulations. This means that the number of potential actions is limited to only a few options. Pedestrians are not subject to such strict limitations, meaning that they have much more flexibility in their potential movements and decisions: ‘Machine learning (AI) methods will be used here to gain an understanding of the complexity of human motion patterns lasting several seconds with the help of pattern recognition, and predict intended actions with a high level of reliability’, Dr. Lehmann noted.

Improved safety for all road users

When it comes to pattern recognition using AI, the quality of the training data is key. To ensure this, the researchers are planning to develop a multi-stage process with a variety of data. As Dr. Gausemeier explained: ‘For the data collection, test subjects will be equipped with eye tracking, mobile electroencephalography (which measures brain activity), multi-sensor mobile measuring systems and motion capturing. This will enable us to classify the effects of situational parameters and cognitive, cerebral decision-making behaviour in terms of the resulting motion patterns.’ After training, the autonomous systems should be able to identify intentions using only the on-board camera images and use these to infer future motion patterns. ‘This solution approach could substantially improve the safety of all road users’, Professor Tröster said. The team expects to have initial results in early 2027.