image: A simulated traffic scenario from the experiments with Autoware, where a red car in the adjacent lane cuts into the path of the autonomous vehicle.
Credit: Duong Dinh Tran from JAIST.
Ishikawa, Japan -- Imagine a future where you can sit back and relax while your car drives you safely to your destination. Self-driving cars promise this convenience, but before we fully embrace this technology, one crucial question remains: Can we truly trust them to handle anything the road throws their way?
In this view, Research Assistant Professor Duong Dinh Tran from Japan Advanced Institute of Science and Technology (JAIST) and his team, including Associate Professor Takashi Tomita and Professor Toshiaki Aoki at JAIST, decided to put the open-source autonomous driving system, Autoware, through a rigorous verification framework, revealing potential safety limitations in critical traffic situations. To thoroughly check how safe Autoware is, the researchers built a special virtual testing system. This system, explained in their study published in the journal IEEE Transactions on Reliability, acted like a digital proving ground for self-driving cars.
Using a language called AWSIM-Script, they could create simulations of various tricky traffic situations—real-world dangers that car safety experts in Japan have identified. During these simulations, a tool called Runtime Monitor kept a detailed record of everything that happened, much like the black box in an airplane. Finally, another verification program, AW-Checker, analyzed these recordings to see if Autoware followed the rules of the road, as defined by the Japan Automobile Manufacturers Association (JAMA) safety standard. This standard provides a clear and structured way to evaluate the safety of Autonomous Driving Systems (ADSs).
Researchers focused on three particularly dangerous and frequently encountered scenarios defined by the JAMA safety standard: cut-in (a vehicle abruptly moving into the ego vehicle's lane), cut-out (a vehicle ahead suddenly changing lanes), and deceleration (a vehicle ahead suddenly braking). They compared Autoware's performance against the JAMA's "careful driver model," a benchmark representing the minimum expected safety level for ADSs.
These experiments revealed that Autoware did not consistently meet the minimum safety requirements as defined by the careful driver model. As Dr. Tran explained, "Experiments conducted using our framework showed that Autoware was unable to consistently avoid collisions, especially during high-speed driving and sudden lateral movements by other vehicles, when compared to a competent and cautious driver model."
One significant reason for these failures appeared to be errors in how Autoware predicted the movement of other vehicles. The system often predicted slow and gradual lane changes. However, when faced with vehicles making fast, aggressive lane changes (like in the cut-in scenario with high lateral velocity), Autoware's predictions were inaccurate, leading to delayed braking and subsequent collisions in the simulations.
Interestingly, the study also compared the effectiveness of different sensor setups for Autoware. One setup used only lidar, while the other combined data from both lidar and cameras. Surprisingly, the lidar-only mode generally performed better in these challenging scenarios than the camera-lidar fusion mode. The researchers suggest that inaccuracies in the machine learning-based object detection of the camera system might have introduced noise, negatively impacting the fusion algorithm's performance.
These findings have important real-world implications, as some customized versions of Autoware were already deployed on public roads to provide autonomous driving services. "Our study highlights how a runtime verification framework can effectively assess real-world autonomous driving systems like Autoware. Doing so helps developers identify and correct potential issues both before and after the system is deployed, ultimately fostering the development of safer and more reliable autonomous driving solutions for public use,” noted Dr. Tran.
While this study provides valuable insights into Autoware's performance in specific traffic disturbances on non-intersection roads, the researchers plan to expand their work to include more complex scenarios, such as those at intersections and involving pedestrians. They also aim to investigate the impact of environmental factors like weather and road conditions in future studies.
###
|
About Japan Advanced Institute of Science and Technology, Japan
Founded in 1990 in Ishikawa prefecture, the Japan Advanced Institute of Science and Technology (JAIST) was the first independent national graduate university that has its own campus in Japan. Now, after 30 years of steady progress, JAIST has become one of Japan’s top-ranking universities. JAIST strives to foster capable leaders with a state-of-the-art education system where diversity is key; about 40% of its alumni are international students. The university has a unique style of graduate education based on a carefully designed coursework-oriented curriculum to ensure that its students have a solid foundation on which to carry out cutting-edge research. JAIST also works closely with both local and overseas communities by promoting industry–academia collaborative research.
About Research Assistant Professor Duong Dinh Tran from the Japan Advanced Institute of Science and Technology, Japan
Dr. Duong Dinh Tran is a Research Assistant Professor at the Japan Advanced Institute of Science and Technology (JAIST). His research focuses on the practical application of formal methods to ensure the trustworthiness of critical systems, particularly in the realm of autonomous driving and next-generation automotive platforms. Currently, he is involved in the "Formal Methods and Verification Tools for Next-generation Automotive System Platforms" project, collaborating with Professors Toshiaki Aoki and Takashi Tomita. Dr. Tran earned his Ph.D. in Information Science from JAIST in September 2023 under the supervision of Professor Kazuhiro Ogata.
Funding information
This work was supported by JST, CREST Grant Number JPMJCR23M1.
Journal
IEEE Transactions on Reliability
Article Title
Safety Analysis of Autonomous Driving Systems: A Simulation-based Runtime Verification Approach
Article Publication Date
1-May-2025