Can a traffic world model generate realistic roadside multimodal data?

To overcome the scarcity of diverse roadside traffic data, researchers developed the Traffic World Model, or TWM, a unified framework that generates controllable roadside images, traffic videos, safety-critical events, and LiDAR-style point clouds.

 

Large-scale, accurately annotated traffic data are essential for autonomous driving and intelligent transportation systems. However, collecting multimodal data from infrastructure-mounted cameras and LiDAR sensors is expensive and time-consuming. Moreover, safety-critical situations, such as road debris, lane-occupying construction equipment, accidents, heavy fog, and snow, occur infrequently and are difficult to collect at scale.

 

The team published their study in Communications in Transportation Research (https://doi.org/10.26599/COMMTR.2026.9640045).

 

“Most existing driving-generation models focus on the ego-vehicle perspective. Roadside traffic generation is more challenging because infrastructure sensors observe wider areas, more complex road structures, and traffic participants at substantially different distances and scales,” says Chunmian Lin, a researcher at Beihang University. “We therefore designed TWM specifically for roadside-view multimodal data generation. The framework first creates a controllable roadside image and then uses that image as a common scene anchor for video, safety-critical event, depth, and LiDAR-style point-cloud generation.”

 

A shared image-centered framework for four generation tasks

At the core of TWM is a conditional Diffusion Transformer, or cDiT. The model receives three complementary types of input: textual descriptions of environmental conditions, projected three-dimensional bounding boxes describing traffic participants, and map-derived priors representing road topology. The textual descriptions specify background characteristics such as the time of day, weather, illumination, and atmospheric conditions. Object boxes control the position, category, and spatial distribution of vehicles and other traffic participants, while the map prior constrains the structure of roads and lanes. Together, these inputs allow the model to control not only the appearance of the generated scene but also its spatial organization. After generating a roadside image, TWM uses it as the visual foundation for three additional generation branches.

 

For video generation, the researchers introduced traffic-aware prompt expansion, which supplements the original text prompt with information about fixed-camera viewpoints, traffic-flow directions, lane-following behavior, and temporal-consistency requirements. The video model is further adapted to roadside-surveillance footage using low-rank adaptation, or LoRA.

 

For safety-critical event generation, the researchers formulate the problem as a conditional image-editing task. Event-related objects are extracted from traffic images, semantically annotated, quality-filtered, reconstructed into coarse three-dimensional assets, and inserted into target scenes at specified locations, poses, and scales. This approach allows the model to create rare traffic situations involving road debris, traffic cones, warning signs, construction equipment, accident-related vehicles, and other potential hazards while maintaining consistency with the original road environment.

 

For LiDAR generation, TWM first predicts a dense pseudo-depth map from the input image. Sparse LiDAR measurements and monocular-depth estimates are combined through outlier filtering, global scale alignment, and edge-aware residual propagation. The predicted depth is then projected into three-dimensional space and transformed into the LiDAR coordinate system to create LiDAR-style point clouds.

 

Improved performance across four multimodal generation tasks

 

The researchers evaluated TWM using several vehicle–infrastructure datasets. The experiments assessed four tasks: roadside image generation, traffic-video generation, safety-critical event generation, and image-to-LiDAR-style point-cloud generation.

 

Compared with the strongest baseline for each task, TWM reduced the Fréchet Inception Distance by 39% for general traffic-image generation, the Fréchet Video Distance by 20% for traffic-video generation, the Fréchet Inception Distance by 73% for safety-critical event generation, and depth error by 11% for point-cloud generation.

 

The researchers also evaluated the model on a camera-disjoint test set containing approximately 2,000 traffic-scene images collected from 23 previously unseen roadside cameras. The test data covered highways and intersections under sunny, cloudy, and rainy conditions.

 

DOI Link:

https://doi.org/10.26599/COMMTR.2026.9640045

 

About Communications in Transportation Research

Communications in Transportation Research was launched in 2021, with academic support provided by Tsinghua University and China Intelligent Transportation Systems Association. The Editors-in-Chief are Professor Xiaobo Qu, a member of the Academia Europaea from Tsinghua University, and Professor Xiaopeng (Shaw) Li from University of Wisconsin–Madison. The journal mainly publishes high-quality, original research and review articles that are of significant importance to emerging transportation systems, aiming to serve as an international platform for showcasing and exchanging innovative achievements in transportation and related fields, fostering academic exchange and development between China and the global community.

 

It has been indexed in SCIE, SSCI, Ei Compendex, Scopus, CSTPCD, CSCD, OAJ, DOAJ, TRID and other databases. It was selected as Q1 Top Journal in the Engineering and Technology category of the Chinese Academy of Sciences (CAS) Journal Ranking List. In 2022, it was selected as a High-Starting-Point new journal project of the “China Science and Technology Journal Excellence Action Plan”. In 2024, it was selected as the Support the Development Project of “High-Level International Scientific and Technological Journals”. The same year, it was also chosen as an English Journal Tier Project of the “China Science and Technology Journal Excellence Action Plan PhaseⅡ”. In 2024, it received the first impact factor (2023 IF) of 12.5, ranking Top1 (1/58, Q1) among all journals in "TRANSPORTATION" category. In 2026, its 2025 IF was announced as 12.7, maintaining the Top1 position (1/66, Q1) in the same category.

 

From Volume 6 (2026), Communications in Transportation Research will be published by Tsinghua University Press on the SciOpen platform with the official journal website at https://www.sciopen.com/journal/2097-5023. We kindly request that all new manuscript submissions be made through the journal’s submission system at https://mc03.manuscriptcentral.com/commtr. For any submission-related inquiries, please contact the Editorial Office at commtr_e@mail.tsinghua.edu.cn.

---

---