Large language models expand capabilities for autonomous driving, survey finds
image: The modularized approach (top) separates the driving task into distinct stages—perception, prediction, planning (including route, behavior, and motion planning), and control—each handled by specialized components. In contrast, the end-to-end approach (bottom) uses a single large neural network to directly map raw sensor inputs (e.g., camera, light detection and ranging (LiDAR)) to control signals, bypassing explicit intermediate modules.
Credit: Yuxuan Zhu, Shiyi Wang, Wenqing Zhong, Nianchen Shen, Yunqi Li, Siqi Wang, Zhiheng Li, Cathy Wu, Zhengbing He, Li Li
A comprehensive review titled A Survey on Large Language Model-Powered Autonomous Driving has been published in Engineering, examining how large language models (LLMs) address persistent technical barriers across two mainstream autonomous driving (AD) development paradigms: modularized pipelines and end-to-end neural network frameworks. Authored by a cross-institutional team from Tsinghua University, the Massachusetts Institute of Technology and Technical University of Munich, the paper structures its analysis around ten specific unresolved challenges distributed across perception, prediction, planning, control and end-to-end system workflows, while evaluating the practical suitability of LLM integration for each technical bottleneck.
The survey first outlines inherent limitations of traditional AD architectures. Modularized systems split driving tasks into separate perception, prediction, multi-layer planning and control modules, yet face integration challenges, leading to gaps and conflicts between independent sub-models. End-to-end AD directly maps raw LiDAR and camera sensor data to vehicle control signals via unified neural networks, but suffers from heavy reliance on driving-specific training data, poor interpretability and inefficient training workflows. Both paradigms struggle with long-tail corner cases, unified data annotation and human–vehicle communication, pain points that researchers argue LLMs may mitigate through built-in commonsense knowledge, logical reasoning and multimodal comprehension capacities.
The paper systematically catalogs existing research integrating LLMs into modular AD components. In perception workflows, multimodal LLMs function as cross-modal bridges linking sensor point clouds, visual frames and textual descriptions to strengthen complex scene comprehension and identify semantic anomalies within rare traffic scenarios. For prediction tasks, LLMs deliver situation-aware trajectory forecasting by reasoning over social interactions between road agents. In planning, LLMs support hierarchical route, behavioral and motion planning via chain-of-thought reasoning and retrieval-augmented generation, while hybrid frameworks pair LLMs with model predictive control to adjust high-level decision logic for vehicle control modules, even if LLMs remain unsuitable for direct low-level real-time control signal generation due to latency constraints.
For end-to-end AD development, the survey details LLM contributions to dataset construction, synthetic traffic scene generation and model training optimization. Researchers reference language-annotated benchmarks including DriveLM-nuScenes and LMDrive, alongside LLM-powered simulation tools that generate diverse long-tail driving scenarios to ease data collection burdens. The work also frames LLMs as standalone driving agents that convert multimodal sensor inputs into interpretable driving decisions, offering a pathway to improve the interpretability of conventional end-to-end neural networks.
Beyond technical implementation, the survey weighs core constraints limiting broad LLM deployment in safety-critical AD systems. These include model hallucination that generates factually inconsistent scene judgments, substantial computational overhead introducing inference latency, limited innate physical world reasoning, privacy risks from sensitive driving data and inconsistent evaluation standards across simulation and real-road testing. The paper further explores a longstanding field debate on whether artificial general intelligence built upon LLMs represents a necessary foundation for high-level autonomous driving, or task-specific dedicated driving intelligence constitutes a more efficient development path. It notes that modularized and end-to-end architectures will likely coexist in near-term commercial AD systems, with LLM-driven hybrid frameworks emerging as a sustained research frontier for balancing stability and generalization.
The paper “A Survey on Large Language Model-Powered Autonomous Driving,” is authored by Yuxuan Zhu, Shiyi Wang, Wenqing Zhong, Nianchen Shen, Yunqi Li, Siqi Wang, Zhiheng Li, Cathy Wu, Zhengbing He, Li Li. Full text of the open access paper: https://doi.org/10.1016/j.eng.2025.07.038. For more information about Engineering, visit the website at https://www.sciencedirect.com/journal/engineering.