For years, self-driving software was built as a modular stack: perception, prediction and planning developed separately and then connected. More recently, end-to-end designs that go from sensor input to driving behavior in one neural network have spread quickly. This article compares the two using landmark research and company disclosures, with particular attention to the question that matters most for deployment: how do you validate it?
The modular stack: build separately, connect by hand
A modular driving system usually works in this order:
- Perception — find vehicles, pedestrians, lanes and signals with cameras, LiDAR and radar.
- Prediction — forecast how surrounding objects will move.
- Planning — choose the ego vehicle’s trajectory based on those forecasts.
- Control — turn the trajectory into steering, throttle and braking commands.
Each stage can be developed and tested on its own, which makes problems easier to find and fix. The UniAD paper, however, argues that errors in early stages can accumulate downstream and that coordination between tasks can be deficient. Information that does not fit the human-defined interfaces between stages — a list of detected objects, for example — never reaches the next stage.
Planning-oriented integration: UniAD
UniAD puts planning, the ultimate goal of driving, at the center and integrates all the driving tasks into one network. Perception, prediction and planning exchange information through unified query interfaces, and the authors report substantially outperforming prior state-of-the-art methods on nuScenes. The key point is not that modules disappear, but that they are connected by learned representations and optimized together for planning performance. In practice this means a perception error that does not affect the planned trajectory costs little during training, while one that leads to a bad plan is penalized directly, which focuses the network on what matters for driving rather than on detection scores alone.
Product case 1: Tesla FSD v12
According to the FSD Beta v12.1 release notes reported by Not a Tesla App (December 2023), Tesla upgraded its city-streets driving stack to a single end-to-end neural network, replacing over 300,000 lines of explicit C++ code. Before that, neural networks handled perception while hand-written code decided how to act in each situation; v12 moved those decisions into the network as well. The scope stated in the notes is the city-streets stack.
Product case 2: Waymo’s EMMA
EMMA, from Waymo researchers, builds on a multimodal large language model such as Gemini and maps raw camera input directly to driving outputs. Non-sensor inputs such as navigation instructions and ego-vehicle status are represented as text, and the model generates planner trajectories, perceived objects and road-graph elements. The paper reports state-of-the-art motion planning on nuScenes and finds that co-training on planning, detection and road-graph tasks improves all three — an attempt to bring a language model’s general knowledge into driving.
Product case 3: NVIDIA’s Alpamayo
At CES in January 2026, NVIDIA announced the Alpamayo family of models and tools. Its core, Alpamayo 1, is a 10-billion-parameter reasoning VLA that takes video input and outputs trajectories together with reasoning traces explaining each decision. NVIDIA positions it not to run in the vehicle but as a teacher model that developers fine-tune and distill into their own stacks. The release also included AlpaSim, an open-source simulation framework with sensor modeling and closed-loop testing, and more than 1,700 hours of driving data.
The approaches compared
| Item | Modular | Tesla FSD v12 | Waymo EMMA | NVIDIA Alpamayo 1 |
|---|---|---|---|---|
| Structure | Separate perception, prediction, planning | Single end-to-end network (city streets) | End-to-end on a multimodal LLM | Reasoning VLA (teacher model) |
| Main input | Multiple sensors | Cameras | Cameras + text | Video |
| Output | Per-module results → trajectory | Driving behavior | Trajectories, objects, road graph | Trajectories + reasoning traces |
| Validation view | Easy per-module testing | Whole-behavior validation needed | Whole behavior + auxiliary outputs | Reasoning traces aid explainability |
Validation is the central issue
The biggest challenge for end-to-end driving is not raw performance but validation. In a modular stack you can point to where a failure happened — “perception missed the pedestrian” — while in a single network it is hard to say why a decision was made. EMMA’s auxiliary outputs of objects and road graph, and Alpamayo’s reasoning traces, can be read as attempts to reduce that explainability problem.
In real deployments, end-to-end models are commonly wrapped with safety constraints and monitoring layers. And whether gains on open-loop benchmarks carry over to closed-loop safety on real roads has to be established separately through simulation and on-road testing. The compute setup for training and testing such models is covered in NVIDIA’s three computers for robotics.
Data becomes the main engineering asset
Moving from hand-written rules to learned behavior also changes what engineering teams spend their time on. In a modular stack, much of the effort goes into writing and tuning code for specific situations. In an end-to-end system, the equivalent effort goes into data: finding examples of rare and difficult situations, deciding which recorded drives represent good behavior, and building simulation scenarios that exercise the cases real driving rarely produces. That is why open datasets and simulators, such as those NVIDIA released alongside Alpamayo, matter as much as the models themselves, and why teams with large fleets collecting driving data see an advantage in this approach.
What it means for robotics
The end-to-end shift in driving mirrors what is happening in robotics. In humanoids and robot arms, too, stacks that separated perception, planning and control are giving way to VLA models that take camera images and instructions and output joint commands directly; their architectures and action-generation methods are covered in VLA architectures compared.
The validation problem carries over as well. Robots also need ways to explain decisions made by a single network, separate validation in simulation and in the real environment, and safety layers outside the learned model. And the criteria for choosing input sensors — starting from the failures you cannot accept — apply just as they do on the road, as discussed in LiDAR fusion vs vision-only perception. Teams that have shipped driving systems are therefore a useful reference for robotics teams planning their own move to end-to-end control: the lessons about data, simulation and safety wrappers transfer more directly than the models themselves.