Eight of eleven runs ended in a collision before the vehicle had cleared its opening corner. That is the headline result from DrivingBench, an experiment built by researchers Aditya Ramabadran, Simon Mahns and Tobias Gessler to establish whether today’s leading general-purpose AI agents are ready for tasks in the physical world. The testbed was a real Toyota Corolla fitted with Comma/OpenPilot vision hardware, and the course was a short point-to-point route marked out with cones in a parking lot. No humans were injured.
The striking part is not simply that the agents failed. It is that most of them never genuinely engaged with the problem at all.
- Eight crashes across eleven runs: every failure occurred inside the first 11% of the route, at that initial curve.
-
A single completion:
GPT-6 Astra was the only agent to finish, and only on its second attempt after the first stalled at the halfway point. - $7.74 per successful lap: the completed run cost roughly €7.28, almost four times the cost of the preceding attempt.
- Failure modes carry the real signal: Grok charged a cone after barely two commands, while another agent invented a rule about cone colours.
How DrivingBench actually works
The architecture is deliberately minimalist. The researchers installed Comma/OpenPilot vision hardware on the Corolla and connected the agents directly to that visual stream. Prompts were transmitted over a wireless hotspot to remote data centres, and the responses arrived back as commands to the vehicle. The latency inherent in that round trip forced frequent stops.
Eleven runs were carried out using four different agents. The route opened with a curve and then continued along a lane defined by cones. All eight of the failed attempts ended in a collision within that very first turn.
What each agent got wrong
The failure modes are where the genuinely useful information sits. Grok interpreted the gap between two cones as a doorway — a legitimate node of passage — and drove straight into one of them after no more than two commands. It is a spatial representation error: the model reads “space between objects” as “opening” without modelling the context in which the cones were defining the correct path rather than blocking it.
One of the GPT agents fabricated a non-existent rule about what the colour of the cones signified and then bent its behaviour to that invented rule. In a text interface, a hallucinated fact produces a faulty article; behind the wheel, it produces a crash.
The most frequent mistake across the field was underestimating the steering angle required at the first turn. Language models derive their understanding of physical space from written descriptions and training data rather than from motor experience. Translating “steer harder” into degrees of wheel rotation from a camera with an unusual perspective is simply not a task they were trained for.
The distance between demos and the road
The dominant industry narrative frames AI agents as the natural next step for language models: hand them tools and access to external systems and they become capable of acting in the world. DrivingBench suggests that transition is far messier than laboratory demonstrations imply. That first curve was not difficult — it was about as basic a physical task as a test bench can offer.
Purpose-built driving stacks show what specialisation buys. Stellantis and Wayve integrated the Wayve AI Driver with the STLA AutoDrive platform across a Fiat 500e and a Maserati Grecale in under four weeks per vehicle, aiming for Level 2++ commercial availability in North America in 2028. XPENG’s robotaxi operation runs a pure-vision stack governed by its VLA 2.0 foundation model on four in-house Turing AI chips delivering 3,000 TOPS. Those systems were trained for the road. General-purpose agents were trained for text, and the difference shows the moment the asphalt begins.
GPT-6 Astra’s completed run is, in a sense, more troubling than the cars that never moved. Finishing a parking-lot task at that price and with that level of uncertainty does not scale to real driving — the cost alone would be absurd at fleet level, before anyone considers the reliability gap. Yet it is the benchmark’s first completion, which is not nothing. The honest reading of DrivingBench is that general agents currently lack the physical grounding to be trusted in open environments, and that the handful of purpose-built systems clearing that bar are doing so precisely because they were engineered for it rather than repurposed from text.