Autonomous vehicles: sensors, perception and levels of automation
Autonomous vehicles combine cameras, radar, lidar, AI software, and high-performance compute to navigate without human input. The SAE defines six levels of automation, from no automation to full automation. Perception systems can be trained end-to-end or with modular components, and safety data from deployments show reduced crash rates.
How an autonomous vehicle works
Autonomous vehicles span passenger cars, robotaxis, trucks, and commercial vehicles. They operate with little or no human direction, perceiving the environment with cameras, radar, and lidar, using artificial intelligence to decide how to drive, and relying on fast onboard computers to put those decisions into action, according to the NVIDIA glossary on autonomous vehicles.
Development involves three specialized computers: a training computer for AI models, a simulation platform for validation, and an in-vehicle computer that runs perception, planning, and control. The in-vehicle computer must process sensor data fast enough for real-time responses.
The training and simulation loop allows continuous refinement using real driving data collected from the fleet.
Levels of automation
The Society of Automotive Engineers (SAE) defines six levels of driving automation, from Level 0 (no automation) to Level 5 (full automation).
Level 0 systems provide momentary warnings but no sustained control. Level 1 controls one function, such as adaptive cruise control. Level 2 manages both speed and steering but requires the driver to remain attentive. Level 2+ and Level 2++ expand the capabilities, with Level 2++ allowing attention-off operation in defined conditions while the driver must still be ready to resume control.
Level 3 allows the vehicle to handle driving in defined conditions, but the driver must be ready to take over when requested. Level 4 operates without human intervention within a defined operational area, and most commercial robotaxi deployments operate at this level. Level 5 would operate in all conditions with no human involvement.
Perception and learning approaches
One approach to perception is end-to-end learning, where a convolutional neural network maps raw camera pixels directly to steering commands. This method was demonstrated in a paper on end-to-end learning for self-driving cars, which showed the system could learn to drive in various environments with minimal training data, as described in the arXiv paper by Bojarski et al..
End-to-end learning optimizes all processing steps simultaneously, unlike modular systems that separate lane detection, path planning, and control. The paper argues this can lead to better performance and smaller systems because internal components self-optimize to maximize overall performance.
To support perception research, large-scale datasets are needed. The Waymo Open Dataset provides 1,150 scenes, each spanning 20 seconds, with synchronized and calibrated LiDAR and camera data across urban and suburban areas, as described in the arXiv paper on scalability in perception.
Safety and real-world deployment
Crash avoidance technologies have demonstrated real-world benefits. Research from the Insurance Institute for Highway Safety shows that front crash prevention combining forward collision warning with automatic braking reduces rear-end crashes by 50%, while forward collision warning alone lowers them by 27%. The institute also notes that fully driverless cars are not yet commercially available.
Waymo publishes safety data from its driverless ride-hailing service. The Waymo safety page reports that compared to an average human driver in its operating cities, the Waymo Driver had 95% fewer serious injury or worse crashes, 82% fewer airbag deployment crashes, and 82% fewer injury-causing crashes. It also shows large reductions in pedestrian, cyclist, and motorcycle crashes with injuries.
The safety page also indicates that Waymo supports Vision Zero goals and uses a safety framework that is documented and pressure-tested.
| Level | Name | Driver role | Vehicle control |
|---|---|---|---|
| 0 | No automation | Driver handles all tasks | Provides warnings only |
| 1 | Driver assistance | Driver manages most functions | System controls one function, e.g., speed |
| 2 | Partial automation | Driver must remain attentive and supervise | System manages speed and steering in defined conditions |
| 2+ | Expanded automation | Driver remains responsible and supervises | Integrates multiple driver assistance functions |
| 2++ | Expanded automation | Driver does not monitor road but must be ready to resume control | Allows attention-off operation in defined conditions |
| 3 | Conditional automation | Driver must be ready to take control when requested | Vehicle handles driving within defined conditions |
| 4 | High automation | No human intervention within defined area | Vehicle operates within operational design domain |
| 5 | Full automation | No human involvement | Vehicle operates in all conditions |
Frequently asked questions
What sensors do autonomous vehicles use?
Autonomous vehicles use cameras, radar, and lidar, along with AI software and high-performance compute, to perceive and respond to their environment.
How do SAE levels classify automation?
The SAE defines six levels, from Level 0 (no automation) to Level 5 (full automation), based on how much of the driving task the system handles and the driver's responsibility.
What is end-to-end learning for self-driving cars?
End-to-end learning uses a convolutional neural network to map raw camera pixels directly to steering commands, optimizing all processing steps simultaneously.
Why is the Waymo Open Dataset important?
The dataset provides 1,150 scenes of 20 seconds each with synchronized LiDAR and camera data, supporting research in perception for autonomous driving.
Do crash avoidance systems reduce crashes?
Yes, front crash prevention with automatic braking cuts rear-end crashes in half, and forward collision warning alone reduces them by 27%.