The BEVDet paper (Huang, Huang, Zhu, Yun and Du, arXiv 2112.11790, 2021) describes a multi-camera 3D detector. It lifts features from a vehicle’s surround-view cameras into a single bird’s-eye-view (BEV) grid and predicts 3D boxes in that grid. On the nuScenes validation set the paper reports 31.2% mAP and 39.2% NDS for BEVDet-Tiny at 15.6 FPS, and 39.3% mAP and 47.2% NDS for BEVDet-Base. Those scores come from nuScenes, recorded on two cars with an identical six-camera layout whose calibration the dataset authors checked on every log. We would not read them as a guide to a fleet where cameras get knocked out of alignment, rigs differ between vehicle types and the weather looks nothing like the training set.
BEVDet uses each camera’s calibration and its own depth estimates to decide where every image feature lands in the grid. Because both inputs decide where features land, corrupting either one moves the boxes without any change to the code.
How the BEVDet paper builds a bird’s-eye view from six cameras
The authors say they built BEVDet “merely” by reusing existing modules. They credit its performance to an exclusive data augmentation strategy and an upgraded non-maximum suppression step. We split the pipeline into four stages.
| Stage | What it does | What it relies on | Data-quality failure that hits it |
|---|---|---|---|
| Image-view encoder | A 2D backbone extracts features from each camera image separately | Image appearance similar to the training data | Lighting, weather and sensor effects such as motion blur |
| View transformer (LSS-style forward projection) | Predicts per-pixel depth and pushes image features to their 3D positions, then pools them into the BEV grid | The depth estimate plus each camera’s intrinsics and extrinsics | Focal-length mismatch, camera pose differences, rig change |
| BEV encoder | Processes the unified grid, reasoning across all cameras at once | Features having landed in the right cells | Inherits and compounds errors from the lifting step |
| Detection head | Regresses box centre, size, orientation and velocity from BEV features | Accurate BEV features from the earlier stages | Localisation error, which shows up as a rising mATE |
The detection head is where box centre, size and orientation are regressed. In BEVDet, though, depth and camera geometry enter one stage earlier, in the view transformer. RoboBEV (2024) describes BEVDet as one of the methods, following LSS, that use an auxiliary depth branch to get from perspective view to BEV. The BEVDepth authors observed that depth estimation in recent approaches was “surprisingly inadequate”, even though depth is essential to camera 3D detection. They fixed it with explicit depth supervision. The current BEVDet repository offers a variant with the same fix, depth supervised from lidar as in BEVDepth.
When a BEVDet box drifts, we would therefore check the depth estimate and the camera transform before the classifier. Those two causes call for different fixes, so a team should find out which one it has before retraining anything.
How does BEVDet compare with BEVFormer and DETR3D for camera-only perception?
The three methods differ in which direction they connect image and 3D space. BEVDet lifts 2D features forward into the BEV grid. BEVFormer goes the other way: it projects 3D reference points from grid-shaped BEV queries back into the camera images and samples features around them. DETR3D takes a third route: a sparse set of 3D object queries indexes into multi-view 2D features through the camera transformation matrices, with no depth network and no NMS step.
| Method | Image-to-BEV mechanism | Explicit depth? | Reported nuScenes figure |
|---|---|---|---|
| BEVDet-Base | Forward projection (LSS-style lifting) | Yes, auxiliary depth branch | 47.2% NDS, val set |
| BEVDet4D-Base | BEVDet plus fusion of the previous frame’s features | Yes | 54.5% NDS, +7.3 over BEVDet-Base, as its authors report |
| BEVFormer | Backward projection with spatial cross-attention and temporal self-attention | No | 56.9% NDS, test set |
| DETR3D | Sparse 3D queries projected to images | No | 47.9% NDS, test set |
| BEVDepth | Forward projection with lidar-supervised depth | Yes, supervised | 60.9% NDS, test set |
Be careful with that last column. BEVDet-Base’s figure is a validation-set number, while the BEVFormer, DETR3D and BEVDepth figures above are test-set numbers. Ranking methods by subtracting one column from the other mixes splits, so we would not read it as a clean comparison.
For a deployment decision, we would weigh robustness more heavily. A cross-sensor study found that dense BEV detectors with backward projection, BEVFormer among them, held up best when the camera setup changed. Sparse, object-centric methods tended to overfit to the training rig. RoboBEV found that even after pre-training improved its robustness, BEVDet still lagged behind depth-free models. Backward projection has its own weak point, however. BEVFormer’s sampling locations are computed from camera extrinsics, so perturbed extrinsics make queries pull features from the wrong image regions. Choosing an architecture changes how calibration error enters the model. It does not remove the exposure.
What BEVDet’s nuScenes scores (mAP, NDS, mATE, mAOE) tell you and what they miss
The scores measure specific things. nuScenes mAP matches each prediction to ground truth by 2D centre distance on the ground plane, not by box overlap, and averages over match thresholds of 0.5, 1, 2 and 4 metres. The true-positive metrics are measured on matches within 2 metres. They include translation error (mATE, in metres), scale, orientation (mAOE), velocity and attribute errors, and if a class never reaches 10% recall, its true-positive errors are set to 1. NDS is a weighted combination of mAP and those five true-positive metrics, so it can be broken back down into its parts.
That decomposition is the useful part. For a BEV deployment we would track mATE and mAOE on their own rather than only the headline NDS. In the cross-sensor study, falling mAP went together with a 57.5% average rise in mATE (+0.34 m) once the camera setup changed.
The scores leave out four things that matter on a road.
- Which misses are dangerous. The 2020 planner-centric metric paper showed NDS staying roughly flat while missed detections moved closer to the ego vehicle or sped up. Its planner-based PKL metric penalised those misses, and human raters sided with PKL over NDS 79% of the time.
- Conditions outside the dataset. nuScenes was not collected during or after rain, and the vehicles drove at no more than 40 km/h.
- Regions outside the dataset. Its train, val and test splits come from the same regions, a deliberate choice for a geofenced robotaxi use case.
- Camera-visible ground truth. Because annotation was lidar- and radar-centric, camera-only methods can be penalised for objects the cameras could not see.
To be fair to NDS, the nuScenes authors note that it correlates better with downstream planning than mAP does, and the detection task has reported PKL since a 2020 update. A good BEVDet number is a real result on a carefully calibrated, quality-checked benchmark. It just was not measured under the conditions where a fleet fails.
Why does camera calibration drift cost BEVDet its 3D localisation accuracy?
Forward projection places each lifted feature using the camera’s intrinsics and extrinsics. If those parameters no longer match the physical camera, features land in the wrong BEV cells and the boxes move with them. The BEVFormer authors put the deployment risk plainly: extrinsics “may be biased due to various reasons such as calibration errors, camera offsets”.
The BEVFormer paper (2022) added rotation noise with variance of i degrees and translation noise with variance of 5i centimetres at level i. Trained on noisy extrinsics, BEVFormer’s NDS dropped by only 8.9%. In a harsher five-camera dynamic stress test, the NCGR authors report BEVFormer at 28.00% NDS.
No source here measures BEVDet’s own sensitivity to extrinsic drift, so the BEVFormer figures above stand in. For BEVDet, the measured evidence concerns a related problem: a change of rig. In the cross-sensor study’s simulated platforms, BEVDet’s forward-projection mAP dropped 12.5% and 18.6% when tested on a camera setup other than the one it was trained on. The authors suggest forward projection is the more sensitive design here. Fleet-wide rollout runs into this directly, because mounting positions differ between vehicle types even when the hardware and camera count are the same.
Catching drift before it reaches the boxes
Calibration has to be repeated whenever camera parameters change, and on a vehicle that can happen often. Static garage calibration with a target does not scale to a fleet. Targetless methods such as SceneCalib (2023) measure a miscalibration rate: they project a lidar map into pairs of camera frames and compare photometric error against perturbed poses. SceneCalib’s authors also say night drives, heavy rain and snow are poor material for automatic calibration. In those conditions they fall back on a result from a better-conditioned recording of the same vehicle. Self-supervised online methods learn intrinsics and extrinsics in real time without targets.
Our position is that data quality and fault tolerance come before model selection. For a BEVDet fleet, that puts calibration and data checks before any argument about switching architectures. One signal we would watch closely is mATE rising on a production slice while mAP holds steady.
How weather, lighting and new driving regions shift BEVDet’s inputs
Weather and lighting corrupt the images and, under Snow and Dark, the depth estimate too. In RoboBEV’s nuScenes-C benchmark (eight corruptions at three severity levels), every model lost performance. Dark and Snow stood out as shared weak spots of BEV models. RoboBEV traced the largest drops in depth-based methods to depth errors under Snow and Dark. A separate CVPR 2023 benchmark found camera-only 3D detectors “extremely vulnerable” to image corruptions, with motion-level corruptions the most damaging for all models.
Pixel statistics alone are not an adequate early warning. RoboBEV found that Motion Blur barely shifted the pixel distribution yet caused a large drop, while Bright and Fog shifted pixels more and did the least damage. RoboBEV also says its synthetic corruptions cannot cover every real out-of-distribution case. Treat a corruption benchmark as a stress test, not a guarantee.
Moving to a new rig adds a geometric shift on top of appearance changes. The domain-generalisation study found that detectors trained on nuScenes showed location bias on Waymo, where intrinsics and extrinsics differ, and a source-only model’s mAP fell almost to zero. The mechanism is easy to follow: a different focal length changes the object’s size in the image at the same distance, so a depth predictor trained on one camera misjudges depth on another. Between nuScenes and Lyft, whose camera intrinsics and poses are close, the source-only model kept a fair amount of capability.
Our judgement is to slice validation metrics by condition (night, rain, region, vehicle type) and to monitor the true-positive errors per slice. Fleet-wide averages hide exactly the cells that degrade.
Where are the BEVDet paper and code, and what does retraining on your own rig take?
The paper is arXiv 2112.11790, with a public reference repository. The repository lists variants including lidar-supervised depth and a long-term temporal mode that concatenates eight history frames. The temporal successor, BEVDet4D, fuses the previous frame’s BEV features at negligible extra compute and reports cutting velocity error by up to 62.9%.
For retraining, nuScenes-style info files hold calibration for each of the six cameras, CAM_FRONT through CAM_BACK_RIGHT, with a cam_intrinsic key per camera for projecting 3D points. Your rig needs the same thing: accurate per-sample intrinsics and extrinsics, and annotations in a vehicle frame consistent with them. Check how far your optics are from the training rig. nuScenes front and side cameras have a 70° field of view and 5.5 mm focal length, and the rear camera has 110° and 4 mm.
When the rig differs, three routes have measured support:
- Augmenting camera pose during training. Yaw and pitch perturbation helped in transfers between datasets with different camera yaw and mounting height.
- Decoupling depth from intrinsics, which in the domain-generalisation ablation raised cross-domain NDS* from 0.178 to 0.393.
- Neural-rendering an existing dataset into the new camera viewpoints, which cut average localisation degradation from 0.34 m to 0.09 m mATE.
Before committing to any of them, rank how hard the transfer will be from rig metadata. The multi-rig benchmark found its Rig Contrastive Distance descriptor to be a reliable proxy for cross-rig difficulty.
Checks a BEVDet stack should pass before its detections reach the planner
This checklist follows from the failure modes above. TechnoLynx’s Production AI Monitoring Harness builds the eval, regression, drift and slice-metric checks this list calls for.
- Calibration validity for each log. Each drive has a calibration result and a miscalibration measure, and conditions that defeat automatic calibration (night, heavy rain, snow) are flagged.
- Rig parity. Per-camera intrinsics and mounting poses are compared with the training rig, and rig distance is recorded.
- Condition-sliced evaluation. Metrics are reported separately for darkness, snow, fog and motion blur, not only on clean validation data.
- Decomposed errors. mATE and mAOE are tracked separately from NDS, per slice.
- Planner-relevant scoring. A planner-centric metric such as PKL is reported next to NDS, so near and fast misses are not averaged away.
- Runtime consistency. Where a second sensor exists, a runtime monitor checks that both perception outputs agree inside a region of interest around the vehicle.
The Dependability Cage work compares camera and lidar object lists inside a dynamically computed region of interest. Its consistency thresholds come from hazard analysis and risk assessment, and a timeout handles the sensors’ different rates. It was demonstrated with YOLO Nano 2D detectors on a model vehicle in static scenarios, and the authors list dynamic scenes as future work. We would use the design as a pattern, not as validation evidence for a road-going BEVDet stack. Teams building a safety case can also read our page on BEVDet’s ASIL evidence implications.
For failures beyond BEV geometry, see our page on data-quality problems that degrade CV systems after deployment. The open question is not whether BEVDet scores well. It is whether they can tell, for each vehicle and each drive, that the geometry the model assumes is still the geometry on the road.
FAQ
Why is LiDAR better than camera for 3D detection?
Inferring 3D position from 2D images is still a hard problem, and the BEVFormer authors note that camera-based methods trail lidar-based ones in both accuracy and efficiency. A CVPR 2023 robustness benchmark also found lidar-camera fusion models more robust to corruptions than camera-only models.
How do you calibrate a car camera?
The traditional method parks the vehicle in a garage and captures views of a purpose-built calibration target. Targetless methods calibrate automatically from ordinary driving data, and self-supervised online methods estimate intrinsics and extrinsics in real time without physical targets.
