Yes, fine-tune rather than train from scratch. For a new production line, the better strategy is to start from a model pretrained on a large public LiDAR corpus such as nuScenes, KITTI or Waymo, then fine-tune it on around 200 labelled scenes with augmentation suited to point clouds. Two LiDAR studies report that, with their pretraining methods, fine-tuning on 20 percent of KITTI or 5 percent of nuScenes labels beats training from scratch on the full dataset. The limit is the domain gap: a model pretrained on a different sensor, beam density or object-size distribution transfers less, and some popular augmentations make that gap worse.
TechnoLynx starts this work by comparing the 200 scenes with candidate source datasets to measure the domain gap. From there it picks transfer and augmentation methods that suit sparse point clouds, builds the fine-tuning pipeline on the target hardware, and tests fine-tuned against scratch models on held-out scenes before deploying with monitoring.
Can a model pretrained on nuScenes or KITTI be fine-tuned on 200 scenes?
Yes. Several LiDAR detection studies show it working. Wang et al. (2020) found that a detector adapted from Lyft or Waymo with only 20 labelled target scenes was on a par with one trained from scratch on 500 target scenes. In the Grounded Point Colorization study, PointRCNN fine-tuned on 5 percent of KITTI rose from 55.2 to 62.7 moderate AP over a scratch baseline. At 20 percent it reached 70.7, beating scratch training on the whole dataset.
KITTI’s training split has 3,712 annotated frames, so 5 percent is about 186 frames (3,712 × 0.05), roughly the size of a 200-scene pilot. When the same authors pretrained on Waymo instead, 20 percent of KITTI gave 70.1 moderate AP against 69.5 for scratch training on 100 percent.
A similar result holds when the sensor changes. In the LiDAR Distillation paper (Wei et al.), PointPillars models were pretrained with Waymo-to-nuScenes adaptation and then fine-tuned on 5 percent of nuScenes labels. They outperformed a model trained from scratch on 100 percent. The DADT study targets the small-sample case directly. It adapts a pretrained detector with about 100 LiDAR frames and reports gains of 4.44 to 20.98 AP over the baseline when fine-tuning on 32 to 128 frames.
Pretraining also reduced run-to-run variance. In the GPC study, the standard deviation of moderate AP across five rounds fell from 2.7 for scratch training to 0.6 with pretraining.
Some reported gains are small. In segmentation, generative pretraining on ShapeNet gave 71.3 percent mIoU on SemanticKITTI, 1.1 points better than scratch. In the Sense Less, Generate More study, masked-autoencoder pretraining with 20 percent of the fine-tuning data on Waymo gave mAP improvements of 0.52 to 4.11 percent. In that study, the gains grew as the target had fewer labels.
Sensor, beam density and object size limit how much transfers
Domain gaps limit how well a pretrained detector transfers. LiDAR detectors adapt poorly to targets with a different sensor setup (sensor type, spatial resolution, field of view) and to a change of location. Wang et al. measured a KITTI-trained detector performing 36 percent worse on Waymo than one trained on Waymo. They also found that Waymo has ten times as many points per car as nuScenes.
Point density is an important factor. Wei et al. found that matching the source data’s point density to the target’s during training is essential for crossing the gap caused by different beam counts. Downsampling high-beam source data loses information, but the paper argues that matching density matters more than that loss. The reverse does not work. Upsampling the sparse target with bilinear interpolation or a super-resolution network did not improve accuracy, because noisy points distorted object shapes. The direction of the gap matters too. Across nine detectors, Eskandar et al. (CVPR 2024) found the high-to-low resolution gap much harder to close than the low-to-high one.
On a production line, that means comparing your sensor with the source dataset’s sensor before you choose a checkpoint. Wei et al. note that industrial training data usually comes from environments similar to deployment, so the main gap is the beam difference between sensors rather than object sizes or weather. If your sensor is sparser than the source’s, plan for density alignment or sparsity augmentation.
Object sizes also cause a gap. Wang et al. traced most of the cross-dataset gap to car sizes, and correcting for them raised 3D easy AP by 41.4 points. Eskandar et al. found that adjusting anchor sizes at test time, with no retraining, improved performance across locations and mainly cut false positives. If objects on a new line differ in size from those in the source data, the result is systematically wrong boxes, not random errors.
Weather is less a transfer problem than a discriminability problem, and augmentation barely helps with it. Eskandar et al. found that the weather gap is not significantly affected by augmentations. Models trained on clean-weather data generalised to bad weather as well as, or better than, models trained on bad-weather data. Features from a distant source can still beat random ones. Yosinski et al. (2014) found that transferability falls as the base and target tasks diverge, but features from distant tasks can still beat random initialisation.
Point-cloud augmentations that stretch 200 scenes, and the ones that backfire
Augmentation can have a large effect. Hahner et al. measured a 17.72 point gap between no augmentation and the PointPillars augmentation policy, so about 23 percent of PointPillars’ performance came from augmentation. A 2023 survey of point-cloud augmentation notes that public point-cloud benchmarks are fewer and less varied than image benchmarks, which makes augmentation matter more for point clouds than for images.
The right combination has to be found by testing, so treat the table below as a starting set, not a recipe.
| Augmentation | Measured effect | Condition or risk |
|---|---|---|
| Global rotation | Most effective single technique, above 10% gain (Hahner et al.) | Unsuitable where absolute orientation carries meaning |
| Global translation, scaling, random flip | About 7% gain each (Hahner et al.) | Scaling changes density; keep ranges physically plausible |
| Sparsity augmentation, line downsampling | Largest robustness effect on sparser targets; line downsampling most consistent across models (Eskandar et al.) | Strongest case is a high-to-low resolution gap |
| Random point drop | Robustness to occlusion and missing data | Too much drop hurts small or density-critical objects |
| GT-sampling (object paste) | Weak alone, useful only combined with others | Can overfit to source shapes and hurt pedestrians on sparser sensors; mixing in samples from a dataset with a different vertical field of view reduces this |
| Pattern-aware GT-sampling | +0.7% for PV-RCNN cars beyond 25 m on KITTI, best at 40% probability | Degraded some mid-range bins through distribution shift |
| Occlusion-aware real-object placement (Real3D-Aug) | +6.65% AP on KITTI “Hard” pedestrians | Reuses real annotated objects with explicit occlusion handling |
| Scene swap and rotate-paste (PolarMix) | Reported gains across perception tasks | Abstract-level evidence only here |
| Local object translation | Can hurt when applied aggressively | Removing it improved the PointPillars policy |
| Ground removal | Harmful in Hahner et al.’s experiments | Deletes context around objects |
| Mixup | Improves corruption robustness in classification | Destroys semantic information; limited gains outside classification |
What decides it is whether the augmented scene is realistic for the sensor’s physics. Many GT-sampling methods do not reproduce the occlusion an inserted object would cause in a real LiDAR scan; the better variants remove the points the new object would hide. In the Point Cloud Recombination study, the simulations tested raised the intensity of reflective points but never reproduced the point cancellation a real Ouster sensor showed. If an augmented scene could not have come from your sensor, the model learns a distribution it will never see at inference.
Transfer learning vs fine-tuning vs feature extraction
Transfer learning is the general idea: reuse what a model learned on one task or dataset as the starting point for another. Fine-tuning is one way to do it. The pretrained weights initialise the new model, and some or all layers keep training on the new data. Feature extraction keeps the pretrained backbone frozen and trains only a new head on its outputs; the research literature calls it linear probing when that head is linear.
The three are not interchangeable. Kumar et al. (2022) found that across ten distribution-shift datasets, full fine-tuning averaged 2 percent higher in-distribution accuracy but 7 percent lower out-of-distribution accuracy than linear probing. Your 200 scenes are the in-distribution data. Next month’s lighting, wear or product mix may not be.
Should we freeze the pretrained layers or retrain them on a small set?
For 200 scenes we would choose neither extreme. Train the head first with the backbone frozen, then unfreeze and fine-tune the whole network at a low learning rate (LP-FT). Kumar et al. showed that fine-tuning with a randomly initialised head distorts the pretrained features, because the lower layers change while the head is still poorly aligned. In their image-classification experiments, LP-FT was 1 percent better in-distribution and 10 percent better out-of-distribution than full fine-tuning, and it changed the features about 20 times less.
In image classification, freezing can win on very small sets. Kumar et al. cite Kornblith et al. (2019), who found that linear probing can beat fine-tuning when there are very few examples, but fine-tuning won on all 11 of their datasets once there were more than about 30 examples per class. One pretraining study froze the encoder and tuned only the decoder and head in a ShapeNet-pretrained segmentation setup, but fine-tuned the entire network for SemanticKITTI.
Regularised fine-tuning is an alternative built for LiDAR. DADT keeps a frozen pretrained teacher that sees density-aligned inputs, and similarity losses keep the fine-tuned student close to it. The authors show that plain supervised fine-tuning on a small set shifts the feature distribution and overfits. Their regularised version produces representations similar to a model fine-tuned on all the data.
One caution comes from outside LiDAR. Image-classification work (ImageNet-RIB, 2024) found that models pretrained on the largest datasets, such as LAION-2B, lost more robustness after small-data fine-tuning than models pretrained on smaller ones. The loss appeared in CLIP models but not in the supervised models tested. Weight averaging combined with continual-learning methods preserved robustness best. We would take this as a reason to check robustness on out-of-distribution scenes, not as evidence about LiDAR detectors.
What about starting from a pretrained LiDAR diffusion model?
Fine-tuning a pretrained LiDAR diffusion model costs far less than training one from scratch, but that evidence covers generating point clouds, not detecting objects in them. LiDARDraft (2025) froze a pretrained unconditional LiDAR diffusion model and fine-tuned only a ControlNet. It reached good conditional generation in 5,000 training steps, 95,000 fewer than training from scratch. The base model was trained on 69,580 KITTI-360 samples.
Generated scenes still have to be checked against real sensor data. Simulation results can look accurate to a person yet deviate beyond acceptable thresholds.
How many labelled scenes does each new line need?
You have to measure the number for each line. Published figures give a range: 20 target scenes matching 500 from scratch, about 100 frames being enough for regularised fine-tuning, and 5 percent of KITTI (about 186 frames) giving large gains. In our view, 200 scenes is a reasonable pilot size when the sensor matches the source data and an optimistic one when it does not.
Data collected in the normal course of operation is biased towards frequent scenes, so rare but critical cases stay underrepresented. To find the number for a line, plot a learning curve: fine-tune on growing subsets (50, 100, 150, 200 scenes) and see where held-out performance levels off.
In a computer vision project, the first things to check are the quality and quantity of the data, whether more can be generated, and what accuracy is realistic, since close to 100 percent is almost never achievable. The workflow has to tolerate errors and leave room for human review. A line that can supply more labelled scenes cheaply needs a different plan from one where every label costs an engineer’s afternoon.
When to stop training a small fine-tuning set
The standard approach is early stopping on a held-out validation set. Keep some scenes the model never trains on, track their metric each epoch, and keep the checkpoint where it peaks. With 200 scenes, we would hold out a stratified group (by shift, product variant and object type) rather than a random slice, and repeat the split several times because single-split scores vary.
Pretraining sped up convergence in one study. In the GPC study, fine-tuning on 5 percent of KITTI passed the scratch baseline’s final score by epoch 55, more than 30 percent more efficient. If validation stalls while training loss keeps falling, the DADT findings point to overfitting through representation drift, which regularising toward the pretrained model prevented in that study. Image-domain evidence adds one qualification. In the ImageNet-RIB study, overfitting was not the main cause of robustness loss in models pretrained on LAION-2B and OpenAI data.
Checking each new line before the pilot model takes over
A good fine-tuned score on held-out scenes from the same line is necessary, but it is not enough to replace manual inspection. Stratified validation across conditions, sites and hardware is a separate step from fine-tuning. For each line, the comparison worth reporting is fine-tuned versus scratch, by stratum, on scenes from that line, and how much the margin shrinks as the sensor and object statistics drift away from the pretraining source.
FAQ
What is the difference between data augmentation and data synthesis?
Augmentation modifies or expands existing data, for example by rotating, dropping or pasting real points, to make it more varied. Synthesis generates new data with simulators or generative models, which gives controlled variation but simplifies how real sensors behave. Synthetic point clouds tend to be over-idealised compared with real scans.
What is the purpose of data augmentation?
Augmentation increases the quantity and variety of a limited training set, which improves robustness and generalisation and reduces overfitting. It is used most where annotated point clouds are scarce.
