SMc2f decouples scenario mining for autonomous driving into a coarse-to-fine pipeline. In such a funnel, each stage sees fewer clips than the one before it. Vehicles upload event clips instead of raw streams. CLIP embedding similarity cuts the log down to candidate time windows. Novelty and out-of-distribution (OoD) scores pull out the unusual cases, and a ranking that weighs exposure, severity, diversity and labelling cost decides what human labellers see first. The reason is the well-documented long-tail problem. Most fleet data is routine highway cruising or waiting at lights, while aggressive cut-ins and pedestrians crossing against signals are exceedingly rare, and sifting petabytes for them by hand is too slow and too costly to be practical. Every design choice below comes down to one question: what does this stage cost per unit of work, and what recall on rare events does it give up?
Finding rare events, not more of the same cut-ins
Retrieval of rare events splits into two steps: cross-modal similarity prunes the search space, and detailed analysis runs only on what survives. Raw “weirdness” is not the same thing as safety relevance, so a pipeline that treats it as a proxy can surface the wrong clips. Through its automotive AI engineering, TechnoLynx builds this kind of scenario-mining pipeline. The work starts by defining rare and safety-relevant event criteria jointly with your scenario taxonomy. It then covers the choice of embedding model, OoD scoring method and incremental or batch architecture, and the scheduling of large-batch offline inference across your GPU fleet.
Hierarchical pruning that keeps recall: a worked example
The SMc2f pipeline (Chen, Greer et al.) reports coarse-to-fine mining gaining both efficiency and accuracy. It splits each 15-second log into 3-second windows with a 1-second stride, samples five frames per camera view and scores the windows against the text query by CLIP cosine similarity. It keeps the top five windows by similarity summed across all seven views, merges adjacent ones and forwards only overlapping trajectories to the LLM-based mining stage. The authors report that this prunes irrelevant frames early and lowers computation while preserving recall. In their ablation, the CLIP filter gave the largest single gain: 8.13 points of HOTA-T and 10.7 points of Log-F1. A contrastive text-trajectory matcher then re-ranks the candidates and removes subtle false positives.
The lesson we draw is that the cheap stage does not have to be accurate. It only has to be accurate enough not to throw away positives, because the precise stage only ever sees a fraction of the data. Faiss handles nearest-neighbour search in the same two-stage way, querying a fast, inaccurate index for a shortlist and computing exact results only on that shortlist.
Statistically weird is not safety-relevant
Prediction-error novelty is one of the most natural signals to build triage on. A model that has not internalised some content should, in principle, find it harder to predict. A 2026 study of V-JEPA 2 prediction-error scoring tested exactly this idea and found that a single checkpoint scored AP 0.89 on a cross-dataset triage benchmark, yet AP 0.29 against a chance floor of 0.24 when positives and negatives came from the same dataset. The novelty score had mostly learned to detect which dataset a clip came from. A supervised probe on the same frozen embeddings reached AP 0.50, so the signal was present in the representation but the unsupervised objective did not surface it. Sampling positives and negatives from every source is a cheap control for this confound.
Human review stays in the loop for a reason documented in naturalistic driving research. The NHTSA/VTTI 100-Car Naturalistic Driving Study concluded that purely quantitative near-crash criteria were not feasible, because the vehicle kinematics of near-crashes were virtually identical to ordinary driving. An edge-case detection survey in IEEE Transactions on Intelligent Transportation Systems draws a similar line. It separates safety-critical cases from safety-relevant near-miss precursors and from non-safety degradations, and it calls for expert review before a candidate is accepted as a true edge case. OoD scores are a triage signal, and a person still confirms what is safety-relevant.
Other methods anchor novelty to a fleet baseline or a proxy reference set. AWS compares embeddings against fleet-wide baselines and flags scenes in the 95th percentile of rarity or risk. It then uses HDBSCAN to isolate “noise” points, which the authors describe as the outliers most likely to hold previously undefined edge cases. An unsupervised Deep Isolation Forest on 100 hours of naturalistic driving data placed 84% of its top-100 events in a rule-based proxy anomaly set, compared with 65% for a plain Isolation Forest. That proxy set came from heuristics and is not verified ground truth.
Benchmarking embedding models against known rare events
The OoDIS benchmark reports AR at 1, 10 and 100 predictions, which captures how many anomalous objects are found when only the top-k are kept. It also shows that objects under 1,000 pixels are the hardest case for every method it tested. Faiss evaluates nearest-neighbour search accuracy with n-recall@k, the fraction of the n ground-truth neighbours found in the first k results. The held-out set itself needs an unbiased rare-class distribution. SearchAD’s authors manually inspected every image, noting that pre-selection by predefined objects risks excluding challenging instances.
Can scenario mining run incrementally over the data lake, or does it need a full re-scan?
Scenario mining can run incrementally over newly appended data. On an Apache Iceberg table, Spark can read only the data appended between a start and end snapshot ID, so each run embeds and indexes just the new fleet logs. The same documentation states that incremental reads currently capture append operations only, not replace, overwrite or delete. Backfills that replace or overwrite old logs therefore fall outside incremental reads. Faiss indexes accept vectors added progressively. Embeddings computed once can also serve many tags: the universal embedding approach for self-driving logs precomputes and stores log embeddings for fast on-demand tagging, and the authors test whether it extends to new attributes.
Drift is the catch. When an embedding model is upgraded, Drift-Adapter maps new queries into the legacy space and recovered 95–99% of full re-embedding retrieval recall on MTEB text and a 1M-item CLIP upgrade, adding under 10 µs latency. That defers full re-computation rather than removing it. Baselines also age as road environments and road users evolve. TechnoLynx runs such pipelines incrementally against the live data lake while monitoring for drift.
Which GPU gives the best throughput per dollar for embedding fleet logs, and on-prem or cloud?
Throughput per dollar for embedding depends on more than the accelerator. LynxBenchAI publishes results for how each device runs the public PyTorch stack your own models run on, which makes them broadly indicative for a custom embedding model when you shortlist hardware. LynxBenchAI does not run or measure your workload. Its published benchmark results are a starting point for the hardware list, not the final answer. Once the hardware is chosen, the Inference Cost-Cut Pack is where TechnoLynx cuts the embedding job’s inference cost.
In Runpod’s embedding benchmark, the serving engine moved the number more than the card did. Runpod’s benchmark, priced at its 10 September 2026 list rates, found that switching serving engines changed embedding throughput by up to 11x on the same GPU, model and workload. Snowflake Engineering found that once its embedding throughput was optimised, the more expensive H200 cost less per token than the cheaper A10g, which challenges the assumption that a lower hourly price means better price-performance. Both benchmarks measured text embedding models, not video or LiDAR encoders. Treat them as evidence that the method matters, not as figures for your pipeline.
Video decoding is itself a bottleneck to remove. AWS lists four video decoders on each L4 GPU in its G6 instances, and fractional sizes down to 1/8 of a GPU with 3 GB of memory. NVIDIA reports that NeMo Curator, using NVDEC/NVENC and Ray, processed 20M hours of video with an 89x speedup on 1K GPUs compared with unoptimised CPU pipelines, which is a vendor benchmark.
On-prem versus cloud breakeven depends on utilisation. Lenovo’s 2025 TCO analysis assumes 100% GPU utilisation over five years (43,800 hours) to define a breakeven point, and says lower utilisation scenarios scale from that reference. A mining job that saturates hardware continuously points toward owned capacity. Either comparison has to include egress, storage and index I/O, not just the GPU hourly rate.
What does it cost to score every clip versus a sample, and how should storage be tiered?
Scoring cost scales with how many clips you score times the cost of each inference, and storage cost scales with volume times retention time times tier price. The levers are therefore to shrink what is uploaded, shrink what is fully scored and shrink how long unflagged data lives. TechnoLynx benchmarks and validates these costs on your fleet data. The first step is to benchmark GPU-hours, latency and storage per clip for full-scan, sampling and cascade scenarios on a representative fleet workload, and to check that rare-event recall holds. TechnoLynx then implements the edge pre-filter, the cloud detector and tag-driven storage tiers, with optimisation run through the Inference Cost-Cut Pack. For how per-clip costs compound as the fleet grows, see our piece on modelling per-unit inference costs from pilot to fleet.
Worked example: the cost model in symbols
Assumptions, stated as assumptions: N clips arrive per period, scoring one clip takes t GPU-seconds, and per-clip cost is constant.
- Full scoring: GPU-hours = N × t / 3600.
- Uniform sample at rate r: GPU-hours = r × N × t / 3600, so a 10% sample cuts scoring GPU-hours by 10x.
- The same sample keeps, in expectation, only a fraction r of the rare events, so a 10% sample discards about nine in ten of the clips you most wanted.
- Storage = Σ over tiers of (volume in tier × retention time × tier price).
The third line is why uniform sampling is the wrong default when rare-event recall is the goal. A cascade gets the compute saving without that loss, because every clip passes through the cheap stage and only the survivors pay for the expensive one. Multi-level OoD detection follows the same logic, handling easy cases with shallow detectors and spending deeper processing only on complex ones.
| Approach | Scoring GPU-hours | Storage | Rare-event recall risk | Fits when |
|---|---|---|---|---|
| Score 100% in the cloud, full retention | Highest: N × t | Highest | Lowest | Volume is small or the budget is not binding |
| Uniform sample (e.g. 10%) | r × N × t | Lower | High: loses about 1 − r of rare events | Rare events are not the priority |
| Edge pre-filter + cloud cascade + tiered storage | Cheap stage on all, expensive stage on few | Lowest | Moderate: the filters need their own validation | Vehicle compute and bandwidth allow a light filter and recall must be kept |
Can a cheap on-vehicle pre-filter cut what reaches the data lake?
An on-vehicle pre-filter can cut upload volume substantially, within the limits of the in-vehicle box’s compute and bandwidth. Resolute Dynamics, an edge-telematics vendor, estimates raw volume at 25+ GB per vehicle per day and post-preprocessing volume at 200–800 MB, with 60–90% transmission savings that skew higher when uploads move from raw video to event clips. Those are vendor estimates for telematics fleets, not measurements on an AV data programme. An edge filter that is too aggressive creates false negatives that no later stage can recover, so validate it against held-out rare events before relying on it.
Tiering storage so flagged events stay and the rest ages out
Object tags set at mining time can drive storage tiers directly. Amazon S3 Lifecycle rules can filter on object tags to transition objects to cheaper classes or expire them, and AWS recommends tagging objects in the application by their retention needs. Tag flagged rare events “retain” at full fidelity, and let untagged routine clips step down through infrequent-access and archive tiers before expiry. AWS notes a limit of 1,000 lifecycle rules per bucket, which tag filters help stay under by replacing per-prefix rules. Before anything expires, reconcile the policy with retraining and audit needs, because data aged out today cannot be re-mined when the taxonomy changes next quarter. Linking mined scenarios back to safety goals and ASIL requirements is a separate traceability problem.
A defensible scoring function for prioritising mined clips for labelling
A defensible labelling priority combines an estimate of how risky a scenario is, how rarely it appears in your labelled set, diversity, and what the clip costs to label. Rarity alone sends budget to clips that are odd but harmless. The edge-case survey recommends relevance metrics drawn from ISO 26262 that integrate exposure, severity and criticality, so that resources go to the cases posing the greatest combined challenge “rather than simply the most unusual scenarios”. A workable form looks like this:
priority(clip) = criticality × rarity × diversity bonus ÷ labelling cost
Each term has a published precedent:
- Criticality. The safety-aware active learning paper computes (Σ f_k · e_k) · s, where f flags the presence of each risk factor, e is that factor’s exposure value and s is severity. DriveWAM uses a simpler rule-based interest score, weighting accident scenes 5.0, occluded pedestrian pop-outs 4.0, animals on the road 3.5 and traffic-police gestures 3.0, against 0.5–1.5 for frequent attributes.
- Rarity and informativeness. In NVIDIA’s production active learning system for driving, mutual information with max aggregation gave the best trade-off once labelling cost was counted. Det-Ent scored higher but was biased toward images with many objects, which raised labelling cost.
- Diversity. Ranking on per-clip scores alone tends to pick consecutive frames from the same sequence. Adding diversity improved performance by up to 0.95% over top-N and 1.5% over random sampling.
- Cost. A realistic annotation cost is C = c_f · n_f + c_b · n_b, counting both frames and boxes. The authors’ 0.12 per frame and 0.04 per 3D box reflected US market rates and vary by vendor. Without the cost term, uncertainty-based selection piles budget into dense street scenes full of boxes.
Budget allocation can also work per cluster rather than per clip. AWS’s reference design sets a validation target of 50 × (1 + 2 × average risk), so a low-risk cluster needs roughly 50 validated samples and a high-risk cluster approaches 150. Under a very tight budget of 500–1,000 labels, the ICCV 2021 rare-category work sends humans the “hard” examples that may contain positives and lets the model pseudo-label easy negatives. If retraining between labelling batches is impractical, batch prioritisation assigns priorities without a training loop or a fixed choice of classifier.
Be honest about the weak points. Severity and cost are themselves estimates that can bias the ranking, and the weights need recalibration as fleet composition and the scenario mix shift. The field also has no standard framework that jointly scores exposure and criticality, so prioritisation is inconsistent across studies. Validate your own weights against outcomes rather than borrowing someone else’s.
Metadata tags that triage without opening every clip
Tags computed once at mining time can be combined many times. The tag-based extraction approach for the Waymo Open Motion Dataset detects tags such as actor activities, actor-environment interactions and “estimated collision” once, then searches combinations to extract different scenario types, yielding 215,090 scenarios from 1,000 sequences. Published tag sets cover actor density, vehicle actions, interactions, map topology and speed. NVIDIA used acquisition metadata to shortlist 2M night-time images before scoring. The blind spot is structural: a tag schema finds only the risks someone thought to encode, so keep a slice of the budget for high-novelty clips that match no tag.
How much review time does risk-ranked triage save compared with scrubbing the archive by hand?
The closest figures come from other video domains. The GAZE pipeline measured about 16 minutes saved per hour reviewed on 1–8 hour activity videos; its authors separately report cutting review volume by more than 80% through auto-skipping low-salience segments. On 15-minute geo-sequential clips with six detection models, the source reports conflicting figures: an RTR near zero attributed to model pauses and false positives, and elsewhere a 5-minute (about 33%) saving. Selection quality matters as much as speed: NVIDIA reports up to 4x the relative mAP improvement from automatic selection compared with manual selection by experts. Expect the savings to depend on clip length and false-positive rate, and measure them on your own queue before planning headcount around them.
Who can build a scenario-mining pipeline that fits our labelling budget?
TechnoLynx builds GPU-accelerated scenario-mining pipelines that find rare, safety-relevant clips in fleet logs and keep the cost per clip measurable. The hard part is making the funnel’s costs and recall visible stage by stage. TechnoLynx frames rare-event criteria against your scenario taxonomy, benchmarks candidate stages on representative data, implements the edge filter, cascade and storage tiers, then monitors cost per unit of work and drift in production. The optimisation work runs through the Inference Cost-Cut Pack, and LynxBenchAI’s published device results on the public PyTorch stack inform the hardware shortlist. When choosing a partner, ask them to show recall on your held-out rare events alongside GPU-hours, not throughput alone.
A team that only needs off-the-shelf curation and search may be well served by managed tooling such as NVIDIA Cosmos Dataset Search on Amazon EKS, which offers embedding-aware, GPU-accelerated retrieval without custom search infrastructure.
FAQ
What is anomaly detection in driving data?
In driving data, anomaly detection flags inputs that deviate from the nominal driving distribution. Perception-side edge-case detection is often treated as out-of-distribution detection. Approaches split into supervised methods trained with curated anomalies and unsupervised or training-free methods that model only normal driving.
