Validating a Shared-Backbone Network for Hull Detection and Debris Segmentation

One shared-backbone network with separate detection and segmentation heads is a candidate to replace a hull detector and a debris segmenter. In published joint detection and segmentation work it saves parameters and latency, and accuracy ranges from close to single-task (YOLOP) to a modest drop (radar-camera). The risk is negative transfer: joint training can let one task dominate and degrade the others. So treat the merge as a hypothesis to test. Train it with a deliberately chosen loss weighting, then A/B it against your existing two-model pipeline on the same held-out set, with confidence intervals on hull mAP and debris recall.

The studies cited here cover driving, occupancy and radar-camera perception, not hull and debris tasks on maritime imagery. What they do give is the architecture patterns, the balancing methods and the statistics to measure the cost yourself. This article covers those three things.

Should one shared-backbone network replace the hull detector and the debris segmenter?

Yes, provided the merged model matches the two-model pipeline’s per-task metrics within a tolerance you set in advance. If debris still regresses past that tolerance after you have tried balancing, keep two branches or give the debris head more capacity.

TechnoLynx takes on this decision through its Generative & Agentic AI R&D engagement, building the shared-backbone model and benchmarking it against your two-model pipeline, and the first step is agreeing the success metrics and the tolerance with your team.

The MDSP authors (2022), working on occupancy monitoring, report that joint detection and segmentation on one backbone cut learnable parameters by over 35% compared with single-task networks. They also report that inference time for those two tasks dropped by almost 30%, at 36 FPS in real time. In a 2026 radar-camera study, a single forward pass for both tasks avoided the roughly 2× backbone cost of separate pipelines. The multi-task extension added 2.21M parameters to a 38.4M detection backbone, a 5.8% increase.

Hard parameter sharing means a subset of parameters (the backbone) is shared between tasks while the rest (the heads) are task-specific. Most deep multi-task models take this form: a shared representation function feeding task-specific decision functions.

One failure mode is over-sharing. The Learning to Branch authors (2020) note that over-sharing can enforce over-generalisation and cause negative knowledge transfer across tasks. Standley et al. (2019) found that task objectives can compete, which often gives worse overall performance than separate networks. Sener and Koltun (2018) argue that minimising a weighted sum of losses only makes sense if one parameter set works for all tasks, which is rarely the case.

In this merge, we judge debris segmentation the task most exposed to this. A segmentation loss can be on a different scale from a detection loss. The same paper lists these exact training difficulties: some tasks are easier, some have noisier labels, and some have losses on very different scales. If the hull loss dominates the shared gradients, debris performance can degrade.

Joint detection and segmentation without trading one task for the other

Several published patterns address this. Panoptic FPN adds a semantic segmentation branch to Mask R-CNN on a shared Feature Pyramid Network backbone, and its authors found it stayed effective for instance segmentation while also being strong at semantic segmentation. YOLOP uses one shared encoder and separate decoders with no complex shared blocks between them, trained end to end. HybridNets (2022) follows the same shared multi-task pattern with a loss designed to balance the tasks, and reports 77.3 mAP on Berkeley DeepDrive.

The detection head type matters. The YOLOP authors saw detection and segmentation conflict in a region-proposal (R-CNN-style) framework but not in their own. They attribute this to their detection head predicting over the whole feature map, as segmentation heads do, while a proposal-based head works on individual regions. For a hull-and-debris merge, a dense, grid-style detector next to a segmentation decoder is therefore the less risky starting point. If you are still choosing a detector family, our guide to object detection tools covers that choice separately.

The TripleNet authors (2018) argue that simple feature sharing does not fully use how closely detection and segmentation are related. They add extra supervision on each decoder layer, and that supervision is not run at test time, so there is no extra inference cost. Lighter options exist too. MTAN puts a task-specific soft-attention module on a shared feature pool. The 2026 radar-camera study’s cross-task attention module improved segmentation by 0.6 mIoU with detection essentially unchanged, using 0.58M extra parameters.

We prefer the plain shared-backbone, separate-heads layout as the first candidate because it keeps the system modular. Each head’s output, loss and failure cases can be inspected on their own, and when debris recall drops you can see which head it shows up in. That visibility has limits: Kendall et al. (2018) observed that failures were highly correlated across outputs conditioned on the same shared features. Per-head monitoring tells you where an error shows up, not always where it started.

Which loss-weighting scheme stops one task dominating training?

Start with a well-tuned static weighted sum. Move to an adaptive method only if the sweep shows the static weights are unstable or too expensive to search. Three adaptive schemes each balance tasks using a different signal. Uncertainty weighting learns task-dependent uncertainty parameters, GradNorm uses gradient magnitudes and relative training rates, and Dynamic Weight Average uses recent loss-reduction ratios.

Uncertainty weighting (Kendall et al., 2018): treats homoscedastic task uncertainty as a learned weight. The authors show performance is highly sensitive to manually chosen weights and that the best weights depend on each task’s scale and noise.
GradNorm (Chen et al., 2018): tunes the weights so that task gradient norms sit on a common scale and tasks train at similar rates. In its synthetic example with equal weights, the task with the larger loss scale stopped the other task from learning.
Dynamic Weight Average: adapts the weights from each task’s rate of loss change. Unlike GradNorm it needs only the loss values, not internal gradients, which makes it simpler to implement. MDSP used it for its detection and segmentation weights.

These methods are not clearly better than a sum of losses. Kurin et al. (NeurIPS 2022) found that unitary scalarization (minimising the plain sum of task losses) with standard regularisation matched or beat complex multi-task optimisers in popular supervised and reinforcement-learning settings. Xin et al. (NeurIPS 2022) found that multi-task optimisation methods gave no improvement beyond what a weighted average of task losses can reach. They also found that the dynamic weights mostly did not move much during training, so in effect those methods behaved like static weighting. The GradNorm and uncertainty-weighting papers, by contrast, report gains over equal or hand-tuned weighting. We find the 2022 comparisons stronger on this point. Kurin et al. evaluate the optimisers in one pipeline, show run-to-run variability, and note that the baseline’s performance had been consistently underestimated.

In those comparisons, a plain loss sum with standard regularisation and early stopping matched the adaptive optimisers. Uncertainty weighting is cheap to add if you want to automate the sweep. Three caveats apply:

Results are sensitive to balancing hyperparameters: Xin et al. found that even subtle choices in the hyperparameter grid could drastically change results.
Uncertainty weighting can misjudge which task needs help. GradNorm’s authors report it putting more weight on keypoint regression while the classification tasks were the undertrained ones.
Weighting comparisons from single runs prove little. A 2026 YOLOv8 study of detection-plus-segmentation weighting reported its rankings only descriptively, because it had one run per configuration and no single-task baselines.

How do you A/B the merged model against the two-model pipeline on the same test set?

Evaluate both configurations on one frozen held-out set using the same evaluation code, IoU thresholds and confidence cutoffs. Report hull mAP and debris recall for each, with confidence intervals, and use a paired test on the per-image or per-object differences. The test set has to be fixed because a mAP score is not a fixed property of a model. It is an estimate from a finite set of images and annotations, shaped by choices such as the IoU threshold and the class definitions.

TechnoLynx runs this head-to-head validation as part of its Computer Vision R&D work, benchmarking the shared-backbone and two-branch architectures on mAP and recall/IoU. The first step is defining the detection and segmentation success metrics.

Procedure for a merged-versus-separate comparison

Freeze one held-out set that neither configuration sees in training, and stratify it so rare debris types are present in useful numbers.
Hold everything fixed except the architecture: same data, augmentation and training budget. Otherwise a recall gap can come from training differences rather than shared capacity.
Train several seeds of each configuration where you can afford it. The YOLOv8 weighting study warns that without repeated seeds, differences may come from initialisation, data ordering or checkpoint variation.
Compute intervals with an image-level bootstrap: resample whole images with replacement, keep all their boxes and predictions, recompute mAP, precision and recall, and repeat thousands of times. Resampling by image keeps the correlations between objects in the same frame.
Test the difference, not just the two scores. Non-overlapping intervals indicate a significant difference, while overlapping ones leave the comparison inconclusive. For models trained only once, Dietterich (1998) found McNemar’s test to be the only one of the tests he reviewed with acceptable type I error. Where ten training runs are affordable, he recommends 5×2 cross-validation.
Report per-class results as well as the aggregate. In one worked example, per-class intervals showed a trade-off between two rare pedestrian classes that the aggregate mAP hid.

Small test sets make rare-debris recall noisy

Recall measured on a small debris sample is a weak estimate. A 2012 analysis of recall intervals found the recall estimator becomes more biased at low prevalence, small sample sizes and low recall. The same analysis recommends against naive binomial and normal-approximation intervals for recall and favours a beta-binomial posterior with a half prior. Shang and Langlois show that oversampling the minority class tightens precision intervals on imbalanced data, and that recall has an optimal oversampling ratio. For segmentation metrics, a 2023 brain MRI study found that a 1%-wide confidence interval typically needed about 100 to 200 test samples at low spread, and more difficult tasks could need over 1,000. It recommends the bootstrap because it makes no assumption about the metric’s distribution.

If the debris interval is wider than the regression you would tolerate, the test cannot settle the merge. The fix is more labelled debris, not a decision made on a point estimate.

In the published joint detection and segmentation results below, the cost ranges from a modest drop to a gain. Your own debris task still has to be measured, because these are different datasets. YOLOP’s multi-task model reached 89.2% detection recall against 88.2% for detection alone (+1.0). Its mAP50 was 76.5% against 76.9% (−0.4), and drivable-area mIoU was 91.5% against 91.6% (−0.1). The 2026 radar-camera multi-task baseline scored 55.7 NDS against 56.8 for its single-task counterpart, a 1.1-point detection drop that the authors call modest negative transfer. MDSP reports the opposite direction: its joint model beat the single-task detection baseline by over 0.04 AP and the segmentation baseline by more than 0.1 mIoU.

These results come from driving, occupancy and radar-camera datasets. Standley et al. (2019) found that training tasks together often gives inferior overall performance because task objectives can compete. Run the measurement on your own data before assuming the merge is free. If the hull detections feed a multi-object tracker downstream, check track continuity as well, because a small per-frame detection change can matter more once it compounds over a track. If the hull estimate also has to survive camera dropout by fusing sonar, DVL and IMU, that is a separate sensor-fusion design question, not a multi-task one.

Keep the merged model or return to two branches

Result on the frozen test set	Decision
Hull mAP and debris recall both within tolerance, intervals narrow enough to rule out a larger loss	Adopt the merged model for its parameter and latency savings
Debris recall regresses beyond tolerance under a fixed weighting	Sweep loss weights, then try uncertainty weighting or DWA, and re-test
Debris still regresses after balancing	Increase debris-head capacity or move the branch point earlier, then re-test
Regression persists, or the tasks clearly compete	Keep a dedicated two-branch model and accept the compute cost
Debris interval too wide to decide	Label more debris to enlarge the test set before deciding

The tolerance in the first row should come from what a missed debris object costs downstream, for safety or operations. It is a requirement you set, not a statistical default. Before reading any gap as the cost of sharing, check that the loss weighting was the same across runs. Otherwise you are measuring a weighting change as well as a capacity effect.

Who can build and validate a merged hull and debris model for us?

TechnoLynx builds the shared-backbone model and proves it against your current two-model pipeline through its Generative & Agentic AI R&D engagement. The engagement covers the design (shared backbone, task-specific heads, balanced loss), implementation and tuning for the target hardware. It also covers the head-to-head validation on hull mAP and debris recall, and integration into production with monitoring for drift in the balance between tasks. The engagement starts by agreeing the success metrics and the acceptable debris-recall tolerance with your team.

The engagement delivers the engineering and its validation evidence. The other route is an in-house team that already runs multi-task training and statistical evaluation. The published shared-backbone designs discussed earlier, such as YOLOP and HybridNets, are a reasonable reference point for either route.