Leaderboard · updated 2026-09-23
Which degradation breaks which segmentation model — and by how much.
Relative mIoU drop on real RELLIS-3D test frames when the same frames are degraded. Lower is more robust. Where an independent (held-out) generator exists we use it, so no model — including ours — has seen the corruption code.
| # | Model | Clean mIoU | DustDust plume | Nightheld-out | Rain on lensRain on lens | Mud / spatterheld-out | Fogheld-out | Motion blurheld-out | Mean drop |
|---|---|---|---|---|---|---|---|---|---|
| 1 | DeepLabV3-MobileNetV3CNN · 11.0M | 35.0 | −36.4% | −63.6% | −31.9% | −26.2% | −37.5% | −27.4% | −37.2% |
| 2 | SegFormer-B0Transformer · 3.7M | 38.0 | −43.7% | −63.6% | −26.1% | −43.3% | −23.0% | −30.8% | −38.4% |
| – | Mask2Former (Swin-T)Mask transformer | running on our GPU now | |||||||
| – | SegFormer-B2Transformer | running on our GPU now | |||||||
| – | OneFormer (Swin-T)Universal transformer | needs a larger GPU — queued | |||||||
| – | SAM 2 (Hiera-T encoder + seg head)Foundation encoder | class-agnostic model; needs a trained semantic head — queued | |||||||
reading it
Night is the worst condition on average (−64%). The biggest architecture split is mud / spatter: Transformer models lose 43%, CNN models 26%. That’s why a single “robustness score” per model hides what you actually need to fix.
Want your model on this board? Send it — we run the same protocol and publish with your permission.
How much a pack wins back
Fine-tuning from each baseline. Control = the same extra training on clean data only, so the gain can’t be extra steps. Held-out degraded mIoU, mean over seeds.
| Model | Control | +200 synthetic frames | + online synthesis | Clean-weather change |
|---|---|---|---|---|
| DeepLabV3-MobileNetV33 seed(s) | 21.7 | 24.5 (+2.8) | 25.6 (+3.9) | -0.5 |
| SegFormer-B03 seed(s) | 22.8 | 25.2 (+2.4) | 26.0 (+3.2) | +0.6 |
Methodology
- Real test frames only. No synthetic frame is ever scored as ground truth.
- Held-out generators. Night = Brooks/Cui unprocessing low-light; mud = ImageNet-C spatter; fog and motion blur = ImageNet-C. None is used to make training data.
- Control arm and seeds. Every gain is measured against identical training on clean data, over several seeds.
- Metric. mIoU over the RELLIS-3D classes present in the test split; relative drop = (clean − degraded) / clean, averaged over 3 severities.
limits, stated plainly
- Degraded test inputs are synthetic (held-out generators): RELLIS-3D contains no real dust or night frames.
- Small models at 640×400 on one consumer GPU; absolute mIoU is below full-resolution published numbers.
- Rows marked queued are not measured yet — no numbers are shown for them.
Read the evaluation protocol & code layout.