Leaderboard · updated 2026-10-08
Which degradation breaks which segmentation model — and by how much.
Relative mIoU drop on real RELLIS-3D test frames when the same frames are degraded. Lower is more robust. Where an independent (held-out) generator exists we use it, so no model — including ours — has seen the corruption code.
| # | Model | Clean mIoU | DustDust plume | Nightheld-out | Rain on lensRain on lens | Mud / spatterheld-out | Fogheld-out | Motion blurheld-out | Mean drop |
|---|---|---|---|---|---|---|---|---|---|
| 1 | OneFormer (Swin-T)Mask transformer · 50.7M | 45.7 | −43.3% | −37.0% | −32.3% | −26.1% | −14.1% | −41.4% | −32.4% |
| 2 | SegFormer-B2Transformer · 27.4M | 42.9 | −47.5% | −53.3% | −24.6% | −28.8% | −17.7% | −24.0% | −32.6% |
| 3 | Mask2Former (Swin-T)Mask transformer · 47.4M | 43.6 | −45.3% | −53.6% | −34.6% | −32.4% | −18.8% | −30.7% | −35.9% |
| 4 | DeepLabV3-MobileNetV3CNN · 11.0M | 35.0 | −36.4% | −63.6% | −31.9% | −26.2% | −37.5% | −27.4% | −37.2% |
| 5 | SegFormer-B0Transformer · 3.7M | 38.0 | −43.7% | −63.6% | −26.1% | −43.3% | −23.0% | −30.8% | −38.4% |
| – | SAM 3 (zero-shot, text prompts)Foundation model | one text prompt per class, no training — planned | |||||||
reading it
Night is the worst condition on average (−54%). The biggest architecture split is fog: CNN models lose 37%, Mask transformer models 16%. That’s why a single “robustness score” per model hides what you actually need to fix.
Want your model on this board? Send it — we run the same protocol and publish with your permission. Or get a private failure-map report.
Our own model, at the camera’s real resolution
SegFormer-B5 trained on every labelled off-road frame we can use — RELLIS-3D, GOOSE, GOOSE-Ex and RUGD, about 22,500 frames — at native resolution (RELLIS-3D 1920×1200, GOOSE 2048×1000), scored on all 1,672 RELLIS-3D test frames. These numbers are a different measurement set from the board above and are printed apart from it on purpose: more training data, more pixels, and a degradation covers a different share of an object at 1920×1200 than at 640×400.
| Run | RELLIS-3D test | GOOSE val | GOOSE val rain/snow | Held-out degraded | Our degradations |
|---|---|---|---|---|---|
| Full schedule, multi-scale30,000 iterations, 640 crops scaled 0.5–2.0. The recipe as designed. | 46.8 | 57.2 | 55.8 | 33.8 | 26.5 |
| Schedule cut to one session10,609 of 30,000 iterations, fixed scale — the budget shortened it to fit a free GPU session. | 44.2 | 56.9 | 55.2 | 32.4 | 26.6 |
| Then fine-tuned on RELLIS-3D alone6,000 further iterations on the benchmark's own training split and nothing else. | 48.0 | 44.5 | 45.5 | 35.4 | 28.2 |
what it says
Running the schedule to the end and training multi-scale is worth +2.6 points of clean mIoU over the same recipe cut short — and essentially nothing under degradation (26.6 → 26.5 on our generators). Capacity and schedule buy clean accuracy; they do not buy robustness. What does is in the sections below. The RELLIS-only fine-tune scores higher still on RELLIS-3D — 48.0 — by training on nothing else, and it pays 12.7 points of GOOSE for it. That is a model tuned to one benchmark, which is the opposite of what a vehicle needs, so it is not the one we use.
Published SegFormer results on RELLIS-3D sit at 51.5–52.9 mIoU. The model we would ship is at 46.8, so finishing the schedule closed part of that gap and not all of it. The remainder is not the schedule — that hypothesis has now been tested and spent — and we would rather say so than quote the 48.0 next to it and leave out what it cost.
How much each kind of training data wins back
Fine-tuning from each baseline, all with the same schedule. Control = extra training on clean data only. Standard aug = off-the-shelf albumentations weather transforms (what most teams already use). Combo = half standard, half Siltframe. Held-out degraded mIoU, mean over seeds; best per row in bold.
| Model | Control | +200 Siltframe frames | Siltframe online | Standard aug | Combo |
|---|---|---|---|---|---|
| OneFormer (Swin-T)1 seed(s) | 32.1 | 34.9 (+2.7) | 34.8 (+2.7) | 35.4 (+3.2) | 35.4 (+3.3) |
| SegFormer-B21 seed(s) | 29.4 | 32.2 (+2.8) | 33.4 (+3.9) | 35.1 (+5.6) | 34.7 (+5.3) |
| Mask2Former (Swin-T)1 seed(s) | 28.1 | 32.0 (+4.0) | 33.0 (+4.9) | 34.7 (+6.6) | 35.6 (+7.6) |
| DeepLabV3-MobileNetV33 seed(s) | 21.7 | 24.5 (+2.8) | 25.6 (+3.9) | 26.9 (+5.2) | 27.1 (+5.4) |
| SegFormer-B03 seed(s) | 22.8 | 25.2 (+2.4) | 26.0 (+3.2) | 28.4 (+5.6) | 27.7 (+4.9) |
Real rain, fog, night and snow
Models trained on RELLIS-3D only. Every checkpoint scored zero-shot on IDD-AW — 475 real adverse-weather frames on unstructured roads. Drivable = IoU of the drivable surface; mean = average over drivable, vegetation, sky and person. Change vs. control in brackets.
| Model | Control | Siltframe online | Standard aug | Combo | ||||
|---|---|---|---|---|---|---|---|---|
| drivable | mean | drivable | mean | drivable | mean | drivable | mean | |
| OneFormer (Swin-T) | 35.4 | 55.9 | 40.8 (+5.3) | 55.4 (-0.5) | 30.7 (-4.7) | 54.4 (-1.6) | 39.9 (+4.5) | 56.0 (+0.1) |
| SegFormer-B2 | 6.0 | 40.8 | 13.0 (+7.0) | 40.4 (-0.4) | 5.5 (-0.5) | 41.9 (+1.0) | 10.6 (+4.6) | 40.8 (-0.0) |
| Mask2Former (Swin-T) | 11.1 | 45.2 | 23.6 (+12.5) | 49.1 (+3.9) | 10.3 (-0.8) | 46.0 (+0.8) | 21.7 (+10.6) | 48.3 (+3.1) |
| DeepLabV3-MobileNetV3 | 22.4 | 36.3 | 24.9 (+2.5) | 34.8 (-1.5) | 15.6 (-6.9) | 36.2 (-0.1) | 19.7 (-2.8) | 33.7 (-2.6) |
| SegFormer-B0 | 25.3 | 44.5 | 30.4 (+5.0) | 41.0 (-3.5) | 25.8 (+0.5) | 44.3 (-0.2) | 27.5 (+2.2) | 41.6 (-2.9) |
what it says
On real bad weather, Siltframe data is the only option that improved drivable-surface detection — on all 5 architectures — while standard augmentation left it flat or worse. The cost: sky accuracy drops on 5 of 5 (by up to 13 points), so the four-class mean improves on only 1 of 5. But this picture changes once the training data itself is diverse — see the next section.
The lesson for anyone buying synthetic data, including from us: synthetic benchmarks and real weather can rank methods differently. Validate on real adverse frames.
What diverse real data does
The same architectures retrained on RELLIS-3D plus 1,500 GOOSE frames (German forest tracks, fields and roads; clear weather only), then fine-tuned with a control arm and with Siltframe (generator v2: label-aware sky, voiding pixels a dense veil hides). Real weather = IDD-AW, four-class mean IoU.
| Model | Real weather (IDD-AW) mean | IDD-AW drivable | Synthetic held-out mIoU | ||||
|---|---|---|---|---|---|---|---|
| RELLIS only | + GOOSE | + GOOSE + Siltframe | + GOOSE | + Siltframe | + GOOSE | + Siltframe | |
| SegFormer-B2 | 40.8 | 75.6 (+34.8) | 75.1 (-0.5) | 87.6 | 86.2 | 27.3 | 32.6 (+5.2) |
| Mask2Former (Swin-T) | 45.2 | 77.7 (+32.5) | 77.9 (+0.2) | 86.4 | 85.4 | 28.1 | 33.0 (+4.9) |
| DeepLabV3-MobileNetV3 | 36.3 | 64.6 (+28.3) | 64.4 (-0.2) | 81.0 | 83.4 | 22.7 | 26.3 (+3.6) |
| SegFormer-B0 | 44.5 | 70.5 (+26.0) | 70.0 (-0.5) | 82.5 | 82.2 | 22.5 | 26.0 (+3.5) |
what it says
Adding 1,500 diverse real frames (GOOSE) lifted the real-weather mean by 26–35 points on every architecture — far more than any augmentation did. With that data in training, Siltframe changed the real-weather mean by -0.5 to +0.2 points (within noise) while still adding +3.5 to +5.2 points on the synthetic held-out degradations, and the sky / vegetation regression disappeared.
Coverage beats augmentation. The first question for a perception team isn’t “which augmentation?” but “which real conditions is my data missing?” — here, overhead tree canopy and paved roads, which RELLIS-3D barely contains. Finding that is what our audit does; synthetic data then fills the conditions you can’t easily capture.
Methodology
- Real test frames only. No synthetic frame is ever scored as ground truth.
- Held-out generators. Night = Brooks/Cui unprocessing low-light; mud = ImageNet-C spatter; fog and motion blur = ImageNet-C. None is used to make training data.
- Control arm and seeds. Every gain is measured against identical training on clean data, over several seeds.
- Metric. mIoU over the RELLIS-3D classes present in the test split; relative drop = (clean − degraded) / clean, averaged over 3 severities.
limits, stated plainly
- Degraded test inputs are synthetic (held-out generators): RELLIS-3D contains no real dust or night frames.
- The comparison models are trained at 640×400 with an identical budget, so architectures are compared like for like; absolute mIoU is therefore below full-resolution published numbers — the relative drops are the point.
- Rows marked queued are not measured yet — no numbers are shown for them.
Read the evaluation protocol & code layout.