Blog / 2026-10-08

Night cuts off-road segmentation by 37–64% — and synthetic data only wins part of it back

Public off-road datasets are almost all daytime, dry and clear. We measured how much segmentation models lose when conditions turn bad, and how much synthetic hard cases recover — with a protocol designed not to flatter the result.

The setup

• Data: 1,200 clear RELLIS-3D training frames; 600 real test frames at 640×400, 19 classes.
• Models: OneFormer (Swin-T), Mask2Former (Swin-T), SegFormer-B0, SegFormer-B2, DeepLabV3-MobileNetV3.
• Held-out degradations: gains are reported on generators never used to make training data — ImageNet-C fog, spatter and motion blur, and an unprocessing-based low-light model. Testing on the code you trained with is how augmentation results flatter themselves.
• Control arm: every synthetic-data fine-tune is compared with an identical fine-tune on clean data, over several seeds.

1 · Night is the universal failure

At night the models lose between 37% and 64% of their mIoU — the largest drop of any condition for most of them. Two independent night models — our headlight-and-sensor-noise simulation and a low-light model from the literature — agree closely on SegFormer-B0: −64.4% vs −63.6%. That agreement is the best evidence the number is real rather than an artefact of one simulator.

2 · Architectures fail differently

Night is the worst condition on average (−54%). The biggest architecture split is fog: CNN models lose 37%, Mask transformer models 16%. A single “robustness score” per model hides exactly what you need to know: which condition your model can’t handle. The full per-condition table is on the leaderboard.

3 · 200 synthetic frames buy back 2.4–4.0 points, honestly measured

Clear-weather mIoU moved by less than a point, so the robustness wasn’t bought with a regression.

4 · The synthetic-to-real gap, in one number

Night mIoU for SegFormer-B0, same model, same training:

• tested with our own night generator (unseen seeds): 13.7 → 26.2 (+12.5)
• tested with an independent low-light model: 13.3 → 18.3 (+4.9)

The gain shrinks by more than half when the test data comes from a different simulator, and real night footage will differ again. If you validate synthetic augmentation on synthetic data, you will overestimate it. Keep a real adverse-condition validation set that never touches training, and select checkpoints on it.

5 · Is physics better than standard augmentation? It depends where you test

The obvious objection: most teams already add fog and darkness with a library such as albumentations. So we fine-tuned every model a third way — off-the-shelf albumentations weather transforms, same schedule — and a fourth, half standard and half Siltframe.

On the synthetic held-out tests, standard augmentation is as good or better — mostly because simple darkening + noise closely matches the low-light test. On real rain, fog, night and snow — with models trained on RELLIS-3D alone — the ranking flips for the thing a vehicle needs most: only the physics-based data improved drivable-surface detection, on every architecture. The price was a drop in sky accuracy (and on most models vegetation). Section 6 shows what happened next.

Two lessons: a benchmark built from a simulator rewards whatever resembles that simulator, and the only honest judge is real adverse-weather data.

6 · Update: the bigger lever was real data

We traced the sky regression with the same tooling we sell: on real frames the models were calling overhead tree canopy “sky” — forest roads and tree tunnels that RELLIS-3D, shot in open terrain, barely contains. Fixing the generator (label-aware sky, voiding pixels a dense veil hides) recovered about half of it. Then we added 1,500 real frames from GOOSE, a German off-road dataset with forest tracks and roads.

Adding 1,500 diverse real frames (GOOSE) lifted the real-weather mean by 26–35 points on every architecture — far more than any augmentation did. With that data in training, Siltframe changed the real-weather mean by -0.5 to +0.2 points (within noise) while still adding +3.5 to +5.2 points on the synthetic held-out degradations, and the sky / vegetation regression disappeared.

So the order of operations matters: first find which real conditions your data is missing and cover them; use synthetic hard cases for what you can’t capture, and judge them on real frames. Details on the benchmark page.

Limitations

• RELLIS-3D has no real dust or night frames, so degraded test inputs are synthetic (from held-out generators). Validation on real adverse captures is the next step.
• The comparison models are trained at 640×400 with an identical budget, so architectures are compared like for like; absolute mIoU is below full-resolution published results. The relative drops are the point.
• We don’t synthesise motion blur yet, and it shows: almost no gain there.