First Forecasting-Based Camera-Radar Pretraining

CRISP: A Spatiotemporal Camera-Radar Backbone for Driving via Forecasting-Based World-Model Pretraining

CRISP learns a reusable camera-radar BEV backbone by forecasting future LiDAR point clouds during pretraining, then deploys with only camera and radar for perception, prediction, and planning.

Jingyu Song*1 Yi Liu*1 Katherine A. Skinner1

*Equal contribution    1University of Michigan

Camera + radar inference LiDAR supervision only in pretraining Full-stack downstream transfer
Paper arXiv Project Repo Code Coming Soon
CRISP pretrains a camera-radar backbone with LiDAR supervision and transfers to downstream driving tasks.
Paper Fig. 1: CRISP uses temporal camera-radar observations, learns through LiDAR-supervised 3D forecasting, and transfers to the full driving stack.

Camera-Radar Pretraining for Deployable Driving

Camera-radar fusion is practical for production vehicles, but most CR systems are trained only for one supervised task. CRISP addresses this gap with forecasting-based representation learning for a CR backbone: historical multi-view images and radar sweeps predict future LiDAR point clouds during pretraining, while downstream inference remains LiDAR-free.

The key shift is to pretrain the fused CR representation itself. Radar is injected into temporal BEV propagation and multimodal feature rendering inside the backbone, so the deployed model carries visual semantics, metric radar range, Doppler motion cues, BEV geometry, and temporal dynamics into downstream heads.

Table 1

Long-Horizon Forecasting Surpasses Camera-LiDAR Pretraining

On nuScenes point cloud forecasting, CRISP is a camera-radar model at inference time, yet it beats the camera-LiDAR LRS4Fusion baseline from 1.5s through 3.0s across every history setting in Table 1.

0.84 3.0s CD with 3s history
1.23 LRS4Fusion C+L at the same horizon
1.73 ViDAR camera-only at the same horizon
Chamfer Distance at long horizons from paper Table 1. Lower is better.
History Method Modality 1.5s 2.0s 2.5s 3.0s
0s LRS4Fusion C+L 0.75 0.96 1.21 1.51
0s CRISP C+R 0.67 0.74 0.82 0.91
1s LRS4Fusion C+L 0.64 0.79 0.99 1.25
1s CRISP C+R 0.60 0.68 0.76 0.86
3s LRS4Fusion C+L 0.61 0.77 0.97 1.23
3s CRISP C+R 0.58 0.65 0.74 0.84

Full-Stack Camera-Radar Solution

One Pretrained CR Backbone Improves the Driving Stack

After pretraining, the forecasting decoder is removed and the same CRISP backbone initializes BEVFormer and UniAD heads. The gains span perception, mapping, prediction, future occupancy, and planning.

53.2

3D Detection mAP

BEVFormer-CRISP reaches 61.3 NDS, outperforming ViDAR-pretrained BEVFormer and SpaRC-AD.

48.6

Tracking AMOTA

UniAD-CRISP improves AMOTA, AMOTP, and recall over camera-only and supervised CR baselines.

35.4

Lane IoU

Online mapping remains strong, with best lane and divider IoU among the reported UniAD variants.

0.53

Motion minADE

Forecasting pretraining transfers to agent motion, with best minADE, minFDE, and EPA.

47.5

Far-Range VPQ

Future occupancy improves in both near and far regions, where radar range cues are especially useful.

0.15%

Planning Collision

UniAD-CRISP lowers average planning error to 0.80 m and collision rate to 0.15%.

Method

Radar Enters the Backbone, Not Just the Head

CRISP performs camera-radar interaction inside the spatiotemporal BEV encoder through ego-aware radar encoding, radar-enhanced temporal self-attention, and multimodal feature rendering with modality innovation gating.

Overall CRISP architecture with camera-radar BEV backbone, LiDAR-supervised pretraining, and downstream finetuning.
Paper Fig. 2: during pretraining, future LiDAR supervises occupancy forecasting; during downstream finetuning, only the CR backbone is retained.

Enhanced Radar Encoder

Radar sweeps become BEV features conditioned on ego motion and refined by residual spatial calibration.

Radar-Enhanced Temporal Attention

Radar primes BEV queries before temporal propagation, guiding how history and current evidence are fused.

Multimodal Feature Rendering

Camera and radar propose separate BEV innovations, then query-conditioned gates admit useful updates.

Radar-enhanced temporal self-attention diagram from the paper.
Radar-enhanced temporal self-attention.
Multimodal feature rendering diagram from the paper.
Multimodal feature rendering with modality innovation gating.

Label Efficiency

Pretraining Reduces Dependence on Dense 3D Labels

With 1/4, 1/2, and full nuScenes training data, CRISP improves both mAP and NDS over training the same BEVFormer-CRISP detection architecture from scratch.

Label efficiency plot comparing CRISP pretraining with scratch training for mAP and NDS.
Paper Fig. 5: CRISP gives more than 3 point mAP and NDS gains in low-label detection finetuning and remains better with full data.

Qualitative Forecasting

Radar Cues Stabilize Future World Geometry

The paper examples compare CRISP with the camera-only ViDAR baseline. Future images are shown only for visualization; the model predicts from historical camera-radar inputs.

Low-light qualitative future point cloud prediction comparison between CRISP and ViDAR.
Low-light scene: CRISP preserves the forward vehicle trajectory more accurately than ViDAR.
Qualitative future point cloud prediction comparison where CRISP preserves scene geometry better than ViDAR.
Billboard-like visual ambiguity: CRISP better anchors future geometry to metric scene structure.
Failure case where CRISP begins to drift after early horizons when traffic signal intent is ambiguous.
Failure case: high-level traffic-signal intent remains difficult when geometry and motion cues are not enough.

BibTeX

@article{song2026crisp,
  title   = {CRISP: A Spatiotemporal Camera-Radar Backbone for Driving via Forecasting-Based World-Model Pretraining},
  author  = {Song, Jingyu and Liu, Yi and Skinner, Katherine A.},
  journal = {arXiv preprint arXiv:2607.04541},
  year    = {2026},
  url     = {https://arxiv.org/abs/2607.04541}
}