Camera-Radar Forecasting Pretraining

CRISP: A Spatiotemporal Camera-Radar Backbone for Driving via Forecasting-Based World-Model Pretraining

Learning transferable bird's-eye-view driving representations from camera and radar, with LiDAR used only as privileged supervision during pretraining.

Jingyu Song*1 Yi Liu*1 Katherine A. Skinner1

*Equal contribution    1University of Michigan

Project Repo Paper Coming Soon Code Coming Soon
CRISP uses camera-radar inputs at inference and LiDAR supervision during pretraining.

CRISP uses practical camera-radar inputs for deployment while learning from future LiDAR point clouds during pretraining.

Abstract

Camera-radar fusion is a practical sensing configuration for autonomous driving, but most CR models are trained only with task-specific supervision, limiting reusable representation learning. CRISP is a spatiotemporal CR backbone pretrained through forecasting. Given historical multi-view images and radar sweeps, CRISP learns a bird's-eye-view representation by predicting future LiDAR point clouds. LiDAR is used only as privileged supervision during pretraining; deployment requires only camera and radar.

To make predictive pretraining effective for CR fusion, CRISP introduces an enhanced radar encoder, radar-enhanced temporal self-attention, and multimodal feature rendering with modality innovation gating. On nuScenes, CRISP improves long-horizon point cloud forecasting and transfers to 3D detection, tracking, online mapping, motion forecasting, future occupancy prediction, and planning.

Method Overview

CRISP fuses dense camera semantics with sparse radar range and Doppler evidence inside a shared spatiotemporal BEV backbone, then optimizes the representation with LiDAR-supervised future occupancy forecasting.

Architecture overview of CRISP.

Enhanced Radar Encoder

Radar sweeps are encoded in BEV with ego-motion-aware conditioning and residual spatial calibration, making Doppler cues easier to align with camera-derived BEV features.

Radar-Enhanced Temporal Attention

Radar primes BEV queries before temporal propagation, allowing range and velocity evidence to guide how historical BEV memory is sampled and fused.

Multimodal Feature Rendering

Camera and radar evidence are composed through modality-specific attention and gated residual updates, letting each BEV token select useful visual and motion evidence.

Radar-enhanced temporal self-attention module.
Radar-enhanced temporal self-attention.
Multimodal feature rendering module.
Multimodal feature rendering.

Results

CRISP is evaluated on nuScenes for point cloud forecasting and downstream autonomous driving tasks. The pretrained backbone transfers to perception, prediction, and planning while retaining LiDAR-free inference.

0.84 3.0s CD with 3s history
53.2 3D detection mAP
61.3 3D detection NDS
48.6 Tracking AMOTA

Long-Horizon Forecasting

CRISP improves future point cloud forecasting from 1.5s onward across history settings, showing the value of radar-aware temporal BEV representations for predicting scene evolution.

Broad Downstream Transfer

The pretrained backbone improves BEVFormer and UniAD variants on 3D detection, tracking, motion forecasting, future occupancy prediction, and planning.

Qualitative CRISP forecasting result example one. Qualitative CRISP forecasting result example two.

BibTeX

@article{song2026crisp,
  title   = {CRISP: A Spatiotemporal Camera-Radar Backbone for Driving via Forecasting-Based World-Model Pretraining},
  author  = {Song, Jingyu and Liu, Yi and Skinner, Katherine A.},
  journal = {To appear},
  year    = {2026}
}