NeurIPS 2026

ProMo3D: Probing Motion Cues in
Frozen 3D Foundation Models

Supervised motion probing on frozen geometric representations, using low-rank probes trained on 160 YouTube-VOS videos.

Xiaoang Zhang1   Mert Kiray1,2,3   Yordanka Velikova1,2
  Manoj Biswanath1,2   Benjamin Busam1,2,3

arXiv coming soon GitHub Models Data
PMD learns a low-rank pairwise probe on frozen geometric tokens and produces dynamic score maps, coarse masks, and optional SAM2-refined masks.
Probing-based Motion Distillation (PMD): from frozen geometry to motion segmentation.

Abstract

Feed-forward 3D foundation models have recently reshaped multi-view reconstruction, yet dynamic 4D understanding still often relies on large-scale motion supervision or hand-crafted, architecture-specific heuristics. We ask whether motion-relevant information is recoverable from frozen 3D tokens using low-capacity, rank-restricted probes trained with limited source-domain supervision. Across three representative backbones, supervised probing exposes static–dynamic discriminative information, and a shuffled-label control tests selectivity to the correct token–label correspondence. Rank-restricted and spectral analyses on π3X characterize compactness of the supervised probe-induced representation. Building on this finding, Probing-based Motion Distillation (PMD) converts pairwise motion-state compatibility against a fixed anchor into dense motion scores and coarse masks, with optional SAM2 refinement. Trained on 160 YouTube-VOS videos using a single 16 GB GPU, PMD transfers zero-shot to DAVIS, SegTrackv2, and FBMS-59. Its strongest coarse variant is competitive with heavier motion decoders, while optional refinement improves mask quality. These results support data-efficient supervised recovery of motion-relevant information from frozen 3D representations.

02 / Probing-based motion distillation

Method overview

PMD segments moving objects from frozen 3D representations using a low-rank pairwise probe.

01 Frozen 3D backbone

Rhino context frame 68 Rhino context frame 72 Rhino target frame 70 Input frames 3D foundation model H = [h0, h1, …, hN] Target-frame tokens

Extract target-frame tokens from selected intermediate layers, based on the layer-wise probing results. The token group is fixed per backbone, and scores are averaged across all its layers.

02 Low-rank quadratic probe

Fixed anchor token and token being scored passed to a low-rank quadratic probe Both are spatial tokens from the same target frame. The blue anchor is a fixed static reference; the purple token is scored against it. They are not attention query and key projections. Anchor token ha Token being scored hi Low-rank quadraticprobe p^istat=f(ha,hi) The anchor a = 0 lies in Pi3X's padded border and is static by design.

Train the rank-8 probe to predict whether two tokens from the same target frame share the same motion state, using 160 YouTube-VOS videos. At inference, compare each token hi with the fixed anchor ha; compatibility with the static anchor gives the static score.

03 Coarse motion mask

Stored PMD dynamic score map for rhino frame 70Dynamic score map Stored PMD coarse mask overlay for rhino frame 70Coarse mask

Average dynamic scores across the selected blocks, bilinearly upsample, and binarize using the same thresholding rule across benchmarks.

04 Optional SAM2 refinement

PMD coarse mask supplied to SAM2 refinementCoarse mask SAM2 Stored SAM2-refined PMD overlay for rhino frame 70Refined mask

Use region masks generated from PMD scores as SAM2 prompts to refine mask extent and boundaries. This stage is optional and separate from probe training.

Pi3X-PMD outputs for the same target frame. Token vectors and the probe diagram are schematic.
03 / Empirical analysis

Representation analysis

Layer-wise motion decodability on Pi3X, DA3, and DUSt3R. Click the figure to enlarge.
Reference

Citation

@inproceedings{zhang2026promo3d,
  title={ProMo3D: Probing Motion Cues in Frozen 3D Foundation Models},
  author={Zhang, Xiaoang and Kiray, Mert and Velikova, Yordanka
          and Biswanath, Manoj and Busam, Benjamin},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  url={https://neurips.cc/virtual/2026/poster/151302}
}