PhyVGGT Paper

PhyVGGT: Feed-Forward Articulated Object Reconstruction from Monocular Videos

Qi Shao1,2, Zhihao Cao3,6, Shuhao Zhai4, Anh Nguyen2,5, Hong Liang1, Baoru Huang2,5, Liang Li1

1Harbin Engineering University   2University of Liverpool   3ETH Zurich   4University of Macau   5Imperial College London   6National University of Singapore

01Abstract

  • Problem. Reconstructing 3D movable parts, joint parameters and unseen articulation states from an uncalibrated monocular video, without depth sensors, camera poses or per-scene optimisation.
  • Challenge. Feed-forward 3D foundation models give dense point maps but no parts or joints, and fitting joints on these point maps fails because moving boundaries blur, motion along the viewing ray is underestimated and small opening angles leave continuous axes ill-conditioned.
  • Method. PhyVGGT anchors part matching and joint estimation to the undistorted stationary base: a Part and Motion Decoder references part queries to a static-base query, and a Kinematic Inference Layer selects the joint axis on the base's Manhattan frame under a Bingham posterior and solves the hinge pivot linearly.
  • Result. Trained only on synthetic data, PhyVGGT achieves state-of-the-art monocular performance across five synthetic and real-world benchmarks, surpasses posed RGB-D ArtiPoint in median error on Arti4D, and runs in 6.8 s per 48-frame video.
Full abstract

Reconstructing an articulated object with its 3D movable parts, joint parameters and unseen articulation states from an uncalibrated monocular video is essential for robot manipulation and interactive digital twins. Existing per-scene optimisation methods require minutes to hours and external depth or camera poses, whereas feed-forward 3D foundation models reconstruct dense point maps in one pass without segmenting movable parts or inferring joints. Even on these monocular point maps, regressing or fitting continuous joints directly fails on casual videos, since cross-frame attention blurs moving part boundaries in deep layers, underestimates motion along the viewing ray and leaves continuous axes ill-conditioned under small opening angles. To address these challenges, we propose PhyVGGT, a single-pass framework on a frozen 3D foundation model that anchors moving-part matching and closed-form joint estimation to the undistorted geometry of the stationary base. First, our Part and Motion Decoder fuses intermediate per-frame tokens with global tokens and references part queries to a static-base query to segment 3D parts, align point maps on the static base and predict learned Bingham priors. Second, our Kinematic Inference Layer fits an orthogonal Manhattan frame to static surface normals, selects the joint axis in closed form under a Bingham posterior combining pairwise motion scatter with the learned priors, and solves a linear system for the hinge pivot to drive unseen states. Trained solely on synthetic data, PhyVGGT achieves state-of-the-art monocular performance across five benchmarks, outperforming FAMOS and monocular ArtiPoint, surpassing posed RGB-D ArtiPoint in median error, and running in 6.8 s per 48-frame video against hours of per-scene optimisation in Articulat3D.

PhyVGGT overview: monocular video to 3D movable parts, joints and unseen states

Swipe horizontally to inspect details

Figure 1. Single-pass 3D articulated object reconstruction from uncalibrated monocular RGB videos. Given casual RGB frames (left), PhyVGGT segments 3D movable parts (teal), estimates joint axes (orange) and hinge pivots (red) across observed articulation states, and drives each part to unseen states in 1.2 s per clip (centre), generalising zero-shot from synthetic training to diverse real-world scenes (right).

Kinematic Inference Layer on real point maps

Figure 2. Static-base normals form an orthogonal Manhattan frame (Candidates). Combining pairwise motion scatter with the learned Bingham prior defines an antipodal Bingham posterior (Criterion) that selects the joint axis, from which the hinge pivot and joint states are solved (Selected).

02Anchor to the Stationary Base

Unlike moving parts, the stationary base is unaffected by cross-frame depth bias and stays geometrically accurate across frames. In cabinets, appliances and furniture, flat faces meet at right angles, so joint axes align with the three orthogonal directions of the base, and hinge pivots lie on the boundary between part and base.

  1. Match parts relative to the baseReferencing every part query to a static-base query cancels handheld camera drift during dense 3D matching.
  2. Select instead of regressRestricting candidate axes to the base's Manhattan frame turns ill-conditioned continuous regression into a closed-form three-way selection, exact whenever the depth-induced tilt stays below 45°. Measured on frozen point maps, sliding motion compresses to 0.60–0.69 along the viewing ray across four backbones, tilting motion scatter by at most 14.5°.
  3. Solve the pivot linearlyGiven the selected axis, the hinge pivot follows from a 3×3 positive-definite linear system, so axis tilt no longer shifts the lever arm.

03Method

Architecture overview of PhyVGGT

Swipe horizontally to inspect the full architecture

Figure 3. Architecture overview of PhyVGGT. Given an uncalibrated monocular video, a frozen VGGT backbone extracts patch, per-frame and global tokens alongside 3D point maps (DPT) and tracking features (Track). The trainable Part and Motion Decoder fuses these streams and updates learnable base and part queries across N attention blocks to output part masks, correspondences and Bingham priors. The Kinematic Inference Layer fits a Manhattan frame to static normals, scores its three axes via a three-way softmax and solves for joint type, hinge pivot and unseen states via forward kinematics.

One block of the Part and Motion Decoder

Figure 4. One block of the Part and Motion Decoder.

Part and Motion Decoder. Per-frame self-attention in VGGT keeps sharp part edges, while deep global attention captures motion but smooths moving boundaries. The decoder fuses per-frame and global tokens from multiple VGGT layers, lets per-frame queries localise boundaries within each frame and clip queries attend across frames, and updates each part query from its difference to the static-base query. Static-base correspondences align all point maps on the base before rigid part motion is recovered.

Kinematic Inference Layer. Fourth-order alignment of static surface normals yields an orthogonal Manhattan frame even when two object dimensions are similar. A Bingham posterior combining motion scatter with the learned prior reduces to a closed-form softmax over the three frame axes, and a linear solve gives the hinge pivot used to drive unseen states.

04Results

Qualitative Results

Qualitative results

Swipe horizontally to inspect details

05Ablation & Backbone Transfer

Accuracy stratified by motion amplitude

Swipe horizontally to inspect details

Figure 5. Joint accuracy at 10° stratified by observed motion amplitude across four development benchmarks. Under small opening angles, learned Bingham priors disambiguate the three Manhattan axes, reaching 75.0% on 10°–20° synthetic rotations and 58.6% on small-motion D3D-HOI-dev clips against 41.7% and 21.8% for continuous Motion estimate.

06BibTeX

@article{shao2026phyvggt,
  title   = {{PhyVGGT}: Feed-Forward Articulated Object Reconstruction from Monocular Videos},
  author  = {Shao, Qi and Cao, Zhihao and Zhai, Shuhao and Nguyen, Anh and Liang, Hong and Huang, Baoru and Li, Liang},
  journal = {arXiv preprint arXiv:2611.22244},
  year    = {2026}
}

07Supplementary Material

The supplementary material provides the following details beyond the main paper.

S1Mathematical Derivations and Proofs

Fisher information of continuous joint estimation, the viewing-ray tilt bound with the Manhattan selection tolerance, and the all-pairs scatter identity with camera-drift cancellation.

S2Architecture, Training and Evaluation Protocol

One block of the Part and Motion Decoder, settings of the Kinematic Inference Layer, two-stage training on synthetic clips, and part matching and metrics for each benchmark.

S3Comprehensive Ablation and Diagnostic Studies

Additive design progression, inference, candidate-frame and same-backbone VGGT-1B comparisons, multi-layer token fusion, revolute pivot error, viewing-ray displacement measured in four frozen 3D foundation models, and accuracy by motion amplitude.

S4Additional Qualitative Results

More results on PartNet-Mobility synthetic test videos, and zero-shot results on real egocentric Arti4D and third-person D3D-HOI videos.