ViT-H · Method Overview

Back to project ↗
Explore the generation-and-tracking process.

Humanoid Motion
Generation from Videos

Input
video
PRE: video
Video condition set
Camera
orientation
BBox &
keypoints
Image
feature
PRE: video
PRE: video
PRE: bbox and keypoints: annotated frame 2
Diffusion-based
motion generator
Direct
G1 motion
prediction
PRE: output motion
Preceding
segment
(AR generator
only)

Post-Training with Motion Tracking Feedback

Motion generation
Motion
generator
PRE: video
PRE: video
PRE: video
B
A
Candidates
POST: generated candidate 1: time 0POST: generated candidate 1: time 1POST: generated candidate 1: time 2
POST: generated candidate 0: time 0POST: generated candidate 0: time 1POST: generated candidate 0: time 2
Motion tracking
A
B
Tracking targets
Deployable tracker
POST: deployable rollout 0: time 0
POST: deployable rollout 0: time 1
POST: deployable rollout 0: time 2
POST: deployable rollout 1: time 0
POST: deployable rollout 1: time 1
POST: deployable rollout 1: time 2
A
B
Privileged tracker
POST: global rollout 0: time 0
POST: global rollout 0: time 1
POST: global rollout 0: time 2
POST: global rollout 1: time 0
POST: global rollout 1: time 1
POST: global rollout 1: time 2
R
local
R
succ
R
global
R
succ
Preference optimization
More
likely
A
B
Less
likely
POST: DPO candidate 0: time 0POST: DPO candidate 0: time 1POST: DPO candidate 0: time 2
POST: DPO candidate 1: time 0POST: DPO candidate 1: time 1POST: DPO candidate 1: time 2
DPO update

Real-World Deployment

Offline generation · Video-to-Humanoid
OFFLINE: input
Motion
tracker
Motion
generator
OFFLINE: robot
Online generation · Video teleoperation
ONLINE MOTION: input
ONLINE OBJECT: input
Locomotion teleoperation
Loco-manipulation teleoperation
Motion
tracker
AR motion
generator
Motion
tracker
AR motion
generator
ONLINE MOTION: robot
ONLINE OBJECT: robot
Preceding segment
Preceding segment