Mimic Human
How a robot copies a person
Copying a human motion takes two separate jobs. Retargeting works out what the motion should look like on the robot's body. Control makes the real robot perform it without falling. This page explains two ways to retarget, and why neither one is enough alone.
The key idea: a person and a robot have different bodies, so you can't just copy joint angles or hand positions. And a motion that looks right on the robot can still be impossible for it to perform.
1
MediaPipe + arm IK
The simple route: from a single camera to upper-body motion, with no account or body model needed.
Google's MediaPipe Pose Landmarker finds 33 points on a person's body in each frame. It also gives their 3D positions in metres, measured from the midpoint of the hips. Arm inverse kinematics (IK) then finds the robot joint angles that point its arms the same way.
The method, step by step
- Read the pose. Each frame gives the shoulder
s, elboweand wristwof each arm, plus the hips, in 3D. - Smooth it. Single-camera depth is noisy, so a filter removes the jitter. One common choice is a Savitzky-Golay filter over about 0.2 seconds.
- Build a frame on the person's chest. This means it doesn't matter which way they face the camera:
up = normalise( (L_shoulder + R_shoulder)/2 − (L_hip + R_hip)/2 ) left = normalise( L_shoulder − R_shoulder ), made perpendicular to up forward = left × up R = [ forward left up ]
- Measure arm directions, not positions. Directions carry over between bodies of different size:
d_upper = Rᵀ · normalise(e − s) d_fore = Rᵀ · normalise(w − e)
- Scale to the robot's arm. Use the robot's own link lengths to get targets for its elbow and wrist:
e* = s_robot + L_upper · d_upper w* = e* + L_fore · d_fore
- Solve the joint angles. Damped least squares moves the joints a little at a time until the elbow and wrist reach their targets. Here
Jis the Jacobian, which says how each joint moves the elbow and wrist:Δq = (JᵀJ + λI)⁻¹ Jᵀ err, err = [ e* − e(q) , w* − w(q) ] q ← clip( q + Δq, joint_min, joint_max )The dampingλkeeps the solution stable when the arm is nearly straight, where the joints would otherwise jump. Starting each frame from the last answer keeps the motion continuous. - Make it safe to send. Resample to the controller's rate, smooth the joint angles, and limit how far any joint can move per step, so one bad frame can't cause a sudden swing.
What it can and can't do
Strengths
- Needs only an ordinary video, and runs on a normal computer
- Needs no SMPL-X account, and no special motion capture
- Simple enough to understand and debug fully
- Easy to adapt to a new robot: it needs only the arm model
Limits
- One camera judges depth poorly, especially when the person is side-on or an arm is hidden
- Copies the arms, not the legs, walking or turning
- Wrists are only partly known, and fingers need a separate hand model
- Matches arm direction, not the exact hand position
2
GMR: General Motion Retargeting
The whole-body route: maps a human's full motion onto many different humanoids.
GMR (Araújo, Ze and colleagues, MIT licence) takes a full-body human motion and solves for the robot's position, orientation and every joint angle, frame by frame, fast enough for live teleoperation. Its README lists more than 20 robots, including the Unitree G1, H1 and H2, Booster T1, and even a wheeled humanoid, the Galaxea R1 Pro.
motion_retarget.py.Step 1: Get a human motion
GMR doesn't work on video directly. It takes motion in SMPL-X format (a standard body model, used by datasets such as AMASS and OMOMO), BVH (LAFAN1, Nokov, Xsens) or FBX (OptiTrack). To start from a video, you first estimate the motion with a tool such as PromptHMR, as GentleHumanoid does. SMPL-X body models need their own registration, and carry their own licence terms.
Step 2: Scale the human to the robot
Each body part's position is scaled relative to the pelvis, so a tall person's motion fits a smaller robot:
GMR's G1 settings assume a 1.8 m person, with a scale of 0.9 for the pelvis, torso and legs and 0.8 for the arms. If you give the person's real height, all scales are adjusted by real height / 1.8.
Step 3: Say which robot part follows which human part
A table links each robot body to a human body, with a weight for position, a weight for rotation, and a fixed offset, because a robot's "zero pose" differs from a person's. GMR's G1 settings solve in two stages with different weights:
| Robot part | Follows human | Stage 1: position / rotation | Stage 2: position / rotation |
|---|---|---|---|
| Pelvis | pelvis | 100 / 10 | 100 / 5 |
| Toes | feet | 100 / 10 | 100 / 50 |
| Hips, knees | hips, knees | 0 / 10 | 10 / 5 |
| Torso | upper spine | 0 / 10 | 0 / 10 |
| Shoulders, elbows, wrists | the same | 0 / 10 | 10 / 5 |
Why two stages? Stage 1 first gets the overall pose right: limb rotations, plus the pelvis and feet where they should be. Stage 2 starts from that pose and adds positions for the other bodies, and also weights foot rotation more strongly. Starting from a good pose helps the solver avoid a poor answer. Note that pelvis and feet carry the highest position weights in both stages, because foot errors are the most harmful to walking.
Step 4: Solve inverse kinematics
GMR uses the mink library, which solves a small optimisation for joint velocities at each iteration:
Here errᵢ is how far robot part i is from its target, and wᵢ comes from the table above. GMR repeats this until the error stops falling by more than 0.001, up to a fixed number of times, then moves on to the next frame, starting from the last solution.
What comes out, and what doesn't
The output for each frame is the robot's root position, root orientation and joint angles. That is a kinematic motion: mass, gravity, friction and motor torque play no part in it. The GMR paper names the artefacts this can leave behind: foot sliding, self-penetration and physically infeasible motion. It also studies how these affect the policy that later tracks the motion, which is why retargeting quality matters even with a strong tracker.
3
Which to use
Choose by how much of the body you need to copy.
| MediaPipe + arm IK | PromptHMR + GMR | |
|---|---|---|
| Copies | Arms | Whole body |
| Input | Ordinary video | Video via PromptHMR; or SMPL-X, BVH, FBX directly |
| Human model | 33 landmark points | Full body model with shape and joint rotations |
| What it matches | Arm directions | Positions and rotations of main body parts |
| Setup | Small: only the arm model of your robot | Larger: robot model, IK table and body model registration |
| New robot | Change the arm links and limits | Write a new IK table, unless GMR already lists the robot |
| Controller still needed | Yes, on legged robots | Yes, on legged robots |
4
Why control is required for quadrupeds and humanoids
Retargeting says where the joints should be. It can't make the body stay up.
Reason 1: no motor moves the body directly
The dynamics of a legged robot can be split into two parts:
Here q holds the body's pose and the joint angles, τ is motor torque, and f is the force from the ground. Look at the last line. The six equations for the robot's overall position and orientation contain no motor torque at all. The body can only speed up or turn through the ground pushing back on the feet. The ground can only push up, not pull, and sideways force is limited by friction:
So a controller has to decide what the feet push with, and those forces must stay legal. A list of joint angles can't express that.
Reason 2: a standing robot is falling, unless something corrects it
A simple model of balance is the inverted pendulum, with a centre of mass at height z. With no correction, a small lean grows:
For a humanoid with its centre of mass at z = 0.8 m, T ≈ 0.29 s. Starting from rest, a 1 cm lean becomes about 3 cm after 0.5 s and about 17 cm after 1 s. Small errors in joint angles, timing or the floor are enough to start this, so the robot has to measure and correct itself constantly.
Reason 3: the reference may be physically impossible
Because retargeting ignores physics, its motion can ask for things the robot can't do:
- Feet slide along the floor, when real feet would need friction to allow it.
- Balance is wrong, because a person's weight is distributed differently from the robot's.
- Motors are too weak or too slow for a fast human movement.
- The body collides with itself, because the robot's limbs are shaped differently.
A controller has to bend the motion so it's possible, and keep balance as the priority over matching the pose exactly.
Reason 4: joint position control only makes joints follow
The low-level motor controller is usually a simple spring and damper at each joint:
It pulls each joint towards its target, one joint at a time, and knows nothing about balance. If you just send retargeted angles to it, the joints may follow, yet the robot will still tip over, since nothing is watching the whole body.
Quadrupeds and humanoids need control for different reasons
| Quadruped | Humanoid | Wheeled humanoid, such as Pepper | |
|---|---|---|---|
| Support area | Wide when several feet are down; only a line in a trot | One or two small feet | Wide and fairly constant |
| Standing still | Stable with no effort to balance, if the joints hold | Needs constant small corrections | Stable |
| While moving | Needs control: during a trot, only two feet are down at a time | Needs control: balance is the main problem | Mostly limits on speed and lean |
| Hardest part of copying motion | There's no natural source of motion: human motion doesn't map to four legs, so animal data or animation is used | Keeping balance while copying the whole body | Staying natural and safe near people |
| If the controller fails | Often stumbles, and low height reduces the fall | Falls from a greater height, within a fraction of a second | Usually stops, or tips only in extreme cases |
| Typical joints to coordinate | About 12 | 20 to over 40 | Mainly arms and head |
What can do this job?
RL tracking policy
A network trained in simulation to follow reference motion while balancing. This is the most common choice today for following human motion. BeyondMimic and GentleHumanoid work this way.
MPC + whole-body control
Optimisation with a physical model: plan ground forces, then compute joint torques. The limits are explicit. See Robot Control.
Simple balance feedback
Enough for a robot that stands or rolls while its arms move. Not enough for dance, jumps or walking copied from a person.