Robotics · Shuja Abidi

Mimic Human

How a robot copies a person

Copying a human motion takes two separate jobs. Retargeting works out what the motion should look like on the robot's body. Control makes the real robot perform it without falling. This page explains two ways to retarget, and why neither one is enough alone.

Personvideo or mocap Pose estimation3D human motion Retargetingarm IK, or GMR ControlRL, or MPC + WBC Robot sensors: the controller reacts to what really happens
Retargeting happens without physics. Control is where gravity, feet and motor limits come in.

The key idea: a person and a robot have different bodies, so you can't just copy joint angles or hand positions. And a motion that looks right on the robot can still be impossible for it to perform.

1

MediaPipe + arm IK

The simple route: from a single camera to upper-body motion, with no account or body model needed.

Google's MediaPipe Pose Landmarker finds 33 points on a person's body in each frame. It also gives their 3D positions in metres, measured from the midpoint of the hips. Arm inverse kinematics (IK) then finds the robot joint angles that point its arms the same way.

The method, step by step

  1. Read the pose. Each frame gives the shoulder s, elbow e and wrist w of each arm, plus the hips, in 3D.
  2. Smooth it. Single-camera depth is noisy, so a filter removes the jitter. One common choice is a Savitzky-Golay filter over about 0.2 seconds.
  3. Build a frame on the person's chest. This means it doesn't matter which way they face the camera:
    up = normalise( (L_shoulder + R_shoulder)/2 − (L_hip + R_hip)/2 ) left = normalise( L_shoulder − R_shoulder ), made perpendicular to up forward = left × up R = [ forward left up ]
  4. Measure arm directions, not positions. Directions carry over between bodies of different size:
    d_upper = Rᵀ · normalise(e − s) d_fore = Rᵀ · normalise(w − e)
  5. Scale to the robot's arm. Use the robot's own link lengths to get targets for its elbow and wrist:
    e* = s_robot + L_upper · d_upper w* = e* + L_fore · d_fore
  6. Solve the joint angles. Damped least squares moves the joints a little at a time until the elbow and wrist reach their targets. Here J is the Jacobian, which says how each joint moves the elbow and wrist:
    Δq = (JᵀJ + λI)⁻¹ Jᵀ err, err = [ e* − e(q) , w* − w(q) ] q ← clip( q + Δq, joint_min, joint_max )
    The damping λ keeps the solution stable when the arm is nearly straight, where the joints would otherwise jump. Starting each frame from the last answer keeps the motion continuous.
  7. Make it safe to send. Resample to the controller's rate, smooth the joint angles, and limit how far any joint can move per step, so one bad frame can't cause a sudden swing.

What it can and can't do

Strengths

  • Needs only an ordinary video, and runs on a normal computer
  • Needs no SMPL-X account, and no special motion capture
  • Simple enough to understand and debug fully
  • Easy to adapt to a new robot: it needs only the arm model

Limits

  • One camera judges depth poorly, especially when the person is side-on or an arm is hidden
  • Copies the arms, not the legs, walking or turning
  • Wrists are only partly known, and fingers need a separate hand model
  • Matches arm direction, not the exact hand position
Best for: gestures, waving, reaching and handshakes, on a robot that keeps standing or rolling. On a legged robot, the lower body still needs a controller that keeps it balanced while the arms move.

2

GMR: General Motion Retargeting

The whole-body route: maps a human's full motion onto many different humanoids.

GMR (Araújo, Ze and colleagues, MIT licence) takes a full-body human motion and solves for the robot's position, orientation and every joint angle, frame by frame, fast enough for live teleoperation. Its README lists more than 20 robots, including the Unitree G1, H1 and H2, Booster T1, and even a wheeled humanoid, the Galaxea R1 Pro.

Human motionSMPL-X, BVH, FBX Scaleper body part Offsetsalign each part IK stage 1rough pose IK stage 2refine positions Outputroot + joints
The steps as they appear in GMR's motion_retarget.py.

Step 1: Get a human motion

GMR doesn't work on video directly. It takes motion in SMPL-X format (a standard body model, used by datasets such as AMASS and OMOMO), BVH (LAFAN1, Nokov, Xsens) or FBX (OptiTrack). To start from a video, you first estimate the motion with a tool such as PromptHMR, as GentleHumanoid does. SMPL-X body models need their own registration, and carry their own licence terms.

Step 2: Scale the human to the robot

Each body part's position is scaled relative to the pelvis, so a tall person's motion fits a smaller robot:

p_scaled = s_part · (p_part − p_root) + s_root · p_root

GMR's G1 settings assume a 1.8 m person, with a scale of 0.9 for the pelvis, torso and legs and 0.8 for the arms. If you give the person's real height, all scales are adjusted by real height / 1.8.

Step 3: Say which robot part follows which human part

A table links each robot body to a human body, with a weight for position, a weight for rotation, and a fixed offset, because a robot's "zero pose" differs from a person's. GMR's G1 settings solve in two stages with different weights:

Robot partFollows humanStage 1: position / rotationStage 2: position / rotation
Pelvispelvis100 / 10100 / 5
Toesfeet100 / 10100 / 50
Hips, kneeships, knees0 / 1010 / 5
Torsoupper spine0 / 100 / 10
Shoulders, elbows, wriststhe same0 / 1010 / 5

Why two stages? Stage 1 first gets the overall pose right: limb rotations, plus the pelvis and feet where they should be. Stage 2 starts from that pose and adds positions for the other bodies, and also weights foot rotation more strongly. Starting from a good pose helps the solver avoid a poor answer. Note that pelvis and feet carry the highest position weights in both stages, because foot errors are the most harmful to walking.

Step 4: Solve inverse kinematics

GMR uses the mink library, which solves a small optimisation for joint velocities at each iteration:

v* = argmin Σ wᵢ ‖ Jᵢ v − errᵢ / Δt ‖² + λ ‖v‖² subject to joint limits (and optionally velocity limits) q ← q + v* Δt

Here errᵢ is how far robot part i is from its target, and wᵢ comes from the table above. GMR repeats this until the error stops falling by more than 0.001, up to a fixed number of times, then moves on to the next frame, starting from the last solution.

What comes out, and what doesn't

The output for each frame is the robot's root position, root orientation and joint angles. That is a kinematic motion: mass, gravity, friction and motor torque play no part in it. The GMR paper names the artefacts this can leave behind: foot sliding, self-penetration and physically infeasible motion. It also studies how these affect the policy that later tracks the motion, which is why retargeting quality matters even with a strong tracker.

Best for: full-body motion, including walking and dance, on a supported humanoid, usually as the reference for an RL tracking policy, or live as part of a teleoperation system.

3

Which to use

Choose by how much of the body you need to copy.

MediaPipe + arm IKPromptHMR + GMR
CopiesArmsWhole body
InputOrdinary videoVideo via PromptHMR; or SMPL-X, BVH, FBX directly
Human model33 landmark pointsFull body model with shape and joint rotations
What it matchesArm directionsPositions and rotations of main body parts
SetupSmall: only the arm model of your robotLarger: robot model, IK table and body model registration
New robotChange the arm links and limitsWrite a new IK table, unless GMR already lists the robot
Controller still neededYes, on legged robotsYes, on legged robots

4

Why control is required for quadrupeds and humanoids

Retargeting says where the joints should be. It can't make the body stay up.

Reason 1: no motor moves the body directly

The dynamics of a legged robot can be split into two parts:

Full body: M(q) q̈ + h(q, q̇) = Sᵀ τ + J_cᵀ f Joints (motors): M_a q̈ + h_a = τ + J_{c,a}ᵀ f Floating body: M_u q̈ + h_u = J_{c,u}ᵀ f ← no τ here

Here q holds the body's pose and the joint angles, τ is motor torque, and f is the force from the ground. Look at the last line. The six equations for the robot's overall position and orientation contain no motor torque at all. The body can only speed up or turn through the ground pushing back on the feet. The ground can only push up, not pull, and sideways force is limited by friction:

f_z ≥ 0 ‖f_xy‖ ≤ μ f_z

So a controller has to decide what the feet push with, and those forces must stay legal. A list of joint angles can't express that.

Reason 2: a standing robot is falling, unless something corrects it

A simple model of balance is the inverted pendulum, with a centre of mass at height z. With no correction, a small lean grows:

ẍ = (g / z) x → x(t) = x₀ cosh(t / T), T = √(z / g)

For a humanoid with its centre of mass at z = 0.8 m, T ≈ 0.29 s. Starting from rest, a 1 cm lean becomes about 3 cm after 0.5 s and about 17 cm after 1 s. Small errors in joint angles, timing or the floor are enough to start this, so the robot has to measure and correct itself constantly.

Reason 3: the reference may be physically impossible

Because retargeting ignores physics, its motion can ask for things the robot can't do:

A controller has to bend the motion so it's possible, and keep balance as the priority over matching the pose exactly.

Reason 4: joint position control only makes joints follow

The low-level motor controller is usually a simple spring and damper at each joint:

τ = K_p (q* − q) − K_d q̇

It pulls each joint towards its target, one joint at a time, and knows nothing about balance. If you just send retargeted angles to it, the joints may follow, yet the robot will still tip over, since nothing is watching the whole body.

Quadrupeds and humanoids need control for different reasons

QuadrupedHumanoidWheeled humanoid, such as Pepper
Support areaWide when several feet are down; only a line in a trotOne or two small feetWide and fairly constant
Standing stillStable with no effort to balance, if the joints holdNeeds constant small correctionsStable
While movingNeeds control: during a trot, only two feet are down at a timeNeeds control: balance is the main problemMostly limits on speed and lean
Hardest part of copying motionThere's no natural source of motion: human motion doesn't map to four legs, so animal data or animation is usedKeeping balance while copying the whole bodyStaying natural and safe near people
If the controller failsOften stumbles, and low height reduces the fallFalls from a greater height, within a fraction of a secondUsually stops, or tips only in extreme cases
Typical joints to coordinateAbout 1220 to over 40Mainly arms and head

What can do this job?

RL tracking policy

A network trained in simulation to follow reference motion while balancing. This is the most common choice today for following human motion. BeyondMimic and GentleHumanoid work this way.

MPC + whole-body control

Optimisation with a physical model: plan ground forces, then compute joint torques. The limits are explicit. See Robot Control.

Simple balance feedback

Enough for a robot that stands or rolls while its arms move. Not enough for dance, jumps or walking copied from a person.