How do you keep a robot arm safe, when safe depends on what it's holding?

A cup of water mustn't pass over a laptop, even with plenty of air to spare. A dry sponge can. This page explains, step by step, the maths that makes a robot arm behave this way. You can drag and change the diagrams as you read.

It's what I learned rebuilding, in simulation, the method of Brunke et al., "Semantically Safe Robot Manipulation" (IEEE Robotics and Automation Letters, 2025).

The big pictureA filter between the controller and the robot

A safety filter doesn't plan the motion. Something else, such as a person, a learned policy or a simple controller, asks for a motion. Many times a second, the filter checks that request and changes it as little as possible to keep the robot safe.

Goal Controller Safety filter Robot arm Safety rules: h(x) u_cmdu_cert where the arm is now
The controller knows nothing about safety. u_cmd is what it asks for, and u_cert is the safe version the robot actually gets. In this project the loop runs 45 times a second.

Three questions need answering, and the rest of this page answers them in turn:

  1. How does a robot know where the unsafe places are? Boundaries, superquadrics and semantic regions.
  2. How does it choose the closest safe motion? The approach limit and QP.
  3. How does that reach the arm's 7 joints? Inverse kinematics and the Jacobian.

Step 1A boundary is where a function crosses zero

Give the filter a function h(x) that takes a position x and returns one number:

h(x)Meaning
h > 0Safe. Larger usually means further away
h = 0Exactly on the boundary
h < 0Inside the unsafe area

The simplest example is a circle with centre c and radius r:

h(x) = |x − c|² / r² − 1

The function has a second gift: its gradient, a = ∂h/∂x. That's an arrow pointing the way h rises fastest, which is the quickest way out. So h says how safe the robot is, and a says which direction is safer. The filter needs both.

h isn't a distance. It has no units, and h = 1 near a small object is a different distance from h = 1 near a large one. It only has to be smooth, and zero on the boundary.

Step 2Superquadrics: one formula for many shapes

Real objects aren't circles. A superquadric is a single formula that can be round, box-like or anything between, and it stays smooth, so it still has a gradient everywhere.

g(x, y, z) = ( |x/ax|2/e₂ + |y/ay|2/e₂ )e₂/e₁ + |z/az|2/e₁,   h = g − 1

Change the roundness

Slide the roundness, and drag the dot. The dashed rectangle is the true object, and the blue curve is the superquadric, seen as a slice through its middle.
Boundary, h = 0Gradient: the quickest way out

Fitting one to an object

Untick Grow to contain the corners above, and you'll see the problem: a superquadric cuts off a box's corners. At a corner of a 3-D box, g = 3, so it lies outside. Growing every axis by a factor of 3e/2 brings the corner back to g = 1. For the 2-D slice above, the factor is 2e/2.

centre = (low + high) / 2,    a = (high − low) / 2 × 3e/2

Here, low and high are the corners of the object's bounding box. An object's points could come from a camera, as in the paper, but in my work they're generated. The paper cites a separate shape-recovery method, and I use this simpler bounding-box fit.

An L-shaped object doesn't fit one convex shape well, so it's better described as a union, with one shape per part.

Step 3The approach limit: slow down near the boundary

This is the key line of the whole method. It's called a control barrier function condition:

ḣ ≥ −α(h)

ḣ is how fast the safety value is changing. If it's negative, the robot is getting closer. The rule limits how quickly that can happen, depending on how close it already is:

The paper uses α(h) = h|h|, with a quarter of that for caution. In this project, a filter is cautious around anything with a semantic rule, such as a laptop when holding water.

The two settings

Move along the chart to read the limit at any h. Cautious allows a quarter of the normal approach speed everywhere, so the robot starts to slow down further away.
Normal: α = h²Cautious: α = h²/4

Because h depends on position, and position changes with velocity u, the rule becomes a limit on the velocity itself:

ḣ = a · u    so the rule is:    a · u ≥ −α(h)

That's a straight line through the space of possible velocities: one side allowed, the other not. Which leads to QP.

Step 4QP: the smallest safe change to the command

A quadratic program (QP) finds the lowest point of a bowl-shaped cost, while obeying straight-line limits. It fits this job exactly: the cost measures how much the command is changed, and the limits are the safety rules.

u_cert = argminu |u − u_cmd|²    subject to   ai · u ≥ −α(hi) one rule per unsafe shape
                                             |u| ≤ speed limit

Watch the filter decide

Drag the robot towards the circle, or move the goal. On the right, the filter's whole choice is visible: if the orange command falls in the red part, it's moved the shortest way to the edge.
Command, u_cmdSafe velocity, u_cert

With one rule, this has a neat exact answer. With many rules, you need a solver. Each rule adds a line, and the answer is the allowed point closest to the command.

Step 5The trap: safe isn't the same as useful

This was the most surprising thing I found. The filter only corrects each command: it has no plan. So if the goal is straight behind a flat-faced shape, the robot can stop there and never arrive, still perfectly safe.

Trap the robot, then free it

The laptop is 0.30 × 0.44 m, and the red line is the filter's shape around it. Slide the roundness up, or drag the goal sideways, until the blue path gets through.
No filterWith the filter

Why it happens: near a flat face, the gradient points straight out and hardly changes. The command pulls towards the middle of the face, and nothing pushes the robot sideways. A curved shape keeps turning the robot away, so it slides past. This means a tighter shape isn't simply better. It blocks less space, but it can stop the robot.

This comes from my simple controller, which steers straight at the goal. A person or a planner might steer around. The paper doesn't report this behaviour, and I can't say whether it happens on its robot.

Step 6Semantic regions: unsafe because of what you're doing

Everything so far protects objects. Semantic safety also protects places that are unsafe because of the task. The air above a laptop is empty, and a collision check sees nothing wrong with it. But it's unsafe if you're carrying water.

The trick from the paper is neat: make the region from the object's own points. For "above", copy the points past the top of the workspace, then fit one superquadric around both sets. The filter's maths doesn't change at all. It just gets one more shape.

Same scene, different rules

Choose what the gripper holds. This is a simplified side view, and the red areas are what becomes unsafe.
RelationshipHow the region is madeExample
AboveCopy the points past the top of the workspaceWater over a laptop
UnderCopy the points below the work surfaceA flame under a balloon
AroundGrow the box sideways by a margin (10 cm is my choice)A flame near a paper towel

This is why a language model fits so well in the paper's design. It only has to list which rule applies to which object, and the filter stays the same. In my version, those rules are typed by hand. The paper has 12 relationships, and I've tried these 3.

Step 7From a point to the Franka arm: inverse kinematics

So far, the robot has been a point that moves as told. A real arm moves its joints, and the safety rules are about where its tool is. Forward kinematics goes from joint angles to tool position. Inverse kinematics (IK) goes the other way, and it's the harder direction: for the Franka FR3 there are 7 joints but only 3 numbers in a position, so endless joint poses reach the same point.

The Franka FR3 arm with its seven joints and tool point labelled
The Franka FR3 with its 7 rotating joints, rendered in MuJoCo. Each red line is the axis that joint turns around. The tool point is between the fingertips.

The Jacobian connects the two

Rather than solving for a final pose, the arm works in small steps, at the level of speeds. The Jacobian J(q) is a table with 3 rows and 7 columns. Each column says how fast the tool moves in x, y and z when that joint turns at 1 rad/s, in the current pose.

ẋ = J(q) u tool velocity = Jacobian × joint speeds

For a rotating joint, each column has a simple meaning, which is axis × (tool − joint position). MuJoCo calculates this directly from the arm's model. The table changes as the arm moves, so it's worked out again at every step.

Differential IK: going backwards

The controller decides a tool velocity, heading to the goal: ẋ = 2 (goal − tool), at no more than 0.3 m/s. Going back to joint speeds needs an inverse of J, which isn't square. So I use the damped pseudo-inverse:

J⁺ = Jᵀ (J Jᵀ + λ²I)⁻¹     u_cmd = J⁺ ẋ + (I − J⁺J) · k (q_home − q)

A small arm with a spare joint

Drag the green circle, or click anywhere. The real Franka has 7 joints for a 3-D position. This 3-joint arm in a 2-D plane has the same kind of spare motion, using the same formula.
Try this: park the goal, then hold a spare joint button. The elbow moves, while the tool barely shifts: that's the null space. Then drag the goal to the edge of the dashed circle, and raise the damping to see the tool lag behind.

Moving the safety rules to the joints

Here's why the Jacobian matters for safety. The rules are about the tool's position, and the filter chooses joint speeds. The chain rule connects them:

ḣ = a · ẋ = a · J(q) u    so the rule is:    a · J(q) u ≥ −α(h)

It's the same straight-line rule as Step 4, now across 7 numbers instead of 2. The QP doesn't care: it's still a bowl with straight-line limits. Joint limits go in the same way, with each joint getting its own barrier 0.05 rad inside its limit.

# The Python version of the rule for each shape
h, a = rule.shape.value_and_gradient(tool_position)
row = a @ J                                   # a · J(q)
limit = rule.caution * h * abs(h)             # alpha(h)
# Then the solver finds u closest to u_cmd, with row · u >= -limit

Step 8Protecting the whole arm, and keeping the cup upright

Rules on one point leave the rest of the arm unprotected. In my test, a tool-only filter let the arm's body pass 8.7 cm into a pillar while it saw no problem.

The arm covered by twenty translucent blue spheres
20 spheres, fitted automatically to the arm's collision meshes. They average 8 cm in radius, so they're cautious.

Spheres: a sphere is the easiest shape to check. The trick is to grow each object by the sphere's radius, plus 1 cm, then treat the sphere's centre as a point. The same rule then applies, using that sphere's own Jacobian.

Self-collision: the arm can hit itself, so pairs of spheres at least 3 links apart get a rule to stay apart.

The counter: spheres also stay above the work surface.

Keeping the cup upright

Staying out of a region says nothing about tilt. The paper adds a rotation penalty to the QP's cost, where e is the rotation needed to get back to the starting orientation, and Jr is the rotation part of the Jacobian:

|u − u_cmd|² + w₁ |e − Jr u Δt|² + w₂ |Jr u Δt|²

It's a soft cost, not a hard rule, so it doesn't guarantee a tilt limit. One thing I had to add: with this penalty and a position-only command, the arm stalled. The command asked for rotation, and the penalty removed it, so the two fought. Using a 6-D command that also asks to hold orientation fixed it. The paper doesn't say how its commands treat orientation, so this may differ from its setup.

All togetherThe full filter

Every piece above is one line of this problem, solved 45 times a second:

u_cert = argmin |u − u_cmd|² + w₁|e − JruΔt|² + w₂|JruΔt|² smallest change, stay upright subject to a · Jtool u ≥ −α(h) for each semantic region tool stays out a · Jsphere u ≥ −α(h) for each sphere and object body stays clear floor and self-collision rules no hitting the counter or itself joint limits and joint speed limits what the arm can do
IdeaPlain meaningWhere it appears
Barrier function hOne number for how safe a position isEach rule
SuperquadricA smooth shape around an object or regionGives h and a
Approach limit α(h)Slow down near the boundaryRight side of each rule
QPSmallest change obeying every ruleThe whole problem
Jacobian JJoint speeds to tool speedLeft side of each rule
Differential IKTool velocity to joint speedsMakes u_cmd
Semantic ruleWhich regions apply, for what's heldChooses the rules
What this isn't: it's a partial reproduction in simulation. There's no real robot, camera, language model or person here, and the rules and object shapes are simplified. The paper's own results come from a real arm.