From simulated motion to a physical robot

1 · What MuJoCo does

MuJoCo is a physics engine. It receives a model of rigid bodies, joints, masses, inertias, actuators, sensors, contacts, and the ground. At every small time step it solves the equations of motion and contact constraints to estimate the next positions and velocities. The website’s recording only replays those saved states; MuJoCo itself runs on the student’s computer.

The model is more than visible geometry. Meshes determine appearance and sometimes collision shape, while the MJCF description supplies the body hierarchy, joint axes, limits, mass distribution, friction, actuator behavior, and simulation settings. A visually correct model can still produce incorrect dynamics if any of these values are wrong.

Two clocks

The physics step can be 0.005 seconds while the policy acts every four steps. The simulator therefore updates contact and dynamics at 200 Hz, and the policy receives a new observation and action at 50 Hz. This separation lets contact evolve smoothly without requiring neural-network inference at every physics step.

What MuJoCo cannot prove

A simulation result is conditional on its model. It cannot establish that the real floor has the same friction, that a motor produces the modeled torque, or that sensor timing matches the simulated timing. Simulation is a controlled experiment and a training environment—not automatic evidence of hardware safety.

2 · Turn walking into a reinforcement-learning problem

Reinforcement learning needs four definitions. The observation is what the policy can measure. The action is what it can request. The reward scores each transition. The termination rule decides when an episode ends.

The documented Microduck movement interface contains 61 observation values: 3 gyroscope values, 3 projected-gravity values, 14 joint-position offsets, 14 joint velocities, 14 previous actions, and 13 command values. The policy returns 14 actions, one for each learned joint target. The mouth motor is outside this locomotion action vector.

An action is not torque delivered instantly. The runtime scales the policy output, adds the home pose, and passes target positions through actuator and safety logic. Motor response, friction, voltage, load, and delay determine the resulting motion.

Reward design is task design

A useful walking reward can combine command tracking with posture, foot clearance, gait timing, smooth actions, limited slip, and collision penalties. The weights encode trade-offs. If standing is rewarded too strongly, the policy may refuse to walk. If forward speed dominates, it may move quickly while drifting or falling. Reward is a teaching signal, not the final acceptance test.

3 · How PPO improves the policy

Microduck uses an on-policy algorithm called Proximal Policy Optimization, or PPO. The actor network maps the current observation to an action distribution. A critic network estimates the future accumulated reward from the current state. During a rollout, many simulated robots use the current actor to collect transitions. PPO then updates the actor and critic from that fresh experience.

The policy gradient asks: which sampled actions produced better outcomes than the critic expected? The difference is called an advantage. Positive advantage increases the probability of similar actions; negative advantage decreases it. PPO clips the probability-ratio update so one batch cannot change the policy too abruptly. Clipping improves stability, but it does not repair a bad reward, missing observation, or inaccurate simulator.

With 4,096 environments and 24 steps per environment, one rollout contains 98,304 transitions. These transitions are efficient to collect in parallel, but they are not 98,304 independent walking experiments: neighboring steps from one trajectory are correlated.

Why the critic is not the deployed controller

The critic helps estimate advantages during training. Deployment normally needs the actor only. Export must preserve any observation normalization used by the actor; otherwise the same physical state produces numerically different inputs after training.

4 · The training loop, step by step

1. Reset. Place each simulated robot near a valid starting pose and sample task conditions.
2. Observe. Read inertial quantities, joint states, the previous action, and the requested motion.
3. Act. Evaluate the actor and convert its 14 outputs into joint targets.
4. Simulate. Advance MuJoCo through several physics steps, including motor and contact dynamics.
5. Score. Calculate reward terms and record termination conditions.
6. Update. After the rollout, estimate advantages and optimize the PPO objectives for several minibatches.
7. Repeat. Continue with the updated actor, periodically saving checkpoints and running separate evaluations.

A rising training reward says that the policy is improving against the chosen reward under sampled training conditions. It does not by itself say that the robot walks straight, turns correctly, stops, survives unseen conditions, or transfers to hardware.

Domain randomization

The released task can vary center of mass, mass and inertia, joint friction, armature, IMU orientation, encoder bias, and external velocity pushes. Randomization teaches the policy not to depend on one exact simulator. Ranges must remain physically plausible: excessive variation can produce a permanently defensive gait, while insufficient variation lets the policy exploit a single ideal model.

5 · Evaluate behavior, not just reward

Freeze a checkpoint before evaluation. Run repeated stand, forward, turn, and stop trials using defined commands and initial conditions. Measure upright time, falls, forward-speed error, angular-speed error, heading deviation, lateral drift, foot lift, and residual speed after a stop command. Preserve the policy checksum, robot model revision, evaluator revision, seed, and raw recording.

Training tests guide design. Confirmation tests estimate performance after choices are frozen. If a result is repeatedly used to select checkpoints or tune rewards, it is no longer independent confirmation evidence.

The public viewer is deliberately labeled “recorded simulation.” Its camera and timeline are interactive, but the motion has already been calculated. A smooth replay shows what one saved trial looked like; it does not establish reliability across trials.

6 · Sim-to-real: transfer one assumption at a time

Sim-to-real is the process of taking a policy developed in simulation and making it work on the physical robot. The main difficulty is the reality gap: differences between simulated and real geometry, mass, friction, motor response, sensing, timing, contacts, and disturbances.

Stage A · Match the interface

First verify the 61 input meanings, ordering, units, signs, reference frames, home pose, 14 output ordering, and action scale. A model can load successfully while controlling the wrong joint. Export the actor to ONNX with normalization and metadata, then test finite inputs and outputs before connecting hardware.

Stage B · Match dynamics and timing

Measure motor position response under safe loads, bus and inference delay, encoder bias, IMU mounting offset, joint friction, battery-dependent performance, and contact behavior. Compare real and simulated trajectories under the same small command. Change one model assumption at a time and retain both traces.

Stage C · Train for bounded uncertainty

Randomize only parameters whose real uncertainty is understood. Add realistic observation noise, delay, actuator limits, disturbances, and mass variation. Validate that the policy still succeeds at the nominal model; robustness should not destroy the core behavior.

Stage D · Escalate hardware tests safely

Begin with model-interface checks, then supported or restrained low-amplitude motion, standing, short steps, low-speed walking, turning, stopping, and finally longer trials. Define abort conditions for tilt, communication loss, excessive target change, current, temperature, or instability. Keep a human-controlled stop available.

Stage E · Decide from evidence

Compare requested motion with measured motion over repeated trials. Report failures as well as successful videos. A policy is ready for a broader test only when its acceptance criteria pass under the intended conditions; simulation success alone never grants that conclusion.