How a walking policy is learned

Learning is repeated trial and feedback

In reinforcement learning, a policy acts in an environment and receives a numerical reward. A training algorithm changes the policy so future behavior tends to obtain more reward under the training conditions. An episode is one run between reset and termination. Many simulated environments can supply experience in parallel. The resulting exported policy is then used for inference.

A reward is a design choice. One term can favor following a commanded speed while another discourages large actions or undesirable contact. The algorithm optimizes the signals provided. It does not automatically understand that the machine should be mechanically gentle, easy to repair or attractive to a person unless the setup captures those goals adequately.

Changing a reward setting changes the instructions for a future training run. It does not rewrite an already exported policy file such as new.onnx. To produce a policy influenced by the new reward, you must run a compatible training process, evaluate the result and export the new learned computation. Restarting a viewer only reloads the existing file.

Read the environment as a specification

Open the pinned velocity_env_cfg.py. Separate observations, actions, commands, rewards, events and terminations. Observations define available information; actions define influence; commands define the requested task. Rewards score behavior, events vary conditions, and terminations decide when an episode ends.

Find physics timestep 0.005 and decimation 4. A deployment loop with different observation scaling or timing is a different environment even when it loads the same network. The current configuration also does not prove how every historical policy export was trained. Keep exact source and export provenance together.

Randomization and evaluation do different jobs

Domain randomization varies modeled quantities such as friction, mass, gains or delay during training. Its purpose is to represent uncertainty. Arbitrary ranges can also create unrealistic or impossible tasks. Choose ranges from measurements or explicit uncertainty estimates.

Evaluation asks how the resulting controller behaves in cases you care about. Keep some conditions separate from training decisions. Measure tracking, falls, recovery and motion limits alongside reward. A high score can hide a poor physical tradeoff. One successful animation is one trial, not a distribution of results.

Choose the next investigation deliberately

First verify the input contract and reproduce the known simulation baseline. Then investigate calibration and model differences. A reversed encoder sign is an integration problem; training longer is not its first repair. A substantially different mass or actuator response motivates a model update and reevaluation before a training decision.

This course does not require full BDX-R training. The Mac already supports the demonstrated inference path. Current mjlab guidance describes limited CPU support on macOS and recommends Linux/NVIDIA for substantial training. BDX-R’s older dependency pin needs its own compatibility check before any upgrade. Keep optional toy training in a separate environment.