Robot Policies: From π(a|o) to Foundation Models

Robot learning is easier to understand once the word policy stops feeling mystical.

A policy is the model inside the physical loop: it looks at observations, chooses actions, the world changes, and then it has to do it again on the next tick. That sounds close to autoregressive generation until you remember the next token is computed by physics, not by the model.

This guide is written for readers who already know modern generative AI and want the robotics map: where PPO still matters, why locomotion and manipulation live in different regimes, what VLAs actually output, why action chunking keeps showing up, why deformable objects are their own problem class, and how the Unitree G1/MuJoCo code paths make the abstractions concrete.

The piece below folds eight research documents into one public research-log article. The evidence tags stay visible because the field is noisy and the distinction between verified, sourced, and background claims matters.

Robot Policies: From π(a|o) to Foundation Models

Guide: Robot Policies — from π(a|o) to Foundation Models Prepared: July 2026, from a multi-agent deep-research pass (105 agents, 23 sources fetched, 115 claims extracted, 25 adversarially verified by 3-vote panels) plus curated background knowledge. Audience: A reader fluent in modern generative AI — VLAs (π0/π0.5, OpenVLA, RT-2), world models (GR00T, Cosmos), LLMs, diffusion models (Stable Diffusion), and flow-matching models (E2-TTS) — who has not studied robot learning. Every part builds bridges from that background.

How to read this guide: the evidence legend

Robotics moves fast and is full of confidently repeated folklore. Every substantive claim in this guide is tagged:

Tag	Meaning
[V]	Verified. Survived a 3-vote adversarial verification panel (independent skeptic agents each tried to refute it against primary sources). Highest confidence.
[S]	Sourced. Extracted from a primary paper or lab publication with a verbatim quote, but not run through the adversarial panel (verification budget). Cite-checked once.
[K]	Background knowledge. Standard, widely-taught material (textbook RL, famous papers) reproduced from the author's training knowledge. Reliable for fundamentals, but re-verify any specific number before quoting in your own work.

Two claims were refuted during verification and are deliberately absent or corrected in this guide:

~~"A direct-torque policy must itself be evaluated at ~1 kHz at runtime"~~ — refuted 1-2. The locomotion survey argues PD-target policies permit low policy rates while torque is produced at 1 kHz, and that torque policies push toward higher evaluation rates, but "must run at 1 kHz" overstates what deployed torque-policy papers actually do.
~~"Diffusion Policy pioneered conditional denoising for control"~~ — refuted 0-3 on the priority claim (e.g., Janner et al.'s Diffuser and other diffusion-for-decision-making work predates it). The correct, verbatim-supported statement is that Diffusion Policy introduced "a new way of generating robot behavior by representing a robot's visuomotor policy as a conditional denoising diffusion process" and made it work on real robots — the analogy to Stable Diffusion is pedagogically sound; the priority claim is not.

Guide map

#	Part	Core question
01	What Is a Robot Policy?	The object itself: π(a\|o), how it differs from controllers, planners, and world models; observation & action spaces; the control-frequency hierarchy.
02	The Taxonomy: How Policies Are Trained	Model-free RL, model-based RL, behavior cloning, diffusion/flow policies, adversarial imitation (GAIL/AMP), motion tracking (DeepMimic), teacher–student distillation, sim-to-real.
03	Locomotion: The "Solved" Problem That Isn't	Massively parallel sim RL, quadrupeds, humanoid whole-body control (HOVER, ExBody2), and a critical audit of "locomotion is solved."
04	Rigid-Body Manipulation: The Frontier	Why manipulation is hard, Diffusion Policy, ACT/ALOHA, the VLA era (RT-2, OpenVLA, π0/π0.5, GR00T), dexterous hands, data engines.
05	Deformable & Soft-Body Manipulation	Infinite-dimensional state, GNN dynamics models, SoftGym/DaXBench, diffusion approaches to cloth and dough, surgical robotics, soft grippers.
06	Frontiers & Cross-Cutting Themes	System 1/System 2 hierarchies (GR00T, Helix), the sim-to-real gap, the benchmark crisis, scaling-law debates, 2025–2026 outlook.
07	Case Study: LocoMuJoCo & the Unitree G1	Grounding every concept from Parts 1–6 in the code that lives in Cybernetic IDE's sibling projects.

The one-paragraph thesis of the whole guide

A policy is a learned closed-loop mapping from observations to actions — the robotics analogue of "the model" in your generative-AI world. The field currently lives in two regimes that barely resemble each other. Locomotion policies are tiny (2–3 layer MLPs, ~10⁵–10⁶ parameters) trained by reinforcement learning in massively parallel GPU simulation (minutes of wall-clock for a walking policy [V]) and transferred to hardware via teacher–student distillation and domain randomization; they run at ~50 Hz above 500 Hz–1 kHz PD loops [V]. Manipulation policies are increasingly huge (billion-parameter VLAs — literally language-conditioned policies built on pretrained VLMs, e.g., π0 = 3B PaliGemma backbone + 300M flow-matching action expert [V]) trained by imitation learning on teleoperated demonstrations, because manipulation's contact-rich, semantically diverse tasks resist both simulation and reward specification. Locomotion is field-robust but explicitly not solved by its own leading researchers [V]; manipulation generalization "remains a widespread and unresolved issue" per the 2025 survey literature [V]. The deformable-object world (Part 5) is roughly a decade behind rigid manipulation, and the frontier everywhere (Part 6) is hierarchical systems that put a slow VLA "System 2" on top of a fast reactive "System 1."

Primary reading list (anchor sources)

Surveys (read first):

Ha, Lee, van de Panne, Xie, Yu, Khadiv — Learning-Based Legged Locomotion: State of the Art and Future Perspectives (arXiv:2406.01152, IJRR 2025) — the locomotion anchor.
A Survey on Imitation-Learning-Based Robotic Manipulation Policies (arXiv:2508.17449, 2025) — the manipulation anchor.
The Anatomy of Vision-Language-Action Models (arXiv:2512.11362, Dec 2025) — the VLA anchor.
A Survey on Robotic Manipulation of Deformable Objects (arXiv:2312.10419, Neurocomputing) — the deformables anchor.

Landmark primary papers, in guide order:

Rudin et al., Learning to Walk in Minutes Using Massively Parallel Deep RL (arXiv:2109.11978, CoRL 2021)
Lee et al., Learning Quadrupedal Locomotion over Challenging Terrain (Science Robotics, 2020)
Miki et al., Learning Robust Perceptive Locomotion for Quadrupedal Robots in the Wild (Science Robotics, 2022)
He et al., HOVER: Versatile Neural Whole-Body Controller for Humanoid Robots (arXiv:2410.21229, ICRA 2025)
Ji et al., ExBody2: Advanced Expressive Whole-Body Control (arXiv:2412.13196)
Chi et al., Diffusion Policy (arXiv:2303.04137, RSS 2023 / IJRR)
Zhao et al., Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT/ALOHA, arXiv:2304.13705, RSS 2023)
Black et al., π0: A Vision-Language-Action Flow Model for General Robot Control (arXiv:2410.24164)
Kim et al., OpenVLA (arXiv:2406.09246) and OpenVLA-OFT (arXiv:2502.19645)
NVIDIA, GR00T N1: An Open Foundation Model for Generalist Humanoid Robots (arXiv:2503.14734)
Lin et al., SoftGym (CoRL 2020, PMLR v155)
Shi et al., RoboCook (arXiv:2306.14447, CoRL 2023)

Each part ends with its own expanded reference list.

Part 01 — What Is a Robot Policy?

Robot Policies field guide. The [V]/[S]/[K] evidence legend appears above and applies throughout.

1. The definition, stripped bare

A policy is a function that maps what the robot senses to what the robot does, executed in a loop:

π : observation → action        (deterministic form)
π(a | o)                        (stochastic form: a distribution over actions)

That's it. Everything else in this guide is detail about (a) what goes into o, (b) what comes out as a, (c) what function class represents π, and (d) how you obtain π's parameters.

The 2025 imitation-learning survey states it exactly this way: "A robotic manipulation policy (RMP) specifies how a robot selects and executes actions based on its sensory observations to achieve a manipulation goal" [V]. Diffusion Policy frames policy learning as "the supervised regression task of learning to map observations to actions" [V] — which should sound instantly familiar: it is sequence modeling, where the sequence is a physical interaction rather than text.

1.1 Bridge from your world

You already know this object under other names:

Your world	Robotics
An LLM maps a token context to a next-token distribution	A policy maps an observation (history) to an action distribution
Sampling temperature / nucleus	Stochastic policy π(a\|o); exploration noise
Autoregressive generation loop	The control loop: observe → act → world changes → observe...
A VLA like π0.5	Literally a policy. "Vision-Language-Action model" = a policy whose observation includes images + a language instruction and whose output is robot actions [V]
Stable Diffusion: noise → image, conditioned on prompt	Diffusion Policy: noise → action sequence, conditioned on observation [S] (Part 2)
E2-TTS flow matching: noise → mel-spectrogram	π0's action expert: noise → 50-step action chunk via flow matching [V]

The key difference from LLM generation: the policy's "decoding loop" passes through physics. The environment — not the model — computes the next "context." Errors compound in a way teacher-forced training never sees (this is the covariate shift problem of imitation learning, Part 2), and the loop must close in milliseconds, not seconds.

1.2 The formal wrapper: MDPs and POMDPs [K]

The standard formalism is the Markov Decision Process (MDP): states s, actions a, transition dynamics p(s′|s,a), reward r(s,a), discount γ. A policy π(a|s) is evaluated by its expected discounted return E[Σ γᵗ rₜ]. Reinforcement learning is the search for π maximizing this.

Real robots never see the state. They see observations — joint encoders, IMU, cameras — which are partial, noisy, delayed views of s. That makes the problem a POMDP (partially observed MDP), and the practical consequence is everywhere in this guide: policies condition on histories of observations (stacked frames, recurrent networks, transformers over past observations) to implicitly reconstruct the missing state. When you see "the student policy observes a history of onboard measurements" in a sim-to-real paper [V], that's POMDP-coping machinery.

2. What a policy is not: the neighboring species

The robotics stack contains several observation-to-action-ish objects. Confusing them is the #1 newcomer error.

2.1 Controller (PD, MPC, WBC)

A controller is a hand-designed feedback law with a given target.

PD controller [K]: τ = kp·(q* − q) − kd·q̇ — a proportional-derivative spring-damper that produces motor torque τ to pull joint position q toward target q*. Two gains per joint. No learning, no perception, runs at 500 Hz–1 kHz. This is the workhorse beneath almost every learned policy (§4).
MPC (Model Predictive Control) [K]: at every step, solve a short-horizon trajectory optimization problem using an explicit dynamics model, execute the first action, re-solve. Optimization at inference time, model-based, no learning required. Boston Dynamics' classical Atlas work is the iconic example.
WBC (Whole-Body Control) [K]: a task-space optimizer (typically a quadratic program solved each control tick) that converts desired body/end-effector motions into joint torques while respecting contact and balance constraints, using the robot's kinematic/dynamic model.

The distinction: a controller tracks a reference it is given; a policy decides. In practice they compose: the policy decides (at 50 Hz) and PD controllers track (at 1 kHz) [V]. Note the boundary is a design choice, not a law of nature — an aggressive MPC that replans goals is policy-like, and a learned network trained only to track is controller-like. The field's working convention: learned + closed-loop + decides ⇒ "policy."

2.2 Planner

A planner [K] computes a sequence of states or actions ahead of time (A*, RRT for motion planning; task planners; an LLM decomposing "make coffee" into steps). Planners are open-loop until re-planned; policies are closed-loop by construction. Modern stacks put planners above policies: the planner emits subgoals, the policy handles the reactive execution (Part 6's System 2 / System 1).

2.3 World model

A world model [K] predicts what happens next: p(o′|o,a) or a latent version of it. It is a simulator distilled into a network. Your reference points — GR00T's video-prediction lineage, Cosmos — are world foundation models. A world model is not a policy, but it can be used to get one, three ways:

Dream training (Dreamer family): train the policy by RL inside the learned model's imagination (Part 2).
Planning (TD-MPC2, MPC-style): search action sequences against the model at inference time.
Data generation: use it to synthesize training trajectories (GR00T N1's "neural trajectories," Part 4 [S]).

Mnemonic: world model = physics, reward model = desire, policy = behavior. The policy is the only one that has to run in the real-time loop.

3. Observation spaces: what π gets to see

A typical modern robot policy consumes some subset of:

Proprioception [K]: joint positions q, joint velocities q̇, base angular velocity and gravity direction from the IMU, previous action(s). Cheap (kHz-rate), low-dimensional (tens of floats), and the sole input to most deployed locomotion policies ("blind" locomotion). HOVER's real-robot student observes exactly this: joint pos/vel, base angular velocity, gravity vector, and an action history stacked over 25 steps [S].
Exteroception [K]: vision (RGB/depth cameras), LiDAR, elevation maps. In locomotion, "perceptive" policies fuse proprioception with terrain height samples (Miki et al. use a recurrent belief encoder over proprioception + noisy height samples, so the policy learns when to distrust its eyes [V]). In manipulation, vision is nearly always required — the object's pose is not in your joint encoders.
Language: an instruction string. This is precisely what makes a VLA a VLA — the policy is conditioned on language the way Stable Diffusion is conditioned on a prompt [V].
Tactile [K]: fingertip force/vision-based touch sensors (GelSight-style optical tactile skins, Meta's Digit). Rich literature, but as of 2025 tactile input remains rare in flagship generalist policies — a known gap (Part 4).
Privileged observations (sim-only): ground-truth contact forces, friction coefficients, exact terrain geometry — visible in the simulator but not on hardware. Used by teacher policies that are then distilled into deployable students (Part 2; verified pipeline in Lee 2020, Miki 2022, HOVER, ExBody2 [V]).

Design principle [K]: every added observation is a sim-to-real liability (it must be simulated faithfully and measured reliably on hardware). The field's habit of proprioception-first minimalism in locomotion is a robustness decision, not a compute one.

4. Action spaces: what π gets to say

This is the most robotics-specific design axis, and the research verified a clear picture:

4.1 The dominant pattern: joint-position targets over a PD loop

The policy outputs desired joint positions q*; a PD controller turns them into torques. The locomotion survey: "Most works in quadrupedal learning use joint target position as the action space (PD policy)" [V]. Concrete verified instances [V]:

Rudin et al. 2022: "The actions are interpreted as desired joint positions sent to the motors. There, a PD controller produces motor torques." Policy at 50 Hz.
Miki et al. 2022: 50 Hz policy emitting leg phase offsets + residual joint-position targets (via leg IK).
ExBody2: 50 Hz policy, 23-dim joint-position targets, 500 Hz low-level interface, deployed on a Unitree G1 with an onboard Jetson Orin NX.
HOVER: 19-dim joint-position targets into PD, on a Unitree H1.

Why this works [K]: the PD loop acts as a learned-policy-friendly abstraction — it is an impedance (spring-damper) around the target, so the policy's outputs are smooth, bounded, and forgiving; the high-rate stabilization burden is delegated to a 1 kHz loop the network never has to model. The action space itself embeds a prior.

Direct-torque policies exist but self-identify as the exception (Chen et al. 2023; Kim et al. 2023) [V]. The survey's argument is that PD-target policies can run slowly (~50 Hz) while torque is still produced at 1 kHz — whereas torque policies push the policy itself toward higher evaluation rates. (Note: the stronger claim "torque policies must run at 1 kHz" was refuted in verification — treat rate requirements as an empirical, per-paper question.)

4.2 The manipulation menu [K, structure; V where tagged]

End-effector deltas: Δ(x, y, z, roll, pitch, yaw) + gripper open/close, with an inverse-kinematics layer mapping to joints. The lingua franca of tabletop manipulation datasets (Open X-Embodiment).
Absolute joint positions: ACT predicts absolute joint targets for both ALOHA arms.
Discretized action tokens: bin each action dimension into 256 buckets and emit them as tokens — action generation becomes next-token classification on a standard transformer stack. This is the RT-2/OpenVLA design: OpenVLA maps 256 bins onto the 256 least-used Llama-2 tokens and trains with cross-entropy [V]. Known costs: quantization error and slow autoregressive decoding at high control rates [V] (the FAST paper's motivation).
Continuous action chunks via generative heads: a diffusion or flow-matching module generates an entire chunk of future actions conditioned on the observation embedding. π0: flow-matching action expert, chunks of H=50 actions, up to 50 Hz control [V].

4.3 Action chunking — the concept your sequence-model intuition needs

Action chunking = predict a block of k future actions at once, execute some prefix, re-observe, repeat. ACT introduced this framing for imitation: it "packs high-frequency, low-level controls into discrete 'action chunks'," shortening the effective decision horizon k-fold and improving sample efficiency in low-demo regimes [V].

The Dec-2025 VLA survey gives the exact formulation to remember: chunking is "a practical compromise: the policy operates autoregressively over coarse time (emitting chunks), but non-autoregressively within each chunk" [V] — the same pattern as ACT, Diffusion Policy, and π0/π0.5. Diffusion Policy pairs this with receding-horizon control: predict a horizon, execute a short prefix, re-plan [S].

Why chunk? [K] (a) compounding-error mitigation — fewer decision points per episode; (b) temporal consistency — one coherent sample instead of k independently-noised decisions (compare: generating a whole image at once vs. pixel-by-pixel); (c) inference amortization — one billion-parameter forward pass buys k control steps. The cost: within-chunk open-loop-ness reduces reactivity in stochastic environments [S] (Bidirectional Decoding, arXiv:2408.17355).

5. The control-frequency hierarchy

Assemble §3 and §4 into the canonical deployed stack:

  ~0.1–10 Hz   Planner / VLM / "System 2"     (subgoals, language, task logic)
  ~10–50 Hz    POLICY (the learned π)          (obs → action / action chunk)
  ~500–1000 Hz PD / whole-body controller      (target → motor torque)
  ~10–40 kHz   Motor drivers / current control (electrical commutation)

Verified anchors: 50 Hz policy over 500 Hz low-level on the G1 (ExBody2) [V]; 50 Hz perceptive policy on ANYmal (Miki) [V]; GR00T N1's System 2 VLM at 10 Hz over a System 1 diffusion-transformer action module at 120 Hz [S]; π0 at up to 50 Hz [V].

The general lesson, which you can port straight from systems thinking: each layer is a rate adapter and an abstraction barrier. The policy is the layer where learning currently pays off most — below it, physics is too fast and too safety-critical; above it, semantics were (until VLAs) too hard.

6. Where policies live in your generative-AI mental model — a summary table

Question	Locomotion answer (typical)	Manipulation answer (typical, 2025)
Function class	2–3 layer MLP (~10⁵–10⁶ params), sometimes GRU/small transformer [V]	Billion-param VLA (π0: 3.3B; OpenVLA: 7B; GR00T-N1: 2.2B) [V]/[S]
Observation	Proprioception (+ optional height map) [V]	Multi-camera RGB + proprioception + language [V]
Action	Joint-position targets → PD [V]	EE deltas / joint targets, as tokens or generative chunks [V]
Trained by	RL in massively parallel sim [V]	Imitation of teleoperated demos [V]
Rate	~50 Hz over 0.5–1 kHz PD [V]	~5–50 Hz chunked [V]
Status	Field-robust, not "solved" [V]	The open frontier; generalization unresolved [V]

Hold this table in mind; Parts 3 and 4 are essentially deep dives into its two columns, and Part 2 explains the training-paradigm row.

References (this part)

IL-manipulation survey — arXiv:2508.17449 (policy definition; ACT chunking)
Legged locomotion survey — arXiv:2406.01152 (action spaces; PPO; teacher–student)
Rudin et al. — arXiv:2109.11978 (50 Hz PD-target policy)
Miki et al. — Science Robotics abk2822 (perceptive locomotion, belief encoder)
ExBody2 — arXiv:2412.13196 (G1 deployment stack)
HOVER — arXiv:2410.21229 (MLP size, observation stack)
VLA anatomy survey — arXiv:2512.11362 (action tokenization; chunking formulation)
OpenVLA — arXiv:2406.09246 (256-bin tokenization)
π0 — arXiv:2410.24164 (flow-matching action expert, 50 Hz)
Diffusion Policy — arXiv:2303.04137 (receding-horizon; "supervised regression" framing)
GR00T N1 — arXiv:2503.14734 (10 Hz / 120 Hz dual rates)
Sutton & Barto, Reinforcement Learning: An Introduction [K] (MDP/POMDP formalism)

Part 02 — The Taxonomy: How Robot Policies Are Trained

Robot Policies field guide. The [V]/[S]/[K] evidence legend appears above and applies throughout.

The function π(a|o) can be represented by anything differentiable. What actually distinguishes the families you'll meet in papers is where the training signal comes from. There are exactly three primal sources — reward (RL), demonstrations (imitation), and predictions (world models) — and everything in the ecosystem is one of these or a hybrid.

1. Model-free reinforcement learning: policies from reward

Idea [K]: roll the policy in an environment, score trajectories with a reward function, ascend the expected return. No dynamics model is learned; the simulator is the model.

1.1 PPO and why it owns locomotion

PPO (Proximal Policy Optimization) [K, mechanics]: an on-policy actor-critic method that takes conservative policy-gradient steps by clipping the importance ratio between new and old policy. Simple, stable, embarrassingly parallel.

The verified field consensus: "PPO has been a particularly popular choice in the legged locomotion community due to its excellent convergence and adaptability to diverse policy architectures" — it is the most popular deep-RL algorithm for legged locomotion and the default in Isaac Lab / RSL-RL pipelines [V]. Even pro-SAC sources treat PPO as the de-facto standard [V].

Why on-policy wins in simulation [K]: PPO's weakness is sample hunger — but massively parallel GPU simulation made samples nearly free. Rudin et al.: 4096 simulated ANYmals, batch size 98,304, 1500 PPO updates → flat-terrain walking in under 4 minutes, rough terrain in ~20 minutes, on one RTX A6000 [V]. GPU simulators (Isaac Gym — note: the survey misnames it "Isaac Sim") collect roughly 0.5–0.7 M state transitions per second [V]. Twenty wall-clock minutes ≈ ~80 hours of simulated robot experience. Off-policy methods (SAC) appear where sample efficiency is the true bottleneck, i.e., learning directly on hardware [V].

Reward design is the real art [K]: locomotion rewards are sums of ~10–20 shaped terms (velocity tracking, torque penalties, action-rate penalties, foot air-time, orientation penalties...). This fragility is exactly what motivates the imitation-flavored methods in §3–4. Curricula matter too — Rudin et al.'s "game-inspired" terrain curriculum promotes a robot to harder terrain when it walks off its tile and demotes it when it covers less than half its commanded distance [S].

1.2 Where model-free RL fails

RL needs (a) a resettable, fast environment and (b) a specifiable reward. Locomotion has both. Tabletop manipulation usually has neither at scale — contact simulation is unreliable (Part 6 [S]), resets are physical labor, and "fold the shirt nicely" defies reward engineering. Hence the imitation-learning takeover of manipulation (§3).

2. Model-based RL: policies from learned dynamics [K]

(Flagged: this section is background — the deep-research verification pass returned no surviving claims here; numbers below are from the primary papers as retained knowledge.)

Dreamer family (Hafner et al.): learn a latent world model (RSSM), then train actor and critic entirely inside its "imagination" — rollouts of the latent model. DreamerV3 (2023) solved diverse benchmark domains with one configuration; DayDreamer (2022) trained a physical quadruped to walk in about an hour on hardware, without a simulator — the canonical demonstration that world models buy real-robot sample efficiency.
TD-MPC2 (Hansen et al., 2023): hybrid — learn a latent dynamics model, plan at inference time with short-horizon sampling (MPPI) guided by a learned value function; one set of hyperparameters across 100+ continuous-control tasks.
Relationship to your world: this is the "policies from world models" cell — the robotics-native ancestor of the Cosmos/GR00T thesis that predictive models of the world are a substrate for action. In deformables, the model-based pattern re-emerges strongly (Part 5: GNN dynamics + planning [S]).

Why it hasn't displaced PPO-in-sim for locomotion [K]: when a GPU simulator gives you 700K real-physics steps per second, a learned approximation of physics has to justify its bias. Model-based methods shine when the environment is the real world (slow, unresettable) or when the dynamics must be learned anyway (deformables).

3. Imitation learning: policies from demonstrations

Behavior cloning (BC) [K]: collect (o, a) pairs from an expert (usually human teleoperation), fit π by supervised learning. It inherits everything you know from supervised generative modeling — and one problem you don't have: covariate shift. The policy's own small errors carry it into states the demonstrator never visited, where it errs worse (compounding). DAgger [K] (Ross et al., 2011) fixes this by iteratively querying the expert on the learner's own states — impractical with humans, but perfect when the "expert" is another neural network (see §5, distillation).

The last three years of manipulation are, at core, three escalating answers to "how do we make BC actually work?":

3.1 ACT (2023): chunking + a CVAE transformer

Zhao et al.'s Action Chunking with Transformers, built for the $20K-class ALOHA bimanual teleoperation rig [K, hardware], predicts chunks of ~100 future joint-position targets (50 Hz control) in one DETR-style forward pass, trained as a conditional VAE. Chunking shortens the effective horizon k-fold, mitigating compounding errors, and was critical in ablations; ~50 demonstrations sufficed for 80–90% success on fine bimanual tasks [V].

3.2 Diffusion Policy (2023): Stable Diffusion for actions — stated carefully

Chi et al. represent "a robot's visuomotor policy as a conditional denoising diffusion process" [S, verbatim]: instead of regressing one action, start from Gaussian noise over an action sequence and iteratively denoise it, conditioned on visual observations — mechanically the same conditional denoising you know from image diffusion, applied to a (horizon × action-dim) array. Inference "learns the gradient of the action-distribution score function and iteratively optimizes via stochastic Langevin dynamics steps" [S].

⚠️ Refuted framing to avoid: "Diffusion Policy pioneered denoising for control" failed verification 0-3 (diffusion for decision-making predates it — e.g., Janner et al.'s Diffuser [K]). Its verified contribution is making diffusion work as a real-time robot policy: receding-horizon control, visual conditioning, and a time-series diffusion transformer [S].

Why diffusion at all? The authors' three reasons: graceful handling of multimodal action distributions (a demonstrator sometimes goes left, sometimes right — a mean-regression policy splits the difference and hits the obstacle), suitability for high-dimensional action spaces (chunks), and training stability [S]. Result: +46.9% average relative improvement across 12 tasks / 4 benchmarks over prior methods [S].

3.3 Flow matching & the VLA era (2024–): π0 and kin

Exactly as E2-TTS replaced diffusion's many-step sampling with flow matching's few-step ODE integration, π0 attaches a flow-matching "action expert" (~300M params, its own weights — not just a head) to a pretrained 3B PaliGemma VLM, 3.3B total, generating 50-step continuous action chunks at up to 50 Hz [V]. The survey's stated rationale for flow over diffusion: high-quality outputs in fewer inference steps [S]. The alternative VLA design tokenizes actions into 256 bins and does next-token classification (RT-2, OpenVLA) [V] — Part 4 treats both in depth.

The punchline for your mental model: a VLA is behavior cloning at foundation-model scale — pretrained VLM backbone, robot-demonstration corpus, generative action decoder. The paradigm is imitation; the novelty is scale and the language interface.

4. Adversarial imitation & motion tracking: reward from demonstrations

Between "reward engineering" (RL) and "supervised copying" (BC) sits a family that converts demonstrations into a reward and then runs RL. This family dominates character-like whole-body motion — and it is exactly what's implemented in the loco-mujoco repo sitting next to this one (Part 7).

GAIL [K] (Ho & Ermon, 2016): a GAN over state transitions. Discriminator D learns to tell policy transitions from expert transitions; the policy is RL-trained with reward = fooling D. No reward engineering; no action labels needed beyond states.
AMP [K] (Peng et al., 2021): Adversarial Motion Priors — GAIL's discriminator repurposed as a style reward, added to a simple task reward (e.g., "reach this velocity"). The policy accomplishes the task in the style of the mocap corpus. This is the standard recipe for natural-looking humanoid/quadruped motion; loco-mujoco's AMPJax implements it as a least-squares discriminator variant of its GAIL trainer.
DeepMimic [K] (Peng et al., 2018): no discriminator — a tracking reward directly measures pose/velocity/end-effector distance to a retargeted reference motion, with exponential kernels (exp(−w·error²)), plus reference state initialization. This is the workhorse of humanoid whole-body control: HOVER's oracle is a motion imitator on retargeted AMASS mocap [V]; ExBody2 tracks CMU mocap with a teacher-filtered curriculum [S]; loco-mujoco's MimicReward is a faithful implementation (qpos/qvel/site-position exponential terms + torque/action-rate penalties).
ASAP [K] (2025, NVIDIA/CMU): the real2sim2real refinement — learn a delta action model from real-robot rollouts that corrects the simulator's dynamics mismatch, then fine-tune the tracking policy inside the corrected sim; used for agile G1 whole-body motions. (Background-flagged; not in the verified set.)

Why adversarial/tracking beats plain BC for locomotion [K]: demonstrations (human mocap) come from a different embodiment — you cannot clone actions that were never recorded (mocap has no torques), so you must let RL discover actions whose outcomes match the reference states. That is precisely what tracking rewards and discriminators do.

5. Teacher–student distillation: the sim-to-real workhorse

The verified standard recipe for getting RL policies onto real legged robots [V]:

Teacher: train with RL in simulation with privileged observations — noiseless states, exact terrain, contact states/forces/normals, friction, external wrenches — things only a simulator can expose.
Student: distill the teacher into a policy that consumes only deployable observations (proprioception histories, noisy height samples), typically with DAgger — roll out the student, label its states with the teacher's actions, supervised-fit (an P2/action-matching loss).
Deploy zero-shot on hardware.

Verified instances [V]: Lee et al. 2020 (ANYmal blind locomotion — proprioceptive student from onboard-measurement history); Miki et al. 2022 (recurrent belief-state encoder fusing proprioception + noisy height samples; zero-shot transfer, no fine-tuning); HOVER (oracle imitator on AMASS → DAgger distillation into one multi-mode policy); ExBody2 (PPO teacher on privileged info → DAgger student from longer non-privileged history, deployed on Unitree G1).

Why it works [K]: the privileged teacher solves an easy, fully-observed RL problem; the student solves an easy supervised problem (with DAgger killing covariate shift, since the teacher-expert is queryable everywhere). The POMDP never has to be solved directly by RL. Distillation also merges: many specialists → one generalist (HOVER distills modes [V]; the embodiment-scaling study distills ~1,000 per-embodiment experts into one cross-embodiment policy [S]).

6. Sim-to-real transfer & domain randomization

Domain randomization [K] (Tobin et al. 2017; OpenAI's Dactyl/Rubik's-cube automatic domain randomization): randomize physics (masses, friction, latencies, motor strengths) and visuals during training so the real world looks like just another sample from the training distribution.
Actuator modeling [K]: ETH's actuator networks — learn the motor's real torque response from data and put it in the simulator.
Verified status: sim-to-real is standard working practice for locomotion ("it is common practice to train the policies in simulation and then transfer them to the real world") [V] — yet for VLA manipulation it "remains a core obstacle," with dynamics discrepancies (friction, latency, actuation response) and perception discrepancies (illumination, textures, sensor noise) severely degrading transfer; randomization and fidelity enhancement are mitigations, not solutions [V].
Counterpoint from late 2025 [S]: ETH (Hutter's group) demonstrated reliable transfer across 13 legged platforms without randomizing dynamic parameters — using systematic dynamic-parameter identification and a physics-grounded actuator (PMSM) energy model instead, while noting that "controllers trained in simulation often fail to transfer reliably." Randomization is the default, not a law.

7. The size spectrum — one taxonomy-wide observation

[V] State-of-the-art humanoid whole-body policies are tiny: HOVER is a 3-layer MLP [512, 256, 128] emitting 19-dim PD targets; corroborating SONIC: "state-of-the-art humanoid control policies are often small neural networks, e.g., three-layer MLPs." Miki's field-proven ANYmal student: a 2×50-unit GRU + a {256,160,128} MLP [S]. Meanwhile π0 is 3.3B, OpenVLA 7B, GR00T-N1 2.2B [V]/[S] — roughly five orders of magnitude apart. (Caveat: some 2025 humanoid trackers use small transformers — still millions, not billions, of parameters [V].)

The interpretation to internalize: network size tracks the semantic bandwidth of the observation-action mapping, not the physical difficulty of the task. Balancing 51 kg of humanoid is stupendously hard physics but a low-dimensional, fast mapping (proprioception → 19 joint targets); "clean the kitchen" is easy physics and an open-world mapping (pixels + language → any of a thousand behaviors). RL-in-sim compresses physics into small networks; open-world semantics demand pretrained internet-scale backbones.

8. Decision chart (which paradigm when) [K]

Reward specifiable + fast simulator?            → model-free RL (PPO) in parallel sim
  ...and deploy on hardware?                    → + teacher-student + DR / system-ID
Demonstrations exist, same embodiment?          → BC with chunking (ACT / Diffusion Policy / VLA fine-tune)
Demonstrations exist, different embodiment
  (human mocap → robot)?                        → tracking reward (DeepMimic) or AMP/GAIL + RL
No simulator, no demos, real robot only?        → model-based RL (Dreamer-style) or offline RL
Open-world language-conditioned generality?     → VLA (pretrained VLM + generative action decoder)

References (this part)

Locomotion survey — arXiv:2406.01152 [V] (PPO consensus, teacher–student, DR practice)
Rudin et al. — arXiv:2109.11978 [V] (walking in minutes; curriculum)
Lee et al. 2020, Science Robotics — [V] (privileged teacher–student)
Miki et al. 2022, Science Robotics abk2822 — [V] (belief encoder, zero-shot transfer)
HOVER — arXiv:2410.21229 [V]; ExBody2 — arXiv:2412.13196 [V]
ACT — arXiv:2304.13705 [V]; Diffusion Policy — arXiv:2303.04137 [S]
π0 — arXiv:2410.24164 [V]; VLA anatomy survey — arXiv:2512.11362 [V]
IL-manipulation survey — arXiv:2508.17449 [V] (flow-vs-diffusion rationale [S])
ETH energy-efficient sim-to-real — arXiv (Sept 2025) [S] (32% CoT reduction, no-DR transfer)
Embodiment scaling — CoRL 2025, PMLR v305 [S]
[K] anchors: Ho & Ermon 2016 (GAIL); Peng et al. 2018 (DeepMimic), 2021 (AMP); Ross et al. 2011 (DAgger); Hafner et al. (Dreamer/DayDreamer); Hansen et al. (TD-MPC2); Tobin et al. 2017 & OpenAI 2019 (domain randomization); He et al. 2025 (ASAP)

Part 03 — Locomotion: The "Solved" Problem That Isn't

Robot Policies field guide. The [V]/[S]/[K] evidence legend appears above and applies throughout.

People keep telling you locomotion is "more or less solved." This part gives you the strongest honest version of that claim, the evidence behind it, and then the expert-sourced case against it. Short version: the recipe is solved; the problem is not.

1. The recipe that changed everything: massively parallel sim RL

Before 2021 [K], learned locomotion meant CPU farms and days-to-weeks of training (ETH's 2019-era Science Robotics results ran on distributed CPU simulation), or careful model-based control (MPC/WBC — the classical Boston Dynamics lineage).

The verified inflection point — Rudin et al., CoRL 2021, "Learning to Walk in Minutes" [V]:

4096 parallel simulated ANYmal robots in Isaac Gym on a single RTX A6000 (+ i9-11900k), batch size 98,304, 1500 PPO updates.
Flat-terrain walking in under 4 minutes; rough terrain in ~20 minutes of wall clock.
Versus a comparable prior fully-learned perceptive approach needing 120 hours.
GPU-simulator throughput: ~0.5–0.7 M state transitions/second (the survey's "~1M" is a generous rounding; and note it's Isaac Gym, Makoviychuk et al. 2021 — the survey misnames it "Isaac Sim") [V].
Training curriculum: "game-inspired" terrain promotion/demotion based on whether the robot walks off its terrain tile or fails to cover half its commanded distance [S].

Wall-clock minutes ≈ tens of simulated hours. Once policy iteration costs minutes, reward shaping, curricula, and architecture search become interactive — that engineering-loop compression, more than any algorithmic insight, is what made the last four years of legged robotics.

The stack that emerged (all of it verified across multiple papers in Parts 1–2): tiny MLP policy [V] → PPO [V] → privileged teacher, DAgger student [V] → joint-position targets into 0.5–1 kHz PD [V] → zero-shot hardware deployment [V].

2. Quadrupeds: the strongest case for "solved"

2.1 Field evidence — ANYmal in the wild [V]

Miki et al. 2022 (Science Robotics) is the canonical robustness result:

50 Hz policy: leg phase offsets + residual joint-position targets; recurrent belief-state encoder fuses proprioception with noisy height-map samples, learning when to distrust exteroception [V].
Teacher trained with privileged contact states/forces/normals, friction, external wrenches; student distilled, deployed zero-shot, no fine-tuning [V].
The hike: ANYmal completed a 2.2 km Alpine route (Mount Etzel, Switzerland; 120 m elevation gain), summiting in 31 min vs. the 35 min posted for human hikers, finishing in 78 min vs. a 76-min planner estimate, no falls [V]. Caveats: single run, and a human supplied directional commands — the policy handled locomotion, not navigation [V].
Deployment record: at the DARPA Subterranean Challenge, the controller carried four ANYmals over 1,700 m of underground terrain without a single fall, staying functional under exteroceptive degradation (reflective floors, snow, transparent obstacles), and walked at 1.2 m/s vs. 0.6 m/s for the blind baseline [S].

2.2 Beyond walking [K]

Parkour policies (Zhuang et al. 2023; Cheng et al. 2023 — climbing, leaping, squeezing on cheap quadrupeds), ANYmal parkour (Hoeller et al., Science Robotics 2024), wheeled-legged hybrids, and commercial adoption (Boston Dynamics shipping an RL controller for Spot, 2024; Unitree's product line) all reinforce the maturity story for quadrupeds. (Background-flagged.)

3. Humanoids: the recipe generalizes upward

Humanoid whole-body control (WBC) is where locomotion research now concentrates. Two verified flagships:

3.1 ExBody2 (UCSD, arXiv:2412.13196) [V]

Two-stage: PPO teacher on privileged info → DAgger student from a longer, non-privileged observation history.
Deployed on Unitree G1 (23-dim joint targets, 50 Hz policy, 500 Hz low-level, onboard Jetson Orin NX).
Small networks: teacher actor MLP [512,256,128]; student MLP [1024,1024,512]; Isaac Gym parallel training [S].
Trained to track human mocap (CMU), with the teacher used to filter dynamically infeasible clips; the student walks, crouches, dances — DeepMimic-style tracking made real [S]. Tracking error (mean per-joint, real robot): 0.1074 rad vs. 0.1465 (ExBody) and 0.1396 (OmniH2O*) [S].

3.2 HOVER (NVIDIA GEAR/CMU, ICRA 2025) [V, with flagged caveats]

The "generalist controller" result:

One policy unifies three command families — kinematic keypoint tracking, local joint-angle tracking, root velocity/height/orientation tracking — via command-space masking (binary mode + sparsity masks; the paper's "one-hot" wording is loose [V]), yielding 15+ usable control modes on a 19-DOF Unitree H1 without per-mode retraining.
Common abstraction: full-body kinematic motion imitation — an oracle trained on retargeted AMASS mocap, distilled by DAgger.
Policy: 3-layer MLP [512,256,128] → 19 PD targets [V]; real-robot student sees only proprioception + 25-step action history [S].
Self-reported: beats specialist controllers in ≥7/12 tracking metrics in every mode, and a from-scratch multi-mode RL baseline on 32/32 metrics. ⚠️ Present with attribution, not as settled fact: comparisons are in Isaac Gym with author-reimplemented baselines, and an independent re-evaluation (BumbleBee, arXiv:2506.12779) reports much lower HOVER success rates in its own setup (63.21% Isaac Gym / 16.12% MuJoCo vs. 89.58%/66.84%), citing gradient conflicts in full-AMASS multi-mode distillation [V-caveat].

The humanoid pattern to remember: mocap tracking is the pretraining objective of whole-body control — human motion data plays the role that internet text plays for LLMs, and the tracking policy is then commanded by higher layers (teleop, planners, VLAs — Part 6).

4. So is locomotion solved? The audit

4.1 The case for "more or less"

Within the settled envelope — velocity-commanded walking over moderate terrain on well-modeled robots — the problem is commodity: minutes of training [V], open-source pipelines (legged_gym/RSL-RL, Isaac Lab, loco-mujoco next door), field-proven robustness [V], products shipping [K]. If your goal is "make a G1 walk," you are executing a known recipe, not doing research. Your acquaintances are right about this envelope.

4.2 The expert-sourced case against [V]

The leading 2024 survey (Ha, Lee, van de Panne, Xie, Yu, Khadiv — arXiv:2406.01152) explicitly treats legged locomotion as not solved, devoting its final section to open problems, item-for-item [V]:

Unsupervised skill discovery — beyond reward-engineered gaits.
Differentiable simulators — gradient-based policy optimization through contact.
Traversing challenging environments — the contact-rich long tail (loose rock, mud, ice, vegetation, degraded perception).
Safety — certification and guarantees for learned controllers; RL policies offer no formal envelopes.
Hybrid wheeled-legged locomotion.
Loco-manipulation — locomotion in service of manipulation: carrying, pushing, opening doors while balancing; the seam between Parts 3 and 4, largely open.
Foundation models for locomotion — cross-embodiment generality (the embodiment-scaling study of Part 6 distills ~1,000 procedurally-generated embodiments into one zero-shot-transferring policy, and self-describes its scaling evidence as preliminary [S]).

Fresh corroboration that the basics still yield: ETH, Sept 2025 — a physics-grounded actuator-energy framework cut ANYmal's full Cost of Transport by 32% (to 1.27) over state-of-the-art methods, while stating plainly that "controllers trained in simulation often fail to transfer reliably" and that most approaches neglect actuator-specific energy losses [S]. A 32% efficiency improvement remaining on the table in 2025 is not the signature of a solved field. Sim-to-real robustness, energy, and safety are all live.

4.3 The synthesis to carry forward

"Locomotion is solved" is true the way "image classification is solved" was true in 2017: the benchmark regime is saturated; the deployment regime is not. What is genuinely settled is the method — parallel-sim PPO + distillation + PD-target actions is to legged robots what the transformer recipe is to language. The open problems have migrated up the stack (loco-manipulation, foundation policies, safety) and down into the physics long tail. Meanwhile manipulation (next part) lacks even the settled recipe — that asymmetry, verified on both sides [V], is the real content of your acquaintances' claim.

References (this part)

Rudin et al., Learning to Walk in Minutes — arXiv:2109.11978 [V]
Makoviychuk et al., Isaac Gym — arXiv:2108.10470 [K]
Lee et al. 2020 — Science Robotics (blind locomotion, teacher–student) [V]
Miki et al. 2022 — Science Robotics abk2822 (perceptive locomotion; Alpine hike; SubT) [V]/[S]
Ha et al., locomotion survey — arXiv:2406.01152 (Section 8 open problems) [V]
HOVER — arXiv:2410.21229 [V]; BumbleBee — arXiv:2506.12779 (independent re-evaluation) [V-caveat]
ExBody2 — arXiv:2412.13196 [V]/[S]
ETH energy-efficient locomotion, Sept 2025 — arXiv:2509.06342 [S]
Embodiment scaling laws — CoRL 2025 (PMLR v305) [S]
[K]: Zhuang/Cheng parkour 2023; Hoeller et al. 2024; Boston Dynamics Spot RL blog 2024

Part 04 — Rigid-Body Manipulation: The Frontier

Robot Policies field guide. The [V]/[S]/[K] evidence legend appears above and applies throughout.

1. Why manipulation resisted the locomotion recipe

Locomotion's recipe (Part 3) requires a fast, faithful simulator and a specifiable reward. Manipulation breaks both legs of that stool:

Contact-rich dynamics defeat simulators. Verified, from the ManipulationNet benchmark paper: simulation-based manipulation benchmarks "offer reproducible tasks and accessibility at scale, but their imperfect approximations of contact dynamics render results incapable of fully reflecting the true manipulation capabilities" — i.e., sim systematically overestimates real capability [S]. Friction, compliance, and intermittent contact are exactly what current physics engines approximate worst [K].
Rewards don't write themselves. "Pour the coffee without splashing" has no clean scalar. Locomotion's velocity-tracking reward has no manipulation analogue at task generality [K].
Semantic diversity. A walking policy meets one distribution (terrain); a kitchen policy meets thousands of objects, layouts, and instructions. This is a representation problem — hence pretrained vision-language backbones [V].
Partial observability: occlusion by the robot's own gripper, unobservable object properties (mass, friction, contents) [K].

The verified status line: as of the 2025 IL-manipulation survey (82 methods reviewed), imitation-learned manipulation policies "suffer from evident overfitting, and the generalization of the policies remains a widespread and unresolved issue" [V]. Even the flagship generalization result, π0.5, self-describes as "far from perfect," calling generalization "the fundamental challenge" [V]. Manipulation is the frontier not because nothing works, but because nothing generalizes the way locomotion's recipe does.

So the field pivoted: if you can't simulate or specify, imitate. The modern history of manipulation policies is the history of scaling behavior cloning.

2. The 2023 duo: making BC actually work

2.1 ACT / ALOHA — chunking (Zhao et al., RSS 2023) [V]

Covered mechanically in Part 2 §3.1: a CVAE transformer that emits ~100-step chunks of absolute joint targets at 50 Hz control on the low-cost ALOHA bimanual rig; ~50 demos → 80–90% success on fine bimanual tasks (ziploc bags, battery insertion). Chunking's k-fold horizon reduction is the transferable idea — every VLA below inherits it [V].

2.2 Diffusion Policy — multimodality (Chi et al., RSS 2023) [S]

Also in Part 2 §3.2: visuomotor policy as conditional denoising over action sequences; receding-horizon execution; +46.9% average over prior SOTA across 12 tasks/4 benchmarks [S]. Its core justification — demonstration data is multimodal, and mean-seeking regression fails where mode-seeking generation succeeds — is the reason generative heads (diffusion, then flow) became the default action decoder in everything below.

3. The VLA era: policies become foundation models

Definition, verified: VLAs are policies — language-conditioned observation-to-action mappings [V]. Two action-decoding paradigms dominate [V]:

Paradigm	Mechanism	Exemplars
Discrete tokens	Bin each action dim (typically 256 bins), emit as next-token classification on the LLM stack	RT-2, OpenVLA (bins mapped to the 256 least-used Llama-2 tokens, cross-entropy) [V]
Continuous generative	Flow-matching/diffusion module generates action chunks conditioned on VLM features	π0 (flow), GR00T N1 (diffusion transformer w/ flow-matching objective) [V]/[S]

Known limits of the discrete route: quantization error and slow autoregressive decoding at high control rates [V] — the stated motivation for Physical Intelligence's FAST tokenizer (DCT-compressed action tokens; universal FAST+ tokenizer trained on ~1M action sequences; ~5× faster training than diffusion VLAs) [S].

3.1 Lineage in five steps [K for narrative; tagged facts]

RT-1 → RT-2 (Google, 2022–23) [K]: RT-2's move — co-fine-tune a VLM on web data + robot trajectories with actions as tokens — established "VLA" as a category.
Open X-Embodiment / RT-X (2023) [K]: the cross-lab dataset pooling (~1M+ episodes, 20+ embodiments) that made open generalist training possible; π0 explicitly complements its own corpus with OXE [S].
OpenVLA (2024) [S]: first fully open-source 7B VLA [V]; Llama-2 backbone + fused DINOv2/SigLIP visual encoder; trained on 970K real robot demonstrations; outperforms the closed 55B RT-2-X by 16.5% absolute across 29 tasks with 7× fewer parameters; LoRA-fine-tunable on consumer GPUs [S].
π0 (Physical Intelligence, Oct 2024) [V]: 3B PaliGemma + 300M from-scratch action expert (a separate-weights transformer expert with block-wise attention — not merely a "head") = 3.3B total; conditional flow matching over H=50 action chunks; up to 50 Hz; pretrained on ~10,000 hours of dexterous demonstrations across 7 robot configurations and 68 tasks + OXE [V]/[S]. Out-of-box evaluations: best across all tasks vs. OpenVLA and Octo, near-perfect on shirt folding and easy bussing; the paper attributes OpenVLA's failures to autoregressive discretization without chunking [S]. Demonstrated on laundry folding, table bussing, grocery bagging, box assembly [S]. Follows an LLM-style pretrain → post-train recipe, including a hierarchical mode where a high-level VLM issues language subcommands to low-level π0 [S]. π0.5 (2025) extends to open-world homes; generalization still self-assessed "far from perfect" [V].
GR00T N1 (NVIDIA, Mar 2025) [S]: the humanoid foundation policy, treated in depth in Part 6 (dual-system architecture; 2.2B params; System 2 at 10 Hz, System 1 diffusion-transformer at 120 Hz; "data pyramid" with 88 h of teleop expanded ~10× by neural trajectories + 780K sim trajectories; ~50K H100-hours of pretraining; beats Diffusion Policy baselines by ~30 points on real GR-1 bimanual tasks, ~76.6% success).

3.2 The plot twist: maybe you don't need the generative head [S]

OpenVLA-OFT (Stanford, Feb 2025): fine-tuning OpenVLA with parallel decoding + action chunking + continuous actions + plain P1 regression hits 97.1% average on LIBERO — beating π0, Octo, Diffusion Policy et al. on that benchmark — with 26× faster action generation than base OpenVLA; with FiLM language conditioning it runs high-frequency control on a real bimanual ALOHA, topping RDT-1B, π0, ACT, and Diffusion Policy on four dexterous tasks [S]. Fine-tuning cost: 50–150K gradient steps, 8×A100/H100, 1–2 days; bf16 inference in ~16–18 GB [S].

Lesson: the chunking + continuous action insight matters more than which generative machinery decodes it — and benchmark rankings among VLA variants remain unstable (Part 6's benchmark crisis).

4. Dexterous hands [K — background; verification pass returned no surviving claims here]

OpenAI Dactyl / Rubik's Cube (2018–19): the proof that domain-randomized RL could produce in-hand reorientation on a 24-DOF Shadow Hand — automatic domain randomization (ADR) at massive sim scale. A legacy result: the program disbanded, but ADR became standard vocabulary.
Hardware democratization: LEAP Hand (~$2K, 16-DOF) and similar open designs moved dexterity research out of the Shadow-Hand price class.
Sim-scale RL dexterity: in-hand rotation and reorientation via parallel-sim RL + distillation (e.g., touch/proprioception-only rotation on real hands from the Berkeley/Meta line of work); bimanual+hands humanoid teleoperation datasets feeding VLA training.
Tactile sensing: GelSight-style optical tactile sensors, Meta Digit — rich sensing exists, but integration into flagship generalist policies remains sparse; most VLAs are vision+proprioception only. Treat "tactile-integrated foundation policy" as an open slot on the 2026 bingo card.

5. The data problem — manipulation's real bottleneck

Every paradigm above is starved by the same constraint: there is no internet of robot actions. The field's countermeasures [K, with tagged instances]:

Teleoperation at scale: ALOHA rigs; Physical Intelligence's ~10K-hour corpus [S]; humanoid teleop via VR. Cost: human-hours ≈ robot-hours.
Portable data collection: UMI (Chi et al., 2024) — hand-held gripper + GoPro turns any human into a demonstration source without a robot [K].
Simulation data generation: MimicGen / DexMimicGen (NVIDIA) — synthesize thousands of demos from a handful by re-composing object-centric segments [K]; GR00T N1's 780K sim trajectories (~6,500 h equivalent) [S].
Neural/world-model data: GR00T N1's "neural trajectories" — model-generated video expansions of real teleop (88 h → 827 h, ~10×) [S]. This is where your Cosmos/world-model background plugs in: world models as data engines for policies.
Cross-embodiment pooling: OXE/RT-X [K]; π0's 7-platform corpus [S].

The data pyramid (GR00T's term [S]) is the field's consensus picture: a broad base of web/human video, a middle of simulation and synthetic trajectories, a small expensive peak of real teleoperation.

6. Status assessment

Dimension	State (2025–26)	Evidence
Single-task, in-distribution BC	Strong (80–97% on benchmarks)	ACT [V], OpenVLA-OFT [S]
Language-conditioned multi-task	Real but brittle; benchmark-dependent rankings	π0 vs OpenVLA vs OFT [S]
Open-world generalization	The unresolved problem	2025 survey; π0.5 self-assessment [V]
Dexterous in-hand manipulation	Demonstrated in narrow regimes; not integrated into generalists	[K]
Tactile integration	Sparse in flagships	[K]
Sim-to-real for manipulation	Core unsolved obstacle (dynamics + perception gaps)	[V]
Standard evaluation	Missing — "no ImageNet of robotics"	[S] (Part 6)

Compare with Part 3's table and the asymmetry your acquaintances gestured at becomes precise: locomotion has a settled method with an open long tail; manipulation has impressive artifacts and no settled method — the recipe itself (which data, which action decoder, RL or not) is still in play. That is what "manipulation is the frontier" means, verified [V].

References (this part)

IL-manipulation survey — arXiv:2508.17449 [V] (overfitting/generalization status)
VLA anatomy survey — arXiv:2512.11362 [V] (tokenization paradigms; chunking)
ACT — arXiv:2304.13705 [V]; Diffusion Policy — arXiv:2303.04137 [S]
π0 — arXiv:2410.24164 + pi.website/blog [V]/[S]; π0.5 — arXiv:2504.16054 [V]; FAST — arXiv:2501.09747 [S]
OpenVLA — arXiv:2406.09246 [S]; OpenVLA-OFT — arXiv:2502.19645 / openvla-oft.github.io [S]
GR00T N1 — arXiv:2503.14734 [S]
ManipulationNet — arXiv (Mar 2026) [S] (sim overestimation; benchmark gap)
[K]: RT-1/RT-2; Open X-Embodiment; UMI (Chi et al. 2024); MimicGen/DexMimicGen; OpenAI Dactyl/Rubik's; LEAP Hand; GelSight/Digit

Part 05 — Deformable & Soft-Body Manipulation

Robot Policies field guide. The [V]/[S]/[K] evidence legend appears above and applies throughout.

Sourcing note: the adversarial-verification budget in the research pass did not reach this part's claims, so nothing here carries a [V] tag — but every [S] claim below was extracted from a primary paper with a verbatim quote. Treat confidence as one notch below Parts 1–4.

0. Two meanings of "soft," disambiguated first

Soft objects as manipulation targets (this part's core): cloth, rope/cables, food, tissue — a rigid robot manipulating deformable stuff.
Soft robots as actuators (§6): grippers and bodies made of compliant material — deformable stuff doing the manipulating.

They share continuum mechanics but are different research communities. When someone says "soft-body manipulation is unsolved," they nearly always mean the first.

1. Why deformables break everything you learned in Part 4

Rigid manipulation is hard; deformable manipulation removes the one mercy rigid objects grant you — a finite state.

Infinite-dimensional state. "Unlike rigid objects whose poses can be fully represented by low-dimensional vectors [6-DoF], deformable objects have an infinite configuration space that is prone to severe self-occlusion" [S] (deformables survey, arXiv:2312.10419). A shirt's "pose" is a continuum surface; a folded shirt hides most of itself from every camera.
Perception, modeling, and control fail simultaneously. "Deformable objects exhibit infinite dimensionality, dynamic shape changes, and complex interactions with their environment, posing significant hurdles for perception, modeling, and control" [S] (2026 DOM survey, arXiv:2602.22998). In rigid manipulation you can often solve perception (pose estimation) and control separately; here the factorization itself breaks.
Analytical models don't rescue you. Mass-spring, position-based dynamics, and FEM/continuum approaches "are not capable of accurately modeling deformable objects due to their infinite state dimensions and the difficulty in acquiring parameters in the real world" [S] — you can't system-identify a towel.
Even the goal is ill-defined. "What does it mean for a cloth to be 'folded,' water to be 'poured,' or fruit to be 'picked'? Defining these tasks in a generalizable way is an open problem at the heart of DOM research" [S]. Rigid tasks have pose goals; deformable tasks have vibes. This wrecks both reward design (RL) and evaluation (benchmarks).
Maturity gap, stated plainly: DOM "has historically been less studied than manipulation of rigid objects" given its compounded difficulty [S], and current methods are narrowly task- and object-specific rather than generalizable [S].

2. The benchmark reality check: SoftGym (CoRL 2020) [S]

SoftGym is the field's SoftGym-shaped mirror: 10 simulated deformable tasks (rope, cloth, fluids) on NVIDIA's FleX particle simulator with a Gym API, split Medium/Hard/Robot [S]. Its findings remain the cleanest statement of why the standard playbook fails:

Vision-based RL flops: CURL-SAC, DrQ, and PlaNet perform "far below the optimal performance on many tasks," especially StraightenRope, SpreadCloth, FoldCloth — with learning curves suggesting more training won't close the gap [S].
It's not (just) the perception: a Full-State Oracle feeding ground-truth positions of all particles to an MLP+SAC policy "performs poorly on all tasks" — the high-dimensional state itself defeats model-free RL. A hand-designed Reduced-State Oracle (4 cloth corners; 10 rope keypoints) succeeds only when the reduction happens to capture task-relevant structure [S]. Moral: representation, not algorithm, is the bottleneck — the finding that motivated graph networks (§3).
Practicalities: ~4× real-time with rendering on a 2080Ti; 1M sim steps = 6 h wall-clock vs ≥35 h of real-robot collection; 1000 pre-sampled task variations each [S].

Successor evidence from DaXBench (differentiable-physics benchmark) [S]: gradient-through-physics methods dramatically beat PPO on deformables — Analytic Policy Gradients reach reward 1.00 vs PPO's 0.34 on Whip-Rope; Short-Horizon Actor-Critic 0.91 vs PPO's 0.32 on Pour-Water; demonstrations + differentiable physics (ILD) achieve 0.85 on sparse-reward Fold-T-shirt [S]. The PPO recipe that owns locomotion does not carry over — deformables reward methods that exploit structure (gradients, models, demonstrations) rather than brute sampling.

3. The method that worked: learned particle/graph dynamics + planning

The dominant successful pattern in deformables is model-based: represent the object as particles/mesh nodes, learn the dynamics with a graph neural network (message passing ≈ local physical interaction), then plan through the learned model (MPC/shooting) [K, pattern; instances below].

Flagship instance — RoboCook (CoRL 2023 best-systems-paper line of work) [S]:

Task: long-horizon elasto-plastic manipulation — a Franka arm making dumplings and alphabet-letter cookies from dough with 15 3D-printed tools.
Method: point-cloud scene → ~300 surface particles → GNN dynamics model, + PointNet++ tool classifier + self-supervised policy nets [S].
Data efficiency: ~20 minutes of real interaction data per tool [S] — contrast π0's 10,000 hours; models-of-structure buy orders of magnitude.
Results: human-evaluated 0.90 ± 0.11 vs 0.17–0.54 baselines on letter shaping; planning in 9.3 s vs 600–1900 s for baselines; robust to a human deforming/replacing the dough mid-task; generalizes across Play-Doh, clay, foam without retraining [S].

Earlier landmarks in the same family [K]: NeRP/VCD for cloth, FlingBot (dynamic flinging to unfold), Ha & Song's smoothing lines, goal-conditioned Transporter Nets for rope/cloth rearrangement.

4. The 2025 turn: diffusion everything, transformers eat the GNN

Two sourced 2025 signals show your generative-AI world arriving here:

UniClothDiff (arXiv:2503.11999) [S] — cloth manipulation as three diffusion models: a Diffusion Perception Model (generative state estimation of the full cloth state from partial RGB-D — perception as conditional generation, exactly your inpainting intuition), a Diffusion Dynamics Model, and an MPC on top. Notably it replaces GNN dynamics with transformers, arguing graph locality throttles long-range propagation; reported ~10× lower long-horizon prediction error than GNN baselines (36-step MSE 0.05 vs 0.75 on cloth), and 9/10 real-robot success on square-cloth and T-shirt folding (vs 2–6/10 baselines) with zero-shot sim-to-real from ~200–500K SAPIEN samples [S].
Dynamics-Informed Diffusion Policy (arXiv:2505.17434) [S] — diffusion policy for dynamic 3D deformable control (whip-like continuum object), learning inverse dynamics in a 20-DoF reduced-order Cosserat-rod space with physics-informed test-time adaptation; 55K simulated trajectories; simulation-only, with success falling from ~94% (10 cm tolerance) to ~62% (2 cm) — precise dynamic deformable control isn't solved even in sim [S]. The paper also documents that most prior deformable work restricts itself to 2D/planar settings for tractability [S].

Pattern to notice: the rigid-manipulation stack (generative policies, big models) is diffusing into deformables with a lag of ~2 years, but model-based structure survives here (diffusion dynamics + MPC, reduced-order physics) because the state problem never went away.

Cloth benchmark anchor [S]: the deformables survey's cited SOTA bimanual garment-folding system (2022-era): 93% success folding randomly placed garments, ~120 s average, after learning from 4,300 human-labeled grasp actions.

5. Application domains, honestly assessed [K unless tagged]

Cloth/laundry: the most-studied; folding from flattened states works (§4); crumpled-to-folded pipelines and garment diversity remain hard. π0's laundry-folding demos [S] show the end-to-end BC route can reach cloth without explicit deformable modeling — at 3 orders of magnitude more data.
Rope/cables: routing, untangling, knot-tying (surgical suturing lineage); industrial cable harness assembly is a live commercial target.
Food/cooking: RoboCook's dough [S]; cutting, scooping, and granular media are active; food is where deformable-DOM meets safety/hygiene constraints.
Surgical robotics: tissue is the ultimate deformable. Supervised autonomy exists (the STAR system's autonomous laparoscopic bowel anastomosis, 2022 [K]); regulatory and safety constraints keep learned policies far from clinical autonomy.
Long-horizon composition is the survey-flagged frontier: "compositional generalization for subskills is a significant challenge for long-horizon tasks" [S].

6. Soft robots as actuators [K]

The inverse problem: compliant hardware (pneumatic grippers, granular jamming, Festo-style continuum arms) makes grasping easier by conforming to objects — morphological computation replacing control precision — at the cost of hard modeling/proprioception problems (infinite-DoF self-state). Learning-based control of soft actuators is its own field; reduced-order models like the Cosserat rods of §4.2 bridge both communities. For policy purposes: soft grippers simplify the action space; soft objects explode the state space.

7. Maturity scorecard vs. rigid manipulation

Axis	Rigid (P4)	Deformable
State representation	Solved-ish (6-DoF pose + shape priors)	Open; infinite-dim, self-occluding [S]
Simulator fidelity	Weak for contact [S]	Weaker (FleX/MPM particle approximations) [K]
Dominant paradigm	End-to-end BC/VLA [V]	Model-based: learned dynamics + planning [S], now diffusion-hybrid [S]
Data cost exemplar	π0: ~10⁴ hours [S]	RoboCook: 20 min/tool [S] (structure ≫ scale)
Benchmarks	Fragmented but numerous [S]	SoftGym/DaXBench, sim-only, aging [S]
Goal specification	Mostly definable	Open research problem [S]
Generalist policies	Emerging (VLAs)	None; task/object-specific [S]

Bottom line: deformable manipulation is where rigid manipulation was around 2019 — pre-foundation-model, benchmark-fragmented, structure-hungry — but with a harder theoretical core (the state space itself). If manipulation is the frontier, deformables are the frontier's frontier; anyone claiming a "solved" story here is selling something.

References (this part)

Deformables survey — arXiv:2312.10419 (Neurocomputing) [S]
2026 DOM survey — arXiv:2602.22998 [S] (goal-definition problem; DaXBench numbers)
SoftGym — Lin et al., CoRL 2020, PMLR v155 [S]
RoboCook — arXiv:2306.14447 [S]
UniClothDiff — arXiv:2503.11999 [S]
Dynamics-Informed Diffusion Policy — arXiv:2505.17434 [S]
π0 laundry demos — pi.website [S]
[K]: FlingBot; VCD; Transporter Nets; STAR surgical system (2022); Festo/jamming grippers; MPM/FleX simulation lineage

Part 06 — Frontiers & Cross-Cutting Themes

Robot Policies field guide. The [V]/[S]/[K] evidence legend appears above and applies throughout.

Four themes cut across Parts 3–5 and define where the field is heading: hierarchy, the reality gap, the evaluation crisis, and the scaling-laws question.

1. Hierarchy: System 2 over System 1

The single clearest architectural trend of 2024–2026: split the policy into a slow deliberative layer (VLM/VLA reasoning about what to do) and a fast reactive layer (generating motor actions), echoing Kahneman's System 2/System 1.

Why it emerged — sourced motivation: monolithic end-to-end VLAs struggle with (a) efficient real-time inference, (b) pretraining cost, and (c) end-to-end fine-tuning complexity on embodied data (domain shift, catastrophic forgetting) [S] (dual-system VLA taxonomy). You already saw the physics reason in Part 1: control needs 50–500 Hz, billion-parameter models deliver 5–10 Hz. Hierarchy is the rate adapter.

Flagship: GR00T N1 (NVIDIA, arXiv:2503.14734, Mar 2025) [S]

Dual-system VLA, jointly trained end-to-end: System 2 = Eagle-2 VLM (SmolLM2 LLM + SigLIP-2 encoder; 1.34B of 2.2B total params) interprets vision + language at 10 Hz; System 1 = diffusion transformer trained with action flow matching generating closed-loop actions at 120 Hz (chunks of H=16, K=4 inference steps, 63.9 ms per chunk on an L40, bf16).
Trained on the data pyramid: 88 h in-house GR-1 teleop, expanded ~~10× to 827 h via neural trajectories (world-model-generated data — your Cosmos intuition operationalized), + 780K simulation trajectories (~~6,500 h), + human egocentric video; ~50K H100-hours of pretraining.
Results (self-reported): beats Diffusion Policy on RoboCasa (32.1% vs 25.6%), DexMimicGen (66.5% vs 56.1%), and real Fourier GR-1 bimanual tasks (~76.6% success; ~+30 pts over DP full-data) [S].

Figure's Helix [S/K]: a dual-system VLA for humanoid control — published only as a company project page, no paper [S]; public materials describe a ~7B VLM System 2 at 7–9 Hz commanding an ~80M visuomotor System 1 at 200 Hz [K — company-reported, unverified]. Treat all Helix numbers as marketing-grade.

The research lineage is dated and real [S]: LCB (IROS 2024), HiRT (CoRL 2024), RoboDual (Oct 2024), OpenHelix survey (May 2025) — hierarchical VLA is a recognized paradigm, not just two companies' branding. π0 itself runs a hierarchical mode: a high-level VLM issues language subcommands to low-level π0 [S].

The full-stack humanoid picture (synthesis): VLA System 2 (10 Hz) → action-chunk System 1 (50–120 Hz) → whole-body tracking policy à la HOVER/ExBody2 (50 Hz) → PD loops (0.5–1 kHz). Locomotion (P3) and manipulation (P4) research are converging into layers of one stack — HOVER explicitly positions kinematic motion tracking as the common abstraction that upstream systems (teleop, VLAs) command [V].

2. The sim-to-real gap — one gap, two very different statuses

Verified asymmetry [V]: For locomotion, train-in-sim → zero-shot transfer is standard working practice (Part 3's entire evidence base). For VLA manipulation, "the sim-to-real gap remains a core obstacle for deploying VLA policies, as discrepancies between simulated and real-world dynamics (friction, latency, actuation response) and perception (illumination, textures, sensor noise) severely degrade policy transfer" — with domain randomization, fidelity enhancement, and robust representations offered as mitigations, not solutions [V] (Dec 2025 VLA survey; corroborated by the 2026 Annual Reviews "Reality Gap" survey).

Why the asymmetry [K, synthesis]: locomotion's contact set is small and rehearsable (feet × ground), its observations are proprioceptive (easy to simulate), and randomization covers the residual. Manipulation's contact set is combinatorial (any gripper–object–scene triple), its observations are photometric (hard to simulate), and — per ManipulationNet — sim benchmarks systematically overestimate real capability because contact dynamics are imperfectly approximated [S].

Live directions [K/S]: real-to-sim-to-real (ASAP's delta-action dynamics correction [K]); identification-over-randomization (ETH's no-DR transfer via actuator physics, Part 3 [S]); and world-model evaluation (assessing policies inside learned simulators).

3. The evaluation crisis: no ImageNet of robotics

Sourced from the ManipulationNet paper (Mar 2026) [S]:

"Progress toward general manipulation systems remains fragmented due to the absence of widely adopted standard benchmarks" — as of early 2026, robot manipulation has no standard benchmark, even for ostensibly identical tasks.
The impossible trinity: existing benchmarks achieve at most two of realism (real-world evaluation), authenticity (verifiable standardized execution), and accessibility (broad participation) — the claimed root cause of the fragmentation [S].
Even at launch, its own baselines were task-specific methods — real-world standardized evaluation of generalist policies (π0, OpenVLA, GR00T) was still nascent [S].

Practical implication for reading papers [K]: success rates are not comparable across papers (different tasks, objects, initial-state distributions, human graders); simulation numbers inflate; and independent re-evaluations can crater self-reported results (HOVER vs BumbleBee, Part 3 [V-caveat]). Robotics circa 2026 is pre-GLUE: trust relative ablations within a paper far more than absolute cross-paper claims.

4. Scaling laws: hypothesis, not law

What your LLM intuition wants — clean loss-vs-compute power laws — does not yet exist for robot policies. The sourced state of play:

Embodiment scaling (locomotion) [S]: CoRL 2025 study — ~1,000 procedurally generated embodiments (humanoids, quadrupeds, hexapods), per-embodiment RL experts distilled into one generalist; generalization to unseen embodiments improves with the number of training embodiments, and the policy zero-shot transfers to real robots. The authors explicitly frame this as "preliminary empirical evidence for embodiment scaling laws" — hypothesis, not established law [S].
Data scaling (manipulation) [K]: UMI-era studies report power-law-ish generalization gains with demonstration diversity (objects/environments, not just count); π0's 10K hours [S] and GR00T's pyramid [S] are bets that scale wins, but no one has published a robotics Chinchilla. The confound: quality/diversity/embodiment matter more than token count, and evaluation noise (§3) obscures the exponent.
The honest 2026 statement: scaling helps; laws unproven. Anyone quoting "robotics scaling laws" as settled is extrapolating from n≈3 points on a noisy benchmark.

5. Where the field is heading, 2026 edition

A defensible short list, each anchored to material from this guide:

Foundation policies consolidate the stack — one pretrained VLA post-trained per robot (π0's LLM-style pretrain→post-train recipe [S]; GR00T's open release [S]), with the locomotion layer (HOVER-style tracking controllers) as the "device driver" underneath [V].
RL returns on top of imitation [K]: RL fine-tuning of VLAs (improving beyond demonstration quality, optimizing real task reward) is the visible next wave — early 2025 work exists (e.g., RL-tuned VLA papers, preference-based post-training), but nothing with verified flagship status yet; watch this space.
World models as data engines and evaluators — neural trajectories [S] today; policy evaluation inside learned simulators next [K].
Cross-embodiment as the generalization axis — OXE pooling [K], π0's 7 platforms [S], embodiment-scaling distillation [S]; the bet is that embodiment diversity is to robotics what language diversity was to LLMs.
Loco-manipulation closes the P3/P4 seam — flagged as open by the locomotion survey [V]; humanoid whole-body VLAs (GR00T, Helix) are the current attempt.
The unsexy blockers decide the timeline: evaluation (§3), sim-to-real for contact (§2), safety certification [V] (survey open-problem list), and tactile integration [K]. Progress here won't demo well but will gate deployment.

References (this part)

GR00T N1 — arXiv:2503.14734 [S]
Dual-system VLA taxonomy — github.com/OpenHelix-robot/awesome-dual-system-vla [S] (LCB, HiRT, RoboDual, OpenHelix dates; Helix status)
π0 — arXiv:2410.24164 [S] (hierarchical mode)
VLA anatomy survey — arXiv:2512.11362 [V] (sim-to-real status)
The Reality Gap in Robotics — arXiv:2510.20808 (Annual Reviews 2026) [V-corroboration]
ManipulationNet — arXiv (Mar 2026) [S] (benchmark crisis; impossible trinity)
Embodiment scaling laws — CoRL 2025, PMLR v305 [S]
Locomotion survey — arXiv:2406.01152 [V] (safety, loco-manipulation, foundation models as open)
HOVER — arXiv:2410.21229 [V]; BumbleBee — arXiv:2506.12779 (re-evaluation caveat)
[K]: Figure Helix public materials; ASAP (2025); UMI data-scaling study (2024); RL-fine-tuning-of-VLA early literature (2025)

Part 07 — Case Study: Every Concept in This Guide, Located in Your Own Repos

Robot Policies field guide. This part maps Parts 1–6 onto the code in ~/wagmi/loco-mujoco and ~/wagmi/cyber-ide — file paths verified against the actual checkouts (July 2026).

The fastest way to make this guide concrete: ~/wagmi/loco-mujoco and ~/wagmi/cyber-ide already implement most of Part 2's taxonomy. Read them side-by-side with the parts.

1. loco-mujoco = Parts 1–3 in executable form

LocoMuJoCo (checked out at ~/wagmi/loco-mujoco) is an imitation-learning benchmark for whole-body locomotion — 12 humanoids (incl. UnitreeG1), 4 quadrupeds, 22K+ retargeted mocap trajectories.

The policy object (Part 1)

Observation space — loco_mujoco/environments/humanoids/unitreeG1.py::_get_observation_specification: the G1's default observation is exactly Part 1 §3's proprioception: FreeJointPosNoXY (root pose, 5-dim — note x,y excluded: position-invariance), 23 joint positions, FreeJointVel (6), 23 joint velocities → 57-dim. No cameras. A locomotion policy in its natural habitat.
Action space — two registered ControlFunctions (Part 1 §4 made flesh):
- core/control_functions/default.py::DefaultControl — normalize [-1,1] → actuator (torque) range: the direct-torque exception.
- core/control_functions/pd.py::PDControl — the dominant pattern: policy emits normalized joint-position targets; generate_action computes ctrl = p_gain·(target_q − q) − d_gain·q̇, and run_with_simulation_frequency = True makes the PD math run at the simulation rate (1 kHz at default timestep=0.001) while the agent acts every n_substeps=10 steps (10 ms → 100 Hz). That is precisely the policy-over-PD frequency hierarchy of Part 1 §5, one flag away.
The carry as POMDP machinery: AdditionalCarry (flax dataclass in core/mujoco_base.py) threads stateful observation/reward/randomizer state through a functionally-pure step — the JAX-native answer to "policies need memory."

The training substrate (Parts 2–3)

Massively parallel sim RL: core/mujoco_mjx.py::Mjx — vmapped/jitted MJX stepping with in-step asynchronous resets (jax.lax.cond(state.done, self._mjx_reset_in_step, ...)), plus optional MjWarp. This is the same GPU-parallelism thesis as Isaac Gym's "walking in minutes" [V], in JAX. MjxUnitreeG1._modify_spec_for_mjx shows the cost of GPU contact: foot meshes → capsules, all contacts pruned except feet-floor — remember this when Part 6 says contact fidelity is the sim-to-real crux.
PPO dominance: algorithms/ppo_jax.py::PPOJax — a single-file JAX PPO where env and training compile into one jitted function. Part 2 §1.1's consensus, instantiated.
Adversarial IL: algorithms/gail_jax.py::GAILJax (discriminator over transitions, policy rewarded for fooling it — Part 2 §4's GAIL) and algorithms/amp_jax.py::AMPJax — a ~30-line subclass swapping in the least-squares discriminator loss and the max(0, 1 − 0.25(D−1)²) reward: AMP is literally GAIL with a different loss, and the code makes that lineage visible.
DeepMimic tracking: core/reward/trajectory_based.py::MimicReward — exponential kernels exp(−w·dist²) over qpos (with proper quaternion angular distance), qvel, and relative site positions/orientations/velocities, plus torque/action-rate/out-of-bounds penalty terms. Compare term-for-term with ExBody2/HOVER's tracking objectives [V] — same family, and the reward-shaping zoo of Part 2 §1.1 is visible in its ~12 weight kwargs.
Mocap as pretraining data: task_factories/imitation_factory.py pulls AMASS/LAFAN1/native datasets (HuggingFace-hosted), retargeted per robot; trajectory/handler.py::TrajectoryHandler filters/reorders/interpolates trajectories to match the model — the unglamorous plumbing behind "trained on retargeted human MoCap" claims in every humanoid paper.
Domain randomization & terrain: core/domain_randomizer/ (DefaultRandomizer; note PDControlState.p_gain_noise/pos_offset/ctrl_mult — gain/offset randomization hooks straight into the PD controller) and core/terrain/RoughTerrain — Part 2 §6's toolkit.
Init/termination: TrajInitialStateHandler = DeepMimic's reference-state initialization; RootPoseTrajTerminalStateHandler = early termination on tracking divergence — two under-cited tricks that make mocap-tracking RL converge [K].

What loco-mujoco does not contain (gap-spotting as a learning exercise)

No teacher–student distillation pipeline (Part 2 §5 — the sim-to-real workhorse [V]), no privileged-observation split, no actuator modeling, no hardware deployment path. It is a benchmark for policy learning research, not a deployment stack — which is exactly why its policies stay in MuJoCo. To get an ExBody2-style G1 result you would add: privileged teacher obs → DAgger student on observation histories → export to the real-robot SDK.

2. cyber-ide = what a robotics stack looks like below and around the policy

The Cybernetic IDE repo contains no learned policy at all — and that makes it the perfect foil. Its Dockerized G1 MuJoCo harness (overlays/unitree-g1-mujoco-protocol/python/g1_protocol_sim.py) implements the classical layer of Part 1 §2:

A PD + gravity-compensation controller, not a policy: apply_hold_control_locked computes tau = qfrc_bias + kp·(target − q) − kd·q̇ per actuator — a hand-tuned controller tracking a given target. It "decides" nothing; per Part 1's distinction, it's a controller. Its documented failure mode ("poses whose center of mass falls outside the support will topple — by design, since there is no whole-body balance controller") is precisely the hole a learned whole-body policy (HOVER/ExBody2-style, or a loco-mujoco-trained tracker) would fill.
Kinematic locomotion: apply_loco_velocity_locked teleports the floating base — the simulator honestly documents that real velocity commands would require the balance/locomotion policy it doesn't have.
The SDK boundary as the future policy seam: the unitree_sdk2py-shaped facade (LocoClient.SetVelocity, rt/lowcmd joint targets with kp/kd) mirrors the real G1 interface — which is exactly where a trained policy's 50 Hz joint-target stream would plug in (ExBody2 drives the same 23 joints through the same kind of interface, on the same robot [V]).
Cybernetic IDE's own todo.md states the gap in guide vocabulary: "Whole-body balance control for low-level commands" — not complete.

The synthesis opportunity (and the reason these two repos share a filesystem): train a G1 tracking policy in loco-mujoco (MjxUnitreeG1 + MimicReward or AMP + PPO on LAFAN1/AMASS clips), then serve its 50–100 Hz joint-position targets through cyber-ide's rt/lowcmd path in place of the static held poses. That single pipe would touch every part: mocap data (P2 §4), parallel-sim PPO (P3 §1), PD-target actions (P1 §4), the sim-to-sim gap between MJX's pruned contacts and the harness's full-mesh 29-DOF scene (P6 §2 — note also the 23-DOF vs 29-DOF joint-mapping problem), and eventually a VLA or agent issuing the commands up top (P6 §1). The neighboring docs g1-yoga-policy-training.md / g1-yoga-policy-handoff-prompt.md in this directory sketch exactly such an effort.

3. Suggested exercises (PhD-problem-set style)

Feel the action space (P1): train MjxUnitreeG1 twice with identical PPO configs — control_type="DefaultControl" (torque) vs "PDControl" (position targets). Compare sample efficiency, final gait quality, and torque smoothness. Predict the winner from Part 1 §4.1 before running.
BC vs. tracking-RL (P2): take one LAFAN1 walking clip. (a) Behavior-clone qpos targets directly; (b) train MimicReward PPO. Measure survival time under a 20 N lateral push. Explain the gap via covariate shift and the embodiment-mismatch argument (P2 §4).
AMP ablation (P2): diff gail_jax.py and amp_jax.py (it's ~30 lines). Swap the AMP reward back to the GAIL log-loss reward and observe discriminator saturation effects on gait naturalness.
Contact fidelity audit (P6): run the same trained policy in MjxUnitreeG1 (capsule feet, pruned contacts) and CPU UnitreeG1 (full meshes). Quantify the sim-to-sim performance drop; relate it to the ManipulationNet claim that contact approximation inflates capability [S].
Close the loop (P7): stream a loco-mujoco-trained policy's joint targets into cyber-ide's /command lowcmd endpoint at 50 Hz and watch Part 1 §5's hierarchy — your policy over the harness's PD loop — render live in the Robot Viewer.

End of guide. The opening one-paragraph thesis, re-read now, should feel like a summary of things you know rather than a preview of things you do not.

Field notes

Occasional dispatches on agent control loops, simulator infrastructure, robot development systems, and the practical details behind what ships.