Turning Codex app-server into simulation workflows

The moment the abstraction clicked was not in a benchmark. It was while I was waiting for the bus in San Francisco.

I opened our site on my phone over 5G and kicked off a workflow:

Create an environment that looks like a high-end yoga studio, put a Unitree G1 in the middle on a yoga mat doing a yoga pose, and make sure the floor and mat are the ground so the robot does not clip through it.

That is a goofy prompt. It is also a pretty good product test.

A phone should be a thin client. A browser tab should not need to know how to allocate a GPU session, wait for Isaac Sim, mint tool access, start a Codex runtime, inspect a scene, repair geometry, save the resulting environment, and keep a run page coherent while the network is doing whatever 5G does near a bus stop.

The workflow should own that.

The result was not a polished demo reel. It was better than that: it was a real run, inside the product, driving a live Isaac Sim scene from a mobile browser.

That is the thing I want more agent systems to take seriously. A useful agent product is rarely "model call plus UI." It is usually a control loop wrapped in boring infrastructure: lifecycle, identity, progress, permissions, logs, retries, observers, artifacts, and a way to decide whether the agent actually did the thing.

This post is about how the Codex packages I wrote fit into that shape, and how the same ideas show up in our Workflows feature for building simulations.

I am going to stay at the broad architectural level because parts of the Cybernetic Physics control plane are not public. The public lesson is the boundary, not the private wiring.

The first bug was a lost event

The original reason I wrote codex-control was embarrassingly small: a race.

If you send turn/start to codex app-server and subscribe afterward, the earliest events can arrive before your collector exists. For a chat UI, that is a visual glitch. For a rollout collector, it is data loss. The trace no longer proves what happened.

That bug is the whole philosophy of the library in miniature.

codex exec is a good human interface. It is not a good systems boundary. A serious harness needs to talk about sessions, threads, turns, subscriptions, interrupts, forks, approvals, workspaces, and traces as objects with lifetimes.

The tiny example looks boring:

import asyncio
from codex_control import CodexSession

async def main():
    async with CodexSession.spawn() as session:
        thread = await session.start_thread()
        turn = await thread.run_turn("Reply with the word: pong")
        print(turn.final_text)
        await thread.archive()

asyncio.run(main())

The boring part is the point. Once a Codex run is a runtime object instead of a subprocess that eventually prints text, the rest of the system can make stronger claims:

• Subscribe before start, so early events are not lost.
• Interrupt without destroying the partial trace.
• Steer the active turn only if the turn id still matches.
• Fork from persisted parent state instead of replaying a prompt and hoping it lands nearby.
• Snapshot the workspace before grading.
• Preserve the model traffic needed to understand how a rollout was sampled.

These are not flashy features. They are the difference between an agent demo and an agent system.

Why this matters for simulations

Robotics simulation makes agent sloppiness expensive.

If a text-only agent says "I built a scene," that sentence is cheap. If a simulation workflow says it built a scene, there should be a stage, a robot, assets, transforms, collisions, grounding, camera evidence, a saved environment, and a run record that explains how it got there.

In practice, simulation work has at least four different kinds of state:

• Product state: who launched the run, which workspace owns it, what status it is in, and what the user should see next.
• Simulator state: the live Isaac session, the stage, prims, materials, physics, cameras, and viewer.
• Agent state: Codex threads, turns, tool calls, observations, feedback, and partial failures.
• Artifact state: saved environments, USD stages, screenshots, traces, logs, and evaluation results.

The failure mode is to cram all of that into one prompt and call it "the agent." That can work once in a screen recording. It does not give you a product surface you can trust.

The better shape is a workflow.

The workflow is not just "run Codex." It is a source-defined recipe with an input contract, a launch view, progress semantics, an execution boundary, and a result view. The user sees one thing: "build me this simulation." The system sees a sequence of responsibilities.

For simulation_from_prompt, the broad shape is:

1. Get or create a simulator session.
2. Wait until the simulator is genuinely ready to be operated, not merely allocated.
3. Give the agent the minimum tool access needed for that session.
4. Run a builder/judge loop against the live scene.
5. Save the finished environment and surface the result.

That sequence is intentionally ordinary. It is easy to reason about. It gives us a place to put failures. It gives the UI a progress model that is not fake.

The Workflows feature

In the product, Workflows are shipped definitions rather than arbitrary blobs of client-side form logic.

The client asks which workflows exist. Each definition describes the kind of run, the form fields, the default budget, the visible steps, and the result view. The launch page can render the workflow without knowing every private implementation detail behind it.

For simulation_from_prompt, the launch modes are deliberately concrete:

• New session: allocate a managed Isaac Sim session and let the workflow own it.
• Existing session: attach to a session the user already has running.
• From environment: start from a saved environment version and publish a new one when the workflow finishes.

That "existing session" mode matters more than it looks. A workflow run is not the same thing as a simulator session. The former is a durable process record. The latter is the live world the agent can operate. Keeping that distinction explicit prevents a lot of product confusion.

The run page is similarly practical. It shows status, prompt, elapsed time, budget, step progress, recent events, structured result fields, and the embedded Isaac viewer. On desktop that gives you a cockpit. On mobile it gives you enough surface area to launch, sync, cancel, and watch without pretending the phone is doing the heavy work.

What I like about this design is that the UI is not trying to be the workflow engine. It is a reader and a controller. It creates a run, shows the current state, polls for progress, and lets the user cancel or inspect. The execution is elsewhere.

That is the same lesson as codex-control: handles beat transcripts. The product does not need "the final assistant message." It needs a handle to the run.

Builder and judge is the smallest useful loop

The first version of a simulation agent can be a single Codex turn:

User prompt -> Codex edits/operates scene -> Done

That is seductive and brittle. The agent can claim success without checking the viewer. It can create plausible-looking objects with bad transforms. It can forget the request halfway through. It can load an asset that matches the word but not the intent. It can leave the robot clipping through the floor and still end with a confident paragraph.

The workflow uses a builder/judge shape instead:

builder attempts scene change
judge inspects live simulator and viewer
judge emits structured verdict
builder receives repair feedback
repeat until satisfied or budget exhausted

The judge is not there to be philosophical. It has a specific job: do not trust the builder's final message. Inspect the live scene. Look at the viewer. Check that the simulator state contains evidence for the user's request. Return structured feedback the next builder attempt can use.

That is enough to unlock a better product behavior. The agent can be wrong locally without the whole workflow being wrong globally. A bad first attempt becomes feedback. Missing grounding, weak asset choice, no viewer check, or a robot clipping through the mat becomes a repair instruction.

This is also where Codex as a coding agent is useful in a robotics setting. The workflow can give Codex a real workspace, real tools, a live desktop surface, and a simulator API. It can write helper scripts, inspect stage state, use computer-use to look at the viewer, call domain tools, and leave artifacts behind.

The agent is not merely generating a USD file from text. It is operating a running system.

Where codex-control fits

codex-control sits at the agent-runtime boundary.

It knows how to connect to codex app-server, start threads, run turns, collect events, install plugins, handle approvals, interrupt, steer, and fork. Those verbs are useful in a local research harness, but they are also exactly the verbs you want inside a product workflow.

A workflow runtime needs to say:

• Start Codex in this workspace.
• Give it these tools.
• Run this role-specific prompt.
• Watch the emitted items.
• Bound the turn.
• Parse structured output when the role requires it.
• Preserve enough trace to debug failures.

Without a runtime handle, you end up with either a giant shell command or a custom integration for every workflow. With a handle, the recipe can be ordinary Python: allocate, prepare, run, judge, finalize.

That is also why research-swarm was a good first proof. It uses Codex as a controller while Python owns the deterministic tool layer: search, fetch, archive, citation checks, run records, and local storage. Codex picks the next action. The harness validates and executes it.

The same split appears in simulation workflows. Codex can decide how to build and repair a scene, but the product owns session lifecycle, access control, progress, artifact saving, and result display.

The agent should be powerful inside a narrow room. The workflow builds the room.

The rest of the package stack

The other packages exist because different failure modes belong at different boundaries.

codex-orchestrate is the conductor. It is about staging workspaces, starting sessions, snapshotting results, invoking verifiers, and cleaning up. The important design idea is that grading should happen against a frozen artifact, not a live directory that can keep changing after the agent says it is done.

codex-env is the task contract. It separates prompts, public workspaces, private evaluator files, expected failures, and verifier results. That separation is boring until you need it, and then it is everything.

codex-proxy is the model wire. It translates Responses API traffic, serves streaming events in the shape Codex expects, and records sampling-time information. If you care about training, you need to know which policy produced which completion and what the behavior logprobs were at sampling time.

codex-train is the learning loop. It consumes rollout groups, drops invalid training data, materializes datums, and updates a backend policy. Its job is not to launch Codex. Its job is to refuse to train on data whose provenance does not satisfy the loss.

Together, these packages sketch a stack:

codex-control      runtime handles for Codex
codex-orchestrate  rollout and verifier lifecycle
codex-env          task and reward contracts
codex-proxy        Responses/SSE serving and policy traffic capture
codex-train        policy updates from valid rollout data
research-swarm     a concrete local-first harness using Codex as controller

The Cybernetic Physics Workflows feature is not a public dump of that whole stack. It is a product-shaped application of the same premise: put Codex behind a durable workflow boundary, connect it to real tools, inspect the world it changes, and save the resulting artifact.

The phone test as an architecture test

The Unitree G1 yoga scene is funny, but it exercises the right things.

The prompt is underspecified. "High-end yoga studio" is taste, not a CAD spec. "Doing a yoga pose" can fail visually even if the robot exists. "Floor and mat are the ground" is a physics requirement disguised as casual language. "Does not clip through it" requires inspecting transforms and the rendered scene.

That forces the workflow to coordinate several layers:

• A user-facing run has to be created quickly and return control to the phone.
• A simulator session has to become actually ready before Codex touches it.
• The agent needs a workspace and tool surface specific to that session.
• The builder needs to make concrete scene changes.
• The judge needs to inspect both programmatic scene state and visual evidence.
• The finalizer needs to save an environment the user can come back to.

If any of those layers is missing, the demo becomes fake in a way that engineers can smell.

The nice part is that the product surface stays small. From the phone, I saw a prompt, status, elapsed time, step progress, and a viewer. Underneath, the workflow carried the heavy state.

That is the UX I want for simulation design. Start from intent. Get a durable run. Watch the scene become real. Iterate from artifacts, not from screenshots pasted into a chat.

Why not just make the agent smarter?

Better models help, but they do not remove the need for structure.

A stronger model still needs the right tools. It still needs to know which simulator session it is allowed to touch. It still benefits from a judge that is required to inspect evidence. It still needs a place to save artifacts. It still needs a trace when something fails after 18 minutes of GPU time.

The workflow boundary is how you keep model progress from becoming product chaos.

There is a useful analogy to CI. A good compiler does not eliminate the need for build jobs, logs, artifacts, test reports, retries, and permissions. The compiler is the clever part, but the pipeline is what makes the clever part usable by a team.

Simulation workflows are the same. Codex can be the clever part. The workflow is the pipeline.

What we deliberately keep out of the public article

Some details are not interesting to publish and not wise to publish.

The exact private endpoint names, credential paths, deployment knobs, provider fallbacks, and internal data model are not the lesson. The lesson is the shape:

source-defined workflow
  -> durable run identity
  -> remote execution boundary
  -> live simulator tools
  -> builder/judge loop
  -> evented progress
  -> saved artifact
  -> result view

That shape is portable. You could swap Isaac for another simulator. You could swap the remote executor. You could add workflow kinds for scene cleanup, asset import, dynamics checks, reinforcement-learning task generation, regression evaluation, or hardware teleop rehearsal.

The private plumbing should remain replaceable. The public contract should remain small.

Sharp edges

This is still early.

Simulator readiness is not the same as session allocation. A system can have a GPU slot and still not be ready for an agent to operate the desktop or simulator APIs. The workflow has to wait for the stronger signal.

Visual judgment is fragile. A judge that cannot see the live viewer should not pretend it verified the scene. Retrying the visual check is not glamorous, but it matters.

Budgets are product language and runtime language at the same time. Users need understandable limits. The runtime needs enough flexibility to continue a repair loop when the first estimate was too small, while still respecting wall-clock and safety bounds.

Saved environments are part of the contract. A workflow that leaves only a transient viewer state is much less useful than one that publishes an artifact the user can reopen, branch from, and compare.

And yes, mobile matters. Not because anyone wants to author all robotics simulations on a phone, but because a phone reveals whether the product is actually a workflow. If the client has to babysit every internal step, the mobile experience collapses immediately.

The bigger bet

I do not think the next useful robotics interface is "chat with a simulator."

I think it is closer to "launch workflows that operate simulators."

The prompt is still there. Natural language is a good way to state intent. But the prompt should enter a system that has memory, ownership, tools, observers, artifacts, and judgment.

That is what codex-control unlocked for me at the Codex layer: a way to treat the running agent as a controllable runtime. The Workflows feature applies the same idea at the product layer: a way to treat a simulation-building agent as a durable run with a live world attached to it.

The bus-stop Unitree G1 scene is a small artifact. The more interesting artifact is the boundary around it.

Once that boundary exists, workflows can get more ambitious without turning into prompt spaghetti. They can plan scenes, route assets, run physical checks, compare screenshots, repair collisions, save versions, generate tasks, train policies, and feed evidence back into the next run.

That is the stack I want to keep building: not one giant autonomous agent, but small runtime handles composed into workflows that make real things happen.

codex-control / research-swarm / codex-orchestrate / codex-proxy / codex-train / codex-env