Train with SimForge Gym

Train driving models with SimForge Gym (the open simforge-gym package) on your own machine. Three recipes, one command each: a starter that fine-tunes Qwen Drive on four scenarios, supervised fine-tuning of Qwen Drive and Alpamayo 1.5, and reinforcement learning with PPO and closed-loop GRPO.

simforge env serve runs episodes of a scenario on the local runtime, with reactive traffic and optional rendered cameras. SimForge Gym is the one place to train driving models on SimForge: supervised fine-tuning, reinforcement learning and closed-loop evaluation. Its open simforge-gym Python package (formerly simforge-rl) is the Gymnasium client: it never contains a simulator of its own, so without a running server there is no episode. Training runs on your machine; the cloud does not run it.

Install it from PyPI:

python -m pip install 'simforge-gym[ppo]'

Recipes

Each recipe is one command on your own machine. Every GPU need and time below was measured; each recipe’s section says how.

Recipe	GPU	Time
Starter: four scenarios to a fine-tuned Qwen Drive 1 4B and a before/after report card	measured on 96 GB (training fits 16 GB)	59.1 min on an RTX PRO 6000
Qwen Drive SFT, one GPU	16 GB (24 GB to prefill online)	5,000 windows on an RTX 5080: about 50 min of prefill, then 10 min per epoch
Qwen Drive SFT, several GPUs	16 GB per GPU	not measured yet
Alpamayo 1.5 SFT, stage 2	40 GB or more	2.3 s per step on an RTX PRO 6000
PPO on state observations	none	37.5 min on a 24-core CPU
GRPO in closed loop, from an SFT checkpoint	48 GB or more	about 20 min per update of 48 drives on an RTX PRO 6000

Nothing runs in the cloud or is billed. Training on SimForge cloud comes later.

Before you start

simforge login
simforge runtime install --accept-license
pip install simforge-gym

Starter, Qwen Drive fine-tuning and GRPO also need the training extras and the Qwen Drive weights. qwen_drive (Apache-2.0, not on PyPI) declares flash-attn and causal-conv1d, which do not build before torch is installed and which its sdpa path never imports, so it installs without its own dependencies:

pip install 'simforge-gym[train]' 'flash-linear-attention>=0.5.1' 'opencv-python-headless>=4.10' 'matplotlib>=3.8' 'tqdm>=4.66'
pip install --no-deps "qwen_drive @ git+https://github.com/QwenLM/Qwen-Drive-1.0@28091c1532e869bc7aee91fc0aef6b3e6fd0b2e0"
huggingface-cli download Qwen/Qwen-Drive-1.0-4B --local-dir Qwen-Drive-1.0-4B

Qwen Drive 1 4B (Qwen/Qwen-Drive-1.0-4B) is Apache-2.0. Local closed-loop evaluation (Starter, GRPO) needs simforge-gym 0.3.0 or later and the next runtime release.

Serve episodes

simforge env serve s.json --socket /tmp/simforge-env.sock --no-sensors --max-sessions 4

The server prints a ready document and runs until you stop it. --no-sensors serves state only and needs no GPU. For camera observations, serve with --rig rig.json instead (see Preview and render for rigs). --max-sessions must cover every environment you open, including the evaluation one.

On Windows use a runtime advertising capabilities.envSocketTransport: "named-pipe". The same --socket path selects a secure local named pipe with a bounded hashed name, so long profile paths do not hit the Unix socket path limit. Older Windows runtimes fail with an upgrade instruction. macOS and Linux use Unix sockets.

Drive it from Python

from simforge_gym import SimForgeEnv, SocketTransport, WeightedTerms

with SimForgeEnv(
    SocketTransport("/tmp/simforge-env.sock"),
    action_mode="setpoint",
    observation="state",
    reward_fn=WeightedTerms({"progress": 1.0, "proximity": 2.0}),
) as env:
    observation, info = env.reset(seed=7)
    observation, reward, terminated, truncated, info = env.step([5.0, 1.0])

• A setpoint action is [target_speed_mps, target_accel_mps2]; action_mode="control" takes [throttle, brake, steer]; None keeps the scenario’s scripted control.
• observation="state" is privileged engine state. "sensors" returns only rendered camera passes keyed <sourceId>/<pass>, with no engine state in info; "state+sensors" returns both.
• reward_fn replaces the reward; info["native_reward"] keeps the runtime’s own.
• Reset after an episode terminates or truncates; nothing resets on its own.

Train with PPO

python -m simforge_gym.examples.ppo --tasks richmond-field-station --runtime --num-envs 8 --total-timesteps 200000 --out run/

State observations need no GPU. --tasks picks a bundled task set: one per released map (belmont-research-center, easterbrook-discovery-school, el-camino-road, richmond-field-station, yale-street), each 24 to 29 weighted, seeded scenario templates (car following, cut-ins, lead braking, crossing pedestrians, unprotected turns, red-light runners) at sites verified on that map. A set is pinned to the map version it was verified on: pull that version. --runtime runs each worker’s episodes in its own runtime process; without it the example starts simforge env serve processes instead, with the same observations and rewards. run/ then holds curve.csv, curve.svg, metadata.json and policy.pt, and each evaluation also scores the scenarios’ scripted driving as a baseline. python -m simforge_gym.examples.ppo --help lists every flag.

The recorded baseline ran 1,228,800 environment steps in 37.5 minutes on a 24-core CPU host with no GPU:

python -m simforge_gym.examples.ppo --tasks richmond-field-station --runtime --num-envs 32 \
  --total-timesteps 1228800 --eval-every 10 --seed 0 --out run/

Mean held-out return rose from 0.79 to 2.74, against 0.71 for the scenarios’ authored driving. The trained policy beats that driving on 12 of the 24 tasks and still crashes on some.

Train on a pulled dataset

simforge datasets pull <dataset-id> --include scenarios --out data/my-dataset
python -m simforge_gym.examples.ppo --dataset data/my-dataset --split train --runtime --max-open 16 --eval-limit 8 --out run/

--dataset takes a pulled folder or a hosted dataset id, which it pulls first (see Getting data out); every file is checked against its digest, and each episode runs exactly the input its scenario’s simulation ran, on the installed map the scenario names. --max-open bounds the sessions each worker keeps open, so a split of thousands of scenarios trains with a bounded number of sessions. In Python, ScenarioSet.from_dataset(...) and ScenarioSet.bundled(...) give the same task sets.

Evaluate a driving model in closed loop

simforge_gym.closed_loop.episode drives one episode of an imported scenario with a camera-based driving model, rendering its cameras live on your GPU with simforge env serve. It is the runner SimForge cloud evaluation uses, so the episode folder is scored the same way:

python -m simforge_gym.closed_loop.episode --plan plan.json --out episode/
python -m simforge_gym.eval.score episode/

The plan names the scenario workspace, its camera ego, the rig and the model: a model host over HTTP (modelHost) or a checkpoint loaded in this process (localPolicy, which needs simforge-gym[train]). The score reports driving score, route completion, collisions, infractions and comfort. The plan fields are in the package README.

Starter recipe: fine-tune Qwen-Drive on your scenarios

One command turns a few scenarios into a fine-tuned Qwen-Drive-1.0 4B planner, then compares it with the base planner in closed loop. It all runs on your GPU; nothing runs in the cloud or is billed:

python -m simforge_gym.recipes.starter --scenarios cut-in-brake,lead-hard-brake,queue-tail,opposing-turn-encroachment \
  --base Qwen-Drive-1.0-4B/ --out starter/

The recipe runs these stages:

1. Each scenario (a bundled template or your own v2 template) is placed at the sites of your installed maps that fit it, with several parameter draws per site. One whole map is held out for evaluation.
2. The training drives (48 by default) are rendered through simforge env serve with the model’s three cameras. These are the same renderer settings the closed-loop evaluation uses.
3. The planner is fine-tuned with the vision-language model frozen. Part of each scenario’s drives is held out as a validation set, and the checkpoint with the lowest validation loss is kept.
4. The base and the fine-tuned planner each drive the held-out variants with the same seed, and every drive is scored as SimForge cloud evaluation scores it.
5. starter/report/report.html shows driving score, route completion, collisions and off-road per scenario and overall, the paired difference per drive, videos, the GPU, and each stage’s time and GPU memory peak.

Re-running the command resumes where it stopped. --baseline-only reports the base planner alone, which shows where it has room to improve. python -m simforge_gym.recipes.starter --help lists every flag.

You need:

• the CLI signed in and the runtime installed;
• two or more maps with their 3D packs (simforge maps pull NAME --for 3d --accept-license, 5 to 14 GB each) and simforge assets pull;
• an NVIDIA GPU: training fits 16 GB, and a whole run has been measured on a 96 GB one;
• the training extras and the Qwen Drive weights, and the closed-loop releases (see Before you start).

On an RTX PRO 6000 with these four scenarios, a run with 48 training drives took 59.1 minutes: 10.4 to render the 48 drives, 12.6 to train and 34.4 to drive the 12 held-out variants with both planners. Training with the prefill cache fits a 16 GB GPU (9.5 GiB peak); a full run on a smaller card has not been measured.

In that run, the fine-tuned planner collided less (4 collisions down to 2) and left the road less (6 off-road down to 1) than the base planner. It also completed less of its route, so its driving score was about the same: 0.694 against 0.710, better on 4 of 12 held-out drives. Fine-tuning teaches the planner how your scenarios are driven. The closed-loop score is what reinforcement learning (GRPO) then optimizes, starting from the recipe’s best checkpoint.

Fine-tune Qwen Drive

python -m simforge_gym.train.sft fine-tunes the Qwen Drive 1 4B planner expert with the vision-language model frozen, on renders exported with python -m simforge_gym.export --format qwen-drive (the package README covers rendering and export):

python -m simforge_gym.train.sft --data qwen-drive-data/ --base Qwen-Drive-1.0-4B/ --out runs/sft --steps 500

--prefill cache (the default) prefills the frozen model once per window to disk, about 140 MB per window, and trains in 9.5 GiB, so a 16 GB GPU fits. --prefill online needs 24 GB and no disk. For 5,000 windows on an RTX 5080, the cache takes about 50 minutes of prefill, then 10 minutes per epoch.

On several GPUs

torchrun --nproc-per-node 4 -m simforge_gym.train.sft --data qwen-drive-data/ --base Qwen-Drive-1.0-4B/ --out runs/sft --steps 500

The same flags train with DDP on one machine. --batch-size counts over all GPUs. With the cache, each GPU peaks at 10.8 GiB (measured with 2 ranks), so 16 GB per GPU fits. Throughput on several GPUs has not been measured.

Fine-tune Alpamayo 1.5

python -m simforge_gym.train.alpamayo runs NVIDIA’s Alpamayo 1.5 SFT recipe on renders exported with python -m simforge_gym.export --format pai. Stage 2 is the default: the vision-language model stays frozen and the trajectory expert trains. It runs in NVIDIA’s recipe environment, with simforge-gym installed into it:

git clone https://github.com/NVlabs/alpamayo-recipes && git -C alpamayo-recipes checkout 6172113260522fbf290db85e21989b0cdb5302b6
cd alpamayo-recipes/recipes/alpamayo1_5_sft && uv venv a1_5_sft && source a1_5_sft/bin/activate && uv sync --active && cd -
uv pip install simforge-gym gymnasium pyarrow
huggingface-cli download nvidia/Alpamayo-1.5-10B --revision 7aba8293c09993f2e125c6819df05d7fa3e873ea --local-dir Alpamayo-1.5-10B
(cd alpamayo-recipes && python scripts/convert_checkpoint.py to-a1 --input ../Alpamayo-1.5-10B --output ../Alpamayo-1.5-10B-A1)

python -m simforge_gym.train.alpamayo simforge.data=alpamayo-data/ simforge.val_data=alpamayo-val/ \
  model.pretrained_model_name_or_path=Alpamayo-1.5-10B-A1/ paths.output_dir=runs/alpamayo-stage2

On one RTX PRO 6000, stage 2 peaked at 36.5 GiB and took 2.3 s per step at batch 1 (30 steps on a synthetic export), so it needs a 40 GB or larger GPU; a 24 GB card does not hold it. Each checkpoint is about 30 GB on disk. A full training run has not been measured.

The model builds its VLM from the configuration and tokenizer of nvidia/Cosmos-Reason2-8B, a gated Hugging Face repository under the NVIDIA Open Model License: request access to it and sign in (huggingface-cli login) before the first run.

simforge-gym ships only Apache-2.0 code, never the weights: you download nvidia/Alpamayo-1.5-10B yourself at revision 7aba8293c09993f2e125c6819df05d7fa3e873ea. The LICENSE file at that revision is OpenMDW-1.1, and the model card at that revision says: “This model is ready for non-commercial use. Commercial licensing available upon request.” Review both, and the NVIDIA Open Model License of nvidia/Cosmos-Reason2-8B, before you train or use the model.

Closed-loop RL with GRPO

python -m simforge_gym.train.grpo continues from a Qwen Drive planner, such as an SFT checkpoint, with reinforcement learning in closed loop. Every rollout is a closed-loop episode scored the way the report card scores it, and the planner learns from the drives that scored better than their group:

simforge rig init --preset qwen-drive-3cam --out rig.json
python -m simforge_gym.train.grpo --base Qwen-Drive-1.0-4B/ --planner runs/sft/planner-000500 \
  --train train.json --heldout heldout.json --rig rig.json --out runs/grpo --updates 20 \
  --actor-closure "$(basename ~/.local/share/simforge/runtime/share/actors/closures/*.json .json)"

--train and --heldout are scenario sets; put the held-out set on another map. Each update drives --scenarios-per-update scenarios --group times each (default 4 x 8), --parallel drives at a time against the one loaded model, then trains the planner on what those drives scored. The held-out set is driven as an evaluation drives it at update 0 and every --eval-every updates; runs/grpo/train.jsonl records both, and --resume continues a stopped run.

Measured on one RTX PRO 6000 (96 GB) from the starter recipe’s checkpoint, 6 scenarios x 8 drives per update:

• GPU memory: the trainer peaks at 32 to 35 GiB (the frozen model, a copy of the starting planner, float32 master weights and AdamW states). The whole GPU peaked at 39.9 GB with two drives rendering at once (a one-update run) and 48.8 GB with four (a 6-update run), so plan for a 48 GB or larger GPU; a 40 GB GPU is at its limit with --parallel 2.
• Time: about 20 min per update of 48 drives of 10 to 13 s: 13 to 15 min driving with --parallel 4 (each model decision takes about 0.8 s, and decisions of drives running at once take turns on the one model), then about 4.5 min training.

torchrun --standalone --nproc-per-node N runs one trainer per GPU, each driving its share of the episodes; its memory per GPU and speed have not been measured. Whether GRPO raises the held-out driving score over the SFT checkpoint is still being measured. python -m simforge_gym.train.grpo --help lists every flag.

Persistent worlds

simforge world serve hosts one shared, realtime world that several clients join and drive, rather than isolated episodes. It authenticates clients with a token from a file you create first and keep private (for example umask 077; head -c 32 /dev/urandom | xxd -p > world.token), passed as --token-file; see simforge world serve --help.