Getting data out

Pull a hosted dataset, a filter of it or chosen scenarios as files with the CLI, an agent, a link list or the API; resumable, checked and verifiable offline.

A dataset is a collection of scenarios in your hosted workspace, or one shared with it. Getting it out means downloading the files SimForge already stores for each scenario (its document, OpenSCENARIO export, trace and simulation record, and, if you ask for them, renders, labels, ground truth and recordings) straight from storage, through signed links. Nothing is packed or zipped for you, so a pull starts at once and an interrupted one picks up where it stopped.

The public SimForge scenario dataset, SIMLOG-25k, is on Hugging Face: huggingface.co/datasets/SIMFORGE-AI/simlog-25k.

Route recordings contain trajectory data, not original camera or sensor video. Renders are generated by SimForge. Public SIMLOG files omit source dataset names, log IDs and collection provenance.

With the CLI

simforge datasets list
simforge datasets pull <dataset-id> --dry-run
simforge datasets pull <dataset-id> --out ./data
simforge datasets verify ./data

--dry-run prints how many scenarios and files the pull has and their size, and downloads nothing. Narrow the pull with --filter 'grade:A tag:cut-in' (the library’s filters), --q <text>, --source-log <recording> or --ids a,b,c. --include chooses what comes with each scenario: renders[:training|showcase|schematic], labels, gt, recordings, or scenarios for nothing else (default renders:training). --passes rgb,depth keeps only some render modalities. --jobs sets the parallel downloads (default 8).

Every file is checked against its sha256 before it is put in place. Run the same command again to resume: it fetches only what is missing or changed. A file SimForge is still preparing (an OpenSCENARIO export, ground truth) is waited for, up to --wait-pending seconds (default 1800); if some are still not ready, the pull lists them and you run it again later. When everything is in place the CLI writes manifest.json, scenarios.jsonl and SHA256SUMS; simforge datasets verify DIR checks the folder against them, offline.

The folder:

manifest.json, scenarios.jsonl, SHA256SUMS
recordings/<sha256>.rec.json.zst
scenarios/<map>/<id[:2]>/<id>/
  document.json, scenario.xosc, trace.json.gz, resolution.json.gz
  labels/labels.json, gt/ego.parquet, gt/actors.parquet
  renders/<preset>-<render id>/...

Scenarios or files that cannot be had (a scenario never saved, an OpenSCENARIO export that failed) are listed with their reason in manifest.json under excluded, never dropped silently.

With an agent

Give your coding agent the setup and the task, for example:

Install the SimForge CLI (curl -fsSL https://simforge.ai/install.sh | sh), sign in with `simforge login`
and install the SimForge skills for yourself (`simforge skills install --claude` or `--codex`).
Then pull dataset <dataset-id>, only grade A scenarios with their training renders and labels:
first `simforge datasets pull <dataset-id> --filter 'grade:A' --include renders,labels --dry-run`
to tell me the size, then the same command with `--out ./data`, then `simforge datasets verify ./data`.

Studio’s Export panel shows this prompt, and the exact command, for the selection you are looking at, after checking that your CLI is installed and signed in.

A link list, without the CLI

curl -H "Authorization: Bearer $SIMFORGE_API_KEY" \
  "https://simforge.ai/api/v1/datasets/<dataset-id>/files?format=urls&include=renders,labels" -o links.txt
aria2c -i links.txt -d ./data -j 8

The list holds a link per file, valid for 24 hours, with the file’s place in the folder and its sha256, which aria2c checks. With wget instead (it does not check digests): grep '^http' links.txt | wget -x -nH --cut-dirs=5 -i - -P ./data. Files still being prepared are listed as comments at the end of the list; download the list again later for them. A link list has no manifest.json: use the CLI when you want the folder verified.

From CI, with an API key

Create a key with datasets:read and artifacts:read (simforge keys create --name ci-pull --scope datasets:read --scope artifacts:read, which needs a login with keys:admin, or the API keys page in workspace settings) and set it as SIMFORGE_API_KEY; the CLI and the API then need no login:

SIMFORGE_API_KEY=sfk_... simforge datasets pull <dataset-id> --include scenarios --out ./data

The API itself: GET /api/v1/datasets/<id>/files pages the selection’s files (signed links valid for an hour; ask for the page again for fresh ones), dryRun=true counts them, and POST takes the same fields as a JSON body for long id lists. See the API reference.

Copy SIMLOG-25k scenarios into your workspace

Every workspace lists SIMLOG-25k read-only. Browse it, open a scenario and watch it in 2D; to edit or run scenarios, copy them into one of your datasets.

simforge datasets copy <simlog-dataset-id> --filter 'map:yale-street split:test' --new 'SIMLOG-25k copy'
simforge datasets copy <simlog-dataset-id> --ids <scenario-id> --into <dataset-id>

In Studio, Copy to dataset… copies the selected scenarios, current filter or whole dataset into an existing or new dataset. Large copies run in the background. A scenario already copied into that dataset is skipped, and each copy identifies its SIMLOG-25k source.