Score a scenario render with a driving model

Open-loop episode evaluation of one scenario by an offered driving model: the scenario’s authored drive stays in control while the model’s cameras render live and every plan is scored against the drive that followed it. modelVersionId names an available model of GET /api/v1/evaluation-options and is required (400 validation_failed, param modelVersionId, detail.code model_required). API versions before 2026-10-24 accept a request without it and run the default model, with a model_defaulted note and a SimForge-Warning header. The episode costs 300 cents per scenario (estimate basis per_scenario; maxCostCents at least 300), charged only when it succeeds with a verified, scored result.

POST /api/v1/evaluations

Score a scenario render with a driving model

cURL

curl --request POST \
  --url https://dev.simforge.ai/api/v1/evaluations \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --header 'Idempotency-Key: <idempotency-key>' \
  --data '
{
  "scenarioId": "<string>",
  "modelVersionId": "<string>",
  "maxCostCents": 1,
  "path": "episode",
  "revisionId": "<string>",
  "predictionIntervalS": 0.5,
  "params": {},
  "seed": 1,
  "numTrajSamples": 32,
  "dryRun": false
}
'

200

{
  "dryRun": true,
  "type": "<string>",
  "scenarioId": "<string>",
  "estimate": {
    "lowCents": 1,
    "highCents": 1,
    "basis": "measured"
  },
  "allowed": true,
  "refusal": {
    "code": "<string>",
    "message": "<string>",
    "detail": {}
  },
  "notes": [
    {
      "code": "<string>",
      "message": "<string>",
      "subject": "<string>",
      "detail": {}
    }
  ],
  "availableCreditsCents": 123,
  "concurrency": {
    "active": 123,
    "limit": 123
  },
  "maxCostCents": 1,
  "cancellationTerms": "<string>",
  "modelVersionId": "<string>",
  "modelName": "<string>",
  "model": {
    "modelVersionId": "<string>",
    "name": "<string>",
    "family": "<string>",
    "revision": "<string>",
    "quant": "<string>",
    "checkpointDigest": "<string>",
    "aliasOf": "<string>",
    "owner": "<string>"
  },
  "modelParams": {},
  "customParams": true
}

Authorizations

Authorization · string · header · required

A sfc_at_… access token from simforge login, or an sfk_… API key. An operation's security lists the scopes the credential must hold.

Headers

Idempotency-Key · string · required

Required string length: 1 - 200

Body

application/json

scenarioId · string · required

The scenario to evaluate. Its latest revision is used unless revisionId names one explicitly.

Required string length: 1 - 128

Pattern: ^[A-Za-z0-9_.:-]+$

modelVersionId · string · required

The model to evaluate: an offered model's id from GET /api/v1/evaluation-options, or a registered model version in this workspace. Required from API version 2026-10-24; older versions run the default model without it and say so.

Required string length: 1 - 128

Pattern: ^[A-Za-z0-9_.:-]+$

maxCostCents · integer · required

Required range: x >= 0

path · enum<string> · default: episode

Open-loop evaluations run as a non-reactive episode: the scenario's authored drive stays in control while the model's cameras are rendered live and each plan is scored against the drive that followed it.

Available options: episode

revisionId · string

The scenario revision to evaluate, including its pinned map; absent: the latest revision.

Required string length: 1 - 128

Pattern: ^[A-Za-z0-9_.:-]+$

textures · enum<string>

The texture tier for the live episode cameras; absent: the model family's open-loop tier.

Available options: uastc-full, bc7-512

predictionIntervalS · number · default: 0.5

Seconds between scored prediction instants. Episode evaluations run at 0.5 s (2 Hz).

params · object

The model's parameters by name (GET /api/v1/evaluation-options lists each model's params with their type, bounds, default and loops); absent ones take their defaults. A name the model does not take in this loop, or a value it does not take, is refused naming params..

Show child attributes

seed · integer

Alias of params.seed (sent both ways, the two must agree). Absent: the model's default seed.

Required range: x >= 0

numTrajSamples · integer

Alias of params.numTrajSamples (sent both ways, the two must agree): trajectory samples per instant. Absent: the model's default, or the one count a model returns (its param's fixed value).

Required range: 1 <= x <= 64

dryRun · boolean · default: false

Response

200

application/json

Quote or idempotent replay.

Option 1

Option 2

dryRun · enum<boolean> · required

Available options: true

type · string · required

Legacy: alpamayo.openloop for the default model or a workspace alias of it, policy.openloop for every other model (before API version 2026-10-24: the lane it ran on, the default model's family lane or policy.openloop). Name the model by model.modelVersionId.

scenarioId · string · required

The scenario it would evaluate.

estimate · object · required

Show child attributes

allowed · boolean · required

refusal · object | null · required

Why the same request without dryRun would be refused now, with that submission's code and detail (cloud_not_open with detail.code model_unavailable while no worker serves the model). Null when allowed.

Show child attributes

notes · object[] · required

Warnings attached to the evaluation while it runs, including model_defaulted before API version 2026-10-24 when the request names no model. Empty when nothing.

Show child attributes

availableCreditsCents · number · required

concurrency · object · required

Show child attributes

maxCostCents · integer · required

Required range: x >= 0

cancellationTerms · string · required

modelVersionId · string

The model version it would run (before API version 2026-10-24, the default model when the request names none); model.modelVersionId.

modelName · string

model.name.

model · object

Show child attributes

modelParams · object

The model's parameters it runs with, by name, defaults filled in (what params sent, else each one's default).

Show child attributes

customParams · boolean

True when a parameter whose comparability is custom (a sampling setting, a navigation instruction) was off its default: compare it only with runs of the same params.