> ## Documentation Index > Fetch the complete documentation index at: https://docs.simforge.ai/llms.txt > Use this file to discover all available pages before exploring further. # Score a scenario render with a driving model > Open-loop episode evaluation of one scenario by an offered driving model: the scenario's authored drive stays in control while the model's cameras render live and every plan is scored against the drive that followed it. modelVersionId names an available model of GET /api/v1/evaluation-options and is required (400 validation_failed, param modelVersionId, detail.code model_required). API versions before 2026-10-24 accept a request without it and run the default model, with a model_defaulted note and a SimForge-Warning header. The episode costs 300 cents per scenario (estimate basis per_scenario; maxCostCents at least 300), charged only when it succeeds with a verified, scored result. ## OpenAPI ````yaml /api-reference/openapi.json post /api/v1/evaluations openapi: 3.1.0 info: title: SimForge Cloud API version: '2026-10-24' description: >- The public SimForge API: what `simforge --cloud`, `simforge login` and the Python client call. Authenticate with a bearer token: the access token `simforge login` stores, an AI agent's access token (its recipe is `/auth.md`), or an `sfk_` API key. A `/api/v1` 401 carries `WWW-Authenticate: Bearer resource_metadata="/.well-known/oauth-protected-resource"`. `/api/v1` errors are an `ErrorEnvelope`; `/api/cli` and `/agent` errors are an OAuth error object. Versions: send `SimForge-Version: ` (2026-10-01, 2026-10-21, 2026-10-22, 2026-10-23, 2026-10-24) and every `/api/v1` response is in that version's shape and names it in the same header; this document describes 2026-10-24. A request without the header gets 2026-10-01, where an error's hint is `detail.hint` and `GET /api/v1/evaluations` names its rows `jobs`. A header that is not a date, or a date before 2026-10-01, is `400 bad_request`. Every response also carries `SimForge-Cli-Minimum`: the oldest `simforge` CLI the server serves. Retries: from 2026-10-21, send a new `Idempotency-Key` (1-255 characters) with every POST, PUT, PATCH and DELETE, and the same key when retrying that request. The first request's answer is kept for 24 hours and a repeat gets it again with `Idempotent-Replayed: true`; the key with any other request, or while the first is still running, is `409 idempotency_conflict` (`retryable` while running). A 5xx answer is not kept. Creating an API key refuses a key (its secret is shown once). license: name: Apache-2.0 identifier: Apache-2.0 servers: - url: https://dev.simforge.ai description: Development (dev.simforge.ai) security: [] tags: - name: Sign-in description: >- `simforge login`: OAuth 2.0 device and browser sign-in for the CLI, the session behind a token. AI agents sign in through auth.md (`/auth.md`): register, a person approves, then the claim and JWT-bearer grants. - name: Account description: The principal, its workspaces, API keys and usage, and system status. - name: Runs description: >- Cloud runs (`--cloud`): renders, simulations, Recast log ingestion and transfers, their events and artifacts. - name: Models description: Customer fine-tunes and their workspace-owned model weights. - name: Datasets description: >- Datasets, their files as signed links (`simforge.dataset-files/v1`, `simforge datasets pull`), and copies out of shared datasets (SIMLOG-25k). - name: Maps and vehicles description: >- Maps the workspace can use, the vehicles a scenario can bind and the asset gallery. - name: Authoring description: >- Hosted scenario authoring: drafts, transactions, validation and map discovery. - name: Distribution description: >- The runtime and map pack agreement and signed downloads (`simforge runtime install`, `simforge maps pull`). - name: Telemetry description: Runtime usage telemetry, a condition of the runtime terms. - name: Benchmark description: Renderer benchmark results and scene packs (`simforge bench`). - name: Challenge description: >- The driving challenge: team, registry login and submissions (`simforge challenge`). paths: /api/v1/evaluations: post: tags: - Runs summary: Score a scenario render with a driving model description: >- Open-loop episode evaluation of one scenario by an offered driving model: the scenario's authored drive stays in control while the model's cameras render live and every plan is scored against the drive that followed it. modelVersionId names an available model of GET /api/v1/evaluation-options and is required (400 validation_failed, param modelVersionId, detail.code model_required). API versions before 2026-10-24 accept a request without it and run the default model, with a model_defaulted note and a SimForge-Warning header. The episode costs 300 cents per scenario (estimate basis per_scenario; maxCostCents at least 300), charged only when it succeeds with a verified, scored result. operationId: createEvaluation parameters: - in: header name: Idempotency-Key required: true schema: type: string minLength: 1 maxLength: 200 requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/EvaluationCreateRequest' responses: '200': description: Quote or idempotent replay. content: application/json: schema: oneOf: - $ref: '#/components/schemas/EvaluationQuote' - $ref: '#/components/schemas/Evaluation' '201': description: Created evaluation. content: application/json: schema: $ref: '#/components/schemas/Evaluation' '400': description: The request failed. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' '401': description: The request failed. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' '402': description: The request failed. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' '403': description: The request failed. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' '404': description: The request failed. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' '409': description: The request failed. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' '429': description: The request failed. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' security: - simforgeToken: - runs:write components: schemas: EvaluationCreateRequest: type: object properties: path: type: string enum: - episode default: episode description: >- Open-loop evaluations run as a non-reactive episode: the scenario's authored drive stays in control while the model's cameras are rendered live and each plan is scored against the drive that followed it. scenarioId: type: string minLength: 1 maxLength: 128 pattern: ^[A-Za-z0-9_.:-]+$ description: >- The scenario to evaluate. Its latest revision is used unless revisionId names one explicitly. revisionId: type: string minLength: 1 maxLength: 128 pattern: ^[A-Za-z0-9_.:-]+$ description: >- The scenario revision to evaluate, including its pinned map; absent: the latest revision. textures: type: string enum: - uastc-full - bc7-512 description: >- The texture tier for the live episode cameras; absent: the model family's open-loop tier. predictionIntervalS: type: number exclusiveMinimum: 0 default: 0.5 description: >- Seconds between scored prediction instants. Episode evaluations run at 0.5 s (2 Hz). params: type: object additionalProperties: anyOf: - type: number - type: string description: >- The model's parameters by name (GET /api/v1/evaluation-options lists each model's params with their type, bounds, default and loops); absent ones take their defaults. A name the model does not take in this loop, or a value it does not take, is refused naming params.. seed: type: integer minimum: 0 description: >- Alias of params.seed (sent both ways, the two must agree). Absent: the model's default seed. numTrajSamples: type: integer minimum: 1 maximum: 64 description: >- Alias of params.numTrajSamples (sent both ways, the two must agree): trajectory samples per instant. Absent: the model's default, or the one count a model returns (its param's fixed value). modelVersionId: type: string minLength: 1 maxLength: 128 pattern: ^[A-Za-z0-9_.:-]+$ description: >- The model to evaluate: an offered model's id from GET /api/v1/evaluation-options, or a registered model version in this workspace. Required from API version 2026-10-24; older versions run the default model without it and say so. maxCostCents: type: integer minimum: 0 dryRun: type: boolean default: false required: - scenarioId - modelVersionId - maxCostCents additionalProperties: false EvaluationQuote: type: object properties: dryRun: type: boolean enum: - true type: type: string description: >- Legacy: alpamayo.openloop for the default model or a workspace alias of it, policy.openloop for every other model (before API version 2026-10-24: the lane it ran on, the default model's family lane or policy.openloop). Name the model by model.modelVersionId. scenarioId: type: string description: The scenario it would evaluate. estimate: type: object properties: lowCents: type: number minimum: 0 highCents: type: number minimum: 0 basis: type: string enum: - measured - unbenchmarked - per_scenario required: - lowCents - highCents - basis allowed: type: boolean refusal: type: - object - 'null' properties: code: type: string message: type: string detail: type: object additionalProperties: {} required: - code - message description: >- Why the same request without dryRun would be refused now, with that submission's code and detail (cloud_not_open with detail.code model_unavailable while no worker serves the model). Null when allowed. notes: type: array items: type: object properties: code: type: string message: type: string subject: type: string detail: type: object properties: {} additionalProperties: {} required: - code - message description: >- Warnings attached to the evaluation while it runs, including model_defaulted before API version 2026-10-24 when the request names no model. Empty when nothing. availableCreditsCents: type: number concurrency: type: object properties: active: type: number limit: type: - number - 'null' description: The workspace's active-job limit; null when it has none. required: - active - limit maxCostCents: type: integer minimum: 0 cancellationTerms: type: string modelVersionId: type: string description: >- The model version it would run (before API version 2026-10-24, the default model when the request names none); `model.modelVersionId`. modelName: type: string description: '`model.name`.' model: $ref: '#/components/schemas/ModelIdentity' modelParams: type: object additionalProperties: anyOf: - type: number - type: string description: >- The model's parameters it runs with, by name, defaults filled in (what params sent, else each one's default). customParams: type: boolean description: >- True when a parameter whose comparability is custom (a sampling setting, a navigation instruction) was off its default: compare it only with runs of the same params. required: - dryRun - type - scenarioId - estimate - allowed - refusal - notes - availableCreditsCents - concurrency - maxCostCents - cancellationTerms Evaluation: type: object properties: id: type: string pattern: ^(run|cmpj)_ kind: type: string description: >- Legacy: alpamayo.openloop for the default model or a workspace alias of it, policy.openloop for every other model (before API version 2026-10-24: the lane it ran on, the default model's family lane or policy.openloop). Name the model by model.modelVersionId. idempotencyKey: type: - string - 'null' description: >- Caller-supplied job identity; qualification namespaces are excluded from evaluation suites. status: type: string enum: - queued - dispatching - dispatch_unknown - running - cancelling - succeeded - partial - failed - cancelled workspaceId: type: string description: >- The workspace it ran in; absent on an evaluation published to every workspace (`published`). published: type: boolean enum: - true description: >- An evaluation SimForge published to every workspace (`GET /api/v1/evaluations?published=true`): read-only, without its workspace, origin, submitter, estimate or cost. origin: type: string enum: - browser - desktop - cli - api description: >- Where it was launched from: Studio (browser), the desktop app, the simforge CLI (cli) or an API key (api). model: type: object properties: family: type: string description: Its model family (architecture and upstream weights). revision: type: string description: The upstream weights revision (40 hex) its family pins. quant: type: string description: Its quantization. modelVersionId: type: - string - 'null' description: >- The registered model version it ran; absent or null on an evaluation from before cloud runs that names none. name: type: string description: >- Its name, as `GET /api/v1/evaluation-options` lists an offered model. checkpointDigest: type: - string - 'null' description: >- The sha256 of the checkpoint it runs; a model job runs only beside a model host serving exactly this checkpoint. aliasOf: type: - string - 'null' description: >- The offered model this workspace's own version copies (the same family, quantization and checkpoint); null for an offered model and any other version. owner: type: string description: >- Whose model it is: `simforge` (a model SimForge offers) or `workspace` (this workspace's own version, an alias of an offered model included). An open string: a reader keeps a value it does not know. required: - family - revision - quant description: >- What it ran: the identity block (`ModelIdentity`); an evaluation from before cloud runs may lack modelVersionId, name, checkpointDigest, aliasOf and owner. params: type: object additionalProperties: {} description: >- The runner's job parameters (its instants, render source and sampling). modelParams: type: object additionalProperties: anyOf: - type: number - type: string description: >- The model's parameters it runs with, by name, defaults filled in (what params sent, else each one's default). customParams: type: boolean description: >- True when a parameter whose comparability is custom (a sampling setting, a navigation instruction) was off its default: compare it only with runs of the same params. estimateCents: type: object properties: lowCents: type: number minimum: 0 highCents: type: number minimum: 0 basis: type: string enum: - measured - unbenchmarked - per_scenario required: - lowCents - highCents - basis reservedCents: type: number settledCents: type: - number - 'null' scored: type: - boolean - 'null' result: type: - object - 'null' properties: metrics: type: object additionalProperties: {} artifacts: type: array items: type: object properties: role: type: string artifactId: type: string mediaType: type: string bytes: type: number minimum: 0 sha256: type: string pattern: ^[a-f0-9]{64}$ required: - role - artifactId - mediaType - bytes - sha256 required: - metrics - artifacts additionalProperties: {} required: - id - kind - status - model - params - scored - result additionalProperties: {} ErrorEnvelope: type: object properties: error: type: object properties: code: type: string enum: - bad_request - validation_failed - unauthorized - insufficient_scope - workspace_required - not_found - conflict - gone - idempotency_conflict - lease_lost - budget_exceeded - max_cost_required - idempotency_key_required - insufficient_credits - checkout_closed - cloud_not_open - email_unverified - terms_acceptance_required - evaluation_launch_disabled - evaluation_admissions_paused - fine_tunes_not_enabled - model_name_reserved - model_name_invalid - model_name_taken - model_quota_exceeded - checkpoint_too_large - checkpoint_format_unsupported - model_family_not_fine_tunable - checkpoint_file_unexpected - checkpoint_is_offered_model - model_version_exists - param_unknown - param_invalid - param_override_not_allowed - upload_incomplete - upload_mismatch - model_in_use - rate_limited - internal_error reason: type: string hint: type: string param: type: string retryable: type: boolean detail: {} path: type: string requestId: type: string pattern: ^req_[0-9A-HJKMNP-TV-Z]{26}$ required: - code - reason - retryable - requestId required: - error ModelIdentity: type: object properties: modelVersionId: type: string description: >- The model version: an offered model's id (`GET /api/v1/evaluation-options`) or one of this workspace's own. name: type: string description: >- Its name, as `GET /api/v1/evaluation-options` lists an offered model. family: type: string description: Its model family (architecture and upstream weights). revision: type: string description: The upstream weights revision (40 hex) its family pins. quant: type: string description: Its quantization. checkpointDigest: type: string description: >- The sha256 of the checkpoint it runs; a model job runs only beside a model host serving exactly this checkpoint. aliasOf: type: - string - 'null' description: >- The offered model this workspace's own version copies (the same family, quantization and checkpoint); null for an offered model and any other version. owner: type: string description: >- Whose model it is: `simforge` (a model SimForge offers) or `workspace` (this workspace's own version, an alias of an offered model included). An open string: a reader keeps a value it does not know. required: - modelVersionId - name - family - revision - quant - checkpointDigest - aliasOf - owner securitySchemes: simforgeToken: type: http scheme: bearer description: >- A `sfc_at_…` access token from `simforge login`, or an `sfk_…` API key. An operation's security lists the scopes the credential must hold. ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.