Models and providers

The model dropdowns in a project (run judge, playground, online rules, simulations) offer the Models added to that project, and each Model calls its vendor through a Provider.

Two levels

A Provider is an LLM vendor credential on the organization, under Settings → Organization → Providers in the sidebar footer. It is stored encrypted, validated with a real call when saved, never returned to the browser, and shared by every project.

A Model belongs to one project, under Settings → This project → Models. It points at one provider model and holds the provider, model id, display name, default parameters and modalities.

Connect a Provider

1. 1

   Pick the vendor

   Click Connect provider and pick from the catalog: frontier, cloud, aggregator, self-hosted, voice and media vendors. The System One column marks the vendors whose decision models can judge as System One, and the System One chip lists only them.

2. 2

   Fill the credential

   Enter a label such as "Prod key", which model pickers show, and the credential fields, such as api_key, api_base, aws_access_key_id or vertex_credentials. For a vendor with several auth modes, such as Bedrock, Authentication picks the mode and its fields.

3. 3

   Test & save

   Click Test & save. One real, minimal call either stores the connection as valid, with the models it discovered, or refuses the credential; for a vendor that serves only decision models, such as TypeSafe or Strands Decider, that call is one yes-or-no question. For a server the platform cannot reach, such as Ollama behind your firewall, turn on Skip validation and click Save without testing; the connection is stored as untested.

The dialog connects a Provider for the whole organization. A connection scoped to one project, which outranks the organization's for that project, can be created only through the API.

The Providers table shows each connection's status (valid, invalid or untested), last error, validation time, and how many Models and projects use it. Owners and Admins can:

• Test the credential again.
• Rotate credentials, which replaces the whole credential map and validates it again while Models keep working. Partial updates are not offered, because a half-rotated credential fails in a way that looks like an outage.
• Disconnect, which deactivates the connection and discards the stored credential. When Models use it, you type the provider name to confirm.

After a disconnect, Models added in the app fail at call time with a message to re-select a connection, even once the vendor is connected again, and no screen re-points them. Runs that use them show judge_error rows.

Add models to a project

In Settings → This project → Models, click Add models, then pick a Provider connection and a Purpose. Generation is for models that write text, images, audio or video; a text model can also judge. System One is for decision models, such as Jev, gpt-6-luna or Clef: they answer the questions an evaluator asks and write no text, so only judge pickers offer them. A Model that does both, such as gpt-6-luna, is listed under each purpose and is added once for each. See System One. Tick models in the catalog list (featured first, with modalities and prices per million tokens); retired models are refused.

Modality decides where a Model appears: text Models in judge, agent and chat pickers, image Models on the Image tab, tts and stt Models on Voice. Judge calls get the Provider's full credential map, so Azure, Vertex and Bedrock Models can judge too. Every member of the project sees every Model. An empty picker links to Add one in project settings.

Remove a Model

Used by on the Models tab counts the online rules that judge with each Model, and a Model its provider retired carries a "retired" pill.

Removing a Model that anything references fails with a 409 listing the references, because an online rule whose judge disappeared would fail at call time with nothing to explain it. Agents, simulation runs, online rules, playground grids and evaluation runs all count, finished runs included, so a Model used in any run cannot be removed from the app. The API removes it anyway with ?force=true.

Removal is a soft delete: past runs keep their results and the Model's name, and pickers stop offering it. POST /models/:id/restore undoes it; the app has no restore button.