Datasets

Golden datasets: import, export and evaluate

Import a golden set as Gemini or OpenAI JSONL, export any dataset in the same shapes, and evaluate a Model or prompt against its expected outputs.

A golden set pairs inputs with expected outputs you trust. Import one from request and response JSONL, export any dataset in the same shapes, and score a Model or prompt against it to see which items regressed.

The JSONL shapes

Each line is one JSON object with a request and an optional response. Two request shapes are accepted, and each line is detected on its own:

Gemini shape: request.contents[] with roles user and model

{"request":{"contents":[{"role":"user","parts":[{"text":"Why is the sky blue?"}]}]},"response":{"candidates":[{"content":{"role":"model","parts":[{"text":"The sky appears blue to the human eye as a result of a phenomenon known as Rayleigh scattering."}]}}]}}

OpenAI shape: request.messages[] with roles system, user and assistant

{"request":{"messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What's the capital of France?"}],"model":"gpt-4o"}}

Multi-turn: the response is the expected reply to the last user turn

{"request":{"contents":[{"role":"user","parts":[{"text":"I'm planning a trip to Paris."}]},{"role":"model","parts":[{"text":"That sounds wonderful! What time of year are you going?"}]},{"role":"user","parts":[{"text":"I'm thinking next spring. What are some must-see sights?"}]}]},"response":{"candidates":[{"content":{"role":"model","parts":[{"text":"For spring in Paris, you should definitely visit the Eiffel Tower, the Louvre Museum, and wander through Montmartre."}]}}]}}

JSONL	Dataset row
the last `user` turn (`request.contents[].parts[].text` or `request.messages[].content`)	`input`
the whole dialogue of a multi-turn line	`turns` on a conversation dataset; dropped on a single-turn one
`response.candidates[0].content.parts[].text` or `response.choices[0].message.content`	`target`, the expected output. The row becomes golden once a person verifies it.
a `system` message, or `request.systemInstruction` (`system_instruction`)	`systemInstructions`
`request.model`	`metadata.model`
Gemini `functionCall` and `functionResponse` parts	dropped, because only text parts are read
OpenAI `tool` messages	`tool` turns; `tool_calls` is not read

Import

Choose Import JSONL in New dataset to create a dataset, or on the Rows toolbar to append to one. Appended rows go after the last row and nothing is renumbered, so runs and snapshots over the old rows still find them.

The dialog takes a .jsonl, .json or .txt file and previews what it detected, with errors by line number. A line without a user turn is an error.

The dataset type defaults to Auto, which makes a conversation dataset when any line has more than one turn. On append, the existing dataset keeps its type, and the dialog warns before multi-turn lines lose their earlier turns.

An import is limited to 50,000 rows and 50 MiB, counting existing rows on append, and the server answers 422 above either. A toast reports rows imported and lines skipped. The Last import card on the Rows tab keeps the latest import's full counts, who ran it and when, and up to 100 skipped lines with reasons.

Export

Export in the dataset header opens a dialog with four fields, and Download saves the file:

Field	Options
Format	JSONL or CSV
Shape	Gemini contents and parts, or OpenAI messages. Not used for CSV.
Write as response	The expected output (`target`) for a golden set, or the generated output to hand a model's answers to someone as a candidate golden set
Rows	All active rows, golden only, finalized only, draft only, or archived

Each JSONL line is rebuilt from the row's fields: its turns or input, system instructions and model form the request, and the chosen column becomes the response. A row with that column empty is written request-only. The file can be imported again without changes.

Evaluate against the golden set

Check the Health tab first: it counts golden rows, so you see how much of the set is verified. Then open Run evaluation, or Run evaluation on v1 in the Snapshots tab to pin the run.

To test a Model or prompt, generate the outputs with it. A run that generates outputs cannot be pinned, so for a pinned run generate on the dataset first, take a snapshot, then run on it with the existing outputs.

Then pick reference-based evaluators. The assertions exact_match, json_equal and numeric_close require a reference by default; contains, regex, json_schema and length are also available. LLM judge evaluators that compare with the expected output are marked needs an expected answer. A row with no expected output is skipped, never scored zero. Embedding-based semantic similarity is not available.

The finished run shows the pass rate and a mean per evaluator. Exact match and a judge are separate evaluators, so a row can pass one and fail the other. The header's Snapshot field names a pinned version, and the row drawer shows input, expected output and output with each verdict and reason, plus the source trace if any.

Compare the run with a baseline and turn on Regressed only to see the rows that got worse. Curating a golden set from traces with human review, and keeping it healthy, is covered in Curate and maintain a golden dataset.