SyntropyLabsDatasets
Golden datasets: import, export and evaluate
Bring a golden set in as JSONL (Gemini contents or OpenAI messages, single- or multi-turn), export any dataset in the same shapes, and evaluate a model or prompt against verified expected outputs with reference-based evaluators.
After this page an existing golden set in request / response JSONL is a project dataset with input, context, target and metadata columns; you can export any dataset back in the same shape without loss; and you can measure a model or prompt against the expected outputs and see exactly which items regressed.
The JSONL shapes
One JSON object per line with a request and an optional response. Two request shapes are accepted and detected per line:
{"request":{"contents":[{"role":"user","parts":[{"text":"Why is the sky blue?"}]}]},"response":{"candidates":[{"content":{"role":"model","parts":[{"text":"The sky appears blue to the human eye as a result of a phenomenon known as Rayleigh scattering."}]}}]}}{"request":{"messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What's the capital of France?"}],"model":"gpt-4o"}}{"request":{"contents":[{"role":"user","parts":[{"text":"I'm planning a trip to Paris."}]},{"role":"model","parts":[{"text":"That sounds wonderful! What time of year are you going?"}]},{"role":"user","parts":[{"text":"I'm thinking next spring. What are some must-see sights?"}]}]},"response":{"candidates":[{"content":{"role":"model","parts":[{"text":"For spring in Paris, you should definitely visit the Eiffel Tower, the Louvre Museum, and wander through Montmartre."}]}}]}}| JSONL | Dataset row |
|---|---|
last user turn (request.contents[].parts[].text or request.messages[].content) | input |
| earlier turns | kept as context (turns[]) — as conversation context on a text dataset, as the row’s turns on a conversation dataset |
response.candidates[0].content.parts[].text / response.choices[0].message.content | target — the expected output; becomes golden once a person verifies it |
system message / request.systemInstruction | systemInstructions |
request.model | metadata.model |
| tool and function parts | kept verbatim as read-only metadata |
| the whole line | metadata.raw, so export round-trips |
Import
New dataset → Import JSONL, or Import on an existing dataset to append after its last row (nothing is renumbered, so runs and snapshots over the old rows still match them; the existing dataset’s type wins). The dialog detects the shape and counts single- and multi-turn lines; dataset type defaults to single-turn rows (earlier turns kept as context) and offers a conversation dataset when any line has more than one user turn. Roles must alternate; invalid lines are reported by line number. Limits: 50,000 rows and 50 MiB per import (existing rows count on append), 422 above either. The result is a banner — 309 imported · 3 skipped (view report) — and the report is stored on the dataset: imported, skipped (the true count), up to 100 error lines with reasons, who imported and when. Only the last import’s report is kept.
Export
Export on any dataset: CSV as stored, or JSONL in the Gemini or OpenAI shape with a choice of which column becomes the response — target for a golden set, or output to hand a run’s answers to someone else as a candidate golden set — plus which rows (all · golden only · a status) and whether to include the system message and metadata. Conversation rows export as several entries with the target as the final response; rows without the chosen column are written request-only. The preview shows the first lines and a warning for rows with no target. The file re-imports through the import flow unchanged.
Evaluate against the golden set
Open Run evaluation (or Run evaluation on v1 from Snapshots to pin the run). The header already shows golden rows against the total, so a set missing expected outputs is visible before the run. In the form: generate outputs with the Model and prompt under test, then the reference-based evaluators — deterministic assertions (exact_match, json_equal, numeric_close default to a required reference; contains, regex, json_schema, length also available) and an LLM judge with reference mode required (“LLM match vs expected”). A reference check confirms how many rows have an expected output before you start; a row without one is skipped, never scored zero. Embedding-based semantic similarity is not available.
The finished run shows pass rate, per-evaluator means (exact and judge match are separate evaluators over the same rows, so a row can pass one and fail the other), the snapshot chip, and a row drawer with expected vs output side by side and the judge’s reason. Compare with baseline lists the regressed items first; each links to its source trace when the row came from one. Curating a golden set from traces with human verification, and keeping it healthy over time, are described in Curate and maintain a golden dataset.