Scenario Simulation

Generate synthetic-user scenarios from your agent’s own prompt and tools, then run them against your real agent code.

Three phases: `generate_scenarios` writes realistic test scenarios for your agent from its instructions and tools (hosted model by default, or bring your own key). `simulate_user` runs those scenarios against your real agent code, in your process — every LLM/tool/HTTP/DB call your agent makes is auto-traced and tagged to the simulation, then each scenario gets lightweight offline scores. `evaluate_simulation` grades a finished run against your project’s evaluator collection (LLM-as-judge), returning per-scenario, per-criterion scores with reasons. Results show under the dashboard’s Simulations tab.

Phase 1 — Generate scenarios

import evalkit, os
evalkit.init(subscription_key="tk_live_...", service_name="billing-agent")

scenarios = evalkit.generate_scenarios(
    "You are a billing support agent for a SaaS product.",   # agent_instructions (required)
    tools=["get_payment_history", "issue_refund", "escalate"],
    count=5,
    categories=["happy_path", "edge_case", "adversarial"],   # optional steering
    # BYOK (optional): use your own model instead of the hosted one
    provider="openai", api_key=os.environ["OPENAI_API_KEY"], model="gpt-4o",
)

Scenario shape

Generated scenarios are multi-turn by default: turns carries ≥3 user messages, turns[0] equals starting_prompt, and conversation_plan is the human-readable script behind them. Use generated scenarios as-is, edit them, or hand-write your own.

{
  "scenario_id": "scn_ab12cd",          // auto-generated if omitted
  "name": "Refund for double charge",
  "starting_prompt": "I was charged twice this month.",  // verbatim Turn 1; == turns[0]
  "conversation_plan": "1. Report the double charge  2. Give the order id when asked  3. Confirm the refund",
  "turns": [
    "I was charged twice this month.",
    "The order id is ORD-1001.",
    "Yes, please refund the duplicate."
  ],
  "expected_tools": ["get_payment_history", "issue_refund"],
  "constraints": { "required_terms": ["refund"], "has_citations": false, "max_turns": 8 },
  "persona": "EXPERT",
  "setup": { "state": { "account_id": "acct_42" } }
}

Phase 2 — Simulate the user against your agent

You provide an entrypoint that runs one turn of your agent: entrypoint(ctx) -> str | AgentTurnResult. The simulator feeds it each user message in order and collects the replies. ctx (SimContext) carries message, session_id/sessionId, state, and turn.

def billing_agent(ctx):                       # ctx.message, ctx.session_id, ctx.state, ctx.turn
    reply = run_my_agent(ctx.message, session_id=ctx.session_id)
    return reply                               # tools auto-captured from traced spans

report = evalkit.simulate_user(
    billing_agent,
    scenarios,
    tags=["release-1.4"],
    max_turns=12,
)
print(report["simulation_id"], report["results"])

Return either a string (tool calls are auto-captured from the spans your agent emits), or an explicit result object for RAG metrics: Python AgentTurnResult(text, tool_calls=[...], retrieved_context=[...]), TS { text, toolCalls?, retrievedContext? }.

Phase 3 — Evaluate the run against a collection

simulate_user records traces and computes lightweight offline scores. To grade a run with your project’s evaluator collection (LLM-as-judge rules with reasons), call evaluate_simulation with the simulation_id and a collection id. The judge is BYOK. Results come back per scenario and per criterion, and are also persisted so they appear in the Tracing dashboard.

result = evalkit.evaluate_simulation(
    report["simulation_id"],
    collection_id="665f0c...",       # Dashboard → Evaluators → Collections
    provider="openai", model="gpt-4o", api_key=os.environ["OPENAI_API_KEY"],
    max_tokens=1024,                 # optional judge output cap
)

print(result["aggregate"])           # {"averageScore", "passRate", "totalScenarios", "evaluatedScenarios"}
for scn in result["scenarios"]:
    print(scn["name"], scn["overallScore"], scn["passed"])

How it traces & scores

• Each scenario runs as its own trace named scenario:<name>, with an isolated context so scenarios never bleed into each other.
• Every LLM/tool/HTTP/DB call your agent makes is auto-captured — returning a plain string is enough.
• Tool metrics use the actual tool calls, pulled from tool_call spans or gen_ai.tool.* events on llm_call spans.
• After the turns, each scenario is scored with evaluate(...) against expected_tools + constraints.
• flush() is called for you at the end of the run.