AI Coding Assistants

# EvalKit integration skill

EvalKit ("syntropylabs-evalkit") is a tracing + evaluation SDK for Python and
TypeScript/Node.js. Use this when asked to add tracing, evaluation, or LLM
observability to an app.

## Install
- Python: pip install syntropylabs-evalkit   (import name stays "evalkit")
- TypeScript/Node: npm install syntropylabs-evalkit

## Core rule: init() does everything
Call evalkit.init() ONCE at app startup, before the app handles traffic.
After that, do NOT manually wrap LLM calls, HTTP requests, or DB queries —
they are already auto-instrumented. Never invent an API like
"evalkit.trace_llm_call()" — it does not exist.

Python:
    import evalkit
    evalkit.init(
        subscription_key="tk_live_...",  # required, from the dashboard
        service_name="my-api",
        environment="production",        # development | staging | production
    )
    # ... app runs ...
    evalkit.flush()  # call before process exit

TypeScript:
    import evalkit from "syntropylabs-evalkit";
    evalkit.init({
      subscriptionKey: "tk_live_...",
      serviceName: "my-api",
      environment: "production",
    });
    // before exit: await evalkit.flush();

## Already auto-instrumented after init() — do not wrap these manually
- LLM providers: OpenAI, Anthropic (incl. Bedrock/Vertex), Google Gemini,
  Cohere, Mistral (Python), LiteLLM (Python), Groq/OpenAI-compatible via
  the OpenAI client.
- HTTP: requests/httpx (Python), fetch/axios/node:http (TypeScript).
- Databases (auto): psycopg/asyncpg/PyMySQL/redis/pymongo (Python),
  pg/mysql2/mongoose/ioredis (TypeScript). SQLAlchemy is the one manual
  call: evalkit.patch_sqlalchemy_engine(engine).
- Logs: the "logging" module (Python) / console.* (TypeScript).
- Tool calls the model makes — attached to the LLM call automatically,
  no manual spans needed.

## Web framework middleware — exactly one line
- FastAPI/Starlette: app.add_middleware(evalkit.EvalKitMiddleware)
- Flask: evalkit.instrument_flask(app)
- Django: add "evalkit.EvalKitDjangoMiddleware" to MIDDLEWARE
- Express: app.use(evalkit.expressMiddleware())
- Fastify: await app.register(evalkit.fastifyPlugin())
- NestJS: await evalkit.enableNestjsAutoTrace(app) — after NestFactory.create()

## Tracing the user's own functions (only if they ask for it)
- Python: @evalkit.trace_function() for one function, @evalkit.trace_tool()
  for a tool the model calls, @evalkit.traced on a class,
  evalkit.trace_module(mod) / evalkit.trace_package(pkg) for a whole
  module/package. Python already auto-traces the caller's own source tree
  by default — don't sprinkle trace_function everywhere redundantly.
- TypeScript: evalkit.traceFunction("name", fn), evalkit.traceTool("name", fn),
  the @Traced() class decorator, evalkit.traceObject({...}, { prefix }).

## Offline evaluation (deterministic, no LLM cost)
Python:
    scores = evalkit.evaluate(output=reply, expected_tools=[...],
                               tool_calls=[...], constraints={...})
TypeScript:
    const { scores } = evalkit.evaluate({ output: reply, expectedTools: [...],
                                           toolCalls: [...], constraints: {...} });

## Scenario simulation (synthetic-user testing of an agent)
Three calls, in order:
  generate_scenarios(...) -> simulate_user(entrypoint, scenarios) ->
  evaluate_simulation(simulation_id, collection_id=..., provider=...,
                       model=..., api_key=...)
The entrypoint function runs ONE turn and receives ctx with message /
session_id / state / turn; return a string or an AgentTurnResult.

## Hard rules — do not violate
1. Never invent SDK function/method names. If unsure whether something
   exists, say so instead of guessing.
2. Never create manual spans for anything already auto-instrumented
   (see the list above) — that duplicates data.
3. subscription_key / subscriptionKey is required and comes from the
   user's own dashboard (Tracing -> create a trace project) — never
   hardcode a placeholder value as if it were real.
4. Always call flush() (Python) / await flush() (TypeScript) before the
   process exits — especially in short-lived scripts or serverless
   functions, where spans can otherwise be dropped.
5. Full docs: https://syntropylabs.ai/docs