FAQ
Short answers to the questions people ask most often about EvalKit.
What is EvalKit?
EvalKit is Syntropylabs’ SDK for tracing, evaluating, and testing AI agents and LLM applications. It’s available for Python and TypeScript/Node.js.
How do I install the EvalKit SDK?
For Python, run "pip install syntropylabs-evalkit" (the import name stays "evalkit"). For TypeScript/Node.js, run "npm install syntropylabs-evalkit".
What gets traced automatically?
After calling evalkit.init(), every major LLM provider (OpenAI, Anthropic, Bedrock, Google, Cohere, Mistral, Groq), HTTP client, and database driver is traced automatically — including tool calls the model makes — along with every function in your own app’s source tree.
Can I bring my own LLM API key (BYOK)?
Yes. Scenario generation and evaluation are bring-your-own-key: pass provider and api_key (Python) or apiKey (TypeScript) to use your own Anthropic, OpenAI, or Google key instead of the hosted default.
What’s the difference between offline and online evaluation?
Offline evaluation is triggered manually — by selecting traces in the dashboard, or by calling evalkit.evaluate() in code. Online evaluation automatically evaluates every new trace in a Trace Project at a configured polling interval (1–60 minutes).
How does scenario simulation work?
EvalKit generates realistic, multi-turn synthetic user scenarios from your agent’s system prompt and tools, plays them against your real agent turn by turn via simulate_user (Python) or simulateUser (TypeScript), and scores the results.
Where do I view traces and simulation results?
In the dashboard: Tracing shows the full waterfall view of spans for each trace, and Simulations shows scenario scores, per-turn traces, tool trajectory, and LLM-judge ratings for each run.
Do I need to self-host anything to use EvalKit?
No — the SDK defaults to the hosted trace-ingest and control-plane endpoints. If you do want to self-host, override base_url/baseUrl and api_url/apiUrl at init() time; see the Configuration reference.
What languages does the SDK support?
Python (syntropylabs-evalkit, imported as evalkit) and TypeScript/Node.js (syntropylabs-evalkit). Both cover the same concepts — tracing, evaluation, and scenario simulation.