LLM Evaluator
Evaluation pipelines that grade tool-calling models on what they did to the database, not on what they said.
- Role
- Associate Software Engineer, Devsarch · client: Turing
- Stack
- FastAPI · PostgreSQL · Pytest
Hover or tap a box to see what it does, or press play.
The problem
When a model can call tools that write to a database, what it says and what it does can drift apart. A response can read perfectly while the tool call underneath wrote the wrong row, or wrote two. Grading the transcript rewards the description of the work, not the work.
What I built
- A snapshot of the database before each run
- Runs of the model with its tool calls, through the real code path
- A second snapshot after the run, diffed against the first
- Assertions that the expected changes happened and nothing else changed
- A regression gate: a model or prompt change ships only if every assertion still holds
How it works
- Snapshot the database before the run.
- Run the model with its tool calls, through the real code path.
- Snapshot the database again.
- Diff the two: did it change what it should, and nothing else?
- A model change ships only if every assertion still holds.
Results
- Ran 50+ LLM evaluation pipelines
- Maintained 95%+ evaluation accuracy
- Model output judged against database state instead of a transcript
More on this in the note Testing LLM tool calls against the database, not the transcript.