IrisEvals is an always-on evaluation agent. It works out what a good answer is for each workflow, scores every production LLM call against it, and sends tested fixes to your repo as pull requests. No review team needed. If you have one, it learns from them too.
1Define12 criteria drafted from prompts and policies
2CheckEvery verdict explained, 140 of 150 match reviewers
3WatchEvery production call graded today
4FixPR #482 verified on 37 failing conversations
No review team needed · nothing merged without your review
100%
of production calls scored, every day
Zero
eval spreadsheets to keep up to date
Every fix
replayed on real conversations before you review it
Your AI is answering customers right now. Nobody can read every conversation.
01
Spot checks see a sliver
Whoever has time reads a small sample of conversations, if anyone does. Most failures are never seen until a customer complains.
02
Good is never written down
Most teams never turn their quality bar into criteria. Rubric workshops stall, and the spreadsheet is stale the week after it ships.
03
Fixes land weeks later
A bad answer becomes a complaint, then an escalation, then a sprint ticket. Customers find the failure before your team does.
How it works
Every conversation in, a better agent out, every week
Define what good means from your own prompts and policies. Grade every call. Fix what fails with a verified PR. Pick the right model. Reviews from your team make each step sharper, but nothing waits on them.
Knows what good looks like from day one
IrisEvals reads your system prompts, policies, and real conversations and drafts the evaluation criteria for each workflow itself. You don't need reviewers to start. If your support leads or compliance team already review conversations, a thumbs up or down and a short reason sharpen the criteria further.
Review · billing-agentOptional team review
Customer
I was charged twice this month. Can I get a refund?
Agent answer
Yes, you'll get a full refund of $49 within 3 days.
👎 Dana, Support Lead Never promise an amount or a date before checking the plan. Annual plans get prorated credit, not a refund.
IrisEvals sharpened a criterion Refund answers must check the plan type first and must not promise an amount or a timeline.
Added to 11 criteria drafted from your prompts and policies
Judge agreementReady to grade
140/150verdicts match your reviewers
Refund and billing answers48/50 agree
Plan and pricing questions47/50 agree
Escalation hand-offs45/50 agree
5 disagreements on escalations Sent back to Dana for a second look. The criterion is updated from her answers.
Matches your team's judgment, and shows its work
Every verdict comes with a plain-English reason you can check in one click. Whenever someone on your team reviews a conversation, IrisEvals compares its own verdict with theirs and shows the agreement rate for each workflow. Disagreements update the criteria automatically.
Scores every production call, not a spot check
Every production conversation gets a verdict and a reason, in real time, from Iris Control or a copy of your logs. When quality drops on a workflow, your team hears about it in Slack or email the same hour, with the failing conversations attached, not from a customer escalation weeks later.
Production · last 24 hours100% of calls graded
support-answers
96%
ticket-routing
94%
billing-agent
81%
#ai-quality IrisEvals · 10:42 AM
billing-agent dropped from 95% to 81% since this morning's prompt change. 37 conversations failed "refund answers check the plan first."
Root cause found · PR #482 opened
Open#482 Check plan type before refund answers
irisevals-bot wants to merge 1 commit into main · prompts/billing-agent.md
- If a customer asks for a refund, confirm it.
+ Before any refund answer, look up the plan type.
+ Never promise an amount or a date. Annual plans
+ get prorated credit per the billing policy.
Verified by replay
Failing conversations now passing35 of 37Golden eval set, billing-agent95.2% to 97.6%Regressions on other criteriaNone
✓ All checks passedWaiting on your review
Turns failures into verified pull requests in your repo
IrisEvals traces each failure pattern to its cause, writes the fix, and opens a pull request in your repo. Every PR is replayed before you see it: how many failing conversations now pass, the golden eval set score before and after, and any regressions. Your engineers review and merge. Nothing ships without them.
Picks the right model at the right price
A new model ships every few weeks. IrisEvals replays your real conversations on it, including your own fine-tuned or self-hosted models, and puts cost and latency next to every score. You get a ship or hold call on evidence, and stop paying for the top model on workflows that don't need it.
Eval set · support-answers
Upgrade check
Replayed on your own prompts
ModelAccuracyGrounded
Current primary
in production
New release
candidate
Hold: grounded rate below current
Current fallback
in production
Golden eval set · support-answersRefreshed today
Billing and refunds28%
Account access22%
Product how-to31%
Past failures kept as edge cases19%
Near-duplicates removedEN · ES · DE · FRNew workflow added: voice
An eval set built from this month's traffic
The golden eval set comes from your production traffic, not a pilot spreadsheet. Near-duplicates are removed, it stays balanced across workflows, tenants, languages, and edge cases, past failures are kept so they can't come back, and new workflows get their own slice as you launch them.
Real results
Newer and cheaper is not always better
We run IrisEvals on IrisAgent's own production support agents. On our golden support eval set, a newer release at a third of the output price still scored below the model it would replace. Without evals, that upgrade ships. IrisEvals held it.
✓Teams running customer-facing AI agents, chatbots, or copilots
✓Teams where one bad answer costs revenue, trust, safety, or compliance
✓Regulated industries where agents make judgment calls
✓Teams already paying people to review AI conversations by hand
✓Platform teams paying for a frontier model on every workflow
✓Teams shipping in-house or fine-tuned models next to frontier ones
Compare
IrisEvals vs building it yourself vs eval platforms
Every option can run an eval. The difference is who decides what good means, who keeps it current, and who turns failures into fixes.
IrisEvals
Build it in-house
Eval and tracing platforms (Braintrust, LangSmith, Langfuse)
Who defines good
IrisEvals, from your prompts and policies, sharpened by your team
Engineers collecting labels from busy experts
Your team writes the scorers
Who drafts the grading criteria
IrisEvals
Your engineers
Your engineers
Agreement with your team
Measured whenever they review
Rarely measured
Up to you
Coverage
100% of production calls
Usually a sample
Whatever you wire up
Eval set
Built from live traffic, refreshed automatically
Written once, goes stale
Datasets you curate
Fixing what's found
Verified PRs in your repo
Tickets, weeks later
Dashboards, then manual work
New model releases
Replayed on your traffic with price
Hand-tested, if at all
Possible, if you build the pipeline
On your request path
Never
Depends
Often, through an SDK or proxy
Ongoing upkeep
Done for you
A part-time team
Your team
What is LLM evaluation?
LLM evaluation is the practice of scoring a large language model's outputs against defined criteria, such as correctness, grounding, format, and policy, so teams can measure quality before and after a prompt or model change. The scores, often called evals, replace spot checks with numbers you can compare across models, prompts, and releases.
The hard part is not running an eval. It is deciding what good means, keeping that definition current, and acting on what the evals find. IrisEvals does all three, and works on any LLM call with a prompt and an output:
Right tool, right arguments, task completed, steps taken
Generated content
Summaries, drafts, scripts
Correctness against the prompt, tone, format, required content
Voice and multilingual
Voice agents, conversations that switch languages
Language fidelity, turn accuracy, response time per turn
IrisEvals at a glance
Product type
An always-on evaluation agent, and the evaluation module of Iris Control, IrisAgent's LLM control plane
Built for
Teams running customer-facing AI agents, and the AI, platform, and CX leaders accountable for their quality
What good means
Drafted from your prompts, policies, and conversations, and sharpened by your team's reviews when you have them
Trust
Every verdict has a reason, and agreement with your reviewers is measured whenever they review
Coverage
100% of production calls, from Iris Control or a copy of your logs
Fixes
Verified pull requests in your repo, replayed on the failing conversations and the golden eval set
Model choice
Any model, including in-house and fine-tuned models, replayed on your traffic with cost and latency
Added latency
None. IrisEvals never sits on your request path
Data handling
PII and secrets redacted before storage, with a deploy-in-your-VPC option
Get started
Start with a sample, go to every call when it pays off
IrisEvals never needs to sit on your request path. Logs are redacted for PII and secrets before storage, and Iris Control can run in your own VPC.
How you start
What you send
On your request path?
What you get
Sample report
10 to 20 example conversations or calls, plus a few expert reviews if you have them
No
Graded results with explanations, and the same prompts replayed on leading models, with price
Log copy
A recurring export of your LLM call logs
No
Evals on every call, alerts, verified PRs, and model comparisons
Iris Control endpoint
Point your OpenAI-compatible client at Iris Control
Yes, server side
Everything above captured automatically, plus routing, failover, spend limits, and redaction
IrisEvals is part of Iris Control, IrisAgent's LLM control plane for cost, failover, and security on every LLM call.
LLM evaluation FAQs
We got you.
LLM evaluation is the practice of scoring a large language model's outputs against defined criteria, such as correctness, grounding, format, and policy, so teams can measure quality before and after a prompt or model change. The scores, often called evals, replace spot checks with numbers you can compare across models, prompts, and releases.
An eval is a repeatable test of an AI system's output: a set of inputs, a way to grade each output, and a score you can track over time. Evals do for LLM apps what automated tests do for code, with one difference. LLM outputs vary, so grading usually needs a judge model rather than an exact match.
IrisEvals is an always-on evaluation agent, and the evaluation module of Iris Control, IrisAgent's LLM control plane. It drafts the evaluation criteria for each workflow from your prompts, policies, and conversations, grades 100% of your production LLM calls, alerts you when quality drops, and opens verified pull requests in your repo to fix what fails. It also replays any new model on your own traffic, with price, before you switch.
It starts from what you already have. IrisEvals reads each workflow's system prompt, your policies, and real conversations, and drafts evaluation criteria from them. Every output is also checked for correctness against its own prompt from the first call. You can read and edit every criterion.
No. IrisEvals grades every call from day one without reviewers. If someone on your team does review conversations, even a few a week, a thumbs up or down and a short reason sharpen the criteria, and IrisEvals shows how often its verdicts agree with theirs.
When a failure pattern shows up, IrisEvals traces it to its cause, usually a prompt, an instruction, or a configuration, and opens a pull request in your repo with the change. Every PR is replayed first: it lists how many of the failing conversations now pass, the score on the golden eval set before and after, and whether anything else regressed. Your engineers review and merge it like any other PR.
No. IrisEvals opens pull requests and recommends whether to ship or hold a model. Code, prompts, and model choices only change when your team merges or switches.
A judge model reads the system prompt, the conversation, and the output, then returns a verdict, a score, and an explanation against the criteria for that workflow. Every verdict comes with a reason, so your team can see why a conversation passed or failed.
A golden dataset, or golden eval set, is the reference set of representative inputs a team scores every model and prompt on, so results stay comparable over time. Sets written once by hand drift away from production. IrisEvals builds yours from your traffic, removes near-duplicates, balances it across workflows, tenants, languages, and edge cases, keeps past failures, and refreshes it as your product changes.
LLM-as-a-judge uses one model to grade another model's output. It suits tasks with no single exact answer, such as support replies or summaries. The weak point is trust, so in IrisEvals every verdict comes with a reason you can check, and agreement with your own reviewers is measured whenever they review.
Those platforms give your engineers tools to trace calls, curate datasets, and write scorers. IrisEvals does that work for you: it drafts the criteria, keeps the eval set current, grades every call, and turns failures into verified PRs. If you want code-first tests in CI or trace-level debugging owned by your team, a framework or observability platform may be the better fit.
Running one eval is easy. Defining what good means, keeping the judge honest, grading every production call, rebuilding the eval set as the product changes, and turning failures into tested fixes is a team's worth of ongoing work. IrisEvals is that team, so your engineers ship features instead of maintaining evals.
Yes. If there is a hard requirement you already know, such as valid JSON, a required link, or a phrase that must never appear, you can add it as a plain-English check next to the criteria IrisEvals drafts.
Yes. In-house and fine-tuned models are graded on the same conversations and criteria as frontier models from OpenAI, Anthropic, Google, and others. You see where your model wins, where it falls behind, and which model should back it up.
Yes. Each agent turn of a transcript is graded with the conversation before it, including conversations where users switch languages mid-sentence, which public benchmarks rarely test.
No. IrisEvals runs on a copy of your logs or on Iris Control's call log, never on the request path.
Send 10 to 20 example conversations or calls, with a few expert reviews if you have them. We return a sample report with graded results and explanations, and the same prompts replayed on leading models with price. Nothing changes in your call path.