IrisEvals · Part of Iris Control

Expert-level judgment, on every AI conversation

IrisEvals is an always-on evaluation agent. It works out what a good answer is for each workflow, scores every production LLM call against it, and sends tested fixes to your repo as pull requests. No review team needed. If you have one, it learns from them too.

By the IrisAgent team · Last updated

IrisEvals · billing-agentLive
1Define12 criteria drafted from prompts and policies
2CheckEvery verdict explained, 140 of 150 match reviewers
3WatchEvery production call graded today
4FixPR #482 verified on 37 failing conversations
No review team needed · nothing merged without your review
100%
of production calls scored, every day
Zero
eval spreadsheets to keep up to date
Every fix
replayed on real conversations before you review it

Your AI is answering customers right now. Nobody can read every conversation.

01

Spot checks see a sliver

Whoever has time reads a small sample of conversations, if anyone does. Most failures are never seen until a customer complains.

02

Good is never written down

Most teams never turn their quality bar into criteria. Rubric workshops stall, and the spreadsheet is stale the week after it ships.

03

Fixes land weeks later

A bad answer becomes a complaint, then an escalation, then a sprint ticket. Customers find the failure before your team does.

How it works

Every conversation in, a better agent out, every week

Define what good means from your own prompts and policies. Grade every call. Fix what fails with a verified PR. Pick the right model. Reviews from your team make each step sharper, but nothing waits on them.

Knows what good looks like from day one

IrisEvals reads your system prompts, policies, and real conversations and drafts the evaluation criteria for each workflow itself. You don't need reviewers to start. If your support leads or compliance team already review conversations, a thumbs up or down and a short reason sharpen the criteria further.
Review · billing-agentOptional team review
Customer
I was charged twice this month. Can I get a refund?
Agent answer
Yes, you'll get a full refund of $49 within 3 days.
👎 Dana, Support Lead
Never promise an amount or a date before checking the plan. Annual plans get prorated credit, not a refund.
IrisEvals sharpened a criterion
Refund answers must check the plan type first and must not promise an amount or a timeline.
Added to 11 criteria drafted from your prompts and policies
Judge agreementReady to grade
140/150verdicts match your reviewers
Refund and billing answers48/50 agree
Plan and pricing questions47/50 agree
Escalation hand-offs45/50 agree
5 disagreements on escalations
Sent back to Dana for a second look. The criterion is updated from her answers.

Matches your team's judgment, and shows its work

Every verdict comes with a plain-English reason you can check in one click. Whenever someone on your team reviews a conversation, IrisEvals compares its own verdict with theirs and shows the agreement rate for each workflow. Disagreements update the criteria automatically.

Scores every production call, not a spot check

Every production conversation gets a verdict and a reason, in real time, from Iris Control or a copy of your logs. When quality drops on a workflow, your team hears about it in Slack or email the same hour, with the failing conversations attached, not from a customer escalation weeks later.
Production · last 24 hours100% of calls graded
support-answers
96%
ticket-routing
94%
billing-agent
81%
#ai-quality IrisEvals · 10:42 AM
billing-agent dropped from 95% to 81% since this morning's prompt change. 37 conversations failed "refund answers check the plan first."
Root cause found · PR #482 opened
Open#482 Check plan type before refund answers
irisevals-bot wants to merge 1 commit into main · prompts/billing-agent.md
- If a customer asks for a refund, confirm it.
+ Before any refund answer, look up the plan type.
+ Never promise an amount or a date. Annual plans
+ get prorated credit per the billing policy.
Verified by replay
Failing conversations now passing35 of 37Golden eval set, billing-agent95.2% to 97.6%Regressions on other criteriaNone
✓ All checks passedWaiting on your review

Turns failures into verified pull requests in your repo

IrisEvals traces each failure pattern to its cause, writes the fix, and opens a pull request in your repo. Every PR is replayed before you see it: how many failing conversations now pass, the golden eval set score before and after, and any regressions. Your engineers review and merge. Nothing ships without them.

Picks the right model at the right price

A new model ships every few weeks. IrisEvals replays your real conversations on it, including your own fine-tuned or self-hosted models, and puts cost and latency next to every score. You get a ship or hold call on evidence, and stop paying for the top model on workflows that don't need it.
Golden eval set · support-answersRefreshed today
Billing and refunds28%
Account access22%
Product how-to31%
Past failures kept as edge cases19%
Near-duplicates removedEN · ES · DE · FRNew workflow added: voice

An eval set built from this month's traffic

The golden eval set comes from your production traffic, not a pilot spreadsheet. Near-duplicates are removed, it stays balanced across workflows, tenants, languages, and edge cases, past failures are kept so they can't come back, and new workflows get their own slice as you launch them.
Real results

Newer and cheaper is not always better

We run IrisEvals on IrisAgent's own production support agents. On our golden support eval set, a newer release at a third of the output price still scored below the model it would replace. Without evals, that upgrade ships. IrisEvals held it.

Support eval setGPT-5.6 Sol (today)GPT-6.1 Sol (new release)
Output price per 1M tokens$30$10
Overall score93.390.6
Answer accuracy98.8%95.3%
Cited sources76.5%70.6%

Source: IrisAgent Support Model Arena, October 15, 2026.

Who it's for

Built for agents where a bad answer is expensive

  • ✓Teams running customer-facing AI agents, chatbots, or copilots
  • ✓Teams where one bad answer costs revenue, trust, safety, or compliance
  • ✓Regulated industries where agents make judgment calls
  • ✓Teams already paying people to review AI conversations by hand
  • ✓Platform teams paying for a frontier model on every workflow
  • ✓Teams shipping in-house or fine-tuned models next to frontier ones
Compare

IrisEvals vs building it yourself vs eval platforms

Every option can run an eval. The difference is who decides what good means, who keeps it current, and who turns failures into fixes.

IrisEvalsBuild it in-houseEval and tracing platforms (Braintrust, LangSmith, Langfuse)
Who defines goodIrisEvals, from your prompts and policies, sharpened by your teamEngineers collecting labels from busy expertsYour team writes the scorers
Who drafts the grading criteriaIrisEvalsYour engineersYour engineers
Agreement with your teamMeasured whenever they reviewRarely measuredUp to you
Coverage100% of production callsUsually a sampleWhatever you wire up
Eval setBuilt from live traffic, refreshed automaticallyWritten once, goes staleDatasets you curate
Fixing what's foundVerified PRs in your repoTickets, weeks laterDashboards, then manual work
New model releasesReplayed on your traffic with priceHand-tested, if at allPossible, if you build the pipeline
On your request pathNeverDependsOften, through an SDK or proxy
Ongoing upkeepDone for youA part-time teamYour team

What is LLM evaluation?

LLM evaluation is the practice of scoring a large language model's outputs against defined criteria, such as correctness, grounding, format, and policy, so teams can measure quality before and after a prompt or model change. The scores, often called evals, replace spot checks with numbers you can compare across models, prompts, and releases.

The hard part is not running an eval. It is deciding what good means, keeping that definition current, and acting on what the evals find. IrisEvals does all three, and works on any LLM call with a prompt and an output:

WorkflowExamplesWhat gets graded
Grounded answersSupport answers, internal search, RAGAnswer accuracy, hallucination resistance, cited sources, grounding
Routing and taggingIntent routing, ticket tagging, classificationAccuracy per label and which labels get confused
Agents and tool useBilling agents, triage agents, multi-step workflowsRight tool, right arguments, task completed, steps taken
Generated contentSummaries, drafts, scriptsCorrectness against the prompt, tone, format, required content
Voice and multilingualVoice agents, conversations that switch languagesLanguage fidelity, turn accuracy, response time per turn
IrisEvals at a glance
Product typeAn always-on evaluation agent, and the evaluation module of Iris Control, IrisAgent's LLM control plane
Built forTeams running customer-facing AI agents, and the AI, platform, and CX leaders accountable for their quality
What good meansDrafted from your prompts, policies, and conversations, and sharpened by your team's reviews when you have them
TrustEvery verdict has a reason, and agreement with your reviewers is measured whenever they review
Coverage100% of production calls, from Iris Control or a copy of your logs
FixesVerified pull requests in your repo, replayed on the failing conversations and the golden eval set
Model choiceAny model, including in-house and fine-tuned models, replayed on your traffic with cost and latency
Added latencyNone. IrisEvals never sits on your request path
Data handlingPII and secrets redacted before storage, with a deploy-in-your-VPC option
Get started

Start with a sample, go to every call when it pays off

IrisEvals never needs to sit on your request path. Logs are redacted for PII and secrets before storage, and Iris Control can run in your own VPC.

How you startWhat you sendOn your request path?What you get
Sample report10 to 20 example conversations or calls, plus a few expert reviews if you have themNoGraded results with explanations, and the same prompts replayed on leading models, with price
Log copyA recurring export of your LLM call logsNoEvals on every call, alerts, verified PRs, and model comparisons
Iris Control endpointPoint your OpenAI-compatible client at Iris ControlYes, server sideEverything above captured automatically, plus routing, failover, spend limits, and redaction

IrisEvals is part of Iris Control, IrisAgent's LLM control plane for cost, failover, and security on every LLM call.

LLM evaluation FAQs

We got you.