Raltic field guide
An AI agent evaluation scorecard that catches real workflow failures
A single quality score hides the failure you need to fix. A useful scorecard separates the final outcome from the path, evidence, tools, risk, and reviewer effort, then runs the same cases again after every change.
Move from the definition to the operating decision without losing the thread.
Direct answer
An AI agent evaluation scorecard should grade at least seven dimensions: task outcome, evidence quality, tool choice and arguments, trajectory efficiency, policy and permission compliance, latency and cost, and the amount of human correction required. Start with 10 to 30 reviewed cases, define pass conditions before running the agent, keep deterministic checks separate from subjective rubrics, and turn production failures into new regression cases.
A failure is useful only when it becomes a named case the next version must pass.
- 01
Define
Write the outcome, constraints, expected evidence, and prohibited actions.
- 02
Run
Capture the full trajectory, tools, sources, output, cost, and version.
- 03
Review
Apply deterministic checks, rubrics, and domain-expert judgment separately.
- 04
Regress
Add failures to a dataset and compare the candidate against the baseline.
Write the scorecard before looking at the result
Predefined criteria prevent a persuasive output from moving the goalposts.
Suppose an agent prepares a launch-readiness brief. The answer may sound polished while relying on old analytics, omitting an open security issue, or recommending launch without naming the decision owner. If reviewers invent criteria after reading the draft, fluency can hide those omissions.
Write the required evidence, freshness window, mandatory sections, prohibited actions, and acceptance owner first. Some checks can be exact: every claim has a source URL, dates fall within 30 days, and no external change occurred. Others require judgment: whether the evidence supports the conclusion and whether uncertainty is explained honestly.
- 01Named task and user outcome
- 02Required sources and freshness
- 03Hard constraints and prohibited actions
- 04Expected tool or route where necessary
- 05Authorized acceptance owner
Use seven dimensions instead of one magic score
Separate failure types so the team knows what to change.
Outcome quality asks whether the artifact solves the task. Evidence quality checks grounding and freshness. Tool quality examines selection, arguments, and side effects. Trajectory quality looks for loops, unnecessary handoffs, and early stopping. Policy checks permissions and prohibited behavior. Efficiency covers latency and cost. Human correction measures how much expert work remained.
Keep dimension scores visible rather than averaging everything into 82 out of 100. A high average can conceal a policy failure or an external action that should have been blocked. For consequential workflows, some dimensions are gates: if the agent violates access policy or omits mandatory evidence, the run fails regardless of its writing quality.
- 01Outcome
- 02Evidence
- 03Tools
- 04Trajectory
- 05Policy
- 06Efficiency
- 07Human correction
Build the first dataset from work people actually recognize
A small, reviewed dataset beats hundreds of synthetic prompts with unclear expectations.
Collect normal cases, edge cases, and known failures from the target workflow. Include ambiguous requests, missing sources, unavailable tools, conflicting evidence, permission denials, and a reviewer rejection. Each case needs enough context to reproduce the decision, not only the final user message.
Ten to thirty cases are enough to expose whether the scorecard is usable. Ask two domain reviewers to score a sample independently. Their disagreements reveal unclear rubrics, contested policy, or genuinely subjective work. Fix the rubric before automating it with an LLM judge.
- 01Common successful task
- 02Missing or stale evidence
- 03Tool or connector failure
- 04Ambiguous or conflicting instruction
- 05High-impact action requiring review
- 06Known production failure
Give code, model judges, and people different jobs
Use the cheapest reliable evaluator for each criterion and preserve human authority over contested outcomes.
Code is best for schemas, required fields, exact tools, forbidden actions, latency, cost, and deterministic policy checks. Model judges help with relevance, completeness, groundedness, and rubric-based comparison, but their scores should include explanations and be calibrated against human examples. People remain necessary for domain judgment, policy interpretation, and disputed cases.
Measure evaluator quality too. Track false passes, false failures, stability across repeated judging, and disagreement with domain reviewers. A model judge that produces a precise decimal without a stable decision rule is adding theater, not confidence.
- 01Code for exact invariants
- 02Model judges for documented qualitative rubrics
- 03Pairwise comparison for close variants
- 04Humans for policy and domain judgment
- 05Escalation path for evaluator disagreement
Connect offline regression to production review
Offline evals protect known behavior; production monitoring finds the cases you did not anticipate.
Run the scorecard on every material prompt, model, tool, routing, or retrieval change. Compare the candidate with the current version on the same dataset and inspect dimension-level regressions. A faster model that increases tool mistakes may still be the wrong release even if its average answer score rises.
After deployment, sample real traces, collect reviewer corrections, and label recurring failure modes. Promote representative failures into the offline dataset with sensitive data removed or transformed. The evaluation system improves when the production workflow and test library share a disciplined feedback path.
- 01Version every evaluated candidate
- 02Compare on identical cases
- 03Set gate dimensions before release
- 04Sample production by risk and uncertainty
- 05Promote corrected failures into regression tests
Primary sources
Product and practice claims in this guide are grounded in current first-party or standards sources. Raltic product claims are limited to shipped behavior described on this site.
FAQ