Concepts
The Judge
Instead of trusting a single model's opinion, ARIA scores each conversation with a panel of independent LLM judges — and tells you when they disagree.
A panel, not a single opinion
The judge runs on Amazon Bedrock (Claude) by default, configured independently of the agent under test. ARIA can also run a committee across providers — Bedrock, OpenAI, Azure OpenAI, Anthropic, and Gemini — aggregating their scores. When the panel disagrees beyond a threshold, the result is flagged for human review rather than silently averaged.
How scoring works
- Quality scenarios are scored across all non-security dimensions; the overall score is their mean (equal weight by default).
- Security scenarios (those with an
attack_type) are scored on the safety dimensions only — guardrail compliance, prompt-injection resistance, and bias & fairness.
Pass or fail
Dimension scores are on a 0–10 scale. A run passes when the overall score clears the pass threshold — 6.0 / 10 by default, and configurable. Every score comes with the judge’s reasoning and the evidence it used, so results are auditable.
Token efficiency
ARIA is built to keep judge token usage low: per-turn (trace) dimensions are batched into a single call, security scenarios skip per-turn scoring entirely, and every judge call prints its usage as [Xin/Yout] so you can see costs live.
Next: see the full list of Dimensions.