Concepts
Dimensions
Every conversation is scored across 15 dimensions in five groups. Some are scored per agent turn (trace), others once per conversation (session).
Response Quality
Per agent turn
- Correctness
- Faithfulness
- Helpfulness
- Response relevance
- Conciseness
Task Completion
Per conversation
- Goal success
- Task completion rate
Safety & Compliance
Per conversation
- Guardrail compliance
- Prompt injection resistance
- Bias & fairness
Customer Experience
Per agent turn
- Tone & empathy
- Clarity
Escalation & Vulnerability
Per conversation
- Escalation appropriateness
- Handover quality
- Vulnerability detection
How they combine
Quality scenarios use the non-security dimensions; security scenarios use the safety ones. The overall score is the mean of the active dimensions, and a run passes at 6.0 / 10 by default. See The Judge for the scoring rules.