What is ARIA Evaluator

You describe what a user wants in a scenario, ARIA runs that scenario against your agent through an adapter, and a panel of independent LLM judges scores the resulting transcript across 15 quality and safety dimensions — with adversarial security testing built in. Everything runs on infrastructure you control: bring your own model provider (Amazon Bedrock / Claude), define scenarios in YAML, and get a shareable report with per-dimension scores and the judges’ reasoning.

The evaluation pipeline

Every run follows the same path:

Scenario (YAML) → Adapter → Conversation runner → Transcript → LLM Judge → EvalResult → Report

• Scenario — a YAML file with a persona, a goal, and how the conversation should run.
• Adapter — connects ARIA to the agent under test (Connect, Lex, Azure, Copilot, OpenAPI, WebSocket…).
• Conversation runner — drives the dialogue, scripted or AI-generated, and records an immutable transcript.
• Judge — a panel of independent LLM judges scores the transcript across the active dimensions.
• Report — an HTML + JSON report with scores, evidence, and reasoning; also shown live in the dashboard.

What it measures

ARIA scores conversations across 15 dimensions grouped into response quality, task completion, safety & security, customer experience, and escalation & vulnerability. Security scenarios add adversarial coverage — prompt injection, jailbreaks, social engineering, and more.

Who it’s for

Engineering, security, and product teams putting conversational AI into production — especially in regulated, safety-critical settings — who want a verifiable quality and safety score before they ship.

Next steps

• Quick Start — clone the repo and score your first scenario.
• Run Locally — start the API + dashboard with npm run dev.
• Architecture — how the pieces fit together.
• View the source on GitHub.