Concepts
Scenarios
A scenario describes what you want to test: who the customer is, what they want, and how the conversation should run. Scenarios are plain YAML files.
Where they live
Scenarios live in scenarios/, organised by domain (banking, finance, healthcare, insurance, legal, compliance) and category (functional, adversarial). A single file can hold several scenarios separated by ---.
Fields
| Field | Description |
|---|---|
name | Human-readable scenario name. |
channel | chat, voice, or both. |
mode | agent (an LLM plays the customer) or script (fixed turns). |
authenticated | If true, the session starts pre-authenticated so the agent skips identity checks. |
opening_message | The customer’s first message. |
goal | What the agent must achieve — used by the judge to score goal success. |
customer_persona | Instructions for the LLM customer in agent mode. |
turns | In script mode, the list of { send, timeout_seconds } messages to play. |
attack_type | Present only on security scenarios — drives adversarial scoring (see below). |
max_turns, default_timeout_seconds, turn_delay_seconds | Pacing and limits for the run. |
Script vs agent mode
- Script mode plays the
turnslist verbatim — deterministic and good for precise adversarial probes. - Agent mode uses a Bedrock-backed driver to generate each customer turn from the persona, goal, and conversation so far — a more realistic, reactive test.
Security scenarios
A scenario with an attack_type (e.g. persona_override, system_prompt_exfiltration, authority_impersonation) is treated as a security test. These are scored only on the safety dimensions — see The Judge.
Templating
Placeholders like {customer_name}, {customer_first_name}, and {customer_id} are substituted at runtime from the synthetic-customer settings, so one scenario can run with different identities.
Example
name: "Account — Balance Enquiry (Authenticated)"
channel: chat
mode: agent
authenticated: true
opening_message: "Hi, I'd like to check my current account balance please"
goal: >
The agent greets the customer by first name ({customer_first_name}),
provides the current account balance as a specific figure, and lists
at least 3 recent transactions.
customer_persona: |
You are {customer_name}, a bank customer. If asked for your date of
birth say "12th March 1990". You are cooperative.
max_turns: 12
default_timeout_seconds: 120
turn_delay_seconds: 2.0Run it with the CLI or from the dashboard.