Conversation is only one layer of the task.
A banking agent must discover the customer's real goal, follow operational policy, call the right tools in the right order, mutate account state exactly once, and communicate what actually happened.
Yotta Content can build clean-room scenarios around transfers, disputes, card controls, authentication, account servicing, fees, limits, and escalation. Each world couples a simulated user policy with a typed tool API and a state machine. Hidden scenario facts force the agent to ask, verify, refuse, or escalate instead of following a scripted happy path.
Original scenarios vary intent, state, and pressure.
Dynamic behavior
The customer can reveal information gradually, correct the agent, resist policy, change goals, or react to tool outcomes.
Hidden but consistent
Accounts, cards, transactions, authentication status, limits, eligibility, and prior actions live in a reproducible database.
Typed operations
Read and write APIs enforce preconditions, idempotency, confirmation rules, and realistic failure responses.
Operational constraints
Rules define when to authenticate, disclose, proceed, refuse, transfer, or escalate for human review.
Success requires both correct state and compliant process.
| Reward layer | What it checks | Failure example |
|---|---|---|
| Goal completion | Requested account outcome or information | Customer asked for a lock; card remains active |
| Policy compliance | Authentication, consent, disclosure, and escalation | Correct transfer made without required confirmation |
| Tool correctness | Arguments, ordering, idempotency, and retries | Duplicate mutation after a timeout |
| Communication | Accurate explanation of action and remaining steps | Claims completion after tool failure |
| Efficiency | Unnecessary turns and calls without rewarding shortcuts | Repeated reads instead of using known state |
Binary task success stays visible, while criterion-level reward makes near-misses useful. Boundary violations receive explicit labels and cannot be offset by polished language.
One episode yields policy and recovery supervision.
- Initial state: seeded customer profile, account state, hidden goal, policy version, and tool schema.
- Interaction: every user turn, agent message, tool call, tool result, confirmation, and state mutation.
- Verification: final state assertions, policy events, duplicate-action checks, communication review, and reward breakdown.
- Derived data: earliest-failure label, compliant critique, repaired trajectory, and counterfactual user variants.
What a τ³-Banking-shaped program includes.
- Original scenario families with user simulators, seeded databases, banking tool APIs, and versioned policy documents.
- Train, development, and private evaluation splits separated by goal structure and state transition—not just wording.
- Exact final-state checks, event invariants, policy graders, replay tests, and adversarial boundary audits.
- Multi-turn transcripts, tool events, state diffs, reward components, failure labels, critiques, and repairs.
- Model-specific hillclimb batches around authentication, confirmation, escalation, idempotency, and truthful completion reporting.
Benchmark relationship and evidence boundary.
This is a proposed independent clean-room data program inspired by the dynamic user-and-tool capability tested by τ³-bench. It is not official Sierra data, a replica, an affiliation.
Primary reference: τ-bench official site and Sierra τ³-bench research overview.