How Tide’s evals work

The real Coach, asked realistic questions on real bank data, graded blind — every failure traced to where it started, fixed there, and proven before it shipped. One cycle, following Hamel Husain and Shreya Shankar’s method.

47 → 54

answers passing, of 100 (round 1 → round 2)

failure types found and named

45 of 53

failures weren’t the model’s reasoning

failure types fixed at the source — zero in round 2

01Setup

What we test

What we did

Asked the live Coach realistic questions on the owner’s real bank data. 100 scenarios built from a grid — 10 jobs a person hires the Coach for × 7 steps of a good answer, all 10 red lines covered. Every answer is traced end to end: each model call, each tool call and exactly what it returned.

Why (the method)

Test the thing you ship, on the data it really sees. Sample the space systematically so coverage is measured, not assumed.

What we found

The previous scripted suite faked every tool result, so it could never see a tool returning the wrong thing — which is where most failures turned out to be.

0a–0cTraces, masking, token countsdone

What

OpenTelemetry spans for each turn (agent), each model call (LLM) and each tool call (tool, with its full output), exported to Arize AX. Names, emails, account numbers and IDs are masked; amounts, dates and merchants are kept so failures can be diagnosed.

Why

Error analysis starts with looking at real traces. The old tests faked every tool result, so they could never see a tool returning the wrong thing.

Result

Content is exported only for the owner’s own account, so future beta users are protected by default.

Where it lives

services/tracing/coach_tracing.ts services/tracing/trace_masking.ts PRs #314, #316

1a–1bDimensions → tuples → 100 questions, owner-revieweddone

What

Four dimensions (the job the person hired the Coach for, which step of a good answer the question strains, how it is asked, single or multi-turn) crossed into tuples, then written as 100 questions covering all 10 jobs, all 7 stages and all 10 red lines. Each is tagged with the tools a good answer needs.

Why

Sample the space systematically so coverage is measured, not assumed.

Where it lives

evals/scenarios/round-1.json PR #317

2aOne Coach entry pointdone

What

The app and the runner call the same function (runCoachTurn); only the model provider differs, behind one small interface.

Why

Test the thing you ship. A test rig that re-implements the loop drifts from production.

Where it lives

services/ai/coach_turn.ts PR #318

2bLive runnerdone

What

`node ace coach:scenarios` runs scenarios on real data, saves nothing to chat tables, and writes a run record with every answer and every masked tool output. A cost cap stops a runaway run; an incomplete run gets “no score”, never a fake 0%.

Why

Live runs discover problems as the real data changes.

Result

Round 1: 100/100 scenarios answered.

Where it lives

commands/coach_scenarios.ts evals/runs/<run_id>.json PR #319

2cFrozen replaydone

What

Re-asks a recorded run’s questions under today’s prompt and code, with the clock set back to the recording and every tool call answered from the recording — never the live database.

Why

Live data moves every day, so a live before/after cannot separate “my change” from “my week”. A replay can: the only difference is the change. Live runs discover; frozen replays prove.

How

Each tool call is matched to the recording: exact (same tool and arguments), approximate (same tool, other arguments — flagged), or off-script (not recorded — a clear error, never invented data). Off-script scenarios are marked as less-clean comparisons.

Frozen	Example
The question	“Can I afford a $90 dinner tonight?”
The moment	The original run’s date and time
Every tool call and its exact output	Safe to Spend → balance $1,250.00, today $84.00, card payments due later in the month
Not frozen (what is under test)	The system prompt, the tool descriptions, the Coach code, the model’s answer

What a recording freezes (illustrative figures)

Result

Limits: a replay cannot show tool, data or engine fixes (it serves the old outputs), so those are proven with live re-runs.

Where it lives

services/evals/scenario_runner.ts PR #330

02Analyze

Grade the answers, group the failures

What we did

The owner graded the first answers (Pass/Fail plus the first thing that went wrong). An AI judge was tuned on one blind set, frozen, and tested once on another before grading the rest. The 53 failures were grouped by first upstream cause into 14 types.

Why (the method)

The owner’s judgement is the standard; a judge may stand in only after agreeing on examples it wasn’t tuned on. Then fix specification failures first.

What we found

Round 1 passed 47 of 100. And 45 of 53 failures weren’t the model’s reasoning — a prompt that never said, a tool that returned the wrong thing, an engine edge case, missing data.

Judge alignment	Agreement
Judge v1, blind set 1	7 / 10
Tuned on set 1, then frozen	10 / 10
Frozen judge, held-out set 2	7 / 9
v2.2 on all 20 owner-graded answers*	20 / 20

*Partly in-sample — certification on fresh labels is on the backlog.

#	Failure type (round 1)	Count	Fix lives in
1	Payday Safe to Spend treated as real money	6	engine
2	Card payments falsely said to be 'already factored in'	5	prompt + tool description
3	Pending + posted treated as a double charge or fraud	6	tools + prompt
4	Negative 'uncategorized' bucket read as refunds or savings	6	data
5	Ambiguous question answered without asking which meaning	6	prompt
6	Distress handled badly: dismissive, no crisis resources	3	prompt
7	Tool logic wrong: anomaly flags habitual merchants, expenses include transfers, recurring misses repeated transfers	3	tools
8	Bills/subscriptions miss card payments and card-billed subscriptions	3	tool
9	Assumes facts it doesn't have, or doesn't check the user's premise	5	prompt
10	Arithmetic or direction errors	3	code check (phase 6)
11	Income detection wrong: transfers called pay, several deposits stated as one certain paycheck	2	engine
12	Stale or partial data not disclosed	3	prompt + tool output
13	Product stance: not a budgeting app; the recommendation must lead	2	prompt

3a–3e20 human grades, an aligned judge for the other 80done

What

Every answer gets Pass/Fail and a critique of the first upstream thing that went wrong (open coding). The owner graded two blind sets of 10 in Arize; the AI judge was tuned on set 1, frozen, then tested once on held-out set 2, and graded the remaining 80.

Why

Train on one set, test once on another: a judge measured on the examples it was tuned on flatters itself.

Step	Agreement with the owner
Judge v1 on blind set 1	7 / 10
Tuned (v2.1) on set 1	10 / 10
Frozen v2.1 on held-out set 2	7 / 9
v2.2 re-grading all 20 owner-graded answers (partly in-sample)	20 / 20

Judge alignment

Result

Round 1: 47 of 100 passed.

Where it lives

evals/labels/round-1.json PR #329

4a14 types (axial coding)done

What

Types include: payday Safe-to-Spend treated as real money; card payments falsely “already factored in”; pending and posted read as a double charge; ambiguous questions answered without asking; distress handled badly; tool logic errors; bills missing card payments; assuming facts; arithmetic; income detection; stale data; product stance; and a prompt describing features that no longer exist.

Why

Fix specification failures first: if the prompt or a tool never told the model the truth, the model is not the problem.

Result

45 of 53 failures were prompt, tool, engine or data problems — not model reasoning.

Where it lives

evals/labels/round-1-failure-types.json PR #330

03Improve

Fix each failure where it lives

What we did

Fixed each failure type in the layer it lives in — prompt, tool, engine or data — and proved every fix before it shipped, graded blind by the same frozen judge on both sides.

Why (the method)

A prompt can’t out-argue a confidently wrong tool, so facts are fixed in tools and data, behaviour in the prompt. Prompt changes are proven on frozen replays; tool and engine changes on live re-runs.

What we found

Prompt iterations on frozen replays took round 1 from 47 to 52 → 56, and the tool fixes took their scenarios from 1/14 to 8/14. Reading every regression — not just the total — caught four new prompt rules overreaching.

Layer	What was fixed	How it was proven
Prompt	Dead features removed; card-payment truth; ask when ambiguous; person first in distress; not a budgeting app	Frozen replays, 3 iterations
Engine	Payday + month end collapsed the look-ahead to one day; a paycheck counted twice	Unit tests on the failing day; unchanged on ordinary days
Tools	Each charge once; card payments in “bills coming up”; smarter anomalies; full pay schedule	Live re-runs of the affected scenarios
Data	New bank category codes left paychecks uncategorized	Backfill + coverage test

Frozen replay, in one picture (illustrative figures)

A live run records each question, the moment it was asked, and every tool output — e.g. “Can I afford a $90 dinner tonight?” with Safe to Spend returning $84.00 for today. To test a prompt change, the same question is asked again with the clock set back and every tool answered from the recording. Only the change differs, so a different verdict is the change’s doing — not a paycheck landing. A replay can’t show tool or engine fixes (it serves the old outputs), so those are proven live.

5aPrompt fix, three iterations on frozen replaysdone

What

Removed two dead tools and prompt content for features Tide no longer has; corrected the Safe-to-Spend description; added rules for card payments, pending vs posted, data freshness, ambiguity, distress (person first; crisis lines only for crisis language) and product stance (not a budgeting app; the recommendation leads).

Why

Prove a prompt change on a frozen replay, graded by the same frozen judge on both sides, blind to which is which.

Iteration	Scope	Pass
Round 1 (before)	100	47
Iteration 1	100	52
Iteration 2	100	56
Iteration 3 (targeted)	21 distress + safety	18 → 19
Final	100	57

Frozen replays of round 1, graded blind by the same judge

Iterations

Iteration 1 fixed 15 of 24 prompt-type failures — and four new rules over-applied (asking instead of answering a clear question; “not debt”; a crisis line for an urgent money question; naming two of three card payments). Reading every regression, not just the total, is what caught them.
Iteration 2 fixed those four; two new side effects of the distress wording appeared and were fixed in iteration 3.
A prompt rule could not stop the Coach defending an impossible number from the engine — evidence that fix belongs in the engine.

Where it lives

services/ai/system_prompt.ts evals/labels/round-1-5a-replays.json PR #331

5bEngine: payday + month end collapsed the horizondone

What

On a day that was both payday and month end, Safe to Spend looked ahead one day, so “this week” multiplied a one-day figure by five unseen days. Now the horizon always covers every window it prices, and on payday the runway rolls to the next paycheck.

Why

A wrong tool number is fixed in the tool. The owner chose the semantics: hold back what is due before the next paycheck; what is due after it is that paycheck’s job.

Result

No change on ordinary days (verified to the cent against the old engine); sane windows on pinned paydays.

Where it lives

services/safe_to_spend_service.ts (v5.6) PR #332

5c–5eTools, glued text, income scheduledone

What

Each charge appears once (a pending charge and its posted twin are one); “bills coming up” includes card payments; insights exclude transfers from expenses, judge habitual merchants against their own norm, and find repeated transfers; text before and after a tool call is two paragraphs; “what comes in next month?” sees the whole pay schedule.

Why

Tool changes cannot show in a replay, so they were proven with live re-runs — and iterating on live data caught three bugs of the fix itself before it shipped.

Result

The 14 affected scenarios went from 1/14 to 8/14 passing; every remaining failure traced to the Coach’s own reasoning, not the tools.

Where it lives

services/ai/coach_insights.ts evals/labels/round-1-5c-live.json PR #333

04Measure

Run it again, automate what’s objective

What we did

A fresh live run of all 100 scenarios after every fix, graded blind, each failure filed under an existing type or marked new. Twelve free code checks now run over every recorded run and in CI.

Why (the method)

Keep coding fresh traces until no new failure types appear — each fix changes the system the next round sees. Automate what’s objective; certify judges for the rest.

What we found

Round 2 passed 54 of 100 (round 1: 47). Failure types fixed at the source fell from 18 to 0. Two new types appeared, and grading surfaced a real product bug — a paycheck counted twice by Safe to Spend — fixed the same day.

	Round 1	Round 2
Pass	47	54
Failures in types fixed at the source	18	0
Arithmetic slips (now a code check)	3	14
New failure types	—	2

4cSaturation check on round 2done

What

After every fix, a fresh live run of all 100 scenarios, graded blind; each failure filed under an existing type or marked new.

Why

Keep coding fresh traces until no new failure types appear — each fix changes the system the next round sees.

	Round 1	Round 2
Pass	47	54
Payday-engine, pending/posted, data, dead-feature failures	18	0
Arithmetic slips (now a code check)	3	14
New failure types	—	2

Round 1 vs round 2 (fresh live runs, graded blind by the same judge)

Result

Round 2: 54 of 100, up from 47. The types fixed at the source (payday engine, pending/posted, uncategorized data, dead features) had zero failures. Two new types appeared: an overcautious “it’s tight, wait” caused by subtracting a payment the window already held back, and meta-commentary leaking into an answer. Card-payment confusion was fixed at the source — the tool now lists which payments each window already holds back — taking the 13 failing scenarios from 0 to 5 passing; the remaining “tight” verdicts are partly a product question for the owner (is a one-day “this week” on Sunday the right yardstick?). Grading also surfaced a real product bug: a confirmed paycheck re-detected as a second income stream was counted twice by Safe to Spend — fixed in the engine (v5.7).

6a12 code checksdone

What

Figure grounding (every money figure must be in the tool outputs or one step from them), right tool, valid tool and arguments, loop sanity, currency format, no internal leakage, account-number and risky-product tripwires, complete trace. They run over any recorded run, free, and in CI.

Why

Use code where the answer is objective; judges only where judgement is needed.

Result

On round 1, the figure-grounding check flagged 14 answers; the judge had failed 12 of them, usually for the very figure flagged. On round 2 a code check caught a regression the judge missed — an internal field name leaking into three answers — fixed and proven on a replay the same hour.

Where it lives

evals/judges/scenario_checks.ts npm run scenario:check

6bWhich types still need a judgedone

What

After Phase 5, the types left for judges are the judgement calls: asking on ambiguity, handling distress, checking the user’s premise, and product stance. Arithmetic is a code check; tool and engine types were fixed at the source.

Why

One judge per failure type, only where code cannot decide.

6c–6ePer-type judges, certified on held-out labelsprovisional

What

Per-type judge briefs are written and run; certifying them needs the owner’s yes/no labels per type, split train/dev/test, with the test set used once.

Why

An AI cannot certify an AI. Until the owner’s held-out labels exist, every judge result is marked provisional.

05Next

What’s next

The cycle is closed; these are on the product backlog.

Certify the judges — on a held-out set of the owner’s own labels, then report corrected pass rates with a range
Switch on the schedule — a weekly live run and a nightly replay, with a monthly read of fresh real traces
Close the feedback loop — fix the in-app thumbs so real usage feeds the next round
Test models before switching — replay the frozen set on any new model and compare
One product question — is a one-day “this week” on a Sunday the right yardstick for “can I afford it this week?”

06Lessons

What this cycle taught

Most failures aren’t the model. 45 of 53 came from the prompt, the tools, the engine or the data. Look at traces before tuning prompts.
Fix it in the layer it lives in. A prompt rule was ignored 5 times out of 5 against an engine returning an impossible number; the engine fix worked.
Prove each change the right way. Frozen replays for prompt and model changes; live re-runs for tool, engine and data changes; the same blind judge on both sides.
A fix can overreach. The total went up while new failures appeared underneath it. Read every regression.
Code checks before judges. The recorded trace is the answer key: every figure must come from the tool outputs. A code check caught a regression the judge missed.