Trend
Pass rate across completed runs since 18 Sep 2026.
Method
- What a test is
- A fixed user question with fixed account data behind it, so the same conversation can be replayed on any day and the only thing that changes is the answer.
- When a test passes
- The Coach produced a complete answer, every rule check passed, and the AI judge agreed.
- The two grades
- Rule checks look for specific words, numbers or lookups. The AI judge reads the answer against written criteria, and is shown the same account data the Coach had.
- Why one run isn't enough
- The Coach words things differently each time, so a test can pass one run and fail the next. Those are marked varies; read them across runs rather than from a single one.
- Incomplete runs
- If the setup behind the tests breaks, the run is marked no score and left out of the trend, rather than being shown as a very low score.
- How tests are grouped
- Two words, used the same way on the archive: a test sits at the stage it is built to catch a mistake at (A–G) and covers one job a person came with (1–10). Its tag says both —
C3is stage C for job 3. - Money formatting
- The Coach is asked to write amounts as $X,XXX.XX. That is measured on every answer and reported on its own, deliberately outside pass and fail, so a single formatting rule cannot mask what each test is really checking.
- History
- Starts 18 Sep 2026, the first run of the current setup.
- More detail
- How this suite works · the real-data results that replaced it
Tide · generated from recorded eval runs