Quotes bucketed by their confidence_score. A well-calibrated model has
actual win-rate ≈ bucket midpoint.
Bucket
Midpoint
Total
Won
Lost
Actual win-rate
Miscalibration (pp)
Win-probability calibration
Same idea using win_probability_at_quoted from each Sonnet quote.
Bucket
Midpoint
Total
Won
Lost
Actual win-rate
Miscalibration (pp)
Margin drift per (segment, delivery mode)
Realized minus projected margin %, in percentage points. Rows with |drift| > 10pp
are flagged for attention.
Segment
Delivery
n
Projected median
Realized median
Drift (pp)
Per-Task Model Selection
Select jobs to change several at once
Job
Model
Hosts, then fallbacks
Thinking
Effort
On
Calls
Cache hit
Avg ms
Errors
Cost = token counts from agent_llm_calls × prices in bench/model_pricing.json.
Cache reads billed at 10% of input price; cache writes at 200% (1h TTL).
Total spend (window)
$0.00
Top purpose
—
Top model
—
Daily spend
Breakdown by purpose
Purpose
Calls
Cost (USD)
$ per call
% of total
Breakdown by model
Model
Calls
Cost (USD)
$ per call
% of total
⚠ Unpriced models in telemetry:
These models appear in agent_llm_calls but aren't in bench/model_pricing.json. Their cost contribution is counted as $0. Add them to the pricing file to get accurate totals.
Run a head-to-head bake-off across models against real production fixtures.
Fixtures + prompts are production-faithful (real April system prompt + RAG retrieval).
Test calls don't write to agent_llm_calls.
How are these fixtures chosen?
Both fixture sets are hand-curated from real production data, picked to span the dimensions where models tend to differ.
Email track (10 fixtures) — pulled from email_compose_audits. Each fixture's ground truth is the actual message Joe sent to the customer in production. Coverage: short SMS acks, standard $595 quotes, dense engagement-letter handoffs, complex pricing defenses, payment confirmations, outbound invoices with Stripe links.
Action track (10 fixtures) — synthesized from real inbox-action runs (where April reads an inbound and proposes next steps + draft). Each fixture has an expected outcome (key points the response should hit; anti-patterns it should avoid) used by the Sonnet judge. Coverage: scope decisions (estates, large M&E inventories), CRM hygiene, out-of-scope detection (insurance disputes, real property), skeptical customers, cold replies, Form 8283 questions.
Source: bench/fixtures/email-prompts.json + bench/fixtures/action-prompts.json. Click any fixture's ▸ below to see exactly what's being tested.
Track
Fixtures
Loading fixtures...
Models
Pick 2-4
Adds ~$0.01-0.03 per cell; ~5s extra per cell.
Saved runs
Every bake-off is saved automatically. Click a row to reload its grid; rename to label important runs.
Loading...
Contact Check
Sales Follow-ups
Last 14 days
Why follow-ups were blocked (7 days)
Drafts to review
Loading...
Stage changes from messages
What April decided to do to a deal's stage after each message to or from a lead (the part of n8n's "Update Attio's stages"). In shadow mode nothing is written to Attio. "Proposal" means n8n would ask a human to approve instead of writing.
Time
Message
Action
Decision
Change
Why
All runs (latest 100)
Outcome:
Mode
Job
Deal
Outcome
Confidence
Blocked By
When
Feedback
Loading...
Every follow-up on the books in the email_jobs queue, whether or not the bot has decided on it yet. Read-only: reschedule or cancel from April chat (reschedule_followup).
Status:
Channel:
Follow-ups due per day
Status breakdown
Follow-ups
Recipient
Deal
Status
Channel
Scheduled
Originally scheduled
Sent
Loading...
Team guidance for the bot
General feedback that applies to every follow-up, not one draft. A new note is Pending and changes nothing until you press Apply. Applied notes (newest 10) are added to the bot's instructions on every decision.
