How Grounds proves a fix worked.
Grounds compares changed questions with an unchanged comparison group before and after a fix. This page explains the calculation and the evidence needed to trust the result.
Before publishing a change, split buyer questions into a changed group and an unchanged comparison group.
After the first follow-up check, Grounds keeps both groups fixed so later results stay comparable.
Sample both groups after 24 hours, 72 hours, 7 days, and 14 days across the configured engines. Matching the timing helps account for changes in model behaviour.
Grounds compares how often each group is picked before and after the change. The result includes an uncertainty range, not just a headline number.
The statistics, plainly
A two-proportion z-test compares the changed group’s pick rate against the comparison group’s, within the same measurement window. The p-value uses the pooled standard error (the standard choice under the null hypothesis that both arms behave the same); the 95% confidence interval uses the unpooled, Wald standard error — the standard choice once the interval itself is meant to describe how different the arms might really be. The diff-in-differences carries its own interval, built from the sum of all four groups’ proportion variances (current changed, current comparison, baseline changed, baseline comparison), since they’re independent question groups, not paired observations.
| Read | Treated | Comparison | Lift | Result |
|---|---|---|---|---|
| 14d | 45% (n=100) | 30% (n=100) | +15 pts | p=0.028, 95% CI [1.7, 28.3] |
This example uses illustrative numbers. Pooled SE gives z≈2.19, two-tailed p≈0.028; Wald SE gives the interval. Run those two proportions through any standard statistics package and you get the same answer.
- Lift before the intervention has been published and verified.
- A confidence label without the p-value it came from, next to it.
- A diff-in-differences without saying so when the baseline arm is missing or small.
- Any read under n=20 without saying the sample is thin, in the same sentence as the number.
Every snapshot, scored observation, and experiment result is written to an append-only table. A Postgres trigger rejects updates and deletions. Reported numbers trace back to these stored records.
What “picked” means
The pick rate is the share of measured buyer questions where an AI answer names your brand as the preferred option, rather than merely mentioning it or using your website as a source. Being named the preferred option is a stated preference, which Grounds records as a pick. This is distinct from a live transactional agent actually completing a purchase, booking, or checkout on a buyer’s behalf — Grounds does not currently measure that separate, stricter event, and never presents the two as the same thing.