How Grounds measures selection share changes.
Grounds compares treated and holdout prompts before and after an intervention. This page explains the calculations and evidence requirements for reviewing a result.
Before publishing a change, assign buyer prompts to treated and holdout groups. The holdout group provides a comparison without the planned intervention.
After the first post-intervention measurement, Grounds locks the treated and holdout assignments so later results use the same design.
Sample both groups after 24 hours, 72 hours, 7 days, and 14 days across the configured engines. Matching the timing helps account for changes in model behaviour.
Grounds calculates a two-proportion z-test for treated and holdout selection rates, plus a difference-in-differences against the baseline. The resulting p-value determines the confidence label.
The statistics, plainly
A two-proportion z-test compares the treated arm’s selection rate against the holdout arm’s, within the same measurement window. The p-value uses the pooled standard error (the standard choice under the null hypothesis that both arms behave the same); the 95% confidence interval uses the unpooled, Wald standard error — the standard choice once the interval itself is meant to describe how different the arms might really be. The diff-in-differences carries its own interval, built from the sum of all four arms’ proportion variances (current treated, current holdout, baseline treated, baseline holdout), since they’re independent query clusters, not paired observations.
| Read | Treated | Holdout | Lift | Result |
|---|---|---|---|---|
| 14d | 45% (n=100) | 30% (n=100) | +15 pts | p=0.028, 95% CI [1.7, 28.3] |
This example uses illustrative numbers. Pooled SE gives z≈2.19, two-tailed p≈0.028; Wald SE gives the interval. Run those two proportions through any standard statistics package and you get the same answer.
- Lift before the intervention has been published and verified.
- A confidence label without the p-value it came from, next to it.
- A diff-in-differences without saying so when the baseline arm is missing or small.
- Any read under n=20 without saying the sample is thin, in the same sentence as the number.
Every snapshot, scored observation, and experiment result is written to an append-only table. A Postgres trigger rejects updates and deletions. Reported numbers trace back to these stored records.
Selection share, defined
Selection share is the share of measured buyer prompts where an answer engine names your brand as the preferred option, rather than merely mentioning or citing it. A citation is evidence use; being named the preferred option is a stated preference, which Grounds records as a selection. This is distinct from a live transactional agent actually completing a purchase, booking, or checkout on a buyer’s behalf — Grounds does not currently measure that separate, stricter event, and never presents the two as the same thing.