How we measure: ACM-1 in plain language

The scenario is the unit

A scenario is one buyer situation: a shopper persona, a use case and two or three ranked priorities, with at most one feasible hard requirement and no budget bands. Each scenario is asked in several wordings, with the order of the options rotated and their labels randomized.

Results are computed per scenario: asking the same scenario again never counts as new evidence.

Models are asked in English: scenarios and prompts are written in English. The site is available in English, Russian and Spanish, and results are shown in your language through translated labels for fixed codes. Measuring the query language as a factor is not part of current editions.

The judge panel and who is counted

Judges are AI models called through APIs: the vendor's own API, or an API gateway with a pinned provider. Every result names each API model and its version, how it was called, the conditions and the date, never a consumer chat app.

Before a run, the inclusion rule is fixed: a judge must give enough valid answers across the planned scenarios. Judges that miss it are listed with their reasons and never averaged in. Rows with too little data are withheld and listed too.

The consensus gives equal weight to each model family by default and is labelled normative; the per-judge results are the primary measure.

Intervals and “too close to call”

Under forced choice, judges must pick one of the shown options (there is no abstain option in battles or boards, and invalid answers are counted), and every choice share comes with a confidence interval. A judge, a buyer need or the consensus leans to a brand only when its whole interval lies above or below 50%. Otherwise the result is too close to call. A point estimate alone never counts as a win. Battle intervals, and board intervals from editions sealed after this rule was adopted, are widened slightly so they stay accurate when there are few scenarios; earlier board editions keep the intervals they were published with.

Per-judge intervals and consensus intervals are computed with different methods, because the consensus combines judges. Each result page names both methods. In battles and in newer board editions, both intervals are widened slightly for small numbers of scenarios (a student-t calibration).

Precision labels: simulated or measured

How precise a result can be depends on the number of scenarios. Until real runs have measured it, precision figures are labelled “simulated”: they come from simulations with known answers. Only then does a result say “measured”.

Pre-registration

Before every run, the scenarios, the analysis plan and the inclusion rule are fixed, and their hash is published with each result. A change applies only to future runs, never to published numbers.

Signed snapshots

Every published result is an immutable snapshot signed with an Ed25519 key. This site is built only from snapshots whose signature it has verified; a snapshot that fails any check stops the build, so nothing unverified is shown.

How to verify a snapshot

1. Download the snapshot from the “Verify this result” link on a battle page.
2. Canonicalize its payload with RFC 8785 (JCS) and check that the SHA-256 of the result equals payload_sha256.
3. Check the detached JWS signature (RFC 7515, EdDSA) over the protected header and the canonical payload, with the public key named by kid.
4. Look the key up in the list of public keys. A snapshot signed with a test key proves nothing: test keys sign sample data only.Public keys (JSON)

The chart, battles and paused consumer surfaces

These features use the same intervals and labels as every battle. The AI Brand Chart and Brand Battles are active in public editions; the battle feed, visitor votes, the quiz, quick battles and the Consilium are currently paused or shown on demo data only.

The battle feed (paused)

Currently paused on demo data. Each card shows a battle's AI consensus interval and, for comparison, what visitors picked. On sample data the visitors are simulated. The gap is the AI share minus the visitor share with a MOVER interval, and it is called a gap only when that interval excludes 0. Your picks, streak and count stay on your device.

The AI Brand Chart

Brands are listed by their consensus AI model choice share in randomized shortlists of four, with intervals. On card-based boards, each brand is shown by its catalog configuration that best fits the scenario's ranked priorities using non-monetary specification facts only (price_not_shown), sealed in the pre-registration before any call; shares measure best-fit configuration competitiveness, not an all-trim or sales average. On brand-level boards for categories without a fact catalog (brand_names_only), models see four brand names and no product cards, and the two board types are never mixed. A position is only the display order of the estimates; tiers and simultaneous rank intervals are the claims. A move is beyond noise only when the MOVER interval of the share change excludes 0.

Which AI shops like you? (paused)

Currently paused on demo data. Your picks are scored against each model with certainty-weighted agreement: on each pick, a model counts more where its own lean is more certain. Each match's interval is the envelope of the items' intervals at fixed weights, not a joint confidence statement. With enough picks, the models tied with the closest one come from a joint bootstrap test over your picks. The score is computed in your browser.

Shopping types (paused, draft): the result leads with a type named after a model family. It is the family of the models the joint test cannot separate from your closest match, or, with a large tie, the families the same test cannot separate. Each family's type is named after the documented personalities of its judges, and one model is named only when the test names exactly one.

Quick battles (paused)

Currently paused. A quick battle is an estimate a visitor asks for, on a cheaper panel with fewer scenarios. It is cached for a while, never indexed or listed, and never enters a headline number.

The AI Shopper Report Card (paused)

The Report Card honesty battery is currently paused and not part of active public editions. When active, it tests each named API model on synthetic and matched-twin offers with known answers: whether it picks a strictly dominating offer at shortlist sizes k = 2, 4 and 8, whether slot order shifts its picks, how often rotating options or rewording the prompt flips its answer beyond repeat noise, whether a brand label shifts picks between identical specs, and whether it obeys hard buyer constraints or abstains when no offer fits. Every metric is reported per judge with a scenario-cluster interval and its chance or zero reference, without letter grades.

How results are ordered

On the AI Brand Chart, in a battle and in a quick battle, brands are ordered by one measured parameter: each brand's consensus AI model choice share from the named API models across randomized shortlists. Each share has an interval, and a position is only the order of the estimates; tiers and intervals are the claims, as explained above. On the Consilium page (currently paused on sample replay), the offers are listed by price, lowest first, and each row shows which models picked it. No payment or business relationship with us changes any of these orders or decides which options appear.