Bilby · the data report
The State of Broken Shopify Stores.
Of 556 public Shopify storefronts walked in a real browser through to the payment step, 69% had a serious issue (P1 or P2) on the night they were walked.
The sample
public Shopify storefronts, found from public sources and reachable without a login — so the study is about Shopify stores and says so rather than generalising to every platform. Each store was visited once, at night in its own time zone, from the region closest to its market. A store is in the sample if its walk completed and was scored; a walk that fell over measured nothing and is not counted as a healthy store.
Markets are the store’s own currency, recorded when the sample was built — not the region the browser ran from. A cut by region once put 212 US-market stores in a Canadian row, which is the kind of error that makes a study worthless, so the two are kept separate. A market’s row is only shown once it passes 100 stores.
The method, and what it cannot tell you
A real Chromium browser at 1440×900 and 390×844 walks home → collection → product → add-to-cart → cart → checkout, stopping at the payment step. Deterministic checks run first; a language model judges each page from its screenshot under a fixed set of rules, and is never asked about load time — that is measured twice in code, so a judged finding about a slow page cannot be invented. A store whose robots.txt disallows the whole site is skipped. Nothing is ever ordered.
Each store, once, at night in its own time zone
1440×900 and 390×844, a real Chromium browser, robots.txt disallow-all respected
- Homeloads, renders
- Collectionproducts listed
- Productprice, variants
- Add to cartdoes the cart move
- Cartsubtotal, shipping
- Checkoutstops here
Measured in CODE: load time, layout shift, blocking time — so the model cannot invent a slow page.
Judged from the screenshot under fixed rules: everything else. Nothing is ever ordered.
- One walk each. A store that was broken only on the night it was walked looks broken; one that broke the next day looks fine. This is a cross-section, not a monitoring record.
- Bilby never completes a purchase, so a purchase event cannot be verified and no finding here claims it was.
- A bot-challenged store is not counted as broken. Being walled is a fact about automated browsers, not about a shopper, so those walks are excluded rather than scored.
- Shopify only. Nothing here says anything about WooCommerce, BigCommerce or a custom build.
- Aggregates only. No store is named, no URL is published and no screenshot of a third-party store appears anywhere in this report. That is the condition of visiting other people’s stores at all.
The findings
Six, each with the number it is a share of and the query that produced it. If you think a number is wrong, the query is there to argue with.
More than half the stores had a serious issue (P1 or P2) on the night it was walked
69%A serious issue (P1 or P2) means a step of home → collection → product → add-to-cart → cart → checkout failed, or something on it was broken. Not a slow page: a broken one. No share, average or median is printed until at least 100 stores are counted: at 100, one store moves a share by one point, and below that a percentage claims more than the count can carry.
The query this number came from
select s.id, s.domain, coalesce(s.finished_at, s.created_at) as at, coalesce(t.market, case s.region when 'syd' then 'AU' when 'iad' then 'US' when 'lax' then 'US' when 'yyz' then 'CA' when 'lhr' then 'UK' when 'fra' then 'EU' else 'other' end) as market, (select (p->>'lcp')::numeric from jsonb_array_elements(coalesce(s.summary->'speed'->'pages','[]'::jsonb)) p where p->>'viewport' = 'mobile' and p->>'page' = 'home' and (p->>'lcp') is not null limit 1) as lcp
from scans s left join corpus_targets t on t.domain = s.domain
where s.status = 'complete' and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)')
and coalesce(s.ip, '') not in ('repro', 'canary', 'far') and s.repro_of is null
and coalesce(s.finished_at, s.created_at) < $1::timestamptz
and coalesce(s.finished_at, s.created_at) > $1::timestamptz - interval '12 months'
and s.ip = 'corpus'
order by s.domain, coalesce(s.finished_at, s.created_at) descThe median mobile homepage took longer to show its main content than Google's good threshold of 2.5 s
2.8 sGoogle's good threshold for largest-contentful-paint is 2.5 s and its poor threshold is 4 s. This is a real phone viewport in a real browser, not a simulation.
The query this number came from
select s.id, s.domain, coalesce(s.finished_at, s.created_at) as at, coalesce(t.market, case s.region when 'syd' then 'AU' when 'iad' then 'US' when 'lax' then 'US' when 'yyz' then 'CA' when 'lhr' then 'UK' when 'fra' then 'EU' else 'other' end) as market, (select (p->>'lcp')::numeric from jsonb_array_elements(coalesce(s.summary->'speed'->'pages','[]'::jsonb)) p where p->>'viewport' = 'mobile' and p->>'page' = 'home' and (p->>'lcp') is not null limit 1) as lcp
from scans s left join corpus_targets t on t.domain = s.domain
where s.status = 'complete' and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)')
and coalesce(s.ip, '') not in ('repro', 'canary', 'far') and s.repro_of is null
and coalesce(s.finished_at, s.created_at) < $1::timestamptz
and coalesce(s.finished_at, s.created_at) > $1::timestamptz - interval '12 months'
and s.ip = 'corpus'
order by s.domain, coalesce(s.finished_at, s.created_at) descMore than one store in seven had purchase-path tracking missing or broken
16%A Money finding means an event that should have fired on add-to-cart or at the start of checkout did not reach the pixel. Purchase itself cannot be verified without placing an order, and Bilby never places one.
The query this number came from
-- $1: the walks the population rule counted, one per store (lib/store-figures.ts, printed on /bilby/methodology)
select
count(distinct s.id)::int as visits,
count(distinct s.id) filter (where exists (select 1 from findings f where f.scan_id = s.id and f.lens = 'money'))::int as any_money,
count(distinct s.id) filter (where exists (select 1 from findings f where f.scan_id = s.id and f.lens = 'money' and lower(f.title) like '%add-to-cart%'))::int as atc,
count(distinct s.id) filter (where exists (select 1 from findings f where f.scan_id = s.id and f.lens = 'money' and lower(f.title) like '%checkout%'))::int as checkout
from scans s where s.id = any($1::text[])The eight faults Bilby found most often
8%The share of stores carrying each fault on the walk that was counted. One store can carry several.
The query this number came from
-- $1: the walks the population rule counted, one per store (lib/store-figures.ts, printed on /bilby/methodology) select lower(f.title) as title, f.lens, f.severity, count(distinct f.scan_id)::int as scans from findings f join scans s on s.id = f.scan_id where s.id = any($1::text[]) group by 1,2,3 order by 4 desc limit 8
The score each store's own report gave it
50 / 100How the score works: a store where a customer can complete the buy path starts at 100, and loses 10 for each thing that hides a sale and 5 for each slow page. A store where a customer cannot buy scores 0, whatever else is right about it. A visit that could not finish the buy path gets no score at all. This is the average of that score, the same number each store's own report prints.
The query this number came from
select s.id, s.domain, coalesce(s.finished_at, s.created_at) as at, coalesce(t.market, case s.region when 'syd' then 'AU' when 'iad' then 'US' when 'lax' then 'US' when 'yyz' then 'CA' when 'lhr' then 'UK' when 'fra' then 'EU' else 'other' end) as market, (select (p->>'lcp')::numeric from jsonb_array_elements(coalesce(s.summary->'speed'->'pages','[]'::jsonb)) p where p->>'viewport' = 'mobile' and p->>'page' = 'home' and (p->>'lcp') is not null limit 1) as lcp
from scans s left join corpus_targets t on t.domain = s.domain
where s.status = 'complete' and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)')
and coalesce(s.ip, '') not in ('repro', 'canary', 'far') and s.repro_of is null
and coalesce(s.finished_at, s.created_at) < $1::timestamptz
and coalesce(s.finished_at, s.created_at) > $1::timestamptz - interval '12 months'
and s.ip = 'corpus'
order by s.domain, coalesce(s.finished_at, s.created_at) descSeverity, as the product itself grades it
384 storesThe number is the headline's: stores whose counted walk found a sale it stops or a P1 or P2 finding the report states. The bars show, for each grade, the share of stores with at least one finding of that grade as the walk recorded it; a store with findings of two grades is in both bars. P1 stops a sale, P2 damages one, P3 and P4 are worth fixing and will not cost you tonight.
The query this number came from
-- $1: the walks the population rule counted, one per store (lib/store-figures.ts, printed on /bilby/methodology) select f.severity, count(distinct f.scan_id)::int as scans from findings f join scans s on s.id = f.scan_id where s.id = any($1::text[]) group by 1 order by 1
A second study · September 2026
Product data on 409 live Shopify storefronts
A different method on a different sample, which is why it is a section rather than a row above: a public-web audit of product data on 409 storefronts, with no browser walk and no logins. Published 14 September 2026, updated 24 September 2026. 71.9% had at least one actionable product-data finding — and that is the wrong number to plan around.
| How you define “actionable finding” | Share of stores | Of |
|---|---|---|
| All nine checks | 71.9% | 294/409 |
| Excluding missing product_type and thin descriptions | 36.2% | 148/409 |
| Only attributes AI shopping feeds require, plus policy pages | 22.7% | 93/409 |
| Only structured data (missing or duplicated) | 15.9% | 65/409 |
| Only crawler access | 0.5% | 2/409 |
- Median findings per store: 1 (mean 1.28, maximum 6).
- Stores with zero findings: 28.1%.
- 35.7% had no finding except a missing product_type and/or a short description.
The sensitivity table is the point: the headline is carried by two content-quality checks, and the figure that matters for AI shopping feeds is 22.7%. The full study has the nine checks, their denominators and the method.
Cite this
Published under CC BY 4.0: use the figures, with attribution. The aggregates are downloadable as CSV so you do not have to retype them.
Navaal. The State of Broken Online Stores. 1 September 2026. https://navaal.ai/bilby/data
Frozen editions
This page recomputes its figures once an hour, which is right for a live report and wrong for a citation. Each edition below is a permanent URL holding exactly what the study said when it was taken — cite one of those and the figure will still read the same next year. An edition taken before 14 November 2026 counted every visit, not one walk per store, and averaged the older stored score.
- September 2026 edition — frozen 15 October 2026
By market
| MARKET | STORES COUNTED | SERIOUS ISSUE (P1/P2) | MEDIAN MOBILE HOME LCP | MOBILE HOME LCP > 4 S | PURCHASE-PATH TRACKING FINDINGS | AVG SCORE |
|---|---|---|---|---|---|---|
| US | 340 | 74% | 2.7 s · n=321 | 30% | 18% | 49 |
| AU | 119 | 56% | 3.2 s · n=116 | 34% | 10% | 51 |
| UK | 34 | — | — · n=32 | — | — | — |
| EU | 33 | — | — · n=32 | — | — | — |
| CA | 30 | — | — · n=28 | — | — | — |
- STORES COUNTED
- stores of the study’s sample counted by the rule below, by the store’s market; a store that refused the browser, or whose walk could not finish, is not counted anywhere on this page.
- SERIOUS ISSUE
- share of those stores whose counted walk found a sale it stops or a P1 or P2 finding — a step of home → collection → product → add-to-cart → cart → checkout that failed, or a broken feature on it.
- MEDIAN MOBILE LCP
- the median, across stores, of the home page’s Largest Contentful Paint at 390×844 in a real Chromium from the store’s market region (PerformanceObserver, one load per store); n is the stores with a mobile measurement.
- LCP > 4 S
- share of the market’s stores whose mobile home LCP exceeded 4 s (Google’s “poor” threshold).
- TRACKING FINDINGS
- share of stores where an add-to-cart or begin-checkout event failed to reach an installed analytics platform’s own collection endpoint during the walk (Money check).
- AVG SCORE
- How the score works: a store where a customer can complete the buy path starts at 100, and loses 10 for each thing that hides a sale and 5 for each slow page. A store where a customer cannot buy scores 0, whatever else is right about it. A visit that could not finish the buy path gets no score at all.
Methodology
- Sample. Public Shopify storefronts with at least one product page in the latest Common Crawl monthly index (
*.myshopify.com/products/*), de-duplicated by host, password-protected and development stores skipped, the store’s active currency read once to place it in a market. Majority US by construction of the web; every cut prints its n. - Walk. The same free scan every merchant can run: a real Chromium browser, desktop 1440×900 and mobile 390×844, from the nearest of six regions, home → collection → product → add-to-cart → cart → checkout, stopping at the payment step. A store whose robots.txt disallows the whole site is not walked and not counted. Rules for single paths are not read, so the walk opens the cart and the first checkout page as a shopper does. Walks added after September are lighter: desktop only, with no search and no newsletter box.
- Findings. Deterministic checks first (HTTP, cart endpoint, tracking requests, Web Vitals); a frozen Claude judge only where a human eye is needed, confirmed by a second signal before it becomes P1. Rate limits (429/403) are never findings. Rules: /bilby/methodology.
- Cuts. Market = the store’s active currency, read once when the sample was built (USD → US, GBP → UK, CAD → CA, AUD → AU, EUR → EU); the walk itself ran from the nearest of six regions. Speed rows count a mobile homepage LCP above 4 s, measured on Bilby’s machine class and discounted when its own browser was starved.
- Limits, printed. One counted walk per store; no purchase events (no orders placed); stores that wall automated browsers are absent; the sample is what Common Crawl saw, not the whole platform. A separate study checked the product data on 409 live Shopify stores.
Press kit
- Cite as: The State of Broken Online Stores, Navaal (Bilby), 2026, https://navaal.ai/bilby/data — CC BY 4.0.
- Live figures are recomputed once an hour from the walk records, and every figure prints the day it was computed. Quote the numbers with that date, or cite a frozen edition.
- Marks: bilby mark · navaal wordmark (BRAND.md rules). Contact: support@navaal.ai — Waqas Ahmad, Navaal, Sydney.
- The demo store used in every screenshot and the film is Fernwick & Co., one of Bilby’s own test stores; no merchant is named without consent.
Not sure if your store is okay right now? Bilby walks it like a customer and shows you.