The State of Broken Online Stores.

The sample

public Shopify storefronts, found from public sources and reachable without a login — so the study is about Shopify stores and says so rather than generalising to every platform. Each store was visited once, at night in its own time zone, from the region closest to its market. A store is in the sample if its walk completed and was scored; a walk that fell over measured nothing and is not counted as a healthy store.

Markets are the store’s own currency, recorded when the sample was built — not the region the browser ran from. A cut by region once put 212 US-market stores in a Canadian row, which is the kind of error that makes a study worthless, so the two are kept separate. A market’s row is only shown once it passes 50 stores.

The method, and what it cannot tell you

A real Chromium browser at 1440×900 and 390×844 walks home → collection → product → add-to-cart → cart → checkout, stopping at the payment step. Deterministic checks run first; a language model judges each page from its screenshot under a fixed set of rules, and is never asked about load time — that is measured twice in code, so a judged finding about a slow page cannot be invented. robots.txt is respected. Nothing is ever ordered.

• One night each. A store that was broken only on the night we visited looks broken; one that broke the next day looks fine. This is a cross-section, not a monitoring record.
• We never complete a purchase, so a purchase event cannot be verified and no finding here claims it was.
• A bot-challenged store is not counted as broken. Being walled is a fact about automated browsers, not about a shopper, so those walks are excluded rather than scored.
• Shopify only. Nothing here says anything about WooCommerce, BigCommerce or a custom build.
• Aggregates only. No store is named, no URL is published and no screenshot of a third-party store appears anywhere in this report. That is the condition of visiting other people’s stores at all.

The findings

Six, each with the number it is a share of and the query that produced it. If you think a number is wrong, the query is there to argue with.

select coalesce(t.market, case s.region when 'syd' then 'AU' when 'iad' then 'US' when 'lax' then 'US' when 'yyz' then 'CA' when 'lhr' then 'UK' when 'fra' then 'EU' else 'other' end) as market, count(*)::int as n,
    count(*) filter (where exists (select 1 from findings f where f.scan_id = s.id and f.severity in ('P1','P2')))::int as broken
  from scans s left join corpus_targets t on t.domain = s.domain
  where s.status = 'complete' and s.score is not null and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)') and s.ip = 'corpus' group by 1 having count(*) >= 50 order by 2 desc

select
    count(*)::int as n,
    percentile_cont(0.5) within group (order by lcp)::int as p50,
    percentile_cont(0.75) within group (order by lcp)::int as p75,
    count(*) filter (where lcp > 2500)::int as over_2500,
    count(*) filter (where lcp > 4000)::int as over_4000
  from (select (select (p->>'lcp')::numeric from jsonb_array_elements(coalesce(s.summary->'speed'->'pages','[]'::jsonb)) p where p->>'viewport' = 'mobile' and p->>'page' = 'home' and (p->>'lcp') is not null limit 1) as lcp from scans s where s.status = 'complete' and s.score is not null and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)') and s.ip = 'corpus') x where lcp is not null and lcp > 0

select
    count(distinct s.id)::int as visits,
    count(distinct s.id) filter (where exists (select 1 from findings f where f.scan_id = s.id and f.lens = 'money'))::int as any_money,
    count(distinct s.id) filter (where exists (select 1 from findings f where f.scan_id = s.id and f.lens = 'money' and lower(f.title) like '%add-to-cart%'))::int as atc,
    count(distinct s.id) filter (where exists (select 1 from findings f where f.scan_id = s.id and f.lens = 'money' and lower(f.title) like '%checkout%'))::int as checkout
  from scans s where s.status = 'complete' and s.score is not null and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)') and s.ip = 'corpus'

select lower(f.title) as title, f.lens, f.severity, count(distinct f.scan_id)::int as scans
  from findings f join scans s on s.id = f.scan_id
  where s.status = 'complete' and s.score is not null and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)') and s.ip = 'corpus' group by 1,2,3 order by 4 desc limit 8

select
    count(*) filter (where s.score >= 90)::int as b90,
    count(*) filter (where s.score >= 70 and s.score < 90)::int as b70,
    count(*) filter (where s.score >= 50 and s.score < 70)::int as b50,
    count(*) filter (where s.score < 50)::int as b0,
    count(*)::int as n
  from scans s where s.status = 'complete' and s.score is not null and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)') and s.ip = 'corpus'

select f.severity, count(distinct f.scan_id)::int as scans
  from findings f join scans s on s.id = f.scan_id
  where s.status = 'complete' and s.score is not null and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)') and s.ip = 'corpus' group by 1 order by 1

A second study · September 2026

Product data on 409 live Shopify storefronts

A different method on a different sample, which is why it is a section rather than a row above: a public-web audit of product data on 409 storefronts, with no browser walk and no logins. Published 14 September 2026, updated 24 September 2026. 71.9% had at least one actionable product-data finding — and that is the wrong number to plan around.

• Median findings per store: 1 (mean 1.28, maximum 6).
• Stores with zero findings: 28.1%.
• 35.7% had no finding except a missing product_type and/or a short description.

The sensitivity table is the point: the headline is carried by two content-quality checks, and the figure that matters for AI shopping feeds is 22.7%. The full study has the nine checks, their denominators and the method.

Cite this

Published under CC BY 4.0: use the figures, with attribution. The aggregates are downloadable as CSV so you do not have to retype them.

Navaal AI. The State of Broken Online Stores. 1 September 2026–2 September 2026. https://navaal.ai/bilby/data

Frozen editions

This page recomputes on every request, which is right for a live report and wrong for a citation. Each edition below is a permanent URL holding exactly what the study said when it was taken — cite one of those and the figure will still read the same next year.

• September 2026 edition — frozen 15 October 2026

By market

STORES VISITED

stores whose visit completed, by the store’s market; a store that refused the browser or errored on our side is not counted as visited anywhere on this page.

BUY PATH BROKEN

share of visited stores with at least one P1 or P2 finding — a step of home → collection → product → add-to-cart → cart → checkout that failed or a broken feature on it — on the night visited, visits to 2 September 2026.

MEDIAN MOBILE LCP

the median, across stores, of the home page’s Largest Contentful Paint at 390×844 in a real Chromium from the store’s market region (PerformanceObserver, one load per store); n is the stores with a mobile measurement.

LCP > 4 S

share of those stores whose mobile home LCP exceeded 4 s on mobile (Google’s “poor” threshold).

TRACKING FINDINGS

share of stores where an add-to-cart or begin-checkout event failed to reach an installed analytics platform’s own collection endpoint during the visit (Money check).

AVG SCORE

mean Bilby health score, 100 minus the severity points of every finding, floored at zero.

Every cell prints from the visits themselves; a market's row is only meaningful once it passes 50 stores. The 26 August 2026 checkout-scripts sunset is the reason the tracking column exists: a store whose add-to-cart or begin-checkout event no longer reaches GA4 / Meta shows up here as a Money check finding on a single night’s visit (purchase itself needs a real order — we never place one).

Methodology

1. Sample. Public Shopify storefronts with at least one product page in the latest Common Crawl monthly index (*.myshopify.com/products/*), de-duplicated by host, password-protected and development stores skipped, the store’s active currency read once to place it in a market. Majority US by construction of the web; every cut prints its n.
2. Visit. The same free scan every merchant can run: a real Chromium browser, desktop 1440×900 and mobile 390×844, from the nearest of six regions, home → collection → product → add-to-cart → cart → checkout, stopping at the payment step. robots.txt is respected; a store that refuses is not visited and not counted.
3. Findings. Deterministic checks first (HTTP, cart endpoint, tracking requests, Web Vitals); a frozen Claude judge only where a human eye is needed, confirmed by a second signal before it becomes P1. Rate limits (429/403) are never findings. Rules: /bilby/methodology.
4. Cuts. Market = the store’s active currency, read once when the sample was built (USD → US, GBP → UK, CAD → CA, AUD → AU, EUR → EU); the visit itself ran from the nearest of six regions. Speed rows count a mobile homepage LCP above 4 s, measured on our machine class and discounted when our own browser was starved.
5. Limits, printed. One night per store; no purchase events (no orders placed); stores that wall automated browsers are absent; the sample is what Common Crawl saw, not the whole platform. Bilby scores its own store publicly under the same rules, and a separate study checked the product data on 409 live Shopify stores.

Press kit

• Cite as: The State of Broken Online Stores, Navaal (Bilby), 2026, https://navaal.ai/bilby/data — CC BY 4.0.
• Live figures update nightly; the table above is the source of truth, and the measurement line under the headline dates every figure. Quote the numbers with that date.
• Marks: bilby mark · navaal wordmark (BRAND.md rules). Contact: support@navaal.ai — Waqas Ahmad, Navaal, Sydney.
• The demo store used in every screenshot and the film is our own labelled store, Fernwick & Co.; no merchant is named without consent.