Bilby · the data report

The State of Broken Shopify Stores.

Across every store Bilby has walked, not just this study’s sample: 874 real stores, each counted once, walks from 27 August to 15 November 2026; 58% had at least one serious issue (P1 or P2), and the average score was 57 / 100. Counted to the end of 15 November 2026, Sydney time. The same figure, from the same function, is on the methodology page, which states the rule.

  • US830 stores · 340 counted
  • AU362 stores · 119 counted
  • EU176 stores · 33 counted
  • UK67 stores · 34 counted
  • CA65 stores · 30 counted
The sample: public Shopify storefronts from the latest Common Crawl index, verified as live Shopify and placed in a market by their currency. Grey is the sample; green is the stores counted by the rule under the market table. Live from the walk records.

Of 556 public Shopify storefronts walked in a real browser through to the payment step, 69% had a serious issue (P1 or P2) on the night they were walked.

556 stores, one counted walk each. The study's walks were made on 1 September 2026 (UTC); the figure was computed afterwards, on 16 November 2026 (Sydney), over the walks up to the end of 15 November 2026. Each real store counts once, by its latest walk in the 12 months to the last midnight in Sydney that reached a verdict (a walk that could not finish is skipped for the one before it); Bilby's own test stores, its canary stores and its own sites are left out, and so are the second walks it runs to double-check a fault or to look from another region. The average is over the stores whose counted walk has a score: a store whose homepage did not open for Bilby counts as a serious issue and has no score. The study is the same rule, narrowed to the stores of its Common Crawl sample. Bilby’s own test stores and its own sites are excluded from every figure. Why this count and share differ from the figure at the top of the page: that figure counts every real store Bilby has walked, each by its latest finished walk, while this study counts only the stores of its Common Crawl sample, each by its latest finished walk made for the study. The sample is only a part of that whole, so the shares differ; neither count includes Bilby's own stores, test walks or unfinished walks. Both shares use the same measure: a store counts as having a serious issue (P1 or P2) when its counted walk found a sale it stops or a P1 or P2 finding.

The sample

public Shopify storefronts, found from public sources and reachable without a login — so the study is about Shopify stores and says so rather than generalising to every platform. Each store was visited once, at night in its own time zone, from the region closest to its market. A store is in the sample if its walk completed and was scored; a walk that fell over measured nothing and is not counted as a healthy store.

Markets are the store’s own currency, recorded when the sample was built — not the region the browser ran from. A cut by region once put 212 US-market stores in a Canadian row, which is the kind of error that makes a study worthless, so the two are kept separate. A market’s row is only shown once it passes 100 stores.

The method, and what it cannot tell you

A real Chromium browser at 1440×900 and 390×844 walks home → collection → product → add-to-cart → cart → checkout, stopping at the payment step. Deterministic checks run first; a language model judges each page from its screenshot under a fixed set of rules, and is never asked about load time — that is measured twice in code, so a judged finding about a slow page cannot be invented. A store whose robots.txt disallows the whole site is skipped. Nothing is ever ordered.

Each store, once, at night in its own time zone

1440×900 and 390×844, a real Chromium browser, robots.txt disallow-all respected

  1. Homeloads, renders
  2. Collectionproducts listed
  3. Productprice, variants
  4. Add to cartdoes the cart move
  5. Cartsubtotal, shipping
  6. Checkoutstops here

Measured in CODE: load time, layout shift, blocking time — so the model cannot invent a slow page.

Judged from the screenshot under fixed rules: everything else. Nothing is ever ordered.

The payment step is where it stops, in every market, on every night. That is the line between a study of whether a store can be bought from and a study of whether its homepage answers.
  • One walk each. A store that was broken only on the night it was walked looks broken; one that broke the next day looks fine. This is a cross-section, not a monitoring record.
  • Bilby never completes a purchase, so a purchase event cannot be verified and no finding here claims it was.
  • A bot-challenged store is not counted as broken. Being walled is a fact about automated browsers, not about a shopper, so those walks are excluded rather than scored.
  • Shopify only. Nothing here says anything about WooCommerce, BigCommerce or a custom build.
  • Aggregates only. No store is named, no URL is published and no screenshot of a third-party store appears anywhere in this report. That is the condition of visiting other people’s stores at all.

The findings

Six, each with the number it is a share of and the query that produced it. If you think a number is wrong, the query is there to argue with.

More than half the stores had a serious issue (P1 or P2) on the night it was walked

69%

A serious issue (P1 or P2) means a step of home → collection → product → add-to-cart → cart → checkout failed, or something on it was broken. Not a slow page: a broken one. No share, average or median is printed until at least 100 stores are counted: at 100, one store moves a share by one point, and below that a percentage claims more than the count can carry.

  • US74% of 340
  • AU56% of 119
n = 556 stores, one walk each
The query this number came from
select s.id, s.domain, coalesce(s.finished_at, s.created_at) as at, coalesce(t.market, case s.region when 'syd' then 'AU' when 'iad' then 'US' when 'lax' then 'US' when 'yyz' then 'CA' when 'lhr' then 'UK' when 'fra' then 'EU' else 'other' end) as market, (select (p->>'lcp')::numeric from jsonb_array_elements(coalesce(s.summary->'speed'->'pages','[]'::jsonb)) p where p->>'viewport' = 'mobile' and p->>'page' = 'home' and (p->>'lcp') is not null limit 1) as lcp
  from scans s left join corpus_targets t on t.domain = s.domain
  where s.status = 'complete' and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)')
     and coalesce(s.ip, '') not in ('repro', 'canary', 'far') and s.repro_of is null
     and coalesce(s.finished_at, s.created_at) < $1::timestamptz
     and coalesce(s.finished_at, s.created_at) > $1::timestamptz - interval '12 months'
     and s.ip = 'corpus'
  order by s.domain, coalesce(s.finished_at, s.created_at) desc

The median mobile homepage took longer to show its main content than Google's good threshold of 2.5 s

2.8 s

Google's good threshold for largest-contentful-paint is 2.5 s and its poor threshold is 4 s. This is a real phone viewport in a real browser, not a simulation.

  • median (p50)2.84 s
  • 75th percentile4.46 s
  • Google's good line2.50 s
  • Google's poor line4.00 s
n = 529 stores with a mobile homepage measurement
The query this number came from
select s.id, s.domain, coalesce(s.finished_at, s.created_at) as at, coalesce(t.market, case s.region when 'syd' then 'AU' when 'iad' then 'US' when 'lax' then 'US' when 'yyz' then 'CA' when 'lhr' then 'UK' when 'fra' then 'EU' else 'other' end) as market, (select (p->>'lcp')::numeric from jsonb_array_elements(coalesce(s.summary->'speed'->'pages','[]'::jsonb)) p where p->>'viewport' = 'mobile' and p->>'page' = 'home' and (p->>'lcp') is not null limit 1) as lcp
  from scans s left join corpus_targets t on t.domain = s.domain
  where s.status = 'complete' and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)')
     and coalesce(s.ip, '') not in ('repro', 'canary', 'far') and s.repro_of is null
     and coalesce(s.finished_at, s.created_at) < $1::timestamptz
     and coalesce(s.finished_at, s.created_at) > $1::timestamptz - interval '12 months'
     and s.ip = 'corpus'
  order by s.domain, coalesce(s.finished_at, s.created_at) desc

More than one store in seven had purchase-path tracking missing or broken

16%

A Money finding means an event that should have fired on add-to-cart or at the start of checkout did not reach the pixel. Purchase itself cannot be verified without placing an order, and Bilby never places one.

  • any tracking finding16% of 556
  • add-to-cart not recorded15% of 556
  • checkout start not recorded7% of 556
n = 556 stores, one walk each
The query this number came from
-- $1: the walks the population rule counted, one per store (lib/store-figures.ts, printed on /bilby/methodology)
select
    count(distinct s.id)::int as visits,
    count(distinct s.id) filter (where exists (select 1 from findings f where f.scan_id = s.id and f.lens = 'money'))::int as any_money,
    count(distinct s.id) filter (where exists (select 1 from findings f where f.scan_id = s.id and f.lens = 'money' and lower(f.title) like '%add-to-cart%'))::int as atc,
    count(distinct s.id) filter (where exists (select 1 from findings f where f.scan_id = s.id and f.lens = 'money' and lower(f.title) like '%checkout%'))::int as checkout
  from scans s where s.id = any($1::text[])

The eight faults Bilby found most often

8%

The share of stores carrying each fault on the walk that was counted. One store can carry several.

  • google analytics 4 didn't record the add-to-cart (P2)8% of 556
  • meta pixel didn't record the add-to-cart (P2)8% of 556
  • google analytics 4 didn't record the start of checkout (P2)5% of 556
  • a product page shows an error instead of the product (P2)4% of 556
  • desktop home page paints its main content after 5.1 s (P3)2% of 556
  • meta pixel didn't record the start of checkout (P2)2% of 556
  • mobile home page paints its main content after 4.3 s (P2)2% of 556
  • the mobile homepage scrolls sideways (P3)2% of 556
n = 556 stores, one walk each
The query this number came from
-- $1: the walks the population rule counted, one per store (lib/store-figures.ts, printed on /bilby/methodology)
select lower(f.title) as title, f.lens, f.severity, count(distinct f.scan_id)::int as scans
  from findings f join scans s on s.id = f.scan_id
  where s.id = any($1::text[]) group by 1,2,3 order by 4 desc limit 8

The score each store's own report gave it

50 / 100

How the score works: a store where a customer can complete the buy path starts at 100, and loses 10 for each thing that hides a sale and 5 for each slow page. A store where a customer cannot buy scores 0, whatever else is right about it. A visit that could not finish the buy path gets no score at all. This is the average of that score, the same number each store's own report prints.

  • 90–10042% of 556
  • 70–8911% of 556
  • 50–691% of 556
  • under 5046% of 556
n = 556 stores, one walk each
The query this number came from
select s.id, s.domain, coalesce(s.finished_at, s.created_at) as at, coalesce(t.market, case s.region when 'syd' then 'AU' when 'iad' then 'US' when 'lax' then 'US' when 'yyz' then 'CA' when 'lhr' then 'UK' when 'fra' then 'EU' else 'other' end) as market, (select (p->>'lcp')::numeric from jsonb_array_elements(coalesce(s.summary->'speed'->'pages','[]'::jsonb)) p where p->>'viewport' = 'mobile' and p->>'page' = 'home' and (p->>'lcp') is not null limit 1) as lcp
  from scans s left join corpus_targets t on t.domain = s.domain
  where s.status = 'complete' and not (coalesce(s.domain, '') ~* '(^|\.)(navaal\.ai|fly\.dev|localhost)$' or coalesce(s.domain, '') ~* '^(www\.)?(fernwick|.*bilby-testsite)')
     and coalesce(s.ip, '') not in ('repro', 'canary', 'far') and s.repro_of is null
     and coalesce(s.finished_at, s.created_at) < $1::timestamptz
     and coalesce(s.finished_at, s.created_at) > $1::timestamptz - interval '12 months'
     and s.ip = 'corpus'
  order by s.domain, coalesce(s.finished_at, s.created_at) desc

Severity, as the product itself grades it

384 stores

The number is the headline's: stores whose counted walk found a sale it stops or a P1 or P2 finding the report states. The bars show, for each grade, the share of stores with at least one finding of that grade as the walk recorded it; a store with findings of two grades is in both bars. P1 stops a sale, P2 damages one, P3 and P4 are worth fixing and will not cost you tonight.

  • P11% of 556
  • P242% of 556
  • P338% of 556
  • P42% of 556
n = 556 stores, one walk each
The query this number came from
-- $1: the walks the population rule counted, one per store (lib/store-figures.ts, printed on /bilby/methodology)
select f.severity, count(distinct f.scan_id)::int as scans
  from findings f join scans s on s.id = f.scan_id
  where s.id = any($1::text[]) group by 1 order by 1

A second study · September 2026

Product data on 409 live Shopify storefronts

A different method on a different sample, which is why it is a section rather than a row above: a public-web audit of product data on 409 storefronts, with no browser walk and no logins. Published 14 September 2026, updated 24 September 2026. 71.9% had at least one actionable product-data finding — and that is the wrong number to plan around.

How you define “actionable finding”Share of storesOf
All nine checks71.9%294/409
Excluding missing product_type and thin descriptions36.2%148/409
Only attributes AI shopping feeds require, plus policy pages22.7%93/409
Only structured data (missing or duplicated)15.9%65/409
Only crawler access0.5%2/409
  • Median findings per store: 1 (mean 1.28, maximum 6).
  • Stores with zero findings: 28.1%.
  • 35.7% had no finding except a missing product_type and/or a short description.

The sensitivity table is the point: the headline is carried by two content-quality checks, and the figure that matters for AI shopping feeds is 22.7%. The full study has the nine checks, their denominators and the method.

Cite this

Published under CC BY 4.0: use the figures, with attribution. The aggregates are downloadable as CSV so you do not have to retype them.

Navaal. The State of Broken Online Stores. 1 September 2026. https://navaal.ai/bilby/data

Frozen editions

This page recomputes its figures once an hour, which is right for a live report and wrong for a citation. Each edition below is a permanent URL holding exactly what the study said when it was taken — cite one of those and the figure will still read the same next year. An edition taken before 14 November 2026 counted every visit, not one walk per store, and averaged the older stored score.

By market

MARKETSTORES COUNTEDSERIOUS ISSUE (P1/P2)MEDIAN MOBILE HOME LCPMOBILE HOME LCP > 4 SPURCHASE-PATH TRACKING FINDINGSAVG SCORE
US34074%2.7 s · n=32130%18%49
AU11956%3.2 s · n=11634%10%51
UK34—— · n=32———
EU33—— · n=32———
CA30—— · n=28———
STORES COUNTED
stores of the study’s sample counted by the rule below, by the store’s market; a store that refused the browser, or whose walk could not finish, is not counted anywhere on this page.
SERIOUS ISSUE
share of those stores whose counted walk found a sale it stops or a P1 or P2 finding — a step of home → collection → product → add-to-cart → cart → checkout that failed, or a broken feature on it.
MEDIAN MOBILE LCP
the median, across stores, of the home page’s Largest Contentful Paint at 390×844 in a real Chromium from the store’s market region (PerformanceObserver, one load per store); n is the stores with a mobile measurement.
LCP > 4 S
share of the market’s stores whose mobile home LCP exceeded 4 s (Google’s “poor” threshold).
TRACKING FINDINGS
share of stores where an add-to-cart or begin-checkout event failed to reach an installed analytics platform’s own collection endpoint during the walk (Money check).
AVG SCORE
How the score works: a store where a customer can complete the buy path starts at 100, and loses 10 for each thing that hides a sale and 5 for each slow page. A store where a customer cannot buy scores 0, whatever else is right about it. A visit that could not finish the buy path gets no score at all.

A share, a median or an average is printed for a market only once 100 of its stores are counted; below that the cell shows a dash. The rule: Each real store counts once, by its latest walk in the 12 months to the last midnight in Sydney that reached a verdict (a walk that could not finish is skipped for the one before it); Bilby's own test stores, its canary stores and its own sites are left out, and so are the second walks it runs to double-check a fault or to look from another region. The average is over the stores whose counted walk has a score: a store whose homepage did not open for Bilby counts as a serious issue and has no score. The study is the same rule, narrowed to the stores of its Common Crawl sample. Counted to the end of 15 November 2026, Sydney time. The 26 August 2026 checkout-scripts sunset is the reason the tracking column exists: a store whose add-to-cart or begin-checkout event no longer reaches GA4 or Meta shows up here as a Money check finding (purchase itself needs a real order, and Bilby never places one).

Methodology

  1. Sample. Public Shopify storefronts with at least one product page in the latest Common Crawl monthly index (*.myshopify.com/products/*), de-duplicated by host, password-protected and development stores skipped, the store’s active currency read once to place it in a market. Majority US by construction of the web; every cut prints its n.
  2. Walk. The same free scan every merchant can run: a real Chromium browser, desktop 1440×900 and mobile 390×844, from the nearest of six regions, home → collection → product → add-to-cart → cart → checkout, stopping at the payment step. A store whose robots.txt disallows the whole site is not walked and not counted. Rules for single paths are not read, so the walk opens the cart and the first checkout page as a shopper does. Walks added after September are lighter: desktop only, with no search and no newsletter box.
  3. Findings. Deterministic checks first (HTTP, cart endpoint, tracking requests, Web Vitals); a frozen Claude judge only where a human eye is needed, confirmed by a second signal before it becomes P1. Rate limits (429/403) are never findings. Rules: /bilby/methodology.
  4. Cuts. Market = the store’s active currency, read once when the sample was built (USD → US, GBP → UK, CAD → CA, AUD → AU, EUR → EU); the walk itself ran from the nearest of six regions. Speed rows count a mobile homepage LCP above 4 s, measured on Bilby’s machine class and discounted when its own browser was starved.
  5. Limits, printed. One counted walk per store; no purchase events (no orders placed); stores that wall automated browsers are absent; the sample is what Common Crawl saw, not the whole platform. A separate study checked the product data on 409 live Shopify stores.

Press kit

  • Cite as: The State of Broken Online Stores, Navaal (Bilby), 2026, https://navaal.ai/bilby/data — CC BY 4.0.
  • Live figures are recomputed once an hour from the walk records, and every figure prints the day it was computed. Quote the numbers with that date, or cite a frozen edition.
  • Marks: bilby mark · navaal wordmark (BRAND.md rules). Contact: support@navaal.ai — Waqas Ahmad, Navaal, Sydney.
  • The demo store used in every screenshot and the film is Fernwick & Co., one of Bilby’s own test stores; no merchant is named without consent.

Not sure if your store is okay right now? Bilby walks it like a customer and shows you.