How does Bilby check a store?

Bilby opens the store in a real Chromium browser — desktop 1440×900, then mobile 390×844 — from the store's home region, visits home → collection → product → add-to-cart → cart → checkout (stopping at the payment step), records everything the browser sees, runs deterministic checks on the recording, and asks Claude only what needs a human eye. Every finding carries its evidence; every number prints its basis; anything that didn't run is listed.

The visit

One visit, one recording, five checks

The journey is the one a shopper takes. Desktop first (judged), then mobile (deterministic), one after the other so the two never compete for the CPU whose speed is being measured. A step that fails becomes a fact or a finding with a screenshot; a step Bilby can't map on a theme is recorded as “couldn't map” and never scored. A free scan takes about two minutes and is stopped at two and a half minutes whatever it has done, and the report says what didn't complete; nightly visits get six minutes per viewport.

Bilby never logs in, never edits a theme, never installs an app and never places an order. It sees what a shopper sees.

The recording

What is captured, and how privacy is handled

Per viewport: every network request via the Chrome DevTools Protocol (URL, type, status, transfer size, timing, initiator, and the POST body for tracking endpoints), console errors, Web Vitals via PerformanceObserver (LCP, CLS, a TBT proxy from long tasks, TTFB, FCP), the main document's headers and certificate, a DOM snapshot (scripts, theme, prices, stock labels, title/meta/schema), a filmstrip at one frame per second (≤ 40 frames at 480 px), and a screenshot per step.

Personal information is blurred before storage: text that looks like an email address or phone number is blurred in Bilby's own render of the page before any screenshot or frame is captured. Recordings are kept 90 days on paid plans and 30 days for free scans.

Deterministic first

What is measured, never guessed

Watch (certificate and domain expiry, mixed content, sitemap top-10 status, the cart endpoint's answer to add-to-cart, search results, form validation), Money (which tracking platforms fired which events on which step, from their own collection endpoints), Speed (vitals per page, weight, the heaviest requests by vendor), Change (the night-to-night diff) and the Answer line (robots.txt, llms.txt, schema) are computed from the recording with no model involved. Speed numbers measured from a region far from the store's market, or while Bilby's own browser was starved (TBT above 8 s), are printed as facts with the caveat — never as findings.

Judgement

Where Claude is used, and the rules it works under

Claude (the judge model the worker recorded on its latest judged walks: claude-sonnet-5) sees the screenshot of the home, collection and product pages and is asked one question each: does this look broken, stale or confusing to a first-time customer? It works under thirteen rules — never analytics, never speed, never cookie banners, never geo modals, never empty ad slots, never console warnings — and must quote literal evidence from the screenshot or the finding is discarded. A code backstop suppresses the known bot-versus-customer artefacts regardless of wording. An AI finding can only become a P1 when a deterministic signal agrees.

Before the free scan opened, I hand-reviewed every scan in a ten-store run against its recorded evidence. The first clean run was on 26 August 2026; later runs turned up mistakes of mine, which I fixed. False alarms can still happen: if a finding on your store is wrong, email support@navaal.ai and I will mark it.

Severity and score

How a score is made

P1 = a sale is stopped (the buy path is broken); P2 = a broken feature or a missing purchase-path event; P3 = degraded; P4 = worth fixing. Findings are de-duplicated across pages by topic. How the score works: a store where a customer can complete the buy path starts at 100, and loses 10 for each thing that hides a sale and 5 for each slow page. A store where a customer cannot buy scores 0, whatever else is right about it. A visit that could not finish the buy path gets no score at all. That is the score each store's own report prints, and the average below is the average of it.

Revenue at risk

Dollars, with their basis

Bilby cannot see a store's traffic or sales. It guesses a monthly revenue bracket from store signals (platform, products listed, third-party apps present), labels it “my guess”, and lets the owner correct it with one slider. Each finding's share of monthly revenue is a range anchored to published conversion research (for example ~7% of conversions per extra second — Akamai, The State of Online Retail Performance, 2017 of load beyond 2.5 s; 30–80% of a path's sales when add-to-cart is broken) and the step's position in the buying path. The method is printed under every figure.

The public figures

Which walks count, and what they say

Each real store counts once, by its latest walk in the 12 months to the last midnight in Sydney that reached a verdict (a walk that could not finish is skipped for the one before it); Bilby's own test stores, its canary stores and its own sites are left out, and so are the second walks it runs to double-check a fault or to look from another region. The average is over the stores whose counted walk has a score: a store whose homepage did not open for Bilby counts as a serious issue and has no score. No share, average or median is printed until at least 100 stores are counted: at 100, one store moves a share by one point, and below that a percentage claims more than the count can carry. Every page that prints a figure about the stores Bilby has walked — this one and the data report — reads the same function under this rule, so the two cannot disagree, and each prints the day its walks are counted to.

Counted to the end of 15 November 2026, Sydney time: 874 real stores, walks from 27 August to 15 November 2026. 58% had at least one serious issue (P1 or P2); average score 57 / 100; median mobile homepage LCP 2.6 s (75th percentile 4.1 s, n = 615 of those stores with a mobile measurement, PerformanceObserver at 390×844). The market-by-market cut and the study of public Shopify storefronts, under the same rule, are at /bilby/data.

Method updated 14 November 2026. Each store now counts once, by its latest walk that reached a verdict, and the average is the score each store's own report prints, where it used to count every visit and average the older stored score.

Benchmarks inside a store's own report come only from Bilby's own walks — never scraped indexes. A percentile is shown only when n ≥ 10 and labelled “early dataset — directional” below 100 visits.

Change and history

Night to night

Each visit writes a normalised snapshot — scripts, theme, prices and stock on the visited product, title/meta/schema, a 64-bit perceptual hash of the homepage, robots.txt and llms.txt state, top-page redirects — and the next visit diffs it. Issues have a state machine (open → escalated after 72 h → resolved) so an alert is sent once, on the transition, never every night.

Limits, printed

What Bilby cannot know

Purchase events (no order is placed). Anything behind a login. Whether a change was intentional. A store that walls automated browsers (a 403 or a challenge page) — reported as an allowlist ask, never as a store defect. Rate limits: a 429 to Bilby is skipped and noted. The same rules run against Bilby's own test stores every night.

Check pages: watch · money · speed · change · rivals · search · answer · BilbyBot

Not sure if your store is okay right now? Bilby walks it like a customer and shows you.