Does my store actually work tonight?
Bilby's Watch check opens your store in a real Chromium browser every night — on desktop and mobile — and visits the path a shopper takes: home, a collection, a product, add-to-cart, the cart, and checkout up to the payment step. It also tries your site search, checks the newsletter form validates, applies a discount code if you gave one, fetches the top ten pages from your sitemap, and reads your SSL and domain expiry. A step that fails becomes a finding with a screenshot; a step Bilby can't map is never a finding.
What the Watch check catches
- Add-to-cart that fails — the store's own cart endpoint refused the item (HTTP 4xx) or showed an error
- A disabled add-to-cart with no sold-out message and no variant chooser
- Checkout that shows an error instead of the information step
- Collection or product pages that return HTTP 4xx/5xx from your own navigation
- Site search that returns nothing for a product you sell
- A newsletter form that accepts an invalid address silently
- A discount code you gave Bilby that the cart rejects
- Top-10 sitemap pages returning errors, and unexpected redirects
- SSL certificate or domain registration expiring within 14 days
- Mixed content (http:// resources on an https page) and images that fail to render
- Expired promotions still showing (a promo phrase with an end date in the past)
- Pages that look broken, stale or confusing to a first-time customer — judged by Claude from the screenshot, with a literal evidence quote or the finding is discarded
Severity — the rule, not a feeling
- P1Anything on the buy path: add-to-cart refused, checkout error, certificate/domain expiring within 3 days.
- P2Search, forms, codes, a navigation page returning an error, certificate/domain expiring within 14 days.
- P3Everything else — mixed content, a broken image, mobile overflow, a stale promotion (P4).
Honest limit. Bilby never places an order and never signs in. It types into three boxes only (the store's search, the newsletter box with a deliberately invalid address, and your discount code if you gave Bilby one) and stops at the payment step. A page that answers 429/403 to Bilby is rate-limiting Bilby, not your customers — it is skipped and noted, never a finding.
Questions people ask
Does Bilby place test orders?
No. Bilby visits to the checkout's information step and stops. It never enters payment details, never logs in and never edits or installs anything. It types into three boxes only: the store's search, the newsletter box with a deliberately invalid address, and your discount code if you gave Bilby one. Purchase tracking therefore can't be verified by Bilby — the Money check prints the two-minute owner check instead.
What happens when a step can't be mapped on my theme?
It is recorded as "couldn't map this step" — not a defect, not a score penalty. Bilby only reports what it saw fail with evidence; a visit it couldn't complete says so plainly.
How do you avoid false alarms?
Deterministic checks first (HTTP status, the cart endpoint's answer, the certificate's dates). Claude judges screenshots only for what a human eye needs, under thirteen rules with a mandatory evidence quote, and a code backstop suppresses known bot-vs-customer artefacts (cookie banners, geo modals, empty ad slots). Before the free scan opened, I hand-reviewed every scan in a ten-store run against its recorded evidence. False alarms can still happen; if one is wrong, report it to support@navaal.ai and I will mark it.
Read next: Why is my add-to-cart button not working? · How often should you test your checkout? · 7 silent problems that kill sales overnight · Is Shopify down, or is it just my store? · Shopify discount code not working
See it on your store
The free scan runs the Watch check tonight — the whole report, nothing locked.
Not sure if your store is okay right now? Bilby walks it like a customer and shows you.