Home Bilby — store monitor Free store scan Free tools AI apps Navaal SEO app Blog Services Studio Contact Sign in

// Guides · The nightly store check

How often should you test your Shopify checkout?

By Published Updated 7 min read

Last updated 28 September 2026.

After every change you make, and at least once every 24 hours regardless — because apps and third-party scripts change your store on their schedule, not yours. A nightly visit of the buy path (home → product → add-to-cart → discount → checkout → tracking) bounds the longest a silent failure can run to one day, and takes about five minutes by hand. Before a launch or a sale, test after every deploy and freeze changes early.

// Your longest silent-failure window

Longest silent window—a failure the night after your last check runs this long unseen
Worst case, one failure—at your average rate · peak hours run 2–3×
Enter your revenue and pick how often you check — the calculator shows how long a silent failure can run, and what that costs against a nightly visit.

method: monthly ÷ 730 hrs = avg hour · window = hours between checks (a failure can start the moment after one) · "only after my own changes" is treated as a month, because app and script updates you didn't make have no check at all. Same maths as the downtime cost calculator.

Ask when stores break and a pattern emerges from every merchant war story: not at random, but right after something changed. The theme tweak at 11pm. The app that updated itself overnight. The pixel "improvement" on Friday afternoon. The breakage clock starts at the change — which is why the frequency of testing, not the thoroughness of any single test, determines how much a failure costs you.

A silent store failure costs whatever your hourly sales rate is, multiplied by how long it takes you to notice — which makes checking frequency, not checking thoroughness, the variable that controls the cost.

The Bilby scan screen mid-visit on the demo store
The scan screen mid-visit on our labelled demo store, Fernwick & Co.: a screenshot per step as it lands, then the seven things it checks read the recording. Run it on your store: free, about a minute, no signup, we never touch your store. Run the free scan →

What should a nightly store check actually cover?

Here is the whole practice. It needs no tools, no purchase, and about five minutes — in a private window, ideally on your phone, because that's what most customers use:

  1. Land like a stranger. Homepage in a private window, on your phone. Does it load fast? Anything visibly broken, overlapping or stale (an expired sale banner counts)?
  2. Open a product page — ideally a bestseller and, weekly, a random one. Images load? Price right? Variants selectable?
  3. Add to cart. The single most breakage-prone step. Did the item actually land in the cart, at the right price?
  4. Apply your discount code if you're running one. A code the cart rejects is a top cart-abandonment trigger — every reason Shopify rejects one.
  5. Begin checkout through to the payment step. You don't need to pay — most silent failures live before payment.
  6. Confirm tracking fired. Weekly at minimum: compare a day of real orders against what your ad platform and GA4 report — the two-minute gap check.

Why isn't "test after changes" enough?

Because you are not the only thing changing your store. Apps push updates on their own release schedules. Third-party scripts — reviews, chat, upsells — change on their vendors' servers without touching your theme. Payment providers deploy. Even a theme can pull remote assets that change under you. The nightly check exists to catch the changes you didn't make — which are exactly the ones you'd never think to test for.

The changes most likely to break a store silently are the ones the merchant didn't make — app auto-updates and third-party script changes — which is why testing only after your own edits leaves the biggest risk untested.

Is it worth paying to automate checkout testing?

Five minutes a night is about thirty hours a year of a founder's attention — the kind of chore that gets skipped exactly when it matters (holidays, launch weeks, a sale). The calculator beside the answer does the comparison for your store: your longest silent window at the frequency you check today, against 24 hours for a nightly visit, priced at your own average hourly rate. If the difference is more than a month of a monitor, the monitor is the cheaper option. If your revenue is small enough that the difference is nothing, do it by hand — the six steps above are genuinely enough.

Bilby is the nightly store check, automated. It visits all six steps in a real browser every night, on desktop and mobile — judged like a first-time customer — and emails you the annotated screenshot the moment anything breaks. Quiet nights send nothing. Run one free visit now and read the report it produces; the nightly version is US$24/mo billed yearly · 7 free nights.

Frequently asked questions

How often should I test my checkout?
After every change, and at least once every 24 hours regardless — the nightly check bounds your maximum silent-failure window to one day. Before high-stakes periods (a launch, a sale), check after every deploy and freeze changes early.
What is a nightly store check?
A six-step visit of the buy path every 24 hours: land → product → add-to-cart → discount → checkout-start → tracking check. Five minutes by hand in a private window, or automated with a monitor.
Can't I just test after I make changes?
Necessary but not sufficient — apps auto-update and third-party scripts change on their vendors' schedules. The nightly check catches the changes you didn't make.
Do I need to place a real order?
Not nightly — reaching the payment step covers most silent breakage. Place a real test order after significant payment, shipping or tax changes, and periodically to verify purchase tracking end to end.

Keep reading: Best store monitoring tools, compared honestly · Did a theme update break my checkout? · Add-to-cart not working? · Downtime cost calculator

// The nightly store check, automated

Bilby visits it
every night.

All six steps, in a real browser, judged like a customer — with the screenshot in your inbox when something breaks. Try the free visit first.

Thin or missing product content is the other half of this problem, and it is the half search engines and AI assistants see. Navaal: AI SEO, AEO & GEO audits a Shopify catalogue for it free, and writes what is missing for review.