AI that has to work tomorrow night as well.
A demo is easy. What is hard is a system that runs every night, costs a predictable amount, and cannot invent the one number somebody will act on. We run two of those ourselves — one of them visits online stores from six regions every night and judges what it sees — and this page shows how they are built, because that is the only proof worth offering.
The three things that decide whether an AI build is any good, stated before the sales part. One: the model is never asked for something that can be measured — measured things are measured in code, so they cannot be invented. Two: every metered call has a ceiling enforced before the call and a cost recorded from the usage the API reports, so the bill is a number you set rather than one you discover. Three: when it cannot do the job it says so, in words, instead of returning something plausible. Everything else is detail.
A system of ours, drawn as it is built.
This is Bilby: an AI customer that visits an online store every night from six regions, in a real browser, on desktop and a phone, and reports what broke with screenshots. Not a diagram of a pipeline — the actual architecture, with the two boxes that matter marked.
What each one does every day
Bilby queues every watched store for 02:00 in that store’s own time zone, walks the buying path to the payment step without ordering anything, judges each page against its screenshot, scores the walk, and sends one alert when something that matters broke. The judging cost of every walk is recorded from the token usage the API reports and shown on an internal cost board beside the machines — which is how we know October’s judging was 91% our own test runs and not customers’.
Navaal: AI SEO, AEO & GEO is a published Shopify app: it reads a catalogue, writes answer-first product content, and holds it for review before anything publishes. It passed App Store review with OAuth, mandatory webhooks, GDPR data requests and Shopify billing — which, if you are building on a platform with a review process, is most of what you are actually buying.
Both are ours, which is the point. A studio selling AI systems should be able to show you one it has to keep running.
What it is made of, and why.
| Layer | What we use, and why |
|---|---|
| Models | Anthropic Claude, called directly over HTTP. Haiku for the per-page judging because it is the cheapest model that can do the job; a bigger model where a bigger model is needed and nowhere else. |
| Runtime | Node and Next.js on Fly.io, in the region closest to the work. Bilby's browser workers run in six regions because a store fails differently from Singapore than from Sydney. |
| Browser | Playwright driving real Chromium. Not a headless fetch: a checkout that breaks under a popup only breaks for something that can see the popup. |
| Data | Postgres. Two databases, one for the platform and one for Bilby, with a connection pool that has an error handler — we learned that the hard way and the postmortem is in the repository. |
| Edge | Cloudflare in front, with the static site on Pages and the application on Fly. Pages Functions where a page needs live data at the edge. |
| Instrumentation | First-party events in our own database. No third-party analytics, no cookie banner, and the owner-facing numbers count humans only, with automation classified and excluded. |
| Cost control | Every metered call priced from the usage the API reports, written to a table, summed on a board, and capped in code with a refusal that says which limit was reached. |
What is in a build.
A ceiling, in code
Enforced before the call, priced from the usage the API reports, with a refusal that names the limit. Not a dashboard somebody has to remember to read.
Measured things measured
Anything a computer can establish is established in code and never asked of the model. It is the single cheapest way to stop an AI system inventing a fact.
Evaluation you can run
A set of cases with expected answers, runnable on demand, so a prompt change is a thing you can test rather than a thing you hope about.
An honest failure path
When it cannot answer it says which limit or which error, in words a customer can act on. Our own tools refuse out loud; yours will too.
Your accounts and keys
From the first commit. Model usage billed to you at cost, with the ceiling you set, and no markup hidden in a retainer.
Somewhere to look
Cost per call, failures, refusals and latency on a board you can read, because “the AI is being weird” is not a diagnosis.
What it costs, from.
| What | From | Covers |
|---|---|---|
| AI chatbot | A$3,500 / US$2,300 | running costs at cost |
| AI agent or automation | A$7,500 / US$4,900 | one workflow, end to end |
| Web app MVP | A$19,500 / US$12,900 | one product, one audience, shipped |
| Mobile app MVP | A$29,000 / US$19,000 | one platform to start |
AUD ex GST for Australian clients, USD everywhere else. Model usage is passed through at cost with a ceiling you set. Hourly work is A$140 / US$95. The cost guide prices your own build with every hour shown, and its form carries the estimate into the message.
Straight answers.
What does “AI development” actually mean here?
Software that uses a model as one component, built the way the rest of the software is built: with tests, cost ceilings, error handling and somewhere to look when it goes wrong. A$3,500 for a chatbot, A$7,500 for an agent or an automation, A$19,500 for a web app that has AI inside it. Not a prompt in a text box with an invoice attached.
How do you stop the running costs surprising me?
A ceiling in code, before the call, priced from the usage the API reports rather than estimated. When the ceiling is reached the thing refuses and says so, instead of silently returning something worse. Our own free generator runs on a US$2 monthly budget with five runs per visitor a day; the page says what happens when either is hit, and the sixth run gets a 429. You get that arrangement with your own numbers, and the usage is passed through at cost.
Will it make things up?
A model will, so the architecture has to make it not matter. The pattern we use is the one in the diagram above: anything that can be measured is measured in code and never asked of the model. Bilby's judge is not allowed to comment on page load speed at all, because load time is measured twice in code — so a judged finding about a slow page cannot be invented. Where a model must produce facts, we make it list what it was not given rather than filling the gap.
Can you work with my existing data?
Usually, and the first question is what it can tell us rather than what it contains. An API or a database: yes. A screen somebody copies numbers off: still yes, but it is a different job and we will say which before quoting.
Which model do you use?
Whichever is cheapest for the job, named in the quote, and swappable afterwards. Bilby judges on Claude Haiku because it is cheap enough to run on every page of every walk; the free description generator uses the same model for the same reason. A system that can only run on one specific model is a system with a single point of failure and a pricing problem.
Do you host it or do I?
You, in your accounts, from the first commit. We will run it for you with a named response time if you would rather, but the accounts and the keys are yours either way — which is the only arrangement in which you can leave.
How long until something works?
A chatbot or a single automation: two to four weeks. An agent doing multi-step work with approvals: four to eight. A web app with AI inside it: the MVP timeline, which is longer and should be. You will have something deployed and usable well before the end in all three cases, because that is how the scope gets cut honestly.
Where to go from here
- AI chatbots — answering from content you already have, with a real “it does not know” branch.
- AI agents — doing multi-step work, with every action sorted into reversible and irreversible.
- The GenFul build — a custom store we built and then measured again for this site, including the figures that are not flattering.
- The EBS build — 10,304 products, and the data layer that makes a catalogue that size usable.
- The cost guide — price your own build, with every hour shown.
Two kinds of AI build, and they are not the same job.
A chatbot answers from your own content. An agent takes actions on your behalf, which needs approvals, limits and an audit trail. Each has its own page, its own architecture and its own failure modes.