← All posts

You checked a different shop

A shop is assembled twice: once from the template, and once from the settings that fill it in. A preview, a staging copy or a test order can use one set of settings while customers get another — and when they differ, the check passes honestly and the customer still sees the wrong thing. Nothing is broken; the two shops were never the same shop.

Published — written and checked by hand.

Most of the mistakes that reach a customer are not mistakes anybody made. They are the difference between two things that were each correct: the version you looked at, and the version you shipped. You opened a page, it was right, you published, and a customer opened a different page. No step in that sequence was careless, which is exactly why it is hard to catch.

The second shop is made of settings

Think about what has to be true before a product page can render. There is the template — the layout, the copy, the markup, the thing that lives in your theme or your code. And then there is everything the template reads at the moment it renders: which currency, which market, which sales channel, which inventory location, whether a feature is switched on for this customer, what the free-shipping threshold is today, which payment methods are enabled, what your business address is set to.

That second list is not in the template. It lives in an admin screen, an environment variable, a feature switch, a per-market override — somewhere a person can change without touching the thing you reviewed. So the template is a half-finished page, and the settings are the other half, and the page a customer sees is the two of them together.

Which means a check is only a check of the pair it ran against. Preview the template with no settings and you have previewed a shop that does not exist: not broken, not wrong exactly, just not yours. The output looks completely plausible, because a sensible template handles a missing setting gracefully — it hides the banner, drops the field, falls back to the generic label. Graceful degradation is good behaviour and it is also excellent camouflage.

Where this opens up in a real shop

The pattern repeats anywhere the thing you inspect is built separately from the thing you publish:

  • A theme preview or a duplicated draft theme. It has its own copy of the settings it was duplicated with, and they stop tracking the live ones the moment somebody changes one.
  • A second market or currency. Prices, taxes, shipping options and sometimes the whole availability of a product can resolve differently, so the page you check from your own country is one of several pages that exist.
  • A sales channel. A product can be published to one channel and not another, which makes 'is this product live?' a question with more than one answer.
  • An email or notification template. It renders against one test customer's data; most of your list has different data, and the field that is empty for them is invisible to you.
  • A feature you switched on by hand. A thing enabled in an admin screen rather than in the template is one somebody can switch back, and nothing in the template knows it happened.
  • A discount, a bundle or a gift-wrap option with a date range. Inside the window it is part of the page; outside it the page is a different page, and whichever side of the window you happen to be standing on is the version you will test.

None of those is exotic. Each one is just a place where the settings are stored apart from the template, which is most places, because storing them apart is what makes them changeable without a developer.

This is a close relative of the problem where a safeguard runs once per item and therefore cannot run at all when the list is empty — emptiness and compliance produce the same clean report. The difference is worth holding onto: that one is a check that never fires, and this one is a check that fires enthusiastically against the wrong input, which is harder to notice because it produces a result.

Our own version of it, this week

We run an automated check over every public page of this site before anything ships. It builds the site from source, reads the resulting pages and asserts a long list of things about them — one heading per page, titles inside the length a search result shows, every structured-data reference resolvable, no page below a word floor. It has been green every day for months.

The build it reads is created fresh, with only the settings needed to make a build possible. Three published settings were not among them: the web address of our own app-store listing, and the two ownership tokens that let a search engine confirm this site is ours. All three are supplied by the host that serves nitaris.com, so the live site has them and the checked site did not.

The consequences were all invisible and none of them was subtle once measured. On the live site, every page carries a line of structured data that names the company behind it and points at a second, independent page about the same product — the one piece of markup that distinguishes us from the unrelated business whose name is spelled like ours. In the checked build, that line was absent from all of them. On the live site, the main call to action reads one way; in the checked build, most pages carried the wording we used before the listing existed. Our checks had been asserting things about a version of this site that we stopped serving in the summer.

The fix is one line per setting: supply them to the build the check reads, so the pair being checked is the pair being shipped. The lesson is the uncomfortable half — the checks were not weak. They were careful, numerous and honest about a shop we do not run.

The habit that actually prevents this is not more vigilance, because vigilance has nothing to look at: a page rendered with the wrong settings looks like a page. What works is making the settings part of what you read.

  • When you check a page, note which market, currency, channel and theme you were in. If you cannot say, you do not yet know what you checked.
  • Check the state a customer is most likely to be in, not the state you happen to be in. For most shops that means a different country, a phone, and not being logged in.
  • Any setting that changes what a page SAYS deserves to be visible beside the page — a line in your release notes, a label in your staging banner, anything that puts the value in front of the person approving the thing.
  • When something is switched on by hand, write down where. A capability enabled in an admin screen and nowhere else is one that nobody can rediscover from the code.

There is a milder version of this that is worth fixing first, because it costs nothing: the same words living in several places, where you correct one copy and leave the others. That is a problem of duplication. This one is a problem of provenance — not where the words are, but which settings were in force when you looked at them.

The question to keep is short, and it is not 'did it pass?'. It is: what was in force when it passed? A check is a sentence about a pair — this template, those settings — and a pair you cannot name is a pass you cannot bank.

Lights up

Read enough? Watch your own product.

One photo in — a cinematic product video and a full set of marketing photos out. Your first one is free.

Free preview · no cardSee it filmed — free