Ask how many it actually opened
Every check you run on a shop has two halves: the rule it applies, and the list of products it applies the rule to. Reviews almost always argue about the rule. Nobody asks how long the list was. A report that says all clear is quietly also claiming it opened everything, and that half is usually untested.
Published — written and checked by hand.
Most shops end up with some sort of recurring check. A spreadsheet of listings to tidy. A monthly look at the photography. A pass over descriptions before a sale. Whatever shape it takes, somebody built it once, and what they built was two separate things that get remembered as one: a rule, and a list of products to apply it to.
The rule gets argued over. It gets refined, written down, disagreed with, improved. The list gets typed once and never looked at again.
A clean report is two claims, and only one was tested
When a check comes back with nothing to fix, it is saying two things at the same time. The first is that the products it looked at were fine. The second is that the products it looked at were all of them. Only the first one went through any kind of test.
The second claim is the one that quietly goes wrong, because lists shrink without anybody deciding to shrink them. A product gets added through a different channel and never reaches the sheet. A seasonal range is imported in bulk and skips whatever step normally registers a new line. Somebody splits a listing into variants and the original row is the only one on the list. Nothing about any of this looks like a failure. It looks like a short report.
The fix is to start from the export, not from the list
There is one list in a shop that cannot quietly get shorter, and it is the full product export — every live item, pulled fresh, on the day of the check. Start there and the arithmetic does the work for you: if the export has four hundred and ten lines and the check opened three hundred and sixty, fifty products were never looked at, and you know that before you read a single finding.
That number is the thing worth putting at the top of the report. Not how many problems were found — how many products were opened, out of how many exist. A check that cannot tell you both is not finished.
Every exclusion needs a reason, and the reason needs re-checking
Some products genuinely should be skipped. Discontinued lines, made-to-order pieces, a wholesale-only range, items that are listed but not photographed yet. Excluding them is correct. Leaving the exclusion unwritten is not.
Write each exclusion down next to the product, with the reason in words, and then check it in both directions. The direction everybody runs is the easy one: is every product either checked or excluded? The direction that catches real problems is the other one: is every exclusion still true? An exclusion list is a place where reasons go to expire. A line marked discontinued eighteen months ago that is still taking orders today has been exempt from every check you have run since, and the exemption is the reason nobody noticed.
This is a different failure from a check that finds nothing because there is nothing there — we wrote about an empty list passing every check you wrote — and a different one again from a claim that no check covers. Here the rule is right, the check runs, the products it opened really were fine. The problem is the word "all".
What this looked like on our own site
We have a checker that runs over every page of this site before it ships, and one of its steps asks whether each section of a page can be linked to directly. It passed every day. It was also only ever looking at a hundred and two of our hundred and twenty-one pages, because the pages it checked came from a list somebody typed when the step was written. Nineteen pages had never been in scope, and the line it printed each run — a count, a floor, a tidy summary — had no way to say so.
The repair was not a better rule. The rule was fine. It was making the scope add up: every page now has to be either checked, or named in an exemptions list with a written reason, and a page that is in neither fails the build until somebody decides which it is. The exemptions are asserted the other way too, so one naming a page that no longer exists also fails. It is the same correction as counting a long report by its causes rather than its rows — the output was never wrong, it was answering a narrower question than the one being asked of it.
What to do this afternoon
You do not need a tool for this. You need one column and one subtraction.
- Export your full product list today, from the shop itself rather than from last quarter's spreadsheet. That export is the denominator.
- Put the export beside whatever list your last check actually used, and count the difference. That number is your real finding, before any individual product is examined.
- For every product in the gap, decide one of two things: it gets checked, or it gets excluded with a reason written next to it in plain words.
- Re-read last year's exclusions. Any line marked discontinued, out of stock or coming soon that is currently taking orders has been invisible to every check since the day it was excluded.
- From now on, start the report with opened out of total. A report that leads with how many problems it found is answering the easier question.
None of this makes your check stricter, and that is the point of doing it first. A stricter rule applied to the same short list finds more problems in the products you were already watching. Widening the list finds the products nobody was watching at all, which is where the thing you did not know about is, by definition. The ones you never opened are not clean. They are unexamined, and a report that cannot tell the difference will keep saying so in the same confident voice.
Read enough? Watch your own product.
One photo in — a cinematic product video and a full set of marketing photos out. Your first one is free.