← All posts

A long list is not a lot of work

Something hands you a report: two hundred problems across your catalogue. The number feels like a measure of how much is wrong, and it is not one. A checking tool emits a row for every item that fails every rule, so its length is causes multiplied by places. Sort the report by its message before you touch anything and most of them collapse into a handful of jobs.

Published — written and checked by hand.

A tool that checks a catalogue works the same way whatever it is checking. It takes a list of rules, visits each of your items, and writes a row every time an item fails a rule. That is the only way it could work. It also means the number at the top of the report is not a count of problems — it is a count of rule-and-item pairs, which is a different quantity and usually a much larger one.

The row count is a multiplication

Two numbers decide how long a report is: how many things are actually wrong, and how many of your items each wrong thing touches. The second number does most of the work. A field missing from one product is one row. The same field missing from the shared template every product is built from is one row per product, and the report cannot tell you that, because it did not look at your template. It looked at the output.

So a report with two hundred rows might be two hundred separate jobs, or it might be two. Nothing about its length distinguishes those two cases, and length is the first thing you see.

Group by the sentence, not by the product

The habit that sorts this out takes about a minute. Before fixing anything, group the report by the message rather than by the item — sort or filter on whatever column holds the words, not the SKU. You now have a much shorter list: the distinct complaints. That number is far closer to the number of jobs than the row count was.

Then read the shape of each group, because the shape says where the cause lives:

  • The same message on nearly every item. Something shared is responsible — a template, a theme setting, a feed mapping, a default that was never filled in. One edit, in a place the report never mentions, retires the whole group.
  • The same message on one whole category or collection. Whatever is specific to that group: its own template, or the import that created it.
  • The same message on exactly the items you added most recently. Not the items — the process that added them. Fixing the items leaves the process in place to do it again next week.
  • A message that appears once or twice, on unrelated items. This is the genuinely per-item pile, and it is real individual work. It is also usually much smaller than it looked.

The uncomfortable implication is that the longest groups are the good news. A report of two hundred rows with three distinct messages is a better morning than a report of twelve rows with twelve, and you cannot tell which one you have by looking at the total.

Fix the cause, then re-run before anything else

Grouping tells you where to look; re-running is what proves you were right. After one edit aimed at one group, run the check again. If the group is gone, it was one cause and you are done with all of it. If half of it is gone, the group had two causes wearing the same sentence — which happens, and is the reason this is a loop rather than a sort.

Working top to bottom instead has a specific cost, and it is not just time. When a hundred rows share a cause you have not found, fixing them one at a time means making a hundred local changes that each compensate for something upstream. Every one of those is a thing you now have to maintain, and the original cause is still there, ready to produce the same report next month.

The other direction, which is also wrong

The opposite mistake is to assume identical wording must mean a shared cause and make one edit confident it has closed the group. Two items can fail the same rule for different reasons. The test is not how the message reads, it is whether one change retires all of them, and the only way to know that is to run the check again rather than to reason about it.

It is worth saying what this habit is not. It is not a reason to distrust the report. The rows are true; a tool reporting what it found is doing its job. What it cannot do is tell you how its findings are related to each other, because the relationship lives in your templates and your settings and it only ever saw the pages.

This is the companion to a different problem with the same reports: a blank field does not say why it is blank. That one is about sorting the rows by kind — whether each is a fact you can go and retrieve or one you would have to invent. This one is about sorting them by how many are the same fact seen from different places. Both sorts are worth doing, and in that order: how many jobs are there, then which of them are real.

Our own case, since it is why this is written down

This pass added a check that compares one published setting against what our live site actually serves. To make sure it worked, we changed the value it compares against and ran it. It reported 113 problems: one for every page, each row perfectly true, and the single changed value named in none of them.

That is a report whose length is entirely the multiplication described above — one cause, 113 places — and it was written by somebody who had, one layer down in the same codebase, already fixed exactly this. An earlier version of our build checks turned one unset setting into 231 findings blaming 231 individual pages, and the fix for that was to notice the uniform case and report it once. The new check reproduced the mistake it was built next to.

It now looks first at whether every page is telling the same story. If they are, that is one finding, and it names the setting rather than the pages. The per-page list only appears when the pages genuinely disagree with each other, which is the only situation where a per-page list is the answer.

The same check also reported two problems on its first run that were not problems. It compared the machine-readable index of our pages against a file listing them for AI crawlers, found two entries in the second that were not in the first, and flagged them — correctly, by its own rule. They were the sitemap and the RSS feed: two files, deliberately listed, neither of them a page. The rule was wrong, the site was fine, and no amount of reading the two findings would have told us that, because both were accurate sentences about the wrong category of thing.

Which is the whole argument, from the other side. A report is a set of observations about places. The number of observations is not the number of problems, the wording of an observation is not its cause, and both of those are things you can only work out by grouping the list and then re-running it. Do that first, and the two hundred rows usually turn out to be a short afternoon.

Lights up

Read enough? Watch your own product.

One photo in — a cinematic product video and a full set of marketing photos out. Your first one is free.

Free preview · no cardSee it filmed — free