An empty list passes every check you wrote

We keep a checker that reads our own marketing pages after every build and refuses to let one ship if something on it has stopped being true. It runs forty-four separate steps. This morning we found that one of them would pass a build in which the thing it checks had been deleted, and would say so in a neatly formatted summary line.

The step is about a video-processing feature that only works if the right software is installed in the right place. It reads two lists. The first it builds itself, by searching our own source code for every piece of the system that uses that software - and it explicitly refuses to continue if that search comes back nearly empty, because a search that has stopped matching would make everything after it true by default. Somebody wrote that guard deliberately and explained why in a comment.

The second list is typed by hand, and every test in the step runs once for each thing on it. There is no guard on that one. The list happens to hold a single item today, so deleting one line turns the whole step into a loop over nothing. It does not fail. It reports zero problems, because zero problems is exactly what a loop over nothing finds.

Emptiness and compliance produce the same report

This is the part worth carrying away, and it has nothing to do with our software. Almost every check that exists in a shop is a per-item check. It opens a row, examines it, and tells you what is wrong with it. That design is correct and it has one structural blind spot: it can only ever speak about rows that exist. Hand it an empty set and it returns an empty complaint, which is formatted identically to approval.

So the state you most need to be warned about is the state your warnings are least able to describe. A check cannot tell you that it had nothing to do. It can only tell you that it found nothing wrong, and those two sentences arrive looking exactly alike.

Where this shows up when you are not looking for it

• A product feed that reports no errors. The error count is a count of rejected items. If the export that feeds it broke and sent nothing, the count is zero - the best number the report can display - and it means the catalogue has gone dark.
• A nightly alert that stops arriving. You built it to tell you when something goes wrong, so silence reads as good news. Silence is also what you get when the alert itself stops running, and nobody has ever been paged by a job that is no longer scheduled.
• A size chart or spec table that renders empty. The template asks for a list of rows and prints them. Given no rows it prints the heading, the borders, and nothing else, and the page still returns successfully.
• A review score with one review behind it. Five stars out of five, truthfully calculated, over a sample of one. The arithmetic is sound and the number is meaningless, which is a different defect from the number being wrong.
• A collection page that has sold out. The rule says every collection shows its products. An empty collection satisfies it, because it shows all none of them, and the page is now a dead end that looks like a category.
• A shipping or tax rule that no longer matches anything. Rules are tested by running them against an address and seeing what they return. A rule that matches no addresses raises no objection and quietly charges nothing.

Each of these is a case where the absence of data is being reported through a channel that was designed to describe data. The channel is not broken. It is answering a different question from the one you are reading it as having answered.

The fix is a floor, and it is unglamorous

There is one move that handles the whole class, and it is less interesting than the problem. For every list your business depends on, write down how many things are in it today, and make something complain if that number ever drops below it. Not below zero - below today's number, or a little under it if the list legitimately fluctuates.

That is all. You are not checking the items; you already do that. You are checking that there are items, which is the one property a per-item check is structurally incapable of noticing. The guard we added this morning is three lines long and says, in effect, this list had one thing in it when I last looked, so stop if you find none.

Write down the number and the date you measured it, in the same place as the rule. A floor with no date behind it gets treated as arbitrary by the next person to meet it, and the first thing they will do when it blocks them is lower it. A floor with a date and a measurement attached is an argument, and arguments survive contact with people in a hurry.

What not to do about it

Do not go looking for the empty lists by hand. It feels productive and it finds the ones that are empty today, which is the small half of the problem - the rest will empty out next quarter, after you have stopped looking. The check has to be the thing that remembers, because the whole reason this failure mode works is that it happens while your attention is on something else.

And do not treat a zero as self-explanatory when you do meet one. Zero rejected items, zero alerts, zero out-of-stock warnings: each of those has two readings, a good one and an alarming one, and they are not distinguishable from the number alone. The only way to tell them apart is to know what the denominator was, which means something has to have recorded the denominator.

The underlying habit is the same one that makes a count in your copy go stale: a number is only as trustworthy as the thing that holds it to the set it describes. This is that idea pointed at the set instead of the number. It is also the near neighbour of yesterday's note on the claim nothing checks - there the check was reading the quieter half of a sentence, here it is reading a list that has nothing left in it - and of the way a list of exclusions goes stale once nobody re-reads it.