The numbers are right and the sum is wrong

We read the same search report every morning. It prints one number for how many people clicked through to us, and underneath it a set of tables breaking that number down - by the thing people searched for, by the page they landed on, and so on. For months we have quoted those tables in our own notes as though each one carved the total into slices.

This morning we added two of them up. The table of search terms accounted for a bit over half the clicks. The table of landing pages accounted for all of them, exactly. Same report, same week, same filter, two tables that disagree about how many clicks there were.

Neither is wrong. They are answering different questions, and only one of them is a question with a single answer per click. Every click lands on exactly one page, so the page table can add up and does. The search-term table cannot show every term, and the rows it leaves out do not stop existing - they just stop being visible. We can see the size of the gap. We cannot see what is in it.

The same report, failing in the other direction

Then we looked at the impressions in the same report - how often we appeared at all, rather than how often we were clicked. Here the first ten rows of the page table already added up to more than the total for the entire site, with forty-five rows still to come.

That is not a contradiction either, once you ask what one impression is. If a single search shows two of your pages, that is one appearance and two page-rows. The total counts occasions. The column counts pages. There is no arithmetic that turns one into the other, and no honest way to say a given page was “such-and-such a share” of the total.

So within one report, over one week: a column that undercounts because rows are withheld, and a column that overcounts because one event touches several rows. You cannot look at a discrepancy and infer which kind you are holding.

Where this lands in a shop

This is not a search problem. It is a property of any report that offers a total and a breakdown, and a shop runs on several of those at once.

• Channel attribution. Your ad accounts, your email tool and your storefront each report the orders they touched. One order touched by three of them is counted by all three. Add the channels together and you will out-earn your own bank account.
• Discount codes. An order with a stacked code can appear under more than one code, so the codes sum high. An order placed with no code appears under none of them, so the same report is also missing rows at the bottom.
• Product views by variant. A customer who tried three colours before buying is three variant-rows and one sale. The variant table is a record of interest, not of purchases, however much it looks like the latter.
• Search terms inside your own store. Rare searches get folded away or dropped. The visible list is the popular tail of what people typed, not the whole of it - which matters most when you are hunting for the thing nobody can find.
• Returns by reason. One return with two reasons ticked appears twice; one with the box left blank appears nowhere. Both failures sit in the same table.

The common thread is that none of these tables is lying. Each is a correct answer to a question slightly narrower than the one you asked.

The check is one subtraction

Add the column. Compare it with the total. Then do the part almost nobody does, which is to write down which direction it went and why, in one sentence, next to the report.

If the column came out low, rows are being withheld or discarded, and anything you conclude from that table is a floor rather than a count. If it came out high, one event is being credited to several rows, and no row can be quoted as a percentage of the total. If it matched exactly, you have found the rare case where each event belongs to exactly one row - and that is the only table in the set you are allowed to treat as slices of a pie.

Do it once per report, not once per question. The direction is a property of how the table is built, so it holds until the report changes.

What not to do about it

The obvious response to two numbers that disagree is to go looking for the broken one. That is the wrong instinct here and it is expensive, because the hunt succeeds: you will always find some tag or pixel that could be misconfigured, you will fix it, and the numbers will still disagree, because they were never supposed to agree.

The second obvious response is to pick the report you trust and ignore the others. That quietly throws away the only useful thing about having two - they disagree in a direction, and the direction is information. A channel table that sums high is telling you about overlap, which is a real thing happening to real customers who see you in three places before buying.

The third is to scale the numbers until they fit, dividing each channel down until the columns match the bank. It feels rigorous and it is not: you have invented a split between channels that the data never contained, and you will then make budget decisions out of the invention.

What is left is less satisfying and more useful. Treat the total as the number of things that happened, treat the breakdown as a description of how they happened, and stop converting between them. Most reporting mistakes we have made - including this one, for months, in our own notes - come from quietly treating a description as a division.

This is the same family as something we wrote about two days ago. A count in your copy goes stale because nothing holds it to the set it counts; a number in a report goes wrong because nothing holds the column to the total. In both cases the sentence looks like the most checkable thing on the screen, and is one of the least. And when the screen itself changes under you, the numbers can stay put while everything around them moves.