A play count is not a result

The first report most people read after adding a product video is how many times it was played. It is the easiest number to find and it is almost always encouraging, which is what makes it dangerous. A play means somebody was curious enough to press it. It says nothing about whether the film changed what they did next.

That is not a complaint about play counts. It is a reminder of what they are for: they tell you a film was noticed. The question you paid to answer is a different one, and it lives in a different report.

Decide what the film was for before you look at anything

Every film is added to answer something the listing could not answer without it — how the product moves, how it catches light, how big it really is. So the thing to measure is the decision that unanswered question was blocking.

• On a product page, it is whether visitors to that product go on to buy it.
• For a product that comes back with reasons like "smaller than expected" or "not the colour I thought", it is the return rate, and the reasons.
• In an ad, it is what each purchase cost you — not how many people watched.

Pick one of those, write it down next to the product, and do it before the film goes live. If you choose the measure after you have seen the data, you will choose the one that moved.

Compare the product with itself

The tempting comparison is between a product with a film and one without. It tells you almost nothing, because two products differ in a hundred ways besides the film: price, demand, how long each has been listed, where it sits in your navigation, whether anybody has heard of it. The comparison that means something is the same product, before and after.

That needs a baseline — a stretch of time before the film went up, the same length as the stretch you will judge it on afterwards. Long enough to contain a normal run of sales for that product: for something that sells a few a week, that means weeks, not days.

Then check that neither window contains something else that moves sales on its own. A promotion. A seasonal peak. A mention somewhere that sent a burst of visitors. A stock-out that sent them away. If one of those landed in either window, move the window, or wait until you have a clean one.

Change one thing

The most common way to waste a test is to make it part of a refresh. The film goes up the same week the price changes, the description is rewritten and a new still set replaces the old photos — and afterwards there is no way to say which of those did the work. Any of them might have. You will be tempted to credit the one you are most pleased with.

There is a quieter version of the same mistake. If the film goes into the first slot of your gallery, look at what your product page now shows before anybody presses play. If that is the film's opening frame rather than your hero photo, you have made two changes, not one: a new video, and a new first image. Either put the film in a later slot so the hero stays where it was, or accept that you are testing the pair, and write that down.

Stagger it, if you have more than one product

If you are adding films to several products, resist launching them all on the same day. Split them into two groups that look alike — similar prices, similar sales — and give one group its films now and the other a few weeks later.

While the second group is waiting, it is the closest thing a small shop has to a control: the same season, the same traffic, the same everything, minus the film. And when it catches up, you get a second before-and-after for free. If both groups moved at the same moment, it was probably the calendar. If each moved when its own films went up, it was probably the films.

Small numbers will lie to you

A product that sells a few units a month cannot tell you anything quickly. One extra order is a large percentage change on a tiny base, and it can arrive for reasons that have nothing to do with the listing. That is not a reason to skip measuring. It is a reason to judge slow sellers as a group rather than one at a time, and over a longer window than feels natural.

The honest outcome for many listings will be "no clear difference". That is a result, not a failure of method, and it usually means one of two things: the film is not answering a question buyers had, or buyers already had the answer before they arrived. Both are worth knowing before you pay for the next film.

Returns are the slow half

Sales answer quickly. Returns answer late, because a return only happens after delivery, after the buyer has had the product in their hands for a while. So a film judged only on sales can look like a success while it is quietly overpromising — winning the order and losing it again a fortnight later.

If the film was meant to settle something physical about the product — its size, its colour, its finish — keep watching the returns, and the reasons people give, for well after the sales window has closed. That is where a film that flatters the product shows up, and it is the one place a play count can never show you.

Keep a one-line log

None of this needs software. It needs a note, kept somewhere you will find it again, with one line per film:

• The product, and the date the film went live.
• Where it sits in the gallery, and whether that changed the first image a shopper sees.
• The one number you decided to judge it on, and what that number was in the window before.
• Anything else that changed on that listing in the same period.
• What the number was afterwards, read on the date you set — not earlier because it looked good, and not later because it did not.

A season of those lines is worth more than any general claim about whether product video works, including the ones on this site. They are about your products and your buyers, and they tell you where the next film should go.

The short version

A play count measures attention, not effect. Before a film goes live, write down the one outcome it is meant to change and the dates you will compare. Compare the product with itself, change nothing else at the same time, stagger launches when you can, and give returns longer than sales. Then let the log, not the enthusiasm, decide where the next film goes.