A monthly CRO report usually contains four different kinds of claim, and most reports present them as one number. Separating them tells you what your agency actually proved, what it only observed, and what it is hoping you will not ask about.

Four Different Things a CRO Report Can Be Telling You

Most reports blur these together. They should not be.

That fifth row is the one most reports omit entirely, and its absence is what makes the first row look like an achievement.

Only the third row is evidence. The others are useful — a directional opportunity is how next month's test gets designed — but they are not proof, and a report that presents them in the same typeface as a validated result is making a claim it cannot support.

"Revenue Is Up 12%" Is Not a Result

A pre/post comparison over a test window proves nothing on its own.

Revenue moved. So did the weather, your media spend, your promotional calendar, your traffic mix, and whatever your competitors did. A pre/post number attributes all of that to the CRO work by omission.

Three things that routinely move a monthly revenue figure more than any test does:

Traffic mix. Scaling paid social shifts your blended conversion rate downward even when every page performs identically. The reverse is also true — a month weighted toward email and branded search flatters everything.

Promotional calendar. A sale period compresses consideration and lifts conversion across the board. Comparing a promo month against a non-promo month is not a comparison.

Media spend. More spend, more revenue, same conversion rate.

None of this means the agency did nothing. It means the report has not separated the agency's contribution from everything else — and the only mechanism that does that reliably is a controlled test.

An Add-to-Cart Gain Is Not a Purchase Win

The most common substitution in CRO reporting.

A test is designed and powered to detect movement in one metric. The report presents a different one.

This happens for an understandable reason: add-to-cart has a far higher base rate than purchase, so it reaches significance faster and on less traffic. A test that could never have resolved on purchase in four weeks will resolve on add-to-cart comfortably.

The problem is that an add-to-cart gain does not reliably propagate to purchase. Users can add more and buy the same amount. The step that moved was not the step that constrains revenue.

The question to ask: which metric was this test powered for, and what did the metric below it do?

If the answer is that purchase showed no significant difference, that is not a failure — it is an honest result. If the answer is that purchase was not measured, the test was designed to produce a reportable number rather than a decision.

Why You Cannot Add Up Test Uplifts

If your report presents a stacked annual total — three tests at 8%, 12% and 6%, therefore 26% improvement — it is making a claim the underlying data does not support.

Three reasons.

Uplift estimates are systematically inflated. Martin Goodson's Most Winning A/B Test Results Are Illusory remains the clearest statement of why. Tests over-estimate uplift through what statisticians call the winner's curse — you are more likely to declare a winner when chance has worked in the variant's favour, so the measured effect is larger than the true one. The effect is strongest at smaller sample sizes.

Many winners are not real. Goodson's simulations show that checking results daily and stopping when either variant leads produces a "successful" result 41% of the time even when there is no real effect at all. Under that practice, false positives can account for as much as 80% of winning results.

Effects do not compound across separate pages. A product page win and a checkout win do not multiply. They may overlap, interact, or address the same underlying friction twice.

CXL's work on estimating the net value of a testing programme covers how to arrive at a defensible number instead. The short version: discount measured uplift, net off the losses, and treat any annualised figure as an estimate with a range rather than a result.

Counted Uplift vs Signed Net Impact

Here is a distinction almost nobody publishes, and it is the single most useful thing to ask about.

Counted uplift sums the wins. Three winning tests, three positive numbers, one total.

Signed net impact nets the losses off. Three wins and five losses, with the losses carrying their measured negative effect for however long they ran before being called.

Those produce very different numbers, and only one of them describes what happened to your business.

A programme reporting counted uplift is not necessarily being dishonest — it is an industry convention. But it is an incomplete accounting, and asking which one you are looking at tells you a great deal about the rigour behind the report.

Loss Prevented Is a Real Outcome

The counterweight to the section above, and it matters.

A test that stopped you shipping something harmful is a result. If a redesign variant lost by four points and you did not ship it, the programme protected four points of conversion rate. That is as real as a win, and it is invisible in every report that only counts winners.

With industry win rates sitting around 20–30%, most of what a testing programme produces is eliminated hypotheses and prevented losses. A report showing only wins is not showing you a better programme — it is showing you less of one.

Absolute vs Relative Lift

A conversion rate moving from 2.0% to 2.4% is a 0.4 percentage point gain and a 20% relative lift. Both are true. Only one of them tells you what happened.

Relative framing is standard in the industry and not inherently misleading. It becomes misleading when it is the only framing offered, because 20% sounds like a fifth of your revenue and it is not.

Ask for both, every time. A report that gives relative lift without the absolute points behind it is choosing the more impressive of two true numbers.

Every Number Needs Five Things

A rule worth applying to every figure in every report: it should carry its client, period, platform, currency and denominator.

Without those, numbers migrate. A figure measured on desktop gets quoted as a site-wide result. A test run on one product category appears in a summary about the whole catalogue. A conversion rate calculated on sessions sits next to one calculated on users, and nobody notices they are not comparable.

The denominator is the one that slips most often. "Conversion rate improved 15%" is not a claim until you know whether the denominator was sessions, users, or product page viewers — and switching between them can manufacture a result on its own.

A Report Line That Is True and Misleading

Variant B delivered a 23% uplift in add-to-cart rate. Annualised, this represents a substantial revenue opportunity.

Everything there can be literally accurate and it still proves nothing about revenue.

• 23% relative might be under two percentage points absolute

• The test was powered for add-to-cart, not purchase — so purchase may show nothing

• Annualising assumes the effect persists, does not regress toward the mean, and does not overlap with other changes

• Nothing nets off the tests that lost in the same period

• "Represents a substantial opportunity" is a projection wearing the clothes of a measurement

The fix is not to stop reporting it. It is to report it as what it is: a validated add-to-cart gain, with the purchase-level result stated alongside, and no annualisation.

Six Questions to Ask Your Agency This Month

1. Which of these numbers came from a controlled test, and which are pre/post observations?

2. For each test, what metric was it powered for, and what did purchase do?

3. How many tests concluded this period, how many won, and can I see the losers?

4. Is this counted uplift or signed net impact?

5. Can I have absolute points alongside every relative percentage?

6. What changed outside our control this month — spend, promotions, traffic mix, seasonality?

Question three is the most revealing. An agency that cannot produce a test log including losses either is not keeping one or does not want to show it. Our guide to choosing a CRO agency covers the version of this question to ask before you sign.

What a Defensible Report Looks Like

Four things:

Claims separated by type. Validated results, linked movement, observations and opportunities each labelled as what they are.

Every test shown, including losses, with the metric it was powered for, the sample size reached, and the absolute and relative effect.

External context stated — spend changes, promotions, seasonality — so the business movement section can be read honestly.

No stacked annual projection, or if there is one, an explicit range with the discounting method stated.

That is less impressive than a single large number, and considerably more useful. If the numbers underneath it are not reliable to begin with, none of this helps — our post on why CRO results take longer than brands expect covers what validated instrumentation requires, and what's actually in a CRO retainer covers what should be in scope to produce it.

For how many tests it takes before any of this shows up in revenue, see our piece on test volume and revenue impact. And for the underlying distinction between measured and assumed results, data-driven CRO vs traditional conversion optimization.

Want a CRO programme that reports what it proved rather than what moved? Talk to FunnelFreaks about a CRO Programme.