Why Your Festive-Season Test Results Will Lie to You

During India's festive window, prepaid orders overtake COD. That is not more of your usual traffic — it is a different population, buying differently. Any test you read inside that window is answering a question about shoppers you will not see again in January.

Your Sale Traffic Is a Different Population

Start with the clearest evidence, because it is the one that settles the argument.

ClickPost tracked a fixed cohort of 200 D2C brands that shipped more than 500 forward orders in both the 2025 and 2026 festive windows. Across that cohort, prepaid shipments grew 47% year on year against 27% for COD — and prepaid's share of D2C shipments rose from 46% to 52%.

GoKwik recorded the same crossing during Diwali 2025: 52.42% prepaid against 47.58% COD, with festive prepaid share having moved from 37.44% the previous year.

Read that again as a measurement problem rather than a market story. The payment mix inverts inside a few weeks. A store that is COD-dominant in September is prepaid-dominant in October and COD-dominant again by December.

Prepaid and COD buyers do not behave the same way at checkout. Prepaid buyers complete a payment; COD buyers confirm an order. They abandon at different steps, respond differently to trust signals, and return at wildly different rates — Unicommerce's data puts COD returns at around 58% during the festive quarter against under 15% for prepaid.

So when you test something at checkout during the window, you are testing it on a population whose majority payment behaviour is temporarily inverted.

What Else Moves in the Window

Payment mix is the sharpest shift. It is not the only one.

The AOV line is the one worth pausing on. Order volume rose 31% while AOV fell 10% — growth came from more orders, not bigger baskets. Median basket values held roughly stable across both payment types.

That is a composition change, not a scaling one. More buyers, buying smaller, skewed toward smaller cities, paying differently.

What we observe across client accounts, labelled as observation rather than measured finding: device mix shifts further toward mobile during the window, new-visitor share rises sharply, and channel weighting moves toward paid and marketplace-adjacent traffic as brands scale spend into the season. We have not quantified these across a controlled sample, and we are not going to pretend otherwise.

Why That Breaks a Test, Not Just Slows It

This is the intellectual core, and it is worth being precise about.

More traffic does not break a test. Different traffic does.

An A/B test measures the average treatment effect across the population that saw it. Statistical significance tells you that effect was real for that population. It tells you nothing about whether that population resembles the one you will have next month.

So a festive test can be perfectly executed — sample size calculated upfront, run to completion, no peeking, clean 95% result — and still produce a wrong decision. The statistics are not lying. They are answering a narrower question than the one you asked.

Consider a checkout variant that performs well on prepaid buyers and poorly on COD buyers. Run it in October, when prepaid is the majority, and it wins. Ship it, and from December it serves a majority-COD population where it loses. The test was valid. The inference was not.

And nothing in your test output will flag this. A significance calculation has no concept of population drift. It will report the same confidence whether your traffic mix was stable or inverted.

The only defence is knowing what your mix was during the window, and asking whether the variant's effect depends on it. Our post on prepaid conversion as a CRO problem covers why payment mix is a conversion variable rather than a fixed property of your audience.

The Pre/Post Trap

Separately, and more commonly: comparing the sale fortnight against the fortnight before it.

That comparison measures the discount. Conversion will be up, and everyone already knew the discount would do that. Attributing it to anything shipped in the same window is not analysis.

This is the seasonal version of the broader problem covered in how to read a CRO agency's monthly report — observed business movement presented as validated result. The festive window makes it worse, because the external influence is so large that it drowns everything else.

If a report lands in November showing a large conversion improvement over October, the honest reading is that you ran a sale.

What's Safe to Run and What Isn't

The dividing line: run things that are true regardless of who the visitor is. A checkout form that throws a validation error on a valid pincode is broken for everyone. Social proof placement is not.

Freeze Rules

Concrete dates relative to your sale start, rather than general caution.

T-21 days — stop launching. Any new test started inside three weeks of the window will have its measurement period overlap the mix shift. Do not start it.

T-14 days — triage what is running. For each live test, ask whether it reaches its pre-calculated sample size before the window opens. If yes, let it run. If no, pause it rather than letting it bleed into the window. A paused test can resume on comparable traffic; a contaminated one cannot be cleaned.

T-7 days — freeze variants and deploys. No variant changes, no new tests, no site deploys beyond fixes. This is also when your tracking should be verified rather than assumed, because a tag that breaks on the first day of the sale costs you the whole window's data.

Window open — run only the safe column. Fixes and stability work only.

T+14 after the sale ends — wait. Traffic mix does not normalise the day the sale closes. The tail runs into the following weeks.

T+21 — resume. Restart the paused tests. Treat anything the window surfaced as a hypothesis, not a finding.

What to Collect Instead

The window is the worst time of year to test and the best time of year to observe. Volume is at its peak, which means qualitative signals that are too sparse to see in February are visible in days.

Collect, without testing:

• Session recordings filtered to high-intent segments — checkout entries that did not complete

• Site-search queries, particularly zero-result searches. The window shows you demand you are not meeting

• Support tickets and WhatsApp queries, categorised. The questions people ask under time pressure are the questions your product pages are not answering

• Stock-out and size-out patterns, by SKU and variant

• Checkout error logs, including payment failure reasons by gateway and method

None of this requires a controlled comparison. All of it becomes January's hypothesis backlog.

Reading the Window Afterwards

Report what happened without claiming the programme caused it.

Separate the claims explicitly. Observed business movement — volume, revenue, AOV — belongs in its own section, with the sale named as the cause. Any fixes shipped during the window are CRO-linked movement at best, not validated uplift. Qualitative findings are directional opportunities.

And report the mix alongside the outcome. A conversion rate that rose during a window where prepaid share rose is partly explained by the mix, not the work.

The Counter-Position, Fairly Stated

Convert.com argues for testing through peak periods, and their case is sound within its context: a mature programme on a high-volume US store during BFCM, where the traffic surge delivers sample sizes that are otherwise unreachable, and where the team already understands how their peak population differs from baseline.

Two things differ here. Market: the Indian festive window carries a payment-mix inversion that BFCM does not, which changes the composition in a way volume alone cannot account for. Method: that case depends on having a characterised baseline to compare against. Most brands running their first or second festive season do not have one.

If you have several seasons of segmented data and know how your festive cohort behaves against your January cohort, test through the window and segment the read. If you do not, the freeze is the safer call.

What I'd Do Differently Next Year

Instrument the cohort before the window rather than after.

The reason festive findings get thrown away is not that the data was bad — it is that nobody can separate the festive cohort from the rest afterwards. If payment method, city tier and new-versus-returning are registered as dimensions before October, the window becomes analysable rather than merely busy. You can then ask whether a variant's effect held within each segment, which is the question that determines whether it carries into January.

That is a week of instrumentation work in August. It converts the busiest month of the year from a measurement blackout into the richest segmented dataset you will collect.

The January Question

When the window closes, sort the findings into three piles.

Retest on normal traffic: anything that depends on buyer intent, trust, or payment behaviour. The festive result is a hypothesis.

Keep without retesting: fixes. A broken form is broken in January too.

Throw away: anything about discounting, urgency or offer framing. It was measured on a population shopping a sale, and it will not transfer.

If you are planning the next quarter's roadmap from festive findings, our post on how many A/B tests it takes before revenue impact covers realistic cadence — and our Indian D2C conversion benchmarks note why festive figures should not be compared against annual ones. RTO also spikes in this window, which is covered in RTO as a measurement problem.

Planning next year's testing calendar around the season? Talk to FunnelFreaks about a CRO Programme.