How to A/B Test a Landing Page for Ecommerce (And What Not to Test)

Landing page tests have the highest failure rate of any experiment type in e-commerce not because the pages are hard to improve, but because the tests are hard to measure correctly.

The conversion being tested happens somewhere else. The page often lives on a different domain, built in a different tool, firing different events. By the time a user completes a purchase, they've moved off the page under test, possibly across a session boundary, and the result you read may have very little to do with the variant you shipped.

This covers why that happens, what's actually worth testing, how to run one properly, and the mistakes that consume a quarter and produce nothing usable.

Why Landing Page Tests Fail

Three reasons, in order of how often they're the cause.

The measurement breaks before the test does. Landing pages built in external tools rarely inherit your store's GA4 ecommerce implementation. If view_item fires on your Shopify product pages but not on your landing page, your funnel comparison is measuring two structurally different things. The LP looks like it generates no product engagement when it may generate plenty.

Sessions fragment at domain boundaries. An LP on offers.yourbrand.com moving users to your main store creates a session break unless cross-domain measurement is configured. Variant assignment lives in one session; the purchase happens in another. The connection is lost, and the LP appears to convert at nearly zero.

Traffic isn't held constant. A test comparing an LP against a product page, where the LP received retargeting traffic and the PDP received cold prospecting, measures audience temperature rather than page quality. Traffic source affects conversion rate more than page design does so any comparison that doesn't control for it is measuring the wrong variable.

Our guide to comparing landing pages against product pages for paid traffic covers how to set that comparison up so it means something.

What to Test

Prioritise by how much of the conversion decision the element carries. Not everything on a landing page is worth an experiment.

High value: test these

Medium value: test after the above

Hero imagery, headline phrasing variants, trust badge selection, secondary CTA presence.

Low value: usually not worth a test cycle

Button colour, minor copy tweaks, font choices, icon styles. These occupy test slots that could be running higher-impact experiments, and at typical D2C traffic volumes you only get three or four cycles a quarter.

The prioritisation rule: rank by potential impact divided by implementation effort, and be honest about traffic. If your volume supports four tests this quarter, spend them on the four elements most likely to change behaviour — not the four easiest to build.

How to Run One Properly

Step 1 : Verify the events before anything else

Five events need checking, each with a landing-page-specific failure mode:

• page_view is the landing page dimension populating, or falling to (not set)?

• view_item does the LP fire it at all, with item_id matching your store's format?

• add_to_cart does the LP's cart mechanism trigger the same event as your store?

• begin_checkout does it fire with the original campaign source intact after the domain transition?

• purchase exactly once, with transaction_id, value, and currency populated?

Then reconcile at account level. Pull GA4's purchase count against your Shopify order count for the same 30-day window:

That last row matters most. An inflated baseline corrupts your sample size calculation, so the test appears conclusive before it is. Our guide to why GA4 and Shopify numbers don't match covers the diagnosis.

Step 2 : Write a specific hypothesis

Structure it so the result is interpretable regardless of direction:

Observation: [what the data shows, with the segment named] Change: [one variable] Expected impact: [direction and rough magnitude] Measured by: [primary metric, sample size, duration]

"Improve the landing page" is not a hypothesis. "Mobile users from Meta prospecting scroll past the CTA without clicking, so moving it above the fold should raise add-to-cart rate for that segment" is.

Step 3 : Calculate sample size before launch

From your verified baseline conversion rate, your minimum detectable effect, and a 95% confidence threshold. Commit to the duration before you see any data.

At typical D2C volumes this usually lands between three and six weeks. If the calculation says eleven weeks, the test isn't viable — pick a change with a larger expected effect or accept that your traffic can't answer this question yet.

Step 4 : QA across devices and browsers

Confirm the variant renders correctly on real mobile devices, that tracking fires identically on both control and variant, and that the variant doesn't break any downstream event. A variant that inadvertently blocks add_to_cart produces a decisive-looking loss that has nothing to do with the change you made.

Step 5 : Run to completion, then decide

No peeking-and-stopping. Analyse at the predetermined sample size, check whether the effect holds across device and source segments rather than only in aggregate, and log the result whether it won or not.

What Not to Do

Don't test without validating events first. The most expensive mistake, because the failure is invisible. You get a result, it looks valid, and you ship it. We documented a full case in our account of A/B testing on broken GA4 data, where a quarter of declared winners were tracking artefacts.

Don't change multiple elements at once. A variant with a new headline, new hero image, and repositioned CTA tells you something changed. It doesn't tell you which change did it, so you can't apply the learning anywhere else.

Don't stop the test when one variant looks ahead. Early leads reverse routinely. Peeking and stopping is the single most common source of false positives in ecommerce testing.

Don't test during a sale period. Sale traffic is a different population, more price-sensitive, more first-time visitors, converting for reasons unrelated to your page. A variant that wins during Diwali may not hold in February.

Don't compare an LP against a PDP without controlling traffic. Same source, same medium, ideally same campaign on both sides. Otherwise you're measuring audience, not page.

Don't run tests your traffic can't support. Below roughly 10,000 monthly sessions to the tested page, most tests won't reach significance on a practical timeline. Fix obvious problems directly instead of testing whether they're problems.

Don't ignore the losers. Roughly seven in ten tests don't produce a significant positive result. That's what rigorous testing looks like. A losing test that eliminates a hypothesis is a real output; log it, or you'll test the same idea again in six months.

Don't assume the tool is the differentiator. Most platforms handle the core job. Where the page lives, whether you need custom event tracking, and flicker tolerance matter more than feature comparisons. Our guide to running A/B tests on Shopify with GTM covers the no-third-party-tool approach and its limits.

The Pre-Launch Checklist

• All five GA4 events verified on the landing page and downstream

• Cross-domain measurement configured if the LP is on a separate domain

• Payment gateway domains on the referral exclusion list

• GA4 purchase count within 10% of Shopify order count

• Baseline conversion rate confirmed from reconciled data

• Single-variable hypothesis written with a measurement plan

• Sample size and duration calculated and committed to

• Variant QA'd on real mobile devices

• Tracking confirmed identical on control and variant

• Date range avoids sale periods and campaign spikes

Our pre-CRO data audit guide covers the validation scope in more depth.

The Short Version

Verify the events, then test one thing, on enough traffic, for a predetermined duration, on a page whose conversion you can actually attribute.

Most landing page tests fail on the first clause. Twenty minutes of verification before launch is cheaper than a six-week test you can't interpret.

Want your tracking verified before your next landing page test? Talk to FunnelFreaks we validate the events so your results mean what they say.