A/B testing: a guide to reliable decisions

By Patricia Jarmek Marketing Strategy & Analytics 29 October 2026

6 min read

A/B testing sounds simple: show two versions, compare the results and keep the winner. In practice, the most damaging mistakes happen before the first visitor enters the experiment—when the problem is unverified, the hypothesis is vague, too many elements change at once or the team checks the result too early.

A reliable test is not a design contest. It is a controlled experiment intended to answer a question set in advance. Google Analytics defines an A/B test as a randomized experiment in which two or more variants of the same page are shown concurrently to random samples of users to determine which performs better against a specific goal [1].

Start with a problem, not a button idea

Before testing, establish where users struggle and why that problem matters to the business. Combine funnel data, on-page behavior, customer feedback, support questions and sales conversations. Only then should you design a change.

A useful hypothesis follows this form: If we change X for audience Y, we expect outcome Z because evidence D indicates problem P. Google Ads similarly recommends a clear hypothesis tied to a business goal and testing one variable at a time; changing several things makes the cause of the result difficult to identify [2].

For example: “If we show the full cost before asking for personal details in mobile checkout, we expect more completed purchases because session recordings and support questions show users returning to the basket to find the final price.”

Control and treatment must differ meaningfully

Variant A is the existing control; variant B is the intentional treatment. If you change the headline, offer structure, field order and button color together, you may observe a difference but gain little reusable knowledge.

This does not restrict teams to tiny cosmetic tests. If evidence points to a broken end-to-end journey, a redesigned checkout can be tested against the existing flow. The hypothesis must describe the package of changes and the mechanism through which it is expected to work.

Official GOV.UK guidance describes A/B testing as a randomized comparison of a control and variation, with sample size, duration and an even random split considered before the test [3].

Choose one primary decision metric

The primary metric decides the test. For checkout, that might be completed purchase rate or revenue per visitor. For a B2B form, it may be sales-qualified leads per visitor rather than raw form submissions.

Add guardrail metrics. A shorter form may increase lead volume while reducing quality. A larger discount may raise conversion while eroding margin. More CTA clicks may increase checkout starts while also increasing abandonment at the next step.

Before launch, record the primary metric, guardrails, target segment, minimum meaningful effect, planned duration and decision rule. GOV.UK technical guidance recommends defining the hypothesis, audience and primary metric before building the test, then quality-assuring devices, browsers, login states and measurement before launch [4].

Do not stop when you like the result

Checking every day and stopping as soon as B leads increases the risk of declaring a false winner. Determine the sample and duration in advance, include a full business cycle and account for conversion delay.

Statistical significance does not say whether an effect is commercially important. Google Ads uses confidence intervals and two-sided significance testing at the 95% level for its experiments, but the decision still needs the effect size, implementation cost and guardrail outcomes [5].

An inconclusive result is not a failed experiment. The change may be too weak, the sample too small or the element irrelevant to users. The correct conclusion is “we did not establish an effect in this range,” not “the variants are certainly identical.”

Use segments for learning, not winner hunting

You can review results by device, traffic source or user type, but treat post-test slicing carefully. The more segments you inspect after the fact, the more likely a random difference will look like an insight.

Predefine a segment when the hypothesis expects a different response—for example, a mobile-only navigation change. Treat an unexpected segment difference as a hypothesis for a new test, not as final proof.

Protect SEO during the experiment

When variants use separate URLs, Google recommends rel="canonical" pointing to the original URL and temporary 302 rather than permanent 301 redirects. Do not show one version to Googlebot and another to users, and remove the experiment as soon as sufficient data has been collected [6].

For dynamically inserted variants, check speed, layout shifts and measurement integrity. An experiment that changes the experience while breaking analytics or Core Web Vitals cannot provide a clean answer.

When A/B testing is the wrong tool

Do not leave an obvious technical bug, incorrect price, legal requirement or accessibility barrier in place for half of users just to run a test. Fix it and verify the repair.

Testing is also a poor fit when traffic cannot support a reasonable sample, business conditions change rapidly or users cannot be consistently assigned to a variant. Interviews, usability tests, session analysis, prototypes or a carefully qualified before-and-after analysis may be more useful.

A dependable testing workflow

Confirm the problem with quantitative and qualitative evidence.
Write the hypothesis and expected mechanism.
Define the control, one key change and eligible audience.
Predefine the primary metric and guardrails.
Estimate the sample, duration and minimum meaningful effect.
QA assignment, analytics, devices and browsers.
Run the test without mid-experiment changes.
Evaluate the interval estimate, commercial value and planned segments.
Document the conclusion, implement the winner or design the next hypothesis.

The most valuable outcome is not always variant B. It is a decision made with less uncertainty and a growing body of company-specific evidence that is more useful than a collection of borrowed “best practices.”

Frequently asked questions

What is A/B testing?

It is a randomized comparison of a control and one or more variants to estimate which performs better against a goal defined in advance.

How long should an A/B test run?

Until it reaches the planned sample and covers a relevant business cycle. Duration depends on traffic, conversion rate, expected effect and conversion delay.

Is statistical significance enough to make a decision?

No. Consider effect size, commercial value, implementation cost and guardrails such as margin and lead quality.

Can several changes be tested together?

A complete redesign can be tested as a package, but the conclusion applies to the package. Change one key variable when you need to learn about an individual element.

What does an inconclusive result mean?

The experiment did not provide enough evidence for the planned effect. It does not prove the variants are perfectly equal.