Landing page A/B testing: a rigorous checklist
Most landing page tests are unreadable before they launch. The sample size was never going to be sufficient, the metric was chosen after the fact, or the variant changed six things at once. A short pre-launch checklist prevents nearly all of it.
Daniel Williams · Jul 15, 2026 · 7 min read
Before you build the variant
Write down the hypothesis as a sentence with a direction and a mechanism: not "a new headline will perform better" but "leading with the integration list will increase demo requests from visitors arriving on integration keywords, because they are evaluating compatibility first."
Then compute the required sample size from your current conversion rate and the smallest lift that would change a decision. If the answer is longer than your traffic can supply in six weeks, do not run the test. Ship the change you believe in, or find a higher-traffic surface to test on.
Before you launch
Five checks, none of which take more than a few minutes.
- One primary metric, declared in writing, before any data exists.
- A minimum runtime covering at least two full business cycles, so weekday and weekend traffic are both represented.
- Variants that differ in one mechanism, even if that mechanism touches several elements.
- An A/A sanity check if you have never validated the assignment mechanism on this surface.
- Confirmation that assignment is sticky across sessions and devices where you can identify the visitor.
While it runs, and after
Do not stop early on a good-looking result. Peeking and stopping is the single most common way teams generate lifts that fail to replicate. Set the runtime, then leave it alone.
When it ends, record the outcome whatever it was — including flat results, which are the majority and the most useful for calibrating future hypotheses. A test log that only contains wins is a record of your stopping rule, not your learning.