---
title: "How Many Emails You Need for a Valid A/B Test"
canonical: "https://shopstack.tech/how-many-emails-needed-for-valid-ab-test/"
author: "ShopStack"
published: "2026-10-24T23:19:06+06:00"
modified: "2026-10-24T23:20:56+06:00"
language: "en-US"
site: "ShopStack"
description: "You can't pin a single number on \"how many emails needed for valid a/b test\" because it depends on four things: your baseline rate, the size of the lift…"
categories: "Email Marketing & Deliverability"
attribution: "ShopStack (https://shopstack.tech/)"
---

# How Many Emails You Need for a Valid A/B Test

You can’t pin a single number on “how many emails needed for valid a/b test” because it depends on four things: your baseline rate, the size of the lift you want to catch, your confidence level, and your statistical power. Change any one and the required sample size shifts, often by thousands of recipients. That’s why generic advice like “just send 1,000 per variant” fails so many marketers.

 

Run those four inputs through a standard sample size formula, and you get an actual answer. As of 2026, most email platforms still default to a 95% confidence level and 80% power, per Kohavi, Tang and Xu’s *Trustworthy Online Controlled Experiments* (Cambridge University Press). With those settings, a 5% baseline rate and a 10% lift target typically needs roughly 30,000 recipients per variant.

 

Smaller lift? Much bigger list. Here’s how to work out your own numbers before you hit send.

 

## how many emails needed for valid a/b test

 

There’s no fixed number. You need enough recipients per variant to detect your target lift with 95% confidence and 80% statistical power. Most tests fall between 1,000 and 50,000 per variant.

 

Your baseline rate and minimum detectable effect drive the exact figure.

 

## How Many Emails You Actually Need for a Valid A/B Test

 

An A/B test is a controlled comparison. You split a random slice of your list into two groups, send Variant A to one and Variant B to the other, then compare a single primary metric. If the difference is bigger than random noise, you call a winner.

 

Noise is the whole problem. Open rates bounce around by a couple of points send to send, even with identical creative. If your sample is too small, that natural jitter looks like a real result.

 

You ship a “winner” that was never real, and your next campaign proves it.

 

![how many emails needed for valid a/b test](https://shopstack.tech/wp-content/uploads/2026/10/how-many-emails-needed-for-valid-a-b-test-mvmnx4vy.webp)

 

Image source: Bing (Web (fair-use with source credit))

 

That’s why sample size comes first, before design, before copy, before you even open your ESP’s test builder. If your list can’t reach the required number, the honest move is usually to skip the test entirely and either make your best guess or run a bigger swing you can actually measure.

 

The minimum sample size per variant is a formula output, not a rule of thumb. Feed it four inputs and it returns the number. Miss one input and you’re guessing.

 

The next three sections walk through those inputs one at a time.

 

## The Four Numbers That Decide Your Sample Size

 

Every sample size calculation runs on the same four inputs. Get them right and the math is trivial. Get them wrong and your test is dead on arrival.

 

Here’s what each one does.

 

### Baseline Rate: Your Starting Point

 

Baseline rate is your current conversion rate for the metric you’re testing. If your last five campaigns averaged 22% open rate, your baseline for an open-rate test is 22%. If your click-through rate sits at 2.4%, that’s your baseline for a CTR test.

 

Pull this from at least four to six recent sends to the same segment. One campaign isn’t enough, because a single promo can swing the number wildly. If you’re testing on a new segment, use the closest existing segment as a proxy.

 

Lower baselines need bigger samples. A 1% conversion rate requires far more volume than a 20% open rate for the same lift. That catches people off guard every time.

 

### Minimum Detectable Effect: What You’re Trying to Catch

 

Minimum detectable effect (MDE) is the smallest lift you care about detecting. A 10% relative lift on a 20% open rate means 22%. A 20% relative lift means 24%.

 

Smaller MDEs demand bigger lists.

 

Pick your MDE based on what would actually change a business decision. If a 3% lift wouldn’t make you alter anything, don’t build a test around detecting it. You’ll burn your list chasing a rounding error.

 

Most practical email tests target 10% to 30% relative lifts. Anything under 5% usually needs a six-figure list and weeks of runtime.

 

### Confidence Level and Statistical Power: The 95% / 80% Default

 

Confidence level is your tolerance for a false positive. At 95%, there’s a 5% chance you’ll declare a winner that isn’t real. That’s the standard.

 

Bump to 99% for high-stakes sends like pricing or major launches.

 

Statistical power is your chance of catching a real effect when one exists. 80% is the baseline, meaning a 20% chance you’ll miss a genuine lift. Bump to 90% if missing a real win is costly.

 

![statistical significance](https://shopstack.tech/wp-content/uploads/2026/10/statistical-significance-mvmnx5x6.webp)

 

Image source: Web (Bing) / fity.club (Web image (fair-use with source credit))

 

Both numbers trade directly against sample size. Raising confidence to 99% or power to 90% inflates the required list fast. Stick with 95% and 80% unless you’ve got a specific reason to tighten.

 

## How to Calculate Sample Size Step by Step

 

You don’t need to hand-solve a formula. But you do need to understand what the calculator is doing, or you’ll misuse it.

 

### The Formula (Plain English Version)

 

The standard sample size formula compares two proportions. It takes your baseline, your target rate after the lift, your confidence level, and your power. Then it returns the minimum recipients you need per variant.

 

Evan Miller’s sample size calculator is the most widely cited free tool in the industry. It handles the math in seconds. So do Optimizely’s and VWO’s, though their docs are written for website tests rather than email.

 

### Worked Example: Small List vs. Large List

 

Here’s how the numbers shake out for two very different setups. Both assume 95% confidence and 80% power.

 

| Scenario | Baseline | Target Lift | Sample per Variant | Total Send |
| --- | --- | --- | --- | --- |
| Newsletter opens | 25% | 15% (to 28.75%) | ~6,500 | ~13,000 |
| Ecommerce CTR | 2% | 30% (to 2.6%) | ~22,000 | ~44,000 |
| Trial conversion | 8% | 20% (to 9.6%) | ~14,000 | ~28,000 |
| Small list, big lift | 30% | 50% (to 45%) | ~1,200 | ~2,400 |

 

Notice how a bigger lift target shrinks the requirement dramatically. That last row only works if a 50% relative lift is realistic. It usually isn’t.

 

### Using a Sample Size Calculator Without Fooling Yourself

 

Garbage in, garbage out. The tool only returns what your inputs deserve.

 

- Enter your real baseline, not your best-ever campaign.
- Set MDE to a number that would actually change a decision.
- Keep it at 95% / 80% unless someone senior is signing off on stricter.
- [Account for bounces and unsubscribes before you split](https://shopstack.tech/when-to-suppress-inactive-email-subscribers-deliverability/). Your sendable list is smaller than your total list.
- Add 10% to 15% to the result as a buffer for deliverability hiccups.

 

![minimum detectable effect](https://shopstack.tech/wp-content/uploads/2026/10/minimum-detectable-effect-mvmnx7zg.webp)

 

Image source: Web (Bing) / cleverutils.net (Web image (fair-use with source credit))

 

If your calculator says 40,000 per variant and your list has 8,000 subscribers, that’s your answer. Don’t run the test. Scale down the MDE ambition or pick a bigger swing.

 

## The Decision Tree: Can You Even Run This Test?

 

This is where the topic turns practical. Before you compute anything, walk through the branches below. If you fail any of them, no sample size math will save you.

 

### If Your List Is Under 10,000

 

Skip the standard A/B test. You can’t reach significance on most metrics. Instead, roll out the change to your whole list, then compare this campaign to the last four to six on the same segment.

 

That’s a pre/post comparison, not a true split test. It’s weaker evidence, but it beats a fake winner. For lists under 5,000, only massive swings (50%+ lifts) are detectable, and even then you need patience.

 

### If You’re Chasing a Lift Under 10%

 

Recheck your ambition. A 5% lift on a 2% CTR test needs tens of thousands per variant. Unless you’re sitting on a million-name house list, that test never reaches significance in a reasonable timeframe.

 

Bump the MDE to 20% or 30%, or change the metric to something with a higher baseline. Open-rate tests need less volume than CTR tests on the same list.

 

### If You Can’t Reach the Number: What to Do Instead

 

You have three options, in order of preference.

 

1. Test a bigger swing. A completely new subject line approach beats a subtle copy tweak for statistical reach.
2. Test on a higher-baseline metric. Opens move faster than conversions.
3. Aggregate across sends. Run the same test concept over five identical weekly sends, then combine the totals.

 

Option 3 is a sequential test. Kohavi’s research warns it inflates false positives if you peek at results after each send. Pre-commit to five sends and only analyze at the end.

 

Now that we’ve covered the full assessment workflow, the next batch covers the mistakes that invalidate tests, the realistic capability of each list size, FAQs, and your final decision guide.