A/B Testing Your Marketing: What to Test, How to Test It, and What to Do With Results

Why Most Marketing A/B Tests Are Useless

The majority of marketing A/B tests share one of two problems:

Problem 1: The variable being tested has low impact on the outcome being measured. Testing button color when the headline is the primary decision driver produces data about the wrong thing.

Problem 2: The sample size is too small to distinguish a real effect from random variation. A test that ran to 50 recipients per variant and showed a 5% difference in open rate is not statistically significant. It is noise.

Good A/B testing means testing high-impact variables with adequate sample sizes.

The Impact Hierarchy for Marketing Tests

Test in this order — highest impact first:

1. The core offer or positioning: Does "content system" vs. "content strategy" in your headline change conversion? Positioning tests have the highest impact of any test type.

2. The hook or headline: The first line of a LinkedIn post, email subject line, or landing page headline. This determines whether content is consumed at all.

3. The CTA: Not the button color — the specific wording and the specific action requested.

4. Social proof placement and type: Does a testimonial above or below the fold convert better? Named vs. anonymous social proof?

5. Format variables: Short vs. long copy, image vs. no image, text vs. video.

Sample Size Requirements

For a test to be statistically valid, you need at minimum:

• —Landing pages: 500 visitors per variant
• —Emails: 200 opens per variant (not sends — opens)
• —Social posts: 1,000 impressions per variant

Running a test with less than these minimums produces unreliable data. A result that looks significant at 50 samples often disappears at 500.