Bad news about your split test

“Always. Be. Testing.” - certain marketing gurus

Have you ever run a split test?

If so, you’re probably doing it wrong.

In email marketing, the most common element to split test (or A/B test) is the subject line.

Most email software will allow you to plug in 2 different subject lines, then track performance of each to see which is “better.”

In this case, “better” usually means the one that results in a higher open rate.

Makes sense, right?

In principle it is, but there’s a couple of problems.

1. How do you know that one option is better than another?

2. How do you know when you know that?

3. In the context of your business goals, are you optimizing the right things?

To illustrate #1 and #2, if subject line A gets a 40% open rate, and subject line B gets a 50% open rate, how confident are you that the open rate difference you observe is actually because of the subject line?

There’s not enough information in this example to know for sure. 

For that, you would need to know how many total data points there were.

(What if subject line A was sent to 10 people, and 4 opened? And subject line B was sent to 10 people, and 5 opened? Have you truly learned anything?)

But hold on, let me back up.  To illustrate what I’m talking about, I’m going to share an absurd example:

Imagine that I own two pairs of socks.  One of them is plain black, the other is green.

I want to measure the impact that each pair of socks has on the outcome of a coin toss.

I put on the black socks, and flip the coin once.

Heads.

I take off the black socks, slip on the green, and flip the coin again.

Tails.

Amazing! It seems that wearing black socks causes the coin to land on heads, and green socks cause it to land on tails.

But it should be obvious that this is fake data.

We all KNOW that if I flipped the coin 100 times in each pair of socks, I’d end up very close to 50/50 in each pair, right?

…but how close?

What if I ended up with 51% heads in black, and 51% tails in green?

What conclusion would you draw and why?

And how is that conclusion any different than your last split test? 😬

Right now, you may be sweating a little, maybe swearing a little.

And rightfully so.

In the email list-building training I delivered last week, I highlighted the dangers of “optimizing” for the wrong thing when building our list (specifically, number of subscribers instead of quality of subscribers).

Then, I explained how to optimize for the right things.

(For a limited time, you can watch Day 1 of the $399 course for free here, where I explain this concept in much greater detail.)

Even after watching, some of you won’t believe me.

But to prove my point, let’s perform a magic trick together.

This week, I want you to run your own split test.

Go to your first email in your automated welcome series (you have an automated welcome series, right?) and set up an “A/B test” for the subject line.

Most email software will let you set this up with a click.

You’ll now have an “A” version and “B” version of that first welcome email.

Now for the fun(ny) part:

In the B version, don’t change anything.

You are going to run a split test where both versions are exactly the same.

By next week, you will have results for each version.

And I predict that they are not exactly the same.

See the problem?

Later, I’ll show you how to mentally untangle the knot we just tied, and how to run split tests that can give you real, actionable data.

See you then,

Greg

P.S. If you’d like to jump ahead and learn how to build a quality email list by optimizing for things that actually matter (read: generating revenue and creating superfans), you can sign up for the course wait list here.