Signal vs. noise

Did you do it?

Did you run the subject line split test I outlined last week?

To recap, I asked you to set up a subject line A/B test, but make sure that A and B are exactly the same.

Why?

To prove that most people who are running split tests aren’t doing it right.

If you did what I asked, then you probably got slightly different open rates for each version, even though they were identical.

So where did the difference come from?

Randomness.

I can’t explain this any better than Jeff Goldbloom with a drop of water, but here goes:

There are more variables impacting each outcome than we realize.  The uncontrolled variables introduce “noise” into the signal we are trying to measure.

And if we do not understand how much “noise” there is, we cannot possibly know whether or not the differences we see are real or random.

More sample points reduce the amount of noise, so split tests with very big lists are more accurate.

How big is big?

Thankfully, there are math tools we can use to help us figure this out.

I found one for you and linked it here.

It does all the math work for you.  If you see a difference between two things you are split testing, punch the numbers into this calculator and it will tell you if the difference is significant or not.

This way, you don’t waste time testing things that don’t matter.

Have a good one,

Greg

P.S. I’m getting ready to take some time off, but not before reaching into my bag of gifts for all the good boys and girls on my list.

Keep an eye out, because I am going to send a series of custom offers and gifts to people based on their engagement.

Those of you who read everything and interact with me are going to get some special things.

But the low open rate zombies are going to get a lump of coal.

(It’s not too late to change your ways, just make sure you open and read everything I send.)