In an ideal world, a marketer sends out 2 versions of an email, the first email is opened 58% of the time and the other only 2% of the time. It’s fairly easy to figure out that, almost regardless to sample size, the first email simply was superior to the second. But what about when the two emails are opened at 13.3% and 12.8% respectively? Was the first email better than the second, or was it just random chance that caused the 0.5% difference between the two messages? The answer to this question drives real decisions for marketers— decisions that lead to budget allocation, creative content, web page design and much more.
Every statistic that drives marketing decision should be checked— Is this stat pulled from a large enough sample to be interesting, or is the sample size insufficient to account for random chance? In my experience, I’ve seen entire marketing campaigns cancelled because of a statistically insignificant metric— don’t be this type of marketer. Making decisions based off of statistically insignificant metrics are essentially making decisions off of hunches, not data.
To determine if there really is a difference between two statistics, or if it is simply “chance”, the key thing to look at is the size of the samples and determine the margin of error. For example, if you send out 180 emails, 90 of subject line A and 90 of subject line B and observe an open rate of 13.3% and 12.8% respectively, there is an error rate of almost 10%, so you really have no clue which one was better. However, increasing the sample size to 90,000 emails of each version brings the error to 0.31%. If the same open rates were observed as before, we could say that the first email did have an actual open rate that was higher than the open rate of the second email.
To calculate this error rate, and thus sift the real differences from the random chance you’ll need a few things. First, you’ll need your numbers: Statistic A, sample size of A, Statistic B, sample size of B, and how confident you need to be (90%, 95%, or 99%). You’ll also need a tool to calculate your error margin. While it is possible to do this from the original formulas, I highly recommend using a tool to do this for you. There are many online calculators you can do this with, or downloads like this Excel spreadsheet for offline access.
Frequent use will probably increase your intuition for when sample sizes are large enough to generate real learning. Using and understanding statistical significance is key to making data-based decisions and taking your marketing efforts to the next level.