Adventures in A/B Testing Marketing Emails

At Supercircuits, we run A/B (or split) tests on our marketing emails on a regular basis. Here are some recent ones — see if you can guess which email was the winner for each. Unless indicated otherwise, the email itself was the same and just the subject lines were changed.

This first A/B test was for an email showcasing our infrared security cameras. Which of these emails had the most opens and clicks?

• 1a: Protect Your Property Day or Night with IR Cameras – Starting at $99
• 1b: Invisible IR, Smart IR, Long Range IR, More! IR Cameras start at $99

We send out a daily deals email every Monday. Did the incumbent, 2a, keep its title or did the challenger take it down?

• 2a: 7 Days, 7 Great Deals!
• 2b: 7 Daily Deals You Don’t Want to Miss!

Here’s an interesting one — for this test we sent a version of the email that included pricing information and one which did not. For both emails we used the same subject line:

• 3a: Advanced Body-Worn Camera Kit is Flexible, Covert and Powerful *[no prices shown]*
• 3b: Advanced Body-Worn Camera Kit is Flexible, Covert and Powerful *[prices shown]*

This fourth test is for another weekly deals email, that is often themed around a particular product type:

• 4a Deals of the Week: Save up to 60% on Domes, Bullets, PTZ Cameras and More!
• 4b: This week only: Save up to 60% on these top selling security cameras!

Sometimes small changes can make a big difference. Here are two Fat Tuesday-related email subject lines:

• 5a: Fat Deals for Fat Tuesday – Save up to 50% all week!
• 5b: Fat Deals for Fat Tuesday – Save up to 50% this week only!

This email is an example of one that goes out to our law enforcement customers:

• 6a: NEW! Affordable, Expandable City-Wide Surveillance Systems
• 6b: City-Wide Surveillance Systems: Do More with Less

This last test is part of a reacquisition email campaign we sent out to customers during March Madness. We sent out a series of emails with different offers and subject lines, depending on when you last made a purchase with us.

• 7a: This offer is a SLAM DUNK! Save 10% on your next video security purchase
• 7b: Come back and Score! Save 10% on your next video security purchase

Answers

1. Email 1b was the winner as far as open rates are concerned, with a 7% improvement. However, 1a won as far as overall clicks were concerned with a 6% improvement over 1b.

2. Short and sweet won the day with email 2a having a 27% improvement over 2b for open rates and an 11% improvement in the number of clicks.

3. For some reason, email 3a — which had no prices — had an 8% higher open rate over 3b, even though both subject lines were the same. More importantly, it also had a 60% increase in the number of clicks compared to the email with pricing information.

4. The subject line for email 4a led to a 25% increase in open rates. However, email 4b had a 50% greater number of clicks. I guess the moral of this story is, don’t even bother trying to guess the winner … just test.

5. Sometimes small changes also have a small effect. Both email 5a and 5b had the same open rate, with email 5a doing slightly better and generating 4% more clicks.

6. The simpler subject line in email 6b resonated more with readers and generated 72% more opens. It also led to 4x the number of clicks as email 6a (which was exactly the same email apart from the subject line).

7. Subject line 7b was the bigger hit with our target segment, generating 15% more opens and 86% more clicks.