Sabotaging Trust: How A/B Testing Can Actually Hurt More Than It Helps

by on December 17, 2013 in Best Practices | Email Marketing | Targeting
13 Flares Twitter 0 Facebook 0 Google+ 1 LinkedIn 5 StumbleUpon 0 Buffer 7 Email -- 13 Flares ×

Trust is an important topic in marketing today

5 Attributes screenshot The-Definitive-Guide-to-Engaging-Email-Marketing Trustworthy - Always Relevant - Conversational - Coordinated - Strategic

Marketo’s 5 key attributes to engage and succeed in email marketing

In Marketo’s recently released 156 page Definitive Guide to Engaging Email Marketing, Trustworthy was the first of five key attributes “to engage and succeed.”

Increased connectivity has evolved customers and their ability to share experiences.  If one angry customer rant goes viral, your A/B tested but slightly misleading subject line can sabotage the trust of every potential or current customer with a Facebook account.

Just as unsettling, broken trust and the resulting unsubscribes and abuse reports can crush your ability to deliver even to those who are interested. Marketers now need to evolve, and nurture the trust of the communities that support their businesses.

Yet every day customers are bombarded by misleading subject lines

Today’s marketing automation tools make it easy to oversimplify the success or failure of a campaign.  By focusing on details and forgetting about the bigger picture, it is easy to make a big mistake.

For example, let’s discuss an electronics retailer wanting to liquidate their overstock Windows and Android tablets.  This retailer knows the value of A/B tests, and so they test the following two subject lines.

A.   20% off all tablet computers*
B.   20% off all Android and Windows Tablets

Subject line A shows an open rate of 38%. Subject line B shows an open rate of 30%. Just by looking at the results of the test Subject A would seem to be the winner, and so our marketer proceeds, patting themselves on the back for crafting a better subject line and increasing engagement.  But there is more to this story.

email subscriber targeting

Image courtesy of imagerymajestic /

Unfortunately, most of the additional opens enjoyed by Subject A left the customer with a bad taste in their mouth. These are people who were excited about an opportunity to buy a discounted iPad.  When they read through the email however, the asterisk led them to a footnote at the bottom of the email that said “Apple products not included.” These customers felt deceived, and many unsubscribed or reported the email as spam.

Customers who opened the email with subject line B were hoping to have an opportunity to buy an Android or Windows tablet on sale. They were correctly targeted by the subject line and more likely to click through and buy something, unfazed by any disqualifying footnote.  Unfortunately, some of these people never got a chance to purchase because increased spam reports from would be iPad buyers led to delivery problems, affecting this and all future emails from our retailer.

By neglecting to consider how a subject might affect customer trust, our electronics retailer mistook a higher open rate for more meaningful customer engagement. By souring the relationship with their customers, what they really got was unhappy customers who disengaged, and a damaged reputation.

Let trust be your guide

Only time and consistency can build trust. Every interaction you have with your customer builds your relationship with them. But it only takes one negative interaction to destroy what you’ve worked so hard to build.

Be careful to measure what is important to your business, look deeper into the data, and find the truth behind the statistics.  Two quick ways this marketer could have better targeted the right customers initially:

  • Identify customers who have opened or clicked previous emails via Android, Windows, or iDevices.  Use this data to segment customers.
  • Identify customers who have historical purchases around those three devices, even accessories.  Use this to segment customers ahead of time.

Increase your success by getting the right message to the right people, not fooling an extra few percent into opening or clicking on something they don’t want.  Your long term revenue will thank you.

13 Flares Twitter 0 Facebook 0 Google+ 1 LinkedIn 5 StumbleUpon 0 Buffer 7 Email -- 13 Flares ×