The Marine Mammal Center has a strong retail component. As any facility with visitors should, it has a gift shop which sells intangible support items (Adopt-a-Seal) as well as very tangible ones, like warm socks with various marine mammals on them. It also has a basic e-retail store (housed and managed within their Convio website)
For two years the marketing team has experimented with featuring e-retail at various points during the calendar year, or with various audiences. Some discoveries we’ve made historically include:
As we approached Valentine’s Day 2013, we decided to try an experiment with our last Valentine’s Day e-retail email before the shipping deadline. We split our list in two (we use Convio Luminate) and sent two emails with different leads.
The original and the variant
Our original format had an animal story at the top with a charismatic photo of the patient.
The variant instead substituted a product as the lead:
The results
Both emails were sent as part of an A/B test to half of our housefile. They got almost the exact same open rate (15% - on the low to average end of our e-retail emails). Of those that opened either version, about 11% clicked through to various places on the website. The difference comes in results. The original generated 8 purchases in our e-retail store, whereas the variant with the product at the top generated 14 purchases and 2 donations, for about double the revenue vs. the original.
When we computed statistical significance we found that statistically it only reaches a 90% confidence level and we generally look for 95% confidence in A/B tests. However, it’s a strong indicator, so we plan to repeat the experiment with a different product and a new animal story to see if we get similar results.
There’s one weird quirk to all of this as well: the original version with the animal at the top got almost 150 more forwards than the one with the product at the top. And the original, once forwarded, generated 31 extra clicks from friends-of-friends onto e-retail product pages.
At first we were excited about this, and then we realized that for all this forwarding, it still did not generate enough additional revenue to make the original as compelling as the variant – it didn’t motivate people to buy or give at that point in their e-retail experience.
As a rule, we try and put an A/B test into every single piece of large-segment email we send out. Tell us in the comments what sorts of A/B tests you’ve run on your e-retail emails and what results you’ve seen.
PDFs of each full email can be seen here and here.
Kate Harle and Dale Anania are the core marketing staff at The Marine Mammal Center, and Shabbir Imber Safdar (of Safdar Analytics) is their analytics consultant.