![\"bang-head-wall\"](\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2014\/09\/bang-head-wall.png\")
Back in 2005, I started my first forays into split testing of emails, landing pages and funnels whilst working at LOVEFiLM.\u00a0 I'm nearly at my 10 year anniversary, and after seeing over 40 million visitors split tested with stuff, I've become rather battle hardened to all the curious, strange and just plain stupid ways I've managed to break my own experiments.<\/p>\n
I'll be doing a talk this Wednesday at the Smart Insights Digital Marketing conference<\/a>\u00a0on how to improve structured testing to boost profits,\u00a0<\/strong><\/span>so to introduce it, I wanted to write a little about one critical way we can all improve\u00a0our AB and Multivariate testing.\u00a0 So, in this article, I'll be exploring device and browser compatibility or 'Getting with what your customers have at their fingertips'.<\/p>\n Firstly, let me use an analogy and an explanation.\u00a0 I once interviewed a developer and asked a question as part of the process - 'What's the best way to code your sites to get maximum reach and compatibility with what customers actually use?' After a long and rambling explanation, he finally finished with the statement 'Well if people want to use Internet Explorer 8, then they're just effing stupid'.<\/p>\n I then looked at his CV and asked a question, as I knew he had two young children - 'So if your wife and you take your double buggy for a shopping expedition and it won't fit through shop doorways - is that because you were too effing stupid to buy the right one then?\u00a0 Nothing to do with the retailer?\u00a0 All the customers fault then?' \u00a0Suffice to say we didn't hire him, but it's a good example of customer unfriendly bias towards what people use.<\/p>\n What happens at most companies is that there is an assumption that is never put to the test. \u00a0'We test all the popular browsers on our website'.\u00a0 For many teams, this testing work is left to the developer,s yet they are often without the key knowledge to make the right call on what should be tested.<\/em><\/p><\/blockquote>\n In a future\u00a0article with Smart Insights, I'll be showing you (at least in Google Analytics) how to draw up a testing list which mirrors real customer devices and browsers.\u00a0 For now, let's assume somebody has a list at your company.\u00a0 Who drew this up?\u00a0 When was it last updated?\u00a0 Has it ever been updated?\u00a0 How much of your customer base does your testing actually reach?\u00a0 Who is responsible for this whole thing?\u00a0 Sadly this is often something that drops through the cracks in an organisation, at least in terms of someone taking clear ownership and responsibility.<\/p>\n To be blunt, because this is costing you money (all of you and yes, you, reading this too).\u00a0 And why can I make this claim?\u00a0 Well let's look at some anecdotal evidence - in the last week, I've seen the following:<\/p>\n These are pretty big issues aren't they?\u00a0<\/strong> At least one of these hits a big chunk of customers and simply prevents them from giving you money!\u00a0 And so what, precisely, has this got to do with AB testing?<\/p>\n Well - it's simply down to the fact that if you have bugs like these on your site, they're probably biasing or breaking your AB testing too!<\/p>\n One company I worked with got 1.5% of their revenue each month from Internet Explorer 8.\u00a0 They couldn't be bothered to do the testing or fixing I advised until I quantified the loss.<\/p>\n I explained that if they could make Internet Explorer 8 convert to checkout at anywhere near the rate of Internet Explorer 9, 10 or 11, this would rise to nearly 6% share of revenue.\u00a0 For two days of work, 8 million pounds per year was the payback.<\/em><\/p><\/blockquote>\n Well - if you were working in a retail store and you could see people struggling to get through a doorway, you'd do something about it.\u00a0 Let's assume you've got a bug on your site though that hits Safari users during checkout. This is dampening your sales but you haven't noticed it because customers haven't complained or your call centre hasn't passed this on.\u00a0 It simply becomes part of the background average conversion on your site - because unlike the retail store owner - you might not be seeing the problem.<\/p>\n Clearly if you have these holes they're a massive opportunity if you can find them and fix them.\u00a0 However, these holes aren't necessarily making you change your digital strategy in major ways - they're just you punching under your weight.<\/p>\n When you start doing AB testing though, these browser and device issues become magnified.<\/p>\n So unlike a sitewide browser or device compatibility issue, let's assume you're now running an AB test between two checkout systems you're trying out.\u00a0 You put the test together, run some basic checking in your company standard browser and then start it going.<\/p>\n In the original checkout, it was just a really long form on one page, which was putting customers off.\u00a0 You decided to test a new checkout, which nicely chunks each step into a separate page.\u00a0 Your developers put this together with some really clever JavaScript and it looks slick.\u00a0 You're confident the new design will win but it fails badly.\u00a0 What happened?<\/p>\n During the test, the new version didn't work and you all scratched your heads. \u00a0'The new design should have won!' 'I can't believe the new one isn't beating the old one' and 'Well that's what the testing tells us - so we have to go with that'.<\/p>\n You then decide as a company to stop building the new design because, after all, the testing told you it didn't work.\u00a0 But what if you were wrong?\u00a0 What if something was broken in the test which biased the result?<\/p>\n I work with some clever developers and product people so I get to see a lot of AB tests being assembled and checked.\u00a0 Roughly 40% of all the builds I've looked at in the last two years have had one or more bugs in particular browsers.<\/p>\n Now that's a fairly high failure rate and I like to think we're pretty experienced at doing this testing lark, so that convinced me I should always be doing QA on every experiment, without fail.<\/p>\n It simply means that if you aren't checking your AB tests work in all the devices (tablets, mobile, desktop, laptop) and browsers (Chrome, Internet Explorer, Firefox, Safari) that your customers use, your test is probably a pile of dog poo.<\/em><\/p><\/blockquote>\n And it's not just the failure of the test that hits you.\u00a0 You get limited takeoff slots for tests to run so making efficient use of every test is important.\u00a0 Having a broken test is going to slow you right down and give you the wrong conclusion.<\/p>\n This is the biggest problem though - when you reach the wrong conclusion, you'll change your business strategy around that insight, even if it was completely wrong.\u00a0 And that's a risk you shouldn't be taking.<\/p>\n There are three things you can do to<\/p>\n This is the subject of a longer article to come but in summary, you need to draw up a list of mobile devices, tablets and desktop\/laptop browsers for your testing.\u00a0 You may not be able to test everything that your customers use but you should be hitting 80+% of the devices and browsers that hit your site.<\/p>\n For testing tablet and mobile devices, you should have a list of the key handsets (and versions) that are arriving at your site.\u00a0 Most of these will be using the pre-installed browser they came with although you should always check the figures.<\/p>\n You have five choices when it comes to testing mobile and tablets - real devices, cloud devices, a device lab, Friends and Family or simulators.<\/em><\/p><\/blockquote>\n This is the best option.\u00a0 If you can buy the mobile handsets or tablets that represent large chunks of your customer base, you can run tests in the office.\u00a0 You can source unlocked handsets pretty cheaply on Amazon if you want to build your own device lab internally.<\/p>\n This is the next best option.\u00a0 Appthwack <\/a>and Deviceanywhere<\/a>\u00a0both offer a unique service.\u00a0 They install phones in a server cabinet and connect their screens and keyboards remotely to any location in the world.\u00a0 You can basically 'remote control' a real phone and see what happens on the screen.<\/p>\n With these services, you can install apps, send text messages, browse the internet or test your AB test - using a real device on a real network.\u00a0 It's the closest thing to having a mobile or tablet device in your hand.<\/em><\/p><\/blockquote>\n These are located all round the world - generous people and companies donate their handsets so that anyone can book a device lab for testing or exploration.\u00a0 You call up, book a time and then go use their devices for testing.\u00a0 It costs nothing and is a good way to find a heap of devices in one place<\/p>\n If you're stuck, you can often find friends, family or colleagues who have handsets or tablets that represent big chunks of your customer base.\u00a0 If you can't find all the handsets you need then testing something is better than nothing.<\/p>\n I've used simulators with developers for testing before and it simply doesn't work.\u00a0 When you're testing a site with lots of JavaScript, you really need the real device as you don't hit the same problems in a simulator.\u00a0 It's fine using one of these to see how a page might render and that might be reasonably successful.\u00a0 However, it's not useful to check all the interactions on a page and expect these to work flawlessly in the real world.<\/p>\n There are some excellent services for testing desktop browsers (like Internet Explorer, Safari, Firefox, and Google Chrome) and they're very reasonable:<\/p>\n If you don't test your site or your AB tests (before they go live) using real devices or tools - then you're going to break something.\u00a0 At the very least, your testing or analytics insights will be skewed and quite possibly, you'll make entirely the WRONG business or strategy decision from following a broken test result.<\/p>\n It's also a huge money opportunity - to make your site work for customer stuff - but when it comes to split testing, it could harm the work of the entire team and make people lose trust in your testing programme.<\/p>\n If you want to get insightful AB tests or discover easy ways to squeeze extra money out of your site, then start doing device and browser testing.<\/p>\n The first step you might take today is to look at the conversion rate for your top browsers and devices coming to your site.\u00a0 You might start asking questions about why there are such big differences and you might find a hole in your site or AB test too.<\/p>\n Hopefully this article has made you curious to find out if you're actually testing real stuff or just your developers\u2019 ability to break things for customers.<\/p>\n Craig will explain more ways to avoid these mistakes at the\u00a0Smart Insights conference on Wednesday 17th September<\/a>.<\/em><\/p>\n <\/p>\n Learning to avoid Dumb\u00a0AB testing mistakes Back in 2005, I started my first forays into split testing of emails, landing pages and funnels whilst working at LOVEFiLM.\u00a0 I’m nearly at my 10 year anniversary, and after seeing over 40 million …..<\/p>\n","protected":false},"author":96,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_glsr_average":0,"_glsr_ranking":0,"_glsr_reviews":0,"footnotes":""},"categories":[379],"tags":[],"acf":[],"yoast_head":"\nAB testing your developers instead of your customers<\/h3>\n
Why should I care about AB testing?<\/h3>\n
\n
Why does it break things so badly?<\/h3>\n
What happens during testing?<\/h3>\n
So how do I fix this after AB testing?<\/h4>\n
\n
Making your test list<\/h3>\n
Testing mobile and tablet devices<\/h4>\n
Real devices<\/h4>\n
Cloud devices<\/h4>\n
Open device lab<\/h4>\n
Friends and Family<\/h4>\n
Simulators<\/h4>\n
Testing desktop browsers<\/h3>\n
\n
Why am I bothering to test this stuff?<\/h3>\n
Key Takeways on AB Testing<\/h3>\n
![\"\"](\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2014\/06\/digitalimpact.png\" "\\"Post")
<\/a><\/p>\n![\"\"](\"http:\/\/www.linkedin.com\/mpr\/pub\/image-4MS9vkWJO0EPNZkQrjUkerg0r3zoWrwzuMUPj8hJrgz1Wm1I4MSP5NJJrW9iddTuwX6X\/craig-sullivan.jpg\")
Thanks to Craig Sullivan<\/a> for sharing their advice and opinions in this post. Craig loves analytics, split testing and running large scale experiments that transform business strategy or performance. You can follow him on Twitter<\/a> or connect on LinkedIn<\/a>.<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"