{"id":78613,"date":"2016-10-19T11:00:00","date_gmt":"2016-10-19T10:00:00","guid":{"rendered":"http:\/\/www.smartinsights.com\/?p=78613"},"modified":"2016-10-19T08:55:59","modified_gmt":"2016-10-19T07:55:59","slug":"developing-test-learn-programme","status":"publish","type":"post","link":"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/","title":{"rendered":"Developing a test and learn programme"},"content":{"rendered":"<h2>How to utilise continuous improvements and \u2018big bets\u2019 to achieve operational and strategic excellence<\/h2>\n<p>As part of any digital transformation agenda, the ability the test and learn gives businesses the opportunity to experiment, iterate and grow skills and competences throughout the organisation. For digital marketers, this opportunity is becoming increasingly important as technology continues to advance and larger companies face new and emerging threats from more nimble, innovative competitors.<\/p>\n<p>To drive meaningful digital transformation at scale, businesses must therefore be open to the adoption of a test and learn culture, which will enable marketers to optimise digital media activation, create first-class digital experiences and develop learning across the organisation. Much of this will be dependent on each organisation\u2019s stage in the digital transformation journey:<\/p>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78615\" rel=\"attachment wp-att-78615\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78615\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/digital-transformation-roadmap.png\" alt=\"digital-transformation-roadmap\" width=\"942\" height=\"524\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/digital-transformation-roadmap.png 942w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/digital-transformation-roadmap-150x83.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/digital-transformation-roadmap-550x306.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/digital-transformation-roadmap-768x427.png 768w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/digital-transformation-roadmap-700x389.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/digital-transformation-roadmap-250x139.png 250w\" sizes=\"(max-width: 942px) 100vw, 942px\" \/><\/a><\/p>\n<h3>Three big digital trends<\/h3>\n<p>The importance of developing a test and learn culture is reflected in the numerous <a href=\"http:\/\/www.smartinsights.com\/managing-digital-marketing\/marketing-innovation\/marketing-trends-2016\/\">digital trends<\/a> impacting businesses today. However, I\u2019d like to highlight three that I believe are particularly significant and should influence test and learn planning for digital marketers across all types of businesses:<\/p>\n<h4>'Big data\u2019 has grown up<\/h4>\n<p>The concept of \u2018big data\u2019 has been a familiar theme within the marketing world for at least the last five years, and yet with some predicting that 2016 would be the \u2018year of the customer\u2019 there has been an increased emphasis on <a href=\"https:\/\/www.entrepreneur.com\/article\/254620\">customer-centric marketing<\/a>, meaning data must be used intelligently to drive results.<\/p>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78616\" rel=\"attachment wp-att-78616\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78616\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Big-data.png\" alt=\"big-data\" width=\"743\" height=\"403\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Big-data.png 743w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Big-data-150x81.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Big-data-550x298.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Big-data-700x380.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Big-data-250x136.png 250w\" sizes=\"(max-width: 743px) 100vw, 743px\" \/><\/a><\/p>\n<p>Advanced analytics, better consumer profiles and the right market and customer insights are becoming essential in tying marketing campaigns together to create more integrated experiences.<\/p>\n<h4>Mobile continues to dominate<\/h4>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78617\" rel=\"attachment wp-att-78617\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78617\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Mobile-internet-use-chart.png\" alt=\"mobile-internet-use-chart\" width=\"865\" height=\"409\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Mobile-internet-use-chart.png 865w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Mobile-internet-use-chart-150x71.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Mobile-internet-use-chart-550x260.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Mobile-internet-use-chart-768x363.png 768w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Mobile-internet-use-chart-700x331.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Mobile-internet-use-chart-250x118.png 250w\" sizes=\"(max-width: 865px) 100vw, 865px\" \/><\/a><\/p>\n<p>Source: Benedict Evans, 2016<\/p>\n<p>More than half of our waking time is spent on media and much of that time is now consumed on mobile devices. Mobile is an increasingly ubiquitous presence in our lives (<a href=\"http:\/\/www.dailymail.co.uk\/sciencetech\/article-3226070\/Generation-mobile-zombie-1-10-look-phone-soon-wake-50-check-50-times-day.html\">49% of Millennials check their phones within 5 minutes of waking up<\/a> in the morning!) \u2013 we\u2019re now in a truly \u2018mobile-first\u2019 world.<\/p>\n<p>As with big data, mobile advertising has come of age:<\/p>\n<ul>\n<li>Mobile accounts for over a half of ecommerce traffic and a third of sales<\/li>\n<li>More than half of Facebook\u2019s base is mobile-only<\/li>\n<li>App usage (90% of time) dominates browsers in mobile usage<\/li>\n<\/ul>\n<p>As Benedict Evans from venture capital firm Andreessen Horowitz demonstrated last year, \u2018mobile\u2019 no longer means mobile - <a href=\"http:\/\/ben-evans.com\/benedictevans\/2016\/3\/29\/presentation-mobile-ate-the-world\">consumers regularly consume media on mobile devices more within the home than outside<\/a>, highlighting the importance of developing a first-class mobile customer experience.<\/p>\n<h4>'Content shock\u2019 is real<\/h4>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78618\" rel=\"attachment wp-att-78618\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78618\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/The-economics-of-content.png\" alt=\"the-economics-of-content\" width=\"694\" height=\"562\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/The-economics-of-content.png 694w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/The-economics-of-content-150x121.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/The-economics-of-content-550x445.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/The-economics-of-content-250x202.png 250w\" sizes=\"(max-width: 694px) 100vw, 694px\" \/><\/a><\/p>\n<p>Source: business2community<\/p>\n<p>\u2018<a href=\"http:\/\/www.businessesgrow.com\/2014\/01\/06\/content-shock\/\">Content shock<\/a>\u2019 (coined by Mark Schaefer in 2014), is the result of consumers rejecting brand content due to exponentially increasing volumes intersecting our limited consumption capacity. Basically, people are fed up of poor quality content and are finding ways to filter this out so they can consume what really matters to them.<\/p>\n<p>We can see examples of content shock in Facebook\u2019s decision to dramatically reduce organic reach and the continuing rise of adblocking across all device types:<\/p>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78620\" rel=\"attachment wp-att-78620\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78620\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Global-Adblocking-rate-2009-2015.png\" alt=\"global-adblocking-rate-2009-2015\" width=\"735\" height=\"417\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Global-Adblocking-rate-2009-2015.png 735w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Global-Adblocking-rate-2009-2015-150x85.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Global-Adblocking-rate-2009-2015-550x312.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Global-Adblocking-rate-2009-2015-700x397.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Global-Adblocking-rate-2009-2015-250x142.png 250w\" sizes=\"(max-width: 735px) 100vw, 735px\" \/><\/a><\/p>\n<p>Source: KPCB, 2016<\/p>\n<p>The average person is exposed to over 400 messages a day. Attention is at a premium and brands must respond accordingly by producing genuinely high quality, relevant content to earn this increasingly precious commodity.<\/p>\n<h3>Introducing a test and learn approach<\/h3>\n<p>Every test and learn programme will differ based on each business\u2019s appetite to testing and their use of digital marketing in general. However one approach I\u2019m currently adopting takes into account two aspects:<\/p>\n<ol>\n<li><strong>Continuous improvements<\/strong> \u2013 running trials to optimise day-to-day activations)<\/li>\n<li><strong>'Big bets'<\/strong> \u2013 to gain operational lessons and strategic insight to apply across the business<\/li>\n<\/ol>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78621\" rel=\"attachment wp-att-78621\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78621\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Strategy-Pyramid-big-bets-continuous-optimisiation.png\" alt=\"strategy-pyramid-big-bets-continuous-optimisiation\" width=\"795\" height=\"466\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Strategy-Pyramid-big-bets-continuous-optimisiation.png 795w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Strategy-Pyramid-big-bets-continuous-optimisiation-150x88.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Strategy-Pyramid-big-bets-continuous-optimisiation-550x322.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Strategy-Pyramid-big-bets-continuous-optimisiation-768x450.png 768w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Strategy-Pyramid-big-bets-continuous-optimisiation-700x410.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Strategy-Pyramid-big-bets-continuous-optimisiation-250x147.png 250w\" sizes=\"(max-width: 795px) 100vw, 795px\" \/><\/a><\/p>\n<p>According to Rosabeth Moss Kanter writing in HBR in 2006, successful innovators use an \u2018innovation pyramid\u2019, with several big bets at the top that get most of the investment; a portfolio of promising mid-range ideas in test stage; and a broad base of early stage ideas or incremental innovations. This concept allows ideas and influence to flow up or down the pyramid.<\/p>\n<p>For the purpose of this post, I thought I\u2019d bring this approach to life by providing examples of two areas of continuous improvements and one \u2018big bet\u2019 to demonstrate:<\/p>\n<ol>\n<li>a) how these areas relate to the digital trends highlighted earlier in the post<\/li>\n<li>b) how these areas could fit into a digital test and learn programme<\/li>\n<\/ol>\n<p>The following examples are just a sample of many others that could be used and I\u2019ve tried to keep these fairly broad so they can be applied to different businesses, small and large, across B2C and B2B.<\/p>\n<h3>Two areas of continuous improvement<\/h3>\n<h4>1. Mobile optimisation<\/h4>\n<p>Although 2016 cannot be regarded as \u2018the year of mobile\u2019 (that came and went at least two years ago!), mobile strategies are maturing and it is essential that brands take a \u2018mobile-first\u2019 approach to meet consumer expectations. Mobile cannot be a less-than-web experience - it has to be <em>the <\/em>experience.<\/p>\n<p><strong>Take advantage of micro-moments<\/strong><\/p>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78622\" rel=\"attachment wp-att-78622\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78622\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/micro-moments.png\" alt=\"micro-moments\" width=\"865\" height=\"482\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/micro-moments.png 865w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/micro-moments-150x84.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/micro-moments-550x306.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/micro-moments-768x428.png 768w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/micro-moments-700x390.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/micro-moments-250x139.png 250w\" sizes=\"(max-width: 865px) 100vw, 865px\" \/><\/a><\/p>\n<p>Source: Google, 2014<\/p>\n<p>Mobile has fundamentally changed consumer behaviour and as consumers we think, search and buy differently from five years ago. Mobile gives marketers the opportunity to take advantage of what Google refers to as \u2018<a href=\"http:\/\/www.smartinsights.com\/digital-marketing-platforms\/google-marketing\/google-micro-moments\/\">micro-moments<\/a>\u2019, those intent-rich moments where decisions are made and preferences shaped.<\/p>\n<p><strong>Test and learn opportunity:<\/strong><\/p>\n<p>Mobile has fractured the consumer journey into hundreds of real-time, intent-driven micro-moments. Consumers are more loyal to their need rather than a brand so it\u2019s important to test being present at the right moments, e.g.:<\/p>\n<ul>\n<li>Deliver targeted advertising during key events<\/li>\n<li>Prompt potential consumers \u2018in-store\u2019 with targeted promotions<\/li>\n<li>Provide help and guidance when it\u2019s needed, e.g. specific searches on YouTube or Google<\/li>\n<\/ul>\n<p><strong>Reach consumers where they\u2019re spending their time<\/strong><\/p>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78623\" rel=\"attachment wp-att-78623\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78623\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Time-spent-on-media-vs-advertising-spending-.png\" alt=\"time-spent-on-media-vs-advertising-spending\" width=\"728\" height=\"392\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Time-spent-on-media-vs-advertising-spending-.png 728w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Time-spent-on-media-vs-advertising-spending--150x81.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Time-spent-on-media-vs-advertising-spending--550x296.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Time-spent-on-media-vs-advertising-spending--700x377.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Time-spent-on-media-vs-advertising-spending--250x135.png 250w\" sizes=\"(max-width: 728px) 100vw, 728px\" \/><\/a><\/p>\n<p>Source: KPCB, 2016<\/p>\n<p>One of the insights that stood out for me from KPCB\u2019s <a href=\"http:\/\/www.kpcb.com\/internet-trends\">2016 internet trends<\/a> was that advertising currently remains over-indexed to legacy media despite the rapid increase in mobile usage.<\/p>\n<p><strong>Test and learn opportunity:<\/strong><\/p>\n<p>There is an opportunity to increase ad spend in mobile channels to reach consumers where they appear to be spending most of their time. This will not only improve ad performance but potentially provide a competitive advantage over slower-moving rivals who have failed to respond to this trend.<\/p>\n<p><strong>Establish a presence across multiple platforms<\/strong><\/p>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78624\" rel=\"attachment wp-att-78624\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78624\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Apps-over-half-of-internet-use.png\" alt=\"apps-over-half-of-internet-use\" width=\"737\" height=\"344\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Apps-over-half-of-internet-use.png 737w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Apps-over-half-of-internet-use-150x70.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Apps-over-half-of-internet-use-550x257.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Apps-over-half-of-internet-use-700x327.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Apps-over-half-of-internet-use-250x117.png 250w\" sizes=\"(max-width: 737px) 100vw, 737px\" \/><\/a><\/p>\n<p>Source: Benedict Evans, 2016<\/p>\n<p>Mobile has led to an \u2018unbundling\u2019 of the web. We now consume content across browsers and apps, although the trend is moving more and more towards native apps which could signal the death of the hyperlink.<\/p>\n<p>A mobile-first approach means that mobile must be the ultimate experience, with the web becoming merely an add-on (a complete 180\u00b0 shift from where this was before).<\/p>\n<p><strong>Test and learn opportunity:<\/strong><\/p>\n<p>Establish a presence across multiple platforms, including responsive\/ adaptive design that works across desktop and mobile, as well as native mobile apps. How is your audience consuming mobile content? What channels and platforms work for them? Where are the optimisation opportunities? Mobile apps don\u2019t work for everyone but if there is an opportunity to test without too much risk it may be worth looking into.<\/p>\n<h4>2. Measurement and analytics<\/h4>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78625\" rel=\"attachment wp-att-78625\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78625\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart.png\" alt=\"obejctive-goal-kpi-marketing-chart\" width=\"727\" height=\"432\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart.png 727w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart-150x89.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart-550x327.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart-700x416.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart-250x149.png 250w\" sizes=\"(max-width: 727px) 100vw, 727px\" \/><\/a><\/p>\n<p>Effective analysis and insight should underpin everything we work on as digital marketers. Without a thorough understanding of what is and isn\u2019t performing, you will not have the right actionable insights to make correct decisions.<\/p>\n<p><strong>Establish a measurement framework<\/strong><\/p>\n<p>A measurement framework\/ model is a way to structure your thinking, prioritise goals and organise the KPIs and metrics you\u2019ll use to measure performance.<\/p>\n<p><a href=\"http:\/\/www.kaushik.net\/avinash\/digital-marketing-and-measurement-model\/\">Avinash Kaushik<\/a>\u00a0is a leading thought leader on this subject and as he explains:<\/p>\n<blockquote><p><em>\u201cThe root cause of failure in most digital marketing campaigns is not the lack of creativity in the banner ad or TV spot or the sexiness of the website. It is not even (often) the people involved. It is quite simply the lack of structured thinking about what the real purpose of the campaign is and a lack of an objective set of measures with which to identify success or failure\u201d.<\/em><\/p><\/blockquote>\n<p><strong>Test and learn opportunity:<\/strong><\/p>\n<p>Before jumping into a digital campaign or project, consider creating a measurement framework to structure how all of the following work together:<\/p>\n<ul>\n<li>Business objectives<\/li>\n<li>Macro and micro goals<\/li>\n<li>Key performance indicators (KPIs)<\/li>\n<li>Targets<\/li>\n<li>Segments<\/li>\n<\/ul>\n<p>The example above shows how goals, KPIs and segments are flow from the high-level business objectives that have been set out. The key is to understand how this framework might apply to your business and test a similar approach.<\/p>\n<p><strong>Run regular A\/B testing to improve relevance<\/strong><\/p>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78626\" rel=\"attachment wp-att-78626\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-78626\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/AB-testing.png\" alt=\"ab-testing\" width=\"605\" height=\"261\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/AB-testing.png 743w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/AB-testing-150x65.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/AB-testing-550x237.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/AB-testing-700x301.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/AB-testing-250x108.png 250w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/p>\n<p>A\/B or multivariate testing should form a key part of your ongoing test and learn programme. By experimenting with different types of content versus a current experience across web pages, social channels and\/ or apps, you\u2019ll be able to more accurately determine which variant performs better for a given conversion or goal.<\/p>\n<p>Using the data from A\/B testing removes guesswork and subjectivity, giving you the confidence to make more informed, data-driven decisions.<\/p>\n<p><strong>Test and learn opportunity:<\/strong><\/p>\n<p>A\/B testing doesn\u2019t have to be a complex or expensive process. Start small and experiment using particular pieces of content that have generated discussion or debate internally.<\/p>\n<p>If you\u2019re testing something more experimental or risky, take \u2018controlled risks\u2019 by showing the new content to only a small proportion of the audience (e.g. 10% of traffic). As you build an understanding and confidence in your hypothesis you can begin to increase the scope of your testing \u2018landscape\u2019.<\/p>\n<h3>One big bet<\/h3>\n<h4>Dynamic content and personalisation<\/h4>\n<p>\u2018Big bets\u2019 can be anything that has the potential to enhance or optimise the organisation across multiple levels. Unlike the more operational continuous improvements highlighted above, \u2018big bets\u2019 often require more planning and investment, but if successful have the potential to future-proof a business.<\/p>\n<p>For this post I\u2019ve chosen to look at dynamic content and personlisation as an example of a \u2018big bet\u2019. This is closely linked to all three of the trends highlighted at the start of the post (data, mobile and \u2018content shock\u2019) and as I\u2019ve tried to do throughout the post, this idea can apply to large and small businesses alike.<\/p>\n<p><strong>Optimise content to drive action<\/strong><\/p>\n<p>The essence of personalisation is about using content that is most relevant to the audience in order to generate higher engagement and conversion. One method of doing this is to use dynamic content, essentially showing the same web page to two people but serving different content within that page based on what we know about them:<\/p>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78627\" rel=\"attachment wp-att-78627\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78627\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Iterative-Platform-Testing.png\" alt=\"iterative-platform-testing\" width=\"704\" height=\"443\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Iterative-Platform-Testing.png 704w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Iterative-Platform-Testing-150x94.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Iterative-Platform-Testing-550x346.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Iterative-Platform-Testing-700x440.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Iterative-Platform-Testing-250x157.png 250w\" sizes=\"(max-width: 704px) 100vw, 704px\" \/><\/a><\/p>\n<p><strong>Test and learn opportunity:<\/strong><\/p>\n<p>First and third-party data can be used to create more relevant and compelling experiences, and iterative platform testing can be used with or alongside A\/B testing tools to learn about what is\/ isn\u2019t working. Take the time to find out what type and quality of data you have and run small tests initially to optimise content for different audience groups.<\/p>\n<p><strong>Create personal video and TV experiences<\/strong><\/p>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78628\" rel=\"attachment wp-att-78628\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78628\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/addressable-advertising-viewer-engagment-.png\" alt=\"addressable-advertising-viewer-engagment\" width=\"865\" height=\"375\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/addressable-advertising-viewer-engagment-.png 865w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/addressable-advertising-viewer-engagment--150x65.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/addressable-advertising-viewer-engagment--550x238.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/addressable-advertising-viewer-engagment--768x333.png 768w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/addressable-advertising-viewer-engagment--700x303.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/addressable-advertising-viewer-engagment--250x108.png 250w\" sizes=\"(max-width: 865px) 100vw, 865px\" \/><\/a><\/p>\n<p>Source: Google\/ DoubleClick, 2016<\/p>\n<p>The way people consume content has fundamentally changed. Whilst ten years ago many of us would watch a show live on one television in a living room, the use of many different devices and streaming services such as Netflix mean that we now watch TV shows across a variety of media and it\u2019s therefore become increasingly difficult to reach audiences with traditional methods of advertising.<\/p>\n<p>However, this challenge has given brands a new opportunity: to develop personalised advertising strategies.<\/p>\n<p>The concept of addressable TV advertising enables brands to reach fragmented, time-shifted audiences and to show different ads to different households who are all watching the same programme.<\/p>\n<p><strong>Test and learn opportunity:<\/strong><\/p>\n<p>Whilst TV is not a marketing channel open to everyone, and the nascent technology involved in addressable advertising has a number of challenges (including effective measurement and achieving reliability at scale), I wanted to include this example to demonstrate how advertising can be smarter through the use of data.<\/p>\n<p>The emergence of addressable advertising may even give smaller businesses the opportunity to experiment with TV advertising in a way that they couldn\u2019t before. Businesses no longer have to book media to reach millions of people. Instead spots can be bought to target specific groups of people watching certain types of programming at certain times of the day.<\/p>\n<p><strong>Deliver tailored advertising with programmatic<\/strong><\/p>\n<p><a href=\"http:\/\/www.smartinsights.com\/?attachment_id=78629\" rel=\"attachment wp-att-78629\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-78629\" src=\"http:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Programmatic-advertising.png\" alt=\"programmatic-advertising\" width=\"725\" height=\"538\" srcset=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Programmatic-advertising.png 725w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Programmatic-advertising-150x111.png 150w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Programmatic-advertising-550x408.png 550w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Programmatic-advertising-700x519.png 700w, https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/Programmatic-advertising-250x186.png 250w\" sizes=\"(max-width: 725px) 100vw, 725px\" \/><\/a><\/p>\n<p>Source: Google, 2016<\/p>\n<p>A core goal for marketers should be about producing content and communications that matter to the audience. Programmatic marketing is something we\u2019ve <a href=\"http:\/\/www.smartinsights.com\/internet-advertising\/internet-advertising-targeting\/programmatic-marketing-demystified\/\">covered previously<\/a>, but in a nutshell it\u2019s about enabling brands to be responsive to their audience in real-time, with highly relevant messaging and creativity. The objective is to tailor messages to the right person, at the right moment, in the right context.<\/p>\n<p><strong>Test and learn opportunity:<\/strong><\/p>\n<p>Depending on device, location and weather, content can be delivered programmatically to different audiences and this is something that can be tested across different campaigns.<\/p>\n<p>Google offers an excellent guide to getting started with <a href=\"https:\/\/www.thinkwithgoogle.com\/programmatic-guide\/\">programmatic marketing<\/a>, with a useful checklist of key steps:<\/p>\n<ol>\n<li>Organise audience insights<\/li>\n<li>Design compelling creative<\/li>\n<li>Execute with integrated technology<\/li>\n<li>Reach audiences across screens<\/li>\n<li>Measure the impact<\/li>\n<\/ol>\n<h3>Summary<\/h3>\n<p>With so many major trends beginning to take shape and impact the work we do as marketers, we must look to adapt and find the best ways to take advantage of these changes.<\/p>\n<p>Whilst it would be both unwise and costly to carry out tests for every trend that arises, an effective test and learn programme can enable us to carry out tactical and strategic experiments to build learnings and help us understand what works for our business.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>How to utilise continuous improvements and \u2018big bets\u2019 to achieve operational and strategic excellence As part of any digital transformation agenda, the ability the test and learn gives businesses the opportunity to experiment, iterate and grow skills and competences throughout &hellip;..<\/p>\n","protected":false},"author":12923,"featured_media":78625,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_glsr_average":0,"_glsr_ranking":0,"_glsr_reviews":0,"footnotes":""},"categories":[379],"tags":[],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.8 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Developing a test and learn programme | Smart Insights<\/title>\n<meta name=\"description\" content=\"Developing a test and learn programme How to utilise continuous improvements and \u2018big bets\u2019 to achieve operational and strategic excellence As part of any digital transformation agenda, the.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Developing a test and learn programme | Smart Insights\" \/>\n<meta property=\"og:description\" content=\"Developing a test and learn programme How to utilise continuous improvements and \u2018big bets\u2019 to achieve operational and strategic excellence As part of any digital transformation agenda, the.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/\" \/>\n<meta property=\"og:site_name\" content=\"Smart Insights\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/smartinsights\" \/>\n<meta property=\"article:published_time\" content=\"2016-10-19T10:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2016-10-19T07:55:59+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart.png\" \/>\n\t<meta property=\"og:image:width\" content=\"727\" \/>\n\t<meta property=\"og:image:height\" content=\"432\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Gavin Llewellyn\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@smartinsights\" \/>\n<meta name=\"twitter:site\" content=\"@smartinsights\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Gavin Llewellyn\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/\",\"url\":\"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/\",\"name\":\"Developing a test and learn programme | Smart Insights\",\"isPartOf\":{\"@id\":\"https:\/\/www.smartinsights.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart.png\",\"datePublished\":\"2016-10-19T10:00:00+00:00\",\"dateModified\":\"2016-10-19T07:55:59+00:00\",\"author\":{\"@id\":\"https:\/\/www.smartinsights.com\/#\/schema\/person\/ec72ac3813aaa3acc8c07a89022070cc\"},\"description\":\"Developing a test and learn programme How to utilise continuous improvements and \u2018big bets\u2019 to achieve operational and strategic excellence As part of any digital transformation agenda, the.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/#primaryimage\",\"url\":\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart.png\",\"contentUrl\":\"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart.png\",\"width\":727,\"height\":432},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.smartinsights.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Developing a test and learn programme\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.smartinsights.com\/#website\",\"url\":\"https:\/\/www.smartinsights.com\/\",\"name\":\"Smart Insights\",\"description\":\"Digital Marketing &gt; The Marketing Strategy Blog\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.smartinsights.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.smartinsights.com\/#\/schema\/person\/ec72ac3813aaa3acc8c07a89022070cc\",\"name\":\"Gavin Llewellyn\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.smartinsights.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d183a179b2b715b48b441ad2e456d701?s=96&d=identicon&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d183a179b2b715b48b441ad2e456d701?s=96&d=identicon&r=g\",\"caption\":\"Gavin Llewellyn\"},\"description\":\"Gavin Llewellyn (LinkedIn) is an independent consultant. He is a Chartered Marketer who specialises in digital marketing, specifically in social media, SEO and online strategy. Gavin blogs at One Too Many Mornings where he offers advice, guidance and ideas on how individuals and companies can use digital marketing effectively to get found online, build engagement and generate conversion. You can Follow Gavin on Twitter.\",\"url\":\"https:\/\/www.smartinsights.com\/author\/gavinllewellyn\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Developing a test and learn programme | Smart Insights","description":"Developing a test and learn programme How to utilise continuous improvements and \u2018big bets\u2019 to achieve operational and strategic excellence As part of any digital transformation agenda, the.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/","og_locale":"en_US","og_type":"article","og_title":"Developing a test and learn programme | Smart Insights","og_description":"Developing a test and learn programme How to utilise continuous improvements and \u2018big bets\u2019 to achieve operational and strategic excellence As part of any digital transformation agenda, the.","og_url":"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/","og_site_name":"Smart Insights","article_publisher":"https:\/\/www.facebook.com\/smartinsights","article_published_time":"2016-10-19T10:00:00+00:00","article_modified_time":"2016-10-19T07:55:59+00:00","og_image":[{"width":727,"height":432,"url":"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart.png","type":"image\/png"}],"author":"Gavin Llewellyn","twitter_card":"summary_large_image","twitter_creator":"@smartinsights","twitter_site":"@smartinsights","twitter_misc":{"Written by":"Gavin Llewellyn","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/","url":"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/","name":"Developing a test and learn programme | Smart Insights","isPartOf":{"@id":"https:\/\/www.smartinsights.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/#primaryimage"},"image":{"@id":"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/#primaryimage"},"thumbnailUrl":"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart.png","datePublished":"2016-10-19T10:00:00+00:00","dateModified":"2016-10-19T07:55:59+00:00","author":{"@id":"https:\/\/www.smartinsights.com\/#\/schema\/person\/ec72ac3813aaa3acc8c07a89022070cc"},"description":"Developing a test and learn programme How to utilise continuous improvements and \u2018big bets\u2019 to achieve operational and strategic excellence As part of any digital transformation agenda, the.","breadcrumb":{"@id":"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/#primaryimage","url":"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart.png","contentUrl":"https:\/\/www.smartinsights.com\/wp-content\/uploads\/2016\/10\/obejctive-goal-KPI-marketing-chart.png","width":727,"height":432},{"@type":"BreadcrumbList","@id":"https:\/\/www.smartinsights.com\/conversion-optimisation\/ab-multivariate-testing\/developing-test-learn-programme\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.smartinsights.com\/"},{"@type":"ListItem","position":2,"name":"Developing a test and learn programme"}]},{"@type":"WebSite","@id":"https:\/\/www.smartinsights.com\/#website","url":"https:\/\/www.smartinsights.com\/","name":"Smart Insights","description":"Digital Marketing &gt; The Marketing Strategy Blog","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.smartinsights.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.smartinsights.com\/#\/schema\/person\/ec72ac3813aaa3acc8c07a89022070cc","name":"Gavin Llewellyn","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.smartinsights.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d183a179b2b715b48b441ad2e456d701?s=96&d=identicon&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d183a179b2b715b48b441ad2e456d701?s=96&d=identicon&r=g","caption":"Gavin Llewellyn"},"description":"Gavin Llewellyn (LinkedIn) is an independent consultant. He is a Chartered Marketer who specialises in digital marketing, specifically in social media, SEO and online strategy. Gavin blogs at One Too Many Mornings where he offers advice, guidance and ideas on how individuals and companies can use digital marketing effectively to get found online, build engagement and generate conversion. You can Follow Gavin on Twitter.","url":"https:\/\/www.smartinsights.com\/author\/gavinllewellyn\/"}]}},"_links":{"self":[{"href":"https:\/\/www.smartinsights.com\/wp-json\/wp\/v2\/posts\/78613"}],"collection":[{"href":"https:\/\/www.smartinsights.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.smartinsights.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.smartinsights.com\/wp-json\/wp\/v2\/users\/12923"}],"replies":[{"embeddable":true,"href":"https:\/\/www.smartinsights.com\/wp-json\/wp\/v2\/comments?post=78613"}],"version-history":[{"count":3,"href":"https:\/\/www.smartinsights.com\/wp-json\/wp\/v2\/posts\/78613\/revisions"}],"predecessor-version":[{"id":78631,"href":"https:\/\/www.smartinsights.com\/wp-json\/wp\/v2\/posts\/78613\/revisions\/78631"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.smartinsights.com\/wp-json\/wp\/v2\/media\/78625"}],"wp:attachment":[{"href":"https:\/\/www.smartinsights.com\/wp-json\/wp\/v2\/media?parent=78613"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.smartinsights.com\/wp-json\/wp\/v2\/categories?post=78613"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.smartinsights.com\/wp-json\/wp\/v2\/tags?post=78613"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}