4 common pitfalls of running conversion rate experiments from Microsoft

At a previous Measurefest conference, one of the speakers, Craig Sullivan, recommended a classic research paper from Microsoft on common pitfalls in running conversion rate experiments. It details five surprising results which took 'multiple-person weeks to properly analyse’ at Microsoft and published for the benefit of all. As the authors point out, this stuff is worth spending a few weeks getting right as ‘multi-million-pound business decisions’ rest on the outcomes. This research ultimately points out the importance of doing A/A Testing. Here follows an executive overview, cutting out some of the technical analysis: 1. Beware of conflicting short-term metrics Bing’s management had two high-level goals: query share and revenue per search. The problem is that it is possible to increase both those and yet create a bad long-term company outcome, by making the search algorithm worse. If you force users to make more searches (increasing Bing’s share of queries), because they can’t find an answer, they will click on more adverts as well. “If the goal of a search engine is to allow users to find their answer or complete their task quickly, then reducing the distinct queries per task is a clear goal, which conflicts with the business objective of increasing share.” The authors suggest a better metric in most cases is lifetime customer value, and the executives should try to understand where shorter-term metrics might conflict with that long-term goal 2. Beware of technical reasons for experiment results The Hotmail link on the MSN home page was changed to open Hotmail in a separate tab/window. The naïve experiment results showed that users clicked more on the Hotmail link when it opened in a new window, but the majority of the observed effect was artificial. Many browsers kill the previous page’s tracking Javascript when a new page loads – with Safari blocking the tracking script in 50% of pages opening in the same window. The “success” of getting users to click more was not real, but rather an instrumentation difference. So it wasn’t that more people were clicking on the link – but actually that just more of the links were being tracked in the ‘open in new tab’ experiment. 3. Beware of peeking at results too early When we release a new feature as an experiment, it is really tempting to peek at the results after a couple of days and see if the test confirms our expectation of success (confirmation bias). With the initial small sample, there will be a big percentage change. Humans then have an innate tendency to see trends where there aren’t any. So the authors give the example of this chart: Most experimenters would see the results, and even though they are negative, extrapolate the graph along the green line to a positive result and four days. Wrong. What actually happens is regression to the mean. This chart is actually from an A/A test (i.e. the two versions being tested are exactly the same). The random differences are biggest at the start, and then tail off - so the long term result will be 0% difference as the sample size increases. The simple advice is to wait until there are enough test results to draw a statistically significant conclusion. That generally means more than a week and hundreds of individual tests. 4. Beware of the carryover effect from previous experiments Many A/B test systems use a bucketing system to assign users into one experiment or another. At the end of one test the same buckets of users may be reused for the second test. The problem is that if users return to your product regularly (multiple times daily in the case of Bing), then a highly positive or negative experience in one of the tests will affect all of that bucket for many weeks. In one Bing experiment, which accidentally introduced a nasty bug, users who saw the buggy version were still making fewer searches 6 months after the experiment ended. Ideally, your test system would re-randomise users for the start of every new test, so those carryover effects are spread as wide as possible. Summary For me the biggest theme coming out of their research is the importance of A/A tests – seeing what kind of variation and results you get if you don’t change anything. Which makes you more aware of the random fluctuations inherent in statistical tests. In conclusion, you need to think about the possible sources of bias before acting on your tests. Even the most experienced analysts make mistakes! Have any comments? Let us know what you think, below!    

2016-11-27

5 tips to avoid a metrics meltdown when upgrading to Universal Analytics

Universal Analytics promises some juicy benefits over the previous standard analytics. But having upgraded 6 different high traffic sites there are some pitfalls to be aware of. Firstly, why would you want to upgrade your tracking script? More reliable tracking of page visitors - i.e. fewer visits untracked More customisation to exclude certain referrers or search terms Better tools for tracking across multiple domains and tracking users across different devices Track usage across your apps for the same web property Ability to send up to 20 custom dimensions instead of the previous limit of only 5 custom variables If you want to avoid any interruption of service when you upgrade, why not book a quick consultation with us to check if Universal Analytics will work in your case. But before you start you should take note of the following. 1. Different tracking = overall visits change If your boss is used to seeing dependable weekly / monthly numbers, they may query why the number of visits has changed. Universal Analytics is likely to track c. 2% more visits than previously (partly due to different referral tracking - see below), but it could be higher depending on your mix of traffic. PRO TIP: Set up a new web property (a different tracking code) for Universal Analytics and run the old and new trackers alongside each other for a month. Then you can see how the reports differ before sharing with managers. Once this testing period is over you'll need to upgrade the original tracking code to Universal Analytics to you keep all your historic data. 2. Different tracking of referrals Previously, if Bob clicked on a link in Twitter to your site, reads, goes back to Twitter, and within 30 minutes clicks on a different link to your site - that would be counted as one visit and the 2nd referral source would be ignored. In Universal Analytics, when Bob clicks on the 2nd link he is tracked as a second visit, and 2nd referral source is stored. This may be more accurate for marketing tracking, but if Bob then buys a product from you, going via a secure payment gateway hosted on another domain (e.g. paypal.com) then the return from the payment gateway will be counted as a new visit. All your payment goals or ecommerce tracking will be attributed to a referral from 'paypal.com'. This will ruin your attribution of a sale to the correct marketing channel or campaign! PRO TIP: You need to add all of the payment gateways (or other third party sites a user may visit during the payment process) to the 'Referral Exclusion List'. You can find this under the Admin > Property > Tracking codes menu: 3. Tracking across domains If you use the same tracking code across different domains (e.g. mysite.co.uk and mysite.com or mysite.de) then you will need to change the standard tracking script slightly. By default the tracking script you copy from Google Analytics contains a line like: ga('create', 'UA-XXXXXXX-1', 'mysite.com');. This will only track pages that strictly end with 'mysite.com'. PRO TIP: It's much safer to change the tracker to set that cookie domain automatically. The equivalent for the site above would be ga('create', 'UA-XXXXXXX-1', 'auto');. The 3rd argument of the function is replaced with 'auto'. 4. Incompatibility with custom variables Only relevant if you send custom data already Custom variables are only supported historically in Universal analytics. That means you will need to change any scripts that send custom data to the new custom dimension format to keep data flowing. Read the developer documentation for more. PRO TIP: You'll need to set the custom dimension names in the admin panel before the custom data can be sent from the pages. You can also only check that the custom dimensions are being sent correctly by creating a new custom report for each dimension. 5. User tracking limitations We wouldn't recommend implementing the new user ID feature just now, as it has some major limitations compared with storing the GA client ID. You need to create a separate view to see the logged-in-user data, which makes reporting pageviews a whole lot more complex. Visits a user made to your site BEFORE signing up are not tracked with that user - which means you can't track the marketing sources by user PRO TIP: See our user tracking alternative. Got more tips on to setting up Universal Analytics? Please share them with us in the comments, or get in touch if you want more advice on how to upgrade!   Get Social! Follow us on LinkedIn, Twitter, and Facebook and keep up-to-date with our Google Analytics insights.

2016-11-26

Widget Tracking with Google Analytics

I was asked recently about the best way to track a widget, loaded in an iframe, on a third-party site with Google Analytics. The difficulty is that many browsers now block 3rd party cookies (those set by a different domain to the one in the browser address bar) – and this applies to a Google Analytics cookie for widgets as much as to adverts. The best solution seems to be to use local storage on the browser (also called HTML5 Storage) to store a persistent identifier for Analytics and bypass the need to set a cookie – but then you have to manually create a clientID to send to Google Analytics. See the approach used by ShootItLive. However, as their comment on line 41 says, this is not a complete solution - because there are lots of browsers beyond Safari which block third party cookies. I would take the opposite approach and check if the browser supports local storage, and only revert to trying to set a cookie if it does not. Local storage is now possible on 90% of browsers in use and the browsers with worst 3rd party cookie support (Firefox and Safari) luckily have the longest support for local storage. As a final note, I would set up the tracking on a different Google Analytics property to your main site, so that pageviews of widgets are not confused with pageviews of your main site. To do list: Build a script to create a valid clientID for each new visitor Call ga('create) function, setting 'storage' : 'none', and getting the 'clientID' from local storage (or created from new) Send a pageview (or event) for every time the widget is loaded. Since the widget page is likely to be the same every time it is embedded, you might want to store the document referrer (the parent page URL) instead Need help with the details? Get in touch with our team of experts and we'd be happy to help!   Get Social! Follow us on LinkedIn, Twitter, and Facebook and keep up-to-date with our Google Analytics insights.

2016-11-25

How to link Adwords and Google Analytics

If you are running an AdWords campaign you must have a Google Analytics account. We will show you how to link these two accounts so you can unleash the full reporting potential of both platforms. 1. Why should you link Analytics and AdWords? When you link Google Analytics and AdWords, you can: See ad and site performance data in the AdWords reports in Google Analytics. Import Google Analytics goals and ecommerce transactions directly into your AdWords account. Import valuable Analytics metrics—such as bounce rate, avg. session duration, and pages/session—into your AdWords account. Take advantage of enhanced remarketing capabilities. Get richer data in the Google Analytics multi-channel funnels reports. Use your Google Analytics data to enhance your AdWords experience. 2. How to link Google Analytics and AdWords The linking wizard makes it easy to link your AdWords account(s) to multiple views of your Google Analytics property. If you have multiple Google Analytics properties and want to link each of them to your AdWords account(s), just complete the linking wizard for each property. Sign into your Google Analytics account at www.google.com/analytics. Note: You can also quickly open Google Analytics from within your AdWords account. Click the tools tab, select analytics, and then follow the rest of these instructions. Click the admin tab at the top of the page. In the account column, select the analytics account that contains the property you want to link to one or more of your AdWords accounts. In the property column, select the analytics property you want to link, and click AdWords Linking. Use one of the following options to select the AdWords accounts you want to link with your analytics property. Select the checkbox next to any AdWords accounts you want to link with your analytics property. If you have an AdWords manager (MCC) account, select the checkbox next to the manager account to link it (and all of its child accounts) with your analytics property. If you want to link only a few managed accounts, expand the manager account by clicking the arrow next to it. Then, select the checkbox next to each of the managed AdWords accounts that you want to link. Or, click all linkable to select all of managed AdWords accounts under that MCC. You can then deselect individual accounts, and the other accounts will stay selected. Click the continue button. In the link configuration section, enter a link group title to identify your group of linked AdWords accounts. Note: Most users will only need one link group. We recommend creating multiple link groups only if you have multiple AdWords accounts and want data to flow in different ways between these accounts and your analytics property. For example, you should create multiple link groups if you need to either link different AdWords accounts to different views of the same Google Analytics property or enable auto-tagging for only some of your AdWords accounts. Select the Google Analytics views in which you want the AdWords data to be available. If you've already enabled auto-tagging in your AdWords account, skip to the next step. The account linking process will enable auto-tagging for all of your linked AdWords accounts. Click advanced settings only if you need to manually tag your AdWords links. Click the link accounts button. Congratulations! Your accounts are now linked. If you opted to keep auto-tagging turned on (recommended), Google Analytics will automatically start associating your AdWords data with customer clicks. For a deeper view and debugging you should also read the Google Analytics guide. Have any questions on setting this up? Get in touch and we'd be happy to help!   Get Social! Follow us on LinkedIn, Twitter, and Facebook and keep up-to-date with our Google Analytics insights.

2016-11-24

Who visited my website? Find out with Google Analytics

In every retail business, knowing your customers is vital to succeeding. All decisions you make about business and marketing strategies must begin from the user's perspective. Let's find out how we can build the user persona with the data that lies in Google Analytics. Even though Google's user profile is not as fancy as Facebook's, you can still have a pretty good idea about your customers. Let's start with the basics, and ask the most basic questions: How many of my customers are men or women? What is the age range of my customers? What devices do they use to access my website? How often do they visit my website? What are their interests? What makes them convert? For the first two questions, you should already have enabled Demographics and Interest reports in your Google account. If not, go to Admin > Property Settings > Enable Demographics and Interest reports. The split of age and sex can be seen in Audience > Demographics. The most interesting thing here is that you can add a second dimension to compare and see how people are different based on more than one vector. If you add a second dimension, such as Device Category, you will get a split like this: You can see from the above screenshot that females prefer mobile and are the majority user. Also when females are on desktop, they are more likely to spend more time on the website. You can go into more depth and analyse the conversion rate also. You can find out the behaviour of new vs. returning customers from the report, New vs. Returning under Audience. Add a second dimension "Gender" and you will see who's more likely to come back to your website. Based on the above screenshot, women are returning about 25% of the time, while men return about 21% of the time. Also, men have a higher bounce rate. Under Audience, you will also find the Frequency & Recency report and the Engagement report. If you create two new segments by gender: female and male, you will find who your most loyal visitors are. The interests (Google reads them from the user behaviour online) can be found under Audience > Interests. It is best to split the report based on females and males. You will now have a full view of your customers. And for the final and most important question: what makes them convert?, you can find this out by going to Aquisition > Channels. Add a second dimension by gender, age or interests and analyse the traffic for each channel. Find out what channel brings the most important users. In the screenshot below, you can see that woman are more likely to buy if the website is referred. This means that the reputation of the website is a big factor in their decision. Don't be shy when creating custom reports because you can drill down the data to multiple levels of understanding. Applying second dimensions has its limitations and you can see only a part of the information at once. If you still need a more detailed view of what each customer does on the website, we strongly recommend the User Explorer menu. We found it useful to find out how different touch points are important and how long the path is for a valuable customer. Also, it was useful in debugging and creating a marketing strategy based on the customer's flow. The bottom line is that you can answer "who is your customer?" with Google Analytics through its reports if you learn to see the report from different perspectives. Feel free to drop us a line if you use any other report that is relevant to this article!   Get Social! Follow us on LinkedIn, Twitter, and Facebook and keep up-to-date with our Google Analytics insights.

2016-11-17

Exclude fake 'bot' traffic from your site with Google Analytics

Ever wondered why so few visitors convert on your site? One answer is that a big chunk of your traffic is from search engine spiders and other web 'bots' which have no interest in actually engaging with you. Google Analytics has a great new feature to exclude this bot traffic from your site. All you need to do is check a box under the Admin > View > View Settings. The new option is down the bottom, underneath currency selection. It uses the IAB /ABC Bots and Spiders list, which is standard for large publishers, and updated monthly. Warning: you will see a dip in traffic from the date you apply the setting. If you're looking for a more comprehensive method to exclude spam and ghost referrals, check out our how-to guide! Have some questions about this? Get in touch with our Google Analytics experts!   Get Social! Follow us on LinkedIn, Twitter, and Facebook and keep up-to-date with our Google Analytics insights.

2016-11-15

What are Enhanced Ecommerce reports?

In May 2014 Google Analytics introduced a new feature: Enhanced Ecommerce tracking. If you run an ecommerce operation, this gets you much more detailed feedback on your checkout process. What will I see? Shopping behaviour: how are people converting from browsers to purchasers? Checkout behaviour: at what stage of your checkout do buyers abandon the process Product performance: which products are driving your sales, and which have a high return rate Real campaign returns: see your real return on marketing investment including promotional discounts and returns How do I set this up? The bad news is it definitely requires an experienced software developer for the setup. The reports require lots of extra product and customer information to be sent to Google Analytics. You can read the full developer information on what you can track, or our own simpler guide for tracking ecommerce via Tag Manager. However, if you already have standard ecommerce tracking and Google Tag Manager, we can set Enhanced reports up in a couple of days with no code changes on your live site - so no business disruption or risk of lost sales. Is it worth implementing? Imagine you could identify a drop-off stage in your checkout process where you could get a 10% improvement in sales conversion or a group of customers who were unable to buy (maybe due to language or browser difficulties) – what would that be worth? Many businesses have that kind of barrier just waiting to be discovered…   Get Social! Follow us on LinkedIn, Twitter, and Facebook and keep up-to-date with our Google Analytics insights.

2016-11-14

How to track time on page with Google Tag Manager

Our script for accurate tracking of time on page beats Google's default measurement to give you an accurate picture of how long users are spending on your page open and in focus. This post translates the approach into Google Tag Manager. The setup consists of two tags (one custom), one firing rule and two variables. Step by step: 1. Add the timer script as a custom HTML tag <script><br /> /*<br /> Logs the time on the page to dataLayer every 10 seconds<br /> (c) LittleData consulting limited 2014<br /> */<br /> (function () {<br /> var inFocus = true;<br /> var intervalSeconds = 10; //10 seconds<br /> var interval = intervalSeconds * 1000;<br /> var eventCount = 0;<br /> var maxEvents = 60; //stops after 10 minutes in focus<br /> var fnBlur = function(){inFocus = false; };<br /> var fnFocus = function(){inFocus= true; };<br /> if (window.addEventListener) {<br /> window.addEventListener ('blur',fnBlur,true);<br /> window.addEventListener ('focus',fnFocus,true);<br /> }<br /> else if (window.attachEvent) {<br /> window.attachEvent ('onblur',fnBlur);<br /> window.attachEvent ('onfocus',fnFocus);<br /> }<br /> var formatMS = function(t){<br /> return Math.floor(t/60) +':'+ (t%60==0?'00':t%60);<br /> }<br /> var timeLog = window.setInterval(function () {<br /> if (inFocus){<br /> eventCount++;<br /> var secondsInFocus = Math.round(eventCount * intervalSeconds);<br /> dataLayer.push({"event": "LittleDataTimer", "interval": interval, "intervalSeconds": intervalSeconds, "timeInFocus": formatMS(secondsInFocus) });<br /> }<br /> if (eventCount>=maxEvents) clearInterval(timeLog);<br /> }, interval);<br /> })();<br /> </script> 2. Add two variables to access the data layer variables One for the formatted time, which will feed through the event label And one for the number of seconds in focus since the last event, which will feed through the event value 3. Add the firing rule for the event 4. Add the tag that reports the timer event to Google Analytics Options and further information You can change the timer interval in the custom HTML tag - the reporting will adjust accordingly. Choosing the interval is a trade-off between the resolution of the reporting and the load on the client in sending events, as well as Google's 500 hit per session quota. We've chosen ten seconds because we think the users who are in 'wrong place' and don't engage at all will leave in under ten seconds, anything more is some measure of success. If you'd like assistance implementing this or something else to get an accurate picture of how users interact with your site, get in touch!   Get Social! Follow us on LinkedIn, Twitter, and Facebook and keep up-to-date with our Google Analytics insights.

2016-11-14

Get the Littledata analytics app

Complete picture of your ecommerce business. Free Google Analytics connection, audit and benchmarks.

Sign up
No Thanks
We respect your privacy. Your information is safe and will never be shared.
Don't miss out. Subscribe today.
×
×