Our core belief at Littledata is that machines are better at spotting significant changes in your website’s performance than a human analyst. We’ve now made it easier for you to get specific alerts, reducing the time spent wading through data. This is the story of how we produced the new trend detection algorithm. Enjoy!
Back in 2014, we developed the first version of an algorithm to detect if today or this week’s traffic was significantly different from previous periods. This allows managers to focus in on the aspects of the traffic or particular marketing campaigns which are really worthy of their attention. Although the first version was very sensitive, it also picked up too many changes for a single person to investigate. In technical language, it was not specific in enough.
In June and July, Littledata collaborated with a working group of mathematicians from around Europe to find a better algorithm. The European Study Group with Industry (ESGI) originated in the University of Limerick’s mathematics department in Ireland and has helped hundreds of businesses link up with prominent mathematicians in the field to solve real-world problems. Littledata joined the latest study group in University College, Dublin in July, and was selected by a dozen mathematicians as the focus for their investigation.
Andrew Parnell from the statistics department at University College, Dublin helped judge the output from the four teams that we split the group into. The approach was to use an algorithm to test the algorithms; in other words, we pitted a group of statistical strategies against each other, from clustering techniques to linear regression, through to Twitter’s own trend detection package, and compared their total performance across a range of training data sets. Initially, the Twitter package looked to be doing well, but in fact, it had been developed specifically to analyse huge volumes of tweets and perform badly when given low volumes of web traffic.
In between our host’s generous hospitality, with Guinness, Irish folk music, and quite a lot of scribbling of formulas on beer mats, myself and our engineer (Gabriel) worked with the statisticians to tweak the algorithms. Eventually, a winner emerged, being sensitive enough to pick up small changes in low traffic websites, but also specific enough to ignore the random noise of daily traffic.
The new trend detection algorithm has been live since the start of August and we hope you enjoy the benefits. Our web app allows for fewer distractions and more significant alerts tailored to your company’s goals, which takes you back to our core belief that machines are able to spot major changes in website performances better than a human analyst.
If you’re interested in finding out how our web app can help you streamline your Google Analytics’ data, please get in touch!
