March 30th, 2020 – Server-side events missing
On March 30th, from 3 PM UTC to 9 PM UTC, we were not able to process and deliver all of the server-side events such as checkout events and orders.
This was due to unusually long data processing times leading to a backlog of events that never got processed and delivered. Unfortunately, most transactions that might have occurred during this period would not have been sent to Google Analytics or Segment (depending on your setup).
The issue was fixed on 9 PM UTC, March 30th, 2020, and since then all events should be coming in as expected.
March 20th, 2020 – Missing attribution issue
From Jan 23rd to March 5th, many of our customers experienced inaccurate attribution for a portion of the events (~15-30%). Most of these events appeared as coming from a (direct) source instead of the actual source.
The primary cause of the issue was a couple of bugs in our codebase as well as in ReCharge’s codebase, that coincidently got introduced simultaneously. Due to the nature of the issue, it took a while for it to surface and get noticed by our team.
This issue has now been fixed and all events should be coming in with accurate attribution for all our customers.
Feb 3rd, 2020 – Database Migration Issue [closed]
From 4 PM GMT through 9 PM GMT, we experienced a brief interruption during our scheduled database migration to IBM Cloud Compose service. The primary cause was an unfortunate coincidence with a global outage of IBM Cloud interface (https://cloud.ibm.com/status?selected=history).
This incident led to a loss of ~1.6% of events across all our customers. Please note that this is an average, the actual volume per store might differ.
The issue was resolved at around 9 PM GMT on the same day. All systems are now working as expected.
Jan 31st, 2020 – Add To Cart missing attribution [closed]
At around 12 PM GMT, we inadvertently deployed a bug that caused Add To Cart events to be sent without GA Client ID values, leading to loss of attribution for these events. By 3 PM GMT, we identified the root cause of the issue and deployed a fix.
This error was limited to any Add To Cart event fired during this period and had no effect on other events. All events (including Add To Cart) should now be firing as normal with accurate attribution.
Jan 14th, 2020 – Server Outage [closed]
We monitored the events for our customers over the past few days. For most customers, we were able to recover and send 70-80% of events. However, despite our best efforts, the remaining events for the period of Jan 9th and Jan 10th could not be recovered. Note – this is an average, the situation might differ from store to store.
Secondly, as Littledata continues to scale, we have now implemented improvements to our queuing infrastructure so that we are better prepared henceforth.
Jan 10th, 2020 – Server Outage [update]
As of midday today, Jan 10th, 2020 all of the backlog has been processed. At this point, most of the events would have been sent to their respective destinations (GA or Segment) with some hours of delay. We will continue to monitor all customer data and issue an update as required.
Jan 9th, 2020 – Server Outage
Around midday January 9th, 2020 GMT our servers experienced an usually high volume of events in a short period, leading to a backlog that didn’t get processed in time. This delay caused a knock-on effect for subsequent events that also got stuck in the queue for the rest of the day. We are currently investigating the issue, more updates to follow