September 24th, 2020 – [Resolved] Temporary server-side service disruption
From 1800 GMT on September 24th to 0500 GMT on September 25th, our servers faced a temporary disruption of communication with Shopify servers. Due to the disruption, server-side events such as checkout steps and orders were not received by our servers and thus, not sent to Google Analytics or Segment.
The reason for the disruption was a temporary outage of Secure Socket Layer (SSL) protocol at our tech partner, Digital Ocean. Unfortunately, this anomaly was beyond our control, one that we could not have anticipated in advance. We are working closely with Digital Ocean to understand the root cause to make sure this doesn’t happen again.
We have been able to retrieve all the data from this time period and send it to its respective destination. This recovery will take up to 24 hours. You may see a temporary drop in the number of orders on September 24th, and a proportional increase in the following day
June 24th, 2020 – [Follow up] Missing orders for CartHook customers
On June 24th, 2020, at 12.17 UTC, 100% of orders were successfully retrieved and added back to GA or Segment, for all our CartHook customers.
Note: all the missing orders will appear as coming in on today’s date (June 24th), as Google Analytics restricts retroactive backfilling of data.
June 23rd, 2020 – [Resolved] Missing orders for CartHook customers
On June 22nd, 2020 13:58 UTC, we experienced an outage that temporarily halted delivery of orders, processed via CartHook checkout, to GA or Segment. The primary cause of the issue was a bug that inadvertently got introduced during a recent maintenance update. All other events such as Add to Cart, Remove from Cart, etc. were not affected.
The issue was resolved on June 23rd, 14:24 UTC and all orders are now being processed as normal for all our customers. Additionally, our team is currently exploring options to retrieve as much data as possible and backfill at the earliest. We’ll share an update about our progress.
June 9th, 2020 – Brief Server outage
On June 9th, between 10.30 am UTC to 12 am UTC (June 10th) we had a server outage due to our Redis database being unresponsive. The outage completely resolved between 12 am to 6 am on June 10th.
The events might have been delayed by a few hours, however, we do not expect any loss of data for any of our customers.
March 30th, 2020 – Server-side events missing
On March 30th, from 3 PM UTC to 9 PM UTC, we were not able to process and deliver all of the server-side events such as checkout events and orders.
This was due to unusually long data processing times leading to a backlog of events that never got processed and delivered. Unfortunately, most transactions that might have occurred during this period would not have been sent to Google Analytics or Segment (depending on your setup).
The issue was fixed on 9 PM UTC, March 30th, 2020, and since then all events should be coming in as expected.
March 20th, 2020 – Missing attribution issue
From Jan 23rd to March 5th, many of our customers experienced inaccurate attribution for a portion of the events (~15-30%). Most of these events appeared as coming from a (direct) source instead of the actual source.
The primary cause of the issue was a couple of bugs in our codebase as well as in ReCharge’s codebase, that coincidently got introduced simultaneously. Due to the nature of the issue, it took a while for it to surface and get noticed by our team.
This issue has now been fixed and all events should be coming in with accurate attribution for all our customers.
Feb 3rd, 2020 – Database Migration Issue [closed]
From 4 PM GMT through 9 PM GMT, we experienced a brief interruption during our scheduled database migration to IBM Cloud Compose service. The primary cause was an unfortunate coincidence with a global outage of IBM Cloud interface (https://cloud.ibm.com/status?selected=history).
This incident led to a loss of ~1.6% of events across all our customers. Please note that this is an average, the actual volume per store might differ.
The issue was resolved at around 9 PM GMT on the same day. All systems are now working as expected.
Jan 31st, 2020 – Add To Cart missing attribution [closed]
At around 12 PM GMT, we inadvertently deployed a bug that caused Add To Cart events to be sent without GA Client ID values, leading to loss of attribution for these events. By 3 PM GMT, we identified the root cause of the issue and deployed a fix.
This error was limited to any Add To Cart event fired during this period and had no effect on other events. All events (including Add To Cart) should now be firing as normal with accurate attribution.
Jan 14th, 2020 – Server Outage [closed]
We monitored the events for our customers over the past few days. For most customers, we were able to recover and send 70-80% of events. However, despite our best efforts, the remaining events for the period of Jan 9th and Jan 10th could not be recovered. Note – this is an average, the situation might differ from store to store.
Secondly, as Littledata continues to scale, we have now implemented improvements to our queuing infrastructure so that we are better prepared henceforth.
Jan 10th, 2020 – Server Outage [update]
As of midday today, Jan 10th, 2020 all of the backlog has been processed. At this point, most of the events would have been sent to their respective destinations (GA or Segment) with some hours of delay. We will continue to monitor all customer data and issue an update as required.
Jan 9th, 2020 – Server Outage
Around midday January 9th, 2020 GMT our servers experienced an usually high volume of events in a short period, leading to a backlog that didn’t get processed in time. This delay caused a knock-on effect for subsequent events that also got stuck in the queue for the rest of the day. We are currently investigating the issue, more updates to follow