At around 2:50 PM on the 17th October (UTC), our Primary database started experiencing connectivity issues. Upon investigating this, our engineers identified a database failure and proceeded with a replica promotion. During this period, our processing queues backed up and we had to enforce limited new event processing. You may notice some inconsistencies in your data between 2:50 and 6:00 PM and although we recovered quickly, we apologise for any inconvenience caused. We are in the process of conducting an investigation into this issue, to identify ways that we can minimize the risk of similar incidents occurring in the future.
Data ingestion is fully back online and working as expected. Incident has now been resolved.
We have escalated our database replica. The application is online and data ingestion is coming back up.
We are experiencing an outage with one of our major databases. Disruptions across the application and data ingestion pipeline will occur. We are working actively to resolve this.