After a period of close monitoring and root cause analysis, we are happy to consider this fully resolved. All aspects of the system have been fully stable since c. 12:15 BST / 07:15 EST / 04:15 PT.
At around 09:55 BST / 04:55 EST / 01:55 PT we first received internal alerts of slowness across the system. We also began to receive reports from system users that they were receiving error pages, and customers online would have experienced similar issues when browsing Spektrix-powered websites.
Shortly after this point, we were able to identify a broad area of our infrastructure relating to memory storage that was returning errors. Our work focused on this area throughout whilst we also investigated a range of other potential causes of the problem. This initial period of investigation took around an hour - during this time period we were able to implement mitigations to try and rectify this, but we could see the core problem was still occurring. During this time there were periods were the system was more useable than others, and the amount of error pages and slowness experienced will have varied.
At approximately 11:30 / 06:30 / 03:30 we were able to fully isolate the part of the infrastructure within the above memory storage area that was causing the largest amount of impact. This was an internal part of our systems that was unrelated to any other system or user activity. Once we implemented a fix to resolve this shortly afterwards, system errors began to rapidly decrease. By 11:50 / 06:50 / 03:50 impact on regular system usage would have been minimal, and by 12:15 / 07:15 / 04:15 errors related to this area were completely eliminated.
As this was a long-running problem with a complex investigation, we then moved into an extended monitoring phase, so we could fully ascertain the cause and ensure that steps were in place to prevent any recurrences of the issue. We appreciate this will have had a significant impact on your operations today and would like to thank you for your patience as we worked to resolve this.
As always after any system issue of this scale we will reflect internally and make improvements to both our system health and our communications process. If you have any feedback you would like to share, or would like to discuss this issue in any more detail with us, please reach out to us at support@spektrix.com.
Posted Oct 09, 2024 - 17:12 BST
Update
We are continuing to stay in a monitoring phase as we undertake further root cause analysis. We are still seeing that the part of infrastructure that had issues earlier is stable, and we aren't anticipating any further slowness or outages. We should be in a position to fully resolve this status page shortly, and at this point we will share more information on the problems you experienced today, including a more detailed timeline.
Posted Oct 09, 2024 - 14:44 BST
Monitoring
After further work we can see that the number of errors you are seeing has vastly reduced - you should no longer have any issues accessing or using the system, and customers online will also be able to checkout successfully without any problems. We are continuing to monitor for any further issues or recurrences of the problem. Thanks for your patience as we worked through this issue today.
Posted Oct 09, 2024 - 12:41 BST
Identified
We are again seeing that action we are taking is reducing the number of errors that you are experiencing currently. We're still investigating and monitoring our changes and taking further action to continue this positive trend. Updates will continue as we work through this.
Posted Oct 09, 2024 - 11:57 BST
Update
We are seeing error messages continue to happen as part of normal clicking around the system, and this behaviour will also be apparent for customers online as well. We are still doing work to understand the cause of this problem - we have identified the part of our infrastructure that is causing issues and are focusing our efforts on this area. We will continue to update here at regular intervals.
Posted Oct 09, 2024 - 11:31 BST
Investigating
We're seeing instances of errors increase again. We're continuing to explore all avenues to resolve this as quickly as possible.
Posted Oct 09, 2024 - 11:03 BST
Identified
The work we are doing is continuing to show signs of improvements. Whilst both users and end-customers may still see errors, we are seeing the frequency of these errors continue to decrease. We will continue to update you as we work to resolve this fully.
Posted Oct 09, 2024 - 10:53 BST
Update
We are still working on investigating this issue. We have taken some initial actions that we believe are starting to have impact, but we are still seeing that users are getting error pages on a regular basis. We are continuing to take steps to resolve this as a priority and will update further shortly.
Posted Oct 09, 2024 - 10:37 BST
Update
We are continuing to investigate this as a priority.
This is impacting systems globally; you may experience slow loading times or green error screens within your system, and error messages on your website.
Posted Oct 09, 2024 - 10:22 BST
Investigating
We are currently investigating slowness across Spektrix systems. We have seen this since around 10am BST / 5am EDT, and are experiencing this in systems as well as on your websites.
We are investigating this as a priority.
Posted Oct 09, 2024 - 10:16 BST
This incident affected: Spektrix System (UK and Ireland, US and Canada).