Over the last few weeks, we have investigated previous slowness and unreliability caused by interruptions in our ability to access the database from your systems and for your customers on your websites.
We have identified the cause of these problems and have made a number of infrastructure changes, which have had a positive impact. We can confirm that it is not a security issue, not due to overload, and that there has been no impact on data stored in Spektrix.
The periods of system slowness and unreliability were: Wednesday 9 October (from 09:55 BST / 04:55 EDT - 12:15 BST / 07:15 EDT) Thursday 10 October (from 11:00 BST / 06:00 EDT - 11:45 BST / 06:45 EDT) Monday 21 October (from 15:55 BST / 10:55 EDT - 17:15 BST / 12:15 EDT)
We are back to monitoring the system and maintaining system stability as usual.
Posted Nov 01, 2024 - 13:07 GMT
Update
On Monday 21 October (from 15:55 BST / 10:55 EDT and 17:15 BST / 12:15 EDT) we again experienced intermittent interruptions in our ability to access the database from both your systems and for your customers on your websites, causing slowness and unreliability.
The root cause is under investigation, and we have been able to confirm that it is not a security issue, not due to overload, and that there has been no impact on data stored in Spektrix.
We remain in an extended monitoring phase. This is so that we can fully ascertain the cause and prevent any recurrences of the issue. Your system can be used as normal.
We are sorry for the disruption you may have experienced. These types of incidents are rare, but we take their impact on you seriously. We will continue to update this page with real-time status information and updates as our investigation develops.
Posted Oct 23, 2024 - 14:56 BST
Monitoring
Earlier today we had number of occurrences of system slowness and error pages. These occurred between 15:55 BST / 10:55 EDT and 17:15 BST / 12:15 EDT.
The issue is not related to system load or security. It is related to the communication between, and/or functioning of, two key infrastructure components. We are continuing to work with our infrastructure provider to identify the root cause, and have implemented and refined additional internal alerting and reporting to ensure we can respond promptly if there are any repeat issues.
We have no reason to expect that problems will reoccur today but understand this may be concerning. To provide backup, we suggest running an Event Instance Occupancy report from your Sales Interface, which will provide you with customer information and availability in the event of any further system issues. The report can be run on a per Instance basis, using the link below the Seating Plan in the Sales Interface.
If you need any support to prepare for today's events, please reach out to us.
Posted Oct 21, 2024 - 20:06 BST
Update
System performance has returned to normal speeds, from 17:15 BST / 12:15 EDT.
We have taken further actions to mitigate the likelihood of recurrence, and we have no reason to expect that problems will reoccur today. We realise that this may be concerning, particularly ahead of events taking place today. To provide backup, we suggest running an Event Instance Occupancy report from your Sales Interface, which will provide you with customer information and seat availability in the event of any further system issues. The report can be run on a per Instance basis, using the reports dropdown below the seating plan in the Sales Interface.
If you need any support to prepare for today's events, please reach out to us.
Posted Oct 21, 2024 - 17:56 BST
Update
We are seeing the system slowness return again.
We have a full team working on restoring stability as a priority.
Posted Oct 21, 2024 - 17:10 BST
Update
We are continuing to closely monitor the system, but have seen significant increases in stability over the last ten minutes.
This is related to the communication between, and/or functioning of, two key infrastructure components. It is not related to system load or security.
We understand the disruption that this causes to you and your patrons, and thank you for your understanding whilst we work on this.
Posted Oct 21, 2024 - 16:58 BST
Update
We are seeing system performance continue to improve and steady, with improvement click speeds and fewer error screens.
We are monitoring this closely to ensure this remains stable.
Posted Oct 21, 2024 - 16:44 BST
Update
We are seeing the problems with system performance recur, causing error screens and significant latency when clicking around systems.
We are re-investigating this to find out why our previous fix hasn't left the system in a stable state, and are treating this as a priority.
Posted Oct 21, 2024 - 16:37 BST
Update
We are seeing system speeds return to normal and a reduction in error screens. You should now be able to access your Spektrix system, though you may still see some occasional slowness as things return to a normal level.
From 15:55 BST / 10:55 EDT to 16:20 BST / 11:20 EDT there was significant slowness across the Spektrix system and frequent error screens were encountered.
Posted Oct 21, 2024 - 16:24 BST
Update
We are continuing to work on this as a priority. Since 15:55 BST / 10:55 EDT we have seen significant slowness when accessing the Spektrix system, and some green error screens.
Currently we are still seeing online booking pathways confirming transactions, though there is some slowness in this processing.
We are working as a priority to restore system access.
Posted Oct 21, 2024 - 16:18 BST
Identified
We are seeing a return of the system slowness and error screens that have been intermittently experienced in the last couple of weeks. This is causing error screens when accessing and moving around your Spektrix system.
Currently, we are seeing online orders and websites continue to work, though they are loading more slowly than usual.
We are in the process of attempting a fix to mitigate the impact of this.
Posted Oct 21, 2024 - 16:05 BST
Update
Last week we had two occurrences of system slowness and error pages. These were on Wednesday 9 October (from 09:55 BST / 04:55 EDT - 12:15 BST / 07:15 EDT) and Thursday 10 October (from 11:00 BST / 06:00 EDT - 11:45 BST / 06:45 EDT).
We’ve identified the issue as being related to the communication between, and/or functioning of, two key infrastructure components. It is not related to system load or security. We are continuing to work with our infrastructure provider to identify the root cause, and have implemented and refined additional internal alerting and reporting to ensure we can respond promptly if there are any repeat issues.
We remain in an extended monitoring phase. This shouldn’t be a cause for concern, but rather is an opportunity for us to ensure we can fully ascertain the cause and prevent any recurrences of the issue. Please continue using your system as normal; there is no need to limit activity or access.
Posted Oct 16, 2024 - 14:17 BST
Update
The system has remained stable over the weekend. We continue to monitor closely and our teams will regroup tomorrow to continue investigating the root cause of the problems experienced with our key infrastructure components.
We’d like to thank you for your patience as we work to resolve this.
Posted Oct 13, 2024 - 16:57 BST
Update
We are continuing to monitor the system status closely, and have not seen a repeat of the slowness and error pages experienced this week. Our teams are keeping a close eye on system performance and we’ve refined internal alerting to ensure we can respond promptly, but please do reach out to us if you’ve any concerns. You can continue using your system as normal in this time.
Posted Oct 12, 2024 - 14:59 BST
Update
On Wednesday 9 October, from 09:55 BST / 04:55 EDT - 12:15 BST / 07:15 EDT, we received internal alerts of slowness across the system, and reports from system users that they were receiving error pages. This occurred again on Thursday 10 October, from 11:00 BST / 06:00 EDT - 11:45 BST / 06:45 EDT.
We’ve not since seen a recurrence of the problems experienced this week, and we’re continuing to investigate the root cause of the issues. We are monitoring system status closely and this monitoring will continue over the weekend. You can continue using your system as normal in this time; there is no need to limit activity or access.
We’ve received some questions asking whether these errors are related to system load or security - and the answer to both of these is no. We’ve identified the issue as being related to the communication between, and/or functioning of, two key infrastructure components. As the root cause of this is still unknown, there remains a possibility that these errors and behaviours may reoccur.
As this is a long-running problem with a complex investigation, we are in an extended monitoring phase, to ensure we can fully ascertain the cause and prevent any recurrences of the issue. We appreciate this will have had an impact on some of your operations this week and would like to thank you for your patience as we work to resolve this.
Posted Oct 11, 2024 - 16:00 BST
Update
System speeds and behaviour have returned to normal, but we’re continuing to monitor its status closely. We’ve received some questions asking whether these errors are related to system load or security - and the answer to both of these is no. We’ve identified the issue as being related to the communication between, and/or functioning of, two key infrastructure components. As the root cause of this is still unknown, there remains a possibility that these errors and behaviour may reoccur. However, please be assured that we have engineers actively working to understand the root cause and are proactively monitoring the system
Posted Oct 10, 2024 - 12:38 BST
Monitoring
We are continuing to monitor the system, but have seen system speeds to return to normal.
Posted Oct 10, 2024 - 11:51 BST
Identified
We are seeing the system stabilise and speeds return to normal. We are continuing to closely look at this.
Posted Oct 10, 2024 - 11:43 BST
Investigating
We are currently investigating reports of system slowness and error screens, both in your system and on websites.
We are treating this as a priority and investigating all channels.
Posted Oct 10, 2024 - 11:33 BST
This incident affected: Spektrix System (UK and Ireland, US and Canada).