Spektrix Status

Monitoring - Earlier today we had number of occurrences of system slowness and error pages. These occurred between 15:55 BST / 10:55 EDT and 17:15 BST / 12:15 EDT.

The issue is not related to system load or security. It is related to the communication between, and/or functioning of, two key infrastructure components. We are continuing to work with our infrastructure provider to identify the root cause, and have implemented and refined additional internal alerting and reporting to ensure we can respond promptly if there are any repeat issues.

We have no reason to expect that problems will reoccur today but understand this may be concerning. To provide backup, we suggest running an Event Instance Occupancy report from your Sales Interface, which will provide you with customer information and availability in the event of any further system issues. The report can be run on a per Instance basis, using the link below the Seating Plan in the Sales Interface.

If you need any support to prepare for today's events, please reach out to us.
Oct 21, 2024 - 20:06 BST

Update - System performance has returned to normal speeds, from 17:15 BST / 12:15 EDT.

We have taken further actions to mitigate the likelihood of recurrence, and we have no reason to expect that problems will reoccur today. We realise that this may be concerning, particularly ahead of events taking place today. To provide backup, we suggest running an Event Instance Occupancy report from your Sales Interface, which will provide you with customer information and seat availability in the event of any further system issues. The report can be run on a per Instance basis, using the reports dropdown below the seating plan in the Sales Interface.

If you need any support to prepare for today's events, please reach out to us.
Oct 21, 2024 - 17:56 BST

Update - We are seeing the system slowness return again.

We have a full team working on restoring stability as a priority.
Oct 21, 2024 - 17:10 BST

Update - We are continuing to closely monitor the system, but have seen significant increases in stability over the last ten minutes.

This is related to the communication between, and/or functioning of, two key infrastructure components. It is not related to system load or security.

We understand the disruption that this causes to you and your patrons, and thank you for your understanding whilst we work on this.
Oct 21, 2024 - 16:58 BST

Update - We are seeing system performance continue to improve and steady, with improvement click speeds and fewer error screens.

We are monitoring this closely to ensure this remains stable.
Oct 21, 2024 - 16:44 BST

Update - We are seeing the problems with system performance recur, causing error screens and significant latency when clicking around systems.

We are re-investigating this to find out why our previous fix hasn't left the system in a stable state, and are treating this as a priority.
Oct 21, 2024 - 16:37 BST

Update - We are seeing system speeds return to normal and a reduction in error screens. You should now be able to access your Spektrix system, though you may still see some occasional slowness as things return to a normal level.

From 15:55 BST / 10:55 EDT to 16:20 BST / 11:20 EDT there was significant slowness across the Spektrix system and frequent error screens were encountered.
Oct 21, 2024 - 16:24 BST

Update - We are continuing to work on this as a priority. Since 15:55 BST / 10:55 EDT we have seen significant slowness when accessing the Spektrix system, and some green error screens.

Currently we are still seeing online booking pathways confirming transactions, though there is some slowness in this processing.

We are working as a priority to restore system access.
Oct 21, 2024 - 16:18 BST

Identified - We are seeing a return of the system slowness and error screens that have been intermittently experienced in the last couple of weeks. This is causing error screens when accessing and moving around your Spektrix system.

Currently, we are seeing online orders and websites continue to work, though they are loading more slowly than usual.

We are in the process of attempting a fix to mitigate the impact of this.
Oct 21, 2024 - 16:05 BST

Update - Last week we had two occurrences of system slowness and error pages. These were on Wednesday 9 October (from 09:55 BST / 04:55 EDT - 12:15 BST / 07:15 EDT) and Thursday 10 October (from 11:00 BST / 06:00 EDT - 11:45 BST / 06:45 EDT).

We’ve identified the issue as being related to the communication between, and/or functioning of, two key infrastructure components. It is not related to system load or security. We are continuing to work with our infrastructure provider to identify the root cause, and have implemented and refined additional internal alerting and reporting to ensure we can respond promptly if there are any repeat issues.

We remain in an extended monitoring phase. This shouldn’t be a cause for concern, but rather is an opportunity for us to ensure we can fully ascertain the cause and prevent any recurrences of the issue. Please continue using your system as normal; there is no need to limit activity or access.
Oct 16, 2024 - 14:17 BST

Update - The system has remained stable over the weekend. We continue to monitor closely and our teams will regroup tomorrow to continue investigating the root cause of the problems experienced with our key infrastructure components.

We’d like to thank you for your patience as we work to resolve this.
Oct 13, 2024 - 16:57 BST

Update - We are continuing to monitor the system status closely, and have not seen a repeat of the slowness and error pages experienced this week. Our teams are keeping a close eye on system performance and we’ve refined internal alerting to ensure we can respond promptly, but please do reach out to us if you’ve any concerns. You can continue using your system as normal in this time.
Oct 12, 2024 - 14:59 BST

Update - On Wednesday 9 October, from 09:55 BST / 04:55 EDT - 12:15 BST / 07:15 EDT, we received internal alerts of slowness across the system, and reports from system users that they were receiving error pages. This occurred again on Thursday 10 October, from 11:00 BST / 06:00 EDT - 11:45 BST / 06:45 EDT.

We’ve not since seen a recurrence of the problems experienced this week, and we’re continuing to investigate the root cause of the issues. We are monitoring system status closely and this monitoring will continue over the weekend. You can continue using your system as normal in this time; there is no need to limit activity or access.

We’ve received some questions asking whether these errors are related to system load or security - and the answer to both of these is no. We’ve identified the issue as being related to the communication between, and/or functioning of, two key infrastructure components. As the root cause of this is still unknown, there remains a possibility that these errors and behaviours may reoccur.

As this is a long-running problem with a complex investigation, we are in an extended monitoring phase, to ensure we can fully ascertain the cause and prevent any recurrences of the issue. We appreciate this will have had an impact on some of your operations this week and would like to thank you for your patience as we work to resolve this.
Oct 11, 2024 - 16:00 BST

Update - System speeds and behaviour have returned to normal, but we’re continuing to monitor its status closely. We’ve received some questions asking whether these errors are related to system load or security - and the answer to both of these is no. We’ve identified the issue as being related to the communication between, and/or functioning of, two key infrastructure components. As the root cause of this is still unknown, there remains a possibility that these errors and behaviour may reoccur. However, please be assured that we have engineers actively working to understand the root cause and are proactively monitoring the system
Oct 10, 2024 - 12:38 BST

Monitoring - We are continuing to monitor the system, but have seen system speeds to return to normal.
Oct 10, 2024 - 11:51 BST

Identified - We are seeing the system stabilise and speeds return to normal. We are continuing to closely look at this.
Oct 10, 2024 - 11:43 BST

Investigating - We are currently investigating reports of system slowness and error screens, both in your system and on websites.

We are treating this as a priority and investigating all channels.
Oct 10, 2024 - 11:33 BST

About This Site

Stay up-to-date on Spektrix's system status here. You can also subscribe to notifications above.

If you're experiencing something that isn't on this page, get in touch:

support@spektrix.com
UK & Ireland: 020 7183 3586
US & Canada: (646) 569-9097 or 1-800-961-5971

Uptime over the past 90 days. View historical uptime.

Spektrix System Operational

90 days ago

99.79 % uptime

Today

UK and Ireland Operational

90 days ago

99.78 % uptime

Today

US and Canada Operational

90 days ago

99.81 % uptime

Today

Payments Processing Operational

90 days ago

99.99 % uptime

Today

Opayo (formerly SagePay) Operational

90 days ago

100.0 % uptime

Today

Spektrix Payments (Adyen) Operational

90 days ago

99.99 % uptime

Today

Worldpay (formerly Vantiv) Operational

90 days ago

100.0 % uptime

Today

Authorize.Net Operational

90 days ago

100.0 % uptime

Today

Moneris Operational

90 days ago

99.99 % uptime

Today

PayPal Operational

90 days ago

100.0 % uptime

Today

dotdigital Operational

90 days ago

100.0 % uptime

Today

Spektrix Link Operational

90 days ago

99.98 % uptime

Today

Support Services Operational

90 days ago

100.0 % uptime

Today

Support Centre and support@spektrix.com Operational

90 days ago

100.0 % uptime

Today

Phone Service - UK/Ireland Operational

90 days ago

100.0 % uptime

Today

Phone Service - North America Operational

90 days ago

100.0 % uptime

Today

APIs Operational

90 days ago

99.98 % uptime

Today

APIs - UK and Ireland Operational

90 days ago

99.98 % uptime

Today

APIs - US and Canada Operational

90 days ago

99.98 % uptime

Today

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Past Incidents

Oct 22, 2024

No incidents reported today.

Oct 21, 2024

Unresolved incident: System slowness and error screens.

Oct 20, 2024

No incidents reported.

Oct 19, 2024

No incidents reported.

Oct 18, 2024

No incidents reported.

Oct 17, 2024

No incidents reported.

Oct 16, 2024

Oct 15, 2024

Spektrix Box Office App not connecting

Resolved - We have continued to monitor the Box Office App performance closely, and are confident it has remained in a stable position.

From 11:30 BST / 06:30 EDT - 17:45 BST / 12:45 EDT on Monday 14 October, the Box Office App was unavailable. This meant that in-person chip & pin payments and ticket printing were not possible in this time period. This was caused by an upgrade to an underlying technology service the Box Office App runs on. That upgrade didn't roll out as expected and took significant resetting to get back to normal. Now this is resolved and stable, these issues shouldn't reoccur.

There may be a small number of cases where you are required to re-install the Box Office App. We have detailed guidance of the steps required on our Support Centre, but if you require assistance or further guidance, please do reach out to our team.

Thank you for your patience and understanding whilst we worked to resolve this.
Oct 15, 12:41 BST

Monitoring - We have been able to fully implement a fix to resolve the connectivity issues with the Box Office App. If you now attempt to restart or re-register your Box Office App, this will be successful.

There may be a small number of cases where you are required to re-install the Box Office App. We have detailed guidance of the steps required on our Support Centre, but if you require assistance or further guidance, please do reach out to our team.

From 11:30 BST / 06:30 EDT - 17:45 BST / 12:45 EDT the Box Office App was unavailable. This meant that in-person chip & pin payments and ticket printing were not possible in this time period. Chip payments would have remained possible for all clients using Spektrix Payments.

Thank you for your patience and understanding whilst we worked to resolve this. The issues experienced with the Box Office App today are unrelated to the problems with system performance that we saw last week, and we understand that this will have had an impact on your operations.
Oct 14, 17:57 BST

Update - We are continuing to work on restoring Box Office App connectivity as a priority.

For those who urgently require a solution to print and take in-person payments, please consider:
- Taking in-person payments using the 'card' option, where you enter the long card number as if you were on the phone
- Running occupancy reports for event admittance, which can be run within the Sales Interface underneath each Event Instance.

Please reach out to the team if you need assistance with these.
Oct 14, 17:26 BST

Update - As part of our ongoing efforts to resolve the connectivity issues with the Box Office App, you may have seen a notification that the Box Office App has now connected successfully.

We are still in the process of rolling out the fix, and this does not yet mean that you'll be able to connect via Local Settings to successfully take in-person chip & pin payments or print tickets.

We are working on restoring this functionality as a priority.
Oct 14, 16:15 BST

Update - We are continuing to work with our infrastructure provider on a resolution to the Box Office App connectivity issues as a priority.

For those who urgently require a solution to print and take in-person payments, then you could consider running occupancy reports (these can be run from the Sales Interface underneath each Event Instance) in place of printing tickets, and taking in-person payments using the long card number (as if you were on the phone).

Please reach out to the Support Team if you need help with these.
Oct 14, 14:32 BST

Update - We’re currently in dialogue with our infrastructure provider to continue with the required work to resolve the connectivity issues with the Box Office App. As soon as we have further understanding on the timeline we will provide an update on when we expect this to be resolved. We don’t believe that any attempts to fix the problem locally by attempting to restart or re-install will succeed currently.

For those who are urgently requiring a solution to print and take in-person payments then you could consider running occupancy reports (these can be run from the Sales Interface underneath each Event Instance) in place of printing tickets, and taking in-person payments using the long card number (as if you were on the phone). Please get in touch with us if you need help with these.
Oct 14, 13:29 BST

Update - We're still working through the fix to the Box Office App. In this time you may see slowness accessing Local Settings within Spektrix, and your Box Office App may be showing as disconnected, which would prevent you from printing tickets via Spektrix, or taking chip and pin/contactless payments. Spektrix Payments users will be able to take in-person payments still, and print receipts from card terminals, but will be unable to print physical tickets.

If you are experiencing issues, and have the relevant permissions to be able to restart or reinstall the App, you can try do to so, although it may not successfully reconnect at this time.

We will continue to update regularly as we process a complete fix for this issue, and will post here with any further steps or instructions you may need to take as required.
Oct 14, 12:32 BST

Update - We are continuing to work on a fix for the Box Office App connectivity problems that some people are experiencing at the moment. You may also see slowness accessing 'Local Settings' from your control panel.

If your App is currently unable to restart we advise waiting a short period of time for an update from us. We can advise further when this is resolved and potential next steps if your App doesn't reconnect automatically.
Oct 14, 12:03 BST

Identified - We have identified an issue that has resulted in several users are currently having issues connecting to the Box Office App. This will be impacting the ability to print or take chip and pin payments. Chip and Pin payments for Spektrix Payments users will be unaffected.

We are working on a fix for this currently. In the meantime if your App is currently unable to restart we would advise waiting a short period of time for an update from us. We can advise further when this is resolved and potential next steps if your App doesn't reconnect automatically.
Oct 14, 11:50 BST

Oct 14, 2024

Oct 13, 2024

Oct 12, 2024

Oct 11, 2024

Oct 10, 2024

Oct 9, 2024

System slowness and error screens

Resolved - After a period of close monitoring and root cause analysis, we are happy to consider this fully resolved. All aspects of the system have been fully stable since c. 12:15 BST / 07:15 EST / 04:15 PT.

At around 09:55 BST / 04:55 EST / 01:55 PT we first received internal alerts of slowness across the system. We also began to receive reports from system users that they were receiving error pages, and customers online would have experienced similar issues when browsing Spektrix-powered websites.

Shortly after this point, we were able to identify a broad area of our infrastructure relating to memory storage that was returning errors. Our work focused on this area throughout whilst we also investigated a range of other potential causes of the problem.
This initial period of investigation took around an hour - during this time period we were able to implement mitigations to try and rectify this, but we could see the core problem was still occurring. During this time there were periods were the system was more useable than others, and the amount of error pages and slowness experienced will have varied.

At approximately 11:30 / 06:30 / 03:30 we were able to fully isolate the part of the infrastructure within the above memory storage area that was causing the largest amount of impact. This was an internal part of our systems that was unrelated to any other system or user activity. Once we implemented a fix to resolve this shortly afterwards, system errors began to rapidly decrease. By 11:50 / 06:50 / 03:50 impact on regular system usage would have been minimal, and by 12:15 / 07:15 / 04:15 errors related to this area were completely eliminated.

As this was a long-running problem with a complex investigation, we then moved into an extended monitoring phase, so we could fully ascertain the cause and ensure that steps were in place to prevent any recurrences of the issue. We appreciate this will have had a significant impact on your operations today and would like to thank you for your patience as we worked to resolve this.

As always after any system issue of this scale we will reflect internally and make improvements to both our system health and our communications process. If you have any feedback you would like to share, or would like to discuss this issue in any more detail with us, please reach out to us at support@spektrix.com.
Oct 9, 17:12 BST

Update - We are continuing to stay in a monitoring phase as we undertake further root cause analysis. We are still seeing that the part of infrastructure that had issues earlier is stable, and we aren't anticipating any further slowness or outages. We should be in a position to fully resolve this status page shortly, and at this point we will share more information on the problems you experienced today, including a more detailed timeline.
Oct 9, 14:44 BST

Monitoring - After further work we can see that the number of errors you are seeing has vastly reduced - you should no longer have any issues accessing or using the system, and customers online will also be able to checkout successfully without any problems. We are continuing to monitor for any further issues or recurrences of the problem. Thanks for your patience as we worked through this issue today.
Oct 9, 12:41 BST

Identified - We are again seeing that action we are taking is reducing the number of errors that you are experiencing currently. We're still investigating and monitoring our changes and taking further action to continue this positive trend. Updates will continue as we work through this.
Oct 9, 11:57 BST

Update - We are seeing error messages continue to happen as part of normal clicking around the system, and this behaviour will also be apparent for customers online as well. We are still doing work to understand the cause of this problem - we have identified the part of our infrastructure that is causing issues and are focusing our efforts on this area. We will continue to update here at regular intervals.
Oct 9, 11:31 BST

Investigating - We're seeing instances of errors increase again. We're continuing to explore all avenues to resolve this as quickly as possible.
Oct 9, 11:03 BST

Identified - The work we are doing is continuing to show signs of improvements. Whilst both users and end-customers may still see errors, we are seeing the frequency of these errors continue to decrease. We will continue to update you as we work to resolve this fully.
Oct 9, 10:53 BST

Update - We are still working on investigating this issue. We have taken some initial actions that we believe are starting to have impact, but we are still seeing that users are getting error pages on a regular basis. We are continuing to take steps to resolve this as a priority and will update further shortly.
Oct 9, 10:37 BST

Update - We are continuing to investigate this as a priority.

This is impacting systems globally; you may experience slow loading times or green error screens within your system, and error messages on your website.
Oct 9, 10:22 BST

Investigating - We are currently investigating slowness across Spektrix systems. We have seen this since around 10am BST / 5am EDT, and are experiencing this in systems as well as on your websites.

We are investigating this as a priority.
Oct 9, 10:16 BST

Oct 8, 2024

No incidents reported.

About This Site

Related