System Slowness
Incident Report for Spektrix
Postmortem

Shortly after 10:00 (GMT) on Friday 14th February a majority of customers booking online saw slow response times when progressing through a sale. From around 10:30-10:45 some customers may have seen a maintenance page when booking online and were not able to complete transactions. From around 10:20-10:50 some users were unable to take card payments over the phone and via chip & pin.

All services returned to normal just after 11:00.

A second issue occurred at around 12:45 which meant that at times Spektrix did not respond to clicks and sometimes behaved unexpectedly for some users. This problem was resolved by 13:15.

The system remained stable from this point onwards.

We have identified that there were a combination of factors that caused this period of instability:

  • Our ‘Basket Web Component’ started to behave inefficiently in a way that hadn’t previously been observed and was not identified in testing. We have already made changes to improve the efficiency of this feature and will continue to monitor and test this.
  • We also identified a bug in our queuing system. We will address this as a high priority but we don’t expect this specific issue to have the same impact in future.
  • The later issue with clicks not completing and the system behaving unexpectedly was coincidental in timing and not related to the earlier problems. This was caused by a problem with one of our web servers, which are the computers that respond to your clicks. This problem has since been resolved.

We are always committed to providing the best possible service for our clients and their customers, and we apologise for the disruption caused on Friday.

Please do contact the Support, Training and Consultancy team if you have any further questions, on support@spektrix.com.

Posted Feb 18, 2020 - 15:42 GMT

Resolved
Everything is remaining stable so we are closing this incident. To confirm, you may need to restart your Spektrix App or PC in order to get your hardware working again, otherwise everything should be up and running normally. Sorry for the disruption and impact this will have had on your day.
Posted Feb 14, 2020 - 11:18 GMT
Monitoring
Your online booking pages should now be back up and running as well. We are still closely monitoring the system to make sure everything stays stable, so please continue to report any unusual behaviour to our team.
Posted Feb 14, 2020 - 11:06 GMT
Identified
The Spektrix App should now be back up and running, meaning that chip and PIN and printing should be working again. You may need to restart the App or your PC in order to see it reconnect again. We are continuing to work on getting your online booking pages up and running and we'll update on here as soon as we have more information.
Posted Feb 14, 2020 - 11:00 GMT
Update
We are still investigating and working through the issues previously outlined with slowness for customers online.
Most of the Spektrix interface itself should be working fine, but the online booking pages may currently see a maintenance page.
The Spektrix App may also be affected for some of you, affecting chip and pin and printing.
Posted Feb 14, 2020 - 10:41 GMT
Update
Apologies, we are aware there are also some issues accessing the Spektrix App (which will affect chip and pin and printing). We are on the case and will update further on this as well.
Posted Feb 14, 2020 - 10:25 GMT
Update
The interface itself should be running normally but we are aware customers online will be having difficulties booking. If customers call you should be able to take over and confirm transactions.

We are working on resolving this as soon as possible and will update shortly.
Posted Feb 14, 2020 - 10:22 GMT
Investigating
We have received reports of system slowness using the interface and for customers booking online.
We're looking at this right now and will have updates very soon.
Posted Feb 14, 2020 - 10:17 GMT
This incident affected: Spektrix System (UK and Ireland).