tag:status.spektrix.com,2005:/historySpektrix Status - Incident History2024-03-28T15:48:45+00:00Spektrixtag:status.spektrix.com,2005:Incident/203870362024-03-28T12:03:12+00:002024-03-28T12:03:12+00:00Slowness running reports, customer lists and Dotdigital mailings<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>12:03</var> GMT</small><br><strong>Resolved</strong> - We're confident that the issue has now been resolved and all jobs should be running normally.<br /><br />Thank you for your patience whilst we worked to resolve this.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>11:52</var> GMT</small><br><strong>Monitoring</strong> - We are now seeing reports and customer lists should now be completing normally. <br /><br />Jobs including Dotdigital mailings that have been delayed should also now start to complete</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>11:48</var> GMT</small><br><strong>Identified</strong> - We've identified the cause of the issue and have implemented a fix for the issue. We'll update here with more details shortly</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>11:35</var> GMT</small><br><strong>Update</strong> - We're implementing a process that should help with the issue, and will update on any impact of that here shortly<br /><br />We continue to advise not resending or reattempting any jobs at this time</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>11:21</var> GMT</small><br><strong>Update</strong> - We are still seeing slowness for system jobs, and are continuing to investigate.<br /><br />Reports and customer lists are running and completing, but slower than usual.<br /><br />Jobs such as Dotdigital Mailings, bulk creation of Opportunities and Activities and bulk tagging are not currently completing. <br /><br />We advise that you don't attempt to resend Dotdigital mailings or re-attempt jobs that aren't completing</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>11:04</var> GMT</small><br><strong>Update</strong> - We are continuing to investigate the issue and will update here again shortly</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>10:49</var> GMT</small><br><strong>Investigating</strong> - We're investigating some slower than expected report and customer list running times currently. Right now we're seeing these have some delays but these are currently completing. If you are waiting for a job or a mailing to finish it should complete so there is no need to run anything again.<br /><br />We're looking into the cause of these delays and will update further with more information shortly.</p>tag:status.spektrix.com,2005:Incident/203275772024-03-22T15:02:47+00:002024-03-22T15:02:47+00:00Reports of system slowness<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>15:02</var> GMT</small><br><strong>Resolved</strong> - After a period of continued close monitoring we're confident that there are no issues with the performance of the Spektrix system. <br />The broader network/internet provider related issues that some have experienced today appear to have resolved now based on the reports we've been receiving from you.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>13:00</var> GMT</small><br><strong>Monitoring</strong> - The majority of users who had reported issues are now seeing a return to normal speeds. <br />Our investigation hasn't found any problems with the Spektrix system itself, and we continue to believe based on the nature of the reports we received, that any issues were related to problems with the BT/Openreach network, and only affected those who use a BT/Openreach affiliated ISP.<br /><br />We are continuing to closely monitor on our side and if there are any further problems we will update here.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>11:47</var> GMT</small><br><strong>Investigating</strong> - We have received several reports of slowness accessing the Spektrix system and also clients' websites.<br />We currently don't believe there are any issues directly with our system or infrastructure, but we are investigating further.<br /><br />We have seen that BT has been having issues in certain locations that matches the locations of clients who have reported slowness - right now we believe this will be the cause of any problems you are experiencing.<br /><br />If you know you do not use BT as an internet service provider, and are experiencing problems, please do get in touch with as much information as possible to aid our own investigation.</p>tag:status.spektrix.com,2005:Incident/198174182024-01-24T14:24:00+00:002024-01-24T14:24:00+00:00Images, Local Settings and Ticket Subscriptions currently unavailable<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>14:24</var> GMT</small><br><strong>Resolved</strong> - After monitoring the system, we're confident that these issues are resolved. As ever, our team are available via email at support@spektrix.com, or on the phone via +44 (0)20 7183 3586 / +1 (646) 569-9097 if you require further assistance.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>12:58</var> GMT</small><br><strong>Monitoring</strong> - We've identified and resolved the issue with images, chip and pin machines, printers and Ticket Subscriptions not being available again for you today. We'll continue to monitor throughout the rest of the day. Thanks as ever for your patience, and if you require any other assistance today please do get in contact with us at support@spektrix.com or via +44 (0)20 7183 3586 / +1 (646) 569-9097.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>12:45</var> GMT</small><br><strong>Investigating</strong> - We're investigating similar issues that were experienced earlier today with Images, Local Settings and Ticket Subscriptions being unavailable both in your Spektrix and Online. We're looking into this and will be update you again as soon as we can.</p>tag:status.spektrix.com,2005:Incident/198157092024-01-24T10:31:55+00:002024-01-24T10:31:55+00:00Images, Local Settings and Ticket Subscriptions currently unavailable<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:31</var> GMT</small><br><strong>Resolved</strong> - Thank you all for your patience whilst we resolved the issues of Images, Printers and Chip and Pin machines and Ticket Subscriptions not appearing as expected. If you notice anything strange or unexpected, please don't hesitate to contact us at support@spektrix.com, or via +44 (0)20 7183 3586 / +1 (646) 569-9097 where we'll be more than happy to help.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:15</var> GMT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:14</var> GMT</small><br><strong>Monitoring</strong> - All images, printers, chip and pin machines and Ticket Subscriptions are now available and appearing as expected! We'll continue to monitor our solution closely throughout the rest of the morning. Thanks as ever for your patience whilst we worked through the issues this morning.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:06</var> GMT</small><br><strong>Update</strong> - We're still investigating the unavailability of images, printers, chip and pin machines and Ticket Subscriptions. All other functions of your Spektrix system (including running reports and customer lists) remain accessible.<br /><br />If you are using a multi-area seating plan, you can still navigate to the different areas by using the drop down menu above the image to change between your different seating areas.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>09:48</var> GMT</small><br><strong>Investigating</strong> - We are continuing to investigate images for Events and Seating Plans being unavailable and Printers and Chip and Pin machines showing as offline in Local Settings. We are also now aware that Ticket Subscriptions are not available in both the Sales and Admin Interface, as well as Online. We'll continue to update here as we know more.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>09:32</var> GMT</small><br><strong>Identified</strong> - We are also aware that Printers and Chip and Pin devices aren't visible from your Local Settings Interface. We believe that we have found the cause of the issue, and are actively working on restoring functionality.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>09:25</var> GMT</small><br><strong>Investigating</strong> - We are currently investigating an issue where images for events and seating plans are currently unavailable. This means that if you're using an image overlay, this will be inaccessible in both the Sales Interface and Online. We'll update you as soon as we know more.</p>tag:status.spektrix.com,2005:Incident/192627272023-12-01T16:55:00+00:002023-12-01T16:55:00+00:00Slowness running reports and other jobs<p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>16:55</var> GMT</small><br><strong>Resolved</strong> - We're happy now this issue has fully resolved and all reports and jobs are back to normal speeds. Thanks for your patience as we worked on this one.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>16:42</var> GMT</small><br><strong>Monitoring</strong> - Our fix has seen all jobs return to normal speeds now and are completing successfully. We are continuing to monitor this and will update further when we are happy the issue is fully resolved.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>16:35</var> GMT</small><br><strong>Update</strong> - We're seeing some initial improvements to the slowness with reports and jobs and are continuing to work to improve this further as we work on the fix for this. We'll continue to update on our work to resolve this here.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>16:24</var> GMT</small><br><strong>Identified</strong> - We believe we have identified the cause of this issue and are currently working through the best way to resolve it. As well as reports currently other system jobs, such as autotags, overlay updates, and integrated mailings will also be delayed. We still expect these to complete but after a longer than usual run time.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>16:10</var> GMT</small><br><strong>Investigating</strong> - We're investigating some slower than expected report and customer list running times currently. Right now we're seeing these have some delays but these are currently completing. If you are waiting for a job to finish it should complete so there is no need to run anything again.<br /><br />We're looking into the cause of these delays and will update further with more information shortly.</p>tag:status.spektrix.com,2005:Incident/189344112023-10-26T17:43:10+01:002023-10-26T17:43:10+01:00Issues with reports and customer lists running slowly<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>17:43</var> BST</small><br><strong>Resolved</strong> - We are confident that the time taken to run these reports and customer lists has returned to normal. <br /><br />Thank you for your patience whilst we worked to resolve this.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>16:35</var> BST</small><br><strong>Monitoring</strong> - There should now be no delays when running reports and customer lists. In addition, tasks such as bulk adding tags and updating overlays will now also be running normally.<br /><br />We will continue to monitor this closely.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>16:08</var> BST</small><br><strong>Update</strong> - We have continued to see improvements in the speed of these tasks completing within the Spektrix system.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>15:50</var> BST</small><br><strong>Identified</strong> - We have identified the cause of the slowness when running these system jobs, and have seen significant improvement in the last 15 minutes with the speed taken to run reports and customer lists. <br /><br />You should now start to see an improvement in the time taken to run bulk tagging jobs, BCC emails and dotdigital sends.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>15:24</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this, as there is still slowness running reports and updating customer lists.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>14:52</var> BST</small><br><strong>Update</strong> - We are continuing to investigate this. You should now be starting to see an improvement in the time it takes to run reports and update customer lists.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>14:04</var> BST</small><br><strong>Investigating</strong> - We are seeing issues with some jobs from the system, such as the running of reports and customer lists, taking a long time to run.<br /><br />We are investigating this at the moment and will update you shortly.</p>tag:status.spektrix.com,2005:Incident/188546172023-10-19T16:19:49+01:002023-10-19T16:19:49+01:00Issues with reports and customer lists running slowly<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>16:19</var> BST</small><br><strong>Resolved</strong> - There should now be no delays when running reports and customer lists. In addition, tasks such as bulk adding tags and updating overlays will now also be running normally.<br /><br />Thank you for your understanding whilst we looked into this.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>15:48</var> BST</small><br><strong>Monitoring</strong> - We are seeing issues with some jobs from the system, such as the running of reports and customer lists, taking a long time to run.<br /><br />We have identified the issue and implemented a fix. You should start to see improvements within the next few minutes.</p>tag:status.spektrix.com,2005:Incident/187311482023-10-09T14:59:51+01:002023-10-09T14:59:51+01:00Spektrix system inaccessible<p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>14:59</var> BST</small><br><strong>Resolved</strong> - Between 12:15 and 1:40 PM (BST) today Spektrix was inaccessible. This affected system users, customers booking tickets online and our APIs. <br /><br />Users attempting to access Spektrix during this time saw a 'Connection Timed Out' error or similar message. Customers were not able to complete a booking online via your website. <br /><br />We identified the cause of the problem shortly after the outage began. Our team developed a fix which was rolled out to the Spektrix system at 1:29 and system access was fully restored by 1:40 PM. After that time, users and customers could access Spektrix as normal and our APIs functioned as usual. We are confident that the system is functioning as normal.<br /><br />During the outage, scheduled Dotdigital emails, reports and customer lists will have continued to run successfully. Open transactions, or reservations which were due to time out will now have timed out. <br /><br />We apologise again for the disruption, and thank you for your patience whilst we looked into this. If you require any further assistance, please reach out at support@spektrix.com or call us on +44 (0)20 7183 3586</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>13:56</var> BST</small><br><strong>Monitoring</strong> - Further investigations confirms that report schedules and customer list schedules will have continued to run successfully whilst the system was down. Open transactions, or reservations which were due to time out, will now begin timing out. <br /><br />We are continuing to monitor the system closely and are confident that all system areas are functional.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>13:44</var> BST</small><br><strong>Update</strong> - We have now fully implemented the fix across the Spektrix system and our monitoring and testing shows that this has been successful. You should now be able to log in to your system, and iframes and the Spektrix API are now accessible again.<br /><br />We are confirming whether there are any further implications of the system having been down, and will update on this shortly.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>13:29</var> BST</small><br><strong>Update</strong> - The initial part of the fix has seen positive improvements, with some system access being restored. We're now working on ensuring we can apply this more widely across the Spektrix system.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>13:09</var> BST</small><br><strong>Update</strong> - We are implementing a fix that we believe will have a positive impact on the system, however we aren't yet able to guarantee this fix will work and we may need to deploy further changes afterwards.<br /><br />We are aware of the impact that this will be having on you, and are working to restore access as soon as we are able.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>12:40</var> BST</small><br><strong>Update</strong> - Currently, the entire Spektrix system (including the iFrames and the Spektrix API) is unavailable. You may be seeing a "Connection Timed Out" error page.<br /><br />We are investigating this as a priority and working to restore access to the system as soon as possible. We will update further as soon as we are able.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>12:30</var> BST</small><br><strong>Update</strong> - When attempting to log into or navigate around the Spektrix system, you may see an error page that says the connection has timed out. <br /><br />We are working on restoring access to the system as a priority.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>12:22</var> BST</small><br><strong>Identified</strong> - We are aware the the Spektrix system is currently inaccessible. We have identified the cause of this and are working on resolving this as a priority.</p>tag:status.spektrix.com,2005:Incident/186770782023-10-03T16:08:15+01:002023-10-03T16:08:15+01:00Issues confirming payments online<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:08</var> BST</small><br><strong>Resolved</strong> - This issue has now been resolved - online payments have been going through successfully since 12.25pm BST.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>12:53</var> BST</small><br><strong>Monitoring</strong> - We're aware that from 12.15pm BST to 12.25pm BST there was an issue where some online transactions were not processing successfully. Opayo have now confirmed that this issue has been resolved and payments attempted after 12.25pm should all have gone through successfully. We'll continue to monitor the situation and will update this page if there are any changes.<br /><br />You may find that there are some open transactions from this time on your system with pending payments. Please check your MyOpayo portal to cross-reference the details and confirm whether the payment was successful before taking any action.</p>tag:status.spektrix.com,2005:Incident/185360632023-09-19T13:00:52+01:002023-09-19T13:00:52+01:00Cloudflare errors across Spektrix system<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>13:00</var> BST</small><br><strong>Resolved</strong> - We have continued monitoring this, and have not seen a recurrence of these Cloudflare errors since 12:15 BST / 07:15 EDT. <br /><br />These problems started around 4pm BST / 11am EDT yesterday, 18 September. We initially understood Cloudflare to have identified the problem and implemented a fix, but they identified an additional issue around 10:30 BST / 05:30 EDT this morning, which has now been resolved. <br /><br />Thank you for your understanding and patience whilst we we worked to resolve this. If you are still experiencing any specific issues, please reach out to our Support teams on support@spektrix.com or on +44 (0)20 7183 3586 (UK) or +1 (646) 569-9097 (North America)</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>11:39</var> BST</small><br><strong>Monitoring</strong> - We have been closely monitoring the frequency of errors following the fix implemented by Cloudflare, and we are confident that this has significantly reduced the frequency of error screens that you and your customers may have been seeing. We have also seen API requests complete without error and websites be updated successfully. <br /><br />Thank you for your patience throughout this.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>11:04</var> BST</small><br><strong>Update</strong> - Cloudflare have been able to identify a further problem which we believe is the cause of the issues that you have been seeing this morning. They are implementing a fix, and we have already seen a large reduction in the number of errors.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>10:26</var> BST</small><br><strong>Update</strong> - We've investigated further this morning, and we believe that clients in Scotland may be adversely affected, due to how their web traffic is being routed. If you are able to alter your routing (e.g. using 4G or a VPN), you may see your connection become more reliable. <br /><br />We have also seen some issues with web integrations displaying incorrect dates or wrong availability. Some API requests are occasionally getting error responses at the moment.Please loop in your web developer who should be able to force another API request in order to retrieve the correct data.<br /><br />We are continuing to work with Cloudflare to escalate this issue, and we thank you for your patience and understanding.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>08:35</var> BST</small><br><strong>Update</strong> - Cloudflare, one of our infrastructure providers, are still experiencing problems this morning, and so users may still see error pages sporadically throughout the Spektrix interfaces, and customers may occasionally be unable to load iframes. Refreshing the page should resolve this. <br /><br />These problems started around 4pm BST / 11am EDT yesterday, 18 September. We initially understood Cloudflare to have identified the problem and implemented a fix, but as we're still seeing reports of this error, we are continuing to reach out to them on this.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>22:02</var> BST</small><br><strong>Update</strong> - Update: We are aware that some users are still having issues connecting to Spektrix, and may be experiencing cloudflare error pages. This may show as 'Error 1016 - Origin DNS error'.<br /><br />We are continuing to investigate this and will update you as soon as we are able.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>17:29</var> BST</small><br><strong>Identified</strong> - Our investigations have identified that this is related to a problem that Cloudflare are experiencing. Cloudflare are one of our infrastructure providers, which is why some users are experiencing these error pages. <br /><br />We have spoken to Cloudflare, who have been able to identify the issue and they are currently implementing a fix. In the meantime, these errors usually resolve upon a refresh of your page.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>16:55</var> BST</small><br><strong>Update</strong> - We are continuing to investigate the cause of some users having difficulties accessing the Spektrix system. We are finding that these are sporadic errors, that usually resolve upon a refresh of your page.<br /><br />We are continuing to investigate this issue and will provide updates as we have them.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>16:30</var> BST</small><br><strong>Investigating</strong> - We are aware that some users are having issues connecting to Spektrix, and may be experiencing cloudflare error pages. This may show as 'Error 1016 - Origin DNS error'.<br /><br />We are currently investigating this and will update you as soon as we are able.</p>tag:status.spektrix.com,2005:Incident/184344492023-09-08T16:29:16+01:002023-09-08T16:29:16+01:00Slowness with system jobs<p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>16:29</var> BST</small><br><strong>Resolved</strong> - After a period of monitoring we can see that there have been no further delays with the running of system jobs (such as reports and customer lists) for the past 2 hours, so we are going to mark this issue as resolved. <br />We will continue to closely monitor the system and will let you know if there are any further problems.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>14:08</var> BST</small><br><strong>Monitoring</strong> - We're confident that the current level of impact is minimal and that for the vast majority of cases, reports are running as normal, as are other system jobs such as customer lists, mailings and bulk updates.<br />We're continuing to monitor this and are in conversations with our infrastructure provider to get further information on the root cause of the issue.<br />We will update here if there are any changes in impact and when we are happy the issue is fully resolved.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>13:39</var> BST</small><br><strong>Update</strong> - We are still investigating the causes of this issue.<br /><br />The current impact is that all bulk-update type jobs should now run normally with no delays.<br />Reports and customer lists will generally run with no delays, however intermittently you may find that a report you run will take 1-2 minutes longer than it would normally.<br /><br />We will continue to provide further updates on this.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>13:07</var> BST</small><br><strong>Investigating</strong> - We are investigating issues with jobs in the system being slow to run. These include running reports, customer lists, and performing bulk activities such as bulk adding tags or updating overlays.<br /><br />These tasks are all currently completing but taking longer than expected to do so. We would currently advise not to 're-run' a task or job and instead wait for the current one to complete.<br /><br />We will provide further updates on this shortly.</p>tag:status.spektrix.com,2005:Incident/183701812023-09-04T09:28:13+01:002023-09-04T09:28:13+01:00Issue with taking payments for Canadian users<p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>09:28</var> BST</small><br><strong>Resolved</strong> - On 3 September at 7:58pm EDT Moneris were able to resolve the Gateway issues and return transaction processing to normal. You may have experienced some transaction slowness while their systems catch up, but there should no longer be transaction time-outs.<br /><br />If you require any further support on this, please reach out on support@spektrix.com or call us on +1 (646) 569-9097</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>15:16</var> BST</small><br><strong>Update</strong> - Moneris are continuing to work on a solution for the issue with payments for users in Canada. Users who are impacted may experience slowness or intermittent timeouts when processing transactions.</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>13:07</var> BST</small><br><strong>Update</strong> - Moneris are continuing to work on restoring normal payment services for users in Canada. They will continue to update the status of this on their Status Page: https://www.moneris.com/servicestatus</p><p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>10:52</var> BST</small><br><strong>Identified</strong> - At around 6:42am BST / 1:42am EDT we became aware of an issue preventing users in Canada from taking successful payments. This issue has been identified by Moneris, one of our payment providers. They are working to restore services and are providing updates on their status page: https://www.moneris.com/servicestatus</p>tag:status.spektrix.com,2005:Incident/183505692023-09-04T08:30:43+01:002023-09-04T08:30:43+01:00Issues with reports and lists not running<p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>08:30</var> BST</small><br><strong>Resolved</strong> - We are happy this issue with reports and lists running slowly was successfully resolved on Friday.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>18:00</var> BST</small><br><strong>Monitoring</strong> - For the majority of cases, reports and customer lists are running successfully. We are still undertaking monitoring and root cause analysis work on this and will update once we are confident the issue is resolved or if there are any further delays.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>17:33</var> BST</small><br><strong>Identified</strong> - We are seeing improvements with report speeds now, however there may still be some delays with these taking place.<br /><br />We are continuing to work on this issue and will update further shortly.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>16:57</var> BST</small><br><strong>Investigating</strong> - We have identified further issues with reports and lists not running following on from the problems earlier in the day. Running reports will not complete currently and continue to look as if they are generating.<br />We are investigating this as a high priority and will continue to update with more news shortly.</p>tag:status.spektrix.com,2005:Incident/183507602023-09-01T17:47:31+01:002023-09-01T17:47:31+01:00Another issue with postcode look-up<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>17:47</var> BST</small><br><strong>Resolved</strong> - We've continued to monitor this problem with UK postcode look-up and address auto-fill and it is resolved.<br /><br />An error was appearing for people trying to use postcode look-up between around 17:00-17:20 BST. <br /><br />The problem was related to the earlier issue, but was not exactly the same. We have been in communication with the partner who provides postcode look-up and we are both confident these problems are resolved.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>17:21</var> BST</small><br><strong>Monitoring</strong> - Postcode look-up and address auto-fill is working again now. We will continue to monitor this today and through the weekend. <br /><br />We're sorry for the bother this has caused you and your customers.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>17:14</var> BST</small><br><strong>Investigating</strong> - We are currently investigating another issue with the UK postcode look-up and address auto-fill. This means an error will be coming up for Spektrix users and customers when they try to use postcode look up when creating a customer record or adding an address. Addresses can still be added manually.</p>tag:status.spektrix.com,2005:Incident/183461432023-09-01T13:51:17+01:002023-09-01T13:51:17+01:00Postcode Lookup not functioning - UK<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>13:51</var> BST</small><br><strong>Resolved</strong> - There have been no more problems with UK/Ireland postcode look-up and address auto-fill since around 9:40 BST this morning. <br /><br />From around 01:00, our partner who provides the postcode and address service experienced an issue which led to the "no credit" error you and your customers will have seen when setting up a new customer record or adding a new address.<br /><br />Please reach out to support@spektrix if you have any questions.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>09:43</var> BST</small><br><strong>Monitoring</strong> - We have now resolved the issue with the postcode look-up and address auto-fill. <br /><br />Between 01:00-09:40 BST, an error about "no credit" was appearing when using postcode look-up when creating a customer or adding a new address. This error is no longer appearing and postcode look-up and address auto-fill is working as expected.<br /><br />We will continue to monitor this through the day.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>09:21</var> BST</small><br><strong>Identified</strong> - Since approximately 01:00 BST the 3rd party system we use to provide a postcode lookup and auto-fill service hasn't been functioning.<br /><br />This means customers will need to manually enter their address when making purchases online. They may see an error message relating to 'no credit' when entering their postcode, this does refer specifically to the postcode lookup and not anything to do with the customer's credit.<br /><br />We are working with the 3rd party provider to resolve this asap and will let you know when this has been fixed and the postcode look up is functioning again.</p>tag:status.spektrix.com,2005:Incident/183474652023-09-01T13:48:54+01:002023-09-01T13:48:54+01:00Issues with reports and lists running slowly<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>13:48</var> BST</small><br><strong>Resolved</strong> - This issue has now been resolved and all jobs on the system should be running normally.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:25</var> BST</small><br><strong>Monitoring</strong> - There should now be no delays when running reports and customer lists. There may still be some delay in running some other tasks, such as bulk adding tags and updating overlays. We'll post updates on these tasks here shortly.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:19</var> BST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:15</var> BST</small><br><strong>Update</strong> - We have identified the issue and are working to fix this. Tasks such as running reports, customer lists, and updating overlays are taking longer than normal. We'll add further updates here soon</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>11:48</var> BST</small><br><strong>Investigating</strong> - We are seeing issues with some jobs from the system, such as the running of reports and customer lists, taking a long time to run.<br />We are investigating this now and will provide further updates shortly.</p>tag:status.spektrix.com,2005:Incident/182590452023-08-25T16:20:35+01:002023-08-25T16:20:35+01:00Larger than usual amount of declined online payments - Mastercards only<p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>16:20</var> BST</small><br><strong>Resolved</strong> - After closely monitoring this issue for the last few hours we're happy that Mastercard have been able to resolve their authorisation issue. We have seen a normal level of declined transactions since approximately 13:00 BST. We will mark this incident as resolved but if we receive any further reports or notice another uptick in declines we will send further information.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>13:59</var> BST</small><br><strong>Monitoring</strong> - We have been consistently seeing a much reduced level of declined payments in the last hour, that matches the number of declines we would usually expect. <br /><br />We are going to continue monitoring as we have yet to hear definitive confirmation that the Mastercard authorisation issues have been resolved but from what we can currently see everything is now operating as we would expect.<br /><br />We will be in touch as we have further updates on this.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>13:35</var> BST</small><br><strong>Identified</strong> - We have been able to isolate this problem as part of a wider issue with authorisations specifically for customers using Mastercards.<br /><br />We are also seeing that the decline rates are now dropping, so we believe the issue is improving. We are yet to receive concrete confirmation of this though so we are remaining in dialogue with our payment processor and will pass on any further updates.<br /><br />Currently if a customer has issues we would still recommend trying to take payment again, and if possible to try using a Visa card instead.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>12:58</var> BST</small><br><strong>Investigating</strong> - We are aware that since approximately 11:00 BST there have been a larger than usual amount of online payments being marked as declined with a 'declined by the bank' error message. Many payments are still going through successfully, so right now we advise customers try again if they have issues. In person and phone payments appear to be unaffected currently.<br /><br />We are investigating this with all appropriate parties and will provide further updates once we have them.</p>tag:status.spektrix.com,2005:Incident/180185352023-08-01T11:37:07+01:002023-08-01T11:37:07+01:00Delays with Scheduled Jobs<p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>11:37</var> BST</small><br><strong>Resolved</strong> - We are happy after running tests and monitoring the system that all scheduled jobs have caught up and are running on time now. As ever we will continue to monitor the system and if you notice any further issues please do contact us and we will investigate further.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>11:10</var> BST</small><br><strong>Monitoring</strong> - We're happy that the changes we made are working and the jobs that have been delayed through the cause of the night are being scheduled. There will still be some delays as these catch up, but we are expecting these all to send and for these to start to catch up quickly.<br />We'll continue to monitor this on our side and keep you updated on the progress of this.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>10:39</var> BST</small><br><strong>Update</strong> - We are continuing to work on a fix for the issues we are experiencing with some scheduled jobs (such as reports, and scheduled emails) being delayed. We are implementing a fix now - this may cause some slowness on some clicks in the system for the next few minutes but this will be temporary. We will provide a further update again shortly.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>10:19</var> BST</small><br><strong>Identified</strong> - We have now confirmed that there is definitely an issue and we are working on the next steps to getting this resolved.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>10:04</var> BST</small><br><strong>Investigating</strong> - We are currently investigating delays with scheduled jobs since approximately 2amBST. At the moment we are seeing that the majority of reports are arriving, but with a significant delay. No other aspects of the system are affected, we'll keep you informed with any updates.</p>tag:status.spektrix.com,2005:Incident/179445312023-07-24T11:55:55+01:002023-07-24T12:42:30+01:00Problems confirming payments online<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>11:55</var> BST</small><br><strong>Resolved</strong> - We've noticed that from c.9:30 BST some customers may have experienced problems confirming payments online. These may have appeared in your MyOpayo portal (formerly MySagepay) as failures due to fraud rules. Customers will not be charged for these failed transactions.<br /><br />We've spoken to Opayo who have confirmed that whilst there was a problem this morning, this has been resolved as of 11:15 BST. Payments made after this time should be successful. <br /><br />If you require further help or support on this, please reach out on support@spektrix.com</p>tag:status.spektrix.com,2005:Incident/177716222023-07-05T20:17:58+01:002023-07-05T20:17:58+01:00Reports of system slowness<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>20:17</var> BST</small><br><strong>Resolved</strong> - We're confident now that all issues are resolved and the Spektrix system is operating as normal following on from our infrastructure provider declaring everything to be resolved at their side.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>17:48</var> BST</small><br><strong>Monitoring</strong> - We believe after communicating with our infrastructure provider that the slowness issues should now be resolved.<br />We are continuing to monitor the system and keep track of any reports of further issues so please let us know if you continue to encounter slowness on your system.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>16:09</var> BST</small><br><strong>Investigating</strong> - We're currently investigating reports of system slowness. This seems to be affecting customers online as well as the Spektrix system.<br /><br />We believe this is due to issues with our third party infrastructure provider, and are coordinating with them to resolve this as soon as possible.<br /><br />You should be able to complete system actions as normal, but with some delays.</p>tag:status.spektrix.com,2005:Incident/175481542023-06-12T17:18:58+01:002023-06-12T17:18:58+01:00Intermittent Mastercard payment issues<p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>17:18</var> BST</small><br><strong>Resolved</strong> - Our partners have advised that the issues processing Mastercard 3D secure challenges are now resolved, and transactions should be processing again as normal. <br /><br />If you have any further questions on this, please don't hesitate to contact Support at support@spektrix.com.</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>15:29</var> BST</small><br><strong>Identified</strong> - We've identified issues processing some Mastercard payments, specifically a high rate of failures on 3D secure challenges. We understand this to be a Mastercard wide issue. <br /><br />We'll continue to work with our partners on this, but would advise that if customers are encountering issues, that they attempt a different card.</p>tag:status.spektrix.com,2005:Incident/172593812023-05-18T16:43:10+01:002023-05-18T16:43:10+01:00Some donations may not be visible under customer profile<p><small>May <var data-var='date'>18</var>, <var data-var='time'>16:43</var> BST</small><br><strong>Resolved</strong> - We have been able now to fix the issue with the behaviour above and all donation history on a customer profile is now fully visible again and ordering correctly. We will continue to monitor the situation here but mark this as a resolved status page. If there are any recurrences we will communicate this as required.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>16:46</var> BST</small><br><strong>Update</strong> - We have resolved the issue of error screens when clicking into the Orders tab on a Customer Record, and are confident that this will not reoccur. <br /><br />We are continuing to work on a resolution to the missing donation information, and will update you on this tomorrow.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>16:32</var> BST</small><br><strong>Update</strong> - We released a potential fix to this in the last half hour, but since implementing this we have noticed that this has caused some error screens when accessing the Orders tab on a Customer record. We have identified the cause of this and are implementing a fix for this now. We will update you once we have completed this and are confident it is resolved. <br /><br />We will continue working on a resolution to the missing donations information, and will update you on this tomorrow.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>17:39</var> BST</small><br><strong>Identified</strong> - The cause of the issue has been identified and we are taking steps to resolve it. We will continue to keep you updated as we work towards a resolution.</p><p><small>May <var data-var='date'>15</var>, <var data-var='time'>22:28</var> BST</small><br><strong>Investigating</strong> - We are currently investigating an issue that may cause you to only see partial donation history under the donations tab in the customer record. This issue is only impacting the donations tab in the customer record. All reports, customer lists, and segmentation using donation data are fully operational. We are working to resolve this issue now and will update you accordingly.</p>tag:status.spektrix.com,2005:Incident/171298222023-05-03T10:00:20+01:002023-05-03T10:00:20+01:00Delays in scheduled jobs<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>10:00</var> BST</small><br><strong>Resolved</strong> - From around 11pm BST/6pm ET yesterday to 8am BST/3am ET today, there were delays in system jobs running. This means, for example, that you would not have received scheduled reports during this time. <br /><br />All delayed jobs have now been scheduled and most completed, though these may have run in a slightly different order to usual. If you notice that a report doesn't contain the information that you were expecting, please try re-running this from the Insights & Mailings Interface. <br /><br />If you need any further support on this please don't hesitate to contact us by opening up a new request via support@spektrix.com or calling us on +44 (0) 20 7183 3586 or +1 (646) 569-9097.</p>tag:status.spektrix.com,2005:Incident/169843372023-04-27T19:03:13+01:002023-04-28T17:31:32+01:00Scheduled reports and E-tickets failed to run<p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>19:03</var> BST</small><br><strong>Resolved</strong> - Between 15:30 BST / 10:30 EDT and 18:30 BST / 13:30 EDT, schedule reports failed to run. If you had a report schedule set to be delivered between these times, this will have failed and will not be reattempted. We have identified the cause of this and put a fix in place, and our monitoring has shown that this was successful and schedules reports are now running. <br /><br />If you wish to resend any missed reports in this time, you would need to run the report manually from the Insights & Mailings Interface. If you have any questions around how to do this, please contact Spektrix Support on support@spektrix.com.</p>tag:status.spektrix.com,2005:Incident/169702962023-04-27T14:43:39+01:002023-04-27T14:43:39+01:00Issues with Dotdigital emails sending<p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>14:43</var> BST</small><br><strong>Resolved</strong> - We’ve been monitoring this alongside dotdigital for some time now, and we are confident that the issue is now fully resolved. Thank you for your patience and understanding whilst we worked to resolve the cause of this. <br /><br />If you need any further support on this please don't hesitate to contact us by opening up a new request via support@spektrix.com or calling us on 020 7183 3586.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>12:16</var> BST</small><br><strong>Monitoring</strong> - Dotdigital have let us know that they've managed to identify the cause of these emails failing to send, and have implemented a fix. We have seen an improvement since they implemented this fix, and will be closely monitoring the system to ensure that this continues. <br /><br />If you need any further support please don't hesitate to contact us by opening up a new request via support@spektrix.com or calling us on 020 7183 3586.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>11:15</var> BST</small><br><strong>Update</strong> - We continue to see some Dotdigital emails failing to send. While the majority of emails are sending successfully, we know that some of you are seeing multiple attempts to send emails fail. We understand how frustrating this is, and we are working with Dotdigital on understanding and resolving the cause of this issue.<br /><br />If you have any time sensitive communications to send, we advise you to check Dotdigital's Sent list to confirm whether the mailing has sent. If it failed, please retry sending this email. <br /><br />Dotdigital is working on resolving this issue as a priority. They are making progress in identifying and working towards a solution. We will provide further updates as soon as we have them.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>18:12</var> BST</small><br><strong>Update</strong> - From around 09:00 BST / 04:00 EDT we have experienced some dotdigital emails failing to send. While the majority of emails are sending successfully, we know that some of you are seeing multiple attempts to send emails fail. We understand how frustrating this is, and we are working with dotdigital on understanding and resolving the cause of these issues.<br /><br />If you have any time sensitive communications to send, we would advise you to check dotdigital to confirm whether the mailing has sent. If it failed, please retry sending this email. <br /><br />We will update you on this tomorrow morning.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>15:58</var> BST</small><br><strong>Update</strong> - We are still working with Dotdigital on understanding the issues some users are experiencing when attempting to send emails through Dotdigital. The vast majority of emails are being successfully sent, however this is sometimes taking multiple attempts to send.<br />We understand the frustration involved, and are working closely with Dotdigital to understand the cause of the issue and will provide further updates in due course.<br /><br />If you have any time sensitive communications to send, we would still advise you to continue to retry sending your email after checking within Dotdigital to confirm the mailing has failed to send and to avoid multiple sends.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>13:44</var> BST</small><br><strong>Update</strong> - We are currently still working on understanding the issue experienced by some when attempting to send emails through Dotdigital. The vast majority of emails are being successfully sent, however this is sometimes taking multiple attempts to send.<br />We understand the frustration involved, and are working closely with Dotdigital to understand the cause of the issue and will provide further updates in due course.<br /><br />If you have any time sensitive communications to send, we would still advise you to continue to retry sending your email after checking within Dotdigital to confirm the mailing has failed to send and to avoid multiple sends.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>12:26</var> BST</small><br><strong>Update</strong> - We're still working on understanding the issues some of your are seeing when sending emails via Dotdigital. Whilst the majority of emails are being sent, we know that some of you are seeing multiple attempts to send emails fail and we completely understand how frustrating this is.<br /><br />If you have any time sensitive communications to send, our current advice is to continue to retry sending your email after checking within Dotdigital to confirm the mailing has failed to send.<br /><br />We are still working closely with Dotdigital to understand the cause of the issue here and will provide further updates in due course.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>11:31</var> BST</small><br><strong>Update</strong> - We're still working to understand this problem with mailings failing to send.<br /><br />We are seeing that retries are often successful and that not every email is failing. We are working to establish the cause of this and will continue to update here as we work on this with Dotdigital.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>10:48</var> BST</small><br><strong>Update</strong> - We're continuing to look at this with Dotdigital to understand a cause. It looks like in the last 20 minutes all mailings have sent successfully but as we are still looking into what the underlying issues are we aren't yet able to say that this issue is resolved. We will continue to update here as we know more.</p><p><small>Apr <var data-var='date'>26</var>, <var data-var='time'>10:32</var> BST</small><br><strong>Investigating</strong> - We're investigating issues with Dotdigital emails not sending and working with them to resolve this.<br />We think this may have started at 9am BST with a small number of failures and some 'false flag' errors where mailings did send. From 10am BST we have started seeing a much larger number of failures.<br />We're talking with Dotdigital and will provide further updates on this as we have them</p>