We are continuing to monitor for any further issues.
Posted Mar 13, 2023 - 23:16 UTC
Update
Currently, we have fixed the software issues and resumed server operations. We are now monitoring the system to ensure stability under traffic.
Posted Mar 13, 2023 - 23:15 UTC
Update
We are actively working to address the current issue, specifically the Kernel bug that we are facing. Our developers are currently working on patching the issue, and we will keep you updated on our progress. We will provide an estimated time of recovery as soon as possible.
Posted Mar 13, 2023 - 20:21 UTC
Update
There has been a recurrence of the issue in FRA7; the engineers are working to identify the root cause and patch the firmware. In the meantime, we have moved traffic away from Frankfurt as we continue to work.
Posted Mar 13, 2023 - 17:34 UTC
Update
We are continuing to monitor for any further issues.
Posted Mar 13, 2023 - 17:00 UTC
Update
We are continuing to monitor for any further issues.
Posted Mar 13, 2023 - 17:00 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 13, 2023 - 16:59 UTC
Update
The majority of our services in Frankfurt POP have been reinstated. During the initial reboot, we detected a bug linked to the server's NIC cards, and we had to mostly revert to an earlier version of the faulty firmware, which consumed some time. At present, we are keeping an eye on our entire system to ensure that all of the services are operational and performing as they were prior to the incident.
Posted Mar 13, 2023 - 16:59 UTC
Update
We have recovered the Database and API services at this POP. Working towards to get the mitigation system up and running right now.
Posted Mar 13, 2023 - 16:44 UTC
Update
We unfortunately had to reboot our devices again to make sure they will run a stable code. Meanwhile, we have rerouted most of the customers traffic away from FRA while we are working on a fix.
Posted Mar 13, 2023 - 15:39 UTC
Update
Due to the Kernel panic we had in this POP, it forced us to manually reboot some of our machines. They are currently booting back and we expect the traffic flow to be normal very soon.
Posted Mar 13, 2023 - 14:27 UTC
Update
We are continuing to work on a fix for this issue.
Posted Mar 13, 2023 - 13:57 UTC
Update
We are continuing to work on a fix for this issue.
Posted Mar 13, 2023 - 13:56 UTC
Identified
Because our Frankfurt Point of Presence (POP) has been rapidly expanding, and primarily due to a Linux kernel issue, we have encountered an unanticipated restriction on our routing equipment which is resulting in sporadic packet loss.
Posted Mar 13, 2023 - 13:56 UTC
This incident affected: Europe (Amsterdam, Netherlands, Frankfurt, Germany) and Remote DDOS Protection.