While optimizing our edge routers for better performance and routing, we encountered a bug in our edge router controlling software that caused it to drop interfaces configuration on some of our major points of presence (PoPs) following a minor configuration change. This resulted in a loss of access, and we quickly opened remote hands support with our data center partners. The entire team split the work and reverted the bad configuration, and we then rebooted the devices. We apologize for the inconvenience this caused.
Although we had tested the new changes in our test environment and a few small PoPs without any issues, we are currently working internally to build a better testing plan and environment to prevent this type of outage from happening again.