US-based operator CenturyLink suffered a major IP outage on 30 August, affecting internet services around the world. The event followed a mis-configuration at one of its data centres and took seven hours to fix.
The error, which involved both the firewall and border gateway protocol routing, spread across CenturyLink’s network and ended up impacting other internet service providers, causing connectivity problems for many companies, including Amazon, Twitter, Microsoft (Xbox Live), EA, Blizzard, Steam, Discord, Reddit, Hulu, Duo Security, Imperva, NameCheap and OpenDNS.
Cloudflare, also severely impacted, noted that CenturyLink's outward-propagating issue led to a 3.5 percent drop in global internet traffic, which would make this one of the biggest internet outages ever recorded.
According to a CenturyLink status page, the issue originated from CenturyLink's data centre in Mississauga, near Canada’s Ontario. The cause was an incorrect Flowspec announcement, ZDnet reports.
Flowspec is an extension for the BGP protocol that allows companies to use BGP routes to distribute firewall rules across their network. Flowspec announcements are usually used when dealing with security incidents, such as BGP hijacks or DDoS attacks, as it allows companies to change their entire network to react and mitigate attacks within seconds. However, The Mississauga data centre sent out an incorrect Flowspec announcement that effectively prevented the company's BGP routes from taking place. Some of the incorrect Flowspec commands also began to announce incorrect BGP routes to other internet services.
CenturyLink fixed the issue by telling the other internet providers to de-peer, and ignore any traffic coming from its network. Companies rarely take these kinds of decisions, as this results in full connectivity loss for all its customers, ZDNet noted.
