AWS Outage Traced To Route Leak

An Amazon Web Services outage Tuesday night that affected Netflix and other popular sites was caused by a BGP route leak at a Boston-area based hosting and colocation provider.

In a service note on its website, AWS said the outage lasted about 40 minutes starting at 5:25 p.m. Pacific Time.

"The root cause of this issue was an external Internet service provider incorrectly accepting a set of routes for some AWS addresses from a third-party who inadvertently advertised these routes. Providers should normally reject these routes by policy, but in this case the routes were accepted and propagated to other ISPs affecting some end users' ability to access AWS resources," the company said.

ThousandEyes, a San Francisco-based network performance monitoring startup, traced the root cause to a BGP route leak from Axcelx in Somerville, Mass. The company did its own analysis after realizing its internal messaging system, SSO provider and corporate website were down.

ThousandEyes, which documented its analysis of the AWS outage in a blog post, found that Axcelx and Axcelx's ISP, Hibernia, were suddenly showing up in the path between its cloud agents in Dallas and New York to an application hosted in Amazon's data centers.

"All in all, the route leak affected a wide range of services including consumer internet sites like Yelp, Netflix and Match: SaaS services such as HipChat and Jobvite; and financial firms such as Experian and Zions Bank," wrote Nick Kephart, director of product marketing at ThousandEyes.

Reddit also was affected, according to a note posted by a reader that prompted a lengthy discussion about BGP route leaks.

Axcelx apologized for the route leak on Twitter. AWS said once it worked with the service provider to "ensure that this does not reoccur."

Other sites reportedly affected by the outage included Pinterest and collaboration tools Slack and Asana.