Trello is unavailable for some users

Incident Report for Trello

Postmortem

Starting at 2:15pm EDT AWS began experiencing significant disruption to many of its services, including interruptions to network connectivity on EC2 instances. This caused some requests to Trello.com to take longer than usual or timeout completely. The worst of the service degradation had passed by 2:40pm EDT while some lingering isolated affects continued, which we believed were the cause of the CDN errors that some of our users were reporting when trying to load Trello.

However, on further investigation and with the help of some details from Trello users on Twitter, we discovered that there was a problem with a recent release of the web client which affected approximately 10% of users' web sessions. The issue caused affected users to load an empty version of our core.css file, which caused Trello to display without any styling and include a message indicating that the browser was "unable to load all of Trello's resources." The mobile versions of Trello were unaffected. Once the issue was identified, we immediately reverted to an older stable version.

These two issues appear to be unrelated, but identifying and fixing the second issue took longer than we would have liked due to the increased noise in our monitoring systems from the AWS service disruption, and the initial assumption that all user issues were related to the AWS problems.

Posted Jul 19, 2016 - 16:48 EDT

Resolved

This issue is resolved. If you're continuing to experience issues loading Trello, please refresh your browser.

Thanks for bearing with us!
Posted Jul 19, 2016 - 16:09 EDT

Update

If you're continuing to see issues with Trello, refresh your browser.

While we're still investigating the root cause, we determined that one of the versions of our web client (available to a small percentage of our users) had an invalid file for core.css.

We'll post more details when we have them. For now, refresh your browser.
Posted Jul 19, 2016 - 16:05 EDT

Monitoring

Most Trello users will continue to be able to use the app without issue. For some users, Trello may be unavailable. We're continuing to monitor Amazon's status and will provide more updates here as we have additional information.

The latest from AWS is:

12:21 PM PDT We have identified the root cause for error rates accessing the EC2 APIs and EC2 Management Console and are currently working to resolve. We observed isolated periods of impaired network connectivity for some EC2 instances, however running instances are currently operating normally.
Posted Jul 19, 2016 - 15:40 EDT

Investigating

Amazon Web Services, which Trello uses to host our servers, is currently having some networking issues. Users may receive an error that their browser is unable to load Trello's resources and be unable to access Trello. In the hour prior to this issue, users may have had intermittent slow response times.

Our engineering team is actively investigating this incident and working to bring Trello back up as quickly as possible. We will update this page as we have additional information.
Posted Jul 19, 2016 - 15:01 EDT