Update
We want to provide more transparency regarding this incident given its duration.
We were optimizing one of our largest databases, specifically the one storing comments and card activity. At the end of this process, a bug caused some database partitions to go live without their indexes. This means queries to those partitions are timing out, which is why approximately 40 percent of boards are experiencing failures.
Your data is safe. Any comments added during this incident are being saved and will reappear once the process is complete.
We are currently rebuilding the missing indexes. We have allocated significant compute resources to this task to move as quickly as possible, but the data volume is enormous. We estimate it will take approximately 20 more hours to reach full resolution.
Most impacted features include, but are not limited to:
- Comments (viewing, adding, notifications)
- Activity views on cards
- Attachments, Power-Ups, board exports
- Home page / 'Updates' view
- Search
- Some cards showing 'Card Not Found'
- Moving/copying cards or lists
Please avoid moving or copying cards between boards for now. If your board is working normally, you can continue your work, as most boards remain unaffected.
We know Trello is where your work lives and we are committed to getting everything back to normal. We will provide next update in eight hours or sooner as we make progress.
We were optimizing one of our largest databases, specifically the one storing comments and card activity. At the end of this process, a bug caused some database partitions to go live without their indexes. This means queries to those partitions are timing out, which is why approximately 40 percent of boards are experiencing failures.
Your data is safe. Any comments added during this incident are being saved and will reappear once the process is complete.
We are currently rebuilding the missing indexes. We have allocated significant compute resources to this task to move as quickly as possible, but the data volume is enormous. We estimate it will take approximately 20 more hours to reach full resolution.
Most impacted features include, but are not limited to:
- Comments (viewing, adding, notifications)
- Activity views on cards
- Attachments, Power-Ups, board exports
- Home page / 'Updates' view
- Search
- Some cards showing 'Card Not Found'
- Moving/copying cards or lists
Please avoid moving or copying cards between boards for now. If your board is working normally, you can continue your work, as most boards remain unaffected.
We know Trello is where your work lives and we are committed to getting everything back to normal. We will provide next update in eight hours or sooner as we make progress.
Update
Our teams continue to work diligently on the mitigation.
During this time, we recommend that you do not move or copy cards or lists between boards. For users whose boards are affected by this issue, you may create and delete cards and edit card descriptions as usual, but you will not be able to see comments or the detailed activity log of the cards.
We estimate that the rebuild will take another 24 hours to complete. We will update this page as we have more information.
During this time, we recommend that you do not move or copy cards or lists between boards. For users whose boards are affected by this issue, you may create and delete cards and edit card descriptions as usual, but you will not be able to see comments or the detailed activity log of the cards.
We estimate that the rebuild will take another 24 hours to complete. We will update this page as we have more information.
Update
Trello continues to experience slowdowns and failures that are affecting comment operations, including creating and deleting comments.
Our teams continue to work on the issue resolution, while it is taking longer than expected. We will provide more updates within the next 8 hours or sooner once we have further information to share.
Our teams continue to work on the issue resolution, while it is taking longer than expected. We will provide more updates within the next 8 hours or sooner once we have further information to share.
Update
Trello is experiencing slowdowns and failures affecting comment operations, including creating and deleting comments. Users may also experience delays when loading activity views on cards, viewing attachments, using Power-Ups, exporting boards and 'Updates' on home page.
Our team is actively working on the resolution and will likely take up to 7 hours to restore the services back to full functionality. We will provide further update at this time, or earlier if we have additional information to share prior to this resolution.
Our team is actively working on the resolution and will likely take up to 7 hours to restore the services back to full functionality. We will provide further update at this time, or earlier if we have additional information to share prior to this resolution.
Update
Trello is experiencing slowdowns and failures affecting comment operations, including creating and deleting comments. Users may also experience delays when loading activity views on cards, viewing attachments, using Power-Ups, or exporting boards. Some users are encountering errors when viewing specific cards.
The team is working hard to restore full functionality and will provide another update within 3h.
The team is working hard to restore full functionality and will provide another update within 3h.
Update
Work by the Trello engineering team continues to restore functionality to comments and activity views. Recovery of the affected systems is taking longer than anticipated.
Next update will be posted within 4h or on a significant change. Thank you for your patience.
Next update will be posted within 4h or on a significant change. Thank you for your patience.
Update
Work by the Trello engineering team continues to restore functionality to comments and activity views.
Next update will be posted within 4h or on a significant change. Thank you for your patience.
Next update will be posted within 4h or on a significant change. Thank you for your patience.
Identified
The team has identified the root cause and is hard at working resolving this issue. Users will see problems creating/updating/deleting comments and loading the activity view for cards at this time. The rest of Trello's features should be working as normal.
We will provide more updates within 2h.
We will provide more updates within 2h.
Investigating
Users may be experiencing slowdowns across various functions of Trello including creating/deleting comments, editing cards, or creating new attachments. The team is actively working to fix the issue and will provide more updates within the hour.
This incident affects: Trello.com and API.