Trello - degraded performance

Incident Report for Trello

Postmortem

Summary

On May 17th 2026 at 12:00 UTC a subset of Trello users experienced a degradation of some core functionality including but not limited to: issues with comments, viewing activity logs, accessing card details, board exports, Power-Ups, attachments, Home page updates, and search functionality. Some moved cards displayed 'Card Not Found' errors.

These issues persisted for multiple days. Most of the issues encountered when interacting with Trello (commenting, viewing activity) were resolved by May 19 2026 at 08:31 UTC, however there was a lingering data integrity issue which manifested as “Card Not Found” errors when trying to open cards. >90% of these cards were repaired by May 28 at 17:20 UTC. The remainder took a little more time and were completely repaired by June 5th 2026 at 21:37 UTC.

We wanted to share a more detailed analysis of the issues we encountered, as well as what we’re doing to ensure nothing similar happens in the future.

Root Cause

The incident was caused by a bug in our database (MongoDB) which surfaced during a project to optimize the performance and scale of one of our largest collections. The affected collection contains card comments and other activity and is used across much of Trello’s functionality.

The optimization project was to reshard the collection onto a new key. This allows us to eliminate a so-called “insert hot spot” which is where all newly created database entries end up on a single database partition (aka shard). After resharding, the load would be spread evenly across the various servers that make up the database cluster. This decreases latency and improves query performance for users.

During the final stages of the resharding operation (which is a long-running, automated process), 12 of the 30 relevant shards ran out of disk space while building indexes. Rather than aborting the resharding process, the database bug caused the cluster to put the new database servers into rotation without their indexes, despite logs and metrics saying otherwise.

Due to the enormous amount of data in this collection, having no indexes means that there is essentially no way to query for the data you want (i.e. “give me all comments for this card”) without performing an extremely slow and inefficient collection scan.

Remedial Actions Plan

The first thing we did was initiate a full rebuild of all the database indexes for the collection. Unfortunately this takes a very long time to complete (multiple days) and is compute-intensive. We scaled all the database servers up in order to help the process move faster, however due to the single-threaded nature of this process, more compute did not equate to more speed.

Additionally we enabled a database setting which caused all queries that would result in a full collection scan to fail-fast rather than drag on in the background and consume database resources before ultimately failing. This helped the parts of Trello that were still working stay responsive.

Once the reindex process had completed, all functionality was restored. However, we discovered that cards which were moved to a different board during the rebuild were left in a broken state. Their activity logs and comments were essentially stranded on the old board while the card was now on the new board. This manifested as a “Card Not Found” error when trying to open such a card.

The team moved into a new phase of writing code to safely remediate all of the remaining data issues in batches, with the final batch of fixes completing by June 5th 2026 at 21:37 UTC.

Next Steps

We know that Trello is where your work lives and issues like this can cause a lot of disruption.

To ensure we don’t run into this issue again in the future, the team is taking the following actions:

  • Working closely with our database vendor to ensure the original bug is fixed (update: the fix is shipping in the next version)
  • Working to create a standard set of tools and processes to ensure index rebuilds proceed as fast as possible in an emergency like this
  • Designing and implementing changes to make card move operations more atomic (either the whole operation succeeds, or the whole operation fails, rather than a partial failure)
Posted Jun 26, 2026 - 15:42 EDT

Resolved

Thank you for your patience as our team worked diligently to resolve outstanding issues.

Although our team encountered edge scenarios through the weekend, we believe all 'card not found' issues are resolved. Minor data inconsistencies listed below may be present if cards were moved between boards during the incident:
- Missing card labels
- Attachment restrictions not being enforced on moved cards
- Checklist assignee information on some cards
- Incorrect due date reminders sent to some card assignees

Please be assured that our team continues the work necessary to resolve above issues; however, given the impact and scope of remaining issues, we are now moving the channel of communication to Support rather than this Statuspage.


If you believe that you continue to see lingering impacts from this incident, please do not hesitate to reach out to our Support team for further assistance.
Posted May 26, 2026 - 17:12 EDT

Monitoring

Our team is still working to fix minor data inconsistencies on cards that were moved between boards during the incident. The following are still being addressed:

-Missing card labels
-Attachment restrictions are not being enforced on moved cards
-Checklist assignee information on some cards.
-Incorrect due date reminders sent to some card assignees

Your cards are safe to edit. If you make changes to a card before our automated fix reaches it, your edits will be preserved, and the automated restore for that card will be skipped. No data will be lost in either scenario.

We will provide another update once the above items are fully resolved. If you need assistance in the meantime, please reach out to our support team.
Posted May 26, 2026 - 09:27 EDT

Investigating

The team continues to remediate potential outstanding data issues that occurred on Boards and Cards during the incident
Posted May 26, 2026 - 09:27 EDT

Update

Users should be experiencing restored functionality to their boards and cards.

Our team has resolved 'card not found' errors, and continues to remediate potential outstanding data issues that occurred on Boards and Cards during the incident:
- Missing card labels
- Attachment restrictions not being enforced on moved cards
- Checklist assignees that are not a member of the current board may incorrectly linger
- Card assignees that are not a member of the current board may have incorrectly received due date reminders

Please note that while cards are safe to edit, they may be excluded from certain restoration work if they are edited.

We will provide another update when above issues are resolved. In the meantime, please reach out to our support team if you need further assistance.
Posted May 22, 2026 - 15:51 EDT

Update

The team continues to remediate potential outstanding data issues that occurred on Boards and Cards during the incident (details are in yesterday's update).

Users should be experiencing restored functionality to their boards and cards. Please note that while cards are safe to edit, they may be excluded from certain restoration work if they are edited.

We greatly appreciate your patience as we work through these issues. Please reach out to our support team if you need further assistance.
Posted May 21, 2026 - 17:29 EDT

Update

Users should be experiencing restored functionality to their boards and cards.

The team continues to remediate potential outstanding data issues that occurred on Boards and Cards during the incident. Below is a list of remaining issues that may still be occurring, but we are currently working to resolve via processing on our end. These issues were caused by card moves between boards during the incident.
- Missing card labels
- Attachment restrictions not being enforced on moved cards
- Checklist assignees that are not a member of the current board may incorrectly linger
- Card assignees that are not a member of the current board may have incorrectly received due date reminders

Please note that cards are safe to edit, but they may be excluded from certain restoration work if they are edited.

We greatly appreciate your patience as we work through these issues.

We will provide another update when above issues are resolved. In the meantime, please reach out to our support team if you need further assistance.
Posted May 20, 2026 - 15:06 EDT

Update

Users should be experiencing restored functionality to their boards and cards.

We are running scripts to replay actions that occurred during the incident. We will update once these are complete.
Posted May 19, 2026 - 14:01 EDT

Monitoring

Index rebuild is complete and functionality should be restored for users. We are aware that cards moved by users during the incident may be in an incomplete state and we are investigating solutions.

We have no indication of data loss during the incident. We will continue to monitor and will provide the next updates in 4 hours or sooner.
Posted May 19, 2026 - 10:26 EDT

Update

We are continuing to monitor the index rebuild process across all affected shards and will provide updates as each phase completes. We will provide next update in eight hours or sooner as we make progress.
Posted May 19, 2026 - 03:03 EDT

Update

We want to provide more transparency regarding this incident given its duration.

We were optimizing one of our largest databases, specifically the one storing comments and card activity. At the end of this process, a bug caused some database partitions to go live without their indexes. This means queries to those partitions are timing out, which is why approximately 40 percent of boards are experiencing failures.

Your data is safe. Any comments added during this incident are being saved and will reappear once the process is complete.

We are currently rebuilding the missing indexes. We have allocated significant compute resources to this task to move as quickly as possible, but the data volume is enormous. We estimate it will take approximately 20 more hours to reach full resolution.

Most impacted features include, but are not limited to:

- Comments (viewing, adding, notifications)
- Activity views on cards
- Attachments, Power-Ups, board exports
- Home page / 'Updates' view
- Search
- Some cards showing 'Card Not Found'
- Moving/copying cards or lists

Please avoid moving or copying cards between boards for now. If your board is working normally, you can continue your work, as most boards remain unaffected.

We know Trello is where your work lives and we are committed to getting everything back to normal. We will provide next update in eight hours or sooner as we make progress.
Posted May 18, 2026 - 18:45 EDT

Update

Our teams continue to work diligently on the mitigation.

During this time, we recommend that you do not move or copy cards or lists between boards. For users whose boards are affected by this issue, you may create and delete cards and edit card descriptions as usual, but you will not be able to see comments or the detailed activity log of the cards.

We estimate that the rebuild will take another 24 hours to complete. We will update this page as we have more information.
Posted May 18, 2026 - 14:09 EDT

Update

Trello continues to experience slowdowns and failures that are affecting comment operations, including creating and deleting comments.

Our teams continue to work on the issue resolution, while it is taking longer than expected. We will provide more updates within the next 8 hours or sooner once we have further information to share.
Posted May 18, 2026 - 06:14 EDT

Update

Trello is experiencing slowdowns and failures affecting comment operations, including creating and deleting comments. Users may also experience delays when loading activity views on cards, viewing attachments, using Power-Ups, exporting boards and 'Updates' on home page.

Our team is actively working on the resolution and will likely take up to 7 hours to restore the services back to full functionality. We will provide further update at this time, or earlier if we have additional information to share prior to this resolution.
Posted May 17, 2026 - 23:51 EDT

Update

Trello is experiencing slowdowns and failures affecting comment operations, including creating and deleting comments. Users may also experience delays when loading activity views on cards, viewing attachments, using Power-Ups, or exporting boards. Some users are encountering errors when viewing specific cards.

The team is working hard to restore full functionality and will provide another update within 3h.
Posted May 17, 2026 - 20:53 EDT

Update

Work by the Trello engineering team continues to restore functionality to comments and activity views. Recovery of the affected systems is taking longer than anticipated.

Next update will be posted within 4h or on a significant change. Thank you for your patience.
Posted May 17, 2026 - 17:03 EDT

Update

Work by the Trello engineering team continues to restore functionality to comments and activity views.

Next update will be posted within 4h or on a significant change. Thank you for your patience.
Posted May 17, 2026 - 12:25 EDT

Identified

The team has identified the root cause and is hard at working resolving this issue. Users will see problems creating/updating/deleting comments and loading the activity view for cards at this time. The rest of Trello's features should be working as normal.

We will provide more updates within 2h.
Posted May 17, 2026 - 11:13 EDT

Investigating

Users may be experiencing slowdowns across various functions of Trello including creating/deleting comments, editing cards, or creating new attachments. The team is actively working to fix the issue and will provide more updates within the hour.
Posted May 17, 2026 - 10:11 EDT
This incident affected: Trello.com, API, Atlassian Support - Support Portal, Atlassian Support Ticketing, and Atlassian Support Knowledge Base.