The persistent disk on one of our database nodes filled up, and the database node using that disk was unable to serve queries. When the node failed, RethinkDB successfully migrated most tables to other database nodes. The user activity table was not able to recover automatically because it had been created with the database-default configuration of only having one replica.
Because we show user activity data as part of the logged-in experience on expo.io, users were unable to load the website. When we were sure that other more critical database tables were operating correctly, we focused on restoring the website first. After about 30 minutes we had tested and deployed a temporary change to not render user activity data while we resolved the underlying database outage.
Once the user-facing outage was mitigated, we resized the underlying disk and restarted the database node. The change to restore the logged-in experience of the website is currently in flight.
We'll take several actions to prevent this kind of problem in the future, including providing a saner default configuration for database migrations and changing our cluster monitoring to alert us to potential disk issues earlier.