Management summary

Impact: Dev environment
Root cause: Skipped steps during DB instance migration in GCP

Incident background

While migrating PostgreSQL databases from old instance to new one, applications were still connected to old instance. They switched to new one after they were redeployed using Github Actions, which caused inconsistent state of data.

Actions took to mitigate the problem

New database was wiped and all applications were redeployed, to force them to connect to new database instance.

Lesson learned

When there is a database migration, all impacted applications should be undeployed and once the migration is done, they should be deployed back with new configuration.

In the case of such migrations, proper plan should be worked out to make sure, that all steps are done and in correct order as well.

Added instruction for DB migration process: DB migration process