12:10 PM EST: Outage detected. The database was down, causing the website, Excel, and API to be non-functioning. Error Message: Any queries attempted returned an error indicating an inability to access the availability database 'xxxxxxxx' due to the database replica not being in the PRIMARY or SECONDARY role. 12:20 PM EST: Initiated a server reboot as a response to the outage. 12:30 PM EST: Opened a case with AWS support for assistance with the issue. 12:35 PM EST: AWS support person contacted their internal team for further assistance. 12:40 PM EST: Started a snapshot restore process to prepare for a potential situation where the original server could not be restored. The snapshot was from midnight, indicating a potential data loss of approximately 12 hours. 1:21 PM EST: The original server was still in the rebooting state, and the new database was still in the creating state. 1:30 PM EST: The server successfully rebooted and came back online.
Posted Dec 15, 2023 - 13:42 EST
Update
We are continuing to investigate this issue.
Posted Dec 15, 2023 - 13:40 EST
Investigating
We are currently investigating database outage
Posted Dec 15, 2023 - 12:28 EST
This incident affected: Valitana Web, Valitana Excel, and Valitana API.