Tender’s downtime postmortem for April 17, 2011

At approximately 3:48 AM PST Tender experienced a technical problem during a routine database backup. The secondary database server started to leak memory while the backup was running, causing Tender to go down.

Service to Tender was restored at 10:24 AM PST. All inbound emails that were queued during this time have been processed and Tender is running normally.

Today’s outage is unfortunate and unacceptable. We’re working as hard as we can to prevent such an outage from happening again.

What we did with the rest of our Sunday

The first order of business was to figure out why our alert notifications failed. We’re a small company and we rely on alert notifications to warn us if anything goes wrong during off hours. Our phones are supposed to blow up the instant something goes wrong. That ball was totally dropped this morning, so we've implemented a fallback for alert notifications should the first line of defense ever fail again.

Next? Deal with the actual problem. Our secondary database server, the source of the downtime, handles routine backups and miscellaneous services that Tender uses. We’ve moved these additional services that Tender relies on to a new server. In the future, should the secondary database server ever have a problem, it won’t affect Tender at all.

If you have any questions, feel free hit us up on our Tender or mention @tenderapp on Twitter.