Early this morning, I was woken up with some urgent Tender news: requests were taking an abnormally long time to run.
As I logged in, some Engine Yard technicians had already checked the basics: memory usage is stable, database is happy, the filesystem is functioning, etc.
Then we came across this graph in New Relic:
Our application processes were being slammed with requests. The culprit?
Since extending email processing in Tender to handle the spam from our public email server, we’ve been seeing a lot more email traffic. Each email is turned into a HTTP web request from our email server to the Tender servers. Most of the time this load is perfectly fine on our current hardware. However, the backlog of spam emails pushed it over the top. It’s definitely eye-opening seeing how much spam emails we receive.
The issue is taken care of for now. For the short term, we’re going to setup a dedicated application server just for emails. This would be something we’d spin up when there’s a large volume of emails to process. This way email traffic won’t affect the web traffic at all.
In the longer term, we’re looking into adding more application servers as our need dictates.
Sorry for the inconvenience with the issues tonight. If you have any questions or problems, let us know.
Loading tweets…
Our Products
Our Open Source Projects
- astrotrain – simply the best email processing solution out there.
- Seinfeld – automatic tracking of your open source contributions to github projects.
- warehouse – still using subversion? Get the best source browser out there.
- xtt – time tracking built for the add & twitter inclined.
- GitNub – OS X GUI for tracking git logs.
3 Comments
Please, please, please fix this issue as soon as possible. You were down for some time today and performance has been terrible all day. We have just converted to Tender for our support and the system is currently barely useable.
Michael
It’s been days now without a resolution. Not having our support site working is killing us. We need this fixed today or we’re going to have to transition to someone else.
We worked over the weekend but due to traffic decreases only received some very small feedback from the server as to how it would handle Monday morning. We’re still currently seeing delayed response times as well and are currently working with our host to resolve the issue as quickly as possible.
We’ll be keeping http://twitter.com/tenderapp updated as things occur and we’ll post a formal announcement here on Hoth as soon as we have a formal announcement on the situation for Monday morning.
Make your voice heard
Sorry, but comments are closed for this item.