Job Queue Issues: It's Not Lupus

Yow, what happened to the queue? We had two separate issues on Thursday night and Friday night that have since been fixed.

The first one had to do with a strange error from parsing one of the many emails we receive. A bug in our email parsing library raises a critical exception that kills the queue process. I found the issue, deployed a workaround, and then submitted a patch to the library author. Typically, our queue catches and logs exceptions, instead of dying. However, the bug caused it to mistakenly raise a fatal SyntaxError, which is typically reserved for a typo in a script that prevents the application from even running.

The second problem has to do with a new job we deployed on Thursday. One of the problems we saw with accepting anonymous comments on Tender, is that you lose all connection to those comments when you create a new profile. The new job automatically links them to you when you create a profile on Tender. The problem is, these jobs started taking a bit longer than anticipated to run, holding up more critical tasks. I’ve since started using a priority system to ensure that critical tasks like spam checks or comment notifications are run before these jobs.

On a final note, it’s clear that the job queue is of utmost importance for Tender. We actually have a munin server setup for monitoring, but we’re still working with Engine Yard on getting the firewall port issues worked out. This incident has moved that ticket to our top priority so we can get on top of these issues sooner. Sorry for the inconvenience, everyone!