Yesterday, we encountered strange job queue behavior that resulted in discussions being delayed with the indexing, assets were not processed, and an isolated case where incoming emails were not processed.
The issues are now resolved.
We had a few symptoms:
- A beta feature was pushed out over the weekend, increasing the number of jobs that go through the system.
- Certain rare conditions were causing the queue to halt.
- Indexing slowed down, due to some large documents.
To fix things, we pushed the following changes:
- Jobs were rearranged so more important jobs (asset processing, bulk edits) are being processed ahead of less important jobs (indexing)
- The job queue was patched so the certain rare conditions quit halting the queue.
- Full indexes are done more often now, keeping the partial indexes from growing too large.
ENTP is a consultancy and web incubator based in Portland, Oregon. We make Lighthouse and Tender Support. If you’ve got a project in mind, please get in touch. We’d love to help you out.
Loading tweets…
Our Products
Our Open Source Projects
- astrotrain – simply the best email processing solution out there.
- Seinfeld – automatic tracking of your open source contributions to github projects.
- warehouse – still using subversion? Get the best source browser out there.
- xtt – time tracking built for the add & twitter inclined.
- GitNub – OS X GUI for tracking git logs.
2 Comments
May I ask what you are using for your Jon queue? A home-grown solution or something like RabbitMQ/ActiveMQ etc?
We’re using Delayed::Job. We’re looking into something like Resque. I’ve been using DJ for awhile, but our use case on Tender is much different. We’re sometimes processing some really weird spam content, for instance.
Make your voice heard
Sorry, but comments are closed for this item.