Lighthouse Database issues

I woke up this morning to some crazy database issues on Lighthouse. It’s hard to tell what’s going on after the fact, it already seemed like it was starting to settle down. I did some looking around, and noticed that there some projects that were deleted. Active projects tend to have a lot of dependent objects that need to be cleaned up, such as Events, Tickets, Messages, Comments. I just pushed a change that will offload this to a background job.

This particular issue was a bit difficult to track down, because the end result was that everything was slower. New Relic just showed the usual suspects at the top of the most active queries: Ticket.find, Spam.find, etc.

Things seem like they’re doing much better now, but I’ll be keeping a closer eye on thing and double checking that we have the proper database indexes in place.