Lighthouse upgrade delays

Last night, we took Lighthouse offline for about 30 minutes to deploy a major performance upgrade. The upgrade includes a new search engine, revamped job queues, and new permalink indexes. These will improve overall system performance, speed and search accuracy within Lighthouse.

We switched our search engine to Sphinx, an engine acclaimed for it’s speed and flexibility. We’re especially excited about the move to Sphinx because it provides and extremely fast search platform to build upon.

After bringing Lighthouse back online, we immediately noticed two crucial bugs in the system, which affected user invites and email notifications. Fixes for those bugs were promptly deployed to Lighthouse before traffic picked back up.

We weren’t aware of an issue with the deployment setup that causes Lighthouse to have problems getting search indexers to run quickly across all three of our slices. This is preventing tickets from being displayed correctly in the search system. To some users, it may appear as if a newly-posted ticket has vanished in their search query or ticket bin.

But rest easy, your data is still there. Your search results are just delayed. Possible delays could last up to 10 minutes, depending on which slice your ticket fell on. Currently, running a simple all query in your search field will bring up a list of all tickets within that project stored directly in the database.

The ticket data is still in the system, but it’s not being displayed by the search in a timely manner.

We’re working with our host, Engine Yard, to get everything consolidated to a single index so tickets can be updated immediately. We’re close to a final solution and hope to have this cleared up soon.

Another bug, which has sifted into the system, is in the search queries. Some users reported inaccuracies with their search results and ticket bins after our update. These seriously hinder the effectiveness of Lighthouse for users and their workflow. Once the indexing issue has been fully settled, we will begin investigating the irregular search results and deploy a fix as soon as possible.

If you wish to keep up with the status of Lighthouse and receive important announcements, you should subscribe to htth://lhstatus.com or follow @lhstatus on Twitter.

We apologize for any inconveniences you’ve had with Lighthouse today.