It's been a rough couple of weeks for Tender customers. We've encountered multiple, significant downtimes and our developers have been working hard to find the source of this issue. Yesterday the culprit was discovered — a memory leak which manifested itself within a timeframe of less than 20 seconds was causing entire servers to disappear from the cloud.
While it's arguable whether a cloud hosting platform should be susceptible to this type of failure or not, the net effect was that our cluster was cycling through servers at an elevated rate. Because the virtual machines would be destroyed when the leak happened (within mere seconds) without any notification, there were no log files available and we had to escalate the issue to get some developers at our hosting provider involved.
After our investigation uncovered the leak, we disabled the code responsible for it, added more fault tolerance to our backend code, and added a ton of additional monitoring that will alert us about things before they become visible to users (such as statistical monitoring, like the standard deviation on the amount of time for a roundtrip email).
We understand our customers rely on our service to communicate successfully with their own customers and we take downtimes very seriously. This should be the end to this downtime.
As most are aware, Lighthouse had some server issues early this morning (or afternoon depending on your timezone).
The application master on our Amazon cloud instance locked up around 3:30am PST (original cause still being checked into). Then our hosting provider stack failed to promote one of the other application servers to Master. We had our hosting provider developers come in and manually remove the instance and add a new one, but the resulting flood of reloads and access attempts brought Lighthouse right to its knees. Once Lighthouse was up again we started seeing errors from various processes that didn't restart as expected or needed time to get cycled up.
We have escalated this issue all the way to the executive level at hosting provider, because a Cloud app solution that doesn't automatically remove a failing server and promote an existing server to cover the bases isn't much of a cloud app solution. The stability of our application stack is very important to us and we're currently evaluating what steps we can take to ensure this doesn't happen again. We are in active (and frequent) communication with hosting provider about this very thing today, and we will do everything we can to ensure the stability of Lighthouse, including changing hosting providers if necessary.
Today we’re rolling out a new feature for Lighthouse that’s been needed for a long time: bi-directional ticket references. Since we deployed this internally, we’ve all been using this new feature like crazy, and it’s really helped get things done here. We’ve tried hard to make it as workflow agnostic as possible, as you’ll see.
While it’s easy to link to other tickets in the same project with a comment like “blocked by #145”, it’s not easy to see what links to a given ticket. This one-way relationship limits the applications of ticket links, so we decided to make it a two-way relationship.
When a ticket links to another ticket, we now create a back-reference that you can see from the linked ticket sidebar. An excerpt of the paragraph surrounding the link is also included, to provide context. This allows you to easily specify ad-hoc relationships between tickets the same way you do now, without having to take any additional steps.
In fact, there’s a good chance you’ll find a use for this feature right away. While back-filling the references for existing tickets, we found that of 12 million tickets, over 300,000 included references to other tickets, with a total of 1.5 million references across all of Lighthouse. This means you are likely to see new sidebar content for commonly referenced tickets in your Lighthouse.
We’ve also hooked up an API endpoint for this first version of the feature, for listing references to a given ticket (references can still only be created by comments, already possible via the API). By accessing the “references” resource on a ticket using the API, you can get a list of references for that ticket. Use it to build your own ticket dependency graph, or who knows what! Of course it’s up to you to parse out the words to infer the type of relationship, and enforce that across your team.
A few details: if a reference is out of place or irrelevant, they’re easy to remove, but they’ll reappear if the referencing comment is updated without removing the reference. References are added in an asynchronous job and may not appear right away. We actually pull these references out using an HTML parser, so even ticket links that are not specifically auto-links should still work as intended.
We hope you find this new feature useful, and we welcome your feedback on how it could be even more useful!
Updates for August 18:
- You can now download a .ics (ical) file for your milestone due dates. You can also subscribe to authenticated feeds if you have an API token – just append ?_token=YOUR_READONLY_TOKEN to the generated url, or use a calendar that supports HTTP authentication. You can append `.ics` to any milestone-type url (like /milestones or /milestone/1234) Feed URLs are also visible as autodiscovery links, if your browser supports that (probably not).
- You can now send attachments via email to Lighthouse and they’ll show up in the ticket.
- Account users are now visible in the users API, so you can see the list of available users to give access to a project
- Total number of pages are now shown in API results, so you can determine whether to grab more pages
- API requests now respect the project_id of the API token on account-level queries
- Fixed iphone layout
- Creating a new project no longer breaks the RSS feed for the account temporarily
Work also continues on LH3, the complete redesign of lighthouse. You may have seen the semi-secret dashboard page. More is coming!
You can now disable the public forum element of your Tender install. You may have users who don't like to help each other, or perhaps it's just not appropriate for people to browse through your public posts.
This increases the flexibility you have in restricting access to certain content on your Tender site. Read more about locking down your tender here.
We are looking to add another software developer to the team here at ENTP.
You'll primarily be working on two Rails applications, Lighthouse and Tender. We recently implemented a Redis based analytics backend to Tender and are working on the next major update for Lighthouse.
It's a pretty fast paced environment. Aside from that, we love open source and solving problems, so the possibilities are endless. If you have an awesome idea, we'll back it. We're always looking at all sorts of fun programming challanges beyond the range of your typical web application.
Zack Hobson and Courtenay Gasking are the two developers you'll be joining here at ENTP. Both are awesome at what they do. You'll be working closely with Hobson, pairing on feature development and helping push our applications to the next level.
Everyone here at ENTP works as a team. We all chip in with customer support and contribute to bug fixes, brainstorming, and work towards a better future for our products.
Ruby, Rails, MySQL, Redis, jQuery, Prototype, Git, and a love for open source software. Bonus points for experience with Chef.
We're a team of eight. Most of us reside in Portland because it's the most awesome place on the planet, but part of the crew is down in Los Angeles. We typically roll in to work around 10:00 AM and have a serious love for coffee shop hack sessions, quality bourbon, and micro-brews.
What we bring to the table
Competitive salary with health, dental and vision. It's a great gig, an awesome team, and an opportunity to work on two extremely well established Rails applications.
How to apply
Send an email to email@example.com with links to your code and a blurb about yourself.
Tender has reached the 1,000 customers mark!
Hugs-and-handpounds go out to all of our customers for their support and feedback over the last two-and-a-half years. Looking back as a small bootstrapped company that was doing client work to cover the development costs of two products, it's been an amazing journey thus far.
Some Tender statistics
- 1,008 subscriptions
- 11% of trial accounts become customers
- 1.2 Million people used a Tender for customer support
- 2.8 Million support discussions
- 1.6 Million closed support discussions
On the roadmap
Growing the team
The time has come! We've got an empty seat here at ENTP and we're ready to add another developer to our team. If you're a Ruby geek and lusting to dig into some awesome web application development, hit us up at firstname.lastname@example.org with links to your code and a blurb about yourself.
You can view the full job listing here.
New features and updates
Where do I even begin? I want to touch on some of our major goals for Tender in the foreseeable future.
- New private support workflow features (disabling the public forum)
- New "Closed Beta" support feature
- New and improved Knowledge Base system
- New and improved workflow for assigning support discussions to your staff
- Twitter integration
There's a lot more to add to that list, but I'll save those items for when we release our public roadmap.
A new plan structure
We're preparing to roll out a new plan structure for Tender. Between Lighthouse and Tender we've tried a few different plan and pricing options over the years and we've come to a few conclusions that will allow Tender to scale better to your needs. We've started migrating Tender's billing system over to Braintree (which is totally awesome BTW!) and once that is complete we'll have some pretty big announcements to post about.
A better trial experience
First impressions last the longest. That's a big reason why it takes all of 10 seconds to sign up for Tender. A while back we decided that new Tender accounts would get some live action after signing up, so we started to generate a support discussion that goes right to my email inbox when you reply. Thanks to the feedback during trials, we have a fresh perspective on how we can improve that first experience.
As always, we're constantly working to roll out bug fixes and improvements to Tender. We have a number of bug fixes in the stack that we'll announce as they ship.
Last night we shipped some bug fixes and additions to Tender.
- Fix - Broken user profile link bug in Tender's search results page.
- Update - Duplicate comments flagging no longer applies to staff members. It's the first step towards canned replied.
- New - Custom CSS for Tender's Widget.
In addition to those, we had a few lingering edge cases with discussion counters on the Dashboard that we've cleaned up. The counters bug is fixed but a few legacy issues where the counters were still wrong may exist on accounts. Feel free to contact us at email@example.com if you have any questions.
Capo helps you learn to play music in your iTunes library by slowing it down and generating a detailed spectrogram. You can even draw atop the spectrogram and it will automatically generate the tablature for you. Capo is also available for iPhone, allowing you reverse engineer your music on the go.
It's probably safe to say that I've tried just about every to-do list app released over the last four years. Many of them have been great, but they all failed to solve one serious problem, I have multiple computers and devices.
In October 2010, I received a beta invite to a new app called Wunderlist. They were interested in using Tender for their customer support and I was really interested in playing around with their app.
Wunderlist supports multiple devices (iPhone, iPad, Android, Windows and Mac OS X) and syncs between them automatically. What sets Wunderlist even further apart from other to-do lists is that it is Open Source Software. The apps and cloud syncing services are completely FREE. What's not to love about that?
We're huge advocates of Open Source Software here at ENTP. We've released a lot of our own code over the years and provide free Tender and Lighthouse accounts to any Open Source projects who want to use them. It's the least we can do to give back to the community we love so much.
It's amazing to see an Open Source project like Wunderlist using Tender and gain so much traction. We're excited to see what the future holds for 6 Wunderkinder.
I've always figured that one day my kids are going to be very confused as to why their father gets more excited about Apple events than Christmas or a birthday. In honor of that excitement, I want to do something awesome for Mac and iOS developers who are looking to provide customer support for their apps.
If you're a Mac or iOS developer and you sign up for a Tender trial this week (it takes 10 seconds and is free for 14 days), we will give you $75 account credit when you upgrade to a paid plan.
Just send an email to firstname.lastname@example.org with your Tender account URL we'll take care of the rest.
Happy WWDC week everyone.
Shortly after WWDC in 2009, we noticed a traffic spike on Tender resulting from a blog post titled Growing Pains. The post was written by Kevin Hoctor, a Mac developer who had signed up for Tender just one week earlier.
"It's only been a week of working with Tender and a few days of having it live for our customers to use, but I'm thrilled with it. The discussion forum is much more structured than our old Google Groups forum—we can mark issues as open or resolved, assign priority queues to issues, and, best of all, support emails go to the forum so they can't get lost in a cluttered inbox. It's not as large and in charge as FogBugz, but that's part of its appeal. We're even more committed to offering timely and effective support as our customer base continues to grow, and Tender will help us stay on top of our game." - Kevin Hoctor
Since Hoctor's blog post in 2009, we have a few more notable Mac and iOS applications using Tender.
Mac and iOS apps on deck
Papers - Your personal library of research.
The Cocoa Bots - The crew behind Hyperspaces.
Realmac Software - Creative Software for Macs.
Sparrow - The new mail for Mac.
Villian - wickedly good games for iOS.
Propane - Mac client for Campfire (We can't even begin to imagine what our life would be like without it).
M Cubed Software - The makers of Lighthouse Keeper, Code Collector Pro, Minim, Syllabus and Storyboards.
Primate Labs - The makers of Geekbench and Wiinote Wunderlist - Task management at its best.
FuelCollective - Makers of Permute, Eon, Pochade and Snippet. They also make a Mac toolbar client for Tender called Meaty!
Textie Messaging - The unlimited texting app for iPhone, iPad, and iPod touch.
Echofon - Full Featured, Super Clean Twitter Apps.
App Notifications - Delivering real-time content to your iPhone.
Concentrate - Concentrate, for the Mac, helps you work and study more productively.
Stunt Software. The makers of On The Job, Downsize, Overflow, 4 in a Row for iPhone, and Freeform for iPad.
Enstore - An e-commerce platform that works with Checkout and AccountEdge.
Test Flight - iOS beta testing on the fly.
Biolithic - The makers of Comic Zeal, ThinkBook and Twiku.
ExpanDrive - Access remote files with any application.
RunKeeper - RunKeeper makes tracking your workouts fun, social, and easy to understand so that you can improve the quality of your fitness.
Postagram - Instantly print and ship photos from Instagram or your Photo Album right from web or iPhone.
Momento - A unique approach to diary writing.
Moneydance - Simplified money management for your life.
Rocketbox - A powerful, new way of searching your e-mail in Apple Mail.app.
CloudApp - Share. Files. Fast.
SuperMegaUltraGroovy - The makers of TapeDeck, Capo and FuzzMeasure Pro 3.
Dolly Drive - Use Apple Time Machine in the cloud!
Carbon Copy Cloner - Make a bootable backup of your digital life.
Shiny Frog Software - The makers of DeliBar, IMG2ICNS and Shiny Droplets.
omnidea - The makers of RapidCart 3, RapidLink, RapidMaps, RapidViewer and Rulers.
Notify - An awesome email notifier for Mac.
Apparent Software - The makers of Socialite, ImageFramer, Cashcalculator and Blast.
Third Street Software - The makers of Sente 6 and Sente for iPad.
Mental Case - The Study App.
YourHead Software - The makers of Kiwi, Accordion, Collage, Blocks, Stacks, Kwix and Carousel.
Ehon - Your Mac loves eBooks.
Chocomoko- The makers of Eavesdrop, Brisk, and SweetFM.
Outpost 2 - A native iOS application for Basecamp.
touchAble - Control Abelton Live from your iPad.
codeREADr - Scan, record, track, validate barcodes and collect data with your smartphone!
Brightkite - Unlimited group texting.
PixelMags - Content distribution, one app at a time.
iWeb Buddy - An extension to Apple’s popular iWeb application. Where iWeb leaves off, iWeb Buddy starts.
Coradine Aviation Systems - Three magical ways to track your flight time.
SourceTree - A Mercurial and Git client for Mac.
There is also an Objective-C library that Matt Newberry wrote for Tender's API.
That's a pretty awesome lineup.
On Tuesday, we noticed a small subset of Tender users have not received email notifications since May 27. The problem was brought to our attention as soon as a few power users noticed their emptying inboxes. We immediately restored notifications to those users and began investigating the problem.
After some digging, we discovered that a bug with our outbound email server had disabled email addresses that had bounced in the past. We're working closely with our outbound email provider to restore notifications to every affected email address. Tender has recieved more than 148,000 email bounces in the last year, so it's taking us some time to confirm everything is 100% OK.
If you've noticed a lack of email notifications from Tender, please email email@example.com, and and we'll kick those into gear right away. In the meantime, we will continue restore email notifications to those who have been affected. We should to have everything back to normal soon.
I'm a little neurotic when it comes to email. Everything in my inbox is labelled and color coded so I can easily skim over incoming support for our two services vs regular email.
Google Apps opened the flood gates for businesses who need a powerful mail client and web based tools at virtually no cost. In fact, a vast majority of Tender customers use Google Apps for email.
One really awesome trick that a lot of people tend overlook is the ability to filter Tender emails from a mailing list within Gmail. Every Tender account has a unique mailing list ID that enables Gmail's filtering by mailing list feature. I use it for a quick way to apply labels to Tender discussions.
How to be like me with Gmail and Tender
Open any email notification from Tender in Gmail
Click on Show Details
Click the Filter messages from mailing list link
Gmail with automatically pull in the filter for emails on this mailing list. You can set any additional fields for the filter you might desire (I just leave it all as is).
Click Next Step for more options so you can apply a label.
Click on Create Filter button (apply to previous Tender emails if you want).
That's it! You'll now have labels applied to all email notifications sent from your Tender account.