A little over 2 years ago I started a tiny side project called Kernl to scratch an itch, solve a problem, and learn some new technologies. It has been successful beyond anything I could have imagined and I’ve learned a ton of new things along the way. One of those things is how to ship new features to customers quickly, safely, and with the commitment to stability and performance that my customers have come to expect.
The full process for developing features on Kernl touches a lot of different areas: Customer Support, Product Management, Software Development, and Marketing. It’s taken me some time to figure out a good process for all of these things, so I’m sharing my learnings in hope that it helps someone out.
Scale
Kernl isn’t that big. Honestly, it isn’t big at all compared to the likes of Facebook, Instagram, Google, etc. But Kernl isn’t small either. It’s just big enough that I have to worry about scale, availability, and consequences to the WordPress sites that it interacts with on a daily basis should it go down.
Remember how I said Kernl was a tiny side project when it started? Well it was developed that way too originally: A full-stack monolith on a single $5 DigitalOcean droplet. As time went on and Kernl grew I eventually split everything up and landed at a highly available setup that can scale horizontally, but I got some scars along the way. Those scars taught me two things:
- Your customers don’t care why you went down.
- Don’t take unnecessary risks.
With those two things in mind, I organically grew into a process that helps release features slowly, gather feedback from targeted customers, and then release it into the wild.
Customer First Development
Kernl has a backlog of about 200 items that should probably get developed in the future. There is a constant stream of new ideas that get added to backlog, sorted, and filed for later. But the one thing that always makes it to the very top of the backlog is a customer feature request.
When a customer feature request comes in, the first thing I do is try to figure out what exactly it is they are requesting. Once I have an OK handle on that, I respond to the customer and explain what I think they are asking for. This helps me clarify the requirements and solidify my understanding of the feature.
Once I’ve got a handle on what they’re asking for I take a few moments and decide not only if I can implement this feature, but also if I should implement the feature. As I don’t yet work on Kernl full-time, the hour (or less!) that I spend working on it every night is precious. If I don’t think that the return on a feature is going to be high enough I often have to turn it down. Most of the time though feature requests are reasonable 4-5 hour projects and I’m happy to implement them.
Developing for Safety
I have an extremely low risk tolerance when it comes to Kernl. One of the things I like about Kernl is that I can leave it alone for a few days and be confident that it won’t have exploded while I’m gone. Every feature I develop for Kernl has the potential to ruin my confidence in it, so I’m very careful about how I do things.
The first step I take is to always create a feature branch. This may seem like a no-brainer to most software developers, but I have very unpleasant memories from 10 years ago before I realized how stupid it was to always develop directly on master. Once I have a branch created I get to work. If the code I’m writing is in one of the critical high traffic paths I’ll do proper TDD because it helps give me confidence that I’m not going to get woken up at 2am because of a coding mistake. If the code is not in the critical path I’ll generally write the tests afterwards or as I go along. The coverage is still good and meaningful, but development tends to be a bit quicker for me this way.
Once the feature is mostly complete, I wrap it in a feature flag so that I can easily toggle it on and off when I need to. If you aren’t familiar with feature flagging, its a process that allows you to decouple feature releases from code releases. For Kernl, I dogfood and use Kernl’s feature flag infrastructure to toggle features on specifically for the customer who requested it.
The Slow Release
As I mentioned above, using Kernl’s feature flagging I can be extremely choosey about who sees a new feature. The first thing that I do once I deploy a new feature is toggle it on for myself. After I’ve taken some time to test it out and verify the functionality I’ll toggle the feature on for the customer who requested it.
One of the great things about having developers as your main customer base is that they’re often very happy to help shake out features (especially when they requested it!). If the customer doesn’t find any bugs over the course of a few days and I’m satisfied that feature won’t cause any trouble, I change the feature toggle to a simple on/off toggle and leave it on for everyone.
The best part about this approach to development is that it gives me peace of mind. If I start getting notified in the middle of the night that the new feature I just released is acting up, I can just log in to Kernl, disable the feature flag, and go back to bed. Obviously this doesn’t work if the offending code is in the feature flag area, but most of the time its a non-issue.
Basic Marketing
As a developer it took me awhile to realize something about releasing new features: People won’t notice them unless you market them. After learning that hard lesson, every new feature that Kernl gets also receives a simple blog post, Twitter message, and marketing email. I personally believe that every new feature should be celebrated and I’ve found that generally customers are happy to see active development happening on a product that they’re paying for.
Kernl customers don’t log in to the product every day. Ideally they have WordPress continuous deployment set up so they only need log in to setup a new product, so chances of them noticing a new feature just by usage are small. Because of this usage pattern basic marketing is essential for existing Kernl customers so they know what’s new.
While this full process can seem a little bit heavy for a side-project, when you have a bunch of paying customers who rely on you for a critical piece of their product you have to be careful.
