How I Develop Features for Kernl.us

A little over 2 years ago I started a tiny side project called Kernl to scratch an itch, solve a problem, and learn some new technologies. It has been successful beyond anything I could have imagined and I’ve learned a ton of new things along the way. One of those things is how to ship new features to customers quickly, safely, and with the commitment to stability and performance that my customers have come to expect.

The full process for developing features on Kernl touches a lot of different areas: Customer Support, Product Management, Software Development, and Marketing. It’s taken me some time to figure out a good process for all of these things, so I’m sharing my learnings in hope that it helps someone out.

Scale

Kernl isn’t that big. Honestly, it isn’t big at all compared to the likes of Facebook, Instagram, Google, etc. But Kernl isn’t small either. It’s just big enough that I have to worry about scale, availability, and consequences to the WordPress sites that it interacts with on a daily basis should it go down.

Kernl Scale Google Analytics Image

Remember how I said Kernl was a tiny side project when it started? Well it was developed that way too originally: A full-stack monolith on a single $5 DigitalOcean droplet. As time went on and Kernl grew I eventually split everything up and landed at a highly available setup that can scale horizontally, but I got some scars along the way. Those scars taught me two things:

  1. Your customers don’t care why you went down.
  2. Don’t take unnecessary risks.

With those two things in mind, I organically grew into a process that helps release features slowly, gather feedback from targeted customers, and then release it into the wild.

Customer First Development

Kernl has a backlog of about 200 items that should probably get developed in the future. There is a constant stream of new ideas that get added to backlog, sorted, and filed for later. But the one thing that always makes it to the very top of the backlog is a customer feature request.

Kernl Trello Board with Backlog

When a customer feature request comes in, the first thing I do is try to figure out what exactly it is they are requesting. Once I have an OK handle on that, I respond to the customer and explain what I think they are asking for. This helps me clarify the requirements and solidify my understanding of the feature.

Once I’ve got a handle on what they’re asking for I take a few moments and decide not only if I can implement this feature, but also if I should implement the feature. As I don’t yet work on Kernl full-time, the hour (or less!) that I spend working on it every night is precious. If I don’t think that the return on a feature is going to be high enough I often have to turn it down. Most of the time though feature requests are reasonable 4-5 hour projects and I’m happy to implement them.

Developing for Safety

I have an extremely low risk tolerance when it comes to Kernl. One of the things I like about Kernl is that I can leave it alone for a few days and be confident that it won’t have exploded while I’m gone. Every feature I develop for Kernl has the potential to ruin my confidence in it, so I’m very careful about how I do things.

The first step I take is to always create a feature branch. This may seem like a no-brainer to most software developers, but I have very unpleasant memories from 10 years ago before I realized how stupid it was to always develop directly on master. Once I have a branch created I get to work. If the code I’m writing is in one of the critical high traffic paths I’ll do proper TDD because it helps give me confidence that I’m not going to get woken up at 2am because of a coding mistake. If the code is not in the critical path I’ll generally write the tests afterwards or as I go along. The coverage is still good and meaningful, but development tends to be a bit quicker for me this way.

Once the feature is mostly complete, I wrap it in a feature flag so that I can easily toggle it on and off when I need to. If you aren’t familiar with feature flagging, its a process that allows you to decouple feature releases from code releases. For Kernl, I dogfood and use Kernl’s feature flag infrastructure to toggle features on specifically for the customer who requested it.

Kernl Feature Flag Overview

The Slow Release

As I mentioned above, using Kernl’s feature flagging I can be extremely choosey about who sees a new feature. The first thing that I do once I deploy a new feature is toggle it on for myself. After I’ve taken some time to test it out and verify the functionality I’ll toggle the feature on for the customer who requested it.

One of the great things about having developers as your main customer base is that they’re often very happy to help shake out features (especially when they requested it!). If the customer doesn’t find any bugs over the course of a few days and I’m satisfied that feature won’t cause any trouble, I change the feature toggle to a simple on/off toggle and leave it on for everyone.

The best part about this approach to development is that it gives me peace of mind. If I start getting notified in the middle of the night that the new feature I just released is acting up, I can just log in to Kernl, disable the feature flag, and go back to bed. Obviously this doesn’t work if the offending code is in the feature flag area, but most of the time its a non-issue.

Basic Marketing

As a developer it took me awhile to realize something about releasing new features: People won’t notice them unless you market them. After learning that hard lesson, every new feature that Kernl gets also receives a simple blog post, Twitter message, and marketing email. I personally believe that every new feature should be celebrated and I’ve found that generally customers are happy to see active development happening on a product that they’re paying for.

Sendgrid Marketing Campaigns

Kernl customers don’t log in to the product every day. Ideally they have WordPress continuous deployment set up so they only need log in to setup a new product, so chances of them noticing a new feature just by usage are small. Because of this usage pattern basic marketing is essential for existing Kernl customers so they know what’s new.

While this full process can seem a little bit heavy for a side-project, when you have a bunch of paying customers who rely on you for a critical piece of their product you have to be careful.

Feature Flags – Managing a WordPress Beta Program

Let’s say that you’re an author of a premium (i.e. paid for) WordPress plugin or theme. You’ve been hard at work on an amazing new feature but it really needs some testing before it goes out to all of your customers. How do you manage this process? What’s the best way to get new code into the hands of your beta users with the least effort on your part? This blog post is going to walk you through the process of using Kernl Feature Flags to easily manage a beta program without having two separate builds of your product.

Table of contents

What is a feature flag and how does it work?

Feature flagging is a software best practice for controlling the release of features (sometimes called “gating”). Feature flagging is important because it allows you to turn features on/off without having to do a deploy. But how does this impact you? How does it make things easier?

Feature flags allow you to manage beta programs with a single version of your product. What most people currently do is have 2 version of their product at any given time: The live version (what everyone sees) and the beta version (what your beta users see).  Wouldn’t it be easier to just have one version, one deployment process, and highly granular control over who sees what features?

Feature Flags Product View

For example, in the image above you can see that I have two feature flags: “GitLab CI” and “Download Invoice”. Right now they are both active and people can see the features that they represent. If I decided to change “Download Invoice” to inactive, the feature would be immediately deactivated in my plugin. I wouldn’t have to do another deploy and release a new version to make it happen. It happens automatically in the code that’s already with your customers.

Seems great right? Let’s do a full example so you can truly appreciate the power of feature flags.

Example: Adding a feature flag to the Kernl Example Plugin

The Kernl Example Plugin is intentionally very simple. The goal of the plugin is to show off Kernl’s various features while not overwhelming the person who is looking at it. To illustrate how and why feature flags are awesome, let’s add a simple setting to the “Settings -> General” menu.

The example plugin currently looks like this:

<?php
/**
* Plugin Name: Kernl Example Plugin
* Plugin URI: https://kernl.us
* Description: The Kernl Plugin for testing.
* Version: 3.3.0
* Author: Jack Slingerland
* Author URI: http://re-cycledair.com
*/
require 'plugin_update_check.php';

$MyUpdateChecker = new PluginUpdateChecker_2_0 (
    'https://kernl.us/api/v1/updates/5544bd7e5b8ae0fc1fa5e7a5/',
    __FILE__,
    'kernl-example-plugin',
    1
);
?>
All it does is require “plugin_update_check.php” and instantiate the update checker. Let’s make things a little more complicated by adding a setting to the example plugin.

Add a setting to the plugin

// This is added below plugin update instantiation.

function feature_flagged_settings_api_init() {
   add_settings_section(
        'feature_flagged_setting_section',
        'Feature Flag Example Settings in General',
        'feature_flagged_setting_section_callback_function',
        'general'
    );
   add_settings_field(
        'feature_flag_setting_name',
        'My Feature Flag Setting',
        'feature_flag_setting_callback_function',
        'general',
        'feature_flagged_setting_section'
    );
   register_setting( 'reading', 'feature_flag_setting_name' );
}
add_action( 'admin_init', 'feature_flagged_settings_api_init' );

function feature_flagged_setting_section_callback_function() {
   echo '<p>This section is hidden completely behind a Kernl Feature Flag.</p>';
}

function feature_flag_setting_callback_function() {
    echo '<input name="feature_flag_setting_name" id="feature_flag_setting_name" type="checkbox" value="1" class="code" ' . checked( 1, get_option( 'feature_flag_setting_name' ), false ) . ' /> This checkbox is hidden behind a feature flag.';
}
 The above code simply uses the ‘admin_init’ hook to call the WordPress Settings API and add a menu item. It looks like this when you run the code:
Feature Flags Example Setting Image
 Awesome! We’re off to a great start. Now let’s wrap this in a feature flag so only our beta user’s can see it.

Create an On/Off feature flag

Kernl has extensive documentation for feature flag usage, but it all boils down to:

  1. Add the feature flag library to your plugin/theme.
  2. Create a feature flag product in Kernl. A good rule here is 1 plugin / theme to 1 feature flag product.
  3. Create a flag.
  4. Instantiate the feature flag library and wrap your code.
  5. Manage who see’s the feature using Kernl.

So let’s do step one. If you’re using Composer, follow the directions in the feature flag documentation. If not, you can go to https://github.com/wpkernl/WPFeatureFlags and download the WPFeatureFlags.php file and drop it into your plugin or theme.

<?php
// ... snip ...
require 'plugin_update_check.php';
require 'WPFeatureFlags.php';
// ... snip ...
Easy. Next, go to Kernl and add a new product in the Feature Flags section.
Feature Flags Add Product Button
When you click “Add Product” you’ll get to choose a product name. Since I’m using feature flags with my example plugin, I’ll name mine “Kernl Example Plugin”.
Feature Flags Add Product Modal
After you click save, you’ll see the new product at the bottom of your feature flag product list.
Feature Flags Created product in product list
Now here is something important! See the “key” next to your product’s name? You’ll need that to instantiate the WPFeatureFlags class. I’d go ahead and copy it to your clipboard now.
Next up, let’s add a simple feature flag for this new setting we’re adding. This is the thing that will control visibility for all of our users. Click “Manage Flags” in the product menu, and then click the “Add Flag” button. You’ll be presented with this screen.
Add / Edit Feature Flags screen
All the options look straight-forward, but let’s go over them anyways.
  • Active – Will this feature be toggled on or off for everyone.
  • Name – A descriptive name. I’m filling this in with “General Setting Checkbox Example”.
  • Identifier – This is how we identify the flag in code, so I like to make it short and easy to understand. I’ll be using “GENERAL_SETTINGS_CHECKBOX”.
  • Flag Type – Kernl has 3 different types of feature flags for your enjoyment.
    • On/Off – As expected, this toggles the feature on and off for every user. No granularity here, but super useful for quickly disabling things. We’ll be starting with this flag.
    • Individual – You can select specific users that a feature will be toggled on for. This is what we’ll be using eventually, but there are some caveats that come with it.
    • Percentage – Kernl will roll out your feature to a percentage of your user base. Nice if you don’t want to specify individual users, but also don’t want the feature turned on for everyone.

Edit feature flags example

When things are filled out, press save and get ready to move on.

Now that we have our feature flag created, let’s instantiate the WPFeatureFlag class and wrap our code.

<?php
/**
* Plugin Name: Kernl Example Plugin
* Plugin URI: https://kernl.us
* Description: The Kernl Plugin for testing.
* Version: 3.3.0
* Author: Jack Slingerland
* Author URI: http://re-cycledair.com
*/
require 'plugin_update_check.php';
require 'WPFeatureFlags.php';

$MyUpdateChecker = new PluginUpdateChecker_2_0 (
  'https://kernl.us/api/v1/updates/5544bd7e5b8ae0fc1fa5e7a5/',
  __FILE__,
  'kernl-example-plugin',
  1
);

// We add the feature flag code inside the init() function
// so that we can have access to who the current user is.
function feature_flagged_settings_api_init() {
  // The feature flag product key. Remember the key I said you should add to your clipboard? This is it.
  $kernlFeatureFlagProductKey = '5a24035ee48da05271310a71';

  // The user identifier is how Kernl identifies the user requesting flags.
  // This should be unique for every user.
  $current_user = wp_get_current_user();
  $user_login = $current_user->user_login;
  $site_url = get_site_url();
  $userIdentifier = "{$site_url} - {$user_login}";

  $kff = new kernl\WPFeatureFlags($kernlFeatureFlagProductKey, $userIdentifier);

  // This says "For the product defined above, does this flag exists, and if so, is it active for the given user?".
  if ($kff->active("GENERAL_SETTINGS_CHECKBOX")) {
    add_settings_section(
      'feature_flagged_setting_section',
      'Feature Flag Example Settings in General',
      'feature_flagged_setting_section_callback_function',
      'general'
    );

    add_settings_field(
      'feature_flag_setting_name',
      'My Feature Flag Setting',
      'feature_flag_setting_callback_function',
      'general',
      'feature_flagged_setting_section'
    );

    register_setting( 'reading', 'feature_flag_setting_name' );
  }
}

add_action( 'admin_init', 'feature_flagged_settings_api_init' );

function feature_flagged_setting_section_callback_function() {
  echo'<p>This section is hidden completely behind a Kernl Feature Flag.</p>';
}

function feature_flag_setting_callback_function() {
  echo'<input name="feature_flag_setting_name" id="feature_flag_setting_name" type="checkbox" value="1" class="code" '.checked( 1, get_option( 'feature_flag_setting_name' ), false ) .' /> This checkbox is hidden behind a feature flag.';
}

?>
Now that we have the feature flag in our code, let’s talk about some of the optimizations the WPFeatureFlag library has. One of the nice things about WordPress and PHP is that the code itself is stateless. Meaning that without some storage mechanism (MySQL, Redis, Memcache) the entire page and all of it’s data is rebuilt from scratch on every request. This is great for fast development cycles but not always for performance.
The WPFeatureFlag library helps with performance by storing flags for a given user as a WordPress transient for 5 minutes. This way repeated page requests by a user don’t constantly call Kernl and introduce network latency on the request. Kernl’s feature flag API is heavily cached but it’s better for the end user if flags are served out of your own store. What this means for you is that your user’s won’t see changes for a maximum of 5 minutes when you toggle a flag on/off. This is usually fine, but if you need shorter or longer intervals you can use the API directly.
That being said, go ahead and toggle the feature flag off in Kernl. In 5 minutes (or less) you’ll see the setting disappear from the admin. No code deploy needed. “That’s great!” you say, but what about beta program management? Easy. Let’s change this flag to an “individual” flag.

Create an individually targeted feature flag

The video below show’s you how to create an individually targeted feature flag. It’s the same as the on/off flag, except that you get to pick which user’s see the feature. One caveat with this is that if Kernl hasn’t seen the user yet we can’t target them. Why? Because we don’t know how to identify a user that we haven’t seen. If you went to target an individual user without having identified them yet, you would need to register them manually.

Now that the flag is targeted at an individual, that selected person will be able to see the menu setting. This is a contrived example, but you can see how this can be easily expanded to running a beta program. The best part is that you don’t need to have multiple versions of your plugin/theme out there. You can simply release one version, and toggle on features for specific people. In the future Kernl will support making groups of users so managing beta programs will be even easier!

If you have questions feel free to drop them in the comments!

Further Reading on Feature Flags

There’s a lot of great reading out there on feature flags and their uses. If you’re looking for more information about them, I highly recommend these resources.