Today we launched the next iteration of Kernl Analytics. The agency plan has been long in the making and we hope that you enjoy the new insights that you can extract with it.
Features
The Kernl Analytics agency plan is very similar to the “small” plan with two key differences:
Increased DataRetention – When the agency plan is selected, Kernl will hold on to your analytics data for 90 days (instead of the small plan’s single day). This also means that you can select a day in the past and see your analytics for it.
Compare Dates – With the agency plan you can select two dates and compare their data against each other. This is extremely useful if you would like to see adoption curves for WordPress versions, PHP versions, and installed versions of your plugin/theme. It allows you to make smart business decisions based on real data.
The Kernl Analytics agency plan is now available to all Kernl customers. The fee is $30/month on top of your existing Kernl plan. Reach out to jack@kernl.us if you have any questions!
It was great month for Kernl! We didn’t do much in the way of user-facing features, but we did accomplish a lot of great infrastructure work.
Features, Bugs, & Infrastructure
Analytics Domain Search Speed Improvements – Prior to this work is could take up to 5 seconds to filter through the list of domains in Kernl Analytics. You can now search with sub-second response times due to a well placed index in Postgres.
Daily Aggregates Cleanup – This task encompassed cleaning up the millions of rows of aggregate data that Kernl was hanging on to. We weren’t in jeopardy of running out of space, but a table of 300K rows is faster to query than a table of 20M rows.
Analytics Domain List Clickable URLs – A customer suggested that the urls in the domain list should be clickable, so now they are!
Marketing Site Scaled Images – Kernl now serves properly scaled images for the marketing site.
GZIP Compression – All /static routes now serve resources using GZIP compression.
MongoDB Upgrade – We’re now on Mongo 3.4 and the database is hosted in AWS via Compose.io
Node.js Upgrade – Kernl now runs on Node.js 8.11.4. This upgrade addresses some security issues and bug fixes.
Below is a post-mortem analysis of Kernl’s August 14 2018 outage. It details what happened, why it happened, and how we can improve things so that this doesn’t happen again in the future.
What?
At 9:29PM EDT alerts were triggered saying that Kernl was down. Initial investigation showed that the marketing site was still up and that some (but not all) update requests were still going through. Upon further investigation, it was found that Kernl’s connection to MongoDB had stopped working.
In most situations the DB connection dropping would have caused only a momentary blip while we failed over to our secondary, but in this case that wasn’t possible.
Why?
Further investigation revealed that our Mongo provider (Compose.io) was experiencing an outage in some of their Digital Ocean environments. Unfortunately this outage effected not only our primary Mongo host but also our backup secondary host. Due to the nature of the outage automatic failover for Mongo wasn’t a possibility.
Actions Taken
Determined the source of the downtime.
Contacted Compose.io support to resolve the issue.
After not hearing back from support for 30 minutes, work was started on an alternate plan for bringing Kernl back up.
After 45 minutes of no response from Compose.io support the alternate plan was enacted.
The alternate plan was to restore Kernl’s Mongo cluster into a different data center using the daily backup. The only downside was that that backup was 6 hours old, which means there is a possibility that customers will need to re-build or re-upload some plugin and theme versions.
Cache lifetime for all Kernl endpoints was doubled. This was done because the new data center was outside of Digital Ocean NYC3. The increased cache lifetime helps combat the increased latency.
Kernl’s downtime ended at roughly 11:00PM. The Compose.io incident wasn’t resolved until several hours after this so we feel that the decision to restore from a backup was the right one.
What’s Next?
Compose.io has been our Mongo provider for years now and we’ve never experienced any significant downtime. That being said, they don’t actually support DigitalOcean anymore and plan to kill their support for it at sometime in the future.
Our next steps are to evaluate how Kernl performs with the database in another data center. If things look good, we will likely move Kernl’s Mongo instances to the new data center permanently where they can be better supported by Compose. It is our suspicion that if we had been in one of their more popular data centers that we would have received help faster.
Once again, apologies for the downtime and we’ll continue to work hard so that it doesn’t happen again!
Over the past 3 years I’ve often received requests from new and existing Kernl customers for some form of analytics on their plugin/theme. I avoided doing this for a long time because I wasn’t sure that I could do so economically at the scale Kernl operates at, but I eventually decided to give Kernl Analytics a whirl and see where things ended up.
Product Versions Graph
Concerns
After deciding to give the analytics offering a try, I had to figure how to build it. When I first set out to build Kernl Analytics I had 3 main concerns:
Cost – I’ve never created a web service from scratch that needs to INSERT data at 75 rows per second with peaks of up to 500 rows per second. I wanted to be sure that running this service wouldn’t be prohibitively expensive.
Scale – How much would I need to distribute the load? This is tightly coupled to cost.
Speed – This project is going to generate a LOT of data by my standards. Can I query it in performant manner?
As development progressed I realized that cost and scale were non-issues. The database that I chose to use (PostgreSQL) can easily withstand this sort of traffic with no tweaking, and I was able to get things started on a $5 Digital Ocean droplet.
Kernl Analytics Architecture & Technology
Kernl Analytics was created to be it’s own micro-service with no public access to the world. All access to it is behind a firewall so that only Kernl’s Node.js servers can send requests to it. For data storage, PostgreSQL was chosen for a few reasons:
Open Source
The data is highly relational
Performance
The application that captures the data, queries it, and runs periodic tasks is a Node.js application written in TypeScript. I chose TypeScript mostly because I’m familiar with it and wanted type safety so I wouldn’t need to write as many tests.
TypeScript FTW!
With regards to size of the instance that Kernl Analytics is running on, I currently pay $15/month for a 3 core Digital Ocean droplet. I upgraded to 3 cores so that Postgres could easily handle both writes and multiple read requests at the same time. So far this setup has worked out well!
Pain Points
Overall things went well while implementing Kernl Analytics. In fact they went far better than expected. But that doesn’t mean there weren’t a few pain points along the way.
WriteVolume – Kernl’s scale is just large enough to cause some scaling and performance pains when creating an analytics service. Kernl averages 25 req/s which translates to roughly 75 INSERTs into Postgres. Kernl also has peaks of 150 req/s which scales up to about 450 INSERTs into Postgres. Postgres can easily handle this sort of load, but doing it on a $5 digital ocean droplet was taxing to say the least.
Hardware Upgrade – I tried to keep costs down as much as possible with Kernl Analytics, but in the end I had to increase the size of the droplet I was using to a $15 / 3-core droplet. I ended up doing that so one or two cores could be dedicated to writes while leaving a single core available for read requests. Postgres determines what actions are executed where, but adding more cores had led to a lot less resource contention.
Aggregation – Initially the data wasn’t aggregated at all. This caused some pain because even with some indexing, plucking data out of a table with > 2.5 million rows can be sort of slow. It also didn’t help that I was writing data constantly to the table, which further slowed things down. Recently I solved this by doing daily aggregations for Kernl Analytics charts and domain data. This has improved speed significantly.
Backups & High Availability – To keep costs down the analytics service is not highly available. This is definitely one of those “take out some tech debt” items that will need to be addressed at a later date. Backups also happen only on a daily basis, so its possible to lose a day of data if something serious goes wrong.
Yay for affordable hosting
Future Plans
Kernl Analytics is a work in progress and there is always room to improve. Future plans for the architecture side of analytics are
Optimize Indexes – I feel that more speed can be coaxed out of Postgres with some better indexing strategies.
Writes -vs- Reads – Once I gain a highly available setup for Postgres I plan to split responsibilities for writing and reading. Writes will go to the primary and reads will go to the secondary.
API – Right now the analytics API is completely private and firewalled off. Eventually I’d like to expose it to customers so that they can use it to do neat things.
Happy (almost) August everyone! This month with Kernl was focused on fixing some technical debt and adding a few features surrounding analytics.
Features & Bug Fixes
Analytics Top Level Menu – Kernl Analytics now has a top-level menu item. Prior to this change you had to enter a plugin/theme page before you could access it.
Analytics Product & Date Selector – Coupled with the analytics top-level menu, you can now select which product you want to see analytics for directly in the page. You can also select the date if you have the “agency” plan or above.
Session Store Moved to Memcached – For most of Kernl’s life sessions have been stored in Mongo. Recently we moved to storing sessions in Memcached.
GitLab Integration Bug Fix – The GitLab integration was broken for a few days after GitLab disabled their v3 API. This has been resolved.
The .kernlignore file had a few bugs related to processing it. These have been resolved.
Along with the session storage change we cleaned up a few collections in Mongo.
I hope everyone in the northern hemisphere is enjoying their summer and for those of you in the southern hemisphere, stay warm! It was a nice month for Kernl with lots of good structural changes and a few new features rolled out.
SendOwl Integration – If you use SendOwl to distribute your plugin or theme you can now validate license keys with Kernl. This means that every time a customer checks to see if an update is available Kernl will first validate their SendOwl license.
Analytics Aggregate Data – Kernl Analytics now uses aggregate data to populate charts. This means that charts load instantly versus taking a few seconds as they did before. This was a big change and enables us to do neat things in the future like calculating changes over time.
Analytics Domains – In addition to using aggregate data to populate charts Kernl Analytics now has improved domain list support. Data is properly paginated, populated via aggregates for speed, and searchable.
Version Number Improvements – Kernl now supports version numbers such as 10.2.2-alpha or 9.2.1-beta. Previously the alpha|beta tags at the end were not supported.
Download Graph Bug Fix – A customer reported that the download chart in the plugin/theme detail pages weren’t quite right. This bug has been fixed.
License Management – The license management page was occasionally showing duplicates. This bug has been fixed.
It’s been a great month for Kernl! Lots of new features, some bug fixes, and few updates to the license checking on the plugin and theme update checker files. Lets dive in!
Features
Gumroad License Validation – You can now use Kernl to validate your Gumroad licenses! This can be enabled for plugins or themes by going to the product edit screen, clicking the “License Management” tab, and then selecting “Validate Gumroad License?” at the bottom.
Kernl Referral Program – Kernl has a referral program. For every 3 referrals you send us we’ll give you a free month. Customers signing up with your referral code get their first 3 months free.
Restrict Updates to a Maximum Version – If you use Kernl’s license management you can now restrict update availability to a maximum version. For example if the current version of your product was 1.5.0 and you gave the user a license for < 1.6.0, then they would receive updates all the way through 1.5.X. This is a great way to drive more sales of your product!
Plugin/Theme PHP Update Check Files – The license error display behavior of these files has been greatly improved. The error dismisses when it’s supposed to, only shows up on the updates and plugin|theme page, and the license error message can now be customized. It is highly recommended that you update. The file is also versioned now so knowing when to update in the future will be much easier.
Other
Purchase Code Deprecation – Kernl’s old purchase code frontend interface has been hidden behind a feature flag. The goal is the have the old purchase code functionality completely removed by the end of July.
Copy Versions from Product to Product – To support special development styles, you can move versions of a product to another product. This is inherently dangerous and only toggled on for the person who requested it. If you think this might be useful please reach out to jack@kernl.us.
You can now easily click to a customer’s page from the License Management page.
A bug was fixed where the customer filter would stay set even after you had navigated away from that page.
Numerous copy changes were made on the License Management page and on the marketing site.
Some feature flags were removed from features that have proved to be stable.
The main route for plugin update checks (also the highest traffic route on Kernl) was refactored to use async/awaitinstead of promise chains. This makes it much easier to maintain and improve.
If you use Gumroad to sell your WordPress plugins or themes you can now use Kernl to validate those licenses before new updates are made available.
How does it work?
When you create a plugin/theme in Kernl you can select the “Validate Gumroad License?” checkbox and fill out the “Gumroad Product Permalink” field. With those two fields completed Kernl will validate the license code passed into the Kernl via the update checker against Gumroad’s public license API. If the license code passes validation Kernl will allow the update to proceed.
That’s all there is to it! If you need any help getting things set up reach out to jack@kernl.us.
Kernl WordPress license management now allows you to limit updates up to a specific version.
How it Works
Assuming that you already use Kernl’s license management, go to the License Management area of Kernl. Once there add or edit a license.
Now you can fill in the “Max Update Version” field. This field can simply be described as “the version of your product which requires a customer to buy a new license”. For example:
Customer A bought a license with “Max Update Version” set to 2.0.0. The product was at version 1.7.0 at the time of purchase. Over the next few months you release 1.8.0, 1.9.0, 1.9.1, 2.0.0, 2.0.1. Customer A only receives product update versions 1.8.0, 1.9.0, and 1.9.1.
Why should I use this?
Drive. More. Sales. This new feature allows you to be extremely granular about what updates a specific customer receives. We also made updates to plugin_update_check.php and theme_update_check.php that allow you to customize the invalid/expired license message. We strongly believe that this combination of better license expiration messages and limiting through specific update version can be instrumental in helping you drive more sales.
For most of Kernl’s life our best channel for adding new customers has been word-of-mouth referrals and now you can be rewarded for referring new customers!
How It Works
When you go to your Kernl profile you’ll see a new section called “Referrals”.
It briefly explains how the Kernl referral program works, gives you your referral link, tells you how many referrals you’ve made, and how many free months you’ve earned.
Rewards
The rewards for the Kernl referral program are as follows:
For the referrer: Every 3 customers you refer to Kernl earn you one free month. No restrictions on plan or usage. There is a max of 24 referrals. If you somehow manage to bump in to this restriction, shoot us an email and we’ll work with you. 🙂
For the new customer: By using your referral code they earn 3 free months of Kernl (instead of the usual 30 day free trial).
That’s it! Kernl’s referral program is intentionally simple. Thanks to everyone who spreads the word about Kernl. The more customers we have the better we become.
