Building & Scaling Kernl Analytics

Over the past 3 years I’ve often received requests from new and existing Kernl customers for some form of analytics on their plugin/theme. I avoided doing this for a long time because I wasn’t sure that I could do so economically at the scale Kernl operates at, but I eventually decided to give Kernl Analytics a whirl and see where things ended up.

kernl analytics product versions
Product Versions Graph

Concerns

After deciding to give the analytics offering a try, I had to figure how to build it. When I first set out to build Kernl Analytics I had 3 main concerns:

  • Cost – I’ve never created a web service from scratch that needs to INSERT data at 75 rows per second with peaks of up to 500 rows per second. I wanted to be sure that running this service wouldn’t be prohibitively expensive.
  • Scale – How much would I need to distribute the load? This is tightly coupled to cost.
  • Speed – This project is going to generate a LOT of data by my standards. Can I query it in performant manner?

As development progressed I realized that cost and scale were non-issues. The database that I chose to use (PostgreSQL) can easily withstand this sort of traffic with no tweaking, and I was able to get things started on a $5 Digital Ocean droplet.

Kernl Analytics Architecture & Technology

Kernl Analytics was created to be it’s own micro-service with no public access to the world. All access to it is behind a firewall so that only Kernl’s Node.js servers can send requests to it. For data storage, PostgreSQL was chosen for a few reasons:

  1. Open Source
  2. The data is highly relational
  3. Performance

The application that captures the data, queries it, and runs periodic tasks is a Node.js application written in TypeScript. I chose TypeScript mostly because I’m familiar with it and wanted type safety so I wouldn’t need to write as many tests.

kernl analytics and typescript
TypeScript FTW!

With regards to size of the instance that Kernl Analytics is running on, I currently pay $15/month for a 3 core Digital Ocean droplet. I upgraded to 3 cores so that Postgres could easily handle both writes and multiple read requests at the same time. So far this setup has worked out well!

Pain Points

Overall things went well while implementing Kernl Analytics. In fact they went far better than expected. But that doesn’t mean there weren’t a few pain points along the way.

  • Write Volume – Kernl’s scale is just large enough to cause some scaling and performance pains when creating an analytics service. Kernl averages 25 req/s which translates to roughly 75 INSERTs into Postgres. Kernl also has peaks of 150 req/s which scales up to about 450 INSERTs into Postgres. Postgres can easily handle this sort of load, but doing it on a $5 digital ocean droplet was taxing to say the least.
  • Hardware Upgrade – I tried to keep costs down as much as possible with Kernl Analytics, but in the end I had to increase the size of the droplet I was using to a $15 / 3-core droplet. I ended up doing that so one or two cores could be dedicated to writes while leaving a single core available for read requests. Postgres determines what actions are executed where, but adding more cores had led to a lot less resource contention.
  • Aggregation – Initially the data wasn’t aggregated at all. This caused some pain because even with some indexing, plucking data out of a table with > 2.5 million rows can be sort of slow. It also didn’t help that I was writing data constantly to the table, which further slowed things down. Recently I solved this by doing daily aggregations for Kernl Analytics charts and domain data. This has improved speed significantly.
  • Backups & High Availability – To keep costs down the analytics service is not highly available. This is definitely one of those “take out some tech debt” items that will need to be addressed at a later date. Backups also happen only on a daily basis, so its possible to lose a day of data if something serious goes wrong.
kernl analytics server load
Yay for affordable hosting

Future Plans

Kernl Analytics is a work in progress and there is always room to improve. Future plans for the architecture side of analytics are

  • Optimize Indexes – I feel that more speed can be coaxed out of Postgres with some better indexing strategies.
  • Writes -vs- Reads – Once I gain a highly available setup for Postgres I plan to split responsibilities for writing and reading. Writes will go to the primary and reads will go to the secondary.
  • API – Right now the analytics API is completely private and firewalled off. Eventually I’d like to expose it to customers so that they can use it to do neat things.

What’s New With Kernl – August 2018

Happy (almost) August everyone! This month with Kernl was focused on fixing some technical debt and adding a few features surrounding analytics.

Features & Bug Fixes

  • Analytics Top Level Menu – Kernl Analytics now has a top-level menu item. Prior to this change you had to enter a plugin/theme page before you could access it.
  • Analytics Product & Date Selector – Coupled with the analytics top-level menu, you can now select which product you want to see analytics for directly in the page. You can also select the date if you have the “agency” plan or above.
  • Session Store Moved to Memcached – For most of Kernl’s life sessions have been stored in Mongo. Recently we moved to storing sessions in Memcached.
  • GitLab Integration Bug Fix – The GitLab integration was broken for a few days after GitLab disabled their v3 API. This has been resolved.
  • The .kernlignore file had a few bugs related to processing it. These have been resolved.
  • Along with the session storage change we cleaned up a few collections in Mongo.
  • Node.js was upgraded to 8.11.3.

What’s New With Kernl – July 2018

I hope everyone in the northern hemisphere is enjoying their summer and for those of you in the southern hemisphere, stay warm! It was a nice month for Kernl with lots of good structural changes and a few new features rolled out.

  • SendOwl Integration – If you use SendOwl to distribute your plugin or theme you can now validate license keys with Kernl. This means that every time a customer checks to see if an update is available Kernl will first validate their SendOwl license.
  • Analytics Aggregate Data – Kernl Analytics now uses aggregate data to populate charts. This means that charts load instantly versus taking a few seconds as they did before. This was a big change and enables us to do neat things in the future like calculating changes over time.
  • Analytics Domains – In addition to using aggregate data to populate charts Kernl Analytics now has improved domain list support. Data is properly paginated, populated via aggregates for speed, and searchable.
  • Version Number Improvements – Kernl now supports version numbers such as 10.2.2-alpha or 9.2.1-beta. Previously the alpha|beta tags at the end were not supported.
  • Download Graph Bug Fix – A customer reported that the download chart in the plugin/theme detail pages weren’t quite right. This bug has been fixed.
  • License Management  – The license management page was occasionally showing duplicates. This bug has been fixed.

Thats it for this month!

 

 

What’s New With Kernl – June 2018

It’s been a great month for Kernl! Lots of new features, some bug fixes, and few updates to the license checking on the plugin and theme update checker files. Lets dive in!

Features

  • Gumroad License Validation – You can now use Kernl to validate your Gumroad licenses! This can be enabled for plugins or themes by going to the product edit screen, clicking the “License Management” tab, and then selecting “Validate Gumroad License?” at the bottom.
  • Kernl Referral Program – Kernl has a referral program. For every 3 referrals you send us we’ll give you a free month. Customers signing up with your referral code get their first 3 months free.
  • Restrict Updates to a Maximum Version – If you use Kernl’s license management you can now restrict update availability to a maximum version. For example if the current version of your product was 1.5.0 and you gave the user a license for < 1.6.0, then they would receive updates all the way through 1.5.X. This is a great way to drive more sales of your product!
  • Plugin/Theme PHP Update Check Files – The license error display behavior of these files has been greatly improved. The error dismisses when it’s supposed to, only shows up on the updates and plugin|theme page, and the license error message can now be customized. It is highly recommended that you update. The file is also versioned now so knowing when to update in the future will be much easier.

Other

  • Purchase Code Deprecation – Kernl’s old purchase code frontend interface has been hidden behind a feature flag. The goal is the have the old purchase code functionality completely removed by the end of July.
  • Copy Versions from Product to Product – To support special development styles, you can move versions of a product to another product. This is inherently dangerous and only toggled on for the person who requested it. If you think this might be useful please reach out to jack@kernl.us.
  • You can now easily click to a customer’s page from the License Management page.
  • A bug was fixed where the customer filter would stay set even after you had navigated away from that page.
  • Numerous copy changes were made on the License Management page and on the marketing site.
  • Some feature flags were removed from features that have proved to be stable.
  • The main route for plugin update checks (also the highest traffic route on Kernl) was refactored to use async/await instead of promise chains. This makes it much easier to maintain and improve.

Kernl Now Supports Gumroad License Validation

If you use Gumroad to sell your WordPress plugins or themes you can now use Kernl to validate those licenses before new updates are made available.

How does it work?

When you create a plugin/theme in Kernl you can select the “Validate Gumroad License?” checkbox and fill out the “Gumroad Product Permalink” field. With those two fields completed Kernl will validate the license code passed into the Kernl via the update checker against Gumroad’s public license API. If the license code passes validation Kernl will allow the update to proceed.

That’s all there is to it! If you need any help getting things set up reach out to jack@kernl.us.

License Management Update: Limit Updates to Specific Version

Kernl WordPress license management now allows you to limit updates up to a specific version.

How it Works

Assuming that you already use Kernl’s license management, go to the License Management area of Kernl. Once there add or edit a license.

Now you can fill in the “Max Update Version” field. This field can simply be described as “the version of your product which requires a customer to buy a new license”. For example:

  • Customer A bought a license with “Max Update Version” set to 2.0.0. The product was at version 1.7.0 at the time of purchase. Over the next few months you release 1.8.0, 1.9.0, 1.9.1, 2.0.0, 2.0.1. Customer A only receives product update versions 1.8.0, 1.9.0, and 1.9.1.

Why should I use this?

Drive. More. Sales. This new feature allows you to be extremely granular about what updates a specific customer receives. We also made updates to plugin_update_check.php and theme_update_check.php that allow you to customize the invalid/expired license message. We strongly believe that this combination of better license expiration messages and limiting through specific update version can be instrumental in helping you drive more sales.

Introducing the Kernl Referral Program

For most of Kernl’s life our best channel for adding new customers has been word-of-mouth referrals and now you can be rewarded for referring new customers!

How It Works

When you go to your Kernl profile you’ll see a new section called “Referrals”.

It briefly explains how the Kernl referral program works, gives you your referral link, tells you how many referrals you’ve made, and how many free months you’ve earned.

Rewards

The rewards for the Kernl referral program are as follows:

  • For the referrer: Every 3 customers you refer to Kernl earn you one free month. No restrictions on plan or usage. There is a max of 24 referrals. If you somehow manage to bump in to this restriction, shoot us an email and we’ll work with you. 🙂
  • For the new customer: By using your referral code they earn 3 free months of Kernl (instead of the usual 30 day free trial).

That’s it! Kernl’s referral program is intentionally simple. Thanks to everyone who spreads the word about Kernl. The more customers we have the better we become.

What’s New With Kernl – May 2018

Welcome to May! Lets go ahead and dive right in.

  • Data Export – You can now export (most) of your Kernl data. To export your data, log in to Kernl, go to your profile, and scroll down to the “Export My Data” button.
  • Node.js Upgraded – Kernl has been updated to use the latest LTS version of Node.js. We are now on 8.11.1.
  • Kernl Analytics Hardware Upgraded – Kernl analytics sees a lot of traffic on a daily basis. As that number continued to grow we were seeing some resource contention issues during peak periods. We were able to mitigate all resource issues by upgrading our hardware from a single vCPU setup to a three vCPU setup.
  • https://kernl.us/wordpress-installation-statistics – Check out our new public WordPress installation statistics page! Most of this information is readily available on WordPress.org, but Kernl’s data is interesting because it only includes data points from websites that purchase plugins or themes. The data is also available via our API for public consumption.

Thats it for this month! I hope everyone in the northern hemisphere is enjoying the start of summer.

Tips to Keep a WordPress Site Safe

If you want to get a website developed for your business you should definitely take a look at WordPress. It can be considered one of the most convenient methods available for getting a website up and running. But when you are building your website with WordPress, you should never overlook security.

When you are done with creating your website you are open to a whole world. If you have not integrated the necessary security precautions your website might face some serious security issues. Your website might contain sensitive data like; customers credit card details and personal details. Therefore, if you have not taken precautions, you and your customers may be exposed to a considerable risk of being hacked.

Once a hacker gets into your website they can not only steal your information but also destroy all the information of your website. The most dangerous part of such destruction is you can never undo such disastrous act. Therefore it is your duty to take the necessary precautions to discourage hackers and keep your data safe and let the business continue to run smoothly.

Below are some security precautions you can think consider for your website. By taking these steps you can help seriously reduce the chance that you’ll get hacked.

1.  Stay Updated

Be aware of the latest threats in the digital world. You should at least know the basic information of the possible threats and which will help you decide what  precautions you should take. There are few websites which can assist you to get up to date information. You can take a look at those websites and then you will be able to stay updated with ease.

2.  Lock Down Your Access Control

Make sure that your basic precautions are tough enough to discourage a hacker. Never use easy-to-guess passwords and usernames. It is much better if you change the default database prefix from wp to a harder and more random prefix. You can even limit login attempts. If there is more than one user on your WordPress website, you need to introduce strong security policies for all of those users. Following these precautions can give you the peace of mind of knowing that your website cannot be hacked by guessing the password.

3.  Software updates are crucial

It is important to update your software regularly. Do not neglect the updates. What if the most recent update is about security vulnerability and you have neglected it? Hackers are quick to become familiar with security issues in WordPress, so take advantage of the security updates that WordPress provides. Ideally you should automatically install all core WordPress updates.

4.  Install Security Ninja

If you’re wondering how to keep your WordPress site safe, you must take a look at the security plugins available as well. That’s where Security Ninja comes into play. It is one of the best and most popular security plugins available WordPress users. Security Ninja makes it easy to secure your WordPress site.

Security Ninja only takes a minute to scan your whole site for security issues and then helps you deal with them one by one. Easy and efficient!

WP Security Ninja

5.  Maintain a strong network security system

You should pay a good attention to the security your network. You never know if users in your office inadvertently provide easy access to your servers. Therefore, make sure that you have taken action to:

  • Expire user logins after a short idle period
  • Frequently change the passwords
  • Use strong passwords and never write them down.
  • Scan all the peripherals whenever you use them with the computer

6.  Try a Web Application Firewall (WAF)

WAF can be software, hardware or can be a combination of both. It stands between your web server and the data connection to read each and every bit of data transmitted. WAFs have the ability of blocking malicious accesses and hacking attempts to provide you the peace in mind. There are some built-in firewall options available in WordPress for you to try out. You can take a look at them as well. 

Again, we advise checking the Security Ninja PRO plugins which now feature a Cloud Firewall that will protect you.

7.  Install security applications

Although not affective as WAF’s, security applications can make hacking attempts difficult to a certain extent. There are free and paid applications available to download. These are good in preventing automated hacking attempts

8.  Use SSL

You have the option of using an encrypted SSL protocol when transferring your users’ sensitive data. In fact, this can protect your information from being read by unauthorized persons. Setting up SSL with WordPress is pretty easy and should be considered an important step in securing any WordPress installation.

9.  Backing up

Last but not least, you should back up your website’s data both online and offline very often. This will be useful if any damage is caused to your website. Use an automated backup system to backup the website multiple times a day.

Introducing the Kernl WordPress Statistics Page

On an average day Kernl handles around 2 million requests from 135,000 unique domains. Our analytics offering lets you use this data to make better product development decisions, but there is also a more holistic view of the Kernl and WordPress ecosystem to be had. I’d like to introduce to you the Kernl WordPress Statistics page!

What is it?

The statistics page gives you a high level overview of Kernl ecosystem. This is a subset of the overall WordPress ecosystem with the important distinction that every domain represented has paid for a plugin or theme. It includes information around WordPress versions, PHP versions, and the language that the site is in.

Kernl WordPress Statistic Image

 

API

There are a lot of neat visualizations that you could make with this data over time, so the API is exposed and can be used by anyone. Check out the Kernl API documentation to get started. No authentication required.

Awesome! Where do I go to see this in action?

Check out https://kernl.us/wordpress-installation-statistics , or go to the Kernl homepage and scroll all the way to the bottom.