What’s New With Kernl – December 2019

Welcome to the end of 2019! I hope that everyone has had as good a year as Kernl. Let’s dive in to the final update of 2019 to see what’s new.

Features, Bug Fixes, & Misc.

  • Improved License Management Search – License management now includes improved search functionality. The previous search functionality was flaky (at best) and not very discoverable. Search is now a first-class citizen, includes free-text search, and should greatly improve the overall usability of Kernl’s WordPress license management.
  • Load Testing Unit & Integration Tests – When we created the WordPress load testing service it was an experiment. Now that we have proved the viability of the service it’s time to work on stability and overall platform longevity. This month’s focus was on the authorization framework that our load testing service uses.
  • JS Bundle Size Reduction – Over the past year Kernl’s JS bundle size for our web app grew to over 2MB. We spent some time this month figuring out why and making changes to reduce it. In the end we were able to reduce the bundle size by over 50% down to 1.1MB.
  • Bug: Inconsistent Webhook & Deploy Key Behavior – After a few customer reported incidents with the automatic webhook and deploy key behavior, we discovered that Kernl wasn’t deleting local references to remote keys and hooks. If you have some issues with deploy keys or webhooks please contact us and we can help resolve the data inconsistency issues that this caused.
  • Node.js Upgrade to 12.14.0 – This month we upgrade all of our servers to use the latest LTS version of Node. This is includes stability and performance improvements.

That’s it for this year. See everyone in 2020!

Yoast SEO: WordPress Plugin Performance Implications

Yoast SEO is a popular SEO enablement plugin for WordPress. It helps you avoid common mistakes when it comes to SEO on your blog and also handles things like social media “og:meta” tags. If your blog has a public audience, then it stands a good chance that you are using Yoast.

As with the previous article in this series, we’ll focus on what performance implications are of using Yoast and how you can make it faster.

What was tested?

For this test we used the lowest-tier (1vCPU, 1GB RAM, $5) DigitalOcean droplet out of their SFO2 datacenter. Our server setup was as follows:

  • Ubuntu 19.04 with all updates installed.
  • Nginx 1.16.1
  • PHP-FPM 7.3
  • MariaDB 10.3

The theme that was used was TwentyTwenty with no modifications and the content tested was 3 “What’s new with Kernl?” posts from earlier this year. We selected a small number due to the effort of filling out all of the SEO data in Yoast.

The WordPress setup was bare-bones. There were no plugins installed except for when we were running the Yoast test.

As with our previous post in this series, the load for this test was generated out of DigitalOcean’s NYC3 datacenter.

Test Methodology

For testing Yoast we only ran two tests:

The tests were with 200 concurrent users over the course of 1 hour.

Max Requests per Second

One method for determining website performance is what is the maximum number of requests that in can field in a given second. For our purposes this is a pretty good indicator of the performance hit you take for installing plugins.

Yoast SEO - Max Requests / Second
Higher is better.

As you can see from the image above you lose about 25% of your maximum capacity from installing Yoast SEO. With no plugins installed we were able to hit 43 req/s, while with Yoast installed that number went down to 30 req/s.

It’s worth noting here that 30 req/s is 2.5 million requests a day.

First Error Occurrence

The next chart shows when we first started to see errors in our two tests.

Yoast SEO - First Error Occurrence
Higher is better.

Without the plugin installed WordPress was able to hit 38 req/s before seeing errors. Once we enabled Yoast that number went down to 28 req/s. Once again, this is consistent with the performance penalty we saw with the maximum requests per second of about 25%.

Average Response Time

The average response time with and without Yoast SEO tells a similar story to the requests per second measures we have done.

Yoast SEO - Average Response Time
Less is better.

The chart above shows us that without any plugins installed, the average response time under load is around 3000ms. With the Yoast plugin installed the response time goes up to about 4300ms. We’re looking at a roughly 25% change in response time.

99th Percentile Response Time

The 99th percentile chart can be read as “99% of all requests finished in under this time”.

Yoast SEO - 99th Percentile Response Time
Smaller is better.

The chart above tells us that without any plugins installed 99% of our requests finished in under ~3600ms. With Yoast SEO installed 99% of our requests finished in ~4900ms. Once again, a roughly 25% penalty for having Yoast installed.

Yoast SEO Performance Conclusions

Yoast is a really good SEO plugin. If SEO matters to you it’s definitely a plugin you should have installed. However you should use caching if you do use it. This goes for WordPress in general, but every plugin you add to WordPress has a performance cost associated with it. If you run a site where caching is difficult you’ll have to carefully weigh the performance cost versus benefit of installing Yoast SEO.

Cloudways WordPress Performance Value Review

Cloudways is a managed WordPress hosting provider that allows you to deploy your WordPress site onto several different platforms. These platforms are:

Cloudways home page
Cloudways!

If you are just starting with Cloudways it can be tough to figure out where your money is best spent! The prices are different for each platform and how they are sized isn’t them same either. This review gathers data using Kernl’s WordPress Load Testing tool and then massages that data into someone easy to digest.

Load & performance test your own WordPress site with Kernl’s WordPress Load Testing service!

What Was Tested?

We tested the following Cloudways setups:

  • Google Cloud – A “small” instance out of their Northern Virginia data center. $39.36 / month.
  • Amazon Web Services – A “small” instance out of their Northern Virginia data center. $36.51 / month.
  • Vultr – “2GB” plan out of their New Jersey data center. $23 / month
  • Linode – “2GB” plan from their New Jersey data center. $24 / month
  • Digital Ocean – “2GB” plan from one of the New York City data centers. $22 / month.

And we had the following load testing setup:

  • Content was an export of this blog
  • 1000 concurrent users
  • 30 minute duration
  • 3 users per second ramp up
  • Load generating machines were in the San Francisco #2 Digital Ocean data center.

Overall Cloudways Performance and Value

Each provider performed extremely well in our tests. No errors were reported and they all reached over 700 requests / s sustained traffic. However not all hosts are created equal.

AWSGCPLinodeVultrDigital Ocean
Costs / Month (cents)36513936240023002200
Total Requests13409671274018144230114557221445759
$ / 1000 req (cents)2.723.091.661.581.52

Given that each load test was exactly the same, you can see that some hosts performed better than others.

Cost per 1000 requests (in cents)

AWS and GCP are a bit more costly than Linode, Vultr, and Digital Ocean and they also performed worse than the lower cost providers.

Cloudways Provider Response Times

While cost per requests is a good metric we also care about the quality of those requests. Quality can be measured in a number of ways, but response time is often a “good enough” measure of how well a particular host performs. For the providers that you can get through Cloudways, the variance in request quality is extremely interesting.

The chart below shows the 99th percentile response time performance for each provider. It means that 99% of all requests during the load test finished at or below this time.

99th percentile performance

You can see that lower cost providers once again out-perform the higher cost AWS and GCP in pretty significant ways. The really interesting result here is that Vultr responded to 99% of requests in 240ms or less! Way to go Vultr!

Conclusions

If you aren’t sure which provider is the best value on Cloudways, from a raw cents per requests perspective you can’t go wrong with Linode, Vultr, and Digital Ocean. It’s the same story when you consider the quality (re: response time) of service: Vultr, Linode, and Digital Ocean are the clear winners here.

Full WordPress Load Test Results:

Load & performance test your own WordPress site with Kernl’s WordPress Load Testing service!

What’s New With Kernl – November 2019

November was a great month for Kernl! After several years of trying, we finally launched team management and also made our pricing structure easier to understand. Let’s dive in!

Features

  • Team Management – With our new Agency and Unlimited plans you can now grant users access to your account! The Agency plan allows for 3 team members and unlimited allows for unlimited team members.
  • Agency & Unlimited Plans – We introduced an unlimited plan for Kernl which has no usage limits (with the exception of load testing, but those limits have been increased substantially) and includes team management and Kernl Analytics. We also updated our agency plan to more closely match the old enterprise plan. The new agency plan has much higher limits than the previous agency plan as well as access to team management.

Bug Fixes & Other

  • Increased Load Test Generator VM Size – We’ve increased the load test generator machines from 1vCPU to 2vCPUs to allow us to scale our load testing up to 50,000 concurrent users.
  • Repository Sync – When we changed how you connected to your Git repositories we overlooked the ability to sync them from that same location. We’ve added that ability back in.
  • Load Test List Page Performance – With some clever SQL querying and awesome Postgres built-in functions, we’ve decreased the average load time on this page by 32%.
  • Bug: Invalid license if using licenses but no versions present – An odd edge case was found where Kernl would say your license was invalid if you didn’t have any plugin/theme versions uploaded. This has been resolved.
  • Bug: IPv6 Issue on Load Generators – The virtual machines that we spin up the Digital Ocean Singapore data center were having issues communicating over IPv6. We disabled IPv6 for all load generators for the time being.
  • Load Testing Service Node.js Upgrade – The WordPress load testing service backend and workers have been upgraded from Node.js 10.x to Node.js 12.x. All packages were upgraded to their latest with this change.
  • Analytics Service Node.js Upgrade – The WordPress analytics service backend was upgraded from Node.js 10.x to Node.js 12.x. All packages were upgraded to their latest with this change.

Blog Posts

Popular WordPress Plugin Performance Implications: Wordfence

That’s it for this month! Have a great December!

Introducing Team Management and the Kernl Unlimited Plan

One of the most requested features that Kernl gets is Team Management. Team management can take on a lot of different forms, but for Kernl’s purposes it is the ability to share access via the Kernl interface with multiple people with different sets of credentials.

Team Management

Starting today you can sign up for the Kernl Unlimited plan (see below) and start managing your team. Team management allows you to share your Kernl account with authorized users. When a user is part of your team, they have a slightly restricted view of Kernl but otherwise see the same things you do. What’s restricted?

  • Continuous Deployment – Adding connected Git repositories is the responsibility of the admin. Non-admin users don’t need to see this information.
  • Billing – Once again, this is an admin responsibility. The admin is able to change plans, add credit cards, etc.
  • Team Management – Only the admin can add and remove users from their account.

Aside from that everything is the same for admin and non-admin users.

Kernl Team Management

The Unlimited Plan & Pricing Changes

In addition to team management we are also reducing the number of plans that Kernl has. Prior to this change we had 5 different plans (solo, agency, enterprise, huge, massive) which seemed like a bit much. To simplify Kernl’s pricing structure and give our customers better options we went down to 3 plans:

  • Solo – This $9 per month plan is our base plan. Most customers start here and then grow into larger plans as their needs change.
  • Agency – The $39 agency plan is essentially our former “Enterprise” plan but with increased limits for plugins, themes, licenses, load testing, and feature flags.
  • Unlimited – This new $79 unlimited plan allows you to have unlimited plugins, themes, licenses, feature flags and very high limited on load testing (50K concurrent users). This plan also includes team management so you can more easily control access to your Kernl account.

As always, current Kernl customers are grandfathered into the plan that they are currently on.

Kernl Pricing

What’s Next?

Now that Kernl has a more robust upper-level offering we hope to start rolling more features into it. For example:

  • In the near future we’ll include Kernl Analytics with Unlimited plan.
  • We’re also planning on rolling out limited team management into the Agency plan.

What’s New With Kernl – October 2019

October held a lot of great work for Kernl! Lots of refactoring, test coverage increases, and a new UI for managing Git deployments. Let’s dive in!

Features

  • Git Continuous Deployment UI Changes – Kernl now has a new and improved interface for adding continuous deployment to your projects.
  • Installed Plugin List Filter – You can now easily filter the installed plugin data in Kernl Analytics. Prior to this change the best you could do is column sort.

Infrastructure, Bugs, & Other

  • Fetching BitBucket repos with more that 20 branches failed. This has been resolved.
  • The plugin/theme webhook build log didn’t handle the case where a user had just signed up and didn’t have any build requests yet.
  • Easy Digital Downloads license validation was failing on newer versions of EDD. Kernl has been updated to support the newer version API as well as the older API.
  • The Git integration has been refactored and had additional test coverage added.
  • Kernl was upgraded to Node 12.13.x. With this upgrade we get async stack traces and a non-trivial decrease in response time (~50ms).

Blog Posts

WordPress Performance on DigitalOcean Managed MySQL

Want to performance test your own WordPress installation? Try Kernl’s WordPress Load Testing service.

The database is the most important part of any WordPress site. If you aren’t adept at managing MySQL, it can be a serious risk for you and your clients. One way to de-risk this portion of your WordPress site is by going with a managed MySQL offering. There are ton of different options out there, but for this post we’re going to look at Digital Ocean’s Managed MySQL.

DigitalOcean DB SaaS background.

Why Use DigitalOcean Managed MySQL?

There are a lot of reasons to use Digital Ocean’s Managed MySQL (or some other offering) including:

  • Simple Setup – With a few simple clicks you can have a high quality MySQL cluster set up.
  • Horizontal Scalability – Site growing fast? You can spin up read-only nodes to help scale out read operations.
  • Automated Daily Backups – No need to set up your own backups. DigitalOcean takes care of it for you.
  • Automated Failover – If for some reason your primary database node fails, you will automatically fail over to your warm spare.
  • Security – MySQL best practices for security are automatically followed by DigitalOcean. In addition to that your database is isolated to your private network so that outside requests can’t access it by default.

Baseline Performance Test

To get things started let’s do a baseline performance test where the MySQL database is on the same box as the Nginx server. The server configuration is as follows:

  • 1GB RAM
  • 3 vCPU
  • Nginx
  • PHP-FPM 7.2
  • MariaDB 10.1
  • No caching

A test of 400 concurrent users was run using Kernl’s WordPress Load Testing service. Without caching the results are predictably bad, but that is to be expected.

Graph of requests per second for self-hosted MySQL with WordPress
Things fall apart at around ~70 req/sec

For completeness, let’s also take a look at the response time distribution.

Response time distribution for self hosted MySQL with WordPress
99% of requests finish in ~2 seconds

Given how many failing requests we had and zero caching, the ones that did manage to get through didn’t perform too poorly. 99% in 2 seconds or less.

DigitalOcean Managed MySQL

When setting up a managed MySQL database on DigitalOcean you get the option to select the underlying hardware that powers it. For this blog post we went with the minimum possible configuration (1GB RAM, 1vCPU).

Knowing that fewer resources were allocated, it was expected that performance would actually be a bit worse on the dedicated MySQL instance. These expectations were confirmed with the load test.

Graph of requests per second for DigitalOcean Managed MySQL with WordPress
Things go sideways at ~50 req/s.

All things considered, 50 req/s against a WordPress site with no caching isn’t all that bad. Once we hit that level, the database charts were showing 100% CPU saturation and thats when we started to see the error rate increase. Now let’s look at the response time distribution.

Response time distribution for managed MySQL with WordPress
99% of requests finish in less than 2.6 seconds

As expected performance is slightly worse here due to fewer resources on the dedicated MySQL machine, but not in a huge way.

Cost & Why Managed MySQL is Important

For our test we used the most basic configuration that DigitalOcean offers:

  • 1 GB RAM
  • 1 vCPU
  • 10 GB Hard Disk
  • No failover
  • Automatic backups

All of this and not having to manage MySQL for $15/month. Not too bad, but to really be getting your money’s worth (automatic failover) you need to spend more money.

  • 2 GB RAM
  • 1 vCPU
  • 25 GB Hard Disk
  • 1 Standby node
  • Automatic failover
  • Automatic backups

With automatic failover you start to reduce the risk to yourself and your customers a lot. However, this level of availability isn’t free 🙂 DigitalOcean doesn’t support failover on their cheapest node, so you have to upgrade to the next level ($30/month). After that, the failover node costs you an additional $20/month. Now we’re looking at $50/month for a managed MySQL cluster with automated failover.

If you run a WordPress based business on DigitalOcean, $50/month buys you a lot of peace of mind. It’s also easy to scale up as traffic increases and the cost is competitive in the landscape of managed MySQL. If you happen to be an excellent DBA though, you can definitely manage your own cluster at a reduced monetary cost, but increased time cost.

Conclusions

Managing your own database cluster is probably fine if cost is a problem and you are good at it. In general though, if you can afford it we recommend using a managed MySQL host so you can push the complexity of operating a highly available MySQL cluster on to someone else.

Want to performance test your own WordPress installation? Try Kernl’s WordPress Load Testing service.

What’s New With Kernl – September 2019

I hope that everyone has had a great September! There has been some interesting changes with Kernl this month, so let’s dive in and discuss them.

Features

  • Credit Card Add Changes – For the past 3 years Kernl has had a simple credit card form (card number, expiration, etc). In an effort to help reduce fraud we’ve migrated out card addition system to use Stripe’s Checkout.js. This gives us advanced fraud protection provided by Stripe and the added benefit that your card details never hit Kernl’s servers.
  • Continuous Deployment Setup – Kernl is in the middle of a big UI change for continuous deployment setup. The first step was making the ‘connect your Gitlab/GitHub/BitBucket account’ piece a lot prettier. We’ve now made it a 3 panel selection, with nice big logos and fewer ways to get confused. Check it out by going to the “Continuous Deployment” section in Kernl.
  • Repository Pruning – Kernl will now prune repositories from your repository list that you no longer have access to or no longer have connected with Kernl. The exception here is that we’ll keep a reference to repositories that you have connected to plugins or themes.
  • Invoice Payment Failure Notifications – We recently had an issue where a customer wasn’t notified that their payment failed (card had expired) 🙁 This should be resolved now as we’ve integrated with Stripe web hooks and immediately catch the event and send a message to the account owner. We’ve also added a notification preference so that you can silence these emails.

Other

  • All packages have been upgraded on all Kernl servers.
  • Marketing pages are now cached in Redis.
  • Fixed a customer reported bug in plugin_update_check.php that threw a warning in newer versions of PHP. Upgrade to version 1.2.2 to get the fix.
  • Feature flag setup wizard had weird behavior if the customer didn’t have any plugins or themes. You can now manually name your feature flag product in the wizard if you so choose.
  • Fixed a few bugs in the feature flag UI where adding/removing individual users from a flag would occasionally fail.
  • The buttons to manually manage deploy keys have been moved to the bottom of the continuous deployment page. They also come with a disclaimer now.

Blog Posts

Beta testing WordPress plugin features with Kernl WordPress Feature Flags

Beta testing WordPress plugin features with Kernl WordPress Feature Flags

Imagine that you are a WordPress plugin author. Maybe you work for an agency or maybe you work for yourself. The point is that you have clients or customers that have come to expect a high level of quality from your work. Great job!

Now imagine that you have been working on a complicated but much sought after feature for your plugin. Complicated means risk. Complicated means that your hard-earned reputation for high-quality could on the chopping-block if you aren’t careful.

Don’t get stressed out like this guy.

So what do you do? You need to release the feature but you also want to limit the risk you take by doing so. You can do few different things, but we’re here to talk about only one of them:

  • “Dog-food” the new feature for as long as you can.
  • Unit and integration tests can help test the validity of your code
  • Run a limited beta program with feature flags

Using Kernl Feature Flags to Manage a Beta Program

First of all, what is a feature flag anyway? A feature flag is a way of toggling sections of code on or off without doing a code deployment. At an extremely high level, it looks like this:

<?php
  if ($flagActive) {
     // enable feature
  }
?>

Thats it. In practice implementing feature flags is a bit more difficult, but it doesn’t have to be much more complicated. For instance, using Kernl’s feature flag library:

<?php
  $kff = new kernl\WPFeatureFlags($kernlFeatureFlagProductKey, $userIdentifier);
  if ($kff->active('MY_FLAG')) {
    // enable feature
  }
?>

The beauty here is that you can toggle this block of code on or off for individual users, all users, or a percentage of users without ever needing to deploy anymore code.

So. Easy.

And just like that ‘jack.slingerland@gmail.com’ gained access to the beta. No code deploys. No complicated anything. Just search, click, save. And what did this look like for the end user?

Before

No Feature Flag Footer Bar

After

Feature Flag Footer Bar Beta Program!

Now this is obviously a contrived example, but it has all the building blocks you need to do far more complicated integrations.

A Beta Program

With the building blocks above you can see it isn’t hard to manage a beta program. Simply ship an update to your plugin wrapped in a feature flag. After that, add specific people to it as you see fit. As you get more feedback, continue to ship updates behind the flag. Once you are confident that the code looks good, you can remove the flag completely!

Now let’s dive in to the actual plugin code.

Tutorial

Adding feature flags to your plugin is a 3 step process.

  1. Create the product & flag in Kernl.
  2. Install the feature flag library via Composer
  3. Add the code to your plugin.

Step 1 is accomplished by signing up for Kernl and adding the product and flag. The product is just a container for all of your flags. That way your plugin can have a bunch of different flags in it but still be logically grouped together. For this case, let’s create a product called ‘Kernl Footer Flag Blog Post’ and a flag named ‘New Footer Beta’.

Step 2 is as simple as installing the Kernl WordPress feature flag library via Composer:

composer require kernl/wp-feature-flags

For step 3, you add some code to your plugin. Let’s take a look at it, followed by discussion of what it’s all doing.

<?php

require __DIR__ . '/vendor/autoload.php';
add_action('init', 'kernl_footer_flags_init');
function kernl_footer_flags_init() {
    add_action('wp_footer', 'beta_program_footer');
}

function beta_program_footer() {
    if (is_user_logged_in()) {
        $currentUser = wp_get_current_user();
        $userIdentifier = $currentUser->user_email;
    } else {
        $userIdentifier = 'Unauthenticated Users';
    }
    $kernlFeatureFlagProductKey = '5d835a2830cbb568728b9bd4';
    $cacheTimeInMinutes = 1;
    $defaultToActive = false;
    $kff = new kernl\WPFeatureFlags(
        $kernlFeatureFlagProductKey,
        $userIdentifier,
        $defaultToActive,
        $cacheTimeInMinutes
    );
    if ($kff->active('NEW_FOOTER_BETA')):
    ?>
        <style>
            .kernl-footer-flag-bar {
                background-color: #5d5d5d;
                font-size: 12px;
                text-align: right;
                color: white;
            }
        </style>
        <div class="kernl-footer-flag-bar">
            You are in the Kernl Footer Flags beta program (<?= $userIdentifier ?>)
        </div>
    <?php endif;
}
?>

That’s the bulk of the plugin code (I excluded the Kernl updater for brevity). Let’s break it down.

Init Action and Composer Auto Load

require __DIR__ . '/vendor/autoload.php';
add_action('init', 'kernl_footer_flags_init');
function kernl_footer_flags_init() {
    add_action('wp_footer', 'beta_program_footer');
}

In this code we auto load the composer dependency that we have. You can see it here. After that we define an ‘init’ action to add our beta program footer. The reason we have this in the ‘init’ action is because if we do it too early we won’t be able to fetch the current user (which we use to create a unique identifier).

User Identifier Creation

function beta_program_footer() {
    if (is_user_logged_in()) {
        $currentUser = wp_get_current_user();
        $userIdentifier = $currentUser->user_email;
    } else {
        $userIdentifier = 'Unauthenticated Users';
    }

After we define our action we create the function that it calls. The first thing we want to do is create a user identifier. The user identifier is used by Kernl to determine what feature flags that the identified user should see. In our case, if the person isn’t logged in they get lumped into an ‘Unauthenticated Users’ bucket. In the person is logged in, we identify them by their email. It is by this identifier that you will enable/disable features when using the ‘individual’ type feature flag. If we were using the ‘enable for a percentage of users’ type feature flag, simply assigning each user a unique identifier (maybe a UUID) would suffice.

Instantiate the Kernl Feature Flag Library

$kernlFeatureFlagProductKey = '5d835a2830cbb568728b9bd4';
$cacheTimeInMinutes = 1;
$defaultToActive = false;
$kff = new kernl\WPFeatureFlags(
    $kernlFeatureFlagProductKey,
    $userIdentifier,
    $defaultToActive,
    $cacheTimeInMinutes
);

The $kernlFeatureFlagProductKey is generated by Kernl when you create your product. The $cacheTimeInMinutes variable is so you can configure how long the flags will be cached in WordPress. If this is set to ‘0’, then the library will call Kernl every time the page loads. In general you probably don’t want this. And last but not least, $defaultToActive is a boolean variable. If true, the ‘active()` function will return true if Kernl can’t find a flag.

Product Key

Active Check

if ($kff->active('NEW_FOOTER_BETA')):

The final piece is simply checking if this feature flag is active for this user.

Putting it all Together

If you want to see the source code for this plugin and/or install it yourself:

Conclusions

Kernl WordPress Feature flags are an incredibly powerful tool for safely releasing your code into the wild. In a situation like WordPress plugins where production is often not a machine that you are responsible for, being able to quickly toggle code on/off without a deploy is of paramount important.

Load Testing Vultr’s New High Frequency Servers with WordPress

Run your own WordPress Load Tests with Kernl!

Back in June Vultr announced the general availability of their new “High Frequency” servers. Reading through the announcement I was intrigued by their claim of using “3+GHZ processors and blazing fast NVMe storage!” and immediately wondered what WordPress performance would look like versus their regular Cloud Compute offering.

What was tested?

To get a better idea of the performance characteristics of the Vultr High Frequency Compute servers, I ran several types of load tests with all Vultr servers sitting in their Silicon Valley data center:

  • 200 concurrent users from New York City (Digital Ocean NYC3)
  • 2000 concurrent users from New York City (Digital Ocean NYC3) & London (Digital Ocean LON1)

And I tested the following scenarios:

All servers with the exception of the “performance tuned” ones were built using the pre-built WordPress image that Vultr offers with no caching or performance tuning done. The “performance tuned” servers were created using Nginx, PHP-FPM, MariaDB, Memcached, and W3 Total Cache.

Results

Starting with the Vultr High Frequency boxes, lets break down performance.

$6 32GB NVMe

The $6 Vultr High Frequency(HF) machine performed well against the roughly equivalent $5 Vultr Cloud Compute(CC) instance. If we take a look at the requests/failures per second charts you can see that the HF machine out-performed the CC instance by quite a lot.

$6 32GB NVMe Vultr High Frequency Requests/Failures
$5 25GB SSD Vultr Cloud Compute Requests/Failures

As you can see the HF server was able to handle roughly twice as many requests per second as the CC server without any errors. Let’s check out the average and median response times as well as the response time distribution.

$62 32GB NVMe Vultr High Frequency Response Time
$6 32GB NVMe Vultr High Frequency Response Time
$62 32GB NVMe Vultr High Frequency Response Time Distribution
$6 32GB NVMe Vultr High Frequency Response Time Distribution

You can see that as the test progressed response times got steadily worse until leveling out at around 2.5s. The response time distribution shows that 99% of requests finished in 3s or under, with the 100th percentile outlier coming in at just under 6s. This is honestly pretty good performance for no tuning at all.

$5 25GB SSD Vultr Cloud Compute Response Time
$5 25GB SSD Vultr Cloud Compute Response Time
$5 25GB SSD Vultr Cloud Compute Response Time Distribution
$5 25GB SSD Vultr Cloud Compute Response Time Distribution

The response times for the $5 cloud compute instance were about 2 seconds worse than the high frequency instance, with the average coming in at around 4.5. The response time distribution was worse from a performance perspective, but better from a consistency perspective with the spread between the 50th percentile and the 100th percentile only being ~1.3s.

At a glance, it looks like the Vultr High Frequency server out-performs the cloud compute in a significant way for only $1 extra. However, our next tests show that it might not be so simple.

$24 128GB NVMe (Performance Tuned)

The next set of tests that were performed were 2000 concurrent users on a performance tuned setup. The traffic originated from a cluster of servers in both New York and London, with the host server being in Vultr’s Silicon Valley data center.

First, lets take a look at the requests per second handled by the $24 High Frequency server and the $20 Cloud Compute server.

$24 128GB NVMe Vultr High Frequency Requests

Now, 2000 concurrent users is a lot. I think that the HF machine did quite well overall, but I confess that I did expect a bit more out of it. It topped out at around 1250 requests per second, with errors hovering in the ~175 requests per second range. If left to run for another 30 minutes I think it would have reached closer to 1350 requests per second.

$20 80GB SSD Vultr Cloud Compute Requests

This is where things start to get interesting. The Vultr marketing team bills the high frequency machines as heads and shoulders above the regular cloud compute machines. Maybe it is for some workloads. But for this test it doesn’t seem much better at all. Looking at both graphs you can see that the HF machine topped out a bit higher, and had fewer errors, but not enough to warrant the extra cash (in my opinion). Lets see what sort of story the response times tell.

$24 128GB NVMe Vultr High Frequency Response Time
$24 128GB NVMe Vultr High Frequency Response Time Distribution

On the performance tuned box you can see that the requests that did complete successfully were all quite fast. The average response time was between 100ms and 200ms throughout the entire test. The response time distribution was solid as well, with 99% of requests finishing in under 500ms. Not bad for 2000 concurrent users.

Now let’s take a look at how the comparable cloud compute server did.

$20 80GB SSD Vultr Cloud Compute Response Time
$20 80GB SSD Vultr Cloud Compute Response Time Distribution

For the Vultr Cloud Compute instance response times also hovered between 100ms-200ms, although just a hair higher than the high frequency servers. The response time distribution was excellent as well, with 99% of requests coming in under 500ms. The main difference here is that there was a 100th percentile outlier here at the high frequency server didn’t have.

Thoughts & Conclusions

From what I can tell, the Vultr High Frequency servers are better than the Cloud Compute servers, but maybe only for certain types of workloads. You can see in the $5/$6 test that they easily out-performed the cloud compute instances, but in the more expensive $20/$24 test the results weren’t so cut and dry.

If we look at it from a requests per dollar standpoint, the cloud compute instance is actually a better value for this type of workload.

Requests per Dollar

I suggest running extensive load tests against the Vultr High Frequency machines before making any effort to switch over. They might be better for you. They might perform the same for more money.

In conclusion: ¯\_(ツ)_/¯

Run your own WordPress Load Tests with Kernl!