What’s New With Kernl – August 2020

Hello everyone! It’s been awhile since the last update message, but now that summer is nearly over in the northern hemisphere Kernl will again be receiving regular updates and enhancements.

This doesn’t mean that we weren’t busy though! Let’s dig in.

New Features

  • License Management Activated Domains – When viewing your license list, you can now see the domains that have activated the license. If you go into the license detail view, you can make changes to that list. This data will give you granular insight into who is using a specific license.
  • Kernl Update Checker Automatic Update Support – With the release of WordPress 5.5 we finally upgraded our update checker library. There are a few differences like it being multiple files now, and the instantiation code changing a bit, but the upgrade path is simple for those who want automatic updates.

Bug Fixes & Miscellaneous Changes

  • Package Upgrades (Analytics) – All supporting packages have been upgraded to their latest version for better performance and security.
  • Date Retention Bug (Analytics) – We weren’t cleaning up some tables we were supposed to be. This lead to holding onto some data for much longer than 365 days.
  • UX Improvements (Analytics) – The UX around selecting and comparing dates for Kernl Analytics was a little bit confusing. We made some changes that make it a lot easier to understand for a first-time customer.
  • Data collection issues (Load Testing) – There was an issue where load testing wasn’t collecting data from the master node after an upgrade to the underlying infrastructure. This was resolved.
  • UX Improvements (Load Testing) – Some load test templates are large and take awhile to return from the server. An indeterminate spinner was added here to let customers know that things are actually happening. The same situation was happening when long running, high volume tests load initially. There is a lot of data that takes some time to return from the server so an indeterminate spinner was added.
  • Infrastructure (Load Testing) – The Kernl WordPress Load Testing box was upgraded from 1vCPU+1GB RAM to 2vCPU+2GB RAM.
  • Meta Tag Parsing (Load Testing) – When you verify a site with Kernl, we look for a meta tag in your HTML. Initially we were attempting to find, capture, and validate this using regular expressions. As is tradition, we found this was a bad idea and switched to using an open-source library (Cheerio.js) instead.
  • All servers have had their packages upgraded to the latest available versions.

Blog Posts

That’s it for this month!

Hummingbird Cache for WordPress Performance Review

The Hummingbird Cache plugin is one of many different caching plugins available in the WordPress ecosystem. Enabling it will increase your site’s performance significantly, but by how much? In this review we’re going to use Kernl’s WordPress Load Testing tool to push our Hummingbird Cache WordPress installation to it’s limits.

HummingBird cache pushed to limits

Test System Setup

As with most of our cache reviews, we used a pretty standard PHP-FPM + Nginx setup.

  • DigitalOcean $5/month 1vCPU 1GB RAM machine
  • Ubuntu 20.04
  • PHP (FPM) 7.4
  • Nginx 1.18
  • MariaDB 10.3
  • Content – For this test I imported the contents of my personal blog and used it for testing.

The test system was located in San Francisco, CA, USA. Load test virtual users were located in New York, NY, USA along with some of the high volume tests spreading virtual users around Europe.

How did we test Hummingbird Cache?

To test the Hummingbird WordPress caching plugin ran 3 different load tests with Kernl WordPress Load Testing.

  1. Baseline – This is a 200 concurrent user test for 60 minutes with no caching enabled.
  2. Cache Enabled – The same test as the baseline run, but with caching enabled. This is the “apples to apples” comparison.
  3. Cache++ – After the “apples to apples” comparison, we pumped up the concurrent users to 400 to see how well the plugin would respond.

Baseline Load Test

The baseline load test is just the bare WordPress setup with no plugins enabled and the base TwentyTwenty theme. As expected performance isn’t great but it isn’t terrible either.

Hummingbird Cache - Baseline Request Throughput
Hummingbird Cache – Baseline Request Throughput

You can see from the throughput chart that the base WordPress installation with no caching enabled settled in at around 34 requests/s. Not too shabby, but what was the quality of those requests?

Hummingbird Cache - Baseline Response Times
Hummingbird Cache – Baseline Response Times

The average and median response times tell a story steady degradation of the user experience before finally settling at just shy of 5 seconds. If I were a reader of that blog, I would be extremely turned off by waiting for 5 seconds just to have the page load start.

Hummingbird Cache - Baseline Response Time Distribution
Hummingbird Cache – Baseline Response Time Distribution

The response time distribution is pretty awful here. 50% of requests finished in under 5s, and 99% of requests finished in under 5.5s. In most load tests we like to see the P50 number be a lot lower than the P99 number. In a perfect world they’re both really low, but that doesn’t happen in most cases.

Cache Enabled Load Test

Our next test was the same as the baseline test, but with HummingBird cache enabled. We went with all the default options making no changes to the settings.

Hummingbird Cache – Cache Enabled Request Throughput
Hummingbird Cache – Cache Enabled Request Throughput

As expected of a caching plugin, throughput goes up a lot and settles in at around 175 requests/second with zero errors. This is a nearly 6x improvement in throughput. But what about the response times? How did this look to the end user?

Hummingbird Cache – Cache Enabled Response Times
Hummingbird Cache – Cache Enabled Response Times

The response time results are extremely promising. The average response time was around 95ms and the median was around 75ms. Most performance best-practices hope for your site to respond within 100ms, which this plugin easily accomplishes even under incredibly heavy load. Let’s break the response time numbers down further.

Hummingbird Cache – Cache Enabled Response Time Distribution
Hummingbird Cache – Cache Enabled Response Time Distribution

For 50% of our users, the response time was 75ms or less. For 99% of our users, response time was less than 160ms. These are great numbers and just what I would expect from a WordPress caching plugin.

Cache++ Load Test

Now that we’ve established that Hummingbird Cache does a great job under (somewhat) normal circumstances, lets see what happens if we double the traffic (400 concurrent users -vs- 200 concurrent users).

Hummingbird Cache++ – Cache Enabled Request Throughput
Hummingbird Cache++ – Cache Enabled Request Throughput

Event at 2X the number of users, we don’t see any errors and we see the throughput settling at about 325 requests per second. If you do the math, this is about 28 million requests a day. On a $5 box. Obviously this test is fairly naive, but it does show that the plugin can handle some serious traffic when needed.

Hummingbird Cache++ – Cache Enabled Response Times
Hummingbird Cache++ – Cache Enabled Response Times

The best part about this test is that even with incredible load the response time average and median are still below 180ms. Most users visiting a site would be extremely happy with response times in that range.

Hummingbird Cache++ – Cache Enabled Response Time Distribution
Hummingbird Cache++ – Cache Enabled Response Time Distribution

The response time distribution still tells a reasonable story. 50% of users see responses in 150ms or less and 99% see responses in 375ms or less. Solid performance from the Hummingbird Cache team.

Hummingbird Cache Conclusions

If you need a caching plugin for your site, Hummingbird Cache is a solid choice. It performs well, was easy to install, and was generally low friction. I found the user interface to be a little immature, but that doesn’t change the excellent performance we saw during our tests.

Want to run your own load tests? Sign up for Kernl!

UpCloud WordPress Hosting Review

If you spend time in the WordPress hosting space and few names constantly come up, and one of them is UpCloud. UpCloud is a European cloud provider with data centers all over the world that claims to have the world’s “fastest cloud servers”. In this review we’re going to take a look at how WordPress performs while hosted on UpCloud and also see how the reliability of the service is.

Want to run your own WordPress load tests? Sign up for Kernl.

Hardware Tested

For our test of WordPress running on UpCloud, we tested 3 different hardware configurations.

  • 1vCPU + 1GB RAM – This is the cheapest possible option for hosting WordPress on UpCloud at $5 / month.
  • 2vCPU + 4GB RAM – At $20 / month, this option is a nice balance of processing power and memory. If I were to host a few sites on UpCloud with moderate amounts of traffic I would probably start here.
  • 4vCPU + 8GB RAM – This machine was as expensive as I was willing to go for this review, coming in at $40 / month.

Software Used

The software used in this test was the latest available from the included repositories with Ubuntu 20.04.

  • Ubuntu 20.04 LTS
  • MariaDB
  • Nginx
  • PHP-FPM 7.4
  • Memcached

What was tested?

For each test we ran on UpCloud we used Kernl’s WordPress Load Testing service to generate load against each virtual machine. The content for the load tests was a copy of this blog. The load generators for each test lived in DigitalOcean’s SFO2 data center and ran against UpCloud’s Chicago data center.

UpCloud WordPress Load Tests

We ran a total of 6 WordPress load tests on UpCloud’s servers. For each machine we tested we ran a test where nothing was cached and a test where everything was cached (using W3 Total Cache backed by Memcached). Each test was for 1000 concurrent users for 45 minutes.

1vCPU + 1GB RAM (No Cache)

For this configuration the requests per second peaked at around 41/s before failure rates started to increase. Once failures started in earnest, successful requests leveled off at around 11/s.

UpCloud 1vCPU 1GB RAM - Requests/Failures
Not bad for no caching!

On the response time front, things weren’t great. As the request (and error) rate increased, the response times started to get pretty unwieldy.

UpCloud 1vCPU 1GB RAM - Response Time
Less than a 1000ms, then 2500-3000

A graph of median and average response times doesn’t always tell the whole story, so lets take a look at the response time distribution.

UpCloud 1vCPU 1GB RAM - Response Time Distribution
No great.

The response time distribution tells us that 50% of requests finished in under 5000ms, but that 99% finished in under 6000ms. Usually you want to see a large difference between 50th and 99th percentile. But here they’re awfully close, meaning that most request response times were pretty terrible.

1vCPU + 1GB RAM (Cached)

Now that we’ve looked at uncached performance, lets take a look at how this UpCloud server handled WordPress with caching enabled.

UpCloud 1vCPU 1GB RAM Cached – Requests/Failures Per Second

As you can see, performance is much better. We end up leveling off at around 436 request per second. The only issue here is that we still have quite a few failures. Certainly more failures than would be acceptable in a production situation.

UpCloud 1vCPU 1GB RAM Cached – Average / Median Response Time

Looking at response times you can see performance is a lot better, and much more in line with what you’d expect from a WordPress site that’s under heavy load. Average response time was a hair less than 1000ms, with the median response time (doesn’t include outliers) closer to 475ms. This is a bit less than an order of magnitude improve in performance just be turning on caching with W3 Total Cache.

UpCloud 1vCPU 1GB RAM Cached – Response Time Distribution

So far everything looked great (with the exception of the request failures) for this UpCloud configuration with caching enabled. However if we take a look at the response time distribution you can see that all is not what it seems. 80% of requests returned in less than 600ms, but the upper 20% too anywhere between 3000ms and 12000ms. 20% of your customers waiting more than 3 seconds for the page to load isn’t awesome.

2vCPU + 4GB RAM (No Cache)

The next machine we tested was far better provisioned than the first machine with twice the CPUs and 4x the RAM.

UpCloud 2vCPU 4GB RAM - Request / Failures per Second

As you can see request throughput peaked at around 50 per second, and then leveled off at 22 per second when the error rate elevated. I honestly expected the throughput to be better for this scenario given how much more hardware there was to work with.

UpCloud 2vCPU 4GB RAM - Average / Median Response Times

Response times slowly increased as load increased on the server, with average response times ending up near 5000ms and the median response times closer to 5500ms. At that point, 500ms probably doesn’t matter that much.

UpCloud 2vCPU 4GB RAM - Response Time Distribution

The response time distribution for this test was predictably bad. The difference between the lower 50% and upper 50% is only 1000ms, which doesn’t give me warm fuzzy feelings. If anything, this is a great example of how much you need caching in a WordPress installation. Throwing twice as much hardware at it barely even makes a difference uncached.

2vCPU + 4GB RAM (Cached)

Now lets take a look at the same machine, but this time with caching enabled via W3 Total Cache and Memcached.

UpCloud 2vCPU 4GB RAM Cached – Requests / Failures per Second

No we’re getting somewhere! Requests per second leveled out at around 600, which is actually quite high. We also didn’t start to see failures increase until around 300 per second, which is pretty amazing for a $20/month machine.

UpCloud 2vCPU 4GB RAM Cached – Average / Median Response Time

The response times for the cached version were pretty great until the failure started accumulating. Up until the failures started we were seeing response times right around the 60ms mark, which is excellent under heavy load. After that they leveled out at around 450ms, which still isn’t bad when you are serving ~600 requests/s.

UpCloud 2vCPU 4GB RAM Cached – Response Time Distribution

The response time distribution wasn’t terrible here either considering all the failures we were seeing. 95% of requests finished in under 1000ms and 50% finished in under 500ms.

4vCPU + 8GB RAM (No Cache)

Our final test was with a fairly robust machine (by my standards anyway).

4vCPU 8GB RAM – Requests / Failures Per Second

As you can see the results in an uncached situation were pretty solid until failures started in a major way. This machine was able to handle about 175 req/s uncached, and then leveled out at 215 req/s once the errors started. Obviously this number of errors isn’t great, but at the 175 req/s mark thats enough to serve 15 million requests a day.

UpCloud 4vCPU 8GB RAM – Average / Median Response Times

As expected in an uncached situation the response times weren’t great, but they weren’t terrible either given the amount of load the system was under. Prior to elevated error rates we were seeing response times around 100ms. Once errors picked up they leveled off at 2500ms.

4vCPU 8GB RAM – Response Time Distribution

The response time distribution is what we would expect from an uncached WordPress installation under heavy load. 50% of requests finished in 2500ms and 99% of requests finished in 3250ms. That’s not all bad considering there wasn’t any caching.

4vCPU + 8GB RAM (Cached)

Our final load test was a cached version of the previous test. Given drastic improvements we saw in the other tests, the same sort of results were expected here.

4vCPU 8GB RAM Cached – Requests / Failures Per Second

Now that we have a more robust machine + caching enabled, you can see that the server was able to process quite a few requests concurrently before errors started. Even once errors started they stayed relatively low. Initial errors didn’t show up until around 500 req/s and eventually we leveled off at 850 req/s. Not bad for $40/month.

4vCPU 8GB RAM Cached – Average / Median Response Times

The response times were also excellent in the cached scenario. Before errors and failures we were seeing 65ms and after we were stilling coming in at around 100ms. Not too shabby for the intense load the server was under.

UpCloud 4vCPU 8GB RAM Cached – Response Time Distribution

My favorite chart this time around is the response time distribution. 99% of all requests finished in under 300ms. With 1000 concurrent users.

UpCloud WordPress Reliability

An often overlooked metric when comparing VPS providers like UpCloud is reliability. To get some objective numbers on that, I ran a reliability test with Kernl for 30 consecutive hours. The test is low volume (25 concurrent users), but enough to make sure the server stays active and that we’ll notice is sometimes goes awry.

Reliability test

Over the course of 30 hours we saw one error spike that was quickly resolved. After that, no errors at all. Over the entire test we saw 400 requests fail, which is roughly 16 seconds of failures. I’ve done quite a few of these reliability tests with different hosts and this is pretty average.

Conclusions

UpCloud is solid choice for VPS hosting though their claim to be the “fastest” may not be completely substantiated (with WordPress at least). If I wanted to use a VPS host based in Europe they would definitely be near the top of my list. The only odd thing about them is their billing model where you add “credits” to your account versus just having a credit card of file. This is likely to prevent fraud, but doesn’t really make for a great user experience.

Vultr One-Click WordPress Review

If you want to host your own copy of WordPress but don’t want to go through all the trouble of setting it up, it might be worth your time to explore using “one-click” installs from some of the major cloud providers. In this article, we’re going to explore the Vultr One-Click WordPress install and see what it’s all about.

Ease of Installation

Before anything else you need to get a WordPress box spun up on Vultr. Overall, the experience was pretty easy:

  1. Log in / sign up for Vultr
  2. Click “Deploy new server”
  3. Select your server type (we used “Cloud Compute”) and server location (Atlanta for us).
  4. Click the “Application” tab and then select WordPress.
  5. Select your server size (25GB SSD $5/month for this test)
  6. Add your SSH key
  7. Fill out your host name
  8. Click “Deploy Now”

So obviously this isn’t a single click to install, but it is a lot easier than setting up and configuring a WordPress installation from scratch.

Whats included?

Aside from the obvious stuff (WordPress, MySQL, etc) which we’ll cover in the next section, what else is included with your One-Click Vultr WordPress installation?

  • XHProf – XHProf is a tool for profiling PHP applications. This could be very helpful if you find your site getting real slow after installing a plugin or theme. It could help you determine the source of the slowness.
  • PHPMyAdmin – The classic MySQL database administration tool for the web. If you haven’t used it before, its a great way to browse your database and make changes.
  • Maldet – Maldet is a Linux malware detection tool that runs periodically on your system to see if it has been infected. This is disabled by default, but can easily be enabled with the directions included with your new server.
  • Cockpit – Cockpit is a web based interface for your server. You can easily see how your server is configured, what resource usage looks like, and update packages all without even SSHing into your server.

What’s WordPress Running On?

The Vultr One-Click WordPress system that we decided to use had the following hardware and software:

  • Hardware
    • 1 vCPU
    • 1GB RAM
    • 25GB SSD
    • 1TB Bandwidth
  • Software
    • Ubuntu 18.04 LTS
    • Nginx
    • PHP 7.2
    • MySQL 5.7

Overall the software is a pretty standard LEMP setup and the hardware is as performant as you would expect for $5 per month. It’s also worth noting that the installation script for this one click install might need some updating because it installs the previous version of WordPress.

Performance Out of the Box

So you’ve spun up your new Vultr One-Click WordPress server, but what sort of performance can you get out of it? Using Kernl’s WordPress Load Testing we can easily see the performance that this $5 per month server gives you.

For both this test and the cached test (next section) we attempted to have 200 concurrent users browse the WordPress site for 30 minutes. The traffic was generated from Digital Ocean’s NYC3 datacenter.

Vultr One-Click WordPress Requests / Failures per second
Requests / Failures per second

As you can see we get up to about 20 requests per second before the server starts returning failures. For a $5 machine this isn’t bad at all. 20 req/s over the course of 24 hours is ~1.7 million requests.

Vultr One-Click WordPress Response Times
Response Times

Initially response times are pretty good, but as resource contention on the server increases the average and median response times start to go up. Even when under full load, the pages returned in under 5 seconds. Not terrible, but you definitely aren’t winning friends with that sort of performance.

Vultr One-Click WordPress Response Time Distribution
Response Time Distribution

The response time distribution is mostly what you would expect after seeing the error rate and the average response time. 99% of requests return in under 5.5 seconds. In general you want to see your 99th and 100th percentiles be low, but given the duress the server was under this isn’t surprising at all.

Overall the system performed well and was configured well enough to handle a small-ish load test.

Performance with Caching

The test above is almost silly in that most people will definitely be using some form of caching on their WordPress site. For a more realistic test, I enabled W3 Total Cache on the WordPress installation. There wasn’t any Memcached or Redis installation available, so all cache settings were tuned to “Disk (enhanced)”.

Requests / Failures per second

As you can see the difference in performance with caching enabled is quite good. We didn’t see any errors at all for the duration of the test and the maximum concurrent requests per second peaked at 160. Over a 24 hour period, that would be 13.8 million requests.

Average / Median Response Time

The response time over the course of the test was also quite good. Initially it hovered around 35ms on average, but as resource contention on the server increased it leveled out at around 140ms. Not too bad for $5 and nearly zero configuration.

Response Time Distribution

Finally we come to the response time distribution. You can see here that 50% of requests finished in under 100ms, 99% finished in under 700ms, and all requests finished by 1800ms. These are good numbers.

Final Thoughts

If you need to host a WordPress site and don’t want to much around with configuration, the Vultr One-Click WordPress service is pretty good. In the long-term you’ll still need to know a little about system administration to apply updates, but it is definitely a great place to start and gets you a lot of the way to a fully functioning WordPress site.

Should I use Memcached or Redis for WordPress caching?

Choosing between Memcached or Redis for your WordPress cache is a tough decision. Not because they have vastly different performance profiles (they don’t), but because either choice is a good one depending on your needs. In this post we’re going to explore the differences between Redis and Memcached, how they perform for WordPress, and a lot of different non-performance things you should consider when making your choice.

What is Memcached?

memcached logo

Memcached is an open-source, high performance, distributed memory object caching system. What does that mean? It means you can store a bunch of strings in memory and access them really fast. From a WordPress perspective, it means that using a caching plugin like W3 Total Cache we can store the results of the complicated SQL queries that WordPress does in memory and have them available instantly.

What is Redis?

Redis Logo

Redis is an open source in-memory store that can be used as a cache or a message broker. It’s a bit different then Memcached because you get a lot more out of the box with it. For example, Redis has built in replication, transactions, disk persistence, and provides high availability and partitioning. All those features means that managing WordPress can be a little harder to do, but not much harder. Especially if you just need to use it as a cache.

Performance

Both Redis and Memcached have excellent performance. They’re both used by some of the largest websites in the world and are fully ingrained in the Fortune 500. Given that all things are not created equal, let’s see how they perform with a read-heavy WordPress site (this blog).

The Setup

The load tests are performed against the DigitalOcean WordPress Marketplace image with either Redis or Memcached installed alongside of it. The machines have 2 vCPUs, 2 GB RAM and live in DigitalOcean’s SFO2 (San Francisco) data center.

The load test configuration:

  • 500 concurrent users
  • 2 users / second ramp up
  • 45 minute test ( ran twice )
  • Traffic comes from Digital Ocean’s NYC3 data center.

The content of the load test is a copy of this blog.

Baseline Performance (No Cache)

The baseline performance for WordPress with no cache isn’t great.

baseline requests/failures
50 requests / second with LOTS of failures

The response time also isn’t great. A little over 2 seconds on average.

baseline response times
~2 seconds response time on average.

Redis Performance

Once we install Redis and configure W3 Total Cache to use it, the number of requests that we can handle increases substantially.

redis requests/failures
300 requests per second

The requests remain steady at around 300 per second and no failures are recorded. The response time also improves quite a bit.

redis response times
Average ~475ms response time

475ms isn’t bad at all. That’s 4 times faster response times then without any caching at all.

Memcached Performance

With Memcached installed and W3 Total Cache configured to use it, we see some excellent performance.

memcached requests/failures
425 requests per second

In this situation, Memcached performs even better then Redis with 425 req/s versus Redis’ 300 req/s. Response time improvements are similar.

memcached response times
115ms response time

The Memcached response time is almost 3 times faster than the Redis response time. In general, the results where Memcached is faster than Redis are surprising. In most benchmarks Redis is equal or faster than Memcached, so it’s likely a configuration problem.

Other Considerations

When deciding what cache to use with your WordPress setup, there are a few other considerations your should be looking at:

  • Ease of setup – As you can see from the performance results above, Memcached has better performance out of the box. Knowing what I know about Redis this is likely a configuration issue, but the fact that I could get that level of performance with no configuration from Memcached is a good data point.
  • 3rd Party Hosting – Do you really want to manage your own Redis or Memcached server? If you don’t, you’ll want to look at the landscape of 3rd party providers. Redis has a robust provider ecosystem. Memcached’s is a little less robust.
  • Persistence – Do you need your cache to survive a reboot? This is important if the cost of re-populating your cache is too high for your system. If you do need persistence, Redis is your best option.
  • High Availability If you need high availability of your caching cluster, Redis is the clear winner here. Memcached can be made to operate this way, but Redis has it baked in to the core of the application.

If you’d like to see the full results of the load testing runs on Kernl, see the links below.

The Crucible – Extreme WordPress Performance Challenge

Load testing is fun. Breaking things is fun. Breaking WordPress with load testing is even more fun. But in the era of highly scaleable WordPress hosting solutions, can we even break WordPress anymore? Oh yes, yes we can. The Crucible Challenge can.

Crucible Challenge

The Crucible WordPress Performance challenge is a deceptively simple test inspired by the poor ops teams that have to handle traffic from Super Bowl advertisements. Given a WordPress site with consistent content and URL mappings:

  • Handle 50,000 (@ 500 per second ramp up) concurrent authenticated users for 2 hours with load test generators in New York, London, Amsterdam, Singapore, Bangalore, San Francisco, Toronto and Frankfurt.
  • Have an error rate below 0.1%.
  • Average response time should be below 800ms.
  • Median response time should be below 700ms.
  • 99th percentile response time should be below 800ms.
  • Half way through the test, you must flush your cache.

Simple, yes. Easy, no. Why is this hard?

  • 50,000 is a LOT of people.
  • 500 per second ramp up does not give any time for warming your cache. It’s like your site getting hit in the face with a sledgehammer.
  • Low error rate doesn’t give you a lot of room for problems.
  • Keeping your response times below 1000ms with traffic coming from all over the world presents interesting problems.
  • Flushing your cache after the load test is already in progress shows us you have a good cache invalidation strategy and can handle dog-piling.

Example Results

To give everyone an idea of what results might look like, I took at $160 / month CPU Optimized Digital Ocean droplet, put Open LiteSpeed + WordPress on it and ran The Crucible against it.

Shows the crucible request per second graph at 18,000 requests per second.
18,000 requests per second

I only ran the test for 2 minutes, but as you can see it started to max out near the 18,000 req/s mark, with failures in 1,800 failures/s area.

Can your service beat this? Want to find out? Drop an email to jack@kernl.us and we can get a test run scheduled.

What’s New With Kernl – January 2020

January was a pretty great month for Kernl. We got a lot of bug fixes, some performance improvements, and even a new beta feature out the door. Let’s dive in!

WordPress Site Health Beta

This month we released a beta of our WordPress Site Health service. The goal of this service is to help you determine where performance problems are and why they are happening. The dashboard below gives you a high level view of how performance looks on each of your sites.

Kernl WordPress Site Health

Once you click in to any site you see data for the last 7 days.

Kernl WordPress Site Health Detail Page

The data here helps you diagnose performance issues. Using the plugin changes panel you can tie performance issues back to adding/removing/updating of plugins. Google Lighthouse scores show trends on you site’s performance, usability, SEO and more over time.

If you’d like the join the beta, send an email to jack@kernl.us.

Bugs & Performance

  • Improved performance of plugin/theme list pages – The API calls to /api/v1/plugins and /api/v1/themes were incredibly inefficient. They hadn’t really been touched since Kernl’s inception and needed some love. After doing some query optimization and stripping out unnecessary data, the payload and response time were reduced by a factor of 10. The worst case example was response size going from ~750KB to ~75KB and response time from ~4000ms to ~250ms.
  • Server resource increases – Kernl’s main Node.js application servers have been running with 1 vCPU and 1GB of RAM for about the last 2 years. Lately we’ve seen some resource exhaustion and decided it was time to upgrade. The new app servers now run with 2 vCPUs and 2GB of RAM.
  • Server disk space / inode exhaustion – The process of building plugins and themes can use up a lot of space and file system resources. We weren’t doing a great job of cleaning those resources up periodically which could cause some performance issue. We now clean up all temporary files once a day which should prevent this from happening anymore.
  • Easy Digital Download License Validation Error – There was a bug in EDD license validation where the source system wouldn’t send back valid JSON. This would break license validation instead of handling the error gracefully.
  • Profile page autocomplete – If you had form auto-completion on it would sometimes cause the profile page to reset your password. We’ve disabled autocomplete on this form to resolve the issue.
  • Theme tiles not showing correct build status – During the course of our performance improvement work we noticed that theme tiles were not showing the correct build status. This has now been resolved.

What’s New With Kernl – December 2019

Welcome to the end of 2019! I hope that everyone has had as good a year as Kernl. Let’s dive in to the final update of 2019 to see what’s new.

Features, Bug Fixes, & Misc.

  • Improved License Management Search – License management now includes improved search functionality. The previous search functionality was flaky (at best) and not very discoverable. Search is now a first-class citizen, includes free-text search, and should greatly improve the overall usability of Kernl’s WordPress license management.
  • Load Testing Unit & Integration Tests – When we created the WordPress load testing service it was an experiment. Now that we have proved the viability of the service it’s time to work on stability and overall platform longevity. This month’s focus was on the authorization framework that our load testing service uses.
  • JS Bundle Size Reduction – Over the past year Kernl’s JS bundle size for our web app grew to over 2MB. We spent some time this month figuring out why and making changes to reduce it. In the end we were able to reduce the bundle size by over 50% down to 1.1MB.
  • Bug: Inconsistent Webhook & Deploy Key Behavior – After a few customer reported incidents with the automatic webhook and deploy key behavior, we discovered that Kernl wasn’t deleting local references to remote keys and hooks. If you have some issues with deploy keys or webhooks please contact us and we can help resolve the data inconsistency issues that this caused.
  • Node.js Upgrade to 12.14.0 – This month we upgrade all of our servers to use the latest LTS version of Node. This is includes stability and performance improvements.

That’s it for this year. See everyone in 2020!

Yoast SEO: WordPress Plugin Performance Implications

Yoast SEO is a popular SEO enablement plugin for WordPress. It helps you avoid common mistakes when it comes to SEO on your blog and also handles things like social media “og:meta” tags. If your blog has a public audience, then it stands a good chance that you are using Yoast.

As with the previous article in this series, we’ll focus on what performance implications are of using Yoast and how you can make it faster.

What was tested?

For this test we used the lowest-tier (1vCPU, 1GB RAM, $5) DigitalOcean droplet out of their SFO2 datacenter. Our server setup was as follows:

  • Ubuntu 19.04 with all updates installed.
  • Nginx 1.16.1
  • PHP-FPM 7.3
  • MariaDB 10.3

The theme that was used was TwentyTwenty with no modifications and the content tested was 3 “What’s new with Kernl?” posts from earlier this year. We selected a small number due to the effort of filling out all of the SEO data in Yoast.

The WordPress setup was bare-bones. There were no plugins installed except for when we were running the Yoast test.

As with our previous post in this series, the load for this test was generated out of DigitalOcean’s NYC3 datacenter.

Test Methodology

For testing Yoast we only ran two tests:

The tests were with 200 concurrent users over the course of 1 hour.

Max Requests per Second

One method for determining website performance is what is the maximum number of requests that in can field in a given second. For our purposes this is a pretty good indicator of the performance hit you take for installing plugins.

Yoast SEO - Max Requests / Second
Higher is better.

As you can see from the image above you lose about 25% of your maximum capacity from installing Yoast SEO. With no plugins installed we were able to hit 43 req/s, while with Yoast installed that number went down to 30 req/s.

It’s worth noting here that 30 req/s is 2.5 million requests a day.

First Error Occurrence

The next chart shows when we first started to see errors in our two tests.

Yoast SEO - First Error Occurrence
Higher is better.

Without the plugin installed WordPress was able to hit 38 req/s before seeing errors. Once we enabled Yoast that number went down to 28 req/s. Once again, this is consistent with the performance penalty we saw with the maximum requests per second of about 25%.

Average Response Time

The average response time with and without Yoast SEO tells a similar story to the requests per second measures we have done.

Yoast SEO - Average Response Time
Less is better.

The chart above shows us that without any plugins installed, the average response time under load is around 3000ms. With the Yoast plugin installed the response time goes up to about 4300ms. We’re looking at a roughly 25% change in response time.

99th Percentile Response Time

The 99th percentile chart can be read as “99% of all requests finished in under this time”.

Yoast SEO - 99th Percentile Response Time
Smaller is better.

The chart above tells us that without any plugins installed 99% of our requests finished in under ~3600ms. With Yoast SEO installed 99% of our requests finished in ~4900ms. Once again, a roughly 25% penalty for having Yoast installed.

Yoast SEO Performance Conclusions

Yoast is a really good SEO plugin. If SEO matters to you it’s definitely a plugin you should have installed. However you should use caching if you do use it. This goes for WordPress in general, but every plugin you add to WordPress has a performance cost associated with it. If you run a site where caching is difficult you’ll have to carefully weigh the performance cost versus benefit of installing Yoast SEO.

What’s New With Kernl – November 2019

November was a great month for Kernl! After several years of trying, we finally launched team management and also made our pricing structure easier to understand. Let’s dive in!

Features

  • Team Management – With our new Agency and Unlimited plans you can now grant users access to your account! The Agency plan allows for 3 team members and unlimited allows for unlimited team members.
  • Agency & Unlimited Plans – We introduced an unlimited plan for Kernl which has no usage limits (with the exception of load testing, but those limits have been increased substantially) and includes team management and Kernl Analytics. We also updated our agency plan to more closely match the old enterprise plan. The new agency plan has much higher limits than the previous agency plan as well as access to team management.

Bug Fixes & Other

  • Increased Load Test Generator VM Size – We’ve increased the load test generator machines from 1vCPU to 2vCPUs to allow us to scale our load testing up to 50,000 concurrent users.
  • Repository Sync – When we changed how you connected to your Git repositories we overlooked the ability to sync them from that same location. We’ve added that ability back in.
  • Load Test List Page Performance – With some clever SQL querying and awesome Postgres built-in functions, we’ve decreased the average load time on this page by 32%.
  • Bug: Invalid license if using licenses but no versions present – An odd edge case was found where Kernl would say your license was invalid if you didn’t have any plugin/theme versions uploaded. This has been resolved.
  • Bug: IPv6 Issue on Load Generators – The virtual machines that we spin up the Digital Ocean Singapore data center were having issues communicating over IPv6. We disabled IPv6 for all load generators for the time being.
  • Load Testing Service Node.js Upgrade – The WordPress load testing service backend and workers have been upgraded from Node.js 10.x to Node.js 12.x. All packages were upgraded to their latest with this change.
  • Analytics Service Node.js Upgrade – The WordPress analytics service backend was upgraded from Node.js 10.x to Node.js 12.x. All packages were upgraded to their latest with this change.

Blog Posts

Popular WordPress Plugin Performance Implications: Wordfence

That’s it for this month! Have a great December!