Happy (almost) November! It was a very bug-fix and infrastructure heavy month for Kernl, so lets dive in!
Features, Bugs, & Infrastructure
Redis robust connection handling – We’ve done some resiliency work so that Redis failing will not cause any issues in Kernl. If Redis goes away, all traffic is handled un-cached. Once it comes back, we automatically re-connect. We will also timeout any calls to Redis after 50ms and assume it failed.
Stripe Checkout – Kernl has been using a deprecated version of Stripe’s Checkout.js for the past year or so. We finally migrated to using Stripe’s Checkout service which directs you to Stripe’s website for card adding, and then back to Kernl. Localization and card handling is a lot better this way, and you should have fewer issues adding different payment types to Kernl.
Composer Installation – You can now install the Kernl update checker via Composer. Go to your plugin or theme detail page and look for the new tab in the “Meta” section.
Node.js – All Kernl servers have been upgraded to 12.19.0. This new version of Node.js brings stability and security improvements with it.
RoboSwarm Email Bug – There was a regression where RoboSwarm was sending an email when Kernl customers signed up. RoboSwarm powers the Kernl WordPress load testing service.
Build pipeline – We use BitBucket’s Pipelines services to power the continuous integration of our code base. There was a test with intermittent failures that was causing this to fail in unusual ways.
Admin account bug – A bug was fixed where an admin could get their account into a bad state by adding themselves as a team member.
License management documentation update – We’ve corrected an omission in the license management API docs. The notes field was missing from the documentation.
Invoices – For the longest time Kernl generated it’s invoices via WKHTMLtoPDF. We’ve finally switched over to using the invoices that Stripe generates for us, removing a significant headache from our code base.
Server host name – In load balanced systems it can be hard to track down issues if you don’t know what origin server it came from. Kernl now returns the origin server host name in the response headers to make this easier.
Development environment upgrades – All of our build packages have been upgraded to latest versions.
Back in 2019 I became aware of a new player in the managed WordPress hosting space: EasyWP by Namecheap. As a Namecheap customer (domain registrar) I was curious about their entry into this market and gave them a try. When I initially did performance tests, the platform was still very new and it showed. But now its October 2020 and the platform has matured. Let’s dig into the performance
How did we test EasyWP Performance?
To performance test EasyWP we first created an export of this blog and imported it into the WordPress instance that EasyWP stood up for us. After that we ran a series of tests:
50 users, 100 minutes
500 users, 100 minutes
200 users, 100 minutes
2000 users, 100 minutes
All tests data generators were distributed in the DigitalOcean data centers in New York City, London, and Singapore.
What do each of these test types mean?
Authenticated Browsing – The load test user will authenticate and then browse around the front end of the website. In most cases this causes no caching to happen.
Unauthenticated Browsing – The load test user browses around the front end of the website. No login is performed. In most cases this means that we’ll hit cached pages.
EasyWP Authenticated Browsing – 50 Users
The first test we ran was an authenticated browsing test with 50 concurrent users. We used this as our baseline, because we assumed that EasyWP would be able to handle it without any issues.
As you can see we settled in at around 40 req/s with few errors. Not bad for un-cached performance.
Next we look at the average and median response times. With a median just under 200ms, this seemed like fairly good performance for being authenticated and with traffic distributed around the globe.
Finally, we take a look at the response time distribution. With the exception of the 100% outlier, 99% of all requests finished in under 660ms. This is great performance for an authenticated test.
EasyWP Authenticated Browsing – 500 Users
Our next test was the same as the previous test, except we increased the number of concurrent users by an order of magnitude. At this level, I expect things to fail when running the authenticated scenario. WordPress has awful performance without caching and you would need some seriously beefy infrastructure to absorb this number of authenticated requests.
As you can see things start off pretty great before going right off the rails. I’m not sure what is happening in the background at EasyWP, but I suspect that the service was scaling up while this test was running, otherwise we would have never made it to > 250 successful requests per second.
After the initial shock from the traffic volume, the median response time settled in at around 200ms. The average is much higher though, meaning we have some pretty serious outliers. These will likely show up in the response time distribution graph.
Finally we take a look at the response time distribution. Not entirely terrible given the amount of traffic we were throwing at EasyWP. 99% of requests completed in ~6.4 seconds. The outliers here were BIG though which skewed the average response time data.
EasyWP Unauthenticated Browsing – 200 Users
Next we move on to unauthenticated front end browsing. This is the use case that most manged WordPress hosting platforms optimize for (because 99% of traffic falls into this category). Let’s see how EasyWP does.
As expected for the unauthenticated baseline scenario, EasyWP did well. We settled in at around ~160 requests per seconds with very few errors throughout the entire duration of the test.
The response time graph looks solid. The median stayed at right around 190ms which is to be expected on a test with load generators spread across the globe. The average is a bit higher, so we probably had some outliers. Lets look at the distribution to see.
99% of requests finished in 820ms, with the outlier at 100% being 12 seconds. I honestly expected the 99% number to be a bit lower, but it’s still in a fine range for a load test of this size.
EasyWP Unauthenticated Browsing – 2000 Users
Finally we take a look at what happens to performance on EasyWP when we increase the size of the load test by an order of magnitude. Some hosts handle this test fine, others struggle. So let’s get to it.
EasyWP performed well in this test (ignore the spike, that was a reporting problem). We settled in at around ~1400 requests per second with very few errors. If you do the math, that’s 120M requests per day.
The performance on the 2000 user test was very similar to the 200 user test, which is great for EasyWP. Normally an order of magnitude increase in users would see a decrease in performance, but EasyWP handled it well.
The response time distribution was also strikingly similar to the 200 user test. The only change is that the 100% outlier is a lot higher, which is to be expected when working at a higher scale.
EasyWP Performance Conclusions
In general the performance of EasyWP was solid, especially considering how cost-effective it is. Is it the fastest WordPress host that we’ve tested? No. But it is in the top 2-3 for managed hosts in the performance per dollar category. EasyWP is definitely worth giving a try.
Hello everyone! It’s been awhile since the last update message, but now that summer is nearly over in the northern hemisphere Kernl will again be receiving regular updates and enhancements.
This doesn’t mean that we weren’t busy though! Let’s dig in.
License Management Activated Domains – When viewing your license list, you can now see the domains that have activated the license. If you go into the license detail view, you can make changes to that list. This data will give you granular insight into who is using a specific license.
Kernl Update Checker Automatic Update Support – With the release of WordPress 5.5 we finally upgraded our update checker library. There are a few differences like it being multiple files now, and the instantiation code changing a bit, but the upgrade path is simple for those who want automatic updates.
Bug Fixes & Miscellaneous Changes
Package Upgrades (Analytics) – All supporting packages have been upgraded to their latest version for better performance and security.
Date Retention Bug (Analytics) – We weren’t cleaning up some tables we were supposed to be. This lead to holding onto some data for much longer than 365 days.
UX Improvements (Analytics) – The UX around selecting and comparing dates for Kernl Analytics was a little bit confusing. We made some changes that make it a lot easier to understand for a first-time customer.
Data collection issues(Load Testing) – There was an issue where load testing wasn’t collecting data from the master node after an upgrade to the underlying infrastructure. This was resolved.
UX Improvements(Load Testing) – Some load test templates are large and take awhile to return from the server. An indeterminate spinner was added here to let customers know that things are actually happening. The same situation was happening when long running, high volume tests load initially. There is a lot of data that takes some time to return from the server so an indeterminate spinner was added.
Infrastructure (Load Testing) – The Kernl WordPress Load Testing box was upgraded from 1vCPU+1GB RAM to 2vCPU+2GB RAM.
Meta Tag Parsing(Load Testing) – When you verify a site with Kernl, we look for a meta tag in your HTML. Initially we were attempting to find, capture, and validate this using regular expressions. As is tradition, we found this was a bad idea and switched to using an open-source library (Cheerio.js) instead.
All servers have had their packages upgraded to the latest available versions.
Kernl had another great month, with some bug fixes, blog entries, and a few new features.
Most importantly we’re announcing a limited beta of our new global update CDN! With our new global CDN powered by Vercel, every Kernl update request is cached at edge nodes around the world. This means <= 50ms response times for updates globally.
Reach out to firstname.lastname@example.org if you’d like to give it a try!
Features, Bugs, and Performance
Docs – Added more descriptions to the “Understanding Graphs” portion of the Kernl documentation.
Analytics Date Selection – The flow for selecting dates and products in analytics has changed. You can now only select dates in the range that the selected product has data.
Examples – The Kernl API Examples repo has been updated to include a license management example.
WHMCS License Validation – Kernl now supports WHMCS license validation for plugins and themes. This integration works similarly to our other 3rd party license validation integrations.
Performance – We added a few database indexes to our load testing database. In situations where a lot of load tests existed performance was abysmal.
Tutorial – On the main dashboard page of Kernl there is now video tutorials for getting started with plugin and/or theme updates.
License Expiration Date Bug – There was a bug in Kernl license management where date editing didn’t work correctly when the saved date started with a leading zero.
License Domain Restriction Bug – An edge case existed where license validation would fail with a server error if domain restrictions are set but no domain was passed in to be validated.
Download Increment Bug – Incrementing download count was being handled in a very naive way. It was originally developed when Kernl wasn’t expected to reach the scale it has. This has been resolved by using the $inc operator in MongoDB instead of a find -> increment -> upsert.
Load Test Infrastructure Bar – The progress bar when you create a load test didn’t have a very good default state (it looked like nothing was happening). Now when below 3% complete, the bar will show 3% so you can actually see that things are progressing.
Site Health Edit – You can now edit your site health entries.
Envato License Validation Bug – Due to the deprecation of an API endpoint we were using, users of our Envato license validation need to regenerate their Envato access token that is saved with Kernl.
Envato License Tied to ID – You can now verify that an Envato license that is being used is tied to a specific plugin or theme.
With all the uncertainty in the world right now due to COVID-19, we are intentionally keeping things boring at Kernl. Bug fixes, new docs, and increased capacity are all we did this month.
Stay safe out there everyone!
Features, Bugs, Updates
NEW DOCUMENTATION! – After 5 years we finally moved to a real documentation generator! In the process of moving docs to the new system, all of the docs were updated and checked for accuracy. Check it out at https://docs.kernl.us.
Site Health – You can now change you wp-json root. Some people change this for security reasons. Site Health needs it for some of the more advanced functionality.
Site Health Timezone Bugs – Timezones are hard and our Site Health service wasn’t using them correctly.
Analytics VM Changes – The Kernl Analytics service was starting to get bogged down with increased traffic. It now has 2 vCPUs and 2GB of RAM (up from 1 vCPU and 1GB).
Analytics Bug – We were logging analytics data for customers that aren’t paying for it. This is no longer the case.
Plugin/Theme Git Status – The plugin and theme list pages were displaying inconsistent results for the Git status of plugins/themes. There was an error in our Mongo query that populated this data which has been resolved.
Load Testing Remain Resources – The remaining resources page can take a while to load sometimes because we have to call out to external services. An indeterminate loading spinner was added to let customers know that it’s still working.
System Updates – All operating system level packages have been upgraded to their latest available version.
I hope everyone had a great February! We didn’t too much feature development this month, but there was a lot of bug fixing and performance improvements, so let’s dive in!
Feature, Bugs, and Performance
Node.js – Kernl is now on Node.js 12.16.1. This release was all about security fixes.
Load Testing Machine Provisioning – We weren’t calculating the correct number of machines to provision on DigitalOcean. This lead to some serious over-provisioning when running load tests. This has been resolved, which means more customers can run more load tests at the same time.
Load Testing Secondary Node Behavior – Kernl uses Locust under the covers to run our WordPress Load Testing service. The Locust primary node has an argument called “–expect-slaves”. It tells Locust “Don’t start the load test until at this this many secondary nodes have connected.”. We weren’t calculating this number correctly which led to some weird behavior. This is now resolved so load tests should start in every situation now.
Easy Digital Downloads Domains – Kernl wasn’t passing the domain along to EDD. We now do this, which allows you to restrict updates to specific domains while using EDD.
Load Testing Snapshots – Kernl used to build up each load testing machine from the ground up every time a load test was started. We now start from a snapshot that gets us 50% of the way there. This has improved load test start times (especially on large tests) by an average of 30%.
GitHub Authorization Changes – The GitHub API is changing how it handles authorization headers. We’ve update Kernl to handle this change, so we’ll be good going forward when GitHub deprecates the old method.
High Traffic Endpoint Audit – We did an audit of our high-traffic API endpoints and cleaned some things up. Slight performance improvements were had (1%-2%), but mostly the improvements have been in code readability and comprehension.
GitLab Deployment Issues – In a recent release of GitLab they changed the required fields when asking for an access token via a refresh token. This broke all GitLab deployments for Kernl for a few days while we tracked down the issue. This has since been resolved.
Load Testing Unit & Integration Tests – When our load testing service was launched we weren’t sure if it was going to be successful. We’ve proven that it is a worthwhile feature, so now we’re focusing on reliability. We’re in the process of adding a suite of unit and integration tests around this functionality.
January was a pretty great month for Kernl. We got a lot of bug fixes, some performance improvements, and even a new beta feature out the door. Let’s dive in!
WordPress Site Health Beta
This month we released a beta of our WordPress Site Health service. The goal of this service is to help you determine where performance problems are and why they are happening. The dashboard below gives you a high level view of how performance looks on each of your sites.
Once you click in to any site you see data for the last 7 days.
The data here helps you diagnose performance issues. Using the plugin changes panel you can tie performance issues back to adding/removing/updating of plugins. Google Lighthouse scores show trends on you site’s performance, usability, SEO and more over time.
Improved performance of plugin/theme list pages – The API calls to /api/v1/plugins and /api/v1/themes were incredibly inefficient. They hadn’t really been touched since Kernl’s inception and needed some love. After doing some query optimization and stripping out unnecessary data, the payload and response time were reduced by a factor of 10. The worst case example was response size going from ~750KB to ~75KB and response time from ~4000ms to ~250ms.
Server resource increases – Kernl’s main Node.js application servers have been running with 1 vCPU and 1GB of RAM for about the last 2 years. Lately we’ve seen some resource exhaustion and decided it was time to upgrade. The new app servers now run with 2 vCPUs and 2GB of RAM.
Server disk space / inode exhaustion – The process of building plugins and themes can use up a lot of space and file system resources. We weren’t doing a great job of cleaning those resources up periodically which could cause some performance issue. We now clean up all temporary files once a day which should prevent this from happening anymore.
Easy Digital Download License Validation Error – There was a bug in EDD license validation where the source system wouldn’t send back valid JSON. This would break license validation instead of handling the error gracefully.
Profile page autocomplete – If you had form auto-completion on it would sometimes cause the profile page to reset your password. We’ve disabled autocomplete on this form to resolve the issue.
Theme tiles not showing correct build status – During the course of our performance improvement work we noticed that theme tiles were not showing the correct build status. This has now been resolved.
Welcome to the end of 2019! I hope that everyone has had as good a year as Kernl. Let’s dive in to the final update of 2019 to see what’s new.
Features, Bug Fixes, & Misc.
Improved License Management Search – License management now includes improved search functionality. The previous search functionality was flaky (at best) and not very discoverable. Search is now a first-class citizen, includes free-text search, and should greatly improve the overall usability of Kernl’s WordPress license management.
Load Testing Unit & Integration Tests – When we created the WordPress load testing service it was an experiment. Now that we have proved the viability of the service it’s time to work on stability and overall platform longevity. This month’s focus was on the authorization framework that our load testing service uses.
JS Bundle Size Reduction – Over the past year Kernl’s JS bundle size for our web app grew to over 2MB. We spent some time this month figuring out why and making changes to reduce it. In the end we were able to reduce the bundle size by over 50% down to 1.1MB.
Bug: Inconsistent Webhook & Deploy Key Behavior – After a few customer reported incidents with the automatic webhook and deploy key behavior, we discovered that Kernl wasn’t deleting local references to remote keys and hooks. If you have some issues with deploy keys or webhooks please contact us and we can help resolve the data inconsistency issues that this caused.
Node.js Upgrade to 12.14.0 – This month we upgrade all of our servers to use the latest LTS version of Node. This is includes stability and performance improvements.
Yoast SEO is a popular SEO enablement plugin for WordPress. It helps you avoid common mistakes when it comes to SEO on your blog and also handles things like social media “og:meta” tags. If your blog has a public audience, then it stands a good chance that you are using Yoast.
For this test we used the lowest-tier (1vCPU, 1GB RAM, $5) DigitalOcean droplet out of their SFO2 datacenter. Our server setup was as follows:
Ubuntu 19.04 with all updates installed.
The theme that was used was TwentyTwenty with no modifications and the content tested was 3 “What’s new with Kernl?” posts from earlier this year. We selected a small number due to the effort of filling out all of the SEO data in Yoast.
The WordPress setup was bare-bones. There were no plugins installed except for when we were running the Yoast test.
As with our previous post in this series, the load for this test was generated out of DigitalOcean’s NYC3 datacenter.
The tests were with 200 concurrent users over the course of 1 hour.
Max Requests per Second
One method for determining website performance is what is the maximum number of requests that in can field in a given second. For our purposes this is a pretty good indicator of the performance hit you take for installing plugins.
As you can see from the image above you lose about 25% of your maximum capacity from installing Yoast SEO. With no plugins installed we were able to hit 43 req/s, while with Yoast installed that number went down to 30 req/s.
It’s worth noting here that 30 req/s is 2.5 million requests a day.
First Error Occurrence
The next chart shows when we first started to see errors in our two tests.
Without the plugin installed WordPress was able to hit 38 req/s before seeing errors. Once we enabled Yoast that number went down to 28 req/s. Once again, this is consistent with the performance penalty we saw with the maximum requests per second of about 25%.
Average Response Time
The average response time with and without Yoast SEO tells a similar story to the requests per second measures we have done.
The chart above shows us that without any plugins installed, the average response time under load is around 3000ms. With the Yoast plugin installed the response time goes up to about 4300ms. We’re looking at a roughly 25% change in response time.
99th Percentile Response Time
The 99th percentile chart can be read as “99% of all requests finished in under this time”.
The chart above tells us that without any plugins installed 99% of our requests finished in under ~3600ms. With Yoast SEO installed 99% of our requests finished in ~4900ms. Once again, a roughly 25% penalty for having Yoast installed.
Yoast SEO Performance Conclusions
Yoast is a really good SEO plugin. If SEO matters to you it’s definitely a plugin you should have installed. However you should use caching if you do use it. This goes for WordPress in general, but every plugin you add to WordPress has a performance cost associated with it. If you run a site where caching is difficult you’ll have to carefully weigh the performance cost versus benefit of installing Yoast SEO.
If you are just starting with Cloudways it can be tough to figure out where your money is best spent! The prices are different for each platform and how they are sized isn’t them same either. This review gathers data using Kernl’s WordPress Load Testing tool and then massages that data into someone easy to digest.
Google Cloud – A “small” instance out of their Northern Virginia data center. $39.36 / month.
Amazon Web Services – A “small” instance out of their Northern Virginia data center. $36.51 / month.
Vultr – “2GB” plan out of their New Jersey data center. $23 / month
Linode – “2GB” plan from their New Jersey data center. $24 / month
Digital Ocean – “2GB” plan from one of the New York City data centers. $22 / month.
And we had the following load testing setup:
Content was an export of this blog
1000 concurrent users
30 minute duration
3 users per second ramp up
Load generating machines were in the San Francisco #2 Digital Ocean data center.
Overall Cloudways Performance and Value
Each provider performed extremely well in our tests. No errors were reported and they all reached over 700 requests / s sustained traffic. However not all hosts are created equal.
Costs / Month (cents)
$ / 1000 req (cents)
Given that each load test was exactly the same, you can see that some hosts performed better than others.
AWS and GCP are a bit more costly than Linode, Vultr, and Digital Ocean and they also performed worse than the lower cost providers.
Cloudways Provider Response Times
While cost per requests is a good metric we also care about the quality of those requests. Quality can be measured in a number of ways, but response time is often a “good enough” measure of how well a particular host performs. For the providers that you can get through Cloudways, the variance in request quality is extremely interesting.
The chart below shows the 99th percentile response time performance for each provider. It means that 99% of all requests during the load test finished at or below this time.
You can see that lower cost providers once again out-perform the higher cost AWS and GCP in pretty significant ways. The really interesting result here is that Vultr responded to 99% of requests in 240ms or less! Way to go Vultr!
If you aren’t sure which provider is the best value on Cloudways, from a raw cents per requests perspective you can’t go wrong with Linode, Vultr, and Digital Ocean. It’s the same story when you consider the quality (re: response time) of service: Vultr, Linode, and Digital Ocean are the clear winners here.