What’s New With Kernl – April 2019

Lots of great stuff came down the pipeline this month at Kernl, so lets dive in!

Features / Bugs / Improvements

  • Share Load Tests – You can now share your load tests publicly! Just click the “share” button that shows up after your load test is completed.
  • Analytics Widget – If you scroll down past the list of plugin/theme versions in the product detail view you’ll now see a widget with analytics data (for Kernl Analytics subscribers).
  • Load Testing Line Graph Performance – For large load testing data sets the line graph performance has been improved by disabling some of the flashier features of Chart.js
  • Analytics page now attempts to show the analytics for the first product in your product list upon entry.
  • NODE_ENV was not set to “production” on our app servers. It is now.
  • The transition to a fully managed load balancer is now complete.

Blog Posts

Thats it for this month!

Introducing Shareable WordPress Load Test Results

For quite awhile now Kernl has had the ability to throw some serious load at your WordPress site, but never a great way to share the results. Today that changes with the introduction of load test result sharing!

🎉

Why would I share my load test results?

The main use case for sharing your load test results is showing your clients that their new WordPress site can handle the traffic load that they expect. For someone like a YouTube or Twitter celebrity this can be a very real problem. Larger organizations also would enjoy this peace of mind.

How do I get started?

Easy! Just click the share button next to the load test that you want to share. You’ll be presented with your sharing URL that can be copy and pasted into Facebook, Twitter, Slack, or Email!

Can I see some examples?

Yes!

As always, feel free to reach out to Jack if you have any questions!

W3 Total Cache Performance Review

Test your own site’s performance with Kernl WordPress Load Testing!

In the world of WordPress performance, you can’t go far without talking about caching plugins. I’ve personally used several different caching plugins throughout my time as a WordPress developer but have never really took the time to see how the plugins perform under pressure. Until now.

Over the next couple of months I’ll be releasing a series of blog posts detailing the performance characteristics of different WordPress caching plugins. This DOES NOT include server caching plugins (LiteSpeed, Varnish, Nginx Fast CGI, etc) because those are in a league of their own and it wouldn’t really be an “apples to apples” comparison.

The Setup

I decided to try and run this comparison as if I were looking for a cheap VPS provider and hoping to get a lot of performance out of it. Since I’m a huge Digital Ocean fanboy, I went ahead and used the pre-built WordPress image in their marketplace and dropped it on a $5/month droplet. This leaves me with a typical LAMP setup with mostly default configurations for everything. I also wanted to test the configuration with Memcachedso that was installed as well.

The Tests

In order to fully test W3 Total Cache I ran the following tests with 200 concurrent users out of Digital Ocean’s NYC3 data center. The server under test was located in Amsterdam.

  • Baseline Test – No caching at all. We just need to see what performance we can get without caching enabled.
  • Fragment Cache Only (Memcached)
  • Database Cache Only (Memcached)
  • Page Cache Only (Memcached)
  • All Caches Enabled (Memcached)
  • All Caches Enabled (OPCode APC)
  • All Caches Enabled (Disk Enhanced[page] & Disk)

It is important to note that W3 Total Cache will also do HTML, CSS, and JS minification. I’m not particularly interested in that, but do know that it can compress these outputs with a variety of different tools and may decrease overall load time.

High Level Results

W3 Total Cache is a solid caching plugin that gives you a lot of different options for getting good performance out of WordPress.

TestMax req/sResponse Time (90%)
Baseline192000ms
Fragment Cache Only192100ms
Database Cache Only202100ms
Page Cache Only601100ms
All Caches (Memcached)131110ms
All Caches (OPCode APC)601800ms
All Caches (Disk)13390ms

Baseline Test

As expected the baseline load test didn’t perform the greatest. It is no secret that WordPress doesn’t perform well under load without caching plugins, but we needed a baseline anyways.

Lots of red.

We peaked at around 19 req/s and then started to trail off towards 5 req/s as the error rate increased. Looking at the response time distribution, you can see that things didn’t look much better.

I mean it could be worse. At least people could access your site in under 5 seconds, but most people bail in 1 second or less.

Fragment Cache Only

The fragment cache in W3 Total Cache is supposed to “reduce time for common operations”. I’m not sure what that means but by itself it didn’t seem to help me out very much.

With fragment caching enabled we still see a similar request and failure profile as when we didn’t have any caching enabled at all.

Response time distribution was similar to no caching. My guess is that fragment caching optimizes a very narrow set of operations and that my blog doesn’t really use any of them.

Database Cache Only

I expected to see some pretty good performance with the database cache enabled. Presumably this cached queries to MySQL with should have really helped us scale. It didn’t 🙁

You can see that we were just slightly better than with the fragment cache, but not even close to where I thought we would be.

The response time distribution was bad too. It actually is a bit higher than the other ones. I assume that this was just a fluke and that if I ran this test several times it would be the same as the others.

Page Cache Only

With full page caching enabled we start to see some good performance improvements.

60 * 60 *60 * 24 == 5.1 Million per day

The graph above shows that we hit about 60 req/s before the wheels start to fall off, but even then we’re able to continue serving requests at a respectable 50-ish req/s.

The response time distribution looks a lot better here as well. 90% of requests finished in 1.1 seconds. This isn’t bad at all considering you’re going at > 50 req/s.

All Caches Enabled (Memcached)

With all caches enabled we start to see some pretty impressive performance out of W3 Total Cache.

Zoom Zoom

130 req/s is solid, but you’ll notice there is a steady stream of errors coming across as well. In general I would say that this setting is good to prevent to your site from going down but that you should look to increasing resources ASAP.

Response time distribution is where we start to see some really excellent gains. The pages were served almost immediately with the 99th percentile being 220ms. Much of that time can be attributed to latency due because the servers are so far apart.

All Caches Enabled (OPCode APC)

I expected the results of this test to be similar to Memcached because they’re both in-memory caches, but that wasn’t even close to true.

What a strange looking chart.

We initially peaked at 60 req/s and then started to see a large uptick in errors. Requests eventually started to climb back up, but the error rate stayed consistent.

I was pretty disappointed in the response time distribution. I expected it to be similar to Memcached but it looks like OPCode APC just isn’t as efficient. 90th percentile at 1.8s isn’t terrible but it isn’t good either.

All Caches Enabled (Disk)

Disk-based caching is very likely the type you would be using if you were in a shared hosting environment. The good news is that the disk-based caching that W3 Total Cache does is quite good.

133 req/s with a fairly small error rate is great! At this level you could easily handle 99% of the traffic scenarios you’ll see as a WordPress developer.

The response time distribution was similar to the Memcached version of this test in that 99% of requests finished in under 170ms. Not bad considering WordPress was fielding > 130 requests per second.

Conclusions

W3 Total Cache is an excellent choice for caching. While some of its configuration can be confusing, you can still get very acceptable performance by enabling page caching or all caching thats available. Its also worth noting that you could very likely get excellent performance with no errors if you tweaked some Apache and MySQL settings in addition to using W3 Total Cache.

Test your own site’s performance with Kernl WordPress Load Testing!

Is More RAM or More CPU Better for WordPress Hosting?

Want to performance test your own WordPress site? Try Kernl WordPress Load Testing!

Throughout the lifetime of Kernl I’ve always kept an eye out for ways to reduce costs, which means I’ve constantly got an eye on Digital Ocean‘s droplet pricing to see if there are any cost efficiencies to be had. Back in January of 2018 Digital Ocean released what they called “flexible” plans. These plans are 3 droplet types that all cost $15/month, but have 3 different configurations:

  • 3GB RAM, 1 vCPU
  • 2GB RAM, 2 vCPUs
  • 1GB RAM, 3 vCPUs

I recently re-discovered these plans and thought to myself “Jack, you should load test these configurations and see which is better for WordPress!”. So here we are, attempting to answer the age-old question “Is RAM or CPU more important for WordPress performance?”.

Server Configuration

As with the $5 VPS shootout, I used the setup guide posted here to get a consistent setup across the different servers. I ended up with the following software versions:

  • Nginx 1.14.0
  • PHP FPM 7.2
  • MariaDB 10.1.1
  • Ubuntu 18.04 LTS

The WordPress server location was in Digital Ocean’s London (lon1) data center and the load generators were located in Digital Ocean’s New York City 3 (nyc3) data center.

No caching was used for this test because I wanted to test WordPress performance, not caching plugin performance.

Content

The test WordPress site was populated with a WP export dump of this blog. That means that the test skewed extremely read-heavy. Your results may be different for a site that is write-heavy, but experience tells me that this probably isn’t true given how chatty WordPress is with MySQL in most situations.

Test A: 3GB RAM, 1 vCPU

The first test that I ran against the $15 droplet was with the 3GB RAM and 1 vCPU configuration. I had expected this to perform the worst out of any of the configurations due to WordPress generally being CPU bound when it comes to performance.

As you can see things went OK until we hit around 18 requests per second, and then everything went off the rails. This isn’t too surprising as I didn’t have any caching plugins installed. This is just raw WordPress with MySQL (MariaDB). I ended up killing the load test after 7 minutes because it was clear what the results were.

The response time distribution was also pretty terrible with a 99th percentile of slightly over 5 seconds.

Test B: 2GB RAM, 2 vCPUs

Next up was the 2GB of RAM with 2 vCPUs $15 configuration. I had actually thought that this configuration would perform the best due to the nice balance of RAM and CPUs. It performed better than the first test, but it wasn’t the best.

As you can see we had far fewer errors this time and were able to reach 60 requests per second. For those doing the math that is 3.3x better performance than a single vCPU (for the same price!).

The response time distribution was starting to look better with 2 vCPUs as well. The 99th percentile was ~2.7 seconds which isn’t too bad while serving 60 requests per second (5.1 million requests per day).

Test C: 1GB RAM, 3 vCPUs

Finally I tested the 1GB RAM, 3 vCPUs $15 configuration. Personally I thought that this would perform 2nd best because I figured the server would become RAM starved while trying to server more requests. I was wrong.

So for the same price as the 3GB RAM, 1 vCPU server we can get 6.5x the performance by simply choosing the configuration with more CPU availability. We peaked at around 118 requests per second (10.1 million requests per day) and then it started to trail off as we started to see errors.

The response time distribution was pretty excellent here and started to look like what we would see from high performance servers (1 big outlier with generally good response times across the rest of the spectrum). With the 99th percentile at 700ms it seems like a no-brainer to go with more vCPUs.

Conclusions

It seems obvious from the data presented above that you are always better off going with more vCPUs than RAM, but I hesitate to always recommend that because reality is more complicated. There are likely situations where a good balance of RAM and vCPUs would perform better (lots of image uploads and processing for example).

So what does this all mean? In most cases you are probably better off with more vCPUs instead of more RAM. However, I strongly recommend that you do some load testing to make sure that this is true for your WordPress site.

Want to performance test your own WordPress site? Try Kernl WordPress Load Testing!

Adventures in Scalable WordPress Hosting: Part 2

Interested in testing your WordPress scalability? Check out the Kernl WordPress Load Testing beta program!

In part 1 of this series I explored scaling WordPress using WP Super Cache and by throwing more expensive hardware at the problem. In part 2 of this series we’ll go on adventure in horizontal scalability using load balancers, NFS, Memcached, and an externally hosted MySQL.

The Plan

To horizontally scale any app is an exercise in breaking things apart as much as possible. In the case of WordPress there are a few shared components that I wanted to break up:

  • File System – The file system is the most problematic part of scaling WordPress. Unless you change how WordPress stores plugins, themes, media, and other things you need to have a shared file system that all nodes in your cluster can access. There are likely some other solutions here, but this one provides a lot of flexibility.
  • MySQL – In many WordPress installs MySQL lives on the same machine as WordPress. For a horizontally scaled cluster this doesn’t work so we need a MySQL that is external.
  • Memcached – It was brought to my attention that during part 1 of this series using WP Super Cache to generate static pages was sort of cheating. In the spirit of making this harder for myself I introduced W3 Total Cache instead and will be using an external Memcached instance as the shared cache.

Now that the basic why and what is out of the way lets talk about the how. I’m a huge fan of Digital Ocean. I use them for everything except file storage so I’m going to use them for this WordPress cluster as well. Here’s how its going down:

  1. Create a droplet that will act as the file system for our cluster. Using NFS all droplets in the cluster will be able to mount it and use it for WordPress. I’m also going to use this for Memcached since NFS doesn’t take up many resources.
  2. Create a base droplet that has Nginx and PHP7.2-FPM installed on it. There is a little bit of boilerplate configuration here, but in general the install is typical. The only change to the Nginx configuration where I set the root directory to be the NFS mount. Use this base droplet to configure WordPress database settings.
  3. Use Compose.io create a MySQL database. I wanted something that was configured well that I didn’t have to think about. Totally worth the $27 / month.
  4. Once the above are done take a snapshot of the base droplet and use it to create more droplets. If all goes well you shouldn’t need to do any configuration.
  5. Using Digital Ocean’s load balancer service add your droplets to the load balancer.
  6. Voila! Thats it.
Ugly architecture diagram

No Cache Smoke Test

200 users, 10 minutes, 2 users/sec ramp up, from London

As with every load test that I do, the first test is always just to shake out any bugs in the load test itself. For this test I didn’t have any caching enabled and only a single app server behind the load balancer. It was effectively the same as the first load test I did during part 1 of this blog series.

As you can see from the graph below, performance was what we would expect from the setup that I used. We settled in to 21 requests / second with no errors.

As Expected.

The response time distribution wasn’t very great. 90% of requests finished in under 5 seconds, but thats still a very long time. Generally if I saw this response time distribution I would think that its time to add caching or scale up/out.

Not bad. Not great.

So. Many. Failures.

2000 users, 120 minutes, 2 users/sec ramp up, from London

The next test I decided to run was the sustained heavy load test. This is generally where I start to see failures from managed WordPress hosting providers. Given that I didn’t add any more app servers to the load balancer and had no caching things went as poorly as you would expect.

All the failures of failure land.

Everything was fine up until ~25 req/s and then the wheels fell off. The response time distribution was bad too. No surprises here.

50% of requests in 5 seconds, 100% in…33 seconds 🙁

Looks like its time to scale.

Horizontal Scalability

2000 users, 120 minutes, 2 users/sec ramp up, from London

Before adding Memcached to the setup I wanted to see how it scaled without it. That means adding more hardware. For this test I added four more application servers (Nginx + PHP) to the load balancer and ran the test again.

Linear Growth

As you can see from the request/failure graph we experience roughly linear growth in our maximum requests/second. Given we originally maxed out at ~20 req/s on one machine, maxing out at ~100 req/s with five machines seems like exactly the sort of result that I would expect to see. The response time distribution also started to look better:

Not perfect, but better.

Obviously a 90% score of 4 seconds isn’t awesome, but it is a lot better than the previous test. I did make a tiny tweak to the load balancer configuration that may have helped though. I decided to use the ‘least connections’ options instead of ’round robin’. ‘Least connections’ tells the load balancer to send traffic to the app server with the least number of active connections. This should help with dog piling on a server with a few slower connections.

Given the results above we can safely assume linear growth tied to the number of app servers that we have for quite some time. Meaning for each app server that I add I can expect to handle an additional ~20 req/s. With that in mind, I wanted to see what would happen if I enabled some caching on this cluster.

Gotta Go Fast

In my previous test of vertical scaling I used WP Total Cache to make things go quick. WP Total Cache generates static HTML pages for your site and then serves those. The benefit being that static pages are extremely fast to serve. In this test I wanted to try a more dynamic approach using Memcached and W3 Total Cache. W3 Total Cache takes a very different approach to caching by storing pages, objects, and database queries in Memcached. In general this caching model is more flexible, but possibly a bit slower. I installed Memcached on the same server as the NFS mount because it was under utilized. In a real production scenario I wouldn’t violate this separation of concerns.

Once I enabled W3 Total Cache and re-ran the last test I got some pretty great results.

Boom.

With W3 Total Cache enabled and 5 app servers we settled in at ~370 requests/second. More impressive is that we only saw 5 failures during the entire test. For perspective Kernl pushed 1,329,470 requests at the WordPress cluster I created. Thats a failure rate of 0.0003%.

My favorite part of this test was the response time distribution. Without having to wait on MySQL for queries the response times became crazy good.

The “bad” outlier is only 2.5s.

99% of requests finished in 29ms. And the outlier at 100% was only 2.5 seconds. Not bad for WordPress.

Going Further

Being the good software developer that I am I wanted to push this setup to it’s limits. So I decided to try a test that is an order of magnitude more difficult:

20,000 users, 10 users/sec ramp up, for 60 minutes, from London

Things didn’t go great but not because of WordPress. I won’t show any graphs of this test but I started to get limited by the network card on the NFS/Memcached machine. Digital Ocean says that I can expect around 30MB/sec out of a given droplet and with this test I was starting to bump in to that limit. If I wanted to test it further I would have had to load balance Memcached which felt a little bit outside of scope. In a real production scenario I would likely pay for a hosted Memcached service to deal with this for me.

Conclusions

With Kernl I’m always weighing the build versus buy question when it comes to infrastructure and services. Given how much effort I had to put in to making this setup horizontally scalable and how much effort it would take to make it reproducible and manageable, it hardly seems worth creating and managing my own infrastructure.

Aside from my time the cost of the hardware was also not cheap.

  • Load Balancer – $10 / month
  • MySQL Database – $27 / month
  • Memcached (if separate from NFS) – $5 / month
  • NFS Mount (if separate from Memcached) – $5 / month
  • Application Servers – $25 / month ($5 / month * 5 servers)
  • Total – $72 / month

At $72 / month I could easily have any of the managed WordPress hosting companies (GoDaddy, SiteGroup, WPEngine, etc) run my setup, handle updates, security, etc. The only potential hiccup is the traffic limits they place on your account. This setup can handle millions of requests per day and while their setups can too, they’ll charge you a hefty fee for it.

As with any decision about hardware and scaling the choice varies from person to person and organization to organization. If you have a dedicated Ops team and existing hardware, maybe scaling on your own hardware makes sense. If you’re a WordPress freelancer and don’t want to worry about it, maybe it doesn’t. IMHO I wouldn’t scale WordPress on my own. I’d rather leave it to the professionals.

Interested in testing your WordPress scalability? Check out the Kernl WordPress Load Testing beta program!

Adventures in Scalable WordPress Hosting: Part 1

If you follow the Kernl Blog you’ll know that recently I’ve been writing about load testing different managed WordPress cloud providers. Half of the reason for doing this is to shake out any bugs in Kernl’s WordPress load testing platform and the other half is to learn whats out there in terms of managed WordPress hosting.

As I went through the first round of tests I kept thinking: “I wonder how they achieve that level of performance with WordPress?”. This blog post and the post that will follow it are a chronicle of my attempts to scale WordPress to the levels that these managed cloud providers are achieving in an economical fashion.

The Tests

Having done a handful of load tests against other cloud providers I figured that I should hold myself to the same tests. The scale I’m going to try and achieve is:

  1. 200 concurrent users for 10 minutes.
  2. 2000 concurrent users for 2 hours.
  3. 20000 concurrent users for 1 hour.

The first test is just to shake out bugs in the load test, but I have seen some providers start to throw errors at that level. The second test is testing for sustained load. And the third test is simulating a heavy traffic spike.

So. Basic.

To get things started I created a super basic WordPress install on a $5/month Digital Ocean droplet. The droplet specs:

  • 1 CPU
  • 1GB RAM
  • 1000GB data transfer
  • Ubuntu 18.10

I chose to use the LEMP stack instead of the LAMP stack mostly because I’m more familiar with tuning Nginx for performance. I followed the guide at https://www.digitalocean.com/community/tutorials/how-to-install-linux-nginx-mysql-php-lemp-stack-ubuntu-18-04 to get things running. The software specs:

  • PHP 7.2
  • Nginx 1.15.5
  • MySQL 5.7.24

The first test went really well. I didn’t performance tune anything and didn’t have any sort of cache enabled. After 10 minutes we had settled into 35 requests / second and didn’t see any failures at all.

So. Much. Blue.

For 90% of people this is probably more performance than they would ever need. The response time distribution was even awesome. 100% of requests finished in ~500ms.

Not bad 1 hour of work and $5

And Then The Wheels Fell Off

After my early success with the basic 200 user load test I thought it was time to throw some serious load at my WordPress install. This time I did the 2000 concurrent users for 2 hours test. At this point there still wasn’t any caching plugin installed.

Things did not go well

As you can see things didn’t go well. We peaked at around 40 requests/s but then our failure rate started to increase is a really bad way. You can also see that we sorta stopped fielding requests after awhile. Looking at the system load information, you can see why things went poorly. The $5 droplet just couldn’t handle anymore.

The poor $5 droplet was tapped out

As you would expect in this situation, the response time distribution was pretty dismal. In fact, this is the worst response time distribution that I’ve seen in all the load testing that I’ve performed 🙂

Thats right: 2% of requests took over 500s to return 🙁

After reaching the max capacity of the $5 droplet with no tuning, it was time to try and scale.

WP Super Cache Me

WP Super Cache is a caching plugin that generates static HTML files of your WordPress site. For read-heavy sites its tough to beat in terms of performance. The blog that I’m load testing with definitely falls into this category so it was the right choice for this test.

This test was simply a repeat of the last test (2000 users, 2 hours, etc) but with caching enabled. The results were pretty great.

135 req/s is respectable for $5/month

With WP Super Cache enabled on the $5 droplet we were able to field around 135 req/s, however you can see that our error rate was elevated during much of the test. If you expect to see this sort of traffic on a regular basis then this isn’t a great outcome but still pretty respectable for $5/month. The response time distribution tells a different story though:

33% of requests finished in > 10 seconds :/

Whats the point of serving 135 req/s if it takes more than 10s per request for 33% of your users? People are just going to close the tab after 1 second so we obviously have some more work to do.

Scale Me Up

When scaling any website you have 2 options (and they aren’t mutually exclusive):

  1. Scale up (vertically)
  2. Scale out (horizontally)

Scaling up is usually the easiest thing to do because you’re basically throwing more hardware at the problem. Digital Ocean makes scaling up really easy so I decided to give that a go first. This test was once again just a repeat of the 2000 users for 2 hours test but with better hardware. I upgraded from 1 CPU to 3 CPUs which seemed like the right choice given that it didn’t appear that memory was the problem in my previous tests.

3 CPUs -vs- 1 CPU

So how did it go? Real good actually. Once all the load test users were sending requests we settled in at 344 request / second. If that rate continued all day that comes out to 29 million requests. Not bad for $15/month.

So. Many. Requests.

We’re still seeing some failures, but relative to the number of requests it is much lower than the previous test. We can do better but that will likely take some more vertical or horizontal scaling. But what about the response times? Turns out adding more CPUs helped out quite a bit.

This is MUCH better.

100% of our requests finished in under 1.6s. While not SUPER fast it is still a respectable showing for the sort of load that this box was receiving. Even more impressive is that 90% of requests finished in under 100ms and some of that could be attributed to latency. The droplet was spun up in NYC3 and the load test generators were in Toronto, Canada.

Conclusions

The biggest selling point (for me) with WordPress is that it’s easy. With very little configuration or effort I was able to get a WordPress installation serving > 300 req/s. Sure it wasn’t perfect. I am still getting elevated error rates and vertical scaling can only take us so far. But this is likely good enough for almost anyone.

Part II

In part 2 of this series I’ll attempt to scale WordPress horizontally by using shared block storage to host the WordPress file system, a dedicated MySQL machine, and a bunch of application servers running behind a load balancer. The goal is serve 20,000 (or more!) concurrent users for 1 hour without any errors and response times below 1 second. Follow @kernl_ on Twitter to be notified when part 2 is published!

Load Testing the ChemiCloud Managed WordPress Hosting Service

At the beginning of December Kernl launched a closed beta for our WordPress Load Testing service. As part of the bug shakedown we’ve been spending some time load testing different managed WordPress hosting services. Some of previous tests include WordPress.com, CloudWays, and GoDaddy. For this test, we turned our sights on ChemiCloud.

How do we judge the platform?

Using Kernl’s load testing feature we run 3 different load tests against the target system.

  • The Baseline – This is a simple baseline load test that we use to verify that our configuration is correct and that the target can handle even minor traffic. It consists of 200 concurrent users, for 10 minutes, ramping up at 2 users / second, with traffic originating in San Francisco.
  • Sustained Traffic – The sustained traffic test mimics what traffic might look like for a read-heavy website with a lot of visitors. This load test consists of 2000 concurrent users, for 2 hours, ramping up at 2 users / second, with traffic originating from San Francisco.
  • Traffic Spike – This test is brutal. We use it mimic the sort of traffic that your WordPress site might experience if a link to it were shared by a Twitter or Instagram celebrity. The load test consists of 10,000 concurrent users, for 1 hour, ramping up at 10 users / second, with traffic originating from San Francisco.

All traffic for this test is generated out of Digital Ocean’s SFO2 data center.

What ChemiCloud plan was used?

ChemiCloud has several different tiers for managed WordPress hosting. We decided on the “Oxygen” plan. At a high level this seemed to align well with the hosting that we tested thus far.

ChemiCloud - Oxygen Plan
ChemiCloud – Oxygen Plan

Caveats

This load test is intentionally simple. It is read heavy. Many WordPress sites have this sort of traffic profile, but not all do. If you need to perform a WordPress load test with a different traffic profile Kernl supports this. Ideally we should also do multiple tests over time to make sure that this test wasn’t an outlier. Future load test articles will hopefully include this sort of rigor but for now this test can give you reasonable confidence in how you can expect ChemiCloud to perform under a read-heavy load.

The Baseline Test

200 concurrent users, 2 users / s ramp up, 10 minutes, SFO

As most of the hosting providers that we test do, ChemiCloud performed well on the baseline test. They settled in at right around 25 requests / second.

ChemiCloud - Requests
ChemiCloud – Requests

We did see a few failures towards the end of the test, but it appears that it was only a spike. Once the spike passed we didn’t see any more errors for the duration of the test.

ChemiCloud - Failures
ChemiCloud – Failures

The response time distribution for ChemiCloud was solid for this baseline test. 99% of requests finished in 550ms. If we go further down the distribution you can see that 95% of requests finished in ~250ms which is quite good. Even the 100% outlier still wasn’t that bad.

ChemiCloud - Response Time Distribution
ChemiCloud – Response Time Distribution

Sustained Traffic Test

2000 concurrent users, 2 users / s ramp up, 2 hours, SFO

For the sustained traffic test ChemiCloud did a great job serving requests while keeping response times down. As you can see from the graph below, the test settled in to right around 260 requests / second. The journey to that many users was smooth and there aren’t any surprises on the graph.

ChemiCloud - Requests
ChemiCloud – Requests

There were a few failures during the test period, but it appears that they were only a temporary blip. You can see that about half-way through the test we ran into ~32 failures. After that we didn’t see any more for awhile, and then we had one more before not seeing any again for the rest of the test. For some perspective, we performed 1,861,230 requests again ChemiCloud and only 33 failed. Thats a failure rate of 0.0017%! Nice work team ChemiCloud.

ChemiCloud - Failures
ChemiCloud – Failures

The response time distribution was pretty great for the sustained test as well. While there was an outlier at 100% (which is common), 99% of requests finished in under 400ms. Thats an effort worthy of praise with WordPress!

ChemiCloud - Response Time Distribution
ChemiCloud – Response Time Distribution

Traffic Spike Test

20000 concurrent users, 10 users / s ramp up, 1 hour, SFO

The traffic spike load test is brutal for any host. Nobody ever expects to see this kind of traffic out of nowhere so few are prepared for it. ChemiCloud handled the traffic rather well though. We eventually reached 1200 requests / second which is pretty impressive for a plan that costs $17.95 a month. There weren’t any surprises on the way up to that level of traffic, but as you’ll see we did start to see error rates increase.

ChemiCloud - Requests
ChemiCloud – Requests per Second

At about 15 minutes into the load test we started to see an uptick in failure rates. The rate of failure stayed consistent throughout the test after that. This is a fairly common pattern when hosts become overloaded with traffic. In general ChemiCloud performed well even with these failures. We sent 4,332,244 requests to ChemiCloud over an hour period and 134,893 failed. For this sort of load test a failure rate of 3.1% isn’t bad.

ChemiCloud - Failures
ChemiCloud – Failures

The most interesting graph from this load test was the response time distribution. You would expect to see a general degradation of response time performance as request failures increased but that wasn’t the case at all. Everything below the 99th percentile performed remarkably well considering the traffic we threw at it. 98% of requests finished in under 370ms. Great work!

ChemiCloud - Response Time Distribution
ChemiCloud – Response Time Distribution

Conclusions

ChemiCloud competes well with the other hosts that we’ve tested. They have a solid price-point and you get a lot of control over your WordPress environment. If you need a host that can handle some solid traffic spikes they are a good choice.

Want to be part of the Kernl WordPress Load Testing Beta? Sign up and then send an email to jack@kernl.us

Load Testing the WordPress.com Managed WordPress Service

This December Kernl launched it’s new WordPress Load Testing service. As part of the bug shakedown we decided to load test as many managed WordPress providers as we could. In this test, we turn our sights to WordPress.com.

How do we judge the platform?

For this series of blog posts we judge the platform via 3 different load tests.

  • Baseline – This test is for 200 concurrent users, for 10 minutes, with a 2 user / second ramp up. We use this test to double-check our configuration before throwing heavier load at the provider.
  • Sustained Traffic – 2000 concurrent users, for 2 hours, ramping up at 2 users / second. This test represents what a high traffic WordPress site might see on a day to day basis.
  • Traffic Spike – The traffic spike test simulates what might happen if a Twitter or Instagram celebrity mentioned your site. 20,000 concurrent users, for 1 hour, ramping up at 10 users/s.

For all 3 tests traffic is generated out of Digital Ocean‘s San Francisco 2 (SFO2) data center.

What WordPress.com plan was used?

For this test we used the Free WordPress.com plan. We didn’t need any of the bells and whistles, plus (as you’ll see later) the performance didn’t suffer at all. If performance had been impacted we would have increased our plan to somewhere around the $10/month mark. As with all of our load tests we don’t do any configuration. We simply import the content of http://www.re-cycledair.com and then start testing.

WordPress.com Free Plan
WordPress.com Free Plan

The Baseline Test

200 concurrent users, 2 users / s ramp up, 10 minutes, SFO

As expected WordPress.com did very well during the baseline test. The test settled in at 26 req/s.

WordPress.com Load Test - Requests per second
WordPress.com Load Test – Requests per second

There also weren’t any failures through the duration of the test.

WordPress.com Load Test - Failures
WordPress.com Load Test – Failures

The response time distribution was also excellent with 99% of requests returning in under 200ms.

WordPress.com Load Test - Response Time Distribution
WordPress.com Load Test – Response Time Distribution

The Sustained Traffic Test

2000 concurrent users, 2 users / s ramp up, 2 hours, SFO

The sustained traffic test is where WordPress.com’s hosting really started to shine. As of now it is hands down the best host that we’ve tested. The throughput settled in at around 258 requests / second. There also weren’t any surprises on our way up to that number of requests.

WordPress.com Load Test - Requests per second
WordPress.com Load Test – Requests per second

The most impressive part about this entire load test was the failure rate. Over a 2 hour test, under heavy load, for more than 1.4 million requests, not a single request failed. Thats some serious stability.

WordPress.com Load Test - Failures
WordPress.com Load Test – Failures

While not as impressive as the 0% failure rate, the response time distribution was still pretty amazing. 99% of all requests finished in well under 100ms. There was an outlier in the ~1500ms range, but that isn’t uncommon for load tests.

WordPress.com Load Test - Response Time Distribution
WordPress.com Load Test – Response Time Distribution

The Traffic Spike Test

20000 concurrent users, 10 users / s ramp up, 1 hour, SFO

WordPress.com is blazing fast. It didn’t event flinch with 20000 concurrent users. The request rate settled in at 1717 requests / second (!). On the way up to that request rate there were no surprises or stutter steps.

WordPress.com Load Test - Requests per second
WordPress.com Load Test – Requests per second

The failure rate was exceptional as well. For an hour long test, with sustained heavy load, and a total of 4.3 million requests, there were 0 errors.

WordPress.com Load Test - Failures
WordPress.com Load Test – Failure Rate

Finally, the most impressive graph in this entire test! For the traffic spike test, WordPress.com’s distribution chart is nothing short of fantastic. 99% of traffic had response times below 50ms, and even the 100% outlier was still only 1 second. Great work WordPress.com team!

WordPress.com Load Test - Response time distribution
WordPress.com Load Test – Response Time Distribution

Conclusions

If you need extremely robust performance and are OK with the restrictions of WordPress.com they seem like a great choice.

Want to be part of the Kernl WordPress Load Testing Beta? Sign up and then send an email to jack@kernl.us