Adventures in Scalable WordPress Hosting: Part 1

If you follow the Kernl Blog you’ll know that recently I’ve been writing about load testing different managed WordPress cloud providers. Half of the reason for doing this is to shake out any bugs in Kernl’s WordPress load testing platform and the other half is to learn whats out there in terms of managed WordPress hosting.

As I went through the first round of tests I kept thinking: “I wonder how they achieve that level of performance with WordPress?”. This blog post and the post that will follow it are a chronicle of my attempts to scale WordPress to the levels that these managed cloud providers are achieving in an economical fashion.

The Tests

Having done a handful of load tests against other cloud providers I figured that I should hold myself to the same tests. The scale I’m going to try and achieve is:

  1. 200 concurrent users for 10 minutes.
  2. 2000 concurrent users for 2 hours.
  3. 20000 concurrent users for 1 hour.

The first test is just to shake out bugs in the load test, but I have seen some providers start to throw errors at that level. The second test is testing for sustained load. And the third test is simulating a heavy traffic spike.

So. Basic.

To get things started I created a super basic WordPress install on a $5/month Digital Ocean droplet. The droplet specs:

  • 1 CPU
  • 1GB RAM
  • 1000GB data transfer
  • Ubuntu 18.10

I chose to use the LEMP stack instead of the LAMP stack mostly because I’m more familiar with tuning Nginx for performance. I followed the guide at https://www.digitalocean.com/community/tutorials/how-to-install-linux-nginx-mysql-php-lemp-stack-ubuntu-18-04 to get things running. The software specs:

  • PHP 7.2
  • Nginx 1.15.5
  • MySQL 5.7.24

The first test went really well. I didn’t performance tune anything and didn’t have any sort of cache enabled. After 10 minutes we had settled into 35 requests / second and didn’t see any failures at all.

So. Much. Blue.

For 90% of people this is probably more performance than they would ever need. The response time distribution was even awesome. 100% of requests finished in ~500ms.

Not bad 1 hour of work and $5

And Then The Wheels Fell Off

After my early success with the basic 200 user load test I thought it was time to throw some serious load at my WordPress install. This time I did the 2000 concurrent users for 2 hours test. At this point there still wasn’t any caching plugin installed.

Things did not go well

As you can see things didn’t go well. We peaked at around 40 requests/s but then our failure rate started to increase is a really bad way. You can also see that we sorta stopped fielding requests after awhile. Looking at the system load information, you can see why things went poorly. The $5 droplet just couldn’t handle anymore.

The poor $5 droplet was tapped out

As you would expect in this situation, the response time distribution was pretty dismal. In fact, this is the worst response time distribution that I’ve seen in all the load testing that I’ve performed ๐Ÿ™‚

Thats right: 2% of requests took over 500s to return ๐Ÿ™

After reaching the max capacity of the $5 droplet with no tuning, it was time to try and scale.

WP Super Cache Me

WP Super Cache is a caching plugin that generates static HTML files of your WordPress site. For read-heavy sites its tough to beat in terms of performance. The blog that I’m load testing with definitely falls into this category so it was the right choice for this test.

This test was simply a repeat of the last test (2000 users, 2 hours, etc) but with caching enabled. The results were pretty great.

135 req/s is respectable for $5/month

With WP Super Cache enabled on the $5 droplet we were able to field around 135 req/s, however you can see that our error rate was elevated during much of the test. If you expect to see this sort of traffic on a regular basis then this isn’t a great outcome but still pretty respectable for $5/month. The response time distribution tells a different story though:

33% of requests finished in > 10 seconds :/

Whats the point of serving 135 req/s if it takes more than 10s per request for 33% of your users? People are just going to close the tab after 1 second so we obviously have some more work to do.

Scale Me Up

When scaling any website you have 2 options (and they aren’t mutually exclusive):

  1. Scale up (vertically)
  2. Scale out (horizontally)

Scaling up is usually the easiest thing to do because you’re basically throwing more hardware at the problem. Digital Ocean makes scaling up really easy so I decided to give that a go first. This test was once again just a repeat of the 2000 users for 2 hours test but with better hardware. I upgraded from 1 CPU to 3 CPUs which seemed like the right choice given that it didn’t appear that memory was the problem in my previous tests.

3 CPUs -vs- 1 CPU

So how did it go? Real good actually. Once all the load test users were sending requests we settled in at 344 request / second. If that rate continued all day that comes out to 29 million requests. Not bad for $15/month.

So. Many. Requests.

We’re still seeing some failures, but relative to the number of requests it is much lower than the previous test. We can do better but that will likely take some more vertical or horizontal scaling. But what about the response times? Turns out adding more CPUs helped out quite a bit.

This is MUCH better.

100% of our requests finished in under 1.6s. While not SUPER fast it is still a respectable showing for the sort of load that this box was receiving. Even more impressive is that 90% of requests finished in under 100ms and some of that could be attributed to latency. The droplet was spun up in NYC3 and the load test generators were in Toronto, Canada.

Conclusions

The biggest selling point (for me) with WordPress is that it’s easy. With very little configuration or effort I was able to get a WordPress installation serving > 300 req/s. Sure it wasn’t perfect. I am still getting elevated error rates and vertical scaling can only take us so far. But this is likely good enough for almost anyone.

Part II

In part 2 of this series I’ll attempt to scale WordPress horizontally by using shared block storage to host the WordPress file system, a dedicated MySQL machine, and a bunch of application servers running behind a load balancer. The goal is serve 20,000 (or more!) concurrent users for 1 hour without any errors and response times below 1 second. Follow @kernl_ on Twitter to be notified when part 2 is published!

Load Testing the WordPress.com Managed WordPress Service

This December Kernl launched it’s new WordPress Load Testing service. As part of the bug shakedown we decided to load test as many managed WordPress providers as we could. In this test, we turn our sights to WordPress.com.

How do we judge the platform?

For this series of blog posts we judge the platform via 3 different load tests.

  • Baseline – This test is for 200 concurrent users, for 10 minutes, with a 2 user / second ramp up. We use this test to double-check our configuration before throwing heavier load at the provider.
  • Sustained Traffic – 2000 concurrent users, for 2 hours, ramping up at 2 users / second. This test represents what a high traffic WordPress site might see on a day to day basis.
  • Traffic Spike – The traffic spike test simulates what might happen if a Twitter or Instagram celebrity mentioned your site. 20,000 concurrent users, for 1 hour, ramping up at 10 users/s.

For all 3 tests traffic is generated out of Digital Ocean‘s San Francisco 2 (SFO2) data center.

What WordPress.com plan was used?

For this test we used the Free WordPress.com plan. We didn’t need any of the bells and whistles, plus (as you’ll see later) the performance didn’t suffer at all. If performance had been impacted we would have increased our plan to somewhere around the $10/month mark. As with all of our load tests we don’t do any configuration. We simply import the content of http://www.re-cycledair.com and then start testing.

WordPress.com Free Plan
WordPress.com Free Plan

The Baseline Test

200 concurrent users, 2 users / s ramp up, 10 minutes, SFO

As expected WordPress.com did very well during the baseline test. The test settled in at 26 req/s.

WordPress.com Load Test - Requests per second
WordPress.com Load Test – Requests per second

There also weren’t any failures through the duration of the test.

WordPress.com Load Test - Failures
WordPress.com Load Test – Failures

The response time distribution was also excellent with 99% of requests returning in under 200ms.

WordPress.com Load Test - Response Time Distribution
WordPress.com Load Test – Response Time Distribution

The Sustained Traffic Test

2000 concurrent users, 2 users / s ramp up, 2 hours, SFO

The sustained traffic test is where WordPress.com’s hosting really started to shine. As of now it is hands down the best host that we’ve tested. The throughput settled in at around 258 requests / second. There also weren’t any surprises on our way up to that number of requests.

WordPress.com Load Test - Requests per second
WordPress.com Load Test – Requests per second

The most impressive part about this entire load test was the failure rate. Over a 2 hour test, under heavy load, for more than 1.4 million requests, not a single request failed. Thats some serious stability.

WordPress.com Load Test - Failures
WordPress.com Load Test – Failures

While not as impressive as the 0% failure rate, the response time distribution was still pretty amazing. 99% of all requests finished in well under 100ms. There was an outlier in the ~1500ms range, but that isn’t uncommon for load tests.

WordPress.com Load Test - Response Time Distribution
WordPress.com Load Test – Response Time Distribution

The Traffic Spike Test

20000 concurrent users, 10 users / s ramp up, 1 hour, SFO

WordPress.com is blazing fast. It didn’t event flinch with 20000 concurrent users. The request rate settled in at 1717 requests / second (!). On the way up to that request rate there were no surprises or stutter steps.

WordPress.com Load Test - Requests per second
WordPress.com Load Test – Requests per second

The failure rate was exceptional as well. For an hour long test, with sustained heavy load, and a total of 4.3 million requests, there were 0 errors.

WordPress.com Load Test - Failures
WordPress.com Load Test – Failure Rate

Finally, the most impressive graph in this entire test! For the traffic spike test, WordPress.com’s distribution chart is nothing short of fantastic. 99% of traffic had response times below 50ms, and even the 100% outlier was still only 1 second. Great work WordPress.com team!

WordPress.com Load Test - Response time distribution
WordPress.com Load Test – Response Time Distribution

Conclusions

If you need extremely robust performance and are OK with the restrictions of WordPress.com they seem like a great choice.

Want to be part of the Kernl WordPress Load Testing Beta?ย Sign up and then send an email to jack@kernl.us

Load Testing the CloudWays Managed WordPress Service

At the beginning of December Kernl launched the closed beta of it’s WordPress load testing service. As a test to shake out any bugs we’ve decided to run a blog series load testing managed WordPress services. Today we’re going to talk about the CloudWays managed WordPress service. In particular, CloudWays deployed to Vultr.

How is the platform judged?

Cloudways will be tested using 3 different load tests:

  • The Baseline – This is a 200 concurrent user, 10 minutes, 2 user/s ramp up test from San Francisco. This test is used to verify the test configuration and to make sure that Cloudways doesn’t go belly-up before we get started ๐Ÿ™‚
  • The Sustained Traffic Test – This test is for 2000 concurrent users, ramps up at 2 users/s, from San Francisco, for 2 hours. The sustained traffic test represents a realistic load for a high traffic site.
  • The Traffic Spike Test – This test is intentionally brutal. It simulates 20000 concurrent users, ramps up at 10 users/s, from San Francisco, for 1 hour. It represents the sort of traffic pattern you might see if a Twitter celebrity shared a link to your blog.

What CloudWays plan was used?

For this test we used the lowest tier plan available while hosting on Vultr. The cost of the plan is $11 / month and includes full SSH access to the box that CloudWays deploys your WordPress instance on.

CloudWays $11 / month plan hosted on Vultr
Selected CloudWays Plan

Where does the traffic originate?

The traffic for this load test originates in Digital Ocean‘s SFO2 (San Francisco) data center. The Vultr server lives in their Seattle data center.

The baseline load test

200 concurrent users, 2 users / s ramp up, 10 minutes, SFO

The baseline WordPress load test that we did with CloudWays is used to test configuration. CloudWays performed well on this test. You can see from the request graph that we settled in at around 25 requests / second.

CloudWays BaseLine Load Test Requests
CloudWays Baseline Test – Requests per second

The failure graph for the baseline load test was empty, which is generally expected for the baseline test.

CloudWays BaseLine Load Test Failures
CloudWays Baseline Test – Failures

Finally the request distribution graph for the baseline test. You can see that 99% of the requests finished in ~200ms. There was at least one outlier at the ~5000ms mark, but this isn’t uncommon for load tests.

CloudWays BaseLine Load Test Response Time Distribution
CloudWays Baseline – Response Time Distribution

The sustained heavy traffic load test

2000 concurrent users, 2 users / s ramp up, 2 hours, SFO

The sustained traffic load test represents what a WordPress site with high readership might look like day over day.  The CloudWays setup responded quite well for the hardware that it was on.

CloudWays Sustained Heavy Traffic Load Test - Requests
CloudWays Sustained Load Test – Requests

You can see that performance was great for the first 10% of the test. The CloudWays setup had no trouble handling the load thrown at it. However once we started getting to around 85 requests / second the hardware had trouble keeping up with the request volume. You can see from the choppy behavior of request graph that the Varnish server which sits in front of WordPress was starting to get overwhelmed by the request volume. Considering that this particular CloudWays plan was deployed to a low-level Vultr VM, this performance isn’t bad at all.

The failure graph was a little disappointing, but not unexpected knowing the hardware that we tested on. It is very likely that if we tested on a more robust underlying Vultr box we would have had much better results. You can see that failures increased in a fairly linear rate through the whole load test.

CloudWays Sustained Heavy Traffic Load Test - Failures
CloudWays Sustained Load Test – Failures

The final graph for this test is the response distribution graph. This graph shows you for a given percentage of requests how many milliseconds they took to complete. In this case CloudWays didn’t perform great, but once again I’ll point to the fact that the underlying Vultr hardware isn’t that robust.

CloudWays Sustained Heavy Traffic Load Test - Response Time Distribution
CloudWays Sustained Load Test – Response time distribution

From the graph you can see that 99% of requests completed in ~95 seconds. Yes, you read that correctly. You can interpret this graph as you like but taking the other graphs into consideration you can see that Varnish and the underlying Vultr hardware were completely overwhelmed. Knowing that makes this a little less terrible. We suspect that a smaller load test (maybe 750 concurrent users?) might yield a far better response time distribution. Once a server becomes overwhelmed the response time distribution tends to go in a bad direction.

The traffic spike load test

20000 concurrent users, 10 users / s ramp up, 1 hour, SFO

Given what we know about the sustained traffic load test your expectations for how this test went are probably spot on. CloudWays did as good as can be expected with how the underlying hardware is allocated, but you would likely need to upgrade to a much larger plan to handle this level of traffic. We ended up stopping this load test after about 30 minutes due to the increased failure rate.

CloudWays Traffic Spike Load Test - Requests
CloudWays Traffic Spike Load Test – Requests per Second

The requests per second never really leveled out. It isn’t clear what the underlying reason was for the uneven level at the top of the graph. Regardless, top-end performance was similar to the sustained traffic test.

The failure chart looks as we expected it to. After a certain point we start to see increased failure rates. They continue up and to the right in a mostly linear fashion.

CloudWays Traffic Spike Load Test - Failures
CloudWays Traffic Spike Load Test – Requests per Second

The response time distribution is really bad for this test.

CloudWays Traffic Spike Load Test - Response Time Distribution
CloudWays Traffic Spike Load Test – Response Time Distribution

As you can see 80% of the requests finished in < 50s which means that 20% of the requests took longer than that. The 99% mark was only reached after > 200s, at which point the user is likely long gone.

Conclusions

For $11 / month the CloudWays managed WordPress installation did a great job, but there are better performers out there in the same price range (GoDaddy for instance). For the sake of this review which only looks at raw performance, CloudWays probably isn’t the best choice. But if you’re looking for good-enough performance with extreme flexibility then you would be hard pressed to find a better provider.

Want to run load tests against your own WordPress sites? Sign up for Kernl now!

What’s New With Kernl – November 2018

It has been a busy few months for Kernl. Lots of great work has gone into the WordPress load testing feature work as well as a few structural changes to increase reliability.

  • Cache moved to Redis – For as long as Kernl has existed our cache backend was powered by Memcached. We have now finished migrating to Redis hosted at Compose.io.
  • AngularJS Upgrade to 1.7.5 – Fairly straight-forward upgrade to Angular 1.7.5. We wanted to take advantage of performance improvements and few bug fixes.
  • WordPress Load Testingย – Over the past few months we’ve been cooking up something new. Imagine if you could easily test performance changes to you or your client’s WordPress installation? Or be able to tell your client with confidence how many customers at a time their site can support (and what their experience will be like!). What if you could do all this without writing a single line of code or spinning up your own testing infrastructure? We’re ready to start beta testing so send an email to jack@kernl.us if you would like to be a part of it.