Cloudways WordPress Performance Value Review

Cloudways is a managed WordPress hosting provider that allows you to deploy your WordPress site onto several different platforms. These platforms are:

Cloudways home page
Cloudways!

If you are just starting with Cloudways it can be tough to figure out where your money is best spent! The prices are different for each platform and how they are sized isn’t them same either. This review gathers data using Kernl’s WordPress Load Testing tool and then massages that data into someone easy to digest.

Load & performance test your own WordPress site with Kernl’s WordPress Load Testing service!

What Was Tested?

We tested the following Cloudways setups:

  • Google Cloud – A “small” instance out of their Northern Virginia data center. $39.36 / month.
  • Amazon Web Services – A “small” instance out of their Northern Virginia data center. $36.51 / month.
  • Vultr – “2GB” plan out of their New Jersey data center. $23 / month
  • Linode – “2GB” plan from their New Jersey data center. $24 / month
  • Digital Ocean – “2GB” plan from one of the New York City data centers. $22 / month.

And we had the following load testing setup:

  • Content was an export of this blog
  • 1000 concurrent users
  • 30 minute duration
  • 3 users per second ramp up
  • Load generating machines were in the San Francisco #2 Digital Ocean data center.

Overall Cloudways Performance and Value

Each provider performed extremely well in our tests. No errors were reported and they all reached over 700 requests / s sustained traffic. However not all hosts are created equal.

AWSGCPLinodeVultrDigital Ocean
Costs / Month (cents)36513936240023002200
Total Requests13409671274018144230114557221445759
$ / 1000 req (cents)2.723.091.661.581.52

Given that each load test was exactly the same, you can see that some hosts performed better than others.

Cost per 1000 requests (in cents)

AWS and GCP are a bit more costly than Linode, Vultr, and Digital Ocean and they also performed worse than the lower cost providers.

Cloudways Provider Response Times

While cost per requests is a good metric we also care about the quality of those requests. Quality can be measured in a number of ways, but response time is often a “good enough” measure of how well a particular host performs. For the providers that you can get through Cloudways, the variance in request quality is extremely interesting.

The chart below shows the 99th percentile response time performance for each provider. It means that 99% of all requests during the load test finished at or below this time.

99th percentile performance

You can see that lower cost providers once again out-perform the higher cost AWS and GCP in pretty significant ways. The really interesting result here is that Vultr responded to 99% of requests in 240ms or less! Way to go Vultr!

Conclusions

If you aren’t sure which provider is the best value on Cloudways, from a raw cents per requests perspective you can’t go wrong with Linode, Vultr, and Digital Ocean. It’s the same story when you consider the quality (re: response time) of service: Vultr, Linode, and Digital Ocean are the clear winners here.

Full WordPress Load Test Results:

Load & performance test your own WordPress site with Kernl’s WordPress Load Testing service!

WordPress Plugin Performance Implications: Wordfence

In the world of WordPress there are a lot of different plugins you can install to extend its functionality. One of the most popular plugins is Wordfence, a security plugin developed by the fine folks over at Defiant.

Test the performance implications of your own WordPress plugins with Kernl WordPress Load Testing.

Most WordPress developers understand that the more plugins you add to your site the slower it goes, but exactly how much slower isn’t something that is often measured. More importantly, nobody has bothered to figure out the performance implications of installing many of the most popular plugins available to WordPress users today.

This all changes now with this series of blog posts exploring the performance implications of different WordPress plugins. In this series of posts we’ll test each plugin in isolation and then with caching enabled using Kernl’s WordPress load testing service. First up, is Wordfence.

Machine Setup

The test machine for these tests was a $5 Digital Ocean droplet in their SFO2 (San Francisco) data center. The machine has 1GB of RAM, 1 vCPU, and a 25GB hard disk.

The software installed on the machine is as follows:

  • Ubuntu 19.04 with all updates installed.
  • Nginx 1.16.1
  • PHP-FPM 7.3
  • MariaDB 10.3

For tests that required caching we used our favorite caching plugin, W3 Total Cache, with memcached as the data store.

The theme that was used was TwentyTwenty with no modifications and the content was an export of this blog.

All traffic was generated out of Digital Ocean’s NYC3 (New York City) data center.

Test Methodology

A series of 4 tests were run to test the performance implications of installing Wordfence. They were:

Each test was with 200 concurrent users for 1 hour.

Maximum Requests per Second

The first metric that we looked at was the maximum requests per second that the site was able to handle under each situation outlined above.

Wordfence - Maximum Requests per Second
Wordfence – Maximum Requests per Second

As you can see the difference between having Wordfence enabled and having Wordfence disabled is huge. The non-cached site with no plugins enabled handled a maximum of 26 requests per second. With Wordfence enabled it could only handle 12.

More interesting though was that with caching enabled (W3 Total Cache) the site could handle 165 requests per second, but only 67 requests per second with Wordfence enabled.

These results were so surprising that we ran the tests twice. The results were the same (within 1%-2%) each time.

First Error Occurrence

The next metric we looked at was when did the first error occur during our load tests.

Wordfence - First Error req/s
Wordfence – First Error req/s

Once again we see that adding Wordfence took a pretty serious toll on our performance. For the baseline test (no plugins, no cache) we see our first error at around 26 requests / second. In the Wordfence test with no cache we saw it at 12 requests / second.

With our caching enabled test, we never saw any errors when Wordfence wasn’t enabled. However with Wordfence enabled we saw our first error at 56 requests / second.

Average Response Time

Now that we’ve looked at server capacity metrics (max requests, first error), let’s take a look at how your end user experience changes with Wordfence enabled.

Wordfence - Avg. Response Time (milliseconds)
Wordfence – Avg. Response Time (milliseconds)

These results were fairly interesting and not at all what was expected. The average response time for the baseline test was around 4.6 seconds, while the average response with Wordfence enabled was about 2.5 seconds. So why might that be? Looking through the data it appears that the baseline test had far more successful requests and that with Wordfence enabled the requests seemed to fail faster. In short, baseline test had higher throughput but slower response time. The Wordfence test had lower throughput and faster response time.

Turning our attention to the caching scenarios, we can see that enabling Wordfence is extremely problematic at scale. With caching enabled, our baseline test had an average response time of 146ms. With caching + Wordfence that time ballooned over 10x to 1874ms.

99th Percentile Response Time

Our final metric was looking at the 99th percentile response times for our different scenarios. What does that mean? It answers the question “How long does it take for 99% of requests to finish?”. This intentionally leaves out the last 1% because those are often outliers.

Wordfence - 99th Percentile Response Time
Wordfence – 99th Percentile Response Time

As you can see above, the un-cached 99th percentile hovers right around 5s when Wordfence is enabled or disabled. Since this is the 99th percentile such a high response time isn’t too surprising in both scenarios.

The more interesting scenario is when caching is enabled. For WordPress with only a caching plugin running, the 99th percentile is 370ms. That means 99% of all requests finished in 370ms. With Wordfence also enabled (Wordfence + caching), that number jumped to 2400ms. That’s a ~7X increase.

Conclusions

From these tests we came to a few conclusions:

  1. Caching does not solve all of your performance problems. If your cache strategy relies on requests making it all the way through to WordPress, then you are very likely to still take a performance hit from other plugins. Things like Cloudflare, Varnish, Litespeed, or Nginx can help alleviate this problem.
  2. Running Wordfence is expensive from a performance standpoint. The data suggests that by simply enabling Wordfence you lose about 50% of your maximum capacity and can increase your response time between 2x-7x.
  3. Wordfence is still worth it for a lot of people. If you’ve operated a WordPress site for any length of time you know how often they get attacked. Wordfence does a great job of reducing attack surface area and making it hard for people to attack you.

Test the performance implications of your own WordPress plugins with Kernl WordPress Load Testing.

Introducing Team Management and the Kernl Unlimited Plan

One of the most requested features that Kernl gets is Team Management. Team management can take on a lot of different forms, but for Kernl’s purposes it is the ability to share access via the Kernl interface with multiple people with different sets of credentials.

Team Management

Starting today you can sign up for the Kernl Unlimited plan (see below) and start managing your team. Team management allows you to share your Kernl account with authorized users. When a user is part of your team, they have a slightly restricted view of Kernl but otherwise see the same things you do. What’s restricted?

  • Continuous Deployment – Adding connected Git repositories is the responsibility of the admin. Non-admin users don’t need to see this information.
  • Billing – Once again, this is an admin responsibility. The admin is able to change plans, add credit cards, etc.
  • Team Management – Only the admin can add and remove users from their account.

Aside from that everything is the same for admin and non-admin users.

Kernl Team Management

The Unlimited Plan & Pricing Changes

In addition to team management we are also reducing the number of plans that Kernl has. Prior to this change we had 5 different plans (solo, agency, enterprise, huge, massive) which seemed like a bit much. To simplify Kernl’s pricing structure and give our customers better options we went down to 3 plans:

  • Solo – This $9 per month plan is our base plan. Most customers start here and then grow into larger plans as their needs change.
  • Agency – The $39 agency plan is essentially our former “Enterprise” plan but with increased limits for plugins, themes, licenses, load testing, and feature flags.
  • Unlimited – This new $79 unlimited plan allows you to have unlimited plugins, themes, licenses, feature flags and very high limited on load testing (50K concurrent users). This plan also includes team management so you can more easily control access to your Kernl account.

As always, current Kernl customers are grandfathered into the plan that they are currently on.

Kernl Pricing

What’s Next?

Now that Kernl has a more robust upper-level offering we hope to start rolling more features into it. For example:

  • In the near future we’ll include Kernl Analytics with Unlimited plan.
  • We’re also planning on rolling out limited team management into the Agency plan.

WordPress Database Performance Showdown: MySQL vs MariaDB vs Percona

WordPress requires that you use a MySQL compatible database for its database backend. It used to be that you could confidently choose MySQL and go on with life, but in 2019 the choice isn’t quite that simple. With the MySQL, MariaDB, and Percona as the attractive options, how do you know which to choose?

Choosing a database isn’t always about performance, but for the sake of this article it will be πŸ™‚

For this series of tests we tested database performance out-of-the-box with no special tweaks. It is quite possible that a professional database administrator could make each database run more performant, but most people hosting WordPress aren’t DBAs. With regards to caching, none was enabled. We wanted to test database performance, not cache performance.

MariaDB Logo

What was tested?

The WordPress host machine was a DigitalOcean CPU Optimized droplet with 16 vCPUs (dedicated hyper-threads) and 32 GB of RAM. This monster of a machine was chosen so that we could be certain that Nginx + PHP-FPM weren’t the cause of any bottlenecks. A minor tweak to the PHP-FPM config allowed for full use of all 16 vCPUs.

The database host machines were DigitalOcean CPU Optimized droplets with 4 vCPUs (dedicated hyper-threads) and 8GB of RAM. CPU Optimized droplets were chosen because we didn’t want our tests to be at the mercy of shared CPU resources.

Each deployment was in DigitalOcean’s SFO2 region with the WordPress server communicating with the database over the internal private network. The traffic producing nodes were deployed in DigitalOcean’s NYC3 data center and communicated via the public internet.

The software versions are as follows:

  • Percona Server for MySQL 8.0.16-7
  • MariaDB 10.4.8-GA
  • MySQL 8.0.18

What tests were run?

Want to test your site’s performance? Sign up for Kernl WordPress Load Testing.

For each database that was tested, we ran a load test with the following parameters:

  • 500 concurrent users
  • 2 req/s ramp up
  • 30 minute duration

The goal here wasn’t to bring the database to it’s knees but instead see how it performed under sustained heavy load, but not so heavy that it falls over.

Left: WordPress (Nginx + PHP-FPM), Right: MySQL

MariaDB WordPress Performance

During the MySQL acquisition of Oracle in 2009 there was a lot of concern amongst the core developers that Oracle would eventually close off MySQL to the world (similar to Oracle’s business model). Before that could take place, a GPL fork of MySQL was created named MariaDB.

MariaDB is open source and in active development. But how does it stand up to our WordPress load tests? Let’s find out.

First we’ll take a look at the requests and failures per second.

~379 req/s with periodic errors

As you can see from the results above the performance scales up very well over time eventually peaking at ~379 req/s. We see periodic database-related errors (the vCPUs on the database were almost completely saturated) but nothing too crazy. Next, let’s see how the response times look.

Median response time of ~190ms.

The median response time of WordPress when backed by MariaDB under heavy load stays remarkably consistent. You can see that the average is slowly creeping up by the end of the test, but nothing that would be noticeable by customers yet.

99% of request finish in 500ms

The response time distribution is a little more interesting than the median response time. 90% of all requests finish in under 300ms and 99% of all requests finish in less than 500ms. Overall, the performance of MariaDB out of the box with no configuration is quite good.

Percona WordPress Performance

Another open-source fork of MySQL, Percona was started in 2006 and has been steadily delivering value-added features and enterprise support on top of MySQL for more than 12 years. With all that accumulated experience, how does Percona measure up?

~320 requests/s, ~3 errors/s

Overall the Percona WordPress performance was pretty good. Not quite a good as MariaDB but nothing to scoff at. The one interesting thing was that the error rate was consistent across most of the test once the request volume passed ~200 requests/s. Lets see if the response time graph adds to the story.

~450ms Median Response Time

The data here is actually pretty interesting. The response time for Percona was comparable to MariaDB right up until the time it started getting consistent errors. After that, it more than doubled and stayed that way for the rest of the test.

99% of requests finished in ~550ms

The response time distribution tells similar story to the response time chart. The difference between the 50th percentile and the 99th percentile shows that performance was very consistent across the entire test, it just wasn’t as good as MariaDB.

MySQL WordPress Performance

Last but not least (well….) in our test is MySQL. It’s been around since 1994 and is now owned by Oracle. In it’s latest releases there has been lot of great features such as window functions and more JSON features to compete with PostgreSQL. So let’s see how it did.

295 requests/s, 3-4 errors/s

MySQL had a similar trajectory to Percona: It did pretty good across the board with a small but consistent series of errors after it started to go past 200 req/s. Also note that it never achieved higher than 300 req/s, which both MariaDB and Percona did.

500ms Median Response Time

The shape of the MySQL response time graph looks nearly identical to Percona, just about 50ms slower once the vCPUs started to get saturated. The detailed look in the response time distribution tells a similar story.

99% of requests finished in 875ms

The response time distribution is where you can see Percona and MySQL diverge a bit. At the 99th percentile MySQL is returning at 875ms, while Percona was more like 550ms. In general, the distribution matches what one would expect given how the response time graph looks

Conclusions

The out of the box numbers for MariaDB make it look like the clear winner here. And compared to both Percona and MySQL it is in the raw performance department. This isn’t to say that you couldn’t tune Percona or MySQL to out-perform MariaDB, but only that you get more performance with zero configuration changes.

Req / sError / sMed. Response Time99% Dist.
MariaDB379~1190ms500ms
Percona3203450ms550ms
MySQL2953500ms875ms

Load Testing Vultr’s New High Frequency Servers with WordPress

Run your own WordPress Load Tests with Kernl!

Back in June Vultr announced the general availability of their new “High Frequency” servers. Reading through the announcement I was intrigued by their claim of using “3+GHZ processors and blazing fast NVMe storage!” and immediately wondered what WordPress performance would look like versus their regular Cloud Compute offering.

What was tested?

To get a better idea of the performance characteristics of the Vultr High Frequency Compute servers, I ran several types of load tests with all Vultr servers sitting in their Silicon Valley data center:

  • 200 concurrent users from New York City (Digital Ocean NYC3)
  • 2000 concurrent users from New York City (Digital Ocean NYC3) & London (Digital Ocean LON1)

And I tested the following scenarios:

All servers with the exception of the “performance tuned” ones were built using the pre-built WordPress image that Vultr offers with no caching or performance tuning done. The “performance tuned” servers were created using Nginx, PHP-FPM, MariaDB, Memcached, and W3 Total Cache.

Results

Starting with the Vultr High Frequency boxes, lets break down performance.

$6 32GB NVMe

The $6 Vultr High Frequency(HF) machine performed well against the roughly equivalent $5 Vultr Cloud Compute(CC) instance. If we take a look at the requests/failures per second charts you can see that the HF machine out-performed the CC instance by quite a lot.

$6 32GB NVMe Vultr High Frequency Requests/Failures
$5 25GB SSD Vultr Cloud Compute Requests/Failures

As you can see the HF server was able to handle roughly twice as many requests per second as the CC server without any errors. Let’s check out the average and median response times as well as the response time distribution.

$62 32GB NVMe Vultr High Frequency Response Time
$6 32GB NVMe Vultr High Frequency Response Time
$62 32GB NVMe Vultr High Frequency Response Time Distribution
$6 32GB NVMe Vultr High Frequency Response Time Distribution

You can see that as the test progressed response times got steadily worse until leveling out at around 2.5s. The response time distribution shows that 99% of requests finished in 3s or under, with the 100th percentile outlier coming in at just under 6s. This is honestly pretty good performance for no tuning at all.

$5 25GB SSD Vultr Cloud Compute Response Time
$5 25GB SSD Vultr Cloud Compute Response Time
$5 25GB SSD Vultr Cloud Compute Response Time Distribution
$5 25GB SSD Vultr Cloud Compute Response Time Distribution

The response times for the $5 cloud compute instance were about 2 seconds worse than the high frequency instance, with the average coming in at around 4.5. The response time distribution was worse from a performance perspective, but better from a consistency perspective with the spread between the 50th percentile and the 100th percentile only being ~1.3s.

At a glance, it looks like the Vultr High Frequency server out-performs the cloud compute in a significant way for only $1 extra. However, our next tests show that it might not be so simple.

$24 128GB NVMe (Performance Tuned)

The next set of tests that were performed were 2000 concurrent users on a performance tuned setup. The traffic originated from a cluster of servers in both New York and London, with the host server being in Vultr’s Silicon Valley data center.

First, lets take a look at the requests per second handled by the $24 High Frequency server and the $20 Cloud Compute server.

$24 128GB NVMe Vultr High Frequency Requests

Now, 2000 concurrent users is a lot. I think that the HF machine did quite well overall, but I confess that I did expect a bit more out of it. It topped out at around 1250 requests per second, with errors hovering in the ~175 requests per second range. If left to run for another 30 minutes I think it would have reached closer to 1350 requests per second.

$20 80GB SSD Vultr Cloud Compute Requests

This is where things start to get interesting. The Vultr marketing team bills the high frequency machines as heads and shoulders above the regular cloud compute machines. Maybe it is for some workloads. But for this test it doesn’t seem much better at all. Looking at both graphs you can see that the HF machine topped out a bit higher, and had fewer errors, but not enough to warrant the extra cash (in my opinion). Lets see what sort of story the response times tell.

$24 128GB NVMe Vultr High Frequency Response Time
$24 128GB NVMe Vultr High Frequency Response Time Distribution

On the performance tuned box you can see that the requests that did complete successfully were all quite fast. The average response time was between 100ms and 200ms throughout the entire test. The response time distribution was solid as well, with 99% of requests finishing in under 500ms. Not bad for 2000 concurrent users.

Now let’s take a look at how the comparable cloud compute server did.

$20 80GB SSD Vultr Cloud Compute Response Time
$20 80GB SSD Vultr Cloud Compute Response Time Distribution

For the Vultr Cloud Compute instance response times also hovered between 100ms-200ms, although just a hair higher than the high frequency servers. The response time distribution was excellent as well, with 99% of requests coming in under 500ms. The main difference here is that there was a 100th percentile outlier here at the high frequency server didn’t have.

Thoughts & Conclusions

From what I can tell, the Vultr High Frequency servers are better than the Cloud Compute servers, but maybe only for certain types of workloads. You can see in the $5/$6 test that they easily out-performed the cloud compute instances, but in the more expensive $20/$24 test the results weren’t so cut and dry.

If we look at it from a requests per dollar standpoint, the cloud compute instance is actually a better value for this type of workload.

Requests per Dollar

I suggest running extensive load tests against the Vultr High Frequency machines before making any effort to switch over. They might be better for you. They might perform the same for more money.

In conclusion: Β―\_(ツ)_/Β―

Run your own WordPress Load Tests with Kernl!

Digital Ocean 30 Hour WordPress Load Test for Reliability and Consistency

Perform your own 30 hour load tests with Kernl!

Over the past 5 months I’ve been writing a lot of different articles testing WordPress performance when under heavy load. One of the comments that I often receive is “Yes, but how reliable is the host over time?”. To determine that answer I made some changes to Kernl that would allow customers to do long duration tests against their providers with a steady load. Given my affinity for Digital Ocean, I figured that would be a great first host to test.

What Was Tested & How I Tested It

Digital Ocean has several data centers across the globe and I figured that I should test each of these data centers to see how reliable they were. For this test I ran a single load test against the following data centers for 30 hours with a 25 concurrent users:

  • New York City (NYC3)
  • Toronto (TOR1)
  • Bangalore (BLR1)
  • Frankfurt (FRA1)
  • London (LON1)
  • Singapore (SGP1)
  • Amsterdam (AMS3)

All requests were made from the Digital Ocean data center in San Francisco (SFO2). The target of each load test was a simple $5 / month droplet with the WordPress image from the Digital Ocean marketplace installed on it.

Results

The table below summarizes the results for all of the long duration load tests. Click the region to see more details of the load test.

Example Load Test Results Page
RegionRequestsFailuresFailure %Req/s Avg
NYC32.49M1390.005%23
TOR12.46M1260.005%22/23
BLR12.17M1150.005%20
FRA12.28M14050.06%21
LON12.33M13350.05%21
SGP12.29M980.004%21
AMS32.30M2280.009%21

As you can see from the results above Digital Ocean’s reliability is excellent across the entire testing period. Even the data centers with the highest error rate (Frankfurt, London) had an incredibly small error rate. I’m not going to add the response time distribution results here because they were uniformly excellent.

Anomalies

  • The Bangalore (BLR1) test averaged only 20 req / s. Even though the geographic distance is far, I expected the response times to go up but to have a similar throughput.
  • The Toronto (TOR1) load test averaged 22 req / s for 28 hours, then jumped up to 23 req / s for that last two hours of the test. Maybe a noisy neighbor went quiet?
  • Digital Ocean’s Frankfurt (FRA1) and London (LON1) data centers had an order of magnitude more errors than the other data centers I tested.

Conclusions

As a whole, Digital Ocean performed very well in all of their data centers with a moderate amount of sustained traffic to a WordPress instance. In the future I would like to try running all of these tests twice with a different origin for each test run. It’s also worth noting that I haven’t done this type of test on any other platform yet. I hope that as I test more providers I’ll find out whether or not Digital Ocean performed as well as I think it did.

Perform your own 30 hour load tests with Kernl!

What’s New With Kernl – April 2019

Lots of great stuff came down the pipeline this month at Kernl, so lets dive in!

Features / Bugs / Improvements

  • Share Load Tests – You can now share your load tests publicly! Just click the “share” button that shows up after your load test is completed.
  • Analytics Widget – If you scroll down past the list of plugin/theme versions in the product detail view you’ll now see a widget with analytics data (for Kernl Analytics subscribers).
  • Load Testing Line Graph Performance – For large load testing data sets the line graph performance has been improved by disabling some of the flashier features of Chart.js
  • Analytics page now attempts to show the analytics for the first product in your product list upon entry.
  • NODE_ENV was not set to “production” on our app servers. It is now.
  • The transition to a fully managed load balancer is now complete.

Blog Posts

Thats it for this month!

Introducing Shareable WordPress Load Test Results

For quite awhile now Kernl has had the ability to throw some serious load at your WordPress site, but never a great way to share the results. Today that changes with the introduction of load test result sharing!

πŸŽ‰

Why would I share my load test results?

The main use case for sharing your load test results is showing your clients that their new WordPress site can handle the traffic load that they expect. For someone like a YouTube or Twitter celebrity this can be a very real problem. Larger organizations also would enjoy this peace of mind.

How do I get started?

Easy! Just click the share button next to the load test that you want to share. You’ll be presented with your sharing URL that can be copy and pasted into Facebook, Twitter, Slack, or Email!

Can I see some examples?

Yes!

As always, feel free to reach out to Jack if you have any questions!

Scaleway WordPress Performance Review

Test your own WordPress performance using Kernl’s WordPress Load Testing Service!

Scaleway is a European cloud service that provides an easy to use cloud infrastructure for a reasonable price. If you’ve ever used Digital Ocean they feel a lot like that but with fewer features. For this WordPress performance review I load tested 5 different server configurations (DEV1-S, DEV1-XL, GP1-XS, GP1-S, GP1-M) with caching both enabled and disabled using Kernl’s WordPress load testing service.

Server Configuration

As with most of my load tests I followed a very simple LEMP setup guide that left me with the following software versions:

  • NginxΒ 1.14
  • PHP FPMΒ 7.2
  • MariaDB 10.1
  • UbuntuΒ 18.04 LTS

Configurations were mostly default for all of these, with the exception Nginx where I bumped up the file upload max size. Each server was in the Scaleway Paris data center and load was generated from Digital Ocean’s Amsterdam data center.

Tests & Test Data

I performed 3 types of tests on Scaleway:

  • Small scale performance (cached & un-cached) – A 200 concurrent user test for 30 minutes. I did this test for DEV1-S, DEV1-XL, GP1-XS, GP1-S, GP1-M machine types.
  • Large scale performance (cached) – A 1000 concurrent user test for 45 minutes. I did this test only on a GP1-XS machine.
  • Reliability (un-cached) – A low volume long duration test. 25 concurrent users for 6 hours repeated a total of 3 times. This test was used to see what reliability under moderate load was like over the course of a day.

As with most of my cloud provider tests I used this blog’s content as my data source. This means that this test skews extremely read heavy.

Best Value

The results for this battery of tests were interesting.

Cost in Euro (ignore the $ in the graph)

The clear winner without cache enabled was the DEV1-S instance, which was nearly 5x cheaper than the closest competitor. But what does that actually mean? It doesn’t mean that DEV1-S is better than GP1-L, only that it is right-sized for this type of workload. What if we look at the data in another format?

In this bubble chart, the x axis is cost in euros and the y axis is max sustained requests. The cluster in the upper-left corner of this chart is the best value for this type of workload. There isn’t a right answer here, but you can select GP1-S if performance is more important than cost, or DEV1-XL if performance is important but not quite as important as cost. It is worth noting that if increased the volume of requests we would likely see this graph shift in dramatic ways.

To see the results that drove this graph, scroll all the way to bottom of this page and you’ll see an image gallery that has all the raw data.

Reliability

The reliability test was performed on a GP1-XS instance in three 6 hour increments over the course of 1 day. It was a low-volume test (25 concurrent users), but enough load to keep the box busy and to test how reliable Scaleway is over an 18 hour period. Over the course of 18 hours I sent 1.5 million requests to the GP1-XS instance.

23 req/s

As you can see the machine stayed consistently at 23 req/s over the course of 6 hours. The response time distribution was good as well.

98ms @ 99th percentile

99% of requests finished in 98ms or less. Solid performance over a 6 hour period.

For the 2nd 6 hour test things were mostly the same with 1 minor change.

23, 22, 23

You can see that for a 2.5 hour period we had some (minor) performance degradation. Why? Maybe a noisy neighbor? While not a huge deal here, in some workloads losing ~5% of your capacity could be quite problematic.

Even though our capacity was slightly lowered, our response time distribution at 99% remained consistent at 98ms.

Our 3rd 6 hour test was similar to the first, except this time with a ~30% reduction in the 99th percentile response time (98ms to 69ms).

Extremely consistent performance

As you can see from the graph above that for our entire 6 hour period performance remained the same.

SO. FAST.

As stated above, the response time for this 6 hour period was 30% faster. I believe that this is due to the time at which this test was run. It was started at ~3PM EDT which is roughly ~10PM in Paris, so most of this test happened when internet traffic in the region wasn’t very high.

High Volume Performance

The final test that I ran was against a GP1-XS instance, for 45 minutes, with 1000 concurrent users. The WordPress install was using caching. Results were fairly good!

~900 requests / second

While the GP1-XS instance was having a little trouble keeping up, it still served the vast majority of the requests without error and managed to do so at 900 requests/s. The response time distribution was equally impressive.

99% @ 190ms

Conclusions

Scaleway seems like a reasonable host if you need one in Europe, although their pricing seems expensive relative to other companies (Hetzner, Digital Ocean). I was a little concerned with the consistency of performance in the long-term test, but I don’t have any data for other providers to compare it against.

I’m not sure what the difference between Scaleway DEV and GP instances is besides price (maybe a better SLA on GP instances?), but the DEV instances seem like a much better value.

In short: Check out Scaleway if you need WordPress hosting in Europe.

Test your own WordPress performance using Kernl’s WordPress Load Testing Service!

Vultr Cloud Compute -vs- Dedicated -vs- Bare Metal WordPress Performance

Load test your own WordPress site with Kernl! Getting started is free!

In the world of cloud computing there are a lot of different options to choose from. Normally you only need to choose how big your instance will be (2 vCPUs or 4, 2GB RAM or 6), but some cloud compute providers are upping their game and providing an even wider array of options and instance types for you to choose from.

Vultr has 3 different types of compute instances:

  • Cloud Compute – You get your own virtual server, but it is sharing hardware resources with lots of friends. Noisy neighbors can definitely be a problem.
  • Dedicated – Dedicated servers, but virtualized. I (think) it is possible to run in to noisy neighbor problems in this situation.
  • Bare Metal – Dedicated servers and hardware. No hypervisor and no noisy neighbors taking up your resources.

In this article we’re going to see how a very basic WordPress install performs on the different types of Vultr compute instances. We’ll do so using Kernl’s WordPress Load Testing service.

The Test

As per usual with Kernl load tests I imported this blog’s content into each load testing environment. The load test skews extremely read heavy. If you have a site that is write heavy or a mix you may see different results.

Each test was performed for 1 hour with 2000 concurrent users generating load from London and New York to Vultr’s data center in New Jersey.

Configuration

For this test I used Vultr’s pre-built WordPress image with no caching. A lot of readers might say “But you can get much better performance using X or Y!”, and they would be right! But I’m not testing Apache vs Nginx performance, or W3 Total Cache vs WP Rocket, I’m testing Vultr hardware under load in a real world scenario. I simply want to know at the end of this article if Vultr Cloud Compute, Dedicated, or Bare Metal is better for WordPress hosting.

Test 1: Vultr Cloud Compute $10 / Month

The first test I performed was against the $10 per month Vultr Cloud Compute offering. As expected of a $10/month VPS performance wasn’t awesome, but it also wasn’t terrible.

All the red of red land

As you can see, lots of failed requests and only maintaining throughput of 16 req/s. Not unexpected with a single core and 1 GB of RAM. After all, I was throwing 2000 concurrent requests per second at the server. The response time distribution was similarly bad.

Bad, but could be a lot worse.

Overall, the results for the $10 VPS were as expected. This isn’t really an apples to apples comparison (we’ll get to that later), but I wanted to give you an idea of what basic VPS instance performance looks like.

Test 2: Vultr Cloud Compute $80 / Month

With this test we’re starting to get closer to the cost of bare metal and dedicated instances. This server had 6 CPUs and 16GB of RAM. Considerably more robust than the $10 server.

Lots of red, but also blue!!!

This graph tells a much different story than the previous test. Performance peaked at 169 req/s and then leveled off at 100 req/s. We still saw a lot of errors, but once again this isn’t unexpected. Honestly if you started to get this much traffic you would likely start breaking up WordPress into its components (file system, PHP + Nginx, MySQL) and start scaling horizontally.

Much Lower Response Time Distribution

The response time distribution was much better for this server as well. The upper end was just as bad as the cheaper box, but the 90% and below ranges were pretty solid for the amount of traffic that was being received.

Test 3: Vultr Bare Metal $120 / Month

The Vultr Bare Metal server was the instance I was most excited about testing. I’ve always had a soft spot for hardware and getting access to a bare metal server is pretty cool. For $120 per month (on sale, price will rise to $300/month eventually) you get 8 CPUs and 32GB of RAM. This is a pretty serious server.

Oooh, 200 req/s.

Lots of blue on this graph but also the expected amount of red. You can see that throwing 2 more non-virtual CPUs and 2X the RAM made a pretty big difference. We peaked at 200 req/s and then leveled out at 125 req/s. For reference that is 17.2 million requests per day.

πŸ™

The lower end of the response time distribution was solid, but the upper end wasn’t great at all. With all of those errors it isn’t surprising that this is the case.

Test 4: Vultr Dedicated $120 / Month

I honestly had a tough time figuring out why Vultr priced the bare metal and dedicated instances so close to each other. Dedicated is clearly inferior (far fewer CPUs and RAM) so why would anyone choose it? Anyway, let’s take a look at the graph.

πŸ’©πŸ’©πŸ’©πŸ’©πŸ’©

This test peaked at 100 req/s and then leveled off at around 70. I really would expect a lot better performance for this sort of money.

Also πŸ’©, but not as much πŸ’©.

Response time distribution was similar to the other boxes. With all the failures it tends to skew pretty hard in the wrong direction. I’m sure that there is a use case for these dedicated Vultr instances, but it definitely isn’t hosting a WordPress site.

Conclusions

With all of this data it was pretty easy to graph which of these is the best value.

Value was calculated by taking the cost per month and dividing it by the maximum number of requests. Based on the performance we saw above the Vultr Cloud Compute instances seem like your best value for WordPress hosting. For WordPress hosting it looks like Vultr Bare Metal and Dedicated instances aren’t a great choice. As mentioned above, there are likely use cases where they are a good choice though (maybe workloads that require very consistent performance).

As with all of these tests, your mileage may vary! I highly recommend that you run load tests on any new host that you use to get an idea of what sort of performance you can expect.

Load test your own WordPress site with Kernl! Getting started is free!