When hosting your WordPress site on DigitalOcean, storing your static assets nearby makes a lot of sense. Since the advent of DigitalOcean Spaces in 2017 this has become a much less arduous process. In this post we’ll go through step-by-step how to host your media (and static css/js if you want) in a DigitalOcean Space.
Setting up a DigitalOcean Space
Before you get started you need to set up your DigitalOcean space. To do that, log in to DigitalOcean and click the “Create” dropdown. From there, choose the “Space” option.
Create a DigitalOcean Space
Select your region, ignore the CDN (for now. You should do this later though), leave the file listing as restricted, and then pick a name. It can be anything, but something descriptive is usually best. In our case, we’re going to use kernl-blog-space.
Once your space is created, you need to make a few minor changes in it’s settings to allow for file uploading. Under CORS Configuration we need to add the domain where files are going to be uploaded from. In our case, we added:
Origin – http://wploadtest.xyz (this is my test domain)
Methods – I selected all of them, because I want the plugin that will be managing the files to be able to make all the modifications that it needs.
The other two options I left blank/default.
DigitalOcean Spaces CORs Configuration
Finally, you’ll need access keys. To generate those, look in the left menu on DigitalOcean, go down to the “Account” section, and click “API”. Now scroll down and click “Generate New Key”, pick a name, then click the check mark. Copy the access and secret key somewhere. You’ll need it for later.
WordPress Configuration for DigitalOcean Spaces
The next step is to set up WordPress to send and server media from your DigitalOcean space. First up, you need to install the WP Offload Media Lite plugin from the WordPress.org repository.
WP Offload Media Lite plugin
Once installed and activated, go to Settings -> Offload Media Lite. Select DigitalOcean Spaces, choose whether to store keys in the database or not, add your keys, and then click “Next”.
Offload Media Lite Configuration
The next step is selecting the region where your space is located and then copying the name of that space into the “bucket” field. Once you’ve done that click “Save Bucket Settings”.
Testing it Out
Now let’s test it out! Go to your media library and upload a few images. Once uploaded, click into the image detail view, and checkout out the source.
Image served from DigitalOcean Spaces!
And just like that, we’re done! Images are now served from DigitalOcean spaces. A next step you should take it to configure to the CDN so your media is globally distributed, but this is a great start.
Kernl had another great month, with some bug fixes, blog entries, and a few new features.
Most importantly we’re announcing a limited beta of our new global update CDN! With our new global CDN powered by Vercel, every Kernl update request is cached at edge nodes around the world. This means <= 50ms response times for updates globally.
Reach out to jack@kernl.us if you’d like to give it a try!
Features, Bugs, and Performance
Docs – Added more descriptions to the “Understanding Graphs” portion of the Kernl documentation.
Analytics Date Selection – The flow for selecting dates and products in analytics has changed. You can now only select dates in the range that the selected product has data.
Examples – The Kernl API Examples repo has been updated to include a license management example.
WHMCS License Validation – Kernl now supports WHMCS license validation for plugins and themes. This integration works similarly to our other 3rd party license validation integrations.
Performance – We added a few database indexes to our load testing database. In situations where a lot of load tests existed performance was abysmal.
Tutorial – On the main dashboard page of Kernl there is now video tutorials for getting started with plugin and/or theme updates.
License Expiration Date Bug – There was a bug in Kernl license management where date editing didn’t work correctly when the saved date started with a leading zero.
License Domain Restriction Bug – An edge case existed where license validation would fail with a server error if domain restrictions are set but no domain was passed in to be validated.
Download Increment Bug – Incrementing download count was being handled in a very naive way. It was originally developed when Kernl wasn’t expected to reach the scale it has. This has been resolved by using the $inc operator in MongoDB instead of a find -> increment -> upsert.
Load Test Infrastructure Bar – The progress bar when you create a load test didn’t have a very good default state (it looked like nothing was happening). Now when below 3% complete, the bar will show 3% so you can actually see that things are progressing.
Site Health Edit – You can now edit your site health entries.
Envato License Validation Bug – Due to the deprecation of an API endpoint we were using, users of our Envato license validation need to regenerate their Envato access token that is saved with Kernl.
Envato License Tied to ID – You can now verify that an Envato license that is being used is tied to a specific plugin or theme.
For a few years now the DigitalOcean Marketplace WordPress image has been available for installation on their droplets. If you want to use WordPress but don’t want to go through the trouble of configuring it, this is a great place to get started.
Ease of Installation
The DigitalOcean Marketplace WordPress image installation isn’t one click, but it is easy. It takes a few steps, including:
Sign up for DigitalOcean or log in.
Click “Create” then “Droplet”.
Select the image
Pick your droplet size
Select your region for deployment
Add your SSH Keys (optional)
Name your VM
Create the droplet
Overall, the UX for creating the droplet with the WordPress image installed on it is super easy to follow. DigitalOcean has done a great job removing friction from the process.
What’s Included?
Compared the to Vultr One-Click WordPress installation the DigitalOcean image has less included out of the box, but this isn’t necessarily a bad thing. The DigitalOcean marketplace WordPress image is configured with the following software:
Ubuntu 18.04 LTS
Apache 2.4
MySQL 8
PHP 7.4
Fail2ban 0.10
Postfix 3.3
Certbot 0.26
Most of this is fairly self-explanatory with the exception of Fail2ban and Certbot. Fail2ban scans Apache logs for malicious patterns and bans those IP addresses. Certbot provides free SSL certificates for your site.
In addition to installing the packages above, the DigitalOcean Marketplace WordPress makes some common sense security changes to help keep you secure.
Enables UFW firewall to allow SSH, HTTP, and HTTPS.
Sets the MySQL root password and runs mysql_secure_installation.
Creates an initial WordPress config file and sets up the salt keys.
Disables XML-RPC to prevent DDOS and brute force attacks.
Modifies PHP settings to increase max file size and execution time.
Enables Apache rewrite module for WordPress permalinks
For the vast majority of WordPress sites, this will be plenty of power for you.
DigitalOcean Marketplace WordPress Performance
So you’ve installed this WordPress image from the DigitalOcean Marketplace and you want to see what kind of punishment it can take. Cool. Kernl’s got your back.
Requests / Failures per second
The requests per second performance of the baseline image isn’t terrible, but it isn’t great either. Right around 14 requests/s the wheels fall off and error rates spike. Eventually the application failed to response to any requests.
Average / Median Response time
You can see the response time average and median stay right around 150ms until the server becomes overwhelmed. The numbers here are misleading though because this chart only takes in to account successful requests.
Response time distribution
The final chart is the response time distribution. Of all requests, 50% finished in under 500ms. 100% finished in 5000ms. The 5000ms mark is interesting because that’s the timeout that was configured by DigitalOcean on the WordPress image. Just because requests finished, doesn’t mean they were successful.
DigitalOcean Marketplace WordPress Performance with Caching
Most people run some sort of caching with their WordPress site. For this test we used W3 Total Cache with all caching features enabled.
Requests / Failures Per Second
You can see that the throughput of the site increased quite a lot once caching was enabled. Failures didn’t start to see and uptick until around 60 requests/s and then stayed relatively consistent at around 10 failures/s with requests leveling out at 130 req/s. To be clear, 130 req/s is a lot of traffic.
Average / Median Response Times
The average response time leveled off at around 110ms, with the median coming in at around 80ms. The difference between the two is interesting because it means we had some pretty big outliers.
Response time distribution
The response time distribution gives an interesting take on the average/median response time graph. We can see that 99% of our requests finished in under 250ms, but that last 1% took up to 9000ms. That explains the difference between the median and the average.
Conclusions
In general the performance of the $5 droplet with the DigitalOcean Marketplace WordPress image on it was acceptable. Once caching was enabled it became quite good. The best part about this image is that it takes care of basic security things for you and doesn’t try to guess what you want.
In short, give this image a try. It might save you some time.
If you want to host your own copy of WordPress but don’t want to go through all the trouble of setting it up, it might be worth your time to explore using “one-click” installs from some of the major cloud providers. In this article, we’re going to explore the Vultr One-Click WordPress install and see what it’s all about.
Ease of Installation
Before anything else you need to get a WordPress box spun up on Vultr. Overall, the experience was pretty easy:
Click the “Application” tab and then select WordPress.
Select your server size (25GB SSD $5/month for this test)
Add your SSH key
Fill out your host name
Click “Deploy Now”
So obviously this isn’t a single click to install, but it is a lot easier than setting up and configuring a WordPress installation from scratch.
Whats included?
Aside from the obvious stuff (WordPress, MySQL, etc) which we’ll cover in the next section, what else is included with your One-Click Vultr WordPress installation?
XHProf – XHProf is a tool for profiling PHP applications. This could be very helpful if you find your site getting real slow after installing a plugin or theme. It could help you determine the source of the slowness.
PHPMyAdmin – The classic MySQL database administration tool for the web. If you haven’t used it before, its a great way to browse your database and make changes.
Maldet – Maldet is a Linux malware detection tool that runs periodically on your system to see if it has been infected. This is disabled by default, but can easily be enabled with the directions included with your new server.
Cockpit – Cockpit is a web based interface for your server. You can easily see how your server is configured, what resource usage looks like, and update packages all without even SSHing into your server.
What’s WordPress Running On?
The Vultr One-Click WordPress system that we decided to use had the following hardware and software:
Hardware
1 vCPU
1GB RAM
25GB SSD
1TB Bandwidth
Software
Ubuntu 18.04 LTS
Nginx
PHP 7.2
MySQL 5.7
Overall the software is a pretty standard LEMP setup and the hardware is as performant as you would expect for $5 per month. It’s also worth noting that the installation script for this one click install might need some updating because it installs the previous version of WordPress.
Performance Out of the Box
So you’ve spun up your new Vultr One-Click WordPress server, but what sort of performance can you get out of it? Using Kernl’s WordPress Load Testing we can easily see the performance that this $5 per month server gives you.
For both this test and the cached test (next section) we attempted to have 200 concurrent users browse the WordPress site for 30 minutes. The traffic was generated from Digital Ocean’s NYC3 datacenter.
Requests / Failures per second
As you can see we get up to about 20 requests per second before the server starts returning failures. For a $5 machine this isn’t bad at all. 20 req/s over the course of 24 hours is ~1.7 million requests.
Response Times
Initially response times are pretty good, but as resource contention on the server increases the average and median response times start to go up. Even when under full load, the pages returned in under 5 seconds. Not terrible, but you definitely aren’t winning friends with that sort of performance.
Response Time Distribution
The response time distribution is mostly what you would expect after seeing the error rate and the average response time. 99% of requests return in under 5.5 seconds. In general you want to see your 99th and 100th percentiles be low, but given the duress the server was under this isn’t surprising at all.
Overall the system performed well and was configured well enough to handle a small-ish load test.
Performance with Caching
The test above is almost silly in that most people willdefinitelybe using some form of caching on their WordPress site. For a more realistic test, I enabled W3 Total Cache on the WordPress installation. There wasn’t any Memcached or Redis installation available, so all cache settings were tuned to “Disk (enhanced)”.
Requests / Failures per second
As you can see the difference in performance with caching enabled is quite good. We didn’t see any errors at all for the duration of the test and the maximum concurrent requests per second peaked at 160. Over a 24 hour period, that would be 13.8 million requests.
Average / Median Response Time
The response time over the course of the test was also quite good. Initially it hovered around 35ms on average, but as resource contention on the server increased it leveled out at around 140ms. Not too bad for $5 and nearly zero configuration.
Response Time Distribution
Finally we come to the response time distribution. You can see here that 50% of requests finished in under 100ms, 99% finished in under 700ms, and all requests finished by 1800ms. These are good numbers.
Final Thoughts
If you need to host a WordPress site and don’t want to much around with configuration, the Vultr One-Click WordPress service is pretty good. In the long-term you’ll still need to know a little about system administration to apply updates, but it is definitely a great place to start and gets you a lot of the way to a fully functioning WordPress site.
With all the uncertainty in the world right now due to COVID-19, we are intentionally keeping things boring at Kernl. Bug fixes, new docs, and increased capacity are all we did this month.
Stay safe out there everyone!
Features, Bugs, Updates
NEW DOCUMENTATION! – After 5 years we finally moved to a real documentation generator! In the process of moving docs to the new system, all of the docs were updated and checked for accuracy. Check it out at https://docs.kernl.us.
Site Health – You can now change you wp-json root. Some people change this for security reasons. Site Health needs it for some of the more advanced functionality.
Site Health Timezone Bugs – Timezones are hard and our Site Health service wasn’t using them correctly.
Analytics VM Changes – The Kernl Analytics service was starting to get bogged down with increased traffic. It now has 2 vCPUs and 2GB of RAM (up from 1 vCPU and 1GB).
Analytics Bug – We were logging analytics data for customers that aren’t paying for it. This is no longer the case.
Plugin/Theme Git Status – The plugin and theme list pages were displaying inconsistent results for the Git status of plugins/themes. There was an error in our Mongo query that populated this data which has been resolved.
Load Testing Remain Resources – The remaining resources page can take a while to load sometimes because we have to call out to external services. An indeterminate loading spinner was added to let customers know that it’s still working.
System Updates – All operating system level packages have been upgraded to their latest available version.
After 5 months of development and 2 months of closed beta, we are happy to announce the public availability of Kernl’s WordPress Site Health service.
What is it?
Kernl’s WordPress Site Health service allows you to understand changes in your site’s health. But what does that mean? We monitor your response times, time to first byte (TTFB), Google Lighthouse scores, plugin changes (added, updated, or deleted), as well as CPU, disk, and memory usage.
Using the data above, you can see when the health of your site started to degrade and why. Maybe it was when a plugin got updated or installed? Perhaps you started to run out of memory? With Kernl’s WordPress Site Health you can figure this out at a glance.
Common Use Cases
While this tool is not a monitoring service, the value it can provide to you and your customers is immense.
Identify the source of performance regressions – The data gathered by our WordPress Site Health service makes it easy to identify the source of performance regressions. Because we capture machine-level information (CPU, Memory, Disk) and when plugins were added/updated/removed, it’s easy to draw a line between response time or TTFB increases and changes in your WordPress site & host.
Monitor changes in Google Lighthouse scores – Google Lighthouse is a series of tests against your site that checks for modern web best practices. Kernl runs Lighthouse against your site once per day and graphs the results. If you see a score go down, you can look at the other data we gather to try and tie it back to a change in your site.
WordPress Site Health Pricing & Limits
Kernl’s WordPress Site Health is included in every Kernl plan. The only thing that changes are the number of sites you can add and the check frequency.
Plan
# of Site
Check Frequency
Solo
2
10 minutes
Agency
10
5 minutes
Unlimited
Unlimited
1 minute
Sign Up Now
You can sign up now for free. Kernl comes with a 30 day free trial and no credit card is required.
I hope everyone had a great February! We didn’t too much feature development this month, but there was a lot of bug fixing and performance improvements, so let’s dive in!
Feature, Bugs, and Performance
Node.js – Kernl is now on Node.js 12.16.1. This release was all about security fixes.
Load Testing Machine Provisioning – We weren’t calculating the correct number of machines to provision on DigitalOcean. This lead to some serious over-provisioning when running load tests. This has been resolved, which means more customers can run more load tests at the same time.
Load Testing Secondary Node Behavior – Kernl uses Locust under the covers to run our WordPress Load Testing service. The Locust primary node has an argument called “–expect-slaves”. It tells Locust “Don’t start the load test until at this this many secondary nodes have connected.”. We weren’t calculating this number correctly which led to some weird behavior. This is now resolved so load tests should start in every situation now.
Easy Digital Downloads Domains – Kernl wasn’t passing the domain along to EDD. We now do this, which allows you to restrict updates to specific domains while using EDD.
Load Testing Snapshots – Kernl used to build up each load testing machine from the ground up every time a load test was started. We now start from a snapshot that gets us 50% of the way there. This has improved load test start times (especially on large tests) by an average of 30%.
GitHub Authorization Changes – The GitHub API is changing how it handles authorization headers. We’ve update Kernl to handle this change, so we’ll be good going forward when GitHub deprecates the old method.
High Traffic Endpoint Audit – We did an audit of our high-traffic API endpoints and cleaned some things up. Slight performance improvements were had (1%-2%), but mostly the improvements have been in code readability and comprehension.
GitLab Deployment Issues – In a recent release of GitLab they changed the required fields when asking for an access token via a refresh token. This broke all GitLab deployments for Kernl for a few days while we tracked down the issue. This has since been resolved.
Load Testing Unit & Integration Tests – When our load testing service was launched we weren’t sure if it was going to be successful. We’ve proven that it is a worthwhile feature, so now we’re focusing on reliability. We’re in the process of adding a suite of unit and integration tests around this functionality.
Choosing between Memcached or Redis for your WordPress cache is a tough decision. Not because they have vastly different performance profiles (they don’t), but because either choice is a good one depending on your needs. In this post we’re going to explore the differences between Redis and Memcached, how they perform for WordPress, and a lot of different non-performance things you should consider when making your choice.
What is Memcached?
Memcached is an open-source, high performance, distributed memory object caching system. What does that mean? It means you can store a bunch of strings in memory and access them really fast. From a WordPress perspective, it means that using a caching plugin like W3 Total Cache we can store the results of the complicated SQL queries that WordPress does in memory and have them available instantly.
What is Redis?
Redis is an open source in-memory store that can be used as a cache or a message broker. It’s a bit different then Memcached because you get a lot more out of the box with it. For example, Redis has built in replication, transactions, disk persistence, and provides high availability and partitioning. All those features means that managing WordPress can be a little harder to do, but not much harder. Especially if you just need to use it as a cache.
Performance
Both Redis and Memcached have excellent performance. They’re both used by some of the largest websites in the world and are fully ingrained in the Fortune 500. Given that all things are not created equal, let’s see how they perform with a read-heavy WordPress site (this blog).
The Setup
The load tests are performed against the DigitalOcean WordPress Marketplace image with either Redis or Memcached installed alongside of it. The machines have 2 vCPUs, 2 GB RAM and live in DigitalOcean’s SFO2 (San Francisco) data center.
The load test configuration:
500 concurrent users
2 users / second ramp up
45 minute test ( ran twice )
Traffic comes from Digital Ocean’s NYC3 data center.
The content of the load test is a copy of this blog.
Baseline Performance (No Cache)
The baseline performance for WordPress with no cache isn’t great.
50 requests / second with LOTS of failures
The response time also isn’t great. A little over 2 seconds on average.
~2 seconds response time on average.
Redis Performance
Once we install Redis and configure W3 Total Cache to use it, the number of requests that we can handle increases substantially.
300 requests per second
The requests remain steady at around 300 per second and no failures are recorded. The response time also improves quite a bit.
Average ~475ms response time
475ms isn’t bad at all. That’s 4 times faster response times then without any caching at all.
Memcached Performance
With Memcached installed and W3 Total Cache configured to use it, we see some excellent performance.
425 requests per second
In this situation, Memcached performs even better then Redis with 425 req/s versus Redis’ 300 req/s. Response time improvements are similar.
115ms response time
The Memcached response time is almost 3 times faster than the Redis response time. In general, the results where Memcached is faster than Redis are surprising. In most benchmarks Redis is equal or faster than Memcached, so it’s likely a configuration problem.
Other Considerations
When deciding what cache to use with your WordPress setup, there are a few other considerations your should be looking at:
Ease of setup – As you can see from the performance results above, Memcached has better performance out of the box. Knowing what I know about Redis this is likely a configuration issue, but the fact that I could get that level of performance with no configuration from Memcached is a good data point.
3rd Party Hosting – Do you really want to manage your own Redis or Memcached server? If you don’t, you’ll want to look at the landscape of 3rd party providers. Redis has a robust provider ecosystem. Memcached’s is a little less robust.
Persistence – Do you need your cache to survive a reboot? This is important if the cost of re-populating your cache is too high for your system. If you do need persistence, Redis is your best option.
High Availability– If you need high availability of your caching cluster, Redis is the clear winner here. Memcached can be made to operate this way, but Redis has it baked in to the core of the application.
If you’d like to see the full results of the load testing runs on Kernl, see the links below.
Load testing is fun. Breaking things is fun. Breaking WordPress with load testing is even more fun. But in the era of highly scaleable WordPress hosting solutions, can we even break WordPress anymore? Oh yes, yes we can. The Crucible Challenge can.
Crucible Challenge
The Crucible WordPress Performance challenge is a deceptively simple test inspired by the poor ops teams that have to handle traffic from Super Bowl advertisements. Given a WordPress site with consistent content and URL mappings:
Handle 50,000 (@ 500 per second ramp up) concurrent authenticated users for 2 hours with load test generators in New York, London, Amsterdam, Singapore, Bangalore, San Francisco, Toronto and Frankfurt.
Have an error rate below 0.1%.
Average response time should be below 800ms.
Median response time should be below 700ms.
99th percentile response time should be below 800ms.
Half way through the test, you must flush your cache.
500 per second ramp up does not give any time for warming your cache. It’s like your site getting hit in the face with a sledgehammer.
Low error rate doesn’t give you a lot of room for problems.
Keeping your response times below 1000ms with traffic coming from all over the world presents interesting problems.
Flushing your cache after the load test is already in progress shows us you have a good cache invalidation strategy and can handle dog-piling.
Example Results
To give everyone an idea of what results might look like, I took at $160 / month CPU Optimized Digital Ocean droplet, put Open LiteSpeed + WordPress on it and ran The Crucible against it.
18,000 requests per second
I only ran the test for 2 minutes, but as you can see it started to max out near the 18,000 req/s mark, with failures in 1,800 failures/s area.
Can your service beat this? Want to find out? Drop an email to jack@kernl.us and we can get a test run scheduled.
January was a pretty great month for Kernl. We got a lot of bug fixes, some performance improvements, and even a new beta feature out the door. Let’s dive in!
WordPress Site Health Beta
This month we released a beta of our WordPress Site Health service. The goal of this service is to help you determine where performance problems are and why they are happening. The dashboard below gives you a high level view of how performance looks on each of your sites.
Kernl WordPress Site Health
Once you click in to any site you see data for the last 7 days.
Kernl WordPress Site Health Detail Page
The data here helps you diagnose performance issues. Using the plugin changes panel you can tie performance issues back to adding/removing/updating of plugins. Google Lighthouse scores show trends on you site’s performance, usability, SEO and more over time.
If you’d like the join the beta, send an email to jack@kernl.us.
Bugs & Performance
Improved performance of plugin/theme list pages – The API calls to /api/v1/plugins and /api/v1/themes were incredibly inefficient. They hadn’t really been touched since Kernl’s inception and needed some love. After doing some query optimization and stripping out unnecessary data, the payload and response time were reduced by a factor of 10. The worst case example was response size going from ~750KB to ~75KB and response time from ~4000ms to ~250ms.
Server resource increases – Kernl’s main Node.js application servers have been running with 1 vCPU and 1GB of RAM for about the last 2 years. Lately we’ve seen some resource exhaustion and decided it was time to upgrade. The new app servers now run with 2 vCPUs and 2GB of RAM.
Server disk space / inode exhaustion – The process of building plugins and themes can use up a lot of space and file system resources. We weren’t doing a great job of cleaning those resources up periodically which could cause some performance issue. We now clean up all temporary files once a day which should prevent this from happening anymore.
Easy Digital Download License Validation Error – There was a bug in EDD license validation where the source system wouldn’t send back valid JSON. This would break license validation instead of handling the error gracefully.
Profile page autocomplete – If you had form auto-completion on it would sometimes cause the profile page to reset your password. We’ve disabled autocomplete on this form to resolve the issue.
Theme tiles not showing correct build status – During the course of our performance improvement work we noticed that theme tiles were not showing the correct build status. This has now been resolved.
