With all the uncertainty in the world right now due to COVID-19, we are intentionally keeping things boring at Kernl. Bug fixes, new docs, and increased capacity are all we did this month.
Stay safe out there everyone!
Features, Bugs, Updates
NEW DOCUMENTATION! – After 5 years we finally moved to a real documentation generator! In the process of moving docs to the new system, all of the docs were updated and checked for accuracy. Check it out at https://docs.kernl.us.
Site Health – You can now change you wp-json root. Some people change this for security reasons. Site Health needs it for some of the more advanced functionality.
Site Health Timezone Bugs – Timezones are hard and our Site Health service wasn’t using them correctly.
Analytics VM Changes – The Kernl Analytics service was starting to get bogged down with increased traffic. It now has 2 vCPUs and 2GB of RAM (up from 1 vCPU and 1GB).
Analytics Bug – We were logging analytics data for customers that aren’t paying for it. This is no longer the case.
Plugin/Theme Git Status – The plugin and theme list pages were displaying inconsistent results for the Git status of plugins/themes. There was an error in our Mongo query that populated this data which has been resolved.
Load Testing Remain Resources – The remaining resources page can take a while to load sometimes because we have to call out to external services. An indeterminate loading spinner was added to let customers know that it’s still working.
System Updates – All operating system level packages have been upgraded to their latest available version.
I hope everyone had a great February! We didn’t too much feature development this month, but there was a lot of bug fixing and performance improvements, so let’s dive in!
Feature, Bugs, and Performance
Node.js – Kernl is now on Node.js 12.16.1. This release was all about security fixes.
Load Testing Machine Provisioning – We weren’t calculating the correct number of machines to provision on DigitalOcean. This lead to some serious over-provisioning when running load tests. This has been resolved, which means more customers can run more load tests at the same time.
Load Testing Secondary Node Behavior – Kernl uses Locust under the covers to run our WordPress Load Testing service. The Locust primary node has an argument called “–expect-slaves”. It tells Locust “Don’t start the load test until at this this many secondary nodes have connected.”. We weren’t calculating this number correctly which led to some weird behavior. This is now resolved so load tests should start in every situation now.
Easy Digital Downloads Domains – Kernl wasn’t passing the domain along to EDD. We now do this, which allows you to restrict updates to specific domains while using EDD.
Load Testing Snapshots – Kernl used to build up each load testing machine from the ground up every time a load test was started. We now start from a snapshot that gets us 50% of the way there. This has improved load test start times (especially on large tests) by an average of 30%.
GitHub Authorization Changes – The GitHub API is changing how it handles authorization headers. We’ve update Kernl to handle this change, so we’ll be good going forward when GitHub deprecates the old method.
High Traffic Endpoint Audit – We did an audit of our high-traffic API endpoints and cleaned some things up. Slight performance improvements were had (1%-2%), but mostly the improvements have been in code readability and comprehension.
GitLab Deployment Issues – In a recent release of GitLab they changed the required fields when asking for an access token via a refresh token. This broke all GitLab deployments for Kernl for a few days while we tracked down the issue. This has since been resolved.
Load Testing Unit & Integration Tests – When our load testing service was launched we weren’t sure if it was going to be successful. We’ve proven that it is a worthwhile feature, so now we’re focusing on reliability. We’re in the process of adding a suite of unit and integration tests around this functionality.
January was a pretty great month for Kernl. We got a lot of bug fixes, some performance improvements, and even a new beta feature out the door. Let’s dive in!
WordPress Site Health Beta
This month we released a beta of our WordPress Site Health service. The goal of this service is to help you determine where performance problems are and why they are happening. The dashboard below gives you a high level view of how performance looks on each of your sites.
Once you click in to any site you see data for the last 7 days.
The data here helps you diagnose performance issues. Using the plugin changes panel you can tie performance issues back to adding/removing/updating of plugins. Google Lighthouse scores show trends on you site’s performance, usability, SEO and more over time.
Improved performance of plugin/theme list pages – The API calls to /api/v1/plugins and /api/v1/themes were incredibly inefficient. They hadn’t really been touched since Kernl’s inception and needed some love. After doing some query optimization and stripping out unnecessary data, the payload and response time were reduced by a factor of 10. The worst case example was response size going from ~750KB to ~75KB and response time from ~4000ms to ~250ms.
Server resource increases – Kernl’s main Node.js application servers have been running with 1 vCPU and 1GB of RAM for about the last 2 years. Lately we’ve seen some resource exhaustion and decided it was time to upgrade. The new app servers now run with 2 vCPUs and 2GB of RAM.
Server disk space / inode exhaustion – The process of building plugins and themes can use up a lot of space and file system resources. We weren’t doing a great job of cleaning those resources up periodically which could cause some performance issue. We now clean up all temporary files once a day which should prevent this from happening anymore.
Easy Digital Download License Validation Error – There was a bug in EDD license validation where the source system wouldn’t send back valid JSON. This would break license validation instead of handling the error gracefully.
Profile page autocomplete – If you had form auto-completion on it would sometimes cause the profile page to reset your password. We’ve disabled autocomplete on this form to resolve the issue.
Theme tiles not showing correct build status – During the course of our performance improvement work we noticed that theme tiles were not showing the correct build status. This has now been resolved.
Welcome to the end of 2019! I hope that everyone has had as good a year as Kernl. Let’s dive in to the final update of 2019 to see what’s new.
Features, Bug Fixes, & Misc.
Improved License Management Search – License management now includes improved search functionality. The previous search functionality was flaky (at best) and not very discoverable. Search is now a first-class citizen, includes free-text search, and should greatly improve the overall usability of Kernl’s WordPress license management.
Load Testing Unit & Integration Tests – When we created the WordPress load testing service it was an experiment. Now that we have proved the viability of the service it’s time to work on stability and overall platform longevity. This month’s focus was on the authorization framework that our load testing service uses.
JS Bundle Size Reduction – Over the past year Kernl’s JS bundle size for our web app grew to over 2MB. We spent some time this month figuring out why and making changes to reduce it. In the end we were able to reduce the bundle size by over 50% down to 1.1MB.
Bug: Inconsistent Webhook & Deploy Key Behavior – After a few customer reported incidents with the automatic webhook and deploy key behavior, we discovered that Kernl wasn’t deleting local references to remote keys and hooks. If you have some issues with deploy keys or webhooks please contact us and we can help resolve the data inconsistency issues that this caused.
Node.js Upgrade to 12.14.0 – This month we upgrade all of our servers to use the latest LTS version of Node. This is includes stability and performance improvements.
I hope everyone has had a great summer (or winter if you are in the southern hemisphere)! Over the past 2 months we’ve gotten a lot great stuff done, so let’s dive in.
Features & Infrastructure
Kernl has upgraded from Mongo 3.x to Mongo 4.2 with WiredTiger. We get improved performance and the latest features with this change.
Our Redis instance has moved to DigitalOcean along with the rest of our infrastructure. Prior to this we were using managed host that lived outside the NYC3 data center. Response times decreased ~50ms or so with this change.
The high traffic plugin and theme update check endpoints had a round of performance tuning done. Resource consumption was lowered in meaningful way.
🔥🔥🔥Kernl Analytics Active Plugins🔥🔥🔥- Kernl Analytics will now track what plugins are most active across your install-base. You only need to be signed up for Kernl Analytics and use the latest plugin_update_check.php file to get this new feature.
Our MongoDB database has been moved to DigitalOcean NYC3. Prior to this we were hosting on Compose.io outside of the datacenter. Originally this decision was made because managing databases is tough, but the quality of hosting at Compose has gone done significantly in the past year. With this change we shaved ~150ms off of response times.
Some tweaks were made to our network firewalls to make them easier to manage. Thanks DigitalOcean!
Load testing machines would fail to provision if the API call to DigitalOcean failed. This has been resolved.
The load testing master node would fail to start sometimes if secondary nodes failed to connect. The threshold for starting tests has been lowered so that this won’t happen anymore.
If a credit card expires and the invoice payment fails, the account isn’t marked as paid when a new card is added and a successful payment happens.
When switching between themes/plugins in Kernl Analytics the domain data wasn’t reloading with the new plugin/theme.
Thanks to a customer bug report and code snippet, the plugin_update_check.php no longer sends headers before the license check fails.
Hello everyone! Kernl got some pretty interesting updates this month, so let’s dive in!
Average & Median Response Times for Load Tests – Kernl’s WordPress load testing services has always had this data available, we just never surfaced it in a way that was easy to consume. You’ll now see a new tab when you run a load test with information about average and median response times during the course of your test.
Progress Bar for Load Test Initialization – When you create a load test you will now see a progress bar that indicates how far in the process of instantiating your infrastructure we are. Prior to this update it was easy to think that the process had stalled out or broke.
Bug Fixes & Other
Load test site ownership verification would fail to certain types of HTML minification. This has been resolve.
Our internal analytics services has been refactors to be a singleton. This reduced each app server’s memory footprint by 5MB.
The public license validate endpoint now has a 10 second cache on it. This helps us deal with any sort of burst traffic from license validations.
All packages have been update on our servers.
Thats it for June! If you have any questions reach out to email@example.com.
There was a lots of work on Kernl’s WordPress Load Testing this month, so lets dive in and learn about it!
Features & Bug Fixes
Reliability Load Testing – Have you ever wondered how a WordPress host performs over an extended period of time? Kernl’s new reliability testing allows you to answer that question. You can now run low volume (25 user) load tests for up to 30 hours.
Download Full Load Test Data – You are now able to download all the data from your load test. This is the full, non-sampled, data set that Kernl produces during a load testing run.
Sampled Load Test Data – Kernl’s WordPress load testing service can create a lot of data, so for data sets over a certain size we now sample the data. This helps all of charts load faster and improve your experience.
March was a great month for Kernl! We did some blogging, a bunch of infrastructure work, and a little bit of unplanned work due to API deprecation at BitBucket.
Features & Updates
Resource Starvation (Load Testing) – We’ve decreased the number of users per machine that Kernl WordPress Load Testing uses. This helps prevent resource starvation on the load generation servers.
Pre-Configured Load Tests – Kernl now has 4 different pre-configured load tests to make testing your site’s performance even easier!
BitBucket API Update – BitBucket API calls now use Oauth2 and their v2 endpoints. The v1 endpoints are going away in April.
Analytics – All of the features available in Kernl Analytics Agency plan have been rolled into one single plan at the same $10/month price point as the small plan. We also increased data retention to 365 days to make our comparison tool more useful.
Load Test Working Indicator – Load tests will now show a “working” indicator when in a state before the load test has started but after you have submitted your request to start a load test.
MongoDb Driver – The MongoDB driver that we use has been upgraded for better connection retry handling.
Updates – Kernl Analytics and Kernl WordPress Load Testing have been upgraded to Node.js 10.15.3 and have had all of their packages updated.
License Management Widget – The license management widget was throwing an error when no error was present. This has been resolved.