Hi everyone,
You might want to go and grab yourself a big cup of tea - this is going to be a big post with lots of technical detail and might take a while to digest. I’ll start with some background information, and then I’ll dive into the specifics of the 2 hour outage we had on Monday, 2nd September.
On Tuesday 27th August we migrated from our old community site of nearly 10 years to a brand new community site. The migration itself went well, despite taking longer than initially expected, however we experienced some issues in the 2-3 hours immediately following throwing the switch to make the new community available to everyone. Thankfully, we were able to resolve these quickly and the community has been very stable since then, with a couple of notable exceptions.
Evening of 28th August – 10 minutes with broken CSS / JavaScript
The first of the exceptions was a brief blip around 21:30 on the 28th August. For a period of 7 or 8 minutes, the community was unable to load our CSS and JavaScript asset files that are central to displaying the content correctly. This was the result of 4 consecutive small deployments of new code by one of our developers who was fixing a few small display bugs.
On further investigation, we found that the problem was not caused by any of the code changes themselves. Instead, it was a product of the way the new platform bootstraps JavaScript and CSS, and how Network File Storage (NFS) works. Successive deployments created what we call a ‘race condition’. We hadn’t seen anything similar before we launched the community, and this was one of those cases of there being no substitute for “real world” experience - we needed the right amount of consecutive deployments and a decent amount of traffic to create this condition.
Our expectation had been that code deployments would be effectively invisible to members, seamlessly being rolled out. We can accomplish this as the site is hosted in the cloud on multiple ‘nodes’ (servers, pretty much) using multiple Kubernetes Pods (or Docker Containers, if you’ve heard of them) that can be removed and created as necessary. During a code deployment, each pod should be sequentially destroyed and recreated. So while one node is unavailable and being updated, the other pods keep the community running, until within a few minutes all of the nodes have been updated.
Having seen that the seamless deployments weren’t working correctly, over the following days we worked on investigating the root cause of the problems and then putting in place changes designed to prevent any repetition of the issue. While we worked on that, we paused any further code releases to the live site while retaining the ability to deploy code to our staging environment. This was important, as it allowed us to continue working on both a fix for the issue we’d seen as well as other updates for the community in general.
Monday 2nd September – 2 hours of problems
On the morning of Monday, 2nd September, we became aware of a critical defect on the community that needed to be addressed urgently. The fix was simple, and we were also confident that we had code ready that would address the problem from the previous week. So, we were keen to get the code released and see how it performed.
Unfortunately, instead of a smooth and seamless release, we suffered 2 hours of the community site being largely unusable. So what happened?
At its simplest, it was the same problem we had on the 28th – except our first attempt to fix it made the problem (replication latency in the Network File System) worse. This time the race condition did not resolve itself, but it had the positive side effect of neatly highlighting exactly what we needed to do to fix it forever.
In more detail, the CSS and JavaScript files are deleted and automatically recreated whenever we release new code. CSS files are the files that govern how a page looks, which was why when you tried to load a page during the issue, you could have seen pages that looked very different between page loads. The mistake was that we centralised all of the assets files (images, CSS, JavaScript) in a single location that would then serve the multiple pods that I mentioned above. This single location is a Network File System, which means it essentially stores multiple copies of the data and these are synchronised over a network. This works perfectly fine for the images files because they do not change multiple times a second, but became a problem for the CSS and JavaScript because of the way it is created when the new community starts up.
What we missed was that each pod, as it received traffic after a code release, was independently attempting to update the JavaScript and CSS files. However, because there was so much traffic on the new pods, even before the CSS and JavaScript written by one pod was replicated across the Network File System it was already being overwritten as pod two came active. We have over a dozen pods, and under the heavy traffic at that time, the Network File System (which replicates at least 3 times), was not able to complete replication and kept getting overwritten, leading to a situation where we never had a CSS or JavaScript file fully written to serve the Community. Ultimately, you might call this race condition a nasty loop where the pods were fighting with each other to delete and recreate the files. On the day, we were able to manually resolve the race condition and force the platform to stop deleting the files.
Tuesday 10th September - Permanent fix implemented
We have now permanently resolved the problem and been able to make a number of code releases seamlessly since the fix was implemented. We did this by implementing a solution where the CSS and JavaScript files are generated for each node individually and not on the shared Network File System (so we eliminated the replication latency). We have also added some automated testing into our code release process that will automatically stop a release if it detects any failure for the CSS and JavaScript files to be generated correctly in our pre-release version of the platform. These tests don’t just test the existence of the files, but they simulate more concurrent users than the community has at any given time to ensure no race conditions are created.
As a result of the fixes put in place, we have now been able to make a number of code releases with no discernible interruption to the functionality of the community platform. To prove this, we have made over 15 deployments since the fix was introduced on Tuesday morning - and no-one ran into any trouble. This means that we will now be able to continue with our plans to iteratively develop the platform based on your feedback. We’ll share more on those plans next week.
If you have any questions about any of the above, please do raise them here. Either I, or one of the team, will do our best to answer them.
