Greetings Members.
It’s time for this week’s dev diary.
I’m one of the engineers on the Community Platform Team. Having been with giffgaff for 26 months, this will likely be my first and last post on the Community as a giffgaff employee. But I have written one before, it just did not move past the draft stage.
@tariqthedev , who joined a week before we went live with Flarum, will be taking the reins of the community team and I’m sure, ride into success. As the outgoing Technical Lead, it was technically my choice to recommend Flarum. So, if it works for you, I have to say it is because of the Grace of God, the amazing team members (@ianmorland, @shivanshuit914, @jatap, @deeple, @bad_at_pool, Rupert, @alex_w and now @annesolano, @tariqthedev and @andrewdyer) and support from the business we have had all along. If it doesn’t, it’s likely because of something I have done or decided incorrectly - so I offer my apologies.
Back to what the dev diary is really about. There has been a lot going on this week, including the upgrade to beta 10, but we thought it would be good to talk about 2 things, specifically:
- What is Trunk Based Development and what technically goes into a release of the giffgaff Community Platform
- Progress on ElasticSearch - @andrewdyer mentioned this in last week’s dev diary.
Trunk based Development and a giffgaff Release
If you know something about software engineering and Git, head over to https://trunkbaseddevelopment.com/ to learn more about Trunk Based Development. Otherwise here is a simplified attempt to explain it.
At some point, controlling development of software became linked with the idea of a tree trunk and it’s branches. If you like mathematics and graph theory is your thing, have a look at this explanation: https://en.wikipedia.org/wiki/Version_control#Graph_structure.
In a nutshell, the ‘trunk’ of the tree is that part of the code which you see running on the platform right now, that is, the approved version. A ‘branch’ may be an offshoot of the trunk that a particular engineer is working on, and once that work is complete, the branch is literally merged into the trunk and becomes part of the Community Platform.
When there are many engineers working on many branches, the code goes in all different directions and bringing it all together becomes difficult. The big risk is that you may introduce and discover new problems when a branch is merged into master. The bigger the branch, the longer the divergence, the bigger the problem and harder to solve. We prefer to have smaller problems - they’re less complicated and easier to test for.
Trunk Based Development addresses this issue and says that everybody should always write code in the trunk ‘version’ all the time. This means we can find problems sooner and write tests that run automatically each time new code comes in to ensure we don’t reproduce any known problems. A lot of the work we have been doing is about getting this automated testing bit right. As a short summary, there are lots of different types of tests we run:
Unit tests - these literally test parts of our code - it’s basically statement like ‘if x goes in, make sure y always comes out of these 12 lines of code’.
Load tests - to ensure the Community Platform can handle appropriate traffic with all the new code we’ve written.
Functional Tests - to ensure that all the different components of the Community work - the code, the database, the cache clusters, etc.
End to End tests - these tests actually simulate a member’s experience on the Community Platform and performs various actions - if any of these actions worked before and does not now, we know something has gone wrong.
Snapshot tests - We take snapshots of various parts of the UI and try to get a pixel perfect match against the ones we have after every bit of new code that goes into our ‘trunk’.
Before we develop anything, we try and write one or more of the above tests and once they pass, the code is automatically released into the Community Platform. These tests are, therefore, essential. To give you an idea of how little and frequent these tests and releases can be, we sometimes release new code to the Platform 2-3 times an hour.
ElasticSearch
Implementing ElasticSearch in Flarum is progressing well. The implementation should speed up almost all pages that load discussions and posts, and it should allow us to start implementing much smarter and responsive searches. Essentially, this comprises the following bits of work:
- Pass any new searchable content created in Flarum into ElasticSearch as it is created.
- Overwrite default search, tag and discussion functionality to use ElasticSearch instead of the relational database.
- Import all existing data into ElasticSearch. This is our most challenging bit, because this has to happen in a repeatable manner as we iteratively enhance this functionality. If we run a simple import, it can take up to 60 hours. We’re trying to get this down to 3 hours, and once we are able to do this, we can begin rolling out ElasticSearch to various parts of the community platform.
As soon as we are able to get 3 to manageable time, we’ll be rolling out our first iteration of ElasticSearch.
Stay tuned, and thank you for reading.
