A vision of continuous integration
A little while ago I read an article by Thierry de Pauw on what he termed “non-blocking, continuous code-review” (I came across this via a Dave Farley video on the same topic). It’s been bouncing around my head ever since; everything in it seems immediately like a good idea but also—and more importantly—most of it is backed up by evidence. In this post I just want to rehash some of those ideas, combine them with generally accepted CI best practice to come up with a vision for how I believe software engineering teams can most effectively work. This is not something I am reporting on from experience, instead, it is the hashing out of a plan as a starting point to try and iterate upon.
Goals
Our goals are broad and lofty: software engineering excellence. We want to achieve CI/CD, focussing first on CI. To break that down slightly, some of the things we want to achieve are:
- Less (zero?) time spent dealing with merge and rebase conflicts
- Fewer bugs
- Higher quality architecture
- Improved team dynamics
- Improved feedback loop with clients
I’m not going to dive into exploring the existing evidence that CI/CD can deliver these things here—that evidence is extremely important but this article will be long enough as it is. I highly recommend the book Accelerate by By Nicole Forsgren, Jez Humble and Gene Kim for an exploration of this.
Terms
It’s worth defining a couple of terms before we continue:
Trunk-based development or Continuous Integration is the practice of a team developing on a single branch, without using feature branches and pull requests.
Continuous delivery is the practice of keeping the project releasable all the time. If anything causes that to be untrue, fixing it is the highest priority. It uses pipelines with automated tests and other checks to ensure this.
Continuous deployment automates the deployment at the end of this pipeline.
Here we’re primarily looking at the first of these.
Trunk-based development or Continuous Integration is the practice of a team developing on a single branch, without using feature branches and pull requests.
Continuous delivery is the practice of keeping the project releasable all the time. If anything causes that to be untrue, fixing it is the highest priority. It uses pipelines with automated tests and other checks to ensure this.
Continuous deployment automates the deployment at the end of this pipeline.
Here we’re primarily looking at the first of these.
Pull Requests and their alternatives
Pull requests are fantastic for FOSS projects where code is being submitted by unknown contributors, but when used in professional teams tend to simply slow things down. Long running branches cause merge conflicts and can introduce bugs and architectural issues. We use pull requests as a way to try and prevent low quality code or bugs making their way into the codebase. So what is the alternative?
If we can implement pair or mob programming, we have continuous code review and there is no need for review to happen discretely when the task is completed. We can then completely remove the need for that phase of a task’s lifecycle and the use of pull requests.
If for some reason (reluctance of the team, dislike of the process or other restrictions), it is decided not to broadly pair or mob, we can utilise a mixture of 0% code reviews and the continuous code reviews proposed in the article linked above. To summarise that approach: commits are made directly to trunk, and reviewed soon after. If changes are necessary, further commits implement those changes.
In this process, the lifecycle of a ticket becomes:
If we can implement pair or mob programming, we have continuous code review and there is no need for review to happen discretely when the task is completed. We can then completely remove the need for that phase of a task’s lifecycle and the use of pull requests.
If for some reason (reluctance of the team, dislike of the process or other restrictions), it is decided not to broadly pair or mob, we can utilise a mixture of 0% code reviews and the continuous code reviews proposed in the article linked above. To summarise that approach: commits are made directly to trunk, and reviewed soon after. If changes are necessary, further commits implement those changes.
In this process, the lifecycle of a ticket becomes:
- The ticket is written, outlining the problem that needs to be solved
- The team discusses the ticket (a “0% code review”) and how it should be technically implemented
- As the engineer working on the ticket pushes commits, these are reviewed soon after by other team members
- User-facing changes are hidden behind feature flags
- Tests are used to catch bugs and regressions
- Feature flags can be removed at a later date to avoid littering the codebase
Review frequency
Review should happen as frequently as possible. I would suggest finding a rhythm where unreviewed commits are picked up at set points during the day—ideally natural breakpoints such as the start of the day and around lunch. I would also suggest these are on the calendar and performed as a team. This has a few benefits:
- The whole team sees all the code coming into the codebase and has a better understanding of the application as a whole
- It is less likely for review to get pushed back when it is on the calendar
- By reviewing as a team the author is also present and can answer questions immediately.
Development workflow and git habits
Achieving a CI workflow is likely to necessitate some changes to the individual developer’s workflow. Primarily, as GeePaw Hill puts it, “many more much smaller steps”. To further paraphrase him, work in very small steps, each of which is a step in the right direction. It is not necessary that each step improves the application, but it should not make anything worse or break tests. As a concrete example, an individual step (and thus commit) could be as simple as adding an attribute to a model, a database migration or adding the frontend interface to logic which has already been completed in a previous commit. Working in this way allows us to keep pushing to trunk more frequently and running together in parallel rather than pulling in different directions.
In my own personal git workflow, I will still commit as a “save point” occasionally without pushing, but every change I make following that is added to the same commit with git commit --amend until that step is complete. The commit can then be pushed, and onto the next step.
In my own personal git workflow, I will still commit as a “save point” occasionally without pushing, but every change I make following that is added to the same commit with git commit --amend until that step is complete. The commit can then be pushed, and onto the next step.
Tests
In this paradigm tests are clearly extremely important. Tests (perhaps with the exception of slower system tests) should be regularly run locally and all tests should be run as part of the CI pipeline. If the CI pipeline breaks, fixing it becomes the immediate priority of the person who’s commit broke it.
Flaky tests are unacceptable and similarly become the priority to fix. Flaky tests are, as a colleague put it, not “intermittent failures” but “intermittent passes”, and that’s not good enough. Flaky tests will slow you down and halt deploys.
I would argue TDD is generally a very good thing and will contribute toward well-designed architecture, but it is not completely necessary. The tests themselves definitely are though, and should come along with the relevant commits.
Flaky tests are unacceptable and similarly become the priority to fix. Flaky tests are, as a colleague put it, not “intermittent failures” but “intermittent passes”, and that’s not good enough. Flaky tests will slow you down and halt deploys.
I would argue TDD is generally a very good thing and will contribute toward well-designed architecture, but it is not completely necessary. The tests themselves definitely are though, and should come along with the relevant commits.
Production vs. Staging
In an ideal world this happens on a single branch which is deployed directly to production. I believe almost all of the benefits can be gained without stepping quite to this conclusion however, by still having a second “production” branch which trunk is merged into prior to production deploys. I would suggest that ideally this should be seen as a step along the journey to true continuous deployment however.
(As an aside, an equivalent approach would be to target deploys against tagged points on trunk, rather than maintaining a separate branch).
(As an aside, an equivalent approach would be to target deploys against tagged points on trunk, rather than maintaining a separate branch).
Tooling
The main drawback of this approach is the severe lack of tooling around reviewing code in this way. Watch this space—I’m working on it.
Metrics
It’s important when making these changes we attempt to record their effectiveness. I’ll be exploring the metrics we can use as my next step.
Other resources
Beyond those linked directly above, the following provided inspiration for these thoughts:
- Clare Sudberry’s talk Continuous Integration (that’s not what they meant).
- Modern Software Engineering by Dave Farley
- The vast library of video’s on Dave Farley’s YouTube channel, Continuous Delivery
Updated at 2023-10-19 16:08 |
2023-10-19 11:10