Having recently been exposed to several presentations and articles aggressively touting the superiority of feature toggles over feature branches, I decided to examine the two techniques a little more closely.
Defining terms: feature branch and feature toggle
A feature branch is a fork in the history of a codebase allowing concurrent development of different features on different version control system (VCS) branches. A feature toggle is a conditional statement that enables or disables a particular feature. All feature development is conducted on mainline, and when features are ready to be deployed the toggle is changed to enable the feature.
The argument against feature branches
There are two fundamental reasons not to use feature branches.
- If the development effort reflected in a feature branch isn’t merged into the mainline until it’s been released to production, then arguably the team is not practicing continuous integration. Without continuous integration, the team cannot practice continuous delivery.
- Two teams working concurrently on two different feature branches may inadvertently make significant conflicting changes to the same areas of the codebase. The team that releases to production second draws the booby prize, and deals with the tedium or stress (depending on time constraints) of the nightmare merge that follows.
The argument for feature toggles
All feature development is done on mainline, which addresses both of the problems highlighted above. Every commit from every team working on every feature is represented in the single source of truth, so continuous integration is happening. Similarly, since every commit is made on mainline, any conflicts between code changes made by different teams are caught when the second team commits their changes, rather than after a protracted stream of development.
How to feature branch effectively
Feature branches have the potential to cause trouble if they’re not used with circumspection, but they are not inherently evil. The biggest mistake you can make when deciding on how to use branches is trying to define “one branching strategy to rule them all”. You wouldn’t (hopefully) try to define a single design pattern to apply when solving every software engineering problem – this is because each piece of software you write fulfils different requirements in a different context in a different way. The same is true for your branching strategy. A catalogue of branching patterns is beyond the scope of this article, but Brad Appleton, Stephen P. Berczuk, Ralph Cabrera, and Robert Orenstein have published the excellent Streamed Lines: Branching Patterns for Parallel Software Development online. Each project you work on may be subject to a different set of forces, so this pattern catalogue will help you resolve these forces in favour of an optimal branching strategy for your particular project.
One of the patterns described by Appleton et al is the Integration Line. From their website: An integration line is a codeline that is primarily for the purpose of integration (merges) from other codelines and branches. Adopting this pattern within a branching strategy allows developers to work in feature branches, practice continuous integration, and catch conflicts early. This doesn’t mean that you should always use an integration branch, but if it’s appropriate for your needs it does solve the same problems as feature toggles.
If developers edit the same line of code in two or more branches of the codebase, when those branches are merged the version control system (VCS) will identify a conflict. This is not a problem that working on the mainline solves, but in this case the conflict alert is brought forward to the point at which the second developer commits their code, rather than during a much later merge. If you’re finding that teams working on different features (or even the same feature) are experiencing a high number of conflicts when they commit to the VCS, this may be symptomatic of one or more of the following underlying problems.
- The application is poorly designed, or has suffered from entropy (so-called “technical debt“) over time.
- There is no cohesive view of the domain, or the domain has not been modelled effectively in code.
- There are multiple logical models of the domain, but bounded contexts have not been applied.
Although identifying conflicts early is good, it’s better to avoid them entirely if possible. The Manifesto for Agile Software Development favours Individuals and interactions over processes and tools, so if we’re being Agile to one degree or another, communication is one means of overcoming all of these problems.
Make sure that all teams have a view of all features currently under development, and which other teams are working on them. Also make sure that there are active lines of communication between all teams, and that all team members are happy to use them.
Constantly editing domain code does not speak well of adherence to the open/closed principle, so lots of conflicts between feature branches may be highlighting a broader issue than the use of feature branches. Check that the team is using ubiquitous language, and ensure that teams working on features with differing perspectives on the domain are not competing to assert their model in common code space.
If a developer encounters code that they feel they could have written better, it’s tempting for them to refactor it not because it provides any benefit, but because it offends their design sense. If this behaviour (refactor-rage) is left unchecked for any length of time, not only will the velocity of the developer drop significantly, but since the developer’s code surface increases so does the likelihood of conflicts with other developers’ work. Rotating pairs means a greater transparency for everyone’s code, and a quiet word from a colleague can often prevent this practice from occurring. If you’re not pair programming, ensure that the team is at least practicing regular peer review, for the same reason.
Risks associated with feature toggles
Although feature toggles do remove some of the complexity associated with developing multiple features concurrently, there are down sides. When you deploy code that’s been developed in a feature branch, you know that all of the bits that you are deploying have been confirmed as production quality, and tested exhaustively. When you deploy from a codebase using feature toggles, you’re deploying the current development bits from all features currently being worked on. The only guarantee you have for features still under development is that the latest build has passed, and although if you are practicing continuous delivery this goes a long way, it’s unlikely that capacity testing, penetration testing or exploratory testing will have all been performed on all features. Whilst there should be no paths of execution that lead to development bits, this is still a risk that the use of feature toggles introduces – a risk that you will need to sell to your chronically risk-averse operations colleagues.
Even if you do sell feature toggles to operations, you still need to manage the toggles themselves. There needs to be a single place to turn the toggle on or off. This could be hard coded, but it’s more likely that you’ll want to manage this in a configuration setting somewhere. You will need to ensure that this configuration is managed in a consistent manner, and also that the conditional statements are removed from your codebase after deployment. In addition, if a feature is cancelled you will need remove every trace of the feature from the mainline as opposed to merely deleting a branch in the VCS.
Mitigating risk with feature toggles
Depending on the experience and maturity of your team, releasing development bits to a production environment may be a manageable risk. However, if you are producing desktop applications this may not be an option. If your application runs on the JVM or the CLR, then it’s trivial for a user to decompile the application to get a snapshot of all features currently under development, and this is almost never going to be a good idea.
In this case you can still use feature toggles, but implemented in a different way. If your compiler supports preprocessor directives, then you can implement build-time instead of run-time feature toggles. This means that you only release the features that have been built, and you only build the features that have been approved for release. Of course, since everything is a trade-off when building IT systems you don’t get this ability for free. Since you are now only building a subset of features, you are no longer practicing continuous integration. If you take this route, you will need to maintain more than one deployment pipeline for the same application – perhaps one to identify release candidates, and another to produce release candidates. You still have the benefits of feature toggles, but you are now building up complexity that you sought to remove by moving away from feature branches.
There is a third way that you can implement feature toggles; at deploy-time. With this technique you can develop all features concurrently on mainline, maintain a single deployment pipeline for all features, and choose which features to deploy. Although this is the cleanest way to implement feature toggles, not many application architectures will support this approach without extensive re-engineering. Even if they do, it requires a high degree of discipline from all developers.
How to decide: feature branch or feature toggle
If you already use feature branches and don’t experience much merge pain, then it’s not a good idea to expend a lot of effort moving to feature toggles. This is not a simple exercise, and requires design input from an experienced developer or architect who knows both the system and the team well. Don’t forget that feature toggles are the solution to a problem, and if you’re not encountering a problem you don’t need a solution.
If you’re producing desktop applications – especially those released to the general public as opposed to a more controlled corporate environment – then you need to think carefully about how you would implement feature toggles, even if you are experiencing merge pain. Releasing development bits is risky if the user has access to them, so run-time feature toggles may not be appropriate in this case. There are other solutions to this problem, for example building composable applications using a framework like Prism (for .NET).
If you’re suffering merge pain and producing websites or enterprise applications where the release bits are access controlled, then feature toggles may be worth assessing. If you have many teams working on different features concurrently though, it may be difficult to trial this approach on one project without impacting all of the other teams. However, it could end up being very costly to try and move all teams out of feature branches and into mainline at one time. If you find yourself in this situation, it may be a good idea to consult with all members of the team in order to perform a feasibility study. If the result of the study indicates that moving away from feature branches is a practical possibility, then produce a plan describing how to achieve this.
If you have a small number of teams working on different web application features concurrently and you’re experiencing merge pain, then if may be easier to move to run-time feature toggles. Even if you do this and even if it solves your problem, you should examine your system structure and development processes – as detailed above, the problems you’re encountering may be evidence of poor design or software engineering practices.
Feature toggles are certainly an interesting concept, and seem to mesh well with the principle of building composable applications. However, they are not without their pitfalls, so if you are thinking about trying them out, make sure that they are the most appropriate solution to the actual problem that you’re trying to solve.
Originally published at Four Nines.