In my previous post on Team Foundation Version Control some of the terminology surrounding branching was introduced. If some of the terminology in this post is not familiar to you then please refer to that post or, by all means post a comment.

Branching is the primary means by which development teams can support concurrent development using TFVC, and while the need for branching is often debated, the fact is that it is a powerful tool in a configuration managers arsenal.

In the life of a software product there are a number of requirements that may trigger the use of branching, some of the ones that I have come across are:

• Releasing patches for systems already in production.
• Preparing interim releases for testing and user evaluation.
• Working on more than one major version of the same code-base.

As I work through each of these triggers I will be sure to refer to some of the concerns expressed by Jim and Matt in the comments of my second post in this branching series.

Specifically Jim asserted that branching is a real pain in the but when it comes to merging and that developing on a single branch, committing to the head and using continuous integration is the way to go. Matt offered the advice that development on a single branch with a large team is all but impossible but also pointed out that the difficulting of merging a branch is directly proportional to how long that branch has existed.

Patching with Branches

Because discussion of branching can muddy the methodology/practices waters I am going to start with a branching requirement which I consider to be constant, regardless of what set of software engineering practices or development methodologies you are using.

Patching for the purposes of this discussion is defined as the process of changing the implementation of an application or solution after it has been released. Depending on whether you are in a shrink wrap scenario you may call it RTM or “in production”.

The problem with issues that require patching is that the development team becomes aware of them some time after active development on the code-base has resumed.

In this scenario you don’t want to patch and ship from the head of the trunk because not only are you getting the patch, but you are also going to suck in all the miscellaneous code changes that have taken place as you march towards the next major revision and typically you don’t want to risk introducing more bugs or incomplete features. The reality is that you have no choice but to use branching.

For single trunk development like Jim advocates it is important to be able to identify the point at which code was released to manufacturing or to production in order to support patching.

In TFVC there are multiple ways to identify a point from which to create a branch, but when you are talking about creating a branch after a release you will typically branch on a specific date, changeset or label. Of those three changeset or label are probably the most reliable/exact.

Using a label is good because it means you have flagged a particular set of items at a particular version as being “the release” but if you are on top of your build management identifying a release based on a specific changeset can work just as well (hint, make sure you leverage the build reports from Team Build to keep track of what you release and what changesets go into that release).

Once you have identified your release point and created the branch you will be well on your way to creating the branch structure below.

One important thing to note about the branch structure is that it differs from the feature development scenarios that I outlined in my second post. The key difference here is that no forward integration takes place before reverse integrating.

The reason for not forward integrating should be obvious. If you forward integrated from the head of the trunk you would get all the changes to source files that you were trying to avoid when creating the branch in the first place.

In addition to not forward integrating, you need to be careful when reverse integrating into the trunk (to make sure the bug is fixed in future releases). It is sometimes possible that through code restructuring that a bug has been removed or that the patch that was created cannot be applied to the head of the trunk successfully, at which point a second patch will need to be created – although this change can and probably should be made to the head of the trunk branch.

Once a patching branch is created there is no need to branch again for subsequent patches to that same release, further fixes can be made to the head of the patching branch without any ill effects, in fact, if you did branch again from the release point you run the risk of having bug regressions because you didn’t take into account previous patches to source files.

One of the software engineering practices that Jim likes to use (and so do I for that matter) is Continuous Integration. By adopting Team Foundation Server you gain the ability to use the Team Build feature. Team Build makes it dead easy to setup specific types of builds which include execution of build verification tests and code coverage analysis.

Fortunately Team Build is branch aware so you should be able to set up a build with the same ease that you did the first one. The only thing that you will need to watch out for is any customisations that you make to the TFSProj.build will need to be replicated across.

Out of the box, Team Build does not support CI, but it can be made to through the TFS extensibility model by hooking up to receive check-in notifications and then communicating back to TFS to initiate a build (which will then be farmed out to a designated build server).

Potential Problems with Patching Branches

In the interests of full disclosure we need to discuss some of the potential problems with patching branches and patching in general (see, I am not completely one-eyed about branching). One common scenario, especially for shrink-wrap product vendors is the need to support patching of more than one version of a released product at a time and even though tackling this product is of major interest to shrink-wrap vendors it can effect enterprise developers as well.

Basically, once you get two or more releases out in the wild you have the potential need to patch both of those versions independently of each other. You therefore need to maintain two patching branches, one for the first release, and one for the second release.

The issue arrises when you have a bug found in either of the releases and you need to propogate patches for that bug onto the patching branch of the other release. To get some context on the probelm refer to the diagram below.

In this scenario we have two versions of the software released and we identify an issue in the first release after the second release has shipped. We apply our normal process of creating the patch on the version one patching branch and then reverse integrating that onto the trunk.

Unfortunately, because we have shipped version two and it has its own patching branch we actually need to not only reverse integrate onto the trunk, but also forward integrate onto the version two patching branch. The danger here is that unless we are extremely careful we may forward integrate incomplete features into the version two patching branch – and because of the merging rules that TFS imposes (refer to my previous post) there is no way to forward integrate without going through the trunk first.

The reverse can also happen. You may find a bug in the second release of the software, in this case you create the patch on the version two patching branch, reverse integrate to the trunk and then forward integrate onto the version one patching branch.

The problem of potentially forward integrating from here is that potential scope for differences is only going to increase over time as the code-base diverges from earlier releases. Fortunately the application of a little process can help here by ensuring that before you patch, you find the earliest supported release that the bug exists in and only ever move forward through the code-base timeline.

A Simpler Approach?

The problem with always working on the single trunk and having to maintain a patch branch for each release is that it virtually ensures that in order to propogate a bug fix across multiple versions of the software that you are going to have to perform a series of reverse and forward integrations.

Another option for mangaging patches is to restructure the way you work with branches. In this scenario you maintain a branch for each major version of the software and after it is released to manufacturing or production the existing branch becomes the patching branch.

Under this regime subsequent major releases occur on a branch created at the release point for the previous major release and only forward integrations are used for patching (this assumes that the process of finding the earliest supported version that exhibits a bug is applied).

This approach also exploits the merging constraints that TFVC imposes and allows you to merge from one branch to another without having to go through intermediate branches because they form a single string of branches. This can be useful when for some reason a version was released but was never used and there is no need to support it anymore.

Concurrent Major Release Development

Configuruation Managers as a geek sub-species have their own unique set of religious arguments that come up during the course of projects and whilst attending geek dinners. One of the religious debates that seems to have spilt over into the general developer population is whether concurrent development on major releases is a good thing(tm).

Personally I find the whole argument somewhat academic because as soon as you admit to patching code you are admitting to a limited form of concurrent development. The extent to which you indulge in the practice of concurrent development is really just the twist of a knob.

Structurally, the branching strategy outlined in the diagram above is the basis for concurrent major release development and it is another reason I prefer it to single trunk development with patching branches – simply because when you eventually do need to support development of major releases concurrently you won’t need to change your patching strategy.

Concurrent major release development occurs when at some point before you release an upcoming version you branch off and commence development of the version “Next + 1”.

As you can see the simple approach to patching on multiple branches outlined above really lays the groundwork for a concurrent development strategy, but that is not to say that it is without its challenges. Firstly, it can be hard for one team to effectively manage two major release projects at once, and secondly high levels of activites on parallel branches can lead to a rapidly diverging code-base which can cause integration nightmares down the track.

Managing Concurrent Major Release Development

Once your development team and the configuration manager has a handle the raw mechanics of using branching in Team Foundation Version Control it them falls to the project manager to effectively manage two concurrent development efforts on the same project.

Probably one of the most important things to do is distinguish the difference between one version of a product with multiple milestones and two distinct versions of the same product. One of the easiest ways to do this is to write up the list of features and get the user representative to draw the line where they think their current level of funding will run out. Everything above the line is managed in one project, and everything below the line in another.

Fortunately Team Foundation Server makes it easy to create a Team Project to store the project tracking information for each logical project and I recommend that for every version of the product you intend to build, that you create a seperate Team Project.

One of the often overlooked steps in Team Project creation is deciding how to setup the version control repository. By default each project gets a nice clean area for developers to commit source files and other version controlled items into. But the screen that supports the provision of that area also enables the creator to create an area with a branch from an existing project’s source files.

Using this feature creates an appropriate level of isolation between the work items for each version but also helps build the branching structure described above.

Avoiding Integration Hell

Matt’s observation about the pain of merging branches being directly proportional to the time since when the branch was created is right on the money, although I would add its not too bad for patching since source code changes are generally limited and localised.

Avoiding integration hell takes effort and no matter what you do there is always going to be a certain amount of pain, but then again, thats why they pay us the big bucks! In my opinion, the key is performing a forward-integration on a regular basis (assuming the structure above) to ensure that features and enhancements being made to an earlier branch are being brought forward.

Of course the joy of developing the next version of a product is that you get to address some of the design decisions made in earlier versions so it is quite concievable that merge compatibility will be lost at some point. One risk projects do run is branching the code-base too early, especially before the major structural issues in the source branch have been worked out – if you do branch too early it may not be feasible to forward integrate and the branch will need to be abandoned.

Coordination between the teams working on each version will be required and to make this work I would suggest that at least one or more of the team members are wired up to receive check-in alerts from the earlier Team Project and that continuity is provided by bringing some of the earlier Team Project members forward into the new development effort.

Preparing for Interim Releases

I was recently involved in a enterprise project where the product of our third iteration would actually be deployed out to end users to help address a pressing business need. In the shrink-wrap product development space there is also a need to ship BETA versions of the software to get user feedback to ensure that its going to satisfy user requirements.

The problem is, as a project grows the likely hood that all the effort to round out a complete feature is going to fit snugly in befor the interim ship date diminishes. This is another scenario where branching can help.

By using a branch a development team can prepare a stabilised (and feature limited) release of a product without significantly disrupting the main game development.

One thing to notice about this is that as you approach the eventual ship date of the code the need to branch off for an interim release should be reduced and your release candidate (if you have one) should not be off a special branch created for that interim release, but the main branch for the version.

Coming Up Next

Phew! What a marathon post, hopefully I’ve addressed some of Jim and Matt’s concerns about branching, or at the very least given them some food for thought. In my next post (which should be shorter) I will look at how branching can be used to reduce contention on source files, avoid code-base contamination and control conflict resolution.