You try not to about it too much while you get settled in and start pulling down the code to build it locally. It compiles, but there are a number of failing tests. You notice a number of other details about the code; domain logic is spread throughout, there is bleed-over between the different layers of abstraction, and A LOT of untested code. Code duplication abounds, and some of the tests are flaky.
You check the build board again, and notice that not only is it still red, but someone has pushed on top with more changes. When you ask one of the other developers about it they chuckle and reply that it has been red for a while, but it's OK, it's just some of the tests. No one seems too bothered by this. You start wondering if you've walked into some parallel universe where a red build is acceptable, or even expected. How did it get this way? Doesn't this bother anyone? You realize you have your work cut out for you.
Broken Window Theory
So what happened? How did the state of the project get to this point? How can a build get into a red state and stay that way? Unfortunately it is entirely too easy for this to happen. It only takes one bad changeset and zero developers who care about fixing it. This happens in software development with maddening frequency, and is an example of the Broken Window Theory.Broken Window Theory goes like this:
Given a building with one or more broken windows which are not quickly repaired, the tendency is for more windows to be broken and for other acts of vandalism to occur. People notice that no one cares about the building, and there is no social pressure to prevent the vandalism. The building quickly falls into disrepair and stays that way.
In software development lots of little changes can contribute to creating an environment which tolerates a continually broken build. The addition of code which is difficult to test leads to fewer tests. Flaky tests can degrade confidence in the build ("Oh yeah, that build is red because of known flaky test, go ahead and push anyway!"). Just plain broken functionality is allowed to be pushed, and a general lack of design and maintenance can cause the codebase to be become chaotic and difficult to work with. Maybe there are developers on the team who simply don't know how to write tests, or how to refactor properly.
All of this together can make it disheartening to try to do the right thing, it becomes easy to fall into the trap of "Well, this is just how the codebase is, we should just learn how to live with it." Obviously this is not the answer.
Fixing Broken Windows
So the windows have been broken and the build is red. How do you fix the situation? How do you get the build back to a green state and keep it that way? The answer is simple, if not easy: get the team to care about the build. Obviously this is easier said than done in many cases. Often the team has been beaten down by the long-standing degradation of the build, so simply shouting about the problems in the codebase won't solve anything. Concrete steps must be taken.Stop Further Damage
To begin with, cordon off the building, draw a line in the sand, stop the bleeding (pick your metaphor here, there are a lot of them), but whatever you do stop the code from getting any worse than it already is. |
| Take your lead from this guy. |
Be prepared to put the breaks on and play the bad guy, because you're about the rock the boat and upset a lot of people who have become comfortable with The Way Things Are.
Your first order of business should be to get the build back to green as quickly as possible. If that means no one pushes code until that happens then so be it. Implement an Evergreen Policy which states that if a build goes red it is either fixed immediately, or the changeset is backed out.
The build is an indication of the health of the project, and ultimately it should tell you whether you are ready to deliver or not, so it should become a priority to get it green and keep it green. The code should compile and all tests should pass every single time.
Make "Done" Include a Healthy Build
For a given unit of work (user story, task, whatever you want to call it) there should be a Definition of Done. This definition should describe the requirements to be met before that unit of work can be considered complete, and the next unit of work is begun. Whatever those requirements currently are, they should be updated to include a clean bill of health for the build. Appropriate tests should be added and the entire build should complete successfully. Nobody gets to pick up another piece of work until this happens. Refuse to allow breaking changes into the build.Increase Confidence
By now you've hopefully gotten the build green, and laid out a plan to keep it green. Even so, there may be tests which you can't always trust. These tests might flap occasionally, leaving developers unsure if they've broken something or not. If you can't trust the build you can't be sure of the quality of the software you're building. Investigate the root cause of the flapping tests and address it as quickly as possible. In the meantime take steps to fix the tests themselves as well. Maybe they need to be rewritten, or moved into another, more stable layer. Maybe there are timing issues which can solved by increasing the timeout values. Maybe they should just be eliminated, as flapping tests are useless in terms of confidence in the build. Whatever the case, do whatever you have to create a reliable and consistent build.Spread the Pain
Don't try to take on the world on your own. Instead, enlist everyone else to your cause.
Make everyone responsible for keeping the build green. Chances are good you're going to need backing from the technical leadership to get everyone in line, as behavior doesn't change overnight and some developers will need an incentive to change. Maybe this means a rotating responsibility in the beginning, but the goal is to make everyone responsible for the entire codebase, and at the very least responsible for the code they are pushing.
No comments:
Post a Comment