In an earlier piece from this series, Jeff Howell gave a good overview of the term quality as it applies to software development. I want to drill into just one of the items he mentioned and expand on it: continuous integration (henceforth abbreviated CI to save some bytes).
If you aren't familiar with CI, these days you can explore the concept by trying out one of many canned products that are built to do CI for your team. With some variances, they all cover the same basic ground: running software builds automatically, whether triggered by time of day or by the event of someone committing changes to source control. Automatically running tests. Running inspections of source code (static analysis). And then there are thousands of plugins to do any number of other ancillary tasks.
So how did these kinds of products come into existence, and why?
Back In the Day
I didn't pay much attention to the arising of CI when it happened around me many years ago at a startup. I ran builds locally after making changes to ensure I wouldn't cause a build breakage, and that was about it. Every once in a while a breakage would be caused by near-simultaneous checkins of some incompatible code by more than one developer, but I became aware of the situation from the audible curses emanating from a nearby developer's desk (the first unlucky one to do a source control update). Oh, and the build/release manager would sometimes do something to publicly humiliate the build-breaker, like leaving a large rubber chicken on his desk. There was a build system, but it was mostly used just by the dedicated Quality Assurance folks to run builds when they were testing code nearly ready for release.
A few things happened during those days that got me more interested in what a deluxe build system could do for a team. The first resulted from a corporate acquisition -- I suddenly found myself working as part of a distributed team that spanned a couple of time zones. The verbal communication across desks in the office wasn't going to cut it anymore. A developer on one part of the team making a bad commit before going home for the day could waste hours of time for teammates in another time zone. The chicken wasn't going to cut it as a shame tactic when that much productivity was on the line -- plus how do you get it across the country? And the build itself became more complex -- feature branches, release branches, and just the sheer amount of stuff being built was growing by the day. Soon the day would come when no one developer would even know how to build every version of every component in the system without going back to the team wiki to (re)learn how. (You have a team wiki, right?)
Some changes were made to the home-grown "build system" in the following months and years to morph it into something more: Builds went from being triggered by time of day to being triggered by source control activity. They became continuous. Email notifications were sent out on failures, at first to static lists of people, eventually to both static lists (of the people who "owned" what a particular build represented) and dynamic lists (of possible build breakers based on recent source control checkins). Now the system could save the time of the unfortunate teammate who decided to update from source control at the wrong time -- a quick check of the build page could serve as an indicator of whether it was safe to pull down the latest code.
Around the same time, the writing of unit tests went from being a niche practice to becoming part of the culture. At first people just wrote and ran their own tests of the code they were responsible for. But most teams learn that it's best to make the tests live close to the code they're designed to test so that they stay up to date, and so that they can serve as indicators of the health of that code when run by people other than the owner. If I run all the tests after building the code with my new changes, I can learn about potential problems I've caused in code I'm not familiar with and that the compiler can't detect. From there's, it's not a big mental leap to imagine the utility of making running the battery of unit tests a part of the build itself, and teaching the build continuous integration system how to run them. After all, if the build "succeeds" but a number of tests are broken, what is the value of that build? It's at least somewhat suspect.
Distributing "Quality Assurance"
What's really happening in this historical progression, if you take a step back and look at it, is that some of the job of inspecting for software quality is being distributed (amongst people and systems) and moved earlier in the cycle of software development. It's no longer just the work of humans with "Quality Assurance" in their titles to do testing at the end of a development cycle. Some of that work -- at first the easiest parts, and eventually more and more sophisticated parts -- is now being done earlier in the cycle and more automatically.
Another thing that's happening is that the notion of "the health of the product" is becoming more visible. If you take a CI product and install it on a server that everybody on the team -- meaning not just developers, but business analysts, QA testers (sometimes these are also the BAs), and managers -- can see, and you tie in tests and static analysis, you've given everybody the ability to at least somewhat answer the question "how are we doing right now?" for themselves.
Increasing Visibility
This visibility goes hand-in-hand with the practice of agile. On a recent project, our team was introducing both agile and the concept of continuous integration into an organization that hadn't attempted either one. The business analyst who served as the product owner for our scrum team was accustomed to waiting for months to see his feature requests turned into code. And the developers were used to keeping him at arm's length (or hallway's length) for those blocks of months while they banged out code, cowboy-style, and then did a frenzied round of bug fixing at the end of those months. They had an antagonistic relationship with the business analysts because they also did the manual QA testing at the end of the cycle -- during which a frequent negative experience for everyone was the discovery that, even when the new feature "worked", it didn't meet the spirit of what the requirements had been driving at.
The practice of agile corrected a lot of the deficiency in this process -- shorter iterations meant more frequent testing, which meant quicker discovery of requirements gaps and misunderstandings by developers about what was supposed to be built. CI was a critical part of this. At first the BA/QA personnel on the team learned to look at the CI server's page for the release they were about to test, but they'd still ask a developer if it was okay to pull down a build of the product and deploy it for testing. Baby steps! We then made it possible for them to trigger deployments of the artifacts of a successful build to a test server, and soon they were hooked. Where before they'd waited months to see feature requests become reality, now they could attend the morning standup meeting, discuss a problem with the developers who were working on it, and potentially see a build containing a fix for the problem get created on the CI server that afternoon, ready for deployment and testing. That's an incredibly powerful experience when it suddenly comes into your life!
A lot of what I've outlined may be old hat for people working at agile startups, but I can't count the number of organizations I encounter and hear about who don't do anything like continuous integration and are spooked by it as a buzzword. This is primarily for the benefit of those like them out there who may be helped by the narrative.
Key Points
I'd like to conclude by briefly enumerating and expanding on some of the important aspects of CI I've mentioned thus far:
- Automated testing: Definitely consider adding automated running of unit tests to your build. If you do, be careful to make this mantra a part of your culture: it is not okay for tests to remain broken. You've got to keep tests up to date to maintain confidence in the value of CI and the health of your system, just like you've got to keep your tires aired up and perform regular oil changes to have the confidence that you're driving a reliable automobile.
- Add system and integration tests too: Automatically running unit tests is the first step. But you can also automate the running of system and integration tests, given a sophisticated enough build/test environment. The overhead may seem high at first, but it pays dividends later. A lot of the value of a system I recently worked on is its communications with 3rd parties behind the scenes. Since most of those partner organizations exposed endpoints for test purposes, we were able to craft tests that interacted with those endpoints. Sure, in some cases, a 3rd party's endpoint being down will cause an otherwise-valid test to appear to fail, but this in and of itself is valuable when the failure saves a BA/QA team member from attempting more detailed manual testing involving that same partner. "It looks like SprocketSoft's endpoint might be down today." That can become a quick discussion point during the morning standup instead of wasted hours.
- Static analysis: We live an age where most developers use IDEs that have code analysis built in, and this is a great thing. But sometimes not every team member uses the same tools, enables the same settings, or bothers to pay attention to the various colored highlights and squiggly underlines that are trying to warn of potential problems. Standardizing on a set of analyses to perform and wiring them into the build, perhaps to generate a report, can add a lot of value. Consider making pulling the static analysis report and discussing it a regular part of code reviews. (You're doing code reviews, right?)
