Continuous Improvement and the Agile Retrospective

Written by: Mark Balbes

Agile development isn’t just a better way to build software, but is also an effective strategy to make the entire enterprise more flexible, durable, and responsive to change. One vital method to help achieve this broader objective is to focus on continuous improvement.

Organizations begin using Agile as a set of prescriptive practices that structures how development teams perform their work. They transition to Agile development through the use of important technical practices such as test-driven development, pair programming, and continuous integration, as well as project management practices like sprints or continuous flow. While it’s a good starting point, it’s not enough. At some point they notice that the unique requirements of their culture, business domain or a particular project aren’t adequately addressed by the boilerplate versions of these practices. At that point, if they stick to their guns and continue to go “by the book,” they miss out on an important opportunity to improve their team, their processes, and their product.

Doing better takes time. And you have to have confidence that the time invested is more than compensated for by what you get back. Often when solving a technical problem with my teams, I ask that we first determine the qualities that the solution must have before we work on an actual solution. So, let's do that here: What are some qualities to the solution for the problem stated above, namely "How do we do better?"

Here's my list:

• The solution must be sustainable, regular, and continuous
• The whole team must buy into it.
• All team members can participate.
• We should be able to engage outside help as needed.
• The solution itself should be adaptable and allow for change.

A solution that meets these qualities is the team retrospective. Let’s take a look.

The Importance of Team Retrospectives
A retrospective is an all-hands team meeting that happens on a regular basis, preferably weekly, where the team comes together to ask the question “How are we doing?” or in some cases “What the heck is going wrong?”

Let me tell you a story.

One of my teams is incredibly agile. We’ve got a mature code base, a large suite of automated tests, and we practice test-driven development and pair programming religiously. It’s very easy to make changes to the software to accommodate new functionality. The team works together effectively, or at least they did. Suddenly, they found themselves constantly bickering over the appropriate solutions to problems, and generally making our war room a rather unpleasant place to work. No one knew why this was happening until we had our next retrospective.

And then the answer came out. We had inherited a large code base from one of our development partners. They didn’t practice Test Driven Development (TDD) or even use automated testing. Their code was a hairball mess of stuff that was hard to work with and extremely buggy. We were trying to put some discipline around the code and get it under control. And this was why our team cohesion started to fall apart.  We were in extremely uncomfortable territory, working on unknown code without the benefit of tests. It made us ill at ease, irritable and a bit scared, leading to the higher tension we were experiencing. I’d like to be able to tell you that everything snapped back to normal after that, but of course, it couldn’t. However, knowing the source of our tension did help us to keep things calmer. Even better, it helped us focus on discussing the real source of our problems; how to get this code base under control.

Retrospectives are nothing new. Companies have been doing them or something similar for decades. For retrospectives to be helpful, they have to be effective. Without a focus, they can easily turn into non-productive gripe sessions. An outside facilitator provides that focus; someone who is not a member of the team but is there to elicit discussion.

I participate in retrospectives with my team and I act as a facilitator for other teams. This allows me to see both sides of the process. There are several factors that I think are important in a good retrospective:

• The retrospective should take place outside the war room.
• Each team member must feel safe to contribute ideas.
• The facilitator should have a plan for the retrospective but be quick to put it aside if the team wants to attack other issues.
• The facilitator can ask questions and make suggestions but they are not leading the discussion.

Games and exercises are an important part of a retrospective. Some teams will balk at these, and would rather opt for discussion. Open discussion is easy but is not always the right choice. Often a team will have one or more members that are not comfortable speaking their opinion publicly. Open discussion tends to focus on the obvious or comfortable issues but avoid the hard conversations. A good game or exercise will draw out people's opinions and unspoken issues by either creating a safer environment for them to speak publicly or by providing anonymity.

Once you've created your safe environment, you’ve played your games and identified issues, what do you do to address these issues? Here are several suggestions:

The “Try-It For-2-Weeks” Approach

One of the approaches that I've found to be successful is the "Let's try it for two weeks and then re-evaluate" approach. This works when the team has an idea for how to address their issue but isn't sure if it's the right approach. By trying it for a limited time, the team isn't locking themselves into anything. It's easier to get buy-in from the team for a limited experiment than a permanent change. Two weeks is usually long enough to determine if something is working. For my team, this is our favorite approach.

The Metrics-Driven Approach 
This approach is harder. It assumes that the team has metrics that are driving a decision to make changes in how they do their work. If the right metrics don't exist, the team will need to collect them on their current process before making changes. Otherwise, there is no basis for comparison. Most of our projects at Asynchrony Solutions collect metrics like cycle time, work in progress and velocity; metrics associated with how fast the team is going. This allows a team to see when a change works and how it impacts speed.

I was involved with a project that was extremely metrics-driven. We were helping a customer transform their business to agile. Some of their developers were very reluctant to pair program or practice test-driven development. At one point in the project, they convinced their managers to let them drop these practices. Because we already had a significant baseline history of metrics on cycle time, velocity and defects created, we could show how the team velocity dropped and code quality suffered dramatically when these practices were not followed.

The Project Charter
The project charter is a document written by the team that describes how that team will work together. Rather than start from scratch, we use a charter template that is itself a living document. The template asks questions about the following areas:

• Project Purpose and Timeline
• Team Members and Workspace
• Planning and Estimation
• Monitoring and Measurement
• Configuration Management
• Development, Testing and Continuous Integration
• Risk Management

Where I work, every project has a charter and we review them on a regular basis in our retrospectives, typically when new people join the team or when something significant happens like starting a new release. Reviewing the charter spurs conversations about how the team currently works, how it has changed since the charter was last updated, and how the team can do better.

The Facilitator
The facilitator is the key to an effective retrospective. A good facilitator knows when to let open discussion continue or when to move to a game or exercise to draw out the team members. The experienced facilitator knows how to ask probing questions without leading the team to their solution. And when the retrospective is breaking down into a gripe session, as they can easily do, the facilitator brings the discussion back on track.

Continuous Improvement – A Little at a Time
My team has come a long way in four years. It has been a slow, steady process that we will never finish. Our early improvements were pretty basic, focusing on doing better with our software development disciplines like pairing and test-driven development. Then we started looking at our iterations. We went from two-week iterations to one-week iterations (which made things worse) to no iterations using Kanban (which made things much, much better).

Our last process change was as recent as two months ago and it was a big one. We reworked our entire Kanban process to focus on minimal marketable features (MMF) rather than individual stories. An MMF is a collection of stories that makes up an actual shippable unit. While each story is required to deliver value to the end user, a single story may not provide enough value to be releasable by itself. By focusing our development on MMFs, we are hoping to improve the cohesiveness of the user experience in our releases. We continue to improve this new process incrementally as we learn from it. And our weekly retrospective plays an important role in identifying how it is helping us and where we still need to improve.

About the Author:
Dr. Mark Balbes serves as Vice President, Architecture at Asynchrony Solutions, and leads multiple Agile projects for Government and Fortune 500 companies. He received his Ph.D. in Nuclear Physics from Duke University. For more information visit,http://www.asynchrony.com.