Disciplined Agile

The Institution of Engineering and Technology has recently published a paper entitled An Academic Approach to Transform Organizations One Engineer at a Time by Eduardo Juarez Pineda, Rocio Aldeco-Perez, and Jose Manuel Velazquez.  The DA toolkit features in this paper so I thought you’d be interested.

Paper Abstract:

The Agile manifesto is only 4 lines, there are only 12 principles for agile software and the original Scrum Guide was less than 20 pages, so how hard can Agile be?

Chess has only a couple of dozen basic moves so how hard can it be to get to checkmate?

How many times have you heard?

• All you need to be Agile is to get the team doing a few funky ceremonies

Or the approach to evolving a team from forming to performing:

• If the team isn’t performing after a month then the manager is not effective.

Despite the apparent simplicity of Agile, transforming to Agile is hard and the learning curve is very steep.  Depending on the team and the individuals, a transformation from waterfall to a high performing Agile team can take anywhere from 6 months to a year or more.  That’s not to say that they won’t be productive over that period of time.  In fact, they will start delivering business value at the end of the first iteration, but the team will be in constant growth and development over that period of time because not only do the people have to learn a new process, they also need to learn a new way of thinking as well as a new way of working as a team.

1.0 A new process

Aside from being excited to work with the thought leaders in the agile world, the reason I joined with SA+A is because of the Discipline Agile Delivery (DAD) decision framework and the way it captured all the tough stuff that I had been wrestling with for years.  With many years of experience building agile teams, I considered myself somewhat of an expert with answers to a lot of the difficult questions.  When I reviewed DAD I realized that the answers I had gained from my own personal experience would not apply in all circumstances or for all teams.  Context counts.  The DAD Agile/Basic and Lean/Advanced project lifecycles have three phases: inception, construction and transition.  Each phase has a collection of process goals, and each goal encompasses several process decision points.  And, each decision point has a collection of options to choose from.  These process goals, decision points and options have been empirically assembled by analyzing many agile teams to see what makes an agile team successful.  This framework gives a team the ability to define a process that works for them.  Is this difficult to do?  Absolutely!  Defining a process to use is always difficult. BUT…..  The framework provides some guidance. Each goal option has default options that the team can use to get started and the options are listed in preference order so the team can easily pick an initial option.  As the team evolves and matures they can chose the more advanced options and evolve and mature their process.  Does the toolkit include all possible options?  Definitely not, but the options are certainly extensive.  The framework has evolved over time and continues to evolve as the coaches and thought leaders gain more experience and work with more diverse teams.

No matter which lifecycle the team has chosen to use; whether the team is using classic Scrum (Basic/Agile) or Kanban (Advanced/Lean) or a tailored process created using a toolkit such as DAD, the team is going to need practice and discipline to get it right.  That means, executing the lifecycle, doing periodic retrospectives, updating the process and then repeating again and again until they get it right.  Changing behaviors is hard and the team will struggle to be successful.

2.0 A new way of thinking

No matter how you cut it, transforming to Agile is a cultural change and culture changes are hard.  Not only is it hard for those who want to make the change but almost inevitably there will be pockets of resistance from people are happy with “the way we’ve always done it”.  Unfortunately, political infighting is the biggest risk to innovation. It is going to happen, you may as well brace yourself for it.

The roles are changing from traditional roles (such as: developer, business analyst (BA), system analyst (SA), quality assurance (QA) and project manager (PM), to new unfamiliar roles (such as: product owner (PO), team lead (TL), architecture owner (AO) and team member).  The traditional roles may be deeply embedded in your HR hiring and performance review processes. Transitioning to the new roles will require a realignment of skills and abilities that is bound to cause great consternation amongst the team members.  Often this is a very difficult transition for project managers, in particular, because they find that a lot of the traditional tasks (i.e. work breakdown structures, planning, estimating, etc.) are now the responsibility of the whole team.

The team needs to start thinking in smaller deliverable pieces.  The concept of delivering a complete system from known specifications at the beginning of the project has been the fallacy of traditional development.  The cost of delaying delivery to the very end of the project and the lack of ability to meet actual customer needs can be eliminated by delivering minimum functionality early and augmenting with new and improved functionality iteration after iteration in close collaboration with the end customer.  But moving to this model of continuously delivering can be a tough adjustment for developers.  Even (or maybe especially) for seasoned developers, the concept of starting to build and deliver not knowing all the detailed specifications up front can be very unsettling. Getting comfortable with this model takes time and experience before developers cease to fall back on old behaviors.

One of the pillars of agile is complete transparency with the business and the stakeholders into the team, the process they are using and the delivery of the business functionality.  With the business embedded in the team on a day to day basis and iteration reviews to demonstration and approve all the deliverables from each iteration, there is no hiding problems when they occur and no hiding late deliveries or failed iterations.  On the flip side, the business is there to celebrate the wins and recognize the deliveries at the end of each iterations and they are part of determining the velocity of the team and adjusting the schedule to match the velocity.

The new way of thinking also requires new tools.  Tools that can build dashboards to automate and display the development intelligence metrics that provide transparency into the team’s performance.  Tools that automate configuration management, continuous integration and continuous integration as well as automated testing.  All of these endeavors are hard and required skilled individuals to lead; none of these comes for free.

3. A new way of working with others

More people skills are required by everyone.  People are now going to be working much closer together physically and will be interacting and collaborating on a much more frequent basis.  Experience working together (often for many years) may give an advantage to some team members, however, they still need to learn how to work together as a team.

3.1 Team Development Phases

Tuckman’s team development model includes the phases: forming, storming, norming, and performing. These phases are all necessary and inevitable in order for the team to grow, to face up to challenges, to tackle problems, to find solutions, to plan work, and to deliver results.  The number of iterations required to move through the phases will vary by team depending on factors such as: company culture, skills, experience, familiarity with each other, leadership (or lack thereof), external influences and coaching etc. Teams may also plateau in a phase before entering the next phase.

Agile teams will go through the four Tuckman phases:

1. Forming. All teams will go through the forming phase. Typically this take 4 to 6 iterations for the team to: understand the lifecycle process, understand how to work together, understand what the team goal is, and most importantly to get comfortable enough with each other to risk the possibility of conflict.  During this phase, the team’s velocity will vary considerably especially for the expected failed iterations although there should generally be a trend upward in the velocity.  Retrospectives in this phase tend to be very cordial and factual because everyone is avoiding conflict.
2. Storming. The storming phase can be very upsetting to both the team members and team managers. Often managers will step in to solve the storming issues without realizing that it is a necessary phase for the team to work through on their own.  Conflict, disagreements and personality clashes must get resolved before the team can move to the next phase.  Unfortunately, this phase is a make or break phase and it is hard to predict just how many iterations it will take but it is not usual to expect 3 or 4 pretty rough iterations.  Some teams will completely disintegrate during this phase and it is certainly not unusual to have a decrease in velocity during this phase.  Some members of the team may find they have basic incompatibilities and need to move to another team and this change in team membership can revert the team back to the forming phase.  The retrospectives tend to be very interesting and intense during this phase as they deal with the inter-personal issues of the team.  You can pretty much expect that there are going to be some hurt feelings no matter how safe an environment you have.
3. Norming. The norming phase is when the team comes back together stronger and more cohesive for having weathered the storming phase. They have resolved their issues and have developed a new synergy where the team just “works well together” and it is fun to be part of the team. The team velocity should trend upward and then level off as the team hits its capacity consistently.  With the new-found synergy of the team, the retrospectives can focus on process improvement and optimizing the operation of the team with the hope of transitioning to the performing phase.
4. Performing. The performing phase is the goal for all teams but very few actually get to this phase. Not all teams can get to the high level of performance of a Special Ops team where each member knows exactly what their team mates are doing and what they need to do to work as a cohesive unit.  A team in performing phase no longer needs a coach or supervision but can operate autonomously and efficiently.

3.2 Team changes everything

With self-organizing teams, decisions are now made at the team level rather than at the PM or manager levels.  This can be a difficult transition for both the team members and the managers.  Team members are often use to others making the decisions for them and wait for decisions or are reluctant to make decisions.  Managers often feel disempowered when decisions are no longer in their hands because the decisions are made by the team.

The team still has the responsibility to do the traditional activities such as; analysis, design, development testing and deployment.  However, under the new model, the team also takes on the inception responsibilities of: developing user stories, sizing the stories, estimating the stories, creating the product and architectural vision, and developing the release plan.   During construction the team also has to: plan the iteration, do look ahead planning, and have an iteration review (i.e. demo) with the stakeholders.  To keep the team working together they need to have the daily coordination meeting (i.e. Scrum) and to keep the team growing and evolving of course they need to do the retrospective at the end of each iteration.

Thinking, analyzing, planning and building just-in-time is a new way of thinking for a traditional team.  The inception phase in particular can be a difficult adjustment.  Designers and developers are used to doing all the analysis and design upfront.  Doing just enough analysis to be confident of a solution and then writing the user stories to do the work and build the plan is a difficult transition.  Writing user stories so they can get started and learn as they go is a very difficult transition.  Building just the simplest framework and then adding required features and functionality as needed requires practice, training and reinforcement.

There is much more focus on delivering quality because the team is responsible for development, delivery as well as product support.

4.0 The Bottom Line

Chess has only a couple of dozen basic moves and so it is simple to learn the basics.  However, after 3 moves there can be several billion moves available making it so complex that it can take a lifetime to master.  Transforming to Agile is a lot like that.  It looks very simple on the surface but under the covers it can get very complex very quickly.  That’s where the value of a good coach comes in.  You need a coach who brings a lot of experience to the table who will be open and honest with you about what needs to be done and how to get a successful transformation.

A common challenge that we run into when working with organizations adopting Disciplined Agile strategies is helping them to identify and then coach people for the Product Owner (PO) role. This is often easier said than done due to the dearth of people with the required sill and mindset. In this blog we explore several strategies to address this challenge.

What Are You Looking for in a Product Owner?

Let’s begin with a review of the requirements for a good product owner:

1. Analysis skills. POs need to be able to elicit requirements, explore them with stakeholders, negotiate priorities, facilitate modeling sessions, and in some cases document requirements.
2. Decision-making authority. POs need to be empowered to prioritize the work of the team AND need to be comfortable with doing so.
3. Good stakeholder contacts. POs need to know who to work with in the entire range of stakeholders, including both business and technical stakeholders.
4. Full-time availability. This is a full time job, and at scale often proves to require more than a single person in the role (more on this in future blog postings). They’re available to the team on a daily basis.
5. You want them in the position for several years. It takes time to grow an effective PO, depending on the background of the person we’ve seen people take between six and eighteen months to truly become comfortable in the role. This is a fairly large investment for your organization, so once you’ve made that investment its reasonable to want someone to stay in the role for at least a few years.
6. They understand both your business domain and IT infrastructure. When taking a Disciplined Agile approach to product ownership the PO is responsible for representing all stakeholders, including both technical and business stakeholders. An implication of that is that POs should have a good understanding of the business domain and direction as well as your existing IT infrastructure and the direction that it’s going in. These understandings will be very important for prioritizing the work effectively.

Given the skill requirements it shouldn’t be surprising to anyone that there is a shortage of candidates for the PO role in most organizations. Let’s explore your options.

Potential Sources

There are several potential sources of new product owners. The following table compares and contrasts these options. As you can see there is no ideal option available to you, and the reality is that you will likely need to obtain PO candidates from whatever source you can find.

An interesting strategy that we’ve found fruitful, albeit one that borders on ageism, is to look for potential candidates whom have been with your organization for a long time and who are getting close to retirement. These are experienced people who therefore are likely to have a good understanding of your organization and where it’s headed, they very likely have a good contacts throughout your organization, and they’re very likely looking for an interesting and stable position that will last until they’re ready to retire.  Given that the investment required to create a Product Owner is rather steep so therefore you want someone willing to stay in the position for at least several years, and given that these are experienced people looking for a position that will last several years, it’s a very good alignment that you should consider taking advantage of.

Have a Clear Career Path

A critical success factor for attracting people to the role of PO is to have a clear and viable career path for them. If it isn’t obvious to people where they would go next after becoming a PO, or worse yet if becoming a PO is seen as a career dead end, then why would anyone choose to step into this role? One option for POs is to become product managers, if a product management function exists in your organization. Another career path is for POs to move into a senior business or IT leadership position. Being a PO gives people a deeper understanding of how IT fits into the larger organization and how it works in practice – key skills for anyone in senior management these days.

One of the interesting trends that we’re seeing within organizations taking a disciplined agile approach to solution delivery is the preference for stable teams. Stable teams, also called stable product teams or simply long-term teams, are exactly as they sound – they remain (reasonably) stable over time, lasting longer than the life of a single project. This blog explores the differences between project teams and stable teams and then overviews the advantages and disadvantages of the stable team approach.  We also explore the issue of how stable teams evolve over time.

Stable Teams

As you can see in the following diagram, with a project team approach we say that we bring the team to the work. What we mean by that is that we first identify the need to run a project, perhaps to build the new release of an existing solution or to build the initial release of a new solution, we build a team to do the work.   Once the work is done, in this case the solution is successfully released into production, the team is disbanded and its team members move on to other things.

The stable team approach is a bit different. In this case we first build an effective team then we continuously bring valuable work to the team to accomplish. In this situation the work never really ends, but instead we replenish the team’s work item list (or work item pool depending on the lifecycle being followed) regularly. The team stays together and continues to produce value for your organization over time.

Of course the term “stable team” is a bit of a misnomer as they do evolve over time. For example, many people like to stay on a team for a couple of years and then move on to another team to gain new skills and perspectives. This is good for them and good for your organization as it helps to keep your teams fresh. Sometimes you will want to grow or shrink a team. Sometimes you will discover that two people aren’t working well together and you need to split them up. The point is that there are very good reasons for your stable teams to evolve over time.

We wouldn’t be disciplined if we didn’t explore the trade-offs involved with stable teams.

The Advantages of Stable Teams

There are several advantages to stable teams:

1. Lower management overhead. There is clearly less “resource management” to be done because you’re not constantly forming and disbanding project teams. In fact, this lower need for resource management activities is one of several factors why agile IT organizations need managers than non-agile IT organizations.
2. Easier team budgeting. The annual budget for a stable team is incredibly straightforward to calculate: Multiply the number of people on the team by the fully burdened cost of an IT person for your organization. Once again, less management work required for this.
3. You build better teams. When you build project teams you tend to take the people who are currently available (often referred to as sitting on the bench). With a stable team approach you’re motivated to build your teams with the right people, and very often its best for the team to build itself by inviting others who they believe will fit in well.
4. There is greater opportunities to build trust within the team.  It takes time to build trust within a team.  Greater trust leads to greater willingness to work together in a more streamlined manner.  As Stephen Covey insightfully points out, trust enables speed.
5. There’s a greater opportunity for safety.  It takes time to build an environment where people feel safe. In safe environments there is a much greater chance that they will share ideas and be willing to try new things because they don’t fear being thought less of or even punished.
6. There is less overhead from team formation. You’re forming teams far less often with a stable approach compared to a project team approach, hence there is less overhead in total for your organization.
7. Better team performance.  Consider the analogy of a train.  Just like it takes time to bring the train up to cruising speed it takes time for the team to jell.  Bringing work to a seasoned team that works together well is like jumping onto a train going at full speed: it’s faster in both cases because you don’t have to get going from a full stop.
8. You have more efficient utilization of staff. With this approach it is far less likely that someone will be “sitting on the bench” because they will instead be an active member of a team. When someone is hired it is directly into a team. Throughout their career they will move from team to team as appropriate. The only time that they might not be utilized is when their on vacation, sabbatical, or if you purposefully disband a team. The first two reasons are something you still have with the project team approach, and the last reason should happen a lot less often.
9. Your teams are more likely to improve. When a team knows that they will be working together for a long time, and particularly when they are responsible for the entire delivery lifecycle from beginning to end, they are more likely to streamline their work so as to make things better for themselves.

The Disadvantages of Stable Teams

There are several disadvantages to stable teams:

1. Teams can become too stable. A real danger of stable teams is the potential for groupthink – everyone on the team starts to think and work in a common way, thereby being in danger to common blindspots. Luckily people still want to move to other teams for career management reasons, offering the opportunity to bring new viewpoints into other teams. In the Disciplined Agile (DA) toolkit we have the continuous improvement process blade which supports sharing of ideas across teams so that can also lessen the chance of groupthink. And, as mentioned earlier, some people may need to be motivated to move on to another team for interpersonal reasons.
2. You still may need to do projects. Sometimes your business team makes promises to their customers. For example, in a software company a sales person makes a big sale and promises that by a certain date your solution will have additional features that the customer needs (in immature organizations they’ll even make such promises without first negotiating this with the delivery team). Another example would be a financial institution that needs to fulfill new industry regulations that require changes to existing solutions. In both of these cases there is a large amount of work to be done that needs to be delivered before a certain date, and this may motivate you to treat this work as a project. You would still bring this work to the appropriate stable team(s) to accomplish as you normally would. However, you would also track the performance of the work to ensure that it is delivered in its entirety as appropriate. The implication is that projects may not completely go away

Evolving Stable Teams Over Time

Stable doesn’t mean stagnant.  Of course you still have basic people management issues such as people wanting to expand their skill set by working on something new by rotating to another team, people leaving the organization, and new people joining the organization.  So the team itself may go on for many years even though the membership of the team evolves over time.  Ideally these membership changes are not too disruptive: It’s not too bad adding a new person every month or so, or losing people at a similar rate, but gaining or losing several people in a short period of time can be painful.

Our Recommendation

Start experimenting with stable teams if you’re not already doing so. For most organizations the advantages clearly outweigh the disadvantages. In fact, you can see this in the Longevity decision point of the Form Initial Team goal diagram below.

What is the point of the retrospective?

The retrospective is one of the most important ceremonies in all of agile.  This is the time the team spends together to assess how they are working together and define steps to improve that process.  It needs to be a “safe place” where people are able to speak openly and honestly.  This is their opportunity to air their dirty laundry and work through their inter-personal issues.  This is a time of growth for the team and for the team to take ownership of improvement.  The team lead will facilitating the retrospective and should manage the interactions to keep the environment safe.

Define the Team

If the retrospective is a team ceremony, then what do we mean by team?   The team includes: the team lead, the architecture owner and all other members that actively contributed to meeting the deliverables for the iteration.  This includes: developers, testers, BAs, QAs, or other specialists such as technical writers, database engineers etc.

What about the Product Owner (PO)?

The PO is NOT a member of the team.  They certainly interact with the team but they do not contribute to meeting the deliverables for the iteration so they are not a member of the team.  They are not allowed to participate in the planning poker for the user stories for the same reason.  The team votes because they are on the hook for delivering based on the sizing and the estimates but the PO is not on the hook, so they don’t get a vote.

Should the Product Owner participate in the retrospective?

In general, I would start by including the PO in the retrospectives because the team does have to learn and adjust to working with the PO. Keep in mind though that the PO may come and go but the team should stay together so it is most important that the team works well together.??As a coach, I usually talk to the PO beforehand to say that they are an invited guest and that it is a privilege to be part of the meeting so they should act accordingly.   I have been in many situations where the PO was welcomed at the retrospective, and felt left out if not included.  I favor building trust between the business (the PO is their representative) and IT (the team).  Including the PO in the retrospective can help the PO assimilate with the team.

I have also had several situations where as the coach I had to ban the PO from the retrospective because they were too commanding and disruptive in the meeting for the team to have an effective retrospective.  I have also seen many situations where the PO is also the resource manager of members of the team (which in itself is not recommended).  Having managers in the room can definitely have a dampening effect on the member’s willingness to be open and honest about problems and solutions.

If the PO doesn’t participate, at least as an observer, the team runs the risk of having to “sell” the cost of their improvement actions (against other backlog items) after coming up with them. Hopefully the PO is engaged enough with the team to understand its weaknesses and support improvement in those areas whether then attend the meetings or not.

Team Decision

Retrospectives are about improving the process, and a non-trivial part of that is optimization of collaboration between the PO and the team.  I would suggest that the team should decide whether or not to include the Product owner.

What about the Stakeholders?

The retrospective should absolutely be a closed door session for the stakeholders since the retrospective must be a “safe space”.

There was a twitter debate lately that talked about a team being subjected to “a drive by criticism from 2 PM’s during a Retrospective”.  This is a good reminder why we constrain attendance.  The “safe place” is affected by the presence of people with positional authority, potential agendas or other implicit impact.??The team may decide to invite such people – usually to ensure that they are communicating improvements needed that are beyond their locus of control.??Having outsiders as guests at the retrospective will change the dynamics but at least it is a team decision to do so.

It is very important that the team own their process.  If they’re uncomfortable that someone is in the room then that person should be asked to either change their behavior or leave (perhaps to be invited back in the future).??The coach should always be thinking along the lines of “do we have the right people in the room” and then act accordingly

Isn’t agile all about transparency?

There was a twitter debate lately that centered on transparency.  I believe that transparency is a key element to making agile successful.   I’m all for transparency in everything about agile; EXCEPT the retrospective!??Sometimes you need to have a family meeting outside of the public eye and that is the retrospective. The retrospective is all about resolving your issues in private so that you can present a united front to the rest of the world. To use a sports analogy, an NHL coach doesn’t invite the business (fans) into the dressing room between periods.  There are lots of other places for transparency; the retrospective doesn’t have to be one of them.

The output of the Retrospective

While the actual retrospective meeting is closed to other observers, I would suggest that the action items coming out of the retrospective need to be made public and posted as an information radiator for everyone to see.  The changes are more likely to get implemented if the team sees them every day.  The team may also want to “radiate” their improvement actions on their dashboard.

The actions and results of the actions may also be shared with other teams through what is often called a Retrospective of Retrospectives. I encourage teams to only choose one or two areas for improvement at a time to provide focus and make meaningful progress.