Disciplined Agile

This posting, the latest in a series focused on a disciplined agile approach to program management, overviews the activities associated with program management. The Disciplined Agile (DA) toolkit promotes an adaptive, context-sensitive strategy.  The framework does this via its goal-driven approach that indicates the process factors you need to consider, a range of techniques or strategies for you to address each process factor, and the advantages and disadvantages of each technique.  In this blog we present the goal diagram for the Program Management process blade and overview its process factors.

The following process goal diagram overviews the potential activities associated with disciplined agile program management.

The process factors that you need to consider for program management are:

1. Allocate work. Work items must be allocated to delivery teams, or to open source efforts in the case of programs which include internal open source components, throughout the lifecycle.  The type of work and the focus of the sub-team are the primary determinants of how work is allocated.  However, team capacity and load balancing concerns, for example a team has run out of work or a team currently has too much work, will also be considered when allocating new work.  Work allocation is the responsibility of your product owners although team capacity planning and monitoring is typically performed by the program manager and team leads.  Regardless, these activities should be performed collaboratively by the available people at the time.
2. Prioritize work.  The work performed by the teams – including new requirements and fixing defects – needs to be prioritized.  There are several ways to prioritize the work, such as by business value, by risk, by severity (in the case of production defects), or by weighted shortest job first (wsjf) to name a few strategies.  Prioritization is an ongoing activity throughout the lifecycle and is the responsibility of your product owners.
3. Plan program.  Traditional programs are often planned on an annual or even ad-hoc basis.  Agile programs, at least the disciplined ones. tend to be planning on a rolling wave basis.
4. Organize teams.  There are three common strategies for how you can organize delivery teams within a program – feature teams, component teams, and internal open source – each of which has advantages and disadvantages.  In addition to delivery teams, in a large program you are likely to find the need for leadership teams – the Product Owner team, the Architecture Owner team, and the Product Delivery/Management team – made up of the product owners, architecture owners, and team leads from the delivery teams respectively.  These leadership teams are responsible for work/requirements coordination, technical coordination, and management coordination within the program respectively.
5. Coordinate teams.  There are several ways that the sub-teams can coordinate with one another.  For example they could choose to have cross-team coordination meetings (also called a Scrum of Scrums (SoS)); they could visualize the work through task boards, team dashboards, and other information radiators such as a modeling wall; they could choose to have “big room” planning sessions where all team members are involved or “small room” agile modeling sessions where a subset of people are involved; or even traditional (or agile) checkpoint meetings.  All of these strategies have their advantages and disadvantages, and all can be applied by the various types of teams mentioned earlier.
6. Coordinate schedules.  There are several strategies that a program can adopt to coordinate the schedules between sub teams.  The easiest conceptually, although often hardest to implement in practice, is to have all sub-teams on the same cadence (e.g. every sub-team has a two week iteration).  This is what both SAFe and LeSS prescribe.  Another option is to have multiplier cadences where the schedules of sub-teams align every so often.  For example, we once worked with a large program where some sub-teams had a one-week iteration, some had a two-week iteration, and a few had a four-week iteration.  We’ve also seen another team where sub-teams had one, two, or three week iterations that provided alignment of iteration endings every six weeks.  Most common, although rarely discussed, is for sub-teams to have disparate cadences.  This is guaranteed to occur when teams are following different lifecycles (remember, the DA toolkit supports several).  For example, when some sub-teams are following the Scrum-based agile/basic lifecycle that has iterations, yet other sub-teams are following the lean or continuous delivery lifecycles that have no iterations, then you have an alignment challenge.  Or if you have sub-teams adopting any iteration length they like (we’ve seen some programs with sub-teams with two, three, four and sometimes even five week iteration lengths) then they also in effect have disparate cadences.
7. Schedule solution releases.  Programs need to schedule their own releases, in accordance to your organization’s release management strategy, which involves coordination between the sub-teams.  When the cadences of the sub-teams are (reasonably) aligned then it is easier to coordinate production releases.  For example, when all sub-teams have two-week iterations (or at least the sub-teams with iterations do) then they could potentially release into production every two weeks.  In the case of multiplier cadences, there is the potential to release into production each time the iteration endings align.
8. Negotiate functional dependencies.  An important responsibility of the Product Owner team is to manage the functional dependencies between the work being performed by various sub-teams.  There are strategies to manage dependencies between two agile sub-teams, between an agile sub-team and a lean sub-team, and even between an agile/lean sub-team and a traditional sub-team (this isn’t ideal, but sometimes happens).
9. Negotiate technical dependencies.  Similarly, an important responsibility of the Architecture Owner team is to work through technical dependencies within the solution being developed by the program.
10. Govern the program.  The program must be governed, both internally within the program itself while still operating under the aegis of your organization’s overall IT governance strategy.  Program-level metrics, particularly those tracking the progress of sub-teams and the quality being delivered, are vital to successful coordination within the program.  Sub-teams should also be working to common conventions, ideally those of the organization but in some cases specific to the program itself (perhaps your solution is pioneering a new user interface look-and-feel or new data storage conventions).  Programs, because of their size and because they are usually higher risk, often have more rigorous reporting requirements for senior management so as to provide greater transparency to them.  The implication is that a program’s dashboard often has a more robust collection of measures on display.

How Scaling Affects Program Management

Although program management primarily addresses the team size scaling factor, your tailoring decisions will still be affected by the other scaling factors:

1. Geographic distribution.  Chances are very good that large teams will also be geographically distributed in some way.  There are two flavors of this: Are teams geographically distributed (e.g. in different physical locations) and are people within a team geographically dispersed (e.g. people are working in cubes, on different floors, in different buildings, or from home)?  Both add risk. Coordination within the program becomes more difficult the more distributed the teams are, and more difficult within teams the more distributed the people are.  Distribution hits the leadership (the product owner team, the architecture owner team, and the team lead/product delivery) teams particuarly hard because members should be located with their delivery sub-teams but also need to work regularly with their counterparts located elsewhere.  The implication is that the team may require more sophisticated tooling to enable collaboration and more importantly be prepared to invest in travel regularly to foster better communication between disparate locations.  Furthermore, when your stakeholders are geographically distributed it may require your Product Owners to get support from agile Business Analysts in the various locations to help elicit requirements from them.
2. Compliance.  Compliance, either regulatory compliance required by law or self-imposed compliance (i.e. CMMI-compliancy), will definitely have an effect on your approach to program management.  In fact, the larger the program the more likely it is to fall under regulatory compliance due to the greater risk involved. Regulatory compliance generally requires greater governance both within the program and outwards facing as well.  Under some regulations your coordination efforts will require proof that they occurred, such as some form of meeting minutes that capture who was involved, the decisions made (if any), and action items taken by people.  Compliancy may also motivate more sophisticated approaches to capturing requirements by your Product Owners and to documenting technical concerns by your Architecture Owners.
3. Organizational distribution. The larger the team, the more likely you are to involve contractors, consultants, or even to outsource portions of the work.  When external organizations are involved the Program Manager will likely be involved it the contract management effort, which in turn may require assistance by the team leads.
4. Technical complexity.  The larger the team, the more likely it is that they are taking on greater technical complexity.  Or, another way to look at it, greater complexity often motivates the creation of larger teams to deal with that complexity.  Greater technical complexity will motivate greater attention to architecture and design, thereby motivating more regular collaboration of the Architecture Owners.
5. Domain complexity.  Similarly, team size and domain complexity tend to go hand-in-hand.  Greater domain complexity will require the Product Owners to work in a more sophisticated manner and may even motivate them to get support from agile Business Analysts (or junior Product Owners as the case may be).

Concluding Thoughts

In some ways “program coordination” is a more accurate term than “program management.”  Unfortunately “program management” is a far more common term within the IT community so we have decided to stick with it.

Our next blog posting in this series will describe the workflow of program management with other key IT activities such as portfolio management, enterprise architecture, and IT delivery teams.

Related Postings

• Large Agile Teams
• Why Do We Need Agile Programs?
• Disciplined Agile Program Management 101
• Discplined Agile Program Management: External Workflow
• Disciplined Agile Program Management: Internal Workflow

An IT program is a large team composed of two or more sub-teams.  The purpose of program management is to coordinate the efforts of the sub-teams to ensure they work together effectively towards the common goal of producing a consumable solution for their stakeholders.  In this blog posting we explore the team structure of a typical large agile program and the type of activities that program management addresses.

Team Structure of a Large Program

We described in detail the team structure of a large agile program in Large Agile Teams.  The key ideas are that a large team is organized as a team of teams and that structures are required to coordinate people, requirement, and technical concerns within the overall program.  Where a “scrum of scrums” may suffice for this coordination on small-to-medium sized programs (say up to five or six sub-teams), it quickly falls apart for larger programs.  As a result large programs will find that they need: a Product Management (or Product Ownership) strategy where the Product Owners coordinate their activities; an Architecture (or Architecture Ownership) strategy where the Architecture Owners coordinate their activities; and a Product Delivery (or Management) strategy where the Team Leads coordinate their activities.  The Program Manager, a specialist role, is responsible for coordinating the overall leadership team.

The following structure shows how the Product Owners of each sub-team are also members of the Product Management/Product Ownership team for the program.  Similar structures, see Large Agile Teams, will also exist for Product Delivery and Architecture as well.

The Program Lifecycle

DAD includes a Program lifecycle for a team of teams, depicted below.

What Program Management Addresses

There are several important concerns that program management needs to address:

• Architecture. The sub-teams must have a common architectural strategy that they are working to.  The initial architectural strategy will be identified early and will evolve throughout the lifecycle, requiring the architecture owners on each team to coordinate their efforts.  Although there are several ways to do this, a common strategy is for all of the AOs to hold  a weekly architecture meeting to work through program-level technical issues.   Subsets of the AOs will also hold impromptu meetings during the week to deal with immediate technical issues as needed.
• Work item management.  The sub-teams work together to develop a consumable solution for their stakeholders.  The implication is that they need mechanisms in place to allot the work between the sub-teams, including development of requirements, change requests, and fixing of defects.  The product owners on each team will need to coordinate their efforts throughout the lifecycle.  Common strategies to do this include the POs holding joint look-ahead modelling sessions (called backlog grooming/refinement sessions in Scrum) with key stakeholders; work coordination sessions where they identify dependencies between requirements and match the work to the sub-teams; and impromptu sessions between a subset of the POs throughout an iteration to work through dependencies between work being performed by different sub-teams.
• Management coordination between sub-teams.  The sub-teams will need to coordinate “management issues” with one another on a regular basis.  These management issues may include conflict resolution between people on different teams, enabling people to move between teams, financial tracking across teams, and many other concerns.  As a result the teams leads, and any appropriate team members involved with a given issue, will need to meet regularly to address any issues.  A common mechanism for this is for the team leads to hold their own daily coordination meeting after their team’s coordination meeting, a strategy that is often referred to as a “scrum of scrums”.
• Sub-team cadences.  The Disciplined Agile tool kit, unlike most agile scaling frameworks, does not insist that sub-teams follow the same iteration cadence (or even that sub-teams follow an iteration-based lifecycle).  For example, one sub-team may have a one-week iteration length, five sub teams have a two-week iteration length, and two sub-teams follow DAD’s continuous delivery life cycle which doesn’t have iterations at all.  Or you may find a program where every sub-team has two-week iterations.  Or a strategy where all teams take a continuous delivery approach.  Each of these strategies make sense in certain situations, but none of them make sense in all situations.  There is no single “best practice“.
• Release cadence.  There are two fundamental issues here.  First, how often will the overall program release into production?  Will it release monthly?  Quarterly?  Bi-annually?  Something else?  Second, how often will the sub-teams release their work internally to other sub-teams so that it may all be integrated and tested as a whole?  At the end of each iteration?  At the end of each day?  Several times a day?
• Governance.  The program will be governed.  This governance is both inward facing, the program should govern itself, and be outward facing, the program is part of your organization’s portfolio and therefore will be governed as part of your organizations overall IT governance strategy.

Relationship with Other Process Blades

There is clearly overlap with some of the activities in the portfolio management, enterprise architecture, release management, product management, and governance process blades.  The issue is one of scope.  Where these process blades address activities across all of your organization, the scope of the related activities within program management is the program itself.  For example, where enterprise architecture addresses architectural issues for the entire organization, the architecture activities encompassed by program management relate only to the architecture of the solution being produced by the program.

Related Resources

• For an understanding of the team structures within a program, read the article Large Agile Teams.
• The Disciplined Agile Program Management Blade

On June 25, 2015 we gave a webinar entitled Disciplined Agile Outsourcing: Making it Work for Both the Customer and the Service Provider (click on the link to see the slides on Slideshare).  We will likely run the webinar again as it was very well received.  We had hoped to record it but unfortunately weren’t able to save the file because the software crashed.  Hopefully it will work out next time.  During the webinar we promised to write a blog summarizing it and addressing any remaining questions from the audience.  This is that blog.

Agile and Outsourcing?  Really?

There are good reasons why organizations are taking an agile approach to outsouring IT delivery projects.  The following table overviews the motivation and concerns of both customer organizations and service providers.

Table 1. Comparing the Viewpoints of Customers and Service Providers

Here are some interesting statistics about outsourcing:

• Gartner’s 2014 IT Outsourcing report estimated IT outsourcing to be $288 billion in 2013 and to grown by 5.9% compounded annually from 2013 through 2018.
• Deloitte’s 2014 Global Outsourcing survey found that 53% of organizations are outsourcing some of their IT function; 26% of respondents who do not outsource today plan to do so (supporting Garter’s growth forecast); 79% of respondents DO NOT believe their service providers are too expensive; 49% of respondents say their service providers are reactive vs proactive (e.g. they are basically order takers, not partners); and IT outsourcing activity is expected to increase.
• The Ambysoft 2013 Agile Outsourcing survey found that half of organizations who are “doing agile” are also involving outsourcing in some way

The Challenges Associated with Agile IT Outsourcing

Here are some common pitfalls that organizations run into with IT outsourcing:

• Organizational culture differences
• Expectations mismatch between the customer and the service provider
• The customer underestimates difficulty of managing outsourced projects
• Total cost of the solution isn’t considered
• Total value of the solution isn’t considered
• Transition to the operations team is mismanaged
• Over-reliance on documentation
• Software licensing issues
• Learning curve for service provider underestimated
• The service provider is understaffed
• Some aspects, e.g. security, cannot be outsourced
• Intellectual property (IP) rights
• Technology connectivity
• Solution doesn’t fit into organizational ecosystem

Here are some common pitfalls with agile IT outsourcing:

• The customer procures the “agile” project via traditional strategies
• The customer takes a Water-Scrum-Fall approach
• The customer governs the service provider via a traditional approach
• The customer really isn’t agile
• The service provider really isn’t agile
• Neither are agile
• Agile is based on trust, yet it behooves you to not trust the service provider

Here are some common pitfalls with geographically distributed teams:

• Project success rates drop as geographic distribution increases
• Communication challenges
• Time zone differences
• Cultural differences
• Customer unwilling to invest in travel

How Disciplined Agile Delivery Supports Outsourcing

Disciplined Agile Delivery (DAD) provides several important mechanisms which enable agile IT outsourcing:

1. Flexible, process goal driven approach.  A team where the work has been outsourced to them will work differently than an insourced team.  A globally distributed team will work differently than a co-located team.  To be successful at agile IT outsourcing you need to adopt a process that is sufficiently flexible to meet the needs of the situation that you face.  DAD’s process goal-driven strategy provides this flexibility while being easy to tailor.  Context counts.
2. Hybrid framework.  The DA toolkit is a hybrid that adopts strategies from a variety of sources, including Scrum, Extreme Programming, Agile Modeling, Unified Process, Kanban, SAFe, traditional sources, and many more.  As a result it provides the robust advice that you require to address the complexities of agile outsourcing.  Choice is good.
3. Full delivery lifecycles. The DA toolkit supports four full delivery lifecycles – An agile Scrum-based on, a Lean Kanban-based on, an exploratory Lean Startup lifecycle, and a continuous delivery lifecycle – that show how to approach IT delivery from beginning to end in a streamlined and lightweight manner.
4. Risk-value lifecycle.  The DA toolkit explicitly includes proven risk reduction techniques in its delivery lifecycles.  These strategies include coming to a stakeholder vision early in the lifecycle, proving the architecture with working code, and delivering a potentially consumable solution on a regular basis.
5. Enterprise awareness.  The DA toolkit promotes the philosophy of enterprise awareness – DAD teams should do what’s best for the customer organization, not what is convenient for them.  This means that delivery teams should adopt organizational conventions whenever appropriate, reuse existing assets (including legacy data sources), follow the organization’s business roadmap, and follow the technology roadmap.
6. Explicit governance.  The DA toolkit explicitly address how to govern agile teams.  The increased risks associated with outsourcing require a more sophisticated approach to governance than what you would require for an insourced delivery team.

Disciplined Strategies to Make Agile Outsourcing Work

Our experience is that the reason why many customer organizations struggle with agile outsourcing is because their intuition (which typically reflects the traditional mindset) leads them astray.  The following table indicates what traditional intuition often leads you to do and the corresponding disciplined agile strategies that we recommend.

Table 2. Traditional intuition vs. what you should actually do.

When Disciplined Agile Outsourcing Makes Sense

We’ve found that for agile outsourcing to be successful you must first meet all three of the following criteria:

1. You are already successful at insourced agile.
2. You understand and accept the risks involved with outsourcing.
3. You are prepared to address those risks.

Our experience is that to make agile and CMMI co-exist effectively is that four things need to occur:

1. You need to adopt an enterprise view of IT.  Adopting an enterprise view of IT can require an open mind be both CMMI and agile practitioners.  An important implication of enterprise awareness is that different teams are in different situations, therefore they need to tailor their strategy accordingly.  This requires enterprise professionals who work with many project teams to be capable of working with and governing those teams accordingly – they need to work with traditional teams in a traditional manner and agile teams in an agile manner.   For agilists being enterprise aware can be very difficult at first due to the prevalence of a project mindset within the development community.   Agilists need to become disciplined enough to leverage and enhance the existing software and data infrastructure, to follow common guidelines, to work with enterprise professionals such enterprise architects and operations professionals, and to be governed effectively.
2. You need to focus on quantifiable business value.  Your organization must focus on delivering quantifiable business value to your stakeholders in all activities that you perform.  In other words, perform real process and organization improvement at squeeze out the needless bureaucracy that is all too prevalent in CMMI environments.  We have yet to work with a CMMI organization, including those rated at L4 and L5, that didn’t have huge opportunities for improving their productivity by adopting more agile ways of working.   For example, we often run into existing development processes where a requirements specification is created, then test plans and test cases are written so that the solution may be validated, and traceability maintained between these artifacts (and more) for good measure.  Yes, business value is being delivered via this process, but we can work more effectively while achieving the same goals.  For example, but adopting an acceptance-test driven approach the acceptance tests become both the detailed tests and detailed requirements specification, with full requirements-to-test traceability between them with no extra work to be performed.   By working smarter, not harder, you not only reduce the work required to provide the same business value as before but you do so with a shorter feedback cycle between requirement elicitation and implementation, thereby reducing project risk.
3. Agile teams must adopt a full delivery lifecycle.  Minimally this lifecycle includes project/team initiation, construction, and deployment activities although enterprise activities such as portfolio management, enterprise architecture, asset management, operations, support, and others should also be considered.  The Disciplined Agile (DA) toolkit includes four delivery lifecycles: The Agile/Basic Lifecycle based on Scrum, The Advanced/Lean Lifecycle based on Kanban, The Exploratory Lifecycle based on Lean Startup, and the Continuous Delivery Lifecycle.
4. You need to adopt a hybrid approach. Many agile methodologies – including Scrum, XP, AM, Agile Data, Kanban, and more – focus on a subset of the activities required to be CMMI compliant.   Before the advent of DAD you needed to cobble together your own agile methodology to get the job done.   DAD adopts ideas from Scrum, Agile Modeling, Agile Data, XP, Kanban, Lean Software Development, and many more.  The bottom line is that if you intend to address all of the CMMI process areas you will need to adopt a hybrid approach such as DAD or do the work to invent your own.

In the next blog in this series we will explore how Disciplined Agile Delivery (DAD) maps to the CMMI framework.

We’re occasionally asked whether agile and CMMI are compatible, so we thought we’d write a short blog posting on the subject.  The quick answer is yes, but you need to know what you’re doing.  In this article we explore whether organizations are actually combining agile and CMMI in practice and then address some of the rhetoric around this topic.

Survey Says…

The Dr. Dobbs Journal (DDJ) Summer 2012 State of the IT Union Survey examined this issue.  The goal of the survey was to explore whether organizations were successful or unsuccessful at various levels of the scaling factors called out in the Situation Context Framework (SCF).  One of the SCF scaling factors is regulatory compliance, including both legal compliance such as Food and Drug Administration (FDA) compliance as well as self-imposed compliance such as CMMI or ISO 900X.  This survey found that of the respondents whose organizations had achieved success apply agile techniques in practice, 44% indicated that one or more of their project teams had done so when self-imposed compliance requirements were in place.  Of the respondents whose organizations had experienced one or more failed agile projects, 30% indicated that one or more of their projects had self-imposed compliancy requirements.  More recently the DDJ Spring 2014 State of the IT Union Survey found that 44% of agile software development teams (and 43% of non-agile teams) face some sort of compliancy requirement.  The following figure shows that agile teams, just like non-agile teams, are in fact working at scale.

The survey results lead me to three important observations.  First and foremost, people are in fact successfully applying agile and CMMI together.  Second, it can be a rocky road when doing so because some organizations are running into problems.   Three, there isn’t any blatantly obvious evidence for or against applying the two together.   Granted, this third observation is based on averages – your organization may have very good reasons to apply the two together.  In particular, I suspect that the organizations applying CMMI and agile together are the ones where they already have a strong-CMMI culture and are now in the process of increasing their productivity through agile and lean techniques.

Reality Over Rhetoric

One only has to spend some time online to discover that when it comes to applying agile and CMMI together there is some questionable rhetoric being bandied about.  We feel it’s important to surface this rhetoric and describe the reality of the situation.  Common agile CMMI rhetoric includes:

1. Agile and CMMI are incompatible. This is clearly not the case as we learned in the aforementioned surveys.  A quick web search results in many publications on the topic, including case studies. From what we’ve seen most of this problem stems from agile protagonists not understanding CMMI and CMMI protagonists not understanding agile.
2. Scrum is CMMI level 5. This is nothing more than marketing hogwash.  The reality is that Scrum is a very, very small part of what you need to do to succeed at agile. Scrum’s focus is on some aspects of project leadership and requirements management, and it relies on other methodologies such as Extreme Programming (XP), Agile Modeling (AM), Unified Process (UP), and many others to fill in the blanks. Yes, Scrum can be used in CMMI environments but Scrum on its own clearly doesn’t even address all CMMI L2 issues let alone higher levels. Similarly, agile methods such as XP and AM can also be applied in CMMI environments to address portions of one or more process areas in an agile manner.
3. CMMI doesn’t add value. Empirically you can observe that this is clearly not the case by simply hopping on a flight to Bangalore to see how the Indian IT service providers have leveraged CMMI into a multi-billion dollar industry.   Furthermore there are numerous studies that have shown that as organizations move up CMMI levels their productivity improves (although some have shown that productivity peaks upon achieving CMMI L3, so be careful).  Our experience is that the secret is to keep your CMMI implementation as agile as possible.
4. CMMI equals needless bureaucracy. The way that an organization addresses CMMI compliancy is up to them. They can choose to adopt a documentation-heavy strategy, which many unfortunately do, or they can choose to adopt a more streamlined agile approach. Many agilists have had very bad experiences in heavy CMMI shops and in many cases that is their only CMMI experience, hence the bitterness regarding CMMI.