Disciplined Agile

On June 25, 2015 we gave a webinar entitled Disciplined Agile Outsourcing: Making it Work for Both the Customer and the Service Provider (click on the link to see the slides on Slideshare).  We will likely run the webinar again as it was very well received.  We had hoped to record it but unfortunately weren’t able to save the file because the software crashed.  Hopefully it will work out next time.  During the webinar we promised to write a blog summarizing it and addressing any remaining questions from the audience.  This is that blog.

Agile and Outsourcing?  Really?

There are good reasons why organizations are taking an agile approach to outsouring IT delivery projects.  The following table overviews the motivation and concerns of both customer organizations and service providers.

Table 1. Comparing the Viewpoints of Customers and Service Providers

Here are some interesting statistics about outsourcing:

• Gartner’s 2014 IT Outsourcing report estimated IT outsourcing to be $288 billion in 2013 and to grown by 5.9% compounded annually from 2013 through 2018.
• Deloitte’s 2014 Global Outsourcing survey found that 53% of organizations are outsourcing some of their IT function; 26% of respondents who do not outsource today plan to do so (supporting Garter’s growth forecast); 79% of respondents DO NOT believe their service providers are too expensive; 49% of respondents say their service providers are reactive vs proactive (e.g. they are basically order takers, not partners); and IT outsourcing activity is expected to increase.
• The Ambysoft 2013 Agile Outsourcing survey found that half of organizations who are “doing agile” are also involving outsourcing in some way

The Challenges Associated with Agile IT Outsourcing

Here are some common pitfalls that organizations run into with IT outsourcing:

• Organizational culture differences
• Expectations mismatch between the customer and the service provider
• The customer underestimates difficulty of managing outsourced projects
• Total cost of the solution isn’t considered
• Total value of the solution isn’t considered
• Transition to the operations team is mismanaged
• Over-reliance on documentation
• Software licensing issues
• Learning curve for service provider underestimated
• The service provider is understaffed
• Some aspects, e.g. security, cannot be outsourced
• Intellectual property (IP) rights
• Technology connectivity
• Solution doesn’t fit into organizational ecosystem

Here are some common pitfalls with agile IT outsourcing:

• The customer procures the “agile” project via traditional strategies
• The customer takes a Water-Scrum-Fall approach
• The customer governs the service provider via a traditional approach
• The customer really isn’t agile
• The service provider really isn’t agile
• Neither are agile
• Agile is based on trust, yet it behooves you to not trust the service provider

Here are some common pitfalls with geographically distributed teams:

• Project success rates drop as geographic distribution increases
• Communication challenges
• Time zone differences
• Cultural differences
• Customer unwilling to invest in travel

How Disciplined Agile Delivery Supports Outsourcing

Disciplined Agile Delivery (DAD) provides several important mechanisms which enable agile IT outsourcing:

1. Flexible, process goal driven approach.  A team where the work has been outsourced to them will work differently than an insourced team.  A globally distributed team will work differently than a co-located team.  To be successful at agile IT outsourcing you need to adopt a process that is sufficiently flexible to meet the needs of the situation that you face.  DAD’s process goal-driven strategy provides this flexibility while being easy to tailor.  Context counts.
2. Hybrid framework.  The DA toolkit is a hybrid that adopts strategies from a variety of sources, including Scrum, Extreme Programming, Agile Modeling, Unified Process, Kanban, SAFe, traditional sources, and many more.  As a result it provides the robust advice that you require to address the complexities of agile outsourcing.  Choice is good.
3. Full delivery lifecycles. The DA toolkit supports four full delivery lifecycles – An agile Scrum-based on, a Lean Kanban-based on, an exploratory Lean Startup lifecycle, and a continuous delivery lifecycle – that show how to approach IT delivery from beginning to end in a streamlined and lightweight manner.
4. Risk-value lifecycle.  The DA toolkit explicitly includes proven risk reduction techniques in its delivery lifecycles.  These strategies include coming to a stakeholder vision early in the lifecycle, proving the architecture with working code, and delivering a potentially consumable solution on a regular basis.
5. Enterprise awareness.  The DA toolkit promotes the philosophy of enterprise awareness – DAD teams should do what’s best for the customer organization, not what is convenient for them.  This means that delivery teams should adopt organizational conventions whenever appropriate, reuse existing assets (including legacy data sources), follow the organization’s business roadmap, and follow the technology roadmap.
6. Explicit governance.  The DA toolkit explicitly address how to govern agile teams.  The increased risks associated with outsourcing require a more sophisticated approach to governance than what you would require for an insourced delivery team.

Disciplined Strategies to Make Agile Outsourcing Work

Our experience is that the reason why many customer organizations struggle with agile outsourcing is because their intuition (which typically reflects the traditional mindset) leads them astray.  The following table indicates what traditional intuition often leads you to do and the corresponding disciplined agile strategies that we recommend.

Table 2. Traditional intuition vs. what you should actually do.

When Disciplined Agile Outsourcing Makes Sense

We’ve found that for agile outsourcing to be successful you must first meet all three of the following criteria:

1. You are already successful at insourced agile.
2. You understand and accept the risks involved with outsourcing.
3. You are prepared to address those risks.

Our experience is that to make agile and CMMI co-exist effectively is that four things need to occur:

1. You need to adopt an enterprise view of IT.  Adopting an enterprise view of IT can require an open mind be both CMMI and agile practitioners.  An important implication of enterprise awareness is that different teams are in different situations, therefore they need to tailor their strategy accordingly.  This requires enterprise professionals who work with many project teams to be capable of working with and governing those teams accordingly – they need to work with traditional teams in a traditional manner and agile teams in an agile manner.   For agilists being enterprise aware can be very difficult at first due to the prevalence of a project mindset within the development community.   Agilists need to become disciplined enough to leverage and enhance the existing software and data infrastructure, to follow common guidelines, to work with enterprise professionals such enterprise architects and operations professionals, and to be governed effectively.
2. You need to focus on quantifiable business value.  Your organization must focus on delivering quantifiable business value to your stakeholders in all activities that you perform.  In other words, perform real process and organization improvement at squeeze out the needless bureaucracy that is all too prevalent in CMMI environments.  We have yet to work with a CMMI organization, including those rated at L4 and L5, that didn’t have huge opportunities for improving their productivity by adopting more agile ways of working.   For example, we often run into existing development processes where a requirements specification is created, then test plans and test cases are written so that the solution may be validated, and traceability maintained between these artifacts (and more) for good measure.  Yes, business value is being delivered via this process, but we can work more effectively while achieving the same goals.  For example, but adopting an acceptance-test driven approach the acceptance tests become both the detailed tests and detailed requirements specification, with full requirements-to-test traceability between them with no extra work to be performed.   By working smarter, not harder, you not only reduce the work required to provide the same business value as before but you do so with a shorter feedback cycle between requirement elicitation and implementation, thereby reducing project risk.
3. Agile teams must adopt a full delivery lifecycle.  Minimally this lifecycle includes project/team initiation, construction, and deployment activities although enterprise activities such as portfolio management, enterprise architecture, asset management, operations, support, and others should also be considered.  The Disciplined Agile (DA) toolkit includes four delivery lifecycles: The Agile/Basic Lifecycle based on Scrum, The Advanced/Lean Lifecycle based on Kanban, The Exploratory Lifecycle based on Lean Startup, and the Continuous Delivery Lifecycle.
4. You need to adopt a hybrid approach. Many agile methodologies – including Scrum, XP, AM, Agile Data, Kanban, and more – focus on a subset of the activities required to be CMMI compliant.   Before the advent of DAD you needed to cobble together your own agile methodology to get the job done.   DAD adopts ideas from Scrum, Agile Modeling, Agile Data, XP, Kanban, Lean Software Development, and many more.  The bottom line is that if you intend to address all of the CMMI process areas you will need to adopt a hybrid approach such as DAD or do the work to invent your own.

In the next blog in this series we will explore how Disciplined Agile Delivery (DAD) maps to the CMMI framework.

We’re occasionally asked whether agile and CMMI are compatible, so we thought we’d write a short blog posting on the subject.  The quick answer is yes, but you need to know what you’re doing.  In this article we explore whether organizations are actually combining agile and CMMI in practice and then address some of the rhetoric around this topic.

Survey Says…

The Dr. Dobbs Journal (DDJ) Summer 2012 State of the IT Union Survey examined this issue.  The goal of the survey was to explore whether organizations were successful or unsuccessful at various levels of the scaling factors called out in the Situation Context Framework (SCF).  One of the SCF scaling factors is regulatory compliance, including both legal compliance such as Food and Drug Administration (FDA) compliance as well as self-imposed compliance such as CMMI or ISO 900X.  This survey found that of the respondents whose organizations had achieved success apply agile techniques in practice, 44% indicated that one or more of their project teams had done so when self-imposed compliance requirements were in place.  Of the respondents whose organizations had experienced one or more failed agile projects, 30% indicated that one or more of their projects had self-imposed compliancy requirements.  More recently the DDJ Spring 2014 State of the IT Union Survey found that 44% of agile software development teams (and 43% of non-agile teams) face some sort of compliancy requirement.  The following figure shows that agile teams, just like non-agile teams, are in fact working at scale.

The survey results lead me to three important observations.  First and foremost, people are in fact successfully applying agile and CMMI together.  Second, it can be a rocky road when doing so because some organizations are running into problems.   Three, there isn’t any blatantly obvious evidence for or against applying the two together.   Granted, this third observation is based on averages – your organization may have very good reasons to apply the two together.  In particular, I suspect that the organizations applying CMMI and agile together are the ones where they already have a strong-CMMI culture and are now in the process of increasing their productivity through agile and lean techniques.

Reality Over Rhetoric

One only has to spend some time online to discover that when it comes to applying agile and CMMI together there is some questionable rhetoric being bandied about.  We feel it’s important to surface this rhetoric and describe the reality of the situation.  Common agile CMMI rhetoric includes:

1. Agile and CMMI are incompatible. This is clearly not the case as we learned in the aforementioned surveys.  A quick web search results in many publications on the topic, including case studies. From what we’ve seen most of this problem stems from agile protagonists not understanding CMMI and CMMI protagonists not understanding agile.
2. Scrum is CMMI level 5. This is nothing more than marketing hogwash.  The reality is that Scrum is a very, very small part of what you need to do to succeed at agile. Scrum’s focus is on some aspects of project leadership and requirements management, and it relies on other methodologies such as Extreme Programming (XP), Agile Modeling (AM), Unified Process (UP), and many others to fill in the blanks. Yes, Scrum can be used in CMMI environments but Scrum on its own clearly doesn’t even address all CMMI L2 issues let alone higher levels. Similarly, agile methods such as XP and AM can also be applied in CMMI environments to address portions of one or more process areas in an agile manner.
3. CMMI doesn’t add value. Empirically you can observe that this is clearly not the case by simply hopping on a flight to Bangalore to see how the Indian IT service providers have leveraged CMMI into a multi-billion dollar industry.   Furthermore there are numerous studies that have shown that as organizations move up CMMI levels their productivity improves (although some have shown that productivity peaks upon achieving CMMI L3, so be careful).  Our experience is that the secret is to keep your CMMI implementation as agile as possible.
4. CMMI equals needless bureaucracy. The way that an organization addresses CMMI compliancy is up to them. They can choose to adopt a documentation-heavy strategy, which many unfortunately do, or they can choose to adopt a more streamlined agile approach. Many agilists have had very bad experiences in heavy CMMI shops and in many cases that is their only CMMI experience, hence the bitterness regarding CMMI.

This blog posting has been replaced by:

• Enterprise Architecture Workflow - Internal
• Enterprise Architecture Workflow - External

This blog posting has been replaced by Enterprise Architecture Practices.