Disciplined Agile

Recently we published a blog which provided a Definition for Disciplined DevOps.  In that posting we worked through the potential scope of DevOps to help gain a better understanding of what DevOps is all about.  One point that we made was that there was no consistent definition of DevOps in the industry due to various reasons.  In this posting we explore those reasons.

We see several key forces in the current marketplace which makes it difficult to settle on a common definition:

1. Specialized IT practitioners.  Many IT professionals still tend to specialize – someone will choose to focus on being a programmer, an operations engineer, an enterprise architect, a database administrator (DBA) and so on.  As a result they tend to see the world through the lens of their speciality.  Programmers will focus on the software development aspects of DevOps, operations engineers the operations aspects of DevOps, enterprise architects on the long-term planning and modelling aspects, and DBAs on the data management aspects.  Few people are looking at the overall “big picture”.
2. Agilists are focused on continuous delivery.  Right now agile and lean developers are investing a lot of effort to figure out continuous delivery practices so as to streamline the regular deployment of value into production.  Advanced teams are releasing daily if not several times a day due to adoption of practices such as automated regression testing, continuous integration (CI), and continuous delivery (CD).  As a result most of the DevOps discussion in these communities focuses on these topics, sometimes straying into other practices such as canary testing, feature toggles, and production monitoring frameworks.  Clearly important techniques, but still not covering the full potential range of DevOps.  These practices and more are described later in this article.
3. Operations professionals are often frustrated.  Many operations groups are overwhelmed already with the rate of updates being foisted upon them by development teams.  This is often exacerbated by the inconsistent use of technologies – the impact of the lack of enterprise awareness within undisciplined development teams is largely felt by the operations group who needs to support the plethora of technology platforms used by the full range of development teams.  Worse yet, the internal operations processes are often based on heavy implementations of ITIL or ITSM and have yet to be streamlined so that operations engineers are in a better position to collaborate with development teams.
4. Tool vendors have limited offerings.  As a result of this the DevOps messaging from tool vendors will focus on just the aspects of DevOps supported by their tools, narrowing the discussion to what they have on offer.  Yes, tools are important, but they are only part of the DevOps picture.  Even if there was a vendor with a full range of tools, and if they actually interoperated smoothly (yes ALM vendors, we’re referring to you), you would still need to understand how to use those tools effectively.  To paraphrase an old saying – A fool with a DevOps tool is still a fool.
5. Service vendors have limited offerings.  Similar to the issues surrounding tool vendors, service vendors are also making great claims about their deep expertise in DevOps.   Upon examination you will often find, like the tool vendors, their definition of DevOps will focus on whatever they can currently support.
6. Tool vendors treat DevOps as a marketing buzzword.  To be blunt, many vendors have taken their existing products, and started marketing them as DevOps products (regardless of how well those products actually support DevOps practices).  Granted, these products may have been very good at supporting traditional ways of working, but when it comes to supporting DevOps they prove to be rather clunky even though they may have added a few new features.
7. The DevOps=Cloud vision.  There is a lot of rhetoric, particularly coming from Cloud vendors, about how cloud-based tooling and deployment environments are critical to success in DevOps.  Yes, having a cloud-based infrastructure clearly enables many DevOps practices and given the choice we prefer to work in an environment which leverages cloud-based technologies whenever appropriate.  But, that doesn’t mean that the cloud is a prerequisite for doing DevOps.

The point is that there are several contributing factors to the lack of agreement within our industry as to what DevOps means in practice.  The implication is that when someone is giving you advice about DevOps that you need to understand the scope of what they’re actually discussing.   Another way to understand what DevOps is and how it may apply to your organization is to explore the various DevOps strategies and practices available to you, which we’re doing in other blog postings.  Please see our first post in that series which overviews General DevOps Strategies.

In a previous blog posting we overviewed the concept of Disciplined DevOps, which is the streamlining of IT solution development and IT operations activities, as well as supporting enterprise activities.  In this blog posting we begin to overview strategies that support DevOps.  This posting overviews general strategies, and future postings will describe development, operations, release management, data management, and enterprise architecture strategies.

There are several “general” strategies that support DevOps:

1. Collaborative work.  A fundamental philosophy of DevOps is that developers, operations staff, and support people must work closely together on a regular basis. An implication is that they must see one other as important stakeholders and actively seek to work together. A common practice within the agile community is “onsite customer,” adopted from Extreme Programming (XP), which motivates agile developers to work closely with the business. Disciplined agilists take this one step further with the practice of active stakeholder participation, which says that developers should work closely with all of their stakeholders, including operations and support staff–not just business stakeholders. This is a two-way street: Operations and support staff must also be willing to work closely with developers.
2. Automated dashboards. The practice of using automated dashboards is called IT intelligence, effectively the application of business intelligence (BI) strategies for IT. There are two aspects to this, development intelligence and operational intelligence. Development intelligence requires the use of development tools that are instrumented to generate metrics; for example, your configuration management (CM) tools already record who checked in what and when they did it. Continuous integration tools could similarly record when a build occurred, how many tests ran, how long the tests ran, whether the build was successful, how many tests we successful, and so on. This sort of raw data can then be analyzed and displayed in automated dashboards. Operational intelligence is an aspect of application monitoring discussed previously. With automated dashboards, an organization’s overall metrics overhead can be dramatically reduced (although not completely eliminated because not everything can be automated). Automated dashboards provide real-time insight to an organization’s governance teams.
3. Integrated configuration management. With an integrated approach to configuration management (CM), development teams not only apply CM at the solution level as is customary, they also consider production configuration issues between their solution and the rest of your organization’s infrastructure. This can be a major change for some developers because they’re often used to thinking about CM only in terms of the solution they are currently working on. In a DevOps environment, developers need to be enterprise aware and look at the bigger picture. How will their solution work with and take advantage of other assets in production? Will other assets leverage the solution being developed? The implication is that development teams will need to understand, and manage, the full range of dependencies for their product. Integrated configuration management enables operations staff to understand the potential impact of a new release, thereby making it easy to decide when to allow the new release to occur.
4. Integrated change management.  From an IT perspective, change management is the act of ensuring successful and meaningful evolution of the IT infrastructure to better support the overall organization. This is tricky enough at a project-team level because many technologies, and even versions of similar technologies, will be used in the development of a single solution. Because DevOps brings the enterprise-level issues associated with operations into the mix, an integrated change management strategy can be far more complex, due to the need to consider a large number of solutions running and interacting in production simultaneously. With integrated change management, development teams must work closely with operations teams to understand the implications of any technology changes at an organization level. This approach depends on the earlier practices of active stakeholder participation, integrated configuration management, and automated testing.
5. Training, education, and mentoring.  As you would expect, people will need help to learn and adopt your DevOps strategies.
6. Continuous improvement.  Disciplined agile teams strive to learn from their experiences as well as from others so that they can continuously improve the way that they work together, including how they approach DevOps.
7. One team.  An important aspect of the DevOps mindset is shifting away from a “them and us mindset” to an “us mindset.”  We all work together as a single, streamlined team.  An extreme form of this is the “you build it, you run it” philosophy where there are no separate development, operations, data administration teams but instead product teams who are responsible for the entire lifecycle of a product.

Our next blog posting in this series will overview development-oriented strategies.

We recently published an article describing how the Disciplined Agile Delivery (DAD) framework is being extended to cover Portfolio Management activities.  The following diagram depicts the DAD goal diagram for Portfolio Management, and as you would expect your organizations has a range of options to choose from.  In this diagram we use the term endeavor to refer to a project, product, or experiment.

The article describes each one of these process factors – Identify Endeavors, Explore Potential Endeavors, and so on – in greater detail.  A future version of the article will describe the strategies/practices associated with each factor.  We have also included a high-level workflow diagram, see below, to overview from the point of view of the Portfolio Management process blade how it fits into the rest of the Disciplined Agile (DA) toolkit.

An interesting aspect of the flow diagram is that it shows the relationship of Portfolio Management to other IT Plan activities.  For example, Enterprise Architecture provides a Technology Roadmap and Product Management provides a Business Roadmap and an indication of stakeholder priorities – all critical information that are inputs into the efforts around prioritizing potential endeavors.  The point is that this diagram reflects workflow, not organizational design.  For example, in some companies there is no Product Management team, instead responsibility for the Product Management activities are spread out amongst several teams.  A common strategy is to have the Portfolio Management team responsible for understanding stakeholder priorities and the Enterprise Architecture team responsible for the business roadmap.  Another approach would be to have the Portfolio Management team subsume both of those activities.  A third approach would be to have three teams, one for each of Enterprise Architecture, Product Management, and Portfolio Management.  Different organizations will make different organizational design decisions, and of course these decisions will evolve over time.  As always, context counts.

I hope that you find the more detailed Portfolio Management article to be of interest.

This posting, the first in a series, overviews a disciplined approach to DevOps.  It begins by defining DevOps, no small task given the continued debate within the DevOps community, and then described a disciplined approach to DevOps.

Defining DevOps

For our purposes we propose the following definition:

DevOps is the streamlining of the activities surrounding IT solution development (dev) and IT operations (ops). 

Figure 1 below summarizes this idea, showing how the development and operations groups within your IT department begin to breakdown the barriers between them that inhibit effective collaboration.  In organizations that have not yet adopted a DevOps mindset we say that there is a “DevOps gap.”  This gap results in lengthy solution deployments and hence higher costs to deploy; a long mean time between deployments (MTBD) which is often measured in terms of months; reduced market competitiveness; and reduced ability to govern your IT efforts due to lack of real-time intelligence.  When organizations close the DevOps gap, by adopting strategies described later in this article, they effectively address these challenges and more.  They do this by adopting a mindset that promotes collaborative, learning-centric ways of working that are supported by agile practices and automation.

Figure 1. Closing the gap between IT software development and IT operations.

Defining Disciplined DevOps

What does it mean to take a disciplined approach to DevOps? It requires discipline to do the things that are good for you that you may not normally choose to do.  Given this, we propose the following definition:

Disciplined DevOps is the streamlining of IT solution development and IT operations activities, and supporting enterprise-IT activities, to provide more effective outcomes to an organization.

To understand the difference, let’s work through the scope implications of what DevOps could mean to your organization.  Figure 2 depicts an initial strategy for scoping your DevOps efforts, basically rethinking the interactions between your Development teams and your IT Operations team.  Notice how in Figure 2 we’ve started to use the term “IT Operations” and not just Operations.  This is because many organizations choose to distinguish between their IT operations activities and their business operations activities (such as selling, distribution, marketing, and so on).  Of course IT operations exists to support business operations.

Figure 2. An initial view of DevOps.

Some organizations will start with the strategy of Figure 3, particularly when there is a separate group responsible for managing the release/deployment of systems into production.  These Release Management teams are often, but not always, a subgroup of your IT Operations team.   Product companies that are creating solutions for the marketplace will typically have a separate Release Management team because the release effort includes both IT release activities, in particular releasing updated solutions into production, as well as customer-facing release activities (such as marketing, sales, and so on).  A disciplined DevOps strategy streamlines both the IT and the business aspects of your Release Management activities.

Figure 3. DevOps with an explicit Release Management activity.

Although Figure 3 is clearly a step in the right direction, it isn’t complete.  A Disciplined DevOps strategy will streamline the supporting activities performed by your Enterprise Architecture and Data Management teams.  Enterprise architecture activities that support DevOps includes the definition, support, and evolution of technical and business roadmaps which guide the efforts of the rest of the organization.  This in turn supports the creation of a common and consistent technical infrastructure within your production environments, enabling common DevOps practices such as continuous deployment, automated end-to-end regression testing, and operational monitoring.  Supporting data management activities include the definition, support, and evolution of data and information standards and guidelines; the creation, support, evolution, and operation of data sources of record within your organization; and the creation, support, evolution, and operation of  data warehouse (DW)/business intelligence (BI) solutions.  All of these activities will be described in greater detail later in future blog postings.

Figure 4. Disciplined DevOps – An enterprise view.

All three views are valid, your organization needs to decide which one is most appropriate for them given their current situation.  Figures 2 through 4 have been presented from the point of view of a potential DevOps maturity model – you could start with the initial strategy presented in Figure 2, then improve it to address release management (Figure 3) and finally evolve your strategy to address other IT activities such as enterprise architecture and data management (Figure 4).

In following blog postings we will show how DevOps strategies are explicitly addressed by the Disciplined Agile (DA) toolkit, thereby taking some of the mystery about how DevOps works in practice.

We are often asked what makes a Disciplined Agile (DA) coach effective.  The features to look for in a DA coach include:

1. People skills.  First and foremost, effective DA coaches need solid people skills.  They need to be prepared to work with a wide range of people coming from different backgrounds, with different learning preferences, and with different learning goals.  As a result DA coaches need to be empathetic, patient, respectful, and open-minded.
2. Experience.  Good coaches understands the situation that you face, and more importantly understands how to guide you to tailor your strategy.  Effective coaches have many many data points from their experiences over the years, and as such they can recognize patterns quickly and provide appropriate advice to those their are coaching.  We’ve seen people who are very good agile coaches for small, co-located teams get into serious trouble the first time they need to deal with scaling factors such as large team size, geographic distribution, or regulatory compliance.   We’ve also seen good development team coaches flounder when they first start to address, in a meaningful way, the Agile IT issues faced by organizations applying agile across their entire IT department.  This is one of the reasons why we suggest that Transformation coaches be Certified Disciplined Agile Coaches (CDACs) as they need significant experience and knowledge to be successful.  The implication is that when you’re hiring a coach, make sure they’ve worked in environments similar to yours otherwise you run the risk of paying for their learning experiences.
3. Pragmatism.  As Mark Lines astutely pointed out a few months ago, DA is pragmatic agile.  Effective DA coaches are willing and able to work closely with the people that they’re coaching, providing practical advice that they follow themselves. They also like to have real-time measures that reveal how their team(s) are doing, enabling them to make fact-based suggestions to help their teams.
4. Knowledge.  It’s reasonable to expect a DA coach to be very knowledgeable about DA and agile in general.  Development team coaches are at least Certified Disciplined Agile Practitioners (CDAPs) and better yet CDACs.  To earn a CDAP you need to have at least two years agile experience, clearly a bare minimum for someone in a coaching role, and be able to pass a challenging test which explores their understanding of the DA toolkit.  CDACs need to have five or more years of experience and must pass a board-level interview.  These are meaningful certifications that people must work for to earn, and are a clear indication that holders of such certification have the requisite DA knowledge.   Transformation coaches, who coach an organization’s executive team through the process of transitioning to agile, should be CDACs.
5. Skill.  Development team coaches must be skilled in fundamental agile techniques such as regression testing,  continuous integration (CI), iteration/sprint planning, look-ahead modelling and planning, requirements envisioning, and many more.  A good team coach should also be skilled in “advanced” agile techniques such as test-driven development (TDD), behaviour driven development (BDD), and continuous deployment.  Transformation coaches should be skilled in organizational change management as well as the fundamentals of IT-level activities such as enterprise architecture, data management, operations, portfolio management, and others.
6. Leadership.  In addition to solid people skills, good coaches often need good leadership skills too as they need to be adept at convincing people to follow their advice.  Team coaches will often be Team Leads, or at least be working closely with the Team Lead, to help lead the team in making the “hard decisions” required to successfully learn the agile mindset.
7. Flexibility.  A fundamental concept of the DA toolkit is that you need to tailor it to meet the needs that you face.  The implication is that DA coaches need to be agile to go beyond the advice in prescriptive methods such as Scrum or SAFe.  Instead of working from a prescriptive playbook, DA coaches will leverage DA’s goal-driven strategy to help guide teams make process-oriented and organization-oriented choices that are right for them. In short, just because someone has several years of Scrum coaching you can’t count on them having the background to be a good DA coach because they may only understand Scrum strategies and not the full range of agile and lean strategies supported by the DA toolkit.