Disciplined Agile

The Disciplined Agile (DA) toolkit describes strategies for how an organization’s IT group can support a lean enterprise.  An important part of this is to have an effective IT operations strategy, and to do that the people involved need to have what we call a “lean IT operations mindset.”  The philosophies behind such a mindset include:

1. Run a trustworthy IT ecosystem.  At a high level the goal is to “keep the lights on.”  At a detailed level anyone responsible for IT operations wants to run an IT ecosystem that is sufficiently secure, resilient, available, performant, usable, and environmentally friendly.  Part of running a trustworthy ecosystem is monitoring running services so as to identify and hopefully avoid potential problems before they occur.  For some systems, and perhaps for your IT ecosystem as a whole, you may have service level agreements (SLAs) in place with your end users that guarantee a minimum level of trustworthiness.
2. Focus on the strategic (long-term) over the tactical (short-term).  Anyone responsible for IT operations needs to have a very good understanding between the long-term implications of a decision versus the short-term conveniences.  A classic example of this right now is the preference of people building micro-services to use what they believe to be the best technologies for each service.  This makes a lot of sense from the narrow viewpoint of that service and it often proves to be incredibly convenient, and fun, for the developers because they often get to work with new technologies.  However, from an operational point of view you end up with a mishmash of technologies that must be operated and evolved over time, resulting in a potential maintenance nightmare.  Yes, you will still make some short-term decisions but you should do so intelligently.  Too great a focus on the long term results in a stagnant IT ecosystem, too great a focus on short-term decisions results in operations teams who spend all their time fighting fires.  The long-term technical vision for your organization is developed by your Enterprise Architecture efforts and the long-term business vision comes from your Product Management activities.
3. Streamline the overall flow of work.  This arguably should be part of everyone’s mindset, but it is particularly important for people doing IT operations work.  IT operations has traditionally been a bottleneck in many organizations, often the result of the need to run a trustworthy ecosystem and to focus on long-term considerations, hence the need to focus on streamlining the overall flow of work. BUT, this isn’t just operational work that we need to streamline, but the overall flow of work into, within, and out of IT operations.  In this case we need a disciplined approach to DevOps that takes all aspects of the development-operations lifecycle into account, including the support of multiple development lifecycles (not just continuous delivery), the release management process, and the operational aspects of data management.  Of course, streamlining the flow of work goes beyond development-operations and is an important goal of your organization’s continuous improvement strategy.
4. Help end-users succeed.  An important goal of people performing operations activities is to ensure that your end users are successfully using your IT systems.  It doesn’t matter how well your systems are built, or how trustworthy they are, if your end users are unable or unwilling to use them effectively.  End users are going to need help – you need to be prepared to provide a support function.
5. Standardization without stagnation.  The more standardized your IT ecosystem is the easier it will be to run, to release new functionality into, and to find and fix problems if they should arise.  However, too much standardization can lead to stagnation where it becomes very difficult to evolve your ecosystem.  You will need to work very closely with people performing enterprise architecture and product management activities to ensure that you understand the long term vision and are working towards it.
6. Regulate releases into production.   Most DevOps strategies reflect the viewpoint of a single product team.  But what about the viewpoint of your overall IT ecosystem, which may comprise hundreds of products?  An interesting question to ask is what is the WIP limit for releases across your overall ecosystem?  In other words, what rate of change can your infrastructure, and your stakeholder community, bear?  In the Disciplined Agile (DA) toolkit this philosophy is an important driver of the Release Management process blade.  Furthermore, some regulatory compliance regimes call out a separation of concerns pertaining to release management – the people building a product are not allowed to release the product into production, someone else must make that decision and do the work (even if “the work” is merely pressing a button to run a script).
7. Sufficient documentation.  Yes, there will be some documentation maintained about your IT ecosystem.  Hopefully this documentation is concise, accurate, and high-level.  Common documentation includes an overview(s) of your infrastructure, release procedures (even if fully automated, there’s still some overview documentation and training), and high-level views of critical aspects of your infrastructure including security, data architecture, and network architecture.  Organizations that operate in regulated industries will of course need to comply to the documentation requirements of the appropriate regulations.  When infrastructure components are discoverable and self-documenting there is a lesser need for external documentation, but there is still a need.  Any documentation that you do create should be maintained under configuration management (CM) control.

Future blog postings in this series about IT operations and support will explore topics such as why you need IT operations and support, what activities you perform, and the workflow of doing such.

On May 17 2016 I gave a webinar entitled Agile Enterprise Architecture: Disciplined and Pragmatic Strategies (at this link you can watch a recording and download a PDF of the slides). During the webinar I received many questions, some of which we had time to answer but some which we did not.  Regardless, here are all of the questions and my answers.  I’ve organized them into the following topics:

1. Long vs. Short Term
2. Deferring Commitment
3. Transitioning to Agile EA
4. General

1. Long vs. Short Term

What about longer term? How do you see the role of EA on portfolio creation?  As you can see in Disciplined Agile’s Enterprise Architecture process blade one of the primary tasks of the EA activity is to consider the longer term. The enterprise architects will collaborate to do so, and will help bring their vision to their stakeholders in various ways.  As you can see in the workflow, the EAs will interact with the Portfolio Management effort (and many others).

How do you balance the needs of today vs. the needs of tomorrow? Would you wait for business to identify it as a priority, which may be too late… or look to get ahead of the curve and build something that can scale for anticipated growth? Experienced enterprise architects will definitely think about the long term needs of the organization. Disciplined Agile EAs will do so but will wait to act to the most appropriate time to do so.

2. Deferring Commitment

Can you point us to some places to read more about deferred commit? Deferring commitment to the last most responsible moment is a concept that comes from lean software development. I recommend the writings of Mary and Tom Poppendieck as well as the book SDLC 3 by Mark Kennaley.

If we defer commitment, how about if we need infrastructure, procurement and so on to implement that architecture? This deferring will delay the entire implementation? No. The advice is to defer commitment to the last most responsible moment.  You need to understand the needs of your stakeholders, which in Disciplined Agile include a wide range of people, and then act accordingly. The problem is that traditionalists will act too early and thus add unnecessary risk.

A good observation from Valentin Tudor-Mocanu is that to be able to defer decisions we need a design that separates the concerns (adaptive design).

3. Transitioning to Agile EA

Is there a strategy to change the mindset of enterprise architects on a large organization?  Yes. It first begins with education.  We offer a one-day workshop called Agile Enterprise Architecture: Disciplined and Pragmatic Strategies that does a very good job of working through the fundamentals.  The next step is coaching of the EA team themselves, and of the teams that they interact with, to help them gain the mindset and habits required to work in a more effective agile/lean manner.  We also provide such coaching services.

One specific question related to a service or consulting organization: One of the specific requirements for us to is to deliver architecture as a deliverable. How can agile can help us?  Can we apply sprints there?  Can we apply any architecture principles? I’ll assume you meant an architecture model as a deliverable. The answer is yes, you can easily create an architecture model in an agile manner. I highly suggest that you visit the Agile Modeling site and read the advice about agile architecture strategies and to agile documentation strategies.

4. General

Can you talk about metrics in agile EA and Architecture in general?  In the Disciplined Agile (DA) toolkit we suggest that you adopt a lightweight Goal-Question-Metric (GQM) approach to your metrics strategy. The fundamental idea behind GQM is that you first identify a goal that you would like to achieve, a set of questions whose answers are pertinent to determining how well you’re achieving that goal, and then the metric(s) that could help you to answer each question. In short, the metrics that you gather will vary based on your need.

Given that Feedback cycle plays an important role, why does DAD say “Reduce feedback cycle”? Is there a pre-defined cycle frequency?  You should find the blog posting Reducing the feedback cycle requires discipline to be an interesting read.

Some forms of architecture (UX/UI) need a different cadence than the delivery team follows. What is your recommendation for handling this… UX research, etc? I disagree with your premise. I’ve heard this claim in the past and it’s never held water with me.  About 10 years ago I wrote an article entitled Introduction to Agile Usability that overviews how UX activities can appear in the agile lifecycle.  The Lean UX community has clearly gone further than what I originally wrote so you might want to go poking around on the web a bit.

When the business case says “many years before payback”, doesn’t that usually mean that the business case is missing some important elements? (or that the org is really small) ?  My experience is that the business case is usually crap in these situations. There in fact a few things in the software world that may have a multi-year payback period, operating systems come to mind, but they are more the exception to the rule.  The problem with multi-year payback schemes in the software world is that the ecosystem changes so quickly.  During the several years it takes for payback your needs change, there are new options on the market, your organization structure and direction may change, and so on.

I realize I’m asking a history question here, but regarding deferring commitment, a superb method for this that I have often used in the past comes from RUP: architecture mechanisms. (I should know this but) Is this in DAD?  If you read the Enterprise Architecture process blade article you will see that it supports several strategies for capturing and communicating EA ideas such as reference architectures, models, architectural runways, interfaces, and many others. You might also find the Reuse Engineering process blade to be of interest.

Governance is a common thread throughout all aspects of the Disciplined Agile (DA) tool kit. Delivery governance activities were built right into Disciplined Agile Delivery (DAD) v1.x, each of the process blades include a governance process factor, and there is an IT Governance process blade that coordinates all IT governance activities.

The following process goal diagram overviews the potential activities associated with a disciplined approach to IT governance.  These activities are typically performed by a variety of people within your organization. In DA each of the process blades includes activities around governing the activities encapsulated by that blade. For example, the Enterprise Architecture blade includes architectural governance activities and the Data Management blade includes data/information governance activities.

The process factors that you need to consider for IT governance are:

1. Governance view. There are many potential aspects, or perhaps subsets, to IT governance including security governance, data governance, delivery governance, and more. Too many organizations make the mistake of treating each of these issues separately, which is easy enough to do given the different focus of each one. The Disciplined Agile toolkit promotes a more holistic, integrated view that results in a streamlined, consistent, and effective governance strategy. Another common mistake is to focus on portfolio governance over other aspects and have your Portfolio and Project Management Office (PPMO) be responsible for IT governance. PPMO-led governance efforts tend to result in documentation-heavy, bureaucratic strategies instead of lean, risk-based approaches.
2. Develop metrics. Metrics define the measurements that you will take to gain insight into what has happened, what is happening, and hopefully what may happen in the future. The least effective approach to metrics is to have a standard set that all teams are expected to collect whereas the most effective is a context-sensitive approach such as a light-weight implementation of Goal Question Metric (GQM).
3. Measure. The measures themselves can be collected either manually or automatically (usually as the result of tool or system usage). We have found that a combination of the two is required – although automated metrics are inexpensive, accurate, and real-time you cannot always automate all of the measurements that you need to collect.
4. Monitor. Information radiators are sufficient in smaller organizations, but automated dashboards in combination with direct interaction with teams are also needed in modern enterprises. Traditional strategies, such as status reports or status meetings, do not prove to be effective in practice nor do they provide an honest assessment of what is actually occurring.
5. Guidance detail. An easy and important way to support governance within your organization is to have commonly accepted guidance, including standards and guidelines, for teams to follow. We suggest that you keep this guidance lightweight, starting with high-level rules but aiming towards principles whenever possible.
6. Develop guidance. Guidance is ideally developed collaboratively with a combination of practitioners and experienced enterprise IT professionals who understand the bigger picture.
7. Define roles and responsibilities. The definition of roles and responsibilities is an important aspect of your overall governance strategy as it describes what is expected of people. We’ve found that the most effective way to do this is to start with the roles and responsibilities described in the Disciplined Agile (DA) toolkit and then collaboratively evolve them from there. The framework ha both delivery team roles as well as enterprise IT roles defined.
8. Manage IT risk. Risk management for the entire IT portfolio, including both in-progress development teams as well as operational risks. Small risks on individual teams can add up to a large risk across your organization. For example, if many teams adopt a new and relatively unproven framework and that framework is then abandoned by its creators (think about how many open source projects or new companies flounder) then you have a serious problem. Similarly, when multiple teams start addressing a similar business opportunity and that line of business dries up or is legislated away.  Although risk management is addressed on delivery teams via the goal Address Risk, their focus is on risk management within a team as opposed to risk management across all of IT, or even across all of your organization.

Related Reading:

• The Lean Governance Mindset
• Lean Governance Strategies
• What is Lean IT Governance?
• Why Lean IT Governance?
• What is Agile Delivery Governance?
• Strategies that Enable Agile Governance
• Supporting Agile Delivery Team Governance
• Lean vs. Traditional IT Governance

The Agile Stalemate

The current state of Agile, with both its advantages and drawbacks, is significantly influenced by Scrum, the most widely used Agile method.

As a drawback, after some (easy to adopt) first steps in Agile introduced by Scrum, many teams have no clear vision of how to advance with the process improvements and with Agile. As a consequence, we could say that the current state of Agile is a stalemate.

A significant part of the parts missing in many Scrum-based Agile adoptions, such as XP (Extreme Programming) engineering practices, was described by Robert C. Martin in his article The Land that Scrum Forgot.

The Advantage of Scrum: First Steps in Agile

Beyond the serious existing criticism, Scrum offers at least the following advantages:

• Agile has become a widely known approach because of a significant number of Scrum adoptions (or adoption attempts)
• Scrum is easy to adopt
• Scrum adoption offers some important quick benefits

This strong ability of Scrum on making the first Agile steps is too often underrated. The main advantage of Scrum practices is not related so much to its direct benefits, but more to the elimination of the side effects of many bad legacy practices that are thus being replaced. For example, if a team uses the Agile-enabled self-organizing power to make poor process choices and disregards the practices for collaborative work, it will have worse results that a Scrum-based team that offers a base for collaborative work.

Going Further

A team using Scrum could go beyond the stalemate situation by using the XP engineering practices–refactoring, TDD (Test Driven Development), Pair Programming, and other–and/or it can go even further with Disciplined Agile (DA). Disciplined Agile has more practices, makes the Agile habits explicit (as Agile life-cycles), and offers guidance support instead of prescriptions.

The following sequence for adopting Agile, could be common, but is not an optimum approach–to a certain degree it inherently results in a waste of resources and in delays:

• First adopt Scrum for easy first steps
• Later on adopt complementary XP practices
• Finally adopt DA to fill the gaps and adapt the process to the context

From a DA adoption point of view, we need to find an optimized approach that must have the following two advantages:

• It enables quick first steps (similar with Scrum)
• It no longer represents a stalemate situation

The First Steps: a Base that Enables all the Others

DA offers a good learning-oriented process decision framework that will help the teams to make their own process decisions and further improves their process. However, the first steps would be the most difficult ones and the main questions that arise are the following:

“How should I start? What are the first factors that should be subjects of my choices?”

We are proposing the following Agile starting points:

• Use life-cycle and iterations as a container for the process
• Start working in a collaborative manner: non-solo work practices
• Start persistent improvement by adopting retrospectives

In the following section we will explain how the use of these quick first steps as a base will enable all the other improvements.

Start Working in a Collaborative Manner: non-solo Work Practices

Agile-based process improvements and Agile retrospectives work only if the teamwork is based on continuous and active collaboration.

  The simple and rather vague Scrum practices for team collaboration have the great advantage of consistently replacing the old habits of non-collaborative work. We can easily go further and get better results by adopting Agile Modeling/Disciplined Agile non-solo work practices, that mean working with others for (few examples):

• Release Envisioning – Requirements and Architecture Envisioning
• Looking ahead look on requirements or design – Look Ahead Modeling
• Just-in-time requirements or design – Model Storming
• Pair Programming (XP practice)

The essential element is to offer guidance to the team in order start to practicing non-solo work as soon as possible.

Start the Improvement by Adopting Retrospectives

The improvement based on Agile and other sources must become continuous and permanent. The very first step is to start using retrospectives (per iterations, releases, and so on).

 Containers for the Process: Life-Cycle and Iterations

The teams will need these containers as a “living space” for their process options and choices.  Scrum uses the Sprint (aka Iteration) as the main work container. XP goes further and has an implicit release life-cycle.

In Disciplined Agile we can find a more robust approach, meaning that we have both the iteration and also more explicit agile and lean life-cycles, whereby its selection is the team’s choice.

Ok, but “How should I start”?

For an “agile first steps” team, here is an option for the main process containers:

• The iteration – This is a generic container for not-very-advanced cases
• The agile-basic life-cycle – (this life-cycle) fits most of the cases and is similar with XP approach

As soon as possible and using proper guidance, the team should be aware of the process goals (DA makes these goals explicit) and how these goals are distributed along the life-cycle and iteration.

As a note, advanced teams could also select and use other types of life-cycles.

Just Do It!

Using this initial framework – process goals across life-cycle and iterations – the team should start working collaboratively (non-solo work practices) and start using retrospectives and other instruments to decide how to improve and adapt their process. Make sure that this approach is permanent and the process will have further optimizations.

There are several strategies that you can adopt to promote a lean approach to governance. These strategies include:

1. Empowered teams. Teams should have both the authority and responsibility to fulfill their mission. Agile teams should be allowed to be self organizing, the implication being that the team itself determines who will do what work (the team doesn’t have a manager telling them what to do).   Related to that the team should own their process, which is the agile way of saying that the team is allowed to determine how it will work together and as the team learns it will evolve the process that it follows – of course the team will be guided and constrained by your organization’s governance strategy.
2. Enterprise awareness. One of the principles of the Disciplined Agile (DA) toolkit is that you should work in an enterprise aware manner. It is based on the observation that your team is only one of many teams, so therefore you need to consider the bigger picture when you’re working and do what is best for your organization, not just what is convenient for you. Promoting enterprise awareness throughout your organization is a fundamental enabler of lean governance.
3. High-level roadmaps. High-level technology and business roadmaps, produced by your Enterprise Architecture and Product Management efforts respectively, will provide important guidance to your teams. These roadmaps capture the vision as to where your organization is heading, helping teams to understand what the overall vision is, to focus on what is important to your organization, and to help guide and constrain their decisions.
4. Collaborative enterprise IT professionals. The DA toolkit includes several process blades, including Enterprise Architecture, Reuse Engineering, Data Management, and others that address enterprise level concerns. The people performing these activities should work closely with their customers, the delivery teams and stakeholders, in a collaborative and evolutionary manner. This promotes better governance for two reasons. First, by getting the vision, knowledge, and skills of the enterprise professionals into the hands of their customers it increases the chance that they work in a manner that is consistent to the needs of your organization. Second, the enterprise professionals want to get feedback from their customers and learn more about what their customers need from them. This enables them to be more effective at serving the enterprise and guiding their customers.
5. Enterprise IT knowledge in teams. Although roadmaps and enterprise IT professionals collaborating with development teams help, it is far more effective to have people with enterprise knowledge embedded within the development teams. This is why we promote the idea that Architecture Owners (AOs) should not only work closely with the enterprise architects but should preferably be a member of the EA team. Similarly, Product Owners (POs) should either work closely with the product management team or preferably be a member of that team.   And it’s possible to do better than that – if team members are truly enterprise aware and are continuous learners, then it is reasonable to expect them to pick up enterprise knowledge over time. The more knowledgeable people are about their organization and its goals the less supervision/governance they will need.
6. Automated dashboards. Automated dashboards, a strategy that we’ve referred to as development/operational/IT intelligence in the past, is a scalable form of information radiator. With just a bit of work you can take the information being generated by your tools and, using business intelligence (BI) technologies, populate team and even portfolio dashboards in real time. These dashboards provide important information that teams can use to manage themselves as well as governors to monitor what is happening within your organization. This enhances governance because when you get better quality information into people’s hands and they are more likely make better decisions.
7. Defined roles and responsibilities. Defined roles and responsibilities help people to understand who does what, what are they are responsible for and when they need to collaborate with others. This is an important aspect of governance because critical guidance about how people will need to interact with one another.
8. Defined organizational structure. You may choose to have a hierarchy of teams, a network of teams, a collection of circles (along the lines of holocracy), or combinations thereof. This is important to your governance efforts because people need to know what are the teams and how do they interrelate within your organization.
9. Common guidance. Guidance – standards and guidelines – is important to your governance effort because it helps people to understand how they can develop consistent assets which in turn are easier to understand and evolve. Common development guidance includes coding standards, data naming conventions, user interface (UI) design guidelines, security standards, and more. This guidance should be straightforward, ideally be supported through automation, and collaboratively developed.
10. IT governance team. People, typically senior people, are responsible for IT governance. The team, who is on it and what they do, should be defined and publicly known by those being governed. Everyone knows who the governors are and what they do, so that your governance strategy is open and transparent.

In the next blog posting in this series on governance we’ll overview the process goal diagram for IT governance.

Related Reading:

• The Lean Governance Mindset
• What is Lean IT Governance?
• Why Lean IT Governance?
• What is Agile Delivery Governance?
• Strategies that Enable Agile Governance
• Supporting Agile Delivery Team Governance
• Lean vs. Traditional IT Governance