Disciplined Agile

We recently published a collection of articles overviewing the seven principles behind the Disciplined Agile (DA) toolkit.  The article is entitled Principles Behind the Disciplined Agile Framework and it in turn is supported by detailed articles, one for each principle:

1. Delight Customers – We delight our customers when our products and services not only fulfill their needs and expectations but surpass them.
2. Be Awesome – Awesome teams are built around motivated individuals who are given the environment and support required to fulfill their objectives.
3. Pragmatism – Let’s be as effective as we can be, and that may mean we go beyond being just agile.
4. Context Counts – Every person, every team, and every organization is unique.  Let’s find and evolve an effective strategy given the situation we actually face.
5. Choice is Good – Different contexts require different strategies. Teams need to be able to own their own process and to experiment to discover what works in practice for them given the situation that they face. Having process options to choose from, and understanding the trade-offs of those options, enables you to home in on better options sooner.
6. Optimize Flow – Your organization is a complex adaptive system (CAS) of interacting teams and groups that individually evolve continuously and affect each other as they do. To succeed you must ensure that these teams are well aligned, remained well aligned, and better yet improve their alignment over time.
7. Enterprise Awareness – When people are enterprise aware they are motivated to consider the overall needs of their organization, to ensure that what they’re doing contributes positively to the goals of the organization and not just to the sub-optimal goals of their team.

We hope that you find these articles insightful.

There are several values that are key to your success when transforming to a leaner, more agile approach to Data Management. Taking a cue from the Disciplined Agile Manifesto, we’ve captured these values in the form of X over Y. While both X and Y are important, X proves to be far more important than Y in practice. These values are:

1. Evolution over definition. The ability to safely and quickly evolve an existing data source, either to extend it to support new functionality or to fix quality issues with it, is absolutely paramount in today’s hyper-competitive environment. Yes, having defined data models and metadata describing them is also important, but nowhere near as important as being able to react to new business opportunities. Luckily agile database techniques, long proven in practice, exist that enable the safe evolution of production data stores.
2. Holistic organization over Data Management. Earlier we said that data is the lifeblood of your organization. Yes, blood is important but so is your skeleton, your muscles, your organs, and many other body parts. We need to optimize the whole organizational body, not just the “data blood.” Traditional Data Management approaches often run aground because they locally optimize for data concerns, whereas a DA approach to Data Management recognizes that we must optimize the overall whole. This implies that sometimes we may need to sub-optimize our strategy from a data point of view, for the sake of organizational level optimization.
3. Sufficiency over perfection. Data sources, like many other IT assets, need to be good enough for the task at hand. The old saw “perfect is the enemy of good” clearly applies in the data realm – too much time has been lost, and opportunities squandered, while development teams were forced to wait on Data Management teams to create (near) perfect models before being allowed to move forward. Traditional data professionals mistakenly assume that production databases are difficult to evolve and as a result strive to get their designs right the first time so as to avoid very painful database changes in the future. The Agile Data method has of course shown this assumption to be wrong, that it is very straightforward and desirable to evolve production databases. A side effect of this revelation is that we no longer need to strive for perfect, detailed models up front. Instead we can do just enough up-front thinking to get going in the right direction and then evolve our implementation (including data sources) over time as our understanding of our stakeholder needs evolve.
4. Collaboration over documentation. We’ve known for decades that the most effective way to communicate information is face-to-face around a shared sketching environment, and that the least effective is to provide detailed documentation to people. The implication is that we need to refocus our efforts to be more collaborative in nature. As data professionals we need to get actively involved with solution delivery teams: to share our knowledge and skills with those teams, and to enable them to become more effective in working with data. Yes, we will still need to develop and sustain data-related artifacts, but those artifacts should be lightweight and better yet executable in nature.
5. Cross-functional people over specialized staff. Agilists have come to realize that people are more effective when they are cross-functional (also known as T-skilled or generalizing specialists). Although specialists are very skilled in a narrow aspect of the overall process, the problem is that you need a lot of specialists to perform anything of value and as a result the overall workflow tends to be error prone, slow, and expensive. The other extreme would be to be a generalist, someone who knows a little bit about all aspects of the overall process. But, the challenge with these people is that although they’re good at collaborating with others they don’t actually have the skills to produce concrete value. We need the best of both worlds – a generalizing specialist with one or more specialties so that they can add value AND a general knowledge so that they can collaborate effectively with others and streamline the overall effort.
6. Automation over manual processes. The only way that we can respond quickly to marketplace opportunities is to automate as much of the bureaucracy as we possibly can. Instead of creating detailed models and documents and then reviewing potential changes against them we capture our detailed specifications in the form of executable tests. This is quickly becoming the norm for specifying both the requirements and designs of code, and the same test-driven techniques are now being applied to data sources. Continuous integration (CI) and continuous deployment (CD) are also being applied to data sources, contributing to improving overall data quality and decreasing the time to safely deploy database updates into production.

As you can see, we’re not talking about your grandfather’s approach to Data Management. Organizations are now shifting from the slow and documentation-heavy bureaucratic strategies of traditional Data Management towards the collaborative, streamlined, and quality-driven agile/lean strategies that focus on enabling others rather than controlling them.

Recommended Reading

• To read more, we have a detailed article overviewing a Disciplined Agile approach to Data Management
• Need training in agile skills for DW/BI?  Consider the workshop Disciplined Agile Data Warehousing(DW)/Business Intelligence (BI).

The Agile manifesto is only 4 lines, there are only 12 principles for agile software and the original Scrum Guide was less than 20 pages, so how hard can Agile be?

Chess has only a couple of dozen basic moves so how hard can it be to get to checkmate?

How many times have you heard?

• All you need to be Agile is to get the team doing a few funky ceremonies

Or the approach to evolving a team from forming to performing:

• If the team isn’t performing after a month then the manager is not effective.

Despite the apparent simplicity of Agile, transforming to Agile is hard and the learning curve is very steep.  Depending on the team and the individuals, a transformation from waterfall to a high performing Agile team can take anywhere from 6 months to a year or more.  That’s not to say that they won’t be productive over that period of time.  In fact, they will start delivering business value at the end of the first iteration, but the team will be in constant growth and development over that period of time because not only do the people have to learn a new process, they also need to learn a new way of thinking as well as a new way of working as a team.

1.0 A new process

Aside from being excited to work with the thought leaders in the agile world, the reason I joined with SA+A is because of the Discipline Agile Delivery (DAD) decision framework and the way it captured all the tough stuff that I had been wrestling with for years.  With many years of experience building agile teams, I considered myself somewhat of an expert with answers to a lot of the difficult questions.  When I reviewed DAD I realized that the answers I had gained from my own personal experience would not apply in all circumstances or for all teams.  Context counts.  The DAD Agile/Basic and Lean/Advanced project lifecycles have three phases: inception, construction and transition.  Each phase has a collection of process goals, and each goal encompasses several process decision points.  And, each decision point has a collection of options to choose from.  These process goals, decision points and options have been empirically assembled by analyzing many agile teams to see what makes an agile team successful.  This framework gives a team the ability to define a process that works for them.  Is this difficult to do?  Absolutely!  Defining a process to use is always difficult. BUT…..  The framework provides some guidance. Each goal option has default options that the team can use to get started and the options are listed in preference order so the team can easily pick an initial option.  As the team evolves and matures they can chose the more advanced options and evolve and mature their process.  Does the toolkit include all possible options?  Definitely not, but the options are certainly extensive.  The framework has evolved over time and continues to evolve as the coaches and thought leaders gain more experience and work with more diverse teams.

No matter which lifecycle the team has chosen to use; whether the team is using classic Scrum (Basic/Agile) or Kanban (Advanced/Lean) or a tailored process created using a toolkit such as DAD, the team is going to need practice and discipline to get it right.  That means, executing the lifecycle, doing periodic retrospectives, updating the process and then repeating again and again until they get it right.  Changing behaviors is hard and the team will struggle to be successful.

2.0 A new way of thinking

No matter how you cut it, transforming to Agile is a cultural change and culture changes are hard.  Not only is it hard for those who want to make the change but almost inevitably there will be pockets of resistance from people are happy with “the way we’ve always done it”.  Unfortunately, political infighting is the biggest risk to innovation. It is going to happen, you may as well brace yourself for it.

The roles are changing from traditional roles (such as: developer, business analyst (BA), system analyst (SA), quality assurance (QA) and project manager (PM), to new unfamiliar roles (such as: product owner (PO), team lead (TL), architecture owner (AO) and team member).  The traditional roles may be deeply embedded in your HR hiring and performance review processes. Transitioning to the new roles will require a realignment of skills and abilities that is bound to cause great consternation amongst the team members.  Often this is a very difficult transition for project managers, in particular, because they find that a lot of the traditional tasks (i.e. work breakdown structures, planning, estimating, etc.) are now the responsibility of the whole team.

The team needs to start thinking in smaller deliverable pieces.  The concept of delivering a complete system from known specifications at the beginning of the project has been the fallacy of traditional development.  The cost of delaying delivery to the very end of the project and the lack of ability to meet actual customer needs can be eliminated by delivering minimum functionality early and augmenting with new and improved functionality iteration after iteration in close collaboration with the end customer.  But moving to this model of continuously delivering can be a tough adjustment for developers.  Even (or maybe especially) for seasoned developers, the concept of starting to build and deliver not knowing all the detailed specifications up front can be very unsettling. Getting comfortable with this model takes time and experience before developers cease to fall back on old behaviors.

One of the pillars of agile is complete transparency with the business and the stakeholders into the team, the process they are using and the delivery of the business functionality.  With the business embedded in the team on a day to day basis and iteration reviews to demonstration and approve all the deliverables from each iteration, there is no hiding problems when they occur and no hiding late deliveries or failed iterations.  On the flip side, the business is there to celebrate the wins and recognize the deliveries at the end of each iterations and they are part of determining the velocity of the team and adjusting the schedule to match the velocity.

The new way of thinking also requires new tools.  Tools that can build dashboards to automate and display the development intelligence metrics that provide transparency into the team’s performance.  Tools that automate configuration management, continuous integration and continuous integration as well as automated testing.  All of these endeavors are hard and required skilled individuals to lead; none of these comes for free.

3. A new way of working with others

More people skills are required by everyone.  People are now going to be working much closer together physically and will be interacting and collaborating on a much more frequent basis.  Experience working together (often for many years) may give an advantage to some team members, however, they still need to learn how to work together as a team.

3.1 Team Development Phases

Tuckman’s team development model includes the phases: forming, storming, norming, and performing. These phases are all necessary and inevitable in order for the team to grow, to face up to challenges, to tackle problems, to find solutions, to plan work, and to deliver results.  The number of iterations required to move through the phases will vary by team depending on factors such as: company culture, skills, experience, familiarity with each other, leadership (or lack thereof), external influences and coaching etc. Teams may also plateau in a phase before entering the next phase.

Agile teams will go through the four Tuckman phases:

1. Forming. All teams will go through the forming phase. Typically this take 4 to 6 iterations for the team to: understand the lifecycle process, understand how to work together, understand what the team goal is, and most importantly to get comfortable enough with each other to risk the possibility of conflict.  During this phase, the team’s velocity will vary considerably especially for the expected failed iterations although there should generally be a trend upward in the velocity.  Retrospectives in this phase tend to be very cordial and factual because everyone is avoiding conflict.
2. Storming. The storming phase can be very upsetting to both the team members and team managers. Often managers will step in to solve the storming issues without realizing that it is a necessary phase for the team to work through on their own.  Conflict, disagreements and personality clashes must get resolved before the team can move to the next phase.  Unfortunately, this phase is a make or break phase and it is hard to predict just how many iterations it will take but it is not usual to expect 3 or 4 pretty rough iterations.  Some teams will completely disintegrate during this phase and it is certainly not unusual to have a decrease in velocity during this phase.  Some members of the team may find they have basic incompatibilities and need to move to another team and this change in team membership can revert the team back to the forming phase.  The retrospectives tend to be very interesting and intense during this phase as they deal with the inter-personal issues of the team.  You can pretty much expect that there are going to be some hurt feelings no matter how safe an environment you have.
3. Norming. The norming phase is when the team comes back together stronger and more cohesive for having weathered the storming phase. They have resolved their issues and have developed a new synergy where the team just “works well together” and it is fun to be part of the team. The team velocity should trend upward and then level off as the team hits its capacity consistently.  With the new-found synergy of the team, the retrospectives can focus on process improvement and optimizing the operation of the team with the hope of transitioning to the performing phase.
4. Performing. The performing phase is the goal for all teams but very few actually get to this phase. Not all teams can get to the high level of performance of a Special Ops team where each member knows exactly what their team mates are doing and what they need to do to work as a cohesive unit.  A team in performing phase no longer needs a coach or supervision but can operate autonomously and efficiently.

3.2 Team changes everything

With self-organizing teams, decisions are now made at the team level rather than at the PM or manager levels.  This can be a difficult transition for both the team members and the managers.  Team members are often use to others making the decisions for them and wait for decisions or are reluctant to make decisions.  Managers often feel disempowered when decisions are no longer in their hands because the decisions are made by the team.

The team still has the responsibility to do the traditional activities such as; analysis, design, development testing and deployment.  However, under the new model, the team also takes on the inception responsibilities of: developing user stories, sizing the stories, estimating the stories, creating the product and architectural vision, and developing the release plan.   During construction the team also has to: plan the iteration, do look ahead planning, and have an iteration review (i.e. demo) with the stakeholders.  To keep the team working together they need to have the daily coordination meeting (i.e. Scrum) and to keep the team growing and evolving of course they need to do the retrospective at the end of each iteration.

Thinking, analyzing, planning and building just-in-time is a new way of thinking for a traditional team.  The inception phase in particular can be a difficult adjustment.  Designers and developers are used to doing all the analysis and design upfront.  Doing just enough analysis to be confident of a solution and then writing the user stories to do the work and build the plan is a difficult transition.  Writing user stories so they can get started and learn as they go is a very difficult transition.  Building just the simplest framework and then adding required features and functionality as needed requires practice, training and reinforcement.

There is much more focus on delivering quality because the team is responsible for development, delivery as well as product support.

4.0 The Bottom Line

Chess has only a couple of dozen basic moves and so it is simple to learn the basics.  However, after 3 moves there can be several billion moves available making it so complex that it can take a lifetime to master.  Transforming to Agile is a lot like that.  It looks very simple on the surface but under the covers it can get very complex very quickly.  That’s where the value of a good coach comes in.  You need a coach who brings a lot of experience to the table who will be open and honest with you about what needs to be done and how to get a successful transformation.

IT Portfolio Management addresses how an IT organization goes about identifying, prioritizing, organizing, and governing their various IT endeavors.  Disciplined Agile Portfolio Management seeks to do this in a lightweight and streamlined manner that maximizes the creation of business value in a long-term sustainable manner. IT endeavors typically include solution delivery initiatives/projects, stable product development teams, business experiments (along the lines of a lean startup strategy), and the operation of existing IT-based solutions.

Being agile, having an agile mindset, is foundational to working in an agile manner. The Disciplined Agile Manifesto and the principles of lean software development provide an important start at this mindset. In this blog we explore similar agile philosophies that are specific to successful portfolio management. These philosophies are:

1. Keep it simple. Keep your portfolio management activities as streamlined and lightweight as can be. Your goal should be to focus on making good decisions and providing guidance to people, not on maintaining extensive documentation or reviewing documentation. You still may choose to maintain artifacts such as a portfolio roadmap and portfolio budget, but those too should be as lightweight and concise as possible.
2. Focus on value over cost. Shifting your mindset from “what is this going to cost?” to “what value will this generate?” is critical to your success because it helps you to focus making better IT investments. You want to invest in IT endeavors that enhance your organization’s ability to produce value for your customers. This in turn provides the profits required for further investment.
3. Reduce the cost of delay. One of the strategies for maximizing stakeholder value is to invest in developing functionality that will provide value to the organization soonest.   For example, if you delay developing functionality that will generate annual revenue of $10 million for six months you have an effective cost of delay of $5 million because you missed out on half a year of revenue.   Disciplined agile portfolio managers consider the cost of developing a solution, the cost of delay that results from putting off said development, and the revenue generated (or cost savings) when calculating the overall value of a solution.
4. Invest in streamlining the creation of value. Not only do we want to produce value for our customers, we also want to be good at doing so. The implication is that we sometimes need to invest in ourselves through process or environment improvements.
5. Prefer stable teams over project teams. Although portfolio management is often believed to be the oversight of project teams, it really is more about the coordination and oversight of teams in general. In Disciplined Agile we recognize that an initiative is seldom finished at the end of a “project”. There are usually subsequent changes required over time requiring future releases of the solution. The agile community has discovered that long-lived stable teams, whose membership evolves over time, have significant advantages over short-lived project teams. A significant productivity improvement occurs when IT organizations shift away from the project mindset of bringing people to the work and instead decide to bring work to the people (the stable teams).
6. Align teams to value streams. These stable teams should be aligned long-term to value streams or lines of business (LOBs). High performing agile teams reliably deliver value frequently to their business stakeholders. As a result the business learns to trust the teams aligned to their areas. This positive feedback cycle maximizes the effectiveness of your agile teams. Additionally, over time they learn more about the business adding to their effectiveness.
7. Enable diversity. Every person, every team, and every organization is unique. Every team faces a unique situation that evolves over time. The implication is that teams must be allowed to organize themselves and to tailor their process to meet the context of the situation that they face. This is why the Disciplined Agile (DA) toolkit focuses on providing, and comparing and contrasting, a wealth of process choices. The implication for portfolio managers is that they need to be flexible in their approach. They will work differently with each team because each team works differently, yet they must still provide good guidance to these teams and monitor the effectiveness of each team appropriately.
8. Trust but verify. Effective governance is based on enabling and then trusting your teams to do the right thing. However, effective teams monitor themselves through the use of automated dashboard technology and close collaboration, and the metrics that teams collect should be made visible to senior management and other stakeholders so that they may monitor what is happening.
9. Govern by risk not by artifacts. Traditional governance often focuses on the review of common artifacts such as requirement documents, architecture models, and test results. Because it is relatively easy for teams to create the documentation that you want to see, in practice there is very little governance value in reviewing these artifacts. Agile governance instead focuses on addressing common risks such as ensuring there is an agree to vision for what the team should accomplish, that the architectural strategy has been proven to be viable early in the lifecycle, and that the team has produced sufficient business value for their stakeholders.
10. Rolling wave over annual planning. Annual planning often begins in earnest mid-year which means that prioritized initiatives may not be actually delivered for up to 18 months in the following year. This is not business agility. Make your planning a continuous “rolling wave” activity year round with more detail devoted to planning initiatives no longer than 6 months out. Initiatives planned beyond 6 months should be described at a very high level.
11. Prefer small initiatives over large initiatives. It is a proven fact that the larger the initiative the greater the chance of failure. Smaller initiatives are easier to plan and are lower risk to execute. Keep your initiatives small to allow for more frequent delivery of value with less investment in work in progress (WIP).
12. Invest in quality. Ensuring that IT delivery teams produce new business value for your organization is clearly important. But so is ensuring that you will still be able to continue doing so a few years from now. To ensure the long-term sustainability of your IT teams you must allow them to make investments in quality. These investments include building high quality assets in the first place but also fixing low quality assets, what agilists refer to as paying down technical debt, as well.
13. Optimize the whole. Disciplined agilists are enterprise aware. We choose to optimize the whole instead of locally optimizing a single activity. The implication is that you cannot consider portfolio management on its own but instead must consider it in the context of the other parts of your organization that it affects. A strategy that may make things easier to manage your portfolio, such as having a single way to fund IT delivery teams, may make it easier for your portfolio management efforts but could inflict undue costs and bureaucracy on the teams that you’re funding. For example, does it really make sense for a team asking for $50,000 in funding to go through the same level of rigor as a team asking for $5,000,000? Likely not. Context counts.

Our experience is that the philosophies describe above enable portfolio managers to be more effective in practice. We hope you have found this blog of value and we welcome your feedback.

In Implementing Lean Software Development, Mary and Tom Poppendieck show how the seven principles of lean manufacturing can be applied to optimize the whole IT value stream. These principles are:

1. Eliminate waste. Lean thinking advocates regard any activity that does not directly add value to the finished product as waste. The three biggest sources of waste in software development are the addition of unrequired features, project churn and crossing organizational boundaries (particularly between stakeholders and development teams). To reduce waste it is critical that development teams be allowed to self organize and operate in a manner that reflects the work they’re trying to accomplish. Walker Royce argues that the primary benefit of modern iterative/agile techniques is the reduction of scrap and rework late in the lifecycle.
2. Build in quality.  Your process should not allow defects to occur in the first place, but when this isn’t possible you should work in such a way that you do a bit of work, validate it, fix any issues that you find, and then iterate.  Inspecting after the fact, and queuing up defects to be fixed at some time in the future, isn’t as effective.  Agile practices which build quality into your process include test driven development (TDD) and non-solo development practices such as pair programming and modeling with others.
3. Create knowledge.  Planning is useful, but learning is essential. You want to promote strategies, such as iterative development, that help teams discover what stakeholders really want and act on that knowledge. It’s also important for a team to regularly reflect on what they’re doing and then act to improve their approach.
4. Defer commitment.  It’s not necessary to start software development by defining a complete specification, and in fact that appears to be a questionable strategy at best. You can support the business effectively through flexible architectures that are change tolerant and by scheduling irreversible decisions to the last possible moment. Frequently, deferring commitment to the last most responsible moment requires the ability to closely couple end-to-end business scenarios to capabilities developed in multiple applications by multiple projects.
5. Deliver quickly.  It is possible to deliver high-quality systems quickly. By limiting the work of a team to its capacity, which is reflected by the team’s velocity (this is the number of “points” of functionality which a team delivers each iteration), you can establish a reliable and repeatable flow of work. An effective organization doesn’t demand teams do more than they are capable of, but instead asks them to self-organize and determine what they can accomplish. Constraining these teams to delivering potentially shippable solutions on a regular basis motivates them to stay focused on continuously adding value.
6. Respect people. The Poppendiecks also observe that sustainable advantage is gained from engaged, thinking people. The implication is that you need a lean approach to IT governance that focuses on motivating and enabling IT teams—not on controlling them.
7. Optimize the whole.  If you want to be effective at a solution you must look at the bigger picture. You need to understand the high-level business processes that individual projects support—processes that often cross multiple systems. You need to manage programs of interrelated systems so you can deliver a complete product to your stakeholders. Measurements should address how well you’re delivering business value, because that is the sole reason for your IT department.