An important aspect of Disciplined Agile Delivery (DAD) is its explicit inclusion of an Inception phase where project initiation activities occur. Although phase tends to be a swear word within the agile community, the reality is that the vast majority of teams do some up front work at the beginning of a project. Some people will mistakenly refer to this effort this Sprint/Iteration 0 it is easy to observe that on average this effort takes longer than a single iteration (the 2009 Agile Project Initiation survey found the average agile team spends 3.9 weeks in Inception and the November 2010 Agile State of the Art survey found that agile teams have Construction iterations of a bit more than 2 weeks in length).
Regardless of terminology, agile teams are doing some up front work. Part of that initial work is identifying an initial technical architecture, typically via some initial architecture envisioning http://www.agilemodeling.com/essays/initialArchitectureModeling.htm. Because your architecture should be based on actual requirements, otherwise you’re “hacking in the large”, your team will also be doing some initial requirements envisioning http://www.agilemodeling.com/essays/initialRequirementsModeling.htm in parallel. Your architecture will be driven in part by functional requirements but more often the non-functional requirements, also called quality of service (QoS) or simply quality requirements. Some potential quality requirements are depicted in the figure below (this figure is taken from the Disciplined Agile Delivery book but was first published in Agile Architecture Strategies ).
Architectural views and concerns
Some architects mistakenly believe that you need to do detailed up front modeling to capture these quality requirements and then act upon them. This not only isn’t true it also proves to be quite risky in practice, see my discussion about Big Modeling Up Front (BMUF) for more details. Disciplined agilists instead will do just enough initial modeling up front and then address the details on a just-in-time (JIT) basis throughout construction. Of course it’s important to recognize that just enough will vary depending on the context of the situation, teams finding themselves at scale will need to do a bit more modeling than those who don’t. It’s also important to recognize that to address non-functional requirements throughout construction that you need to have more than just architectural modeling skills. This topic will be the focus of my next blog posting in this series.
Recently at the Scott W. Ambler + Associates site we received a series of questions from someone who wanted to better understand how architecture issues are addressed on agile project teams. It seemed to me that the questions were sufficiently generic to warrant a public response instead of a private one. So, over the next few days I’m going to write several blog postings here to address the issues that were brought up in the questions. It’s important to note that I will be answering from the point of view of Disciplined Agile Delivery (DAD), and not agile in general. Other agile methods may provide different advice than DAD does on this subject, or no advice at all in some cases.
The goal of the first blog posting in this series is to address several potential misconceptions that appeared in the email. I want to start here so as to lay a sensible foundation for the follow-on postings.
Partial Misconception #1: Agile can be prefixed in iteration 0 by architectural design
I’ve named this a “partial misconception” for a few reasons:
Disciplined agile teams do some up-front work. This is called the Inception Phase in DAD, although other methods may refer to it as iteration/sprint 0, warm up, initiation, or other names. Up-front work is an explicit part of DAD.
Iteration 0 isn’t an accurate term. Although I have used this term in the past when discussing project initiation, the reality is that the average agile team spends about a month doing project initiation activities whereas the average iteration length is two weeks. So, Inception really isn’t a proper iteration.
Disciplined agile is against needless waste, not “big” things. Many traditional modeling and planning practices prove to be quite wasteful in practice. A serious cultural challenge that the traditional community has is that they are afraid to throw out the bathwater because they assume that the baby will go with it. I believe that Disciplined Agile Delivery (DAD), and Agile Modeling before it, make it quite clear that it’s possible to gain the benefit of thinking before doing without taking on the very serious problems around doing too much thinking before doing. So, have the discipline to keep the thinking “baby” yet discard any needless documentation “bathwater”.
In rare situations it’s appropriate to create “big” artifacts. Disciplined agilists aim for sufficient artifacts, the size of which will depend on the context of the situation that your team finds itself in. In a recent article for Dr. Dobb’s Journal, Disciplined Agile Architecture, I explicitly explored how initial architecture envisioning on an agile project may result in “big” artifacts in some situations. These situations are very rare mind you, ignoring cultural imperatives to create big artifacts because some people still haven’t made the jump to a disciplined agile approach, but they do happen. One of the strengths of the DA toolkit is that it is goal driven, not prescriptive, and explicitly explores the tradeoffs surrounding the amount of detail to capture and when to do so.
Partial Misconception #3: Refactoring system architecture beyond mid-implementation is much more expensive than refactoring components
Once again, this is a partial misconception. I suspect part of the problem is a lack of understanding of what refactoring is really all about, a recurring problem with experienced traditionalists, and part because of a lack of understanding of how architecture is address by disciplined agile teams. Some thoughts:
Refactorings are simple, not difficult. The goal of refactoring is to make SMALL changes to your design that improve the quality without changing the semantics of the design in a practical manner. This is true of code refactorings, database refactorings, user interface refactorings, and other types of refactorings. Small changes are inexpensive to make given the appropriate skills, tools, and organizational environment.
Architectural rework (not refactoring) is often difficult. Rework, or rewrites, are very large changes the goal of which is typically to replace large portions of your solution. Yes, the later in the lifecycle such rework occurs very likely the more expensive it will be because you’ve built more based on that architecture that is now being reworked. This is a general issue, not just an agile one.
Disciplined agile teams get going in the right direction to begin with. The practice of initial architecture envisioning, which we describe in detail in Chapter 9 of Disciplined Agile Delivery, aims to think through the architectural strategy before getting into construction.
Disciplined agile teams prove their architecture works early. The first construction milestone, prove the architecture, reduces the risk of architectural rework. The goal is to prove that the architecture works by building a working end-to-end skeleton of the solution which implements critical/difficult technical requirements. This is an agile “fail fast” strategy, or as we say in DAD a “succeed early” strategy, that reduces technical risk on your project. As an aside, including explicit light-weight milestones such as this is one of many agile governance aspects built right into DAD.
Disciplined agile teams have an architectural role. This role is called Architecture Owner and one of the responsibilities of the person in this role is to guide the team in architectural issues throughout the entire DAD lifecycle.
There are no guarantees. No matter how smart your approach, there’s still a chance that rework can happen. For instance, you can be mid-way through a project and the vendor of a major architectural component of your solution decides to withdraw it from the market. Or the vendor goes out of business. Or perhaps your firm is taken over by another firm and the new owners decide to inflict, oops I mean bless you with, their architectural strategy. Stuff happens. Once again, this is a general issue, not specifically an agile one.
Quality decreases the cost of rework. Disciplined agilists will write high-quality code, with a full regression test suite in place, at all times during Construction. It’s easier to rework high quality artifacts compared with low quality artifacts, so if you get stuck having to perform rework at least the pain is minimized. My article Agile Testing and Quality Strategies overviews many techniques.
In short, disciplined agile teams do what they can to avoid architectural rework to begin with by having an explicit architecture owner role who focuses on architectural issues throughout the entire lifecycle, by identifying a viable architectural strategy early in the project, proving that architectural strategy works early in Construction, and producing high-quality artifacts throughout the lifecycle that are easier to rework if needed. With continuous documentation practices and a focus on producing artifacts which are just sufficient enough for the situation at hand, this proves to be far more effective than traditional strategies that assume you require large up-front investments in “big” artifacts, that rely on validation techniques such as architecture reviews instead of the far more concrete feedback of working code, and that often leave quality strategies to the end of the lifecycle (thereby increasing the cost of any rework).
I plan two follow-on blog postings in this series, one exploring how initial architecture envisioning works and one about how to address initial quality requirements (also called non-functional requirements or quality of service requirements) on disciplined agile projects. Stay tuned!
At Scott W. Ambler + Associates we offer a one-day workshop entitled Agile Architecture: A Disciplined Approach that you should consider if you’re interested in this topic. We also offer coaching and mentoring services around agile architecture.
Because Disciplined Agile Delivery (DAD) addresses the full delivery lifecycle it explicitly addresses the effort required to transition your solution into production, or in the case of product teams into the marketplace. This transition effort may be referred to as release, deployment, or even the “end game”. For teams relatively new to agile this transition effort is a phase, or if you don’t like the term phase then an iteration which very likely has a different time frame than construction iterations. However, as teams gain more experience with agile and lean techniques the “transition phase” can be evolved into a “transition activity” with a little bit of discipline. That is the focus of this blog posting.
The DA tool kit is goal-driven, not prescriptive, and as a result it describes the transition effort in terms of goals:
Ensure that the solution is ready to deploy. To achieve this goal you will need to address several issues, such as planning the transition effort, verifying that the solution meets the stakeholder needs for the current release, and validating that the solution is of sufficient quality. As with other DAD goals, there are several ways each of these issues can be addressed, each with advantages and disadvantages. There is no single strategy that’s right for all situations.
Ensure that stakeholders are ready to receive the solution. Similarly, there are several issues you need to consider when addressing this goal, including how you communicate that the release is coming, how to train and educate stakeholders in the new solution, and how to prepare the operations and support environment for the new release (one of the many aspects of a disciplined DevOps strategy built right into DAD).
Deploy the solution. Issues to consider when achieving this goal is identifying what should be released, how you can back out the release if you run into trouble, and how to determine that the release was successful. The last issue, determining success, isn’t as obvious as you would think. Does success mean that the solution is up and running in production? Or does success mean that you have satisfied, or better yet, delighted stakeholders?
It is straightforward to empirically observe that the complexity of your transition effort can vary depending on the context of your situation. For example, a simple standalone application such as a website can be easily deployed regularly because it effectively involves copying some files to a server (farm). Organizations in this sort of situation may choose to deploy their software many times a day. However, as we showed in the DAD book, for non-trivial enterprise deployments the transition effort can be significant. In fact, the November 2010 Agile State of the Art survey found that the average agile team spent six weeks in their transition efforts, with some respondents indicating spending one day or less and some indicating twelve weeks or more. There are several reasons why this happens:
Transition isn’t agile yet. Many organizations are adopting agile techniques for construction but not for other aspects of the delivery lifecycle. This is often referred to as “Water-Scrum-Fall”. As a result these organizations are inflicting traditional, labor intensive transition practices on their agile teams.
We’re dealing with more than just software. It’s often a bit more difficult than simply copying some files onto a production server. Mainstream agile methods describe the goal of creating “shippable software” at the end of every iteration. That’s a wonderful, overly simplistic philosophy that doesn’t hit the mark in practice. In DAD we prefer to use the phrase “consumable solution” indicating that shippable does not necessarily mean usable software. Furthermore, in many situations more needs to be delivered than just software – there may also be hardware upgrades, the business processes surrounding the use of the system may evolve, and even the organization structure of the people using the system may evolve. In short, disciplined agilists adopt a solutions over software mindset.
Transition occurs infrequently. Some agile project teams adopt a long release cycle, often six months or more, and as a result don’t have sufficient opportunities or motivation to get good at deployment. Sometimes this is due to the nature of the solution they’re working on but more often than not it’s because of the cost of transition – the greater the effort to transition into production the greater the cost to do so, therefore you need to spend more time building the solution to add sufficient value to justify the cost of transition.
Lack of vision. Many IT organizations don’t believe it’s possible that releasing solutions into production can be evolve from a multi-week phase to a multi-hour, or even multi-minute, activity. Or they realize it’s possible for others to do so but not them (I find it amazing how many people believe they’re in a special situation or blame others, often senior management, for not being able to adopt fundamental strategies that would improve their approach).
Because of the potential complexities of releasing a solution in most mid-to-large sized organizations the deployment of solutions is carefully controlled. This is particularly true when the solutions share architectures and have project interdependencies, one of the reasons why DAD promotes the need for enterprise awareness within agile teams. For many reasons release cycles to your stakeholders are less frequent that you would like because of existing complexities within the environment. However, the ability to frequently deploy value to your stakeholders is a competitive advantage; therefore you should reduce the release cycle as much as possible. To do so requires a great degree of discipline in areas such as:
Quality practices. First and foremost, testing and any related fixing should be done as much as possible during construction iterations. Transition is not a “stabilization phase” nor is it a “test phase”, it’s a deployment effort. Granted, you should still run your regression test suite(s) one last time as part of your overall deployment effort, with the potential that if tests fail you will need to do some fixing, but this should be a minimal effort. The implication is that quality practices – including testing, refactoring, reviews, code analysis, and so on – be performed continually throughout construction.
Pre-production integration and deployment testing. These are two of the most challenging types of testing, and sadly they’re often given short-shrift in the agile literature. The goal of pre-production integration testing, sometimes called production testing, is to test the solution in the best approximation of the production environment as possible. This can be particularly challenging in larger organizations where hundreds of systems are already running in production and dozens are being worked on at any given time. To make this testing economical during construction many organizations find that they need to go beyond the “whole team testing” approach with a parallel independent testing team that focuses on complex forms of testing that development teams can struggle with. Similarly deployment testing can also be a challenge because its goal is to determine whether you can successfully deploy into production.
Regular coordination between project teams and with operations and support staff. Throughout construction you will want to ensure that the development, operations, and support teams are all aligned. One aspect of DAD’s enterprise awareness is to include DevOps strategies, such as treating operations and support staff as key stakeholders that you need to work closely with throughout the lifecycle, in the process itself. The better the coordination the smoother, and therefore quicker, transition will go.
Adoption of continuous deployment practices. The fundamental idea behind continuous deployment is that the more often you choose to deploy the better at it you will get. This happens because you have more opportunities to hone your deployment skills and also because you’re motivated to find ways to simplify or automate deployment activities.
I’ve seen teams evolve transition phases of several weeks down to several hours through adoption of the disciplined strategies mentioned above. A disciplined agile team may start with relatively long Inception, Construction, and Transition phases and over time shrink all three down. Over time the Inception phase mostly disappears, particularly if you maintain team consistency between releases, and as I’ve argued in this posting the Transition phase shrinks to a very small activity. When deployment becomes inexpensive it enables you to have shorter construction phases and thus more regular releases – teams go from annual releases to bi-annual, then to quarterly releases, then monthly, weekly, and yes, sometimes even daily. Your team will need to choose a release cadence that makes sense for you.
These days it is fairly easy to observe multi-billion dollar companies, particularly e-commerce companies but even staid organizations such as financial institutions, deploy on a monthly, weekly, and even daily basis. If other organizations choose to work this way then why can’t you?
The Disciplined Agile Delivery (DAD) process framework includes an explicit Inception phase – sometimes called a project initiation phase, startup phase, or iteration/sprint zero – which is conducted before actually starting to build a solution. The goals of this phase include: clarifying the business problem that needs to be solved; identifying a viable technical solution; planning your approach; setting up your work environment and team; and gaining stakeholder concurrence that it makes sense to proceed with investing in the implementation of the chosen strategy. These goals are listed in the following diagram.
In the Disciplined Agile Delivery book we devoted a lot space to describing how to effectively initiate a DAD project. Unfortunately in our experience we have seen many organizations that are still new to agile treat this phase as an opportunity to do massive amounts of upfront documentation in the form of project plans, charters, and requirements specifications. Some people have referred to the practice of doing too much temporary documentation up front on an agile project as Water-Scrum-Fall. We cannot stress enough that this is NOT the intent of the Inception phase. While we provide many alternatives for documenting your vision in Inception, from very heavy to very light, you should take a minimalist approach to this phase and strive to reach the stakeholder consensus milestone as quickly as possible.
According to the 2013 Agile Project Initiation survey the average agile team invests about 4 weeks performing project initiation activities, including initial requirements envisioning, initial architecture envisioning, forming the team, initial release planning, and so on. Of course this is just an average, some respondents reported investing less than a week to do so and some reporting investing more than two months – the amount of time required varies depending on the complexity of the effort, your stakeholders’ understanding of their requirements, your team’s understanding of the solution architecture, whether this is a new solution or merely a new release of an existing solution, and many others.
If you are spending more than a few weeks on this phase, you may be regressing to a Water-Scrum-Fall approach. It takes discipline to be aware of this trap and to streamline your approach as much as possible. You can do this in several ways:
Recognize that the goal is to get going in the right direction. For any project or product of reasonable complexity you want to spend a bit of time up front ensuring that you understand the problem and have what you believe to be a viable strategy to addressing that problem. This doesn’t imply that you need a comprehensive requirements specification, a detailed project plan, nor a comprehensive architecture model but it does mean you need to do a bit of initial thinking and organizing. In a small handful of cases, typically at scale, you may find that your team does require detailed models and plans but this is a very uncommon exception (regardless of what traditional THEORY may claim).
Educate people that details can safely come later. If you have the ability to plan or model something in detail today, won’t you also have that same ability a few months from now when you actually need those details? Of course you do. In lean software development they recommend deferring decisions – including planning decisions, detailed design decisions, and even requirements – until the most appropriate moment. The observation is that by waiting you can make a better decision because you have better information at hand. This strategy of course assumes that you’re able to overcome basic logistical problems such as having the appropriate people available at the time to help explore an issue, provide requisite information, and make the decision. It’s far less risky, and far less expensive, in most cases to address basic logistical issues than it is to apply the process band-aid of writing detailed documentation at the beginning of a project.
Promote a sense of urgency. This is the most important thing that you can do. Just as there is risk associated with not sufficiently thinking about your strategy for approaching a new project or product there are also risks associated with doing too much up front work. My experience is that far too many IT professionals are complacent regarding the risks associated with the project initiation activities of the Inception phase. The longer you put off building a consumable solution the greater the risk that you’re building the wrong thing.
Keep your modeling efforts as light as possible. You very likely need to do some initial requirements envisioning and architecture envisioning early in the project lifecycle to help you think through what you’re doing. But in most cases this modeling should be high level and light, not detailed and heavy. In every project I’ve ever been involved with the team has been asked to identify what they’re going to deliver (at least giving a rough sense of the scope) and how they’re going to do it (at least providing a rough idea of the technical strategy) to secure funding for construction. In short, they need to do a bit of up front thinking. You will often find that you need to reign-in some of your staff who are experienced with traditional approaches to modeling and specification. These people have a lot of value to add to your project, modeling is an important skill needed on disciplined agile teams, but they may need help keeping their approach light-weight and incremental. The details can come later. See the process goals Identify Scope and Identify Technical Strategy for more thoughts on this subject.
Keep your planning efforts as light as possible. Similarly you need to invest some time in high-level release planning to answer basic questions such as how long do you believe (roughly) it will take to get a release of your solution deployed and how much (roughly) this will cost. Planning details can come throughout the Construction phase when it’s more appropriate to invest in such decisions.
I think that it’s very clear that the secret to keeping Inception short is to have the discipline to know that you need to invest some time thinking your approach through but that you want to avoid getting bogged down in too many details. You need the discipline to do some planning but not too much. You need the discipline to do some modeling but not too much. You need the discipline to get going in the right direction knowing that the details will come out in time.
Whether you like it or not, as you adopt agile you will constrained by the organizational ecosystem, and you need to act accordingly. It takes discipline to work with enterprise professionals such as enterprise architects, data administrators, portfolio managers, or IT governance people who may not be completely agile yet, and have the patience to help them. It takes discipline to work with your operations and support staff in a DevOps manner throughout the lifecycle, particularly when they may not be motivated to do so.
Despite the fact that governing bodies such as project management offices (PMOs), architecture and database authorities, and operations may indeed be a source of impediments to your DAD adoption, these authorities serve important functions in any large enterprise. Therefore a disciplined approach to proactively working with them and being a positive change agent to make collaboration with them more effective is required.
Although it takes discipline to work in an enterprise aware manner, the payoff can be dramatic. Enterprise architects can help guide your team early in the lifecycle to adopt a viable architecture strategy. Portfolio managers can help to govern your team to ensure that you truly are spending the stakeholder’s investment wisely. Enterprise administators, including data management professionals and operations professionals, can help you to leverage existing legacy assets. These are just a few examples of the potential ways that enterprise professionals can help a DAD team work more effectively.
