Disciplined Agile

This blog posting has been replaced by Disciplined Agile Enterprise Architecture

Is your organization concerned with the cost and time required to adopt agile strategies?  Are you just starting out with agile and hoping to improve your chances of success by learning from the experiences of those who have gone before you?  Are you part way into your agile transformation but struggling to figure out how it all fits together?  If you answered yes to any of these questions please read on.

In this blog posting we discuss what it means to “right size” your software process to meet the needs of the unique situation that your team finds itself in.  We discuss two common anti-patterns: starting with a process framework that is much too large for your needs and starting with one that is far too small for your needs.  We argue that it’s much better to avoid these extremes and instead take a middle-ground approach by starting with a toolkit that is much closer to your actual needs.

Extreme #1: Large process repository

The first process right-sizing anti-pattern is to start with a large process repository, the classic example being IBM’s Rational Unified Process (RUP).  Although RUP is much maligned within the agile community the fact is that if you were to examine it with an open mind that there are many very good ideas promoted by RUP.  Be that as it may. The basic strategy with RUP is that you need to tailor down the process, often dramatically, to meet the unique needs of the situation your team finds itself in.  To do this requires significant process expertise, time, and money.

In practice, however, many organizations ran aground with RUP when they tailored it to be something similar to the waterfall-style processes from yesteryear that they were familiar with.  This is often referred to as RUPifall.  Another common mistake was to say to themselves “wow, there’s a lot of great ideas here, we need to do them all” and as a result they would create a process that was far too heavy to meet their needs.  In either case the problem was that they often didn’t have the process expertise required to right-size their process.  We also see this with organizations starting from frameworks such as ITIL, COBIT, and CMMI so this clearly isn’t just a RUP problem.

Extreme #2: Small methodology

At the other extreme is the right-sizing anti-pattern of starting with a small methodology, the classic example in this case being Scrum.  Although there is significant support for Scrum within the agile community, or at least among people who are new to agile, we’ve been seeing for a long time now that organizations are also running into trouble with this approach too.  With Scrum the idea is that you add in the techniques that Scrum doesn’t address to right-size your process.  Because Scrum proves to be only a very small part of the overall picture, to do this requires significant process expertise, time, and money.

In practice organizations run aground with Scrum because they don’t have the process expertise to expand it to meet their needs.  One only has to count the number of “How does X fit into Scrum?” conversations occurring in agile discussion forums online, or at user group meetings or conferences, to see that this is true.  Step back for a moment and ask yourself how much time and effort has your organization invested in trying to adopt Scrum.  Could it not have been more streamlined?

A few years ago Forrester Research discovered that the majority of organizations “doing Scrum” had actually tailored it into what they called Water-Scrum-Fall (others call this Scrumifall).  As we describe in Going Beyond Scrum this occurs for several reasons.  First, Scrum doesn’t address the full delivery lifecycle, instead choosing to focus on the construction portion of it.  As a result organizations tend to stick with what they know, a heavy project initiation phase and a heavy solution deployment phase.  Second, Scrum only addresses only a small portion of what you need and explicitly leaves technical issues up to you.  These issues include topics such testing, programming, architecture, governance, documentation, deployment and many others.  As a result teams are left to piece together a process strategy that works for them, at the very same time that they’re struggling to understand the fundamentals of agile and lean software development.  They rarely have the process expertise to do that and as a result end up having to hire hordes of expensive Scrum coaches, few of whom seem to understand the enterprise-class realities that your teams actually face.  This is a risky and expensive proposition indeed.

The Effective Middle Ground

Both of these anti-patterns represent extremes: start with something large and cut it down to size, or start with something small and build it up to meet your needs.  Why not start with something much closer to what you actually need?  Doesn’t that make a lot more sense?  Why do you need to do all this process work?  Because someone wants to sell you tooling?  Because someone else wants to sell you expensive coaching and questionable certification strategies?  Isn’t it time to consider a more pragmatic strategy?

The Disciplined Agile (DA) toolkit is a more effective middle ground.  It addresses the full delivery lifecycle, as does RUP (but not Scrum), and even gives you several choices from which to select.  Sometimes a Scrum-based lifecycle is appropriate, sometimes a Lean lifecycle is, other times a continuous delivery lifecycle is best, and sometimes an exploratory “lean start up” lifecycle is.  Different teams, different situations.  DAD starts with a lightweight approach, as does Scrum but not RUP, helping you to avoid the bloatware of RUP and filling in the numerous blanks left by Scrum.  DAD also gives you lightweight tailoring advice in the form of process goal diagrams, in many ways a cross between mind-maps and decision trees, that make your process choices explicit.  The RUP process tailoring advice, if you bother to read it at all, is rather heavy handed and the Scrum tailoring advice boils down to “you’re smart, you can figure it out and if your run into trouble hire a coach.”  Isn’t it time to abandon the extremes?

This middle ground strategy isn’t without its faults.  A challenge with DAD is that it explicitly reveals that agile software development, or as we prefer to say agile solution delivery, is complex.  This is particularly true in enterprise-class situations where teams are often facing combinations of scaling factors such as larger team size, geographic distribution, and regulatory constraints.  DAD makes it explicit that teams need to invest a bit of time up front to perform initial scoping, initial architectural modeling, and initial planning (all in a lightweight manner of course).  This sort of pragmatic thinking can be inconvenient for less-experienced developers who just want to jump in and start coding.  Because DAD promotes the philosophy of enterprise awareness it purposely bakes in strategies for governance, DevOps, and working with IT-level groups such as your enterprise architects and data management team to name a few.  This can also prove to be inconvenient for developers who want to narrowly focus on doing what’s convenient for their team as opposed to what’s best for their organization.

In Summary

The following infographic summarizes the main points in this blog posting.

We hope that you’ve found this blog posting enlightening.  Even if you are well along the way of your Scrum adoption, or of evolving your RUP-based approach, you can still benefit from switching over to DAD.  Scrum teams will find that it addresses many of the issues that you’re still struggling with, and RUP teams will find that it shows how to work in a far more lightweight manner.  Organizations will find that it provides a much better foundation from which to scale agile strategies.

In this continuing series about how the Disciplined Agile (DA) tool kit supports and enhances DevOps, we overview several strategies to help your organization adopt a Disciplined DevOps strategy:

1. Invest in your people.  In our experience 80 to 90% of your overall effort will be in helping people to learn new skills and ways of thinking and to rethink if not abandon many of the “best practices” of yesteryear.  This requires training, education, and coaching over a long period of time – most people will require many months, and sometimes years, to make the transition to this new mindset.
2. Create a safe learning environment.  Teams must be free to experiment, to try new strategies to discover what works for them in the situation that they face.  Very often this will work out well, with a few stumbles along the way, but every so often the experiment will show that the strategy just isn’t right for this team.  That should still be considered a successful experiment, learning what doesn’t work is just as valuable as learning what does, and the team should not be worried about recrimination for “failed” experiments.
3. Look at the “whole system”.  Disciplined DevOps is about more than just continuous delivery (although that is a great start) and in most enterprises it’s about more than just streamlining development and operations.  With a Disciplined DevOps mindset we strive to improve flow through the entire ecosystem, including development, operations, support, enterprise architecture, data management, release management, and most importantly to the business itself.
4. Improve locally, transform globally.  Each team, including your solution delivery teams, your enterprise architecture team, your operations staff, and many others must strive to improve and streamline the way that they work.  These local improvement efforts must be supported by a “global” transformation effort that focuses on improving DevOps across your entire ecosystem.  Every team will affect other teams, motivating them to make improvements which in turn affects how they work with others.  Your IT department is a complex adaptive system where people and teams learn and improve over time in a dynamic, evolutionary manner.  If you just focus on local improvements your DevOps effort is likely to devolve into chaos.  If you just focus on a company-wide, global transformation it is likely to get bogged down in bureaucracy.  An “improve locally, transform globally” approach is a viable middle ground that benefits from these two extremes while avoiding the disadvantages.
5. Have a communication plan (and work it).  Adopting a Disciplined DevOps strategy within your organization typically requires a lot of (often small) changes.  Although it may be clear to you why this is important it isn’t always clear to everyone else.  People need to understand why you’re making these changes, what’s in it for them, what the overall change strategy is, how far along the plan you currently are, what changes are coming soon, and so on.  You communication plan may include regular newsletters, posters overviewing key concepts, brown bag lunches where people share their experiences, electronic discussion forums, management presentations, and many more.  The keys to success are to have a constant drum beat of information, to be as open and honest about what is actually occurring, to provide opportunities to everyone to learn, and to motivate everyone to share their learnings.
6. Think long-term.  Disciplined DevOps is a journey, not a destination.  It takes time for people to adopt a new mindset, months and often years before it is truly ingrained in their way of thinking.  This paradigm shift does not occur by management fiat, nor does it occur as the result of a day or two of training (although training is important), nor does it occur because you’ve invested in new tooling.  Organizations that successfully make this paradigm shift do so by investing in their people, their process, and their tooling over the long term.

An important financial governance concern in modern enterprises is how to expense costs properly.  This of course includes the costs of IT, including the costs associated with agile software development teams.  There is more to this of course than just adding up expenses, these expenses need to be categorized appropriately and in some cases can have a measurable impact on your bottom line.  In this blog posting we explore how to categorize agile software development costs into either capital expenses (CapEx) or operational expenses (OpEx).

Some Basics

Let’s begin with a few important observations:

1. How you expense cost (and revenue for that matter) is driven first by accounting rules, such as Financial Accounting Standards Board (FASB) in the US, Accounting Standards for Private Enterprises (ASPE) in Canada, and International Financial Reporting Standards (IFRS) IAS 38 in general.  The implication is that CapEx/OpEx decisions in part are driven by geography.  At the end of this posting is an appendix summarizing these regulations.
2. There are a few categories of expense, such as training and education (including coaching related expenses), which automatically fall into the operational expense category regardless of when they occur in the lifecycle.
3. Where the appropriate accounting rules provide leeway, for example financial minimums for what to consider for capitalization, your corporate policy will need to guide you.  A common problem in established organizations is that some of this guidance was set before agile approaches were adopted and reflect waterfall-style governance.
4. How you account for expenses is orthogonal to development paradigm.  It doesn’t matter what method the team follows, the actual capitalization rules remain the same.  The implication is that you need to determine how to apply these rules effectively for each paradigm, and as you’ll see even by lifecycle.
5. This decision, or as we’ll soon show reporting activity, is typically the purview of Finance, not IT.  You really shouldn’t have to worry too much about this.
6. You must engage the appropriate Finance, IT, and Audit people to ensure that everyone understands the implications of how you’re working and agrees to how you will approach accounting for expenses.

Why is This Important?

For publicly traded companies CapEx can potentially boost book value through the increase in assets (in this case a software-based solution) and increase in net income (due to lower operating expenses that year). On the other hand, OpEX is accounted for in the year that it occurs and thereby reduces net income which in turn reduces your organization’s taxes for that year.

How DAD Helps

As you can see in the appendix most of the rules for when to capitalize boil down to “start capitalizing when you know its real”, something that works well with DAD’s risk-based milestones.

As you can see in the diagram, we’ve identified two potential points where you can start capitalizing software development costs:

• Conservative: Stakeholder vision milestone.  This is the point which most organizations use as their CapEx starting point.  As you can see in the appendix, this milestone lines up perfectly with the regulations – you’ve made the decision to go forward with development, you’ve got a technical architecture strategy in place, you have an understanding of what you hope to produce, and so on.
• Aggressive: Project/product team start. Some organizations will choose to start capitalizing at the very beginning of a software development project because they have effectively made the decision to move forward with the effort at that point, they know what the architecture is (due to a good understanding of their existing environment and technical road map), and due to their culture it is highly unlikely that they will cancel the effort.

Of course, there are a few more issues to consider:

• You cancelled the project.  If you cancel a project (or release of a product) most organizations will choose to reverse their capitalization decision for any expenses incurred with that project/release.
• You’re prototyping.  Prototyping, including time spent on proof-of-concept (PoC) or spikes, is usually considered to be an operational expense because it is a learning activity.  This is the case regardless of when in the lifecycle this work occurs.
• Your team follows the continuous delivery lifecycle.  Teams following this lifecycle are in the Construction and Transition efforts all the time, so they’re effectively past the Stakeholder Vision milestone (the Conservative point) at all times.
• Your team follows the exploratory/lean start up lifecycle.  Teams following this lifecycle are not yet at the point where they’ve made the decision to continue with development, and when they do so the typically kick into one of the other lifecycles anyway.  The implication, at least from a conservative point of view, is that these expenses should be operationalized.

Time Tracking

Needless to say, many developers, particularly agile ones, feel that time tracking is a waste.  Spending 5 minutes a week is ok, and to be quite blunt should be more than sufficient, but spending fifteen minutes or more a day doing so is far too much.  For CapEx/OpEx purposes you merely need to track broad categories of software development work such as Development, Prototyping, Vacation, and Training/Education.  These categories will be determined by the applicable accounting rules and should be kept to a minimum.

Our advice is to keep your time tracking strategy as simple as possible, make it very clear to everyone why it’s important to capture their time accurately, and do not use the data to directly measure individuals.  You may find the article Why Time Tracking Might Be the Most Valuable Activity an Agile Developer Performs to be an interesting read.  And, as you would expect from Disciplined Agile, we compare and contrast several Strategies for Tracking Time on Agile Teams.

It’s important to note that tracking time simply for CapEx/OpEx reasons is very hard to justify in practice due to the overhead that it injects plus the loss in team morale (nobody likes tracking time).  Another option is to simply track the amount of time that the entire team spends in a week and then assign a percentage of that time to each category.  This would require a slight amount of effort to track, likely something done by your Team Lead.

The Bottom Line

CAPEX/OPEX decision is fairly straightforward for teams that are following a Disciplined Agile Delivery (DAD) approach.  The team simply needs to keep track of important milestone dates (either the Project/Team Start date or the Stakeholder Vision acceptance date) and track their time in a few broad categories as described earlier.  If you do that, it’s a simple matter of running a report against your time data.

Appendix: What The Regulations Say

Disclaimer: You must read the actual regulations for yourself, what follows is just a summary and may become out of date overtime as the regulations evolve.

US FASB 

• Internal and external costs incurred during the preliminary project stage shall be expensed as they are incurred
• Internal and external costs incurred to develop internal-use computer software during the application development stage shall be capitalized
• Costs to develop or obtain software that allows for access to or conversion of old data by new systems shall also be capitalized
• Training costs are not internal-use software development costs and, if incurred during this stage, shall be expensed as incurred
• Data conversion costs, except as noted above, shall be expensed as incurred
• Internal and external training costs and maintenance costs during the post implementation-operation stage shall be expensed as incurred

International Financial Reporting Standards (IFRS) – IAS 38

You must be able to demonstrate ALL of following BEFORE you can START capitalizing costs:

• Technical feasibility of completing the intangible asset so that it will be available for use or sale
• Its intention to complete the intangible asset and use or sell it
• Its ability to use or sell the intangible asset
• How the intangible asset will generate probable future economic benefits. Among other things, the entity can demonstrate the existence of a market for the output of the intangible asset or the intangible asset itself or, if it is to be used internally, the usefulness of the intangible asset

Note: It is expected that FASB will recognise IAS 38 in 2016.

Canada – Accounting Standards for Private Enterprises (ASPE)

Based on ASPE, you must be able to demonstrate ALL of following BEFORE you can START capitalizing costs:

• Technical feasibility of completing the intangible asset so it will be available for use / sale
• Intention to complete the intangible asset and use / sell it
• Ability to use / sell the intangible asset
• Availability of adequate technical, financial and other resources to complete development and use / sell the intangible asset
• Ability to reliably measure the expenditure related to the intangible asset during development
• How the intangible asset will generate probable future economic benefits

Related Reading

• The Secure Funding process goal
• The Finance process blade

In this posting we briefly explore why both you and your organization should consider adopting a Disciplined DevOps mindset.  For yourself as an individual there are several interesting benefits.  First, you become more productive as an IT professional, increasing your chance at promotion and making you more attractive in the marketplace.  Second, you are in a position where you can focus on interesting, value-added work, which should lead to greater job satisfaction for you.  Third, much of the dysfunctional politics exhibited in traditional IT  organizations is effectively squeezed out as you move to a Disciplined DevOps mindset, making your work environment a more enjoyable place to be.

There are many reasons why your organization should consider adopting a Disciplined DevOps mindset.  The following table summarizes the potential benefits and how they are achieved:

As always, we welcome any feedback that you may have.