Disciplined Agile

On Wednesday October 21, 2015 we hosted a webcast entitled “Crushed by Technical Debt?” A PDF of the slides can be found on Slideshare.  The webcast addressed the following issues:

• What is technical debt?
• How can we avoid technical debt?
• How do we identify technical debt?
• How do we fix technical debt?
• When should we accept technical debt?
• How can we fund technical debt?

As you can imagine we ran out of time to answer all of the questions that we received during the presentation. Below are answers to all of the questions we received, even the ones that were answered during the Q&A portion of the webcast. The questions, and their answers:

• Stories for technical debt?
• How to convince product owners about technical debt?
• Too many warnings from your code analysis tools?
• Too much technical debt?
• Measuring technical debt?
• Fix or start from scratch?
• A negative awareness score for senior business management?
• Forecasting technical debt?
• Can architecture owners work on multiple teams?
• Where are the goal diagrams?

Additional topics:

• A special thank you
• Related reading

Question: What’s your opinion regarding separating tech debt into separate stories?

This is one way of doing it. At one point during the presentation we discussed three strategies for how to fund paying down technical debt.  These three strategies are:

1. Technical debt removal as regular investment. The idea is that developers automatically pay down technical debt, just like you would automatically clean up spilled juice in the middle of your kitchen floor, whenever you run into it.  In effect paying down technical debt becomes part of your culture.
2. Stories to address technical debt. The idea is that you have requirement artifacts, such as technical stories, to remove a small amount of technical debt within your environment.
3. Projects to address technical debt. With this approach you have specific projects to rewrite portions of an IT solution, or perhaps even to remove that solution completely.

Question: How to convince Product Owners to address Technical Debt?

Product Owners (POs) are responsible for prioritizing the work. When technical debt is addressed via stories then the PO can easily deprioritize such stories.  The implication is that the team members, and in particular the Architecture Owner, must work closely with the PO to help them to understand the implications of technical debt and why it’s important to pay it down.

When technical debt is addressed automatically via regular investment, it is a built in overhead that affects your team’s overall velocity (negatively in the short term, but positively in the long term.

Question: What do you do when there are to many warnings thrown by your code analysis tools?

Regarding using code-analysis output as a metric: do you have any specific advice for dealing with the scenario with legacy code where it will initially overwhelm you with thousands of issues, from the trivial (i.e. spelling mistakes) to major : i.e. should one start by switching off the trivial warnings and deal with the major, or start fixing the big-numeric-count issues no matter how trivial just to get the numbers down?

It’s harsh to say, but the reason why you’re getting a lot of warnings is because the code sucks. When this is the case, it’s important to recognize the situation for what it is and get going on fixing the problems.

Initially I would turn off a lot of the lower priority checks so that the team could focus on the critical issues with their code. Once they refactor those away then I would turn on the next group of high-priority checks and so on until the problems are dealt with.  This could take a long time, so you’ll need the discipline to stay at it.  Perhaps start each retrospective with the question “How many code checks can we turn on this coming iteration?”

Question: Lack of (enough) clean code from the first time?

This is always a major frustration for people.   It is incredibly rare to be in a situation where clean code was written from the very beginning and then kept clean via techniques such as refactoring.   Instead most people find themselves swimming in legacy code, data sources, user interfaces, and so on that contain lots of technical debt.  Our advice is to accept:

• The situation that you find yourself in
• That the technical debt isn’t going to fix itself
• It’s only going to get worse
• You need to start paying down your technical debt now

If it makes you feel any better, and it likely won’t, the 2014 Software Development at Scale survey found that 92% of agile teams were facing some form of technical complexity.

Question: How we should measure technical debt?

There are many different ways to measure technical debt, all focused around solution quality. You may choose to measure code quality, perhaps in an automated fashion via code analysis tools, for example.  You may also choose to track defect trends over time, although that can easily become a measure of how good you are at testing instead of the quality of what you’re producing.

Question: How do I know if it’s better to fix technical debt or start all over from scratch?

This typically comes down to a gut feel based on your experience doing such work in the past.

Question: What does a negative score for technical debt awareness means? How should that be read & interpreted?

This is a reference to one of the charts from the 2015 Agile Cadences and Technical Debt Survey, see below , where senior business management had a negative score.  The range is from -10 (very unaware) to +10 (very aware).  So on average business management leaned towards being unaware of the implications of technical debt.  This is a problem seeing as the business needs to fund the removal of technical debt.

Question: Is there any best practice/method to forecast Technical Debt?

Unless you really have your act together, the forecast is that technical debt is going to get worse over time.  Seriously though, that’s why you need to have a technical debt metrics program in place. In this case you want to track the trends over time, and simply extrapolate those trends to forecast the implications to your organization of technical debt.

Question: Can one architect span multiple teams? That is, is it a good idea to have the same person as the architect across multiple teams?

In Disciplined Agile we explicitly include the role of Architecture Owner (AO), which was adopted from the Agile Modeling methodology. This is effectively an agile solution/application architect who works in a collaborative, evolutionary, and facilitative manner.  Every delivery team should have someone in the AO role to help guide them through architectural issues.  Ideally each team has one person in this role and they are dedicated to the team full time, just like other team members.

Having said that, as we like to say “you go to war with the army that you’ve got.” For example, you may have ten agile teams but currently only four people with the skills to be an AO.  In this situation each AO is going to be a member of two or three teams.  As a result they will suffer a personal productivity loss from task switching and their teams will suffer a productivity loss from having to sometimes wait for them to be available.  This isn’t ideal but it’s the reality that many organizations face.  The solution is for the AOs to work collaboratively with other team members so as to start passing on their skills and knowledge to them, to help grow their fellow team members.

Question: Are the goal diagrams available in a single document?

Great question. Sadly the quick answer is that there isn’t a print-based one at the current time.  Our focus has been on publishing to the web, ignoring the fact that we recently published the 104-page book Introduction to Disciplined Agile Delivery. We have published a landing page for the Process Goal Diagrams for Disciplined Agile Delivery. We are also actively publishing the goal diagrams for the process blades, such as Enterprise Architecture and Portfolio Management, as well.  The easiest way to access them is from the diagram on the Disciplined Agile Home Page (just click on the corresponding bubbles). This is a work in progress and we hope to have the process blades descriptions in place by December.

Having said that, we have been thinking about publishing some sort of “Disciplined Agile for Coaches” book that would include all of the goal diagrams.

A Special Thank You

We’d like to give a shout out to Valentin Tudor Mocanu for answering several of the questions in the chat window while Scott was speaking.  

Related Resources

• Article: 11 Strategies for Dealing with Technical Debt
• Slides (PDF): Crushed by Technical Debt?

Imagine this: You go home tonight, you walk into your kitchen, and you see that there is orange juice spilled on the floor.  You’re the only one home, you know that you didn’t spill the juice, but there it is on the floor anyway.  This is clearly a problem.  What do you do?

Odds are that you clean it up, because you’re an adult and that’s what adults do.  You don’t walk through the orange juice because that would track it through the rest of your home, making the problem worse.  You could leave the juice there on the floor in the hopes that someone else will come along and clean it up. But if that doesn’t happen you run the risk of your dog lapping it up, and then getting sick from it because citrus is poisonous to dogs.  The problem will have gotten worse. Or perhaps the juice will slowly dry up, attracting all sorts of bugs, once again the problem gets worse.  Additionally, you’ll be forced to constantly walk around the spilled juice thereby slowing you down.  But luckily you’re an adult so you take a few minutes to clean up the juice and thereby avoid these obvious problems.

Now imagine this: You go to work tomorrow, you open up a Word document or begin working with a data from an existing database, and you discover some technical debt.  Perhaps the wording in the document isn’t very clear or perhaps the data has inconsistencies, or there’s a myriad of other issues.  This is clearly a problem.  What do you do?

Odds are that you do nothing about it.  You’d like to do something about it, you yearn to do something about it, but you perceive yourself as not having the authority to do so.  Or perhaps you don’t have the time to fix the problem because you need to focus on getting that document to your boss.  Or perhaps you don’t have the skills to fix the problem, or the tools to do so, or even recognize that the problem can be fixed (for example, many data quality problems exist because the vast majority of data professionals haven’t learned fundamental techniques such as database testing and database refactoring).  Or perhaps you don’t have the funding to fix the technical debt because your leadership doesn’t understand it and as a result chooses to fund new work over paying down technical debt.  Or perhaps you see the problem as being overwhelming, not realizing that you can chip away at it a little bit at a time.  Or perhaps you realize that you can safely ignore the problem and let the next person to run into it to worry about it, just like someone in the past has clearly done to you. The point is that you have a litany of excuses to choose from for why you can’t fix the technical debt, so unlike the orange juice spill you don’t clean up the mess.

Unfortunately, just like the orange juice spill, the technical debt problem will only get worse.  You’ll build write new information in addition to the poor quality writing, adding to your overall technical debt.  You’ll propagate the poor quality data through yet another part of your application, adding to your overall technical debt.  You’ll add new tests (hopefully) for the new functionality you’re adding but won’t take the time to add tests in for the existing functionality, adding to the cost of that existing technical debt.  How can this possibly be a good thing for you, or for your organization?

The solution of course is that we need to take responsibility.  This is a choice.  It takes time and it takes investment in yourself.  Just as it took you years to build up the habit of cleaning up spilled juice when you see it (I’m going to go out on a limb and assume that when you were a child you had to be told to clean up the juice many times before you built up the automatic habit) it will also take time to build up the habit of automatically addressing technical debt when you discover it.  While you are on your learning path you’ll also need to convince your colleagues to do the same.  You’ll also need to work with others to understand the implications of technical debt and help convince them to invest in paying it down.  As an organization you need to choose to dig your way out of the technical debt pit.

Related Reading

• 11 Strategies for Dealing With Technical Debt

The Disciplined Agile tool kit is guided by the following principles:

1. Choice is good, and making informed choices is better.  Every team is a collection of unique individuals that face a unique situation within the context of a unique organization. One process size does not fit all. To provide choice the DA toolkit supports several delivery lifecycles and is process goal driven. Most importantly the DA toolkit describes the tradeoffs involved with a myriad of agile and non-agile practices enabling people to make intelligent decisions regarding which practices to adopt given the current situation that they face.
2. Optimize the whole. The DA toolkit addresses the full IT lifecycle, showing how it all fits together. Without an understanding of the larger process environment teams run the risk of locally optimizing their own processes to the detriment of the whole. For example, your data management team may have their own streamlined process based on traditional DAMA strategies, your delivery team may have their streamlined process based on the principles of the Agile Manifesto, and your operations team may have their streamlined process based on ITIL. Yet your overall process is ineffective because these three locally optimized strategies contradict and degrade one another when combined.
3. Every team owns its process. Teams, and the individuals on them, must be free to improve the way that they work based on their learnings over time. In agile parlance we say that these teams “own their process”.
4. Improve continuously. Individuals, teams, and organizations must strive to continuously learn and improve the way that they work. The DA toolkit includes the process goal Improve Team Process and Environment which describes options for doing exactly what its name implies. It also has an explicit process blade Continuous Improvement that describes strategies for sharing improvements across teams, thereby speeding up your organization’s process improvement efforts.
5. Embrace process change. IT departments are complex adaptive systems. One implication of this is that any improvements that a team makes may change the way that they work with other teams, motivating process improvements within those teams. Those changes may motivate improvements within other teams and so on. Disciplined agile teams are enterprise aware and understand that they will need to work with other teams to help them to understand and adopt new innovations, and be prepared to be helped by others to do the same.
6. Repeatable results are far more important than repeatable processes. Effective teams focus on producing repeatable results, such as delivering high-quality software that meets stakeholder needs in a timely and cost effective manner.  Because each team finds themselves in a unique situation, to be most efficient they need to follow a unique process tailored to reflect that situation.  That “unique process” may be comprised of a relatively standard lifecycle and common practices such as architecture envisioning, database regression testing, non-solo development, and many others (granted, those practices may be tailored to reflect the situation too).  Each team in your organization must be allowed to follow their version of the process, ideally sharing similar process components defined by a common process framework, to achieve the results required of them.
7. Empiricism is far more important than theory. Observing how well a technique works in practice, and more importantly the context of the situations in which it (doesn’t) work is far more valuable to practitioners than theories or prognostications about what should work. Theory has its place, but it is a poor cousin to empiricism. The DA toolkit was originally developed based on observations of dozens of organizations worldwide, and has evolved since then based on learnings from many more. Furthermore it is backed up by our ongoing industry research.

IT departments are unique, complex adaptive systems. Anyone working in such environments needs a process framework that is sufficiently flexible to address the range of situations faced by your teams. The Disciplined Agile process decision framework is light-weight yet sufficiently flexible to support scaling at both the tactical and strategic levels.

Translations

• Spanish

This posting, the latest in our series focused on a disciplined agile approach to continuous improvement, overviews the activities associated with it. The Disciplined Agile (DA) toolkit promotes an adaptive, context-sensitive strategy.  The framework does this via its goal-driven approach that indicates the process factors you need to consider, a range of techniques or strategies for you to address each process factor, and the advantages and disadvantages of each technique.  In this blog posting we present the goal diagram for the Continuous Improvement process blade and overview its process factors.

The following process goal diagram overviews the potential activities associated with disciplined agile continuous improvement. These activities are performed by, or at least supported by, a process improvement team (sometimes referred to as a Software Engineering Process Group, or SEPG).

The process factors that you need to consider for continuous improvement are:

1. Identify improvements.  There are several ways that your process improvement group can support the identify of potential improvements within your organization.  One of the more effective strategies is to help teams adopt the practice of holding regular retrospectives.  Although it is common for disciplined agile delivery teams to hold retrospectives this is often a new concept for enterprise architecture teams, IT governance teams, data management teams, and so on.  We also get very good traction with value stream mapping and brainstorming sessions, which invariably proves to be far more effective than the traditional approach of creating current and proposed (business) process models that rarely seem to result in lasting acceptance of the proposed new way of working.
2. Share improvements.  As you can see there are multiple ways that you can share improvement ideas between teams, many of them being free or at least very inexpensive to implement.  We’ve had very good experiences with internal discussion forums such as Jive or ActiveBoard, practitioner presentations (often called lunch and learns) where someone presents  their learnings to others, lean coffee sessions where people voluntarily meet at a regular time to share ideas, and communities of practice (sometimes called guilds) who purposely collaborate to educate themselves in a given topic.
3. Capture improvements.  There are various ways that identified improvements may be captured to retain them over time.  Possible strategies include describing each learning in a document and then managing that document in a documentation repository such as Sharepoint or more simply in a shared folder; capturing the learnings in a shared wiki such as Confluence; describing your evolving process using a process repository such as Stages or Rational Method Composer; or via an expert system such as Enterprise Transformation Advisor.
4. Support teams. Your process improvement team can help others to adopt process improvement techniques through training, education, and coaching.  They can also facilitate team assessments and retrospectives (a great idea is to co-facilitate a few retrospectives with someone on the team to transfer those skills to them).  A very effective strategy is to help a team run a process improvement experiment or two, particularly in situations where the team isn’t being supported by a coach.  This helps them see that they can safely try new ideas within their environment for a few iterations to determine whether it works for them or not.  Many teams, particularly those new to agile, often do not feel empowered to run such experiments and thus need help to do so.
5. Organize Communities of Practice (CoPs).  A Community of Practice (CoP) is a collection of people who share a craft or profession who have banded together to ‘learn’ from each other to develop themselves and often even the  organization.  We’ve seen CoPs for testing, architecture, agile/lean, business analysis, technical debt, and many others.  CoPs will often perform the activities called out by the Identify Improvements, Share Improvements, Capture Improvements, and Support Teams process factors.  A CoP will often start up when one or more practitioners within your organization recognize the need for it, although sometimes it may also start to support the efforts of a corresponding Center of Excellence (CoE).  Participation in a CoP is typically voluntary.
6. Organize Centers of Excellence (CoEs).  A Center of Excellence (CoE) is a group of people with specialized skills and expertise whose job is to provide leadership and purposely disseminate that knowledge within your organization.  CoEs are often created by an organization to support the adoption of a new technology or technique, and in fact  the creation of an Agile CoE is often a key component of your organization’s overall Agile transformation efforts.  Over the years we’ve seen CoEs for object technology (particularly in the 90s when it was new to most companies), solution architecture, testing automation, and of course agile/lean.  Participation as a member of a CoE will be part of, or the entire job for someone.
7. Govern improvement.  It is very common for senior management to want to know whether or not the organization is benefiting from your investment in adopting agile and lean techniques (or other potential improvements for that matter), how much things are improving, and how widespread the adoption is.  The implication is that there needs to be some way to monitor and report on, preferably in a lightweight and streamlined manner, the improvement activities.

Future blog postings in this series will explore the workflow between continuous improvement and other process blades as well as the internal workflow of a process improvement team.

The purpose of the Continuous Improvement process blade is to enable people within your organization to easily share their improvement learnings with one another in a systematic way. There are several reasons why you want to have a continuous improvement program within your organization:

1. Shorten the time from idea to implementation. Improvement ideas can come from anyone, at any time, from anywhere in your organization. As a result you want to have organizational mechanisms to identify and explore those ideas so that they get to the person(s) most suitable to implement them quickly.
2. Increase skills and knowledge sharing. The high-collaboration environments that are typical of agile teams are wonderful for sharing skills and knowledge within each team, but fellow team members aren’t the only people within your organization that you can learn from. An important goal of a continuous improvement program is to motivate and enable people to share their skills and knowledge outside of their immediate team. You can do this through strategies such as communities of practice, online discussion forums, practitioner presentations and many others.
3. Maximize your “failure ROI”. A fundamental of lean thinking is to learn from your failures, to treat each “failure” as an opportunity to improve. Having said that, every team doesn’t need to experience all of the same failures. One team, or a handful of teams in some cases, can fail in similar ways and then share those learnings with others. This way other teams can avoid that type of failure and thereby increase the value of the learnings to your organization. But we can only do that when it’s safe to fail and better yet recognize that failures should be celebrated and the learnings shared with others.
4. Increase the opportunity for radical improvements. The challenge with the Japanese concept of kaizen, which is to continuously make small incremental improvements, is that you can get on an improvement path that will never lead to a quantum leap in your process. Yes, things are getting better, but you may be missing opportunities to make things a lot better. For example a team following the Scrum-based Agile/Basic lifecycle may never identify the strategy of continuous deployment (CD) on their own because having a two-week iteration may preclude the idea of releasing several times a day into production. Yet, if people on your team were to hear about other teams in your organization working that way, they might soon choose to start experimenting with CD techniques. This in turn could lead to the “radical” process improvement of abandoning the idea of time-boxed iterations and moving to something much closer to DA’s Continuous Delivery lifecycle.

In short, your organization needs a strategy for communicating potential improvements across teams. Ideally the flow of work should be streamlined to make it as easy as possible for teams to learn from one another. There are many options for doing so available to you, the topic of a future blog posting.