Agile Today: An Interview with Jim Highsmith

2. A move from engineering and team practices to management practices. In parallel, we’re seeing a move from engineering, engineering practices and team practices to something that is more focused on management and management practices. This is something that is really just getting underway in the last few years, where management and executives have taken a look at this and said, “What do we need to do to make our organizations more Agile?” (as opposed to just making our software engineering practices more Agile)  This sort of change from an engineering to a management focus is further demonstrated by the Project Management Institute getting into Agile and Agile certification; not everyone in the Agile community is happy with this, but at least it shows that there is a lot of interest within the project management community as well as within the technical community.

3. A move from Agile development to continuous innovation.The last thing that I would talk about is Agile Development moving from strictly development to continuous delivery and deployment. So your software, in addition to being developed on a short cycle, is being released on a very short cycle in those situations where it makes the most sense. So you have a situation where you are automating that last mile. On the front end of that, the exploratory or discovery phase, the lean startup movement has provided some practices and philosophy around getting starting faster with product inception and product requirements. Agile development to continuous innovation.

I’ve been focused on the management side of these changes, which I call adaptive leadership: 

• Why is it more important for managers and leaders in organizations to be more Agile or adaptive?
• What do these managers and executives need to do differently? What are the actions that they need to take?
• What are the behaviors that we need to exhibit as Agile managers?

Q.  Could you expand a bit on what it means to be Agile?

In fact, I just wrote a blog posting yesterday entitled “What is Agility?”.  To me there are two aspects of Agility:

1. The ability to create or respond to change in order to profit in a turbulent business environment.  So it’s not just responding to change, but the ability to create change in innovative ways to challenge competitors.

2. The ability to balance flexibility and stability.  A lot of people think that agility promotes sort of a lack of structure. But really, if you don't have any structure at all you just kind of go off into chaos. So you really do need structure. The really critical management capability here is to decide how much structure you need and balancing that with the flexibility that you need to respond to the marketplace.  Traditional approaches have come down on the side of more structure, more process and more standards, and have really restricted our ability to innovate because we don't have that flexibility, adaptability and learning ability that we need to be competitive.

Q.  When trying to understand the limits of Agile practices, most people focus on project size. Yet you talk more about complexity and uncertainty, which are related but not as easy to get your head around. Generally speaking, what is the best way for practitioners to think about these two attributes of programs and projects and how each of them should affect their approach? Any thoughts on this specific to planning?

I think that there’s sort of a myth in the community around project size.  Part of this comes from the early Agile work that focused more on small teams and small projects. So the idea was that Agile worked well on small projects, but didn’t scale well to larger projects – but there are plenty of examples now of Agile working well in very large projects and large organizations.

You should really think about this is terms of a few key questions that Agile helps answer:

• At what size does delivering value to customers fail to be important? (there is no such size)
• At what size does the core value of creating better workplaces fail to be important? (there is no such size)
• Can large organizations afford to be inflexible rigid and unresponsive? (of course they can’t)

So there are a lot of reasons that we want to be able to apply Agile as we scale.  So we want to scale Agile, but what are some of the things that we need to do to make that happen? That’s where complexity and uncertainty come into play.  Complexity and uncertainty are two dimensions that we need to think about.  The complexity dimension is the one that we are most familiar with; team size, distribution, domain knowledge, computational complexity all play into overall complexity.  Common uncertainty factors can include the newness of the technology you are using, business and marketing uncertainty, new/emerging product requirements. You can have any combination of high and low complexity and uncertainty on any given project, but the general strategy for scaling projects are as follows:

• Increased complexity calls for increased structure: organizationally, from a communications perspective, documentation and use of tools.
• Increased uncertainty calls for the incorporation of practices that make you more agile: increasing learning, shortening feedback cycles and increasing experimentation.

Obviously, the more difficult projects are the ones that have high complexity and high uncertainty.  With those kinds of projects you usually have a smaller subset with high uncertainty that you can deal with separately. I think the real message here is that you have to adapt and use some more traditional strategies and some Agile strategies as appropriate. Often you end up using an Agile strategy overall and incorporate traditional elements into it as needed.

Q.  A key challenge for organizations that we often hear about is marrying Agile projects with more traditional Project Portfolio Management efforts. The clear goal is to gain the benefits of Agile approaches while attempting to maximize and manage return on investment.  Do you have any thoughts, guidance or best practices that you could share in this area?

I think in terms of exploration (certainty) factors with my projects along two dimensions: technology and requirements. If the technology is very familiar to me, the technology exploration factor is low.  If the technology is new, the exploration factor is high.   The same is true for the project’s requirements. If I have an exploration factor 10 project, things are pretty uncertain.  If I’m dealing with a factor of 1, things are pretty certain. Most PPM approaches standardize/homogenize projects, making them seem equally predictable – which doesn't make any sense. For example, to expect predictability within +/-10% of time or cost on a exploration factor 10 project just isn’t reasonable.

So we need to change governance structures and govern different types of projects based on different criteria. With projects that have a low exploration factor, we can use more traditional portfolio measures like scope, schedule and cost. With higher exploration factor projects, we have to ask:

• What’s the value the project will generate?
• What’s the minimum viable product?
• What can I release out the door as soon as possible in order to get constructive feedback? 

Then the other things like cost and scope become constraints – not drivers. So you have to either run two different types of portfolio management systems or one governance process that combines both traditional systems and Agile. One of the things that I’ve seen lately is continuous monitoring and management of the portfolio, using sort of a Kanban approach. It’s very similar to a traditional approach in some ways, but all of the actions are continuous.

Q.  Uncertainty is something that every company is dealing with today. It’s a key concern for every executive.  Moving Agile up the food chain, are there steps that can be taken to make a traditional PPM process more adaptive and perhaps more effective that are usually well received by the executive team?  Are there ways of coaching executives to make them more Agile-friendly and ultimately more effective in a world that is filled with uncertainty?

The transition is more a management transition than it is a project or portfolio management transition.  There is more of a realization now that managers themselves need to learn to be more adaptive. In the past, we’ve seen the attitude that the development teams need to be more Agile and adaptive, but managers thought, “It doesn't really impact me.” I think people are struggling with changes in the marketplace.  For example, Amazon fought for years against applying state taxes to Internet sales.  Now they’ve changed strategies entirely because they are moving toward same-day delivery. That’s an industry-wide disruption and an example of management thinking in a more Agile way. So that’s how the mindset changes.

The “doing” side of it really involves creating a continuous stream of value:

• Being able to speedily deliver value.
• Doing less, figuring out minimum viable products that deliver value.
• Doing those at very high quality so you can do it continuously. It’s not about doing release 1 and 2, but doing release 1, 2, 3, 4, 5, 6 in very rapid succession.
• Having a well-tuned engine to make all of this happen in a rapid, high-quality way.

The “being” side of that (management behaviors) has to do with creating an innovative culture:

• Thinking about continuous change and adaptability as the norm.
• How do I get my mind around having an “adaptive mindset”?
• How do I measure progress differently?
• How do I look at change as an opportunity versus as a problem?
• How do I adapt an “envision-explore” mindset versus a “plan-do” mindset?

With a plan-do mindset, I plan out everything that I want to do – then I do it.  With an envision-explore mindset, I envision where I want to go and how I might get there.  With envision-explore, I'm creating an innovative culture.  With plan-do, you’re really not.

So in terms of dealing with uncertainty, the things that I just talked about help you deliver a continuous stream of value, while creating an innovative culture.

Q. Following the theme of uncertainty, what are your thoughts on planning in an uncertain environment? 

I think planning is definitely something you do, but really you’re “speculating”. In my book, I actually replace the word “planning” with “speculating”, because that’s what you’re really doing in a highly uncertain environment.

When a lot of people think about planning, they think about a Microsoft Project plan with 6,000 tasks and someone is going through and checking them off as they are completed. I think that kind of plan isn’t really appropriate in a rapidly changing environment.

What is appropriate is a higher level of planning where you are dealing with shorter periods of time and you’re dealing with larger chunks of work.  Again, you would plan an exploration factor 10 project differently than you would plan an exploration factor 5 project. If it’s a 10, you might just run an iteration at a time with a broad goal in mind and a few constraints – then run a series of experimentations on it until you figure out what it's all about.  With a 5, you might be able to lay out a reasonably detailed plan for three or six months.

Another key difference between traditional an Agile planning is what you are actually tracking. Traditional planning tends to be task or activity based.  In Agile planning, you are doing a product breakdown. So you’re tracking pieces of the product that have value to the customer versus activities where their value is less clear.

Q.  Organizational effectiveness has also been a popular theme in recent years. At various levels you see Agile, Lean and even Six Sigma playing roles in helping organizations produce more with less. Do you see these disciplines becoming more integrated or even somewhat merging at some point in the future? How do you see them complementing or detracting from one another?

The ones that I see complementing each other are more the Agile and Lean – particularly the Lean startup movement. The idea of doing less and focusing on high-value stuff is really a complement to Agile.

I haven’t had as much experience with Six Sigma, but it’s more about operational efficiency. There may be places where you want to be more operationally efficient. Then it’s a matter of creating a balance.

Organizations have to think about:

• Do I really want to be focused on being more responsive? This is the driver that Agile and Lean help with.
• Do I want to drive down costs (become more efficient)? This is a constraint that Six Sigma helps with.

Q. Organizational effectiveness often means standardization. Many large organizations struggle to "standardize" their teams.  What are your thoughts on this type of standardization? 

I think it depends on whether your focal point is responsiveness or efficiency. If your focus is on becoming more efficient, then standardization probably helps with that.  Unfortunately, that drive toward standardization often operates against the ability to be more adaptable, flexible or responsive. The more standard you get, the less flexibility you have. In many large organizations, this drive to be standard encompasses too many things.  There may be some areas where standardization is appropriate, but “teams” is not one of them. You need diversity and a set of guidelines, but not much in terms of standardization.

Q. Customer value is obviously critical, but maximizing it sometimes feels in conflict with getting things done. If you had one piece of advice for those trying to maximize value in a tough environment, what would that be?

I would go back to two of the four things that I think adaptive leaders need to do:

• Deliver value to the customer
• Do less to deliver that value

One of the things traditional approaches assume is that you’re successful if you’ve delivered all of the scope in whatever you’ve defined. There have been studies that say that 60% of all software developed is rarely or never used. So the measure really needs to be, “Have I delivered value to the customer today?” not “Have I delivered everything in scope?” Then you might ask the question, “What offers value now?” Then you can grow that value over time.

So the Lean idea of doing more with less, combined with an Agile approach, really produces more meaningful results.