Disciplined Agile

On Tuesday, August 7 I facilitated a workshop about Database DevOps at the Agile 2018 conference in San Diego.  I promised the group that I would write up the results here in the blog. This was an easy promise to make because I knew that we’d get some good information out of the participants and sure enough we did.  The workshop was organized into three major sections:

1. Overview of Disciplined DevOps
2. Challenges around Database DevOps
3. Techniques Supporting Database DevOps

Overview of Disciplined DevOps

We started with a brief overview of Disciplined DevOps to set the foundation for the discussion. The workflow for Disciplined DevOps is shown below.  The main message was that we need to look at the overall DevOps picture to be successful in modern enterprises, that it was more that Dev+Ops.  Having said that, our focus was on Database DevOps.

Challenges around Database DevOps

We then ran a From/To exercise where we asked people to identify what aspects of their current situation that they’d like to move away from and what they’d like to move their organization towards.  The following two pictures (I’d like to thank Klaus Boedker for taking all of the following pics) show what we’d like to move from/to respectively (click on them for a larger version).

I then shared my observations about the challenges with Database DevOps, in particular the cultural impedance mismatch between developers and data professionals, the quality challenges we face regarding data, the lack of testing culture and knowledge within the data community, and the mistaken belief that it’s difficult to evolve production data source.

Techniques Supporting Database DevOps

The heart of the workshop was to explore technical techniques that support database DevOps.  I gave an overview of several Agile Data techniques so as give people an understanding of how Database DevOps works, then we ran an exercise.  In the exercise each table worked through one of six techniques (there are several supporting techniques that the groups didn’t work through), exploring:

• The advantages/strengths of the technique
• The disadvantages
• How someone could learn about that technique
• What tooling support (if any) is needed to support the technique.

Each team was limited to their top three answers to each of those questions, and each technique was covered by several teams.  Each of the following sections has a paragraph describing the technique, a picture of the Strategy Canvas the participants created, and my thoughts on what the group produced. It’s important to note that the some of the answers in the canvases contradict each other because each canvas is the amalgam of work performed by a few teams, and each of the teams may have included people completely new to the practice/strategy they were working through.

Vertical Slicing

Just like you can vertically slice the functional aspects of what you’re building, and release those slices if appropriate, you can do the same for the data aspects of your solution.  Many traditional data professionals don’t know how to do this, in most part because traditional data techniques are based on waterfall-style development where they’ve been told to think everything through up front in detail.  The article Implementing a Data Warehouse via Vertical Slicing goes into this topic in detail.

The advantages of vertical slicing is that it enables you to get something built and into the hands of stakeholders quickly, thereby reducing the feedback cycle.  The challenge with it is that you can lose sight of the bigger picture (therefore you want to do some high-level modeling during Inception to get a handle on the bigger picture). To be successful at vertically slicing your work, you need to be able to incrementally model, or better yet agilely model, and implement that functionality.

Agile Data Modeling

There’s nothing special about data modelling, you can perform it in an agile manner just like you can model other things in an agile manner.  Once again, this is a critical skill to learn and can be challenging for traditional data professionals due to their culture around heavy “big design up front (BDUF)”.  The article Agile Data Modelling goes into details, and more importantly an example, for how to do this.

The advantages of this technique is that you can focus on what you need to produce now and adapt to changing requirements.  The disadvantages are that existing data professionals are resistant to evolutionary strategies such as this, often because they prefer a heavy up-front approach.  To viably model in an agile manner, including data, you need to be able to easily evolve/refactor the thing that you’re modelling.

Database Refactoring

A database refactoring is a simple change to your database that improves the quality of its design without changing the semantics of the design (in a practical manner).  This is a key technique because it enables you to safely evolve your database schema, just like you can safely evolve your application code.  Many traditional data professionals believe that it is very difficult and risky to refactor a database, hence their penchant for heavy up-front modeling, but this isn’t actually true in practice.  To understand this, see the article The Process of Database Refactoring which summarizes material from the award-winning book Refactoring Databases.

Database refactoring is what enables you to break the paradigm of “we can’t change the database” with traditional data professionals.  This technique is what enables data professionals to rethink, and often abandon, most of their heavy up-front strategies from the 1970s.  DB refactoring does require skill and tooling support however.  Just like you need automated tests to safely refactor your code, to safely refactor your database you need to have an automated regression test suite.

Automated Database Regression Testing

If data is a corporate asset then it should be treated as such.  Having an automated regression test suite for a data source helps to ensure that the functionality and the data within a database conforms to the shared business rules and semantics for it.  For more information, see the article Database Testing.

An automated test suite enables your teams to safely evolve their work because if they break something the automated tests are likely to find the problem.  This is particularly important given that many data sources are resources shared across many applications.  Like automated testing for other things, it requires skill and tooling to implement. To effectively regression test your database in an automated manner you need to include those tests in your continuous integration (CI) approach.

Continuous Database Integration

Database changes, just like application code changes, should be brought into your continuous integration (CI) strategy. It is a bit harder to include a data source because of the data.  The issue is side effects from tests – in theory a database test should put the db into a known state, do something, check to see if you get the expected results, then put the DB back into the original state.  It’s that last part that’s the problem because all it takes is one test to forget to do so and there’s the potential for side effects across tests. So, a common thing is to rebuild (or restore, or a combination thereof) your dev and test data bases every so often so as to decrease the chance of this.  You might choose to do this in your nightly CI run for example. For more information, see the book Recipes for Continuous Database Integration.

Operational Data Monitoring

An important part of Operations is to monitor the running infrastructure, including databases.  This information can and should be available via real-time dashboards as well as through ad-hoc reporting.  Sadly, I need to write an article on this still.  But if you poke around the web you’ll find a fair bit of information.  Article to come soon.

Concluding Thoughts

This was a really interesting workshop.  We did it in 75 minutes but it really should have been done in a half day to allow for more detailed discussions about each of the techniques.  Having said that, I had several very good conversations with people following the workshop about how valuable and enlightening they found it.

This workshop, plus other training and service offerings around agile database and agile data warehousing skills, is something that we can provide to your organization.  Feel free to reach out to us.

Today in Canada we tested our nationwide emergency response system.  Apparently the test failed in the province of Quebec.  It did in fact succeed in Ontario, where I live.  Knowing about the test I purposefully had my phone on this afternoon because I was interested in what would actually happen.  Sure enough, my phone made a very annoying noise and a message came up to inform me that it was just a test.  So that was good.

An important aspect of IT Operations, and Business Operations for that matter, is to be prepared to respond to emergencies.  While the Canadian government is worried about responding to inclement weather, terrorist attacks, military attacks, and coffee being sold out at the local Timmies your IT department should be concerned about ensuring that your systems are running properly, that they are repelling cyber attacks, and that your data centres are operational (to name a few potential issues). This is why the IT Operations process goal includes a process decision point called Mitigate Disasters (see the pic above).

By running this scheduled disaster simulation, after careful planning and communication (which I why I had heard about it), the Canadian government has discovered in a controlled test that their strategy needs work.  This is exactly the type of thing you want to find out when you have the luxury of safely addressing any problems that you do find.  The government certainly wouldn’t have wanted to discover their emergency alert system didn’t work as expect in the middle of an actual emergency.

What your organization should ask itself is what would happen if:

• One of your data centres lost power?  Or connectivity?
• Some of your servers went down?
• Outsourced services you rely on (think SAAS, PAAS, and other cloud solutions) went down?
• An application/system went down?
• A denial of service (DoS) attack succeeded?
• And many more issues.

Will your IT ecosystem respond properly?  Will it recover automatically?  Are you guessing at these answers or do you know for sure because you’ve actually simulated them?

I hope this blog has been food for thought.  Time for a Timmies.

In this blog posting we overview the decision point, Choose an SCM Branching Strategy, of the Accelerate Value Delivery process goal.

Figure 1. The goal diagram for Accelerate Value Delivery (click to enlarge).

As you know, the Disciplined Agile (DA) tool kit helps you to make better decisions by describing the trade-offs associated with a practice/strategy.  The following is an excerpt from the Choose Your WoW! book:

Accelerate Value Delivery==>Choose an SCM Branching Strategy

We need to identify our team’s branching strategy for our source code repository.  A branch is a copy or clone of all, or at least a portion of, the source code within the repository. We branch our code to support concurrent development, capture of solution configurations, multiple versions of a solution, and multiple production releases of a solution. so that it may be worked on in parallel.  As you can see in the following table there are many branching strategies available to us, strategies which may be applied in combination.

By now many of you will have heard about the tragic loss of Mike Beedle, one of the signatories of the Agile Manifesto and the thought leader behind Enterprise Scrum.  Mike passed away on March 24th and leaves behind a family that includes six children.

For those of us who were lucky enough to know Mike we’ll remember him as a generous and intelligent gentleman.  He made significant and ongoing contributions to the agile community and we’ll all miss him.

The Disciplined Agile Consortium is proud to have donated to the Memorial Fund for Mike Beedle’s Family and we hope that you consider doing the same.

Rest in peace Mike.

A common question that we get is what is the difference between Product Owners (POs) and Product Managers? From a Disciplined Agile (DA) perspective, it’s a matter of strategy vs. tactics:

• • Product Owners are more tactical in practice.  POs work closely with delivery teams to ensure they build the right functionality in a timely manner. POs will transform the high-level vision of the Product Manager into detailed requirements. To do this they work closely with a range of stakeholders for the product, including non-customer stakeholders such as finance, security, operations, support, audit, and others.  Tactical activities such as attending team coordination meetings, organizing demos, doing sufficient analysis to ensure that requirements are ready to be worked on, and being involved with ongoing testing efforts easily add up to a full-time job.
  • Product Managers are more strategic in practice. They should be focused on the long-term vision for the product, on observing trends in the marketplace, on identifying new potential outcomes or themes to be supported by the product, on supporting the sales/adoption of the product, and on ensuring the product meets the needs of the value stream(s) the product is involved with. Effective Product Managers tend to be very customer focused, although recognize that this needs to be tempered by the constraints and capabilities of your organization. The activities that Product Managers are responsible for – product marketing, supporting product sales/adoption, budgeting, long-term envisioning, customer care, and of course supporting the solution delivery team(s) – can easily add up to a full time job.

We Need to Collaborate

As you can see in the following diagram, the role of Product Manager is different, yet overlapping, with that of a Product Owner (PO).  The PO will spend the majority of their time on tactical activities, including working with the team to communicate stakeholder needs to them and working with stakeholders to elicit and prioritize their needs. The Product Manager, on the other hand, spends most of their time on more strategic issues, collaborating closely with customers (and potential customers) to identify their potential needs.

Figure 1. Example of rolling wave planning for product functionality (click on image for larger version).

There is clearly overlap between strategic, long-term thinking and tactical, short-term implementation.  Product Owners are responsible for the Product Backlog in Scrum, what Disciplined Agile DAD (DAD) teams might refer to as a Work Item List or in the case of teams who have adopted one of the lean lifecycles a Work Item Pool, and some of the items in the backlog/list/pool might be several months away from being implemented (if ever).  In Figure 1, these are items that fall into the yellow or red timing areas, or even the grey area.  Product Managers, being responsible for strategic thinking, will be focused on high-level outcomes or themes for the product.  They may even be focused on more concrete, yet still high level, epics or features.  So we see overlap in the Product Manager’s high-level strategic focus and the Product Owner’s tactical focus, indicating the need for collaboration between the two roles so that the tactical decisions reflect the overall strategy, and the overall strategy is informed by the realities faced on the ground by the delivery team.

Please note that the timing of “short term” and “long term” will vary by product.  In the case of Figure 1 the long-term planning horizon is around the three month point (where the diagram shifts from yellow to red).  That’s just an example, from one team.  We’ve worked with some teams where the long-term planning horizon was anything more than a month.  We’ve also worked with other teams where the long-term planning horizon was closer to a year (they’ve since shortened that considerably).

Shouldn’t Product Owners Also Address Strategic Issues?

Here are a few thoughts to help answer this question:

1. Everyone should consider strategic issues.  Some people, particularly those focused on Scrum, will tell you that Product Owners should also be focused on strategic issues.  It’s certainly good for POs to understand the long-term strategy for the product that they are focused on. In short, POs, like everyone else, should be Enterprise Aware.
2. Each role requires a different, and comprehensive, skillset.  Each of these roles are challenging enough by itself. You’ll have a much better chance of finding someone with the skills to work tactically, and someone with the skills to work strategically, than finding a single person with both skillsets (or the time and inclination to pick up both).
3. There is often too much work for one person.  As we argued earlier, the day-to-day tactical work tends to be a full-time job (and often more) as does the strategic Product Management work.  As a result, you are often motivated to tease these two roles out into separate positions.
4. These are roles, not positions. In straightforward, non-scaled situations, it is common to see a single person taking on both of these roles.  This is common in start-up organizations where the company simply can’t afford to have two people to do this work.  It’s also common with new products in general because it isn’t yet obvious whether the product will be sufficiently successful in the marketplace to warrant much investment in long-term strategic thinking around it.

So, as usual, the answer is “it depends.”  As we like to say in DA, context counts which is why choice is good.

Related Reading

• The Product Management process blade
• Where Do Product Owners (POs) Come From?
• Rolling Wave Planning for Product Roadmaps