Project Management

Disciplined Agile

by , , , , , , ,
#ChooseYourWoW | #ContinuousImprovement | #Kaizen | #ProcessImprovement | Adoption | agile | Agile certification | agile transformation | Analogy | Architecture | architecture | book | Business Agility | Certification | Choose your WoW | CMMI | Coaching | Collaboration | Compliancy | Configuration management | Construction phase | Context | Continuous Improvement | COVID-19 | Culture | culture | DAD | DAD discussions | DAD roles | Data Management | database | DevOps | Discipline | disciplined agile delivery | Documentation | DW/BI | Enterprise Agile | Enterprise Architecture | Enterprise Awareness | Essence | estimation | Evolving DA | Experiment | Financial | GDD | Geographic Distribution | global development | Goal-Driven | goal-driven | goals | Governance | Guideline | Improvement | inception | Inception phase | Large Teams | layer | Lean | Lifecycle | lifecycle | Metrics | mindset | News | News and events | Non-Functional Requirements | non-functional requirements | Operations | Outsourcing | People | Philosophies | Planning | PMI | PMI and DA | Portfolio Management | Practices | Principle | Process | process improvement | Product Management | Product Owners | Program Management | Project Management | Promise | quality | Release Management | Requirements | requirements | Reuse Engineering | Risk management | RUP | Scaling | scaling | scaling agile | Scrum | serial | Support | Surveys | Teams | Technical Debt | Terminology | Testing | testing | Toolkit | Transformation | velocity | Workflow | show all posts

About this Blog


View Posts By:

Scott Ambler
Glen Little
Mark Lines
Valentin Mocanu
Daniel Gagnon
Michael Richardson
Joshua Barnes
Kashmir Birk

Recent Posts

Would you like to get involved with the 20th Anniversary of Agile?

The Four Layers of the Disciplined Agile Tool Kit

The Disciplined Agile Foundation Layer

The Team Lead Role: Different Types of Teams Need Different Types of Leaders

Disciplined Agile is a Hybrid

How Do You Coach an Agile DW/BI Team?

Agile2019 Agile DW/BI Coaching Workshop
The Agile DW/BI Coaching Workshop at #Agile2019

At the Agile 2019 conference in DC I facilitated a workshop with about 70 people where we explored the topic of how do you coach an agile data warehousing (DW)/business intelligence (BI) team. To do this we worked through four issues:

  1. What challenges do you face?
  2. What skills/knowledge does an agile DW/BI coach require?
  3. What strategies should you apply to engage with a DW/BI team?
  4. What are the qualities you should look for in an agile DW/BI coach?

The basic strategy was to introduce the issues to the class one at a time, then at their tables they would discuss the issue and write up to five ideas on sticky notes, then we’d share the ideas. Pictures of the flipcharts for each issue follow below. After the groups shared their ideas I then shared my thoughts with the class.

Issue #1: What Challenges Do You Face Coaching DW/BI Teams?

As you can see the class identified a lot of the classic issues that agile coaches face in general, such as trust issues, the teams being management-driven instead of self organizing, lack of agile skills within the team, cross-team dependencies, and being overwhelmed with work. Certainly there were DW/BI flavours of common problems, such as how to do vertical slices of DW/BI functionality and which lifecycle (agile, lean, CD, …) to follow. But there were also DW/BI specific issues, such as lack of access to data sources, knowing the actual data, and DW/BI architecture and design strategies. These DW/BI specific issues are where agile coaches tend to get hung up.

Agile DW/BI Coaching Challenges
Challenges commonly faced when coaching agile DW/BI teams (click to enlarge).

In my discussion of the challenges that we face when coaching agile DW/BI teams I shared my thoughts on the cultural impedance mismatch that exists between the agile and data communities. This mismatch makes it a bit more difficult to engage with data teams as opposed to application development teams. I also shared results of studies (2009, 2013,2016, 2018) around data quality challenges and practices – it is certainly common for teams to have to deal with technical debt, but data technical debt is both different in nature than code quality debt and the traditional data culture has led them down a very questionable (read dysfunctional) path regarding quality practices.

Issue #2: What Skills/Knowledge Does an Agile DW/BI Coach Require?

The second issue that we explored was what skills/knowledge does an agile DW/BI coach need. Once again the groups identified both classic agile coaching ideas as well as DW/BI specific ideas. Clearly you need coaching skills in order to coach a DW/BI team. But you also need to be knowledgeable about critical skills such as data modeling, data analysis, database testing, database refactoring, and others. You might not be an expert at these things, but you need to know of them and be able to guide the team in their adoption. You’ll also need to be able to speak intelligently about why some of the traditional strategies that they likely hold near and dear to their hearts (remember the cultural impedance mismatch) need to be abandoned for better, more effective strategies.

Agile DW/BI coaching skills
Skills/knowledge required of an agile DW/BI coach (click to enlarge).

In my discussion I overviewed the “agile database techniques stack,” a collection of agile strategies and practices for database development. The stack is:

As you can see, this list of techniques is fairly common from an agile point of view, albeit the corresponding data(base) versions of those techniques. The point is that the techniques exist that enable data professionals to work in an agile, and far more effective, manner. As a coach you will need to be aware of these strategies and be able to help your DW/BI team adopt them. And of course there are agile data management strategies that you need to be aware of as well.

Issue #3: What Strategies Should You Use To Engage Successfully With An Agile DW/BI Team?

The groups identified a collection of great strategies for engaging with DW/BI teams. Once again there were a lot of standard coaching strategies, a DW/BI team is still a group of people after all, but there was also a focus on strategies to address the DW/BI challenges identified earlier.

Agile DW/BI Coaching Strategies
Strategies to engage successfully with a DW/BI team (click to enlarge).

The discussion that followed the sharing of the stickies a very interesting point was brought up. I had earlier stated that my experience with coaching DW/BI teams was that it was different than coaching other types of teams, mostly because of the cultural impedance mismatch. A handful of agile DW/BI coaches in the audience disagreed with that, pointing out that the critical issue was gaining the trust and respect of the team at the start. This is true of any team, and certainly true of DW/BI teams. To do this you need to understand and appreciate the issues that they deal with and be able to show that you know how to guide them through addressing their issues. You might not be an expert in the techniques of the agile database technique stack, or other important agile data techniques, but you do know of them and can help the team learn them. So yes, engaging with an agile DW/BI team is no different on the surface, but it does require the coach to have different skills and knowledge than what your typical agile coach has.

Issues #4: What Are The Qualities You Should Look For In An Agile DW/BI Coach?

For this exercise I pretty much asked the groups to put together a list of qualities for a job ad for an Agile DW/BI coach. This is what they came up with.

Agile DW/BI Coach Qualities
Potential qualities of an agile DW/BI coach (click to enlarge).

Our Learnings

Here are our main learnings regarding coaching an agile DW/BI team:

  • You need fundamental agile coaching skills
  • You need to at least be aware of, but better yet experienced with, agile data and agile database development strategies
  • To effectively engage with a DW/BI team, you need to gain their trust and respect – to do that you need to show that you understand the challenges that they face and can help them to address them
  • To do so can be more challenging than with an application team due to the cultural impedance mismatch, the nature of data-oriented technical debt, and the lack of modern development skills within the data community – all of these challenges point to a greater than normal need for coaching on such teams
Posted by Scott Ambler on: August 24, 2019 07:35 AM | Permalink | Comments (0)

Database DevOps at Agile 2018

On Tuesday, August 7 I facilitated a workshop about Database DevOps at the Agile 2018 conference in San Diego.  I promised the group that I would write up the results here in the blog. This was an easy promise to make because I knew that we’d get some good information out of the participants and sure enough we did.  The workshop was organized into three major sections:

  1. Overview of Disciplined DevOps
  2. Challenges around Database DevOps
  3. Techniques Supporting Database DevOps

Overview of Disciplined DevOps

We started with a brief overview of Disciplined DevOps to set the foundation for the discussion. The workflow for Disciplined DevOps is shown below.  The main message was that we need to look at the overall DevOps picture to be successful in modern enterprises, that it was more that Dev+Ops.  Having said that, our focus was on Database DevOps.The workflow of Disciplined DevOps

Challenges around Database DevOps

We then ran a From/To exercise where we asked people to identify what aspects of their current situation that they’d like to move away from and what they’d like to move their organization towards.  The following two pictures (I’d like to thank Klaus Boedker for taking all of the following pics) show what we’d like to move from/to respectively (click on them for a larger version).

Database DevOps Moving From


Database DevOps Moving To

I then shared my observations about the challenges with Database DevOps, in particular the cultural impedance mismatch between developers and data professionals, the quality challenges we face regarding data, the lack of testing culture and knowledge within the data community, and the mistaken belief that it’s difficult to evolve production data source.


Techniques Supporting Database DevOps

The heart of the workshop was to explore technical techniques that support database DevOps.  I gave an overview of several Agile Data techniques so as give people an understanding of how Database DevOps works, then we ran an exercise.  In the exercise each table worked through one of six techniques (there are several supporting techniques that the groups didn’t work through), exploring:

  • The advantages/strengths of the technique
  • The disadvantages
  • How someone could learn about that technique
  • What tooling support (if any) is needed to support the technique.

Each team was limited to their top three answers to each of those questions, and each technique was covered by several teams.  Each of the following sections has a paragraph describing the technique, a picture of the Strategy Canvas the participants created, and my thoughts on what the group produced. It’s important to note that the some of the answers in the canvases contradict each other because each canvas is the amalgam of work performed by a few teams, and each of the teams may have included people completely new to the practice/strategy they were working through.


Vertical Slicing

Just like you can vertically slice the functional aspects of what you’re building, and release those slices if appropriate, you can do the same for the data aspects of your solution.  Many traditional data professionals don’t know how to do this, in most part because traditional data techniques are based on waterfall-style development where they’ve been told to think everything through up front in detail.  The article Implementing a Data Warehouse via Vertical Slicing goes into this topic in detail.

Database DevOps - Vertical Slicing

The advantages of vertical slicing is that it enables you to get something built and into the hands of stakeholders quickly, thereby reducing the feedback cycle.  The challenge with it is that you can lose sight of the bigger picture (therefore you want to do some high-level modeling during Inception to get a handle on the bigger picture). To be successful at vertically slicing your work, you need to be able to incrementally model, or better yet agilely model, and implement that functionality.


Agile Data Modeling

There’s nothing special about data modelling, you can perform it in an agile manner just like you can model other things in an agile manner.  Once again, this is a critical skill to learn and can be challenging for traditional data professionals due to their culture around heavy “big design up front (BDUF)”.  The article Agile Data Modelling goes into details, and more importantly an example, for how to do this.

Database DevOps - Agile Data Modeling
The advantages of this technique is that you can focus on what you need to produce now and adapt to changing requirements.  The disadvantages are that existing data professionals are resistant to evolutionary strategies such as this, often because they prefer a heavy up-front approach.  To viably model in an agile manner, including data, you need to be able to easily evolve/refactor the thing that you’re modelling.


Database Refactoring

A database refactoring is a simple change to your database that improves the quality of its design without changing the semantics of the design (in a practical manner).  This is a key technique because it enables you to safely evolve your database schema, just like you can safely evolve your application code.  Many traditional data professionals believe that it is very difficult and risky to refactor a database, hence their penchant for heavy up-front modeling, but this isn’t actually true in practice.  To understand this, see the article The Process of Database Refactoring which summarizes material from the award-winning book Refactoring Databases.

Database DevOps - DB Refactoring
Database refactoring is what enables you to break the paradigm of “we can’t change the database” with traditional data professionals.  This technique is what enables data professionals to rethink, and often abandon, most of their heavy up-front strategies from the 1970s.  DB refactoring does require skill and tooling support however.  Just like you need automated tests to safely refactor your code, to safely refactor your database you need to have an automated regression test suite.


Automated Database Regression Testing

If data is a corporate asset then it should be treated as such.  Having an automated regression test suite for a data source helps to ensure that the functionality and the data within a database conforms to the shared business rules and semantics for it.  For more information, see the article Database Testing.

Database DevOps - DB Testing
An automated test suite enables your teams to safely evolve their work because if they break something the automated tests are likely to find the problem.  This is particularly important given that many data sources are resources shared across many applications.  Like automated testing for other things, it requires skill and tooling to implement. To effectively regression test your database in an automated manner you need to include those tests in your continuous integration (CI) approach.


Continuous Database Integration

Database changes, just like application code changes, should be brought into your continuous integration (CI) strategy. It is a bit harder to include a data source because of the data.  The issue is side effects from tests – in theory a database test should put the db into a known state, do something, check to see if you get the expected results, then put the DB back into the original state.  It’s that last part that’s the problem because all it takes is one test to forget to do so and there’s the potential for side effects across tests. So, a common thing is to rebuild (or restore, or a combination thereof) your dev and test data bases every so often so as to decrease the chance of this.  You might choose to do this in your nightly CI run for example. For more information, see the book Recipes for Continuous Database Integration.

Database DevOps - Continuous DB Integration 

Operational Data Monitoring

An important part of Operations is to monitor the running infrastructure, including databases.  This information can and should be available via real-time dashboards as well as through ad-hoc reporting.  Sadly, I need to write an article on this still.  But if you poke around the web you’ll find a fair bit of information.  Article to come soon.

Database DevOps - Operational Monitoring


Concluding Thoughts

This was a really interesting workshop.  We did it in 75 minutes but it really should have been done in a half day to allow for more detailed discussions about each of the techniques.  Having said that, I had several very good conversations with people following the workshop about how valuable and enlightening they found it.

This workshop, plus other training and service offerings around agile database and agile data warehousing skills, is something that we can provide to your organization.  Feel free to reach out to us.

Posted by Scott Ambler on: August 09, 2018 08:54 AM | Permalink | Comments (0)

Bridging the Gap Between Agile and Data

At the Agile 2017 conference this week Lynn Winterboer was kind enough to invite me to her workshop which explored how to apply agile strategies in the data space.  She did a great job of facilitating the group of about 40 people through identifying the challenges currently faced by teams. The main issues that the group explored were:

  • Product ownership and data
  • Agile data architecture
  • Data testing and tooling support
  • How to include data people, and activities, in development

Lynn will soon be blogging about the results so I’m not going to dive into that here.  I suspect that her blog post will be very interesting.

What I’d like to do here is share a few thoughts about what I observed:

  1. The discussion was very healthy. This was a group of very smart people coming from different backgrounds.  Everyone was interested in sharing their experiences and working together to solve some tough problems.
  2. Context counts. “Agile and data” is a big topic.  A few people were dealing with the issue of how to address data issues on application development projects, some were focused on agile data warehousing/business intelligence, and some were focused on complex data analytics.  In our conversations it was very clear that strategies which worked for app development wouldn’t work for analytics, and vice versa.  This is why Context Counts is one of Disciplined Agile’s fundamental principles.
  3. The challenges are tough. Everyone was working in organizations that were struggling with these challenges.  For each of issue we could have spent much longer exploring the potential solution(s).
  4. Every challenge we discussed are solved issues.  I’ve always found it frustrating when people, who are very smart and who have been struggling with a problem for awhile, have clearly never thought to simply google “database testing tools” or “agile architecture” to find out what advice is already out there.  When we discussed testing I even asked why people hadn’t done such as search, and then pointed out that there are a lot of tools available and even a few people maintaining lists of such tools (such as 40+ database testing tools).  All of these challenges, and more, have solutions described by techniques of the Agile Data and Agile Modeling methods from about 15 years ago.  These techniques have of course been adopted, and put into context, by the Disciplined Agile (DA) toolkit and in particular Disciplined Agile Delivery (DAD).
  5. The “factions” behaved differently.  Long ago I pointed out that there’s a cultural impedance mismatch between the data and developer communities, and it was pretty easy to observe during this workshop.  For example, during the workshop we were asked to identify solutions to the challenges.  Lynn wanted us to write this information on flip-chart paper so that she could later scan it and share it with others.  For awhile the groups dominated by data people discussed the solutions without writing anything down.  Their conversations were good, but they quickly got stuck in analysis paralysis mode because they seemed to be afraid to write down information for fear that they couldn’t easily update it (the challenge with having paper to write on instead of whiteboards). The groups dominated by agile people, the ones focused on development and architecture, started by writing on sticky notes and putting them on the flip chart paper.  This fundamental agile modelling strategy enabled them to visualize and share their work transparently, enhancing their conversation and enabling them to move forward easily.
  6. Database (tool) vendors need to up their game. Speaking out tools, database vendors and data warehouse tool vendors really need to up their game.  Here’s a few very harsh questions: Does your database tool vendor or ETL tool vendor offer testing tools that enable both black box and white box database testing?  Answer: very likely no, because their customers don’t demand that of them.  Did your data team even think to ask for such tools?  Answer: very likely not.  How many database testing books do you think have been written? Answer: very few, go do a search and see for yourself.  Why does the data community have such a huge blind spot when it comes to testing? Answer: this is a huge side effect of the cultural impedance mismatch.

I’m very happy to see that Lynn is actively trying to bridge the agile and data communities, helping us to learn from each other.  This is something she’s been doing for years and doing it quite well.  My experience is that both communities would benefit greatly from more collaboration with each other.

Posted by Scott Ambler on: August 11, 2017 08:37 AM | Permalink | Comments (0)

How Does Data Management Fit In?

Categories: Data Management, Workflow

Key tenets of agile and lean are to work collaboratively and to streamline your workflow respectively. This includes all aspects of your workflow, not just the fun software development stuff that we all like to focus on.  This blog posting explores how Data Management activities fit into your overall process.

In the process flow diagram below we see that Data Management is a collaborative effort that has interdependencies with other DA process blades and the solution delivery teams that Data Management is meant to support. Due to the shortened feedback cycles and collaborative nature of the work this can be very different than the current traditional strategies. For example, with a DA approach, the Data Management team works collaboratively with the delivery teams, Operations, and Release Management to evolve data sources. The delivery teams do the majority of the work to develop and evolve the data sources, with support and guidance coming from Data Management. The delivery teams follows guidance from Release Management to add the database changes into their automated deployment scripts, getting help from Operations if needed to resolve any operational challenges. Evolution of data sources is a key aspect of Disciplined DevOps.

Data Management external workflow

This highly collaborative strategy is very different than the typical traditional strategy that requires delivery teams to first document potential database updates, have the updates reviewed by Data Management, then do the work to implement the updates, then have this work reviewed and accepted, then work through your organizations Release Management process to deploy into production.

In the next blog posting in this series we will explore the internal workflow of a Disciplined Agile approach to Data Management.  Stay tuned!

Posted by Scott Ambler on: May 20, 2017 07:07 AM | Permalink | Comments (0)

Disciplined Agile Values for Data Management

Data Management Mindset

There are several values that are key to your success when transforming to a leaner, more agile approach to Data Management. Taking a cue from the Disciplined Agile Manifesto, we’ve captured these values in the form of X over Y. While both X and Y are important, X proves to be far more important than Y in practice. These values are:

  1. Evolution over definition. The ability to safely and quickly evolve an existing data source, either to extend it to support new functionality or to fix quality issues with it, is absolutely paramount in today’s hyper-competitive environment. Yes, having defined data models and metadata describing them is also important, but nowhere near as important as being able to react to new business opportunities. Luckily agile database techniques, long proven in practice, exist that enable the safe evolution of production data stores.
  2. Holistic organization over Data Management. Earlier we said that data is the lifeblood of your organization. Yes, blood is important but so is your skeleton, your muscles, your organs, and many other body parts. We need to optimize the whole organizational body, not just the “data blood.” Traditional Data Management approaches often run aground because they locally optimize for data concerns, whereas a DA approach to Data Management recognizes that we must optimize the overall whole. This implies that sometimes we may need to sub-optimize our strategy from a data point of view, for the sake of organizational level optimization.
  3. Sufficiency over perfection. Data sources, like many other IT assets, need to be good enough for the task at hand. The old saw “perfect is the enemy of good” clearly applies in the data realm – too much time has been lost, and opportunities squandered, while development teams were forced to wait on Data Management teams to create (near) perfect models before being allowed to move forward. Traditional data professionals mistakenly assume that production databases are difficult to evolve and as a result strive to get their designs right the first time so as to avoid very painful database changes in the future. The Agile Data method has of course shown this assumption to be wrong, that it is very straightforward and desirable to evolve production databases. A side effect of this revelation is that we no longer need to strive for perfect, detailed models up front. Instead we can do just enough up-front thinking to get going in the right direction and then evolve our implementation (including data sources) over time as our understanding of our stakeholder needs evolve.
  4. Collaboration over documentation. We’ve known for decades that the most effective way to communicate information is face-to-face around a shared sketching environment, and that the least effective is to provide detailed documentation to people. The implication is that we need to refocus our efforts to be more collaborative in nature. As data professionals we need to get actively involved with solution delivery teams: to share our knowledge and skills with those teams, and to enable them to become more effective in working with data. Yes, we will still need to develop and sustain data-related artifacts, but those artifacts should be lightweight and better yet executable in nature.
  5. Cross-functional people over specialized staff. Agilists have come to realize that people are more effective when they are cross-functional (also known as T-skilled or generalizing specialists). Although specialists are very skilled in a narrow aspect of the overall process, the problem is that you need a lot of specialists to perform anything of value and as a result the overall workflow tends to be error prone, slow, and expensive. The other extreme would be to be a generalist, someone who knows a little bit about all aspects of the overall process. But, the challenge with these people is that although they’re good at collaborating with others they don’t actually have the skills to produce concrete value. We need the best of both worlds – a generalizing specialist with one or more specialties so that they can add value AND a general knowledge so that they can collaborate effectively with others and streamline the overall effort.
  6. Automation over manual processes. The only way that we can respond quickly to marketplace opportunities is to automate as much of the bureaucracy as we possibly can. Instead of creating detailed models and documents and then reviewing potential changes against them we capture our detailed specifications in the form of executable tests. This is quickly becoming the norm for specifying both the requirements and designs of code, and the same test-driven techniques are now being applied to data sources. Continuous integration (CI) and continuous deployment (CD) are also being applied to data sources, contributing to improving overall data quality and decreasing the time to safely deploy database updates into production.

As you can see, we’re not talking about your grandfather’s approach to Data Management. Organizations are now shifting from the slow and documentation-heavy bureaucratic strategies of traditional Data Management towards the collaborative, streamlined, and quality-driven agile/lean strategies that focus on enabling others rather than controlling them.

Recommended Reading

Posted by Scott Ambler on: April 27, 2017 01:09 PM | Permalink | Comments (0)

"When you want to test the depths of a stream, don't use both feet."

- Chinese Proverb