Disciplined Agile

On August 8th I facilitated a workshop at Agile 2018 that focused on agile architecture. I promised everyone, we had over 150 people in the workshop, that I would post the pictures of their strategy canvases online so that they could have copies of it.  So here’s my promised blog!

How architecture fits into agile ways of working (WoW) has been an important topic from the very beginning.  As with all other aspects of agile, the Disciplined Agile (DA) toolkit provides you with choices – because every team is unique and faces a unique situation, they need to choose their WoW to reflect this (we like to say that context counts).  To enable teams to effectively choose their WoW they need to know what choices they have available to them (we also believe that choice is good) and what the trade-offs of those choices are (because there’s no such thing as a best practice).

So in this workshop we worked through twelve agile architecture strategies.  Granted, there’s a lot more than this, but we only had 75 minutes so I lead everyone through some of the key strategies.  This blog overviews each strategy and then shares the strategy canvases that the class developed.  Our approach was to assign each strategy to several tables and then ask each group to discuss the advantages of the strategy, the disadvantages, and when you would use it. There are pictures below of each strategy canvas – it’s important to note that I don’t agree with all of the sticky notes, but for the most part it’s solid stuff.  Our upcoming book, trending to arrive in October 2018, explores these strategies (and more).

During the workshop we explored three fundamental questions:

1. How Do You Explore Your Initial Architecture Strategy?
2. How Do You Bring Architecture Skills Into the Team?
3. How Do You Evolve Your Architecture?

How Do You Explore Your Initial Architecture Strategy?

Each group was given one of three initial architecture strategies to discuss:

1. No modeling
2. Lightweight modeling
3. Heavy modeling

No modeling

At the beginning of our project/endeavor we will perform no architecture modeling or exploration at all.  We believe that all aspects of the architecture will emerge over time during Construction.

Lightweight modeling

At the beginning of our project/endeavor we will invest a bit of time, potentially several hours or even a few days, to perform some lightweight agile architecture modeling.  We believe that we should think through the high-level architectural issues now but that the design details will emerge over time. Our “models” will be created using inclusive modeling tools such as whiteboards, paper or sticky notes. We’re likely to create one or more high-level sketches, each of which explores an architectural view (such as the technical architecture, the data architecture, the communications architecture, and so on).

Heavy modeling

At the beginning of our project/endeavor we will invest a fair bit of time, potentially several weeks or even a few months, to work through the architecture of our solution.  We believe that we need to think through most, if not all, aspects of the architecture in detail before we can safely proceed with Construction.  Our models may be either visual (i.e. UML component diagrams, Free-form architecture diagrams, Data Models, …) or textual (e.g. white papers) in nature.  Our visual models will be supported by sufficient documentation to capture the details overviewed by the diagrams.  Our models will be captured with software-based diagramming tools (e.g. Visio, OmniGraffle), documentation tools (e.g. Word processors or Wikis), or modeling tools (e.g. Enterprise Architect).

How Do You Bring Architecture Skills Into the Team?

Each group was given one of five architecture skills/roles strategies to explore:

1. No architect
2. Part-time architect
3. Modeler architect
4. Developer architect
5. Architecture owner

No architect

On our project/endeavor we do not have anyone on the team responsible for architecture work nor do we have anyone with sufficient architecture skills to do so.  Luckily we are a group of smart, experienced developers and we believe that we will be able to evolve the architecture of our solution through teamwork and experimentation.

Part-time architect

On our project/endeavor we have someone on our team acting as a part-time architect.  This person is very experienced and is a member of our corporate architecture team, supporting several solution delivery teams including our own.  The part-time architect guides our team in architecture decisions, providing advice, educating us in any existing corporate architecture guidance, and reviewing any work that we do on our own.

Modeler architect

On our project/endeavor we have someone with deep architecture experience and skills.  In the past they were a developer but over time they worked their way out of that role to become an architect.  Their job is to work through the architectural strategy for the team, describe that architecture strategy to us, and develop and maintain sufficient architecture models and documentation throughout the lifecycle.  Although they are technically adept and well versed in the domain, they do not have technical experience with some of the platforms and technologies that the team is likely to work with. 

Developer architect

On our project/endeavor we have someone who is a full-time member of our team who has deep architecture experience as well as development skills. They are responsible for formulating the architecture strategy for the team although will often work with other team members to get their input into architecture decisions. When they are not doing architecture work they are an active developer on the team.

Architecture owner

On our project/endeavor we have someone in the role of Architecture Owner (AO).  This is a full-time team member who has architecture experience, is well versed in our corporate architecture guidance/strategies, and also has development skills.  The AO:

• Guides/facilitates the team through architectural strategy and related work
• Mentors/coaches team members in architecture thinking and skills
• Work with our corporate architecture team as needed to ensure that our team is following, to the extent appropriate, the overall architectural vision
• Is an active developer on the team when they are not performing their other responsibilities

How Do You Evolve Your Architecture?

Each group was given one of four architecture evolution strategies to explore:

1. Architectural Guidance
2. Architecture Spikes
3. Walking Skeletons
4. Just-in-time (JIT) Model Storming

Architectural Guidance

One of the approaches that our team uses to evolve our architecture is refer to our organization’s existing architectural guidance. We typically do this by talking with someone on our team familiar with that guidance, such as an Architecture Owner (AO).  This person may decide to look up the current version of the architecture guidance as needed (hopefully this is captured in an agile, lightweight manner).  Such architecture guidance may include:

• Technologies we’re moving towards
• Technologies we’re moving away from
• Technologies that we mustn’t use
• Previously identified architecture experiments to run (important in multi-team environments to coordinate learning efforts)
• Results from architecture spikes performed by other teams

Architecture Spikes

One of the approaches that our team uses to evolve our architecture is to run architecture spikes (or simply called spikes). Whenever we run into an architectural issue, such as how a technology works within our environment or how a combination of technologies work together, we write a bit of code to explore the issue.  This is typically throwaway prototyping code because we want to quickly experiment to gather sufficient data to make a better-informed decision.  Spikes typically take a few hours to a few days. A spike can be thought of as a very small proof of concept (PoC).

Walking Skeletons

One of the approaches that our team uses to evolve our architecture is to develop a walking skeleton, also known as a working skeleton or steel frame, of our solution early in Construction.  We do this by focusing on a small collection of requirements early in the lifecycle that implement sufficient functionality to address our highest-risk items so as to show that our architectural strategy works in practice.  Once the walking skeleton is in place we spend the rest of Construction putting the functional flesh onto it.

Just-in-time (JIT) Model Storming

One of the approaches that our team uses to evolve our architecture is to explore requirement and design details at the last most responsible moment.  We do this via light-weight agile modeling (sketching, sticky notes, …).  Model storming sessions are often impromptu and last for several minutes, often starting with a request along the lines of “Hey, can you help me work through this…”.

Parting Thoughts

After the workshop I received a lot of great feedback from attendees.  It really resonated with people and seemed to provide them with a lot of value.  I was glad to hear that.  

Further Reading

• Architecture Owner
• Identify Initial Architecture Strategy
• Enterprise Architecture
• Agile Architecture Strategies

On Tuesday, August 7 I facilitated a workshop about Database DevOps at the Agile 2018 conference in San Diego.  I promised the group that I would write up the results here in the blog. This was an easy promise to make because I knew that we’d get some good information out of the participants and sure enough we did.  The workshop was organized into three major sections:

1. Overview of Disciplined DevOps
2. Challenges around Database DevOps
3. Techniques Supporting Database DevOps

Overview of Disciplined DevOps

We started with a brief overview of Disciplined DevOps to set the foundation for the discussion. The workflow for Disciplined DevOps is shown below.  The main message was that we need to look at the overall DevOps picture to be successful in modern enterprises, that it was more that Dev+Ops.  Having said that, our focus was on Database DevOps.

Challenges around Database DevOps

We then ran a From/To exercise where we asked people to identify what aspects of their current situation that they’d like to move away from and what they’d like to move their organization towards.  The following two pictures (I’d like to thank Klaus Boedker for taking all of the following pics) show what we’d like to move from/to respectively (click on them for a larger version).

I then shared my observations about the challenges with Database DevOps, in particular the cultural impedance mismatch between developers and data professionals, the quality challenges we face regarding data, the lack of testing culture and knowledge within the data community, and the mistaken belief that it’s difficult to evolve production data source.

Techniques Supporting Database DevOps

The heart of the workshop was to explore technical techniques that support database DevOps.  I gave an overview of several Agile Data techniques so as give people an understanding of how Database DevOps works, then we ran an exercise.  In the exercise each table worked through one of six techniques (there are several supporting techniques that the groups didn’t work through), exploring:

• The advantages/strengths of the technique
• The disadvantages
• How someone could learn about that technique
• What tooling support (if any) is needed to support the technique.

Each team was limited to their top three answers to each of those questions, and each technique was covered by several teams.  Each of the following sections has a paragraph describing the technique, a picture of the Strategy Canvas the participants created, and my thoughts on what the group produced. It’s important to note that the some of the answers in the canvases contradict each other because each canvas is the amalgam of work performed by a few teams, and each of the teams may have included people completely new to the practice/strategy they were working through.

Vertical Slicing

Just like you can vertically slice the functional aspects of what you’re building, and release those slices if appropriate, you can do the same for the data aspects of your solution.  Many traditional data professionals don’t know how to do this, in most part because traditional data techniques are based on waterfall-style development where they’ve been told to think everything through up front in detail.  The article Implementing a Data Warehouse via Vertical Slicing goes into this topic in detail.

The advantages of vertical slicing is that it enables you to get something built and into the hands of stakeholders quickly, thereby reducing the feedback cycle.  The challenge with it is that you can lose sight of the bigger picture (therefore you want to do some high-level modeling during Inception to get a handle on the bigger picture). To be successful at vertically slicing your work, you need to be able to incrementally model, or better yet agilely model, and implement that functionality.

Agile Data Modeling

There’s nothing special about data modelling, you can perform it in an agile manner just like you can model other things in an agile manner.  Once again, this is a critical skill to learn and can be challenging for traditional data professionals due to their culture around heavy “big design up front (BDUF)”.  The article Agile Data Modelling goes into details, and more importantly an example, for how to do this.

The advantages of this technique is that you can focus on what you need to produce now and adapt to changing requirements.  The disadvantages are that existing data professionals are resistant to evolutionary strategies such as this, often because they prefer a heavy up-front approach.  To viably model in an agile manner, including data, you need to be able to easily evolve/refactor the thing that you’re modelling.

Database Refactoring

A database refactoring is a simple change to your database that improves the quality of its design without changing the semantics of the design (in a practical manner).  This is a key technique because it enables you to safely evolve your database schema, just like you can safely evolve your application code.  Many traditional data professionals believe that it is very difficult and risky to refactor a database, hence their penchant for heavy up-front modeling, but this isn’t actually true in practice.  To understand this, see the article The Process of Database Refactoring which summarizes material from the award-winning book Refactoring Databases.

Database refactoring is what enables you to break the paradigm of “we can’t change the database” with traditional data professionals.  This technique is what enables data professionals to rethink, and often abandon, most of their heavy up-front strategies from the 1970s.  DB refactoring does require skill and tooling support however.  Just like you need automated tests to safely refactor your code, to safely refactor your database you need to have an automated regression test suite.

Automated Database Regression Testing

If data is a corporate asset then it should be treated as such.  Having an automated regression test suite for a data source helps to ensure that the functionality and the data within a database conforms to the shared business rules and semantics for it.  For more information, see the article Database Testing.

An automated test suite enables your teams to safely evolve their work because if they break something the automated tests are likely to find the problem.  This is particularly important given that many data sources are resources shared across many applications.  Like automated testing for other things, it requires skill and tooling to implement. To effectively regression test your database in an automated manner you need to include those tests in your continuous integration (CI) approach.

Continuous Database Integration

Database changes, just like application code changes, should be brought into your continuous integration (CI) strategy. It is a bit harder to include a data source because of the data.  The issue is side effects from tests – in theory a database test should put the db into a known state, do something, check to see if you get the expected results, then put the DB back into the original state.  It’s that last part that’s the problem because all it takes is one test to forget to do so and there’s the potential for side effects across tests. So, a common thing is to rebuild (or restore, or a combination thereof) your dev and test data bases every so often so as to decrease the chance of this.  You might choose to do this in your nightly CI run for example. For more information, see the book Recipes for Continuous Database Integration.

Operational Data Monitoring

An important part of Operations is to monitor the running infrastructure, including databases.  This information can and should be available via real-time dashboards as well as through ad-hoc reporting.  Sadly, I need to write an article on this still.  But if you poke around the web you’ll find a fair bit of information.  Article to come soon.

Concluding Thoughts

This was a really interesting workshop.  We did it in 75 minutes but it really should have been done in a half day to allow for more detailed discussions about each of the techniques.  Having said that, I had several very good conversations with people following the workshop about how valuable and enlightening they found it.

This workshop, plus other training and service offerings around agile database and agile data warehousing skills, is something that we can provide to your organization.  Feel free to reach out to us.

Today in Canada we tested our nationwide emergency response system.  Apparently the test failed in the province of Quebec.  It did in fact succeed in Ontario, where I live.  Knowing about the test I purposefully had my phone on this afternoon because I was interested in what would actually happen.  Sure enough, my phone made a very annoying noise and a message came up to inform me that it was just a test.  So that was good.

An important aspect of IT Operations, and Business Operations for that matter, is to be prepared to respond to emergencies.  While the Canadian government is worried about responding to inclement weather, terrorist attacks, military attacks, and coffee being sold out at the local Timmies your IT department should be concerned about ensuring that your systems are running properly, that they are repelling cyber attacks, and that your data centres are operational (to name a few potential issues). This is why the IT Operations process goal includes a process decision point called Mitigate Disasters (see the pic above).

By running this scheduled disaster simulation, after careful planning and communication (which I why I had heard about it), the Canadian government has discovered in a controlled test that their strategy needs work.  This is exactly the type of thing you want to find out when you have the luxury of safely addressing any problems that you do find.  The government certainly wouldn’t have wanted to discover their emergency alert system didn’t work as expect in the middle of an actual emergency.

What your organization should ask itself is what would happen if:

• One of your data centres lost power?  Or connectivity?
• Some of your servers went down?
• Outsourced services you rely on (think SAAS, PAAS, and other cloud solutions) went down?
• An application/system went down?
• A denial of service (DoS) attack succeeded?
• And many more issues.

Will your IT ecosystem respond properly?  Will it recover automatically?  Are you guessing at these answers or do you know for sure because you’ve actually simulated them?

I hope this blog has been food for thought.  Time for a Timmies.

In this blog posting we overview the decision point, Choose an SCM Branching Strategy, of the Accelerate Value Delivery process goal.

Figure 1. The goal diagram for Accelerate Value Delivery (click to enlarge).

As you know, the Disciplined Agile (DA) tool kit helps you to make better decisions by describing the trade-offs associated with a practice/strategy.  The following is an excerpt from the Choose Your WoW! book:

Accelerate Value Delivery==>Choose an SCM Branching Strategy

We need to identify our team’s branching strategy for our source code repository.  A branch is a copy or clone of all, or at least a portion of, the source code within the repository. We branch our code to support concurrent development, capture of solution configurations, multiple versions of a solution, and multiple production releases of a solution. so that it may be worked on in parallel.  As you can see in the following table there are many branching strategies available to us, strategies which may be applied in combination.

By now many of you will have heard about the tragic loss of Mike Beedle, one of the signatories of the Agile Manifesto and the thought leader behind Enterprise Scrum.  Mike passed away on March 24th and leaves behind a family that includes six children.

For those of us who were lucky enough to know Mike we’ll remember him as a generous and intelligent gentleman.  He made significant and ongoing contributions to the agile community and we’ll all miss him.

The Disciplined Agile Consortium is proud to have donated to the Memorial Fund for Mike Beedle’s Family and we hope that you consider doing the same.

Rest in peace Mike.