Disciplined Agile

For teams that are applying agile strategies to Data Warehouse (DW)/Business Intelligence (BI) development is it fairly common for them to take a Disciplined Agile (DA) Approach to DW/BI due to DA’s robustness.  A common question that comes up is how do you write user stories for DW/BI solutions?  Here are our experiences.

First, user stories should focus on business value.  In general, your stories should answer the question “What value is being provided to the user of this solution?”  In the case of a DW/BI solution, they should identify what question the DW/BI solution could help someone to answer, or what business decision the solution could support.  So “As a Bank Manager I would like the Customer Summary Report” or “Get Customer data from CustDB17 and TradeDB” would both be poorly written user stories because they’re not focused on business value.  However, “As a Bank Manager I would like to know what services a given customer currently have with BigBankCo so that I can identify what to upsell them” is.  The solution to implement that story may require you to create the Customer Summary Report (or there may be better ways at getting at that information) and it may require you to get data from CustDB17 and TradeDB (and other sources perhaps).

Here are some examples of user stories for a University DW/BI solution:

• As a Professor I would like to analyze the current grades of my students so that I can adjust the difficulty of future tests and assignments
• As a Student I would like to know the drop out rates by course and professor from previous years to determine the likely difficulty of my course choices
• As a Registrar I would like to know the rate of enrollments within a class over time to determine the popularity of them
• As a Student I would like to know the estimated travel time between back-to-back classes so that I can determine whether I can make it to class on time

Second, user stories on their own aren’t sufficient.  User stories/epics are only one view into your requirements, albeit an important one.  You’ll also want to explore the domain (e.g. do some data modelling), the user interface (e.g. explore what reports should look like), the business process (e.g. what are the overall business process(es) supported by your DW/BI solution), and technical views (e.g. how does the data flow through your solution architecture).

Third, data requirements are best addressed by domain modelling.  As you are exploring the requirements for your DW/BI solution you will hear about data-oriented requirements.  So capture them in your domain model as those sorts of details emerge over time.  Consider reading Agile/Evolutionary Data Modeling: From Domain Modeling to Physical Data Modeling for a detailed discussion of this topic.

Fourth, technology issues are best captured in architecture models.  You will also hear about existing legacy data sources and in general you will need to capture the architecture of your DW/BI solution.  This sort of information is best captured in architecture models, not in stories.

A few more thoughts:

1. User stories are only one option for usage modelling.  There are several ways that we can explore usage, user stories/epics are just one way.  You could also create light-weight use cases, usage scenarios, or personas to name a few strategies.
2. Take a usage-driven approach.  The primary modelling artifact on a Disciplined Agile DW/BI team is the usage model (e.g. your user stories) not the data model.  Data modelling is a secondary consideration compared with usage modelling, and this can be a difficult concept for experienced DW/BI professionals to come to terms with.
3. Keep your initial modelling light-weight.  The agile rules still apply to DW/BI solutions – keep your modelling efforts sufficient for the task at hand and no more.
4. Get trained in this.  This is a complex topic.  

Over the past few days there has been a Tweetstorm around Agilia Conference 2016 that is being held April 4-8 in Olomouc, Czech Republic.  I’ve decided to call this Agiliagate as every “scandal” these days seems to named in honour of the Watergate scandal that brought down Richard Nixon in the 1970s. The scandal started with a tweet from Samantha (Sam) Laing.  For those of you who don’t know her, Sam is an agile coach based out of Cape Town South Africa.  She has co-authored several excellent books and is a regular speaker at conferences worldwide.  I’ve known and respected her for several years now and was lucky enough to take a coaching workshop with her a few years ago at the Agile conference in the U.S.  Her tweet was:

wow @agiliaconf really? 2 out of 27 speakers are woman? You couldn’t find any more?

Then Things Spun Out of Control

Sam made a perfectly valid observation.  Sadly, the initial response to Sam’s question was very, very unfortunate.  @agiliaconf responded with:

@samlaing Why we shoud worry? We do not play feminist games here nor politics. Whoever pass our extensive screenening, he may come.

Yikes.  Just yikes.  Sam then maturely responded:

@agiliaconf not a feminist game, just shocked at lack of diversity and now, lack of caring about diversity. It’s ok, just my opinion.

A very fair and level-headed response in my opinion.  Digging their hole even deeper, @agiliaconf responded with:

@samlaing Yes, as I said, we are not political party. Go to EU in Bruxelles, they love to engineer diversity games. Our focus is different.

That response clearly didn’t help.  By this point the Tweetstorm was in full swing, although @agiliaconf chose to make matters even worse by tweeting:

You have enemies? Good. It means you’ve stood up for something, sometime in your life.- Sir Winston Churchill – short rep to todays attack

@Agiliaconf’s first tweet could be chalked up as an unfortunate mistake.  Their second one was highly questionable, but the third one spectacularly stupid.  Dozens of people from all over the world, many of them important contributors in the agile community, were actively criticizing the conference organizers for the lack of gender diversity at the event.  Some people were also choosing to try to publicly shame several of the speakers, including myself, for being involved with the conference.  Some people tried to bully speakers to drop out of participating in the conference.  Others started going after the vendors who were sponsoring the conference.  Villains had been clearly identified and the forces of righteous indignation were out for blood.

Out of the Mess, Some Very Good Suggestions

The good news is that during all of this several good suggestions came out of the cacophony.  These suggestions included:

1. Invite more female speakers.  Clearly a great idea for future events, but considering this conference was less than a week away it wasn’t a viable consideration to address the immediate problem.
2. People should only speak at conferences that support diversity.  That’s also a great idea, and I’ll going start asking about this in the future, so lesson learned for me.
3. Invite a woman to co-present at the conference.  This is also a great idea.  I’ve co-presented with many people in the past and that works very well and is a great way to shepherd people to become public speakers.
4. Invite one or more women to be involved with the conference committee.  This is a great idea for future events.  Having been involved with many conference committees over the years my experience is that the more diverse your conference committee then generally the more diverse, and interesting, your speaker line-up will be.

Let’s Add Some Context

As we like to say at my company, context counts.  Had the people involved taken a few minutes to pause and consider the situation, I suspect they would have identified some important contextual factors:

1. This happened over a low-fidelity communication medium.  Communication via electronic text, and in particularly the 140-character messages of Twitter, is problematic at best (see my article Agile Communication for some detailed thoughts on this topic).  It is very difficult to have meaningful conversations via Twitter, and it takes people to be a bit more careful in the way that they word things and thoughtful in how they react.  This clearly wasn’t happening.
2. Organizing a conference is a lot of hard, stressful work.  It’s particularly stressful in the days leading up the the conference, which is the period when all of this happened.  @Agiliaconf was very likely dealing with a fair bit of stress at the time that Sam’s tweet came in.  Some of the people critical of @Agiliaconf may not have much, if any experience, with how conferences are run so I can see them being unaware of this.  However, several of the detractors are regular conference speakers and should have been a lot more empathetic.
3. There are cultural differences.  Aguarra, the organization running the conference, is based in the Czech Republic.  The Czechs have their own unique culture, just like the Canadians do, the Brits, the Germans, the South Africans, and so on.  There are some great similarities between all of these cultures, but important nuances too.  When people from different cultures are interacting with one another it often requires a bit more patience and latitude.

The point is that instead of responding with a knee-jerk reaction I decided to act in a respectful manner and contact the Agilia folks and see what they had to say for themselves.

Agilia’s Side of the Story

I had a email conversation with Michal, one of the two organizers of event.  Here’s a few things that I learned in the conversation:

1. Their native language isn’t English.  In hindsight this is clear given the wording of some of their tweets.  It’s hard enough to have a conversation on Twitter, but doing so in a language that isn’t your first one is very difficult.
2. They’re fairly new to Twitter.  Since joining Twitter they’ve sent less than 250 tweets, many of which are focused on advertising presentations at conferences.  As you can see from their tweets they perceived Sam’s initial tweet as an attack coming from a complete stranger, and with just a bit of empathy I think it’s pretty easy to see how this would be the case.  Furthermore, they honestly thought that their first response was sarcastic and would be taken as such.  I explained that sarcasm is an incredibly bad idea on Twitter at the best of times, and that they really needed to avoid it in the future.
3. Half of the conference organizers are female.  Granted, there are only two conference organizers, a man and a woman.  As you can see from the link, this information is fairly easy to discover yet none of their critics bothered to look into this (or if they did they certainly didn’t tweet about it).
4. The critics are from outside of Central and Eastern Europe (CEE).  The conference organizers did some analysis of where the critical tweets were coming from, and for the most part it was from outside of the CEE. This is a reflection of my earlier point about cultural differences between the people involved.  When this happens everyone needs to be more patient and understanding.
5. The critics didn’t bother to reach out to the Agilia organizers.  Because I didn’t know what was going on behind the scenes, I asked if any of the critics had bothered to reach out to the Agilia organizers to get their side of the story.  Here was the response: “From speakers, only Ben Linders and Olli Pietikainen. From people, who posted some articles about this situation, nobody. From most visible twitter screamers such as [NAMES WITHHELD BY ME] nobody. From other twitter people – nobody.”  Given that the agile community preaches values such as respect, communication, and collaboration many of our more prominent people have clearly failed to fulfill these ideals in the “AgiliaGate” situation.  I am embarrassed for us.
6. Their responses reflect their culture.  When I pointed out to them that their responses to Sam were inappropriate, something that many others pointed out via Twitter, here was their response: “I will appreciate, if I could get more detailed info of what has happened as seen from other side. In our cultural environment, there is general dissagreement in society about attempts to impose quotas on anything, including number of women or minorities or anything.  Asking for it might be interpreted as an assalt, which I did.”  Yes, many of us may not agree with that but this is the cultural environment in which the Agilia conference is operating.  Asked about what they were thinking with their second tweet, they responded “I responded with lite sarcasm, true, that we praise meritocracy here.  In second tweet the my message was – go to elsewhere, we do not want you here. I have reffered to Bruxelles, because this place is in CEE region seen as source of such ideas on regulation.”  OK, that message certainly didn’t come across at all but it does reflect the cultural environment that the organizers exist in.
7. There’s a surprising amount of support for the Agilia organizers.  You only need to look at @Agiliaconf’s Twitter feed to see this.  A lot of this support is coming from people in the CEE (whom the conference is targeted at) and in some cases even coming from…. you guessed it… women.  The point is that this gender diversity concern, and it is an important concern, is coming from outsiders who have a different cultural context than that of the target audience of the conference.  Once again, we’re seeing cultural differences get in the way of mutual understanding.
8. Acceptance of talk proposals from women was 100%.  One of the things that I asked the conference organizers was to provide some stats around their proposal process.  They received one talk proposal from a woman for this event and they accepted it.  Furthermore they reached out to another lady to give a talk at the conference.  Apparently she had been scheduled at a previous event but had to drop out due to personal reasons, so for this event they reached out to her again and invited her to speak at the Olomouc conference.

So, by choosing to have more respectful interactions with the conference organizers I easily discovered that they aren’t the evil, misogynistic bastards that some people want to portray them to be.  They are clearly guilty of having poor Twitter skills,  imperfect English, and a culture that is different from that of their critics.  I’m not sure that we should vilify them for that.  With a bit of investigation I discovered that they have a gender diverse conference committee, had not only accepted all of the proposals from women but had gone out of their way to invite another female speaker.  Could they have invited more women?  You bet.

We Need to Do Better

The organizer’s of the Agilia conference certainly made some serious mistakes.  But so did many of the people criticizing them.  Yes, the Agilia people reacted poorly, but that didn’t mean we needed to respond in kind by trying to publicly shame or in some cases bully anyone involved with the conference.

If I may be so bold, here are a few suggestions to consider in the future:

1. Let’s strive for respectful communication over indignant posing.
2. When we see questionable behaviour, let’s investigate first so that we can avoid jumping to conclusions.
3. Contact people privately to discuss and understand the situation before attacking them in public (particularly when you might not understand the full picture).
4. Just because others have jumped on a bandwagon, that doesn’t mean that you need to do so too.  Or, at least if you do, make sure you understand what’s actually going on first.
5. If you honestly believe that future Agilia conferences could do with a more diverse group of speakers, then take the opportunity to submit a proposal or point someone whom you think would be a great speaker to this link.
6. When someone is behaving in a way that offends you, for whatever reason, try to see it from their point of view first.  Yes, that means you’ll likely need to have a conversation with them rather than simply join the stone-throwing crowd.

In short, if you believe it’s appropriate to vilify someone, then at least have the integrity to make sure that they’re actually a villain before doing so.

Written by Kaushik Saha, Certified Disciplined Agile Coach (CDAC)

Many people focus on Scrum process framework when they talk about Agile because of several reasons:

1. Scrum is a lightweight method
2. Scrum is an improved process framework
3. Scrum is a very focused approach and event-driven
4. Scrum is easy to understand

But we must extend beyond Scrum wherein agile can be implemented in lean way and the Scrum could be one approach for the execution the construction phase.  Disciplined Agile Delivery (DAD) takes a “hybrid” approach and beginning-to-end approach that extend Scrum’s Construction lifecycle to explicitly address the full delivery lifecycle.  Disciplined Agile calls out three light-weight phases:

1. Inception –> Inception is a pre-Sprint phase in terms of Scrum which can talk about envisioning and planning where you  develop a common vision with stakeholders ; initial release planning; initialize the environment and infrastructure; identify initial risks; and adopt a goal-driven, non-prescriptive approach to development.
2. Construction –> During Construction the team incrementally and iteratively builds business value in the form of a potentially shippable, or better yet consumable, solution each iteration.
3. Transition –> During Transition you ensure that the solution is ready to be shipped and then ship it.

 Comparing Scrum and DAD:

1. Scrum delivery is focuses on “working software” but DAD goes further for a “complete end-to-end solution“.
2. Scrum is prescriptive, but DAD is pragmatic.
3. DAD is easily tailored.
4. Scrum focuses only on “Construction” phase; but DAD includes Inception. Construction, and Transition phases.
5. Scrum is targeted from single team to multiple teams. DAD goes further to be scalable at both the tactical and strategic levels.
6. Scrum is one approach process framework, but DAD is a Hybrid Framework that includes several lifecycles:  Agile (Scrum-based), Lean (Kanban-based), Continuous Delivery:Agile, Continuous Delivery: Lean, Exploratory (Lean Startup), and Program (team of teams).

The DAD Approach brings Agile and Lean Practices under one umbrella:

• DAD applies Lean development principles to enable scaling, wherein system can be optimized as whole by eliminating non-value added activities and build the quality inside with amplifying learning.
• DAD also adopts visualization of Kanban workflow to measure & manage flow of work by limiting work-in-progress.
• DAD’s Agile lifecycle applies the Scrum process framework in the Construction phase.
• DAD brings XP engineering practices in the Construction phase to reduce feedback cycle and develop quality code faster.

One of the key aspects of a disciplined agile approach is to be enterprise aware.   The fundamental observation is that your team is only one of many in your organization, except in the case of very small organizations, and as a result should act accordingly.  This means that what your team does should reflect your organization’s overall business and technical roadmaps, that you should strive to leverage as much of the existing infrastructure as possible, that you should try to enhance the existing infrastructure, and that you should work with other teams appropriately so that your overall efforts are complimentary to one another.  This is a straightforward idea conceptually, but in practice acting in an enterprise aware manner can prove more difficult than one would initially think.

Over the years we’ve been asked by several customer organizations to help them to understand how to account for the expense of agile software development.   In particular, incremental delivery of solutions into production or the marketplace seem to be causing confusion with the financial people within these organizations.   The details of accounting rules vary between countries, but the fundamentals are common.  For public companies capital expenses (CapEx) are preferable because they can boost book value through the increase in assets (in this case a software-based solution) and increase in net income (due to lower operating expenses that year).  On the other hand, operational expenses (OpEx) are accounted for in the year that they occur and thereby reduce net income which in turn reduces your organization’s taxes for that year.   Furthermore, in some countries you can even get tax credits for forms of software development that are research and development (R&D) in nature.  In order to get properly account for the expenses incurred by software development teams, and potentially to earn R&D tax credits, you need to keep track of the amount of work performed and the type of work performed to develop a given solution.  Time tracking doesn’t have to be complex: at one customer developers spend less than five minutes a week capturing such information.  The point is that the way that a software developer’s work is accounted for can have a non-trivial impact upon your organization’s financial position.  This in turn implies that the need for agile developers to their track time is a fairly simple example of acting in an enterprise aware manner.

So, I thought I’d run a simple test.  On LinkedIn’s Agile and Lean Software Development group I ran a simple poll to see what people thought about time tracking.  It provided five options to choose from:

• Yes, this is a valuable activity (33% of responses)
• Yes, this is a waste of time (39% of responses)
• No, but we’re thinking about doing so (2% of responses)
• No, we’ve never considered this (18% of responses)
• I don’t understand what you’re asking  (5% of responses, one of which was mine so that I could test the poll)

The poll results reveal that we have a long way to go when it comes to working in an enterprise aware manner.  Of the people inputting their time more of them believed it was a waste of time than understood it to be a valuable activity.  When you stop and think about it, the investment of five minutes a week to track your time could potentially save or even earn your organization many hundreds of dollars.  Looking at it from a dollar per minute point of view, it could be the highest value activity many developers perform that week.

The discussion that ensued regarding the poll was truly interesting.  Although there were several positive postings, and several neutral ones, many more were negative when it came to time tracking.  Some comments that stood out for me included:

• It’s a colossal waste of time unless you’re billing a customer by the hour.
• We record time spent on new development work (as distinct from other tasks such as bug fixing in legacy code and so on) as this is capitalised as an asset and depreciated.
• I think the *most* pointless example was where the managers told us what we should be putting in.
• One day we will move past the “just do it” mentality and have some meaningful conversations and the reasons for what we do.
• In my experience time tracking is a massive waste of time. It’s a poor substitute for management.
• Why do you need to know more than the info available through Sprint Backlog, Sprint burndown and the daily standup?
• Some of my teams (I am SM for three teams) are skeptical about this. They do not think that keeping track of task hours this way will be any more useful than the daily standup reports.  And they do not believe that Management can resist the temptation to use task hours as a measure.

So what can we make of this?  First, it’s clear that delivery teams need a better understanding of the bigger picture, including mundane things such as tax implications of what they’re doing.  Second, it’s also clear that management needs to communicate more effectively regarding why they’re asking people to track their time.  To be fair, management themselves might not be aware of the tax implications themselves so may not be making effective use of the time data they’re asking for.  Third, management needs to govern more effectively.  Several people were clearly concerned about how management was going to use the time data (by definition they are measures) which could be a symptom of both poor communication as well as poor governance (unfortunately many developers have experiences where measures have been used against them, a failure of governance, and no longer trust their management teams to do the right thing as a result).  Fourth, some of the team-focused agile practices, such as burndown charts (or better yet ranged burndown charts) and coordination meetings may be preventing people from become enterprise aware because they believe that all of their management needs are being met by these practices.  Finally, many organizations are potentially leaving money on the table by not being aware of the implications of how to expense software development.

In Summary

Disciplined agilists are enterprise aware.  This is important for two reasons: First, you want to optimize your organizational whole instead of sub-optimize on project-related efforts; second, you can completely miss opportunities to add real value for your organization.  In the anecdote I provided it was clear that some agile developers believe that an activity such as time tracking is a waste, when that clearly doesn’t have to be the case.  Worse yet, although someone brought up the issues around capitalizing software development expenses early in the conversation a group of very smart and very experienced people still missed this easy opportunity to see how they could add value to their organization.  It makes me wonder if some of the agile rhetoric is getting in our way of being more effective as professionals (and, BTW, there are light-weight options for tracking time available to you).

Related Reading

• Strategies for Tracking Time on Agile Teams
• Agile Software Development and CapEx/OpEx

Previously, in Agile Transformation: Being Agile, Doing Agile, and Supporting Agile and Agile Transformation: Comparing Transformation Strategies I discussed the need for your agile transformation efforts to address three factors: People-oriented issues (being agile), process-oriented issues (doing agile), and tooling issues (supporting agile).  I argued that you must focus to a different extent on each of these factors – 80-85%, 10-15%, and 5-10% respectively – and that you need to address all three at once if you’re to successfully transition to agile.  But what is the impetus for becoming more agile in the first place?

The answer is that you want to help people to become more effective so that they can work together to address the success criteria that their stakeholders have set out for them.  The challenge of course is that success criteria varies by team.  Some teams want better time to market, some want better quality, some want improved staff morale, some want improved stakeholder satisfaction with what gets delivered, and some want improved return on investment (ROI) in IT.  Many of course need to deliver on a combination of several of these criteria.

The point is that every team has their own success criteria that they should fulfill.  To do that effectively, agile coaches need to help these teams to “be agile” so that they have the proper mindset and culture to provide a foundation from which they can “do agile”.  To “do agile” teams need to understand, and have the skills to execute, agile practices in such a way that they perform the right practices at the right time to the right extent.  And to do that they need the appropriate tools to support these practices.

Your stakeholders could care less about whether your agile or even about what agile is.  They do care deeply about whether your team is able to meet, and better yet exceed, the criteria set out for them.