Disciplined Agile

Are you tired of being embarrassed when you tell people what agile certifications you have? Are you tired of dancing around what little you had to do to “earn” your certification or what little knowledge about agile that effort actually imparted? Are you tired of explaining that you got certified only because it looks good on your resume, when in fact it only looks good to organizations that really don’t know what they’re asking for?  If you answered yes to any of those questions, it’s time to up your game.

Disciplined Agile certification takes a principled approach that provides real value to practitioners. Disciplined Agile certifications are respected because they are earned. There are several benefits of Disciplined Agile certification for practitioners:

• Increase your knowledge.  Disciplined Agile certification requires you to have a comprehensive understanding of Disciplined Agile Delivery, which in turn describes how all aspects of agile principles and practices fit together in an enterprise-class environment.
• Improve your employability.  Disciplined Agile certification indicates to employers that you’re dedicated to improving your knowledge and skills, a clear sign of professionalism.
• Improve your career options.  Disciplined Agile certification can help you gain that new position or role as the result of your increased knowledge base and desire to improve.

Disciplined Agile Certification is for agile professionals working in enterprise-class settings such as banks, insurance companies, retailers, and government agencies. You’re not working in ideal situations – you have legacy cultures, legacy systems, and legacy processes to overcome – but that doesn’t mean you can’t make things better. You take pride in your work and you want to create environments where you can be effective, and you can do that by adopting Disciplined Agile strategies.

The Disciplined Agile Certification Program

The Disciplined Agile Certification program has three main certifications for practitioners – Certified Disciplined Agilist (CDA), Certified Disciplined Agile Practitioner (CDAP), and Certified Disciplined Agile Coach (CDAC) – that build upon each other. There is an additional designation, Disciplined Agilist (DA) and a fifth designation for trainers, Certified Disciplined Agile Instructor (CDAI).

Certified Disciplined Agilist (CDA): Shu (Beginner)

This certification indicates that the holder has comprehensive knowledge of how the Disciplined Agile solution delivery process works from beginning to end. To earn this Shu-level certification you need to pass a comprehensive test. It typically takes between 10 and 15 hours of classroom or reading time to prepare for the test. The primary benefits of this certification are that it:

• Shows that you have an understanding of how agile solution delivery works in enterprise-class settings;
• Is a meaningful certification that sets you apart from the multitude of “certified masters”;
• Indicates that you have the desire to go beyond “cargo cult agile”;
• Directs you down a path that reflects the realities faced by agile teams working in enterprise-class settings, enabling you to recognize and avoid the time consuming pitfalls common to Scrum teams.

Certified Disciplined Agile Practitioner (CDAP): Ha (Intermediate)

This certification indicates that the holder has comprehensive knowledge of how the Disciplined Agile solution delivery process works from beginning to end and has experience applying agile strategies in practice. To earn this certification you must have earned the CDA first, have at least two years of agile work experience (you are required to provide references), and you have passed the CDAP test. The primary benefits of this certification are that it shows you’re:

• Proficient at agile development and on the path towards mastery;
• Ready to start helping others learn, potentially in a junior coaching role supervised by someone more experienced, such as a CDAC.

Certified Disciplined Agile Coach (CDAC): Ri (Expert)

This certification indicates that the holder has comprehensive knowledge of how the Disciplined Agile solution delivery process works from beginning to end, has experience applying it in practice, and has proven giveback to the community. To earn this certification you must have earned the CDAP first, have at least five years of agile work experience (you are required to provide references), and have gone through a board-level interview. The primary benefit of this certification is that it shows you’re qualified to coach agile delivery teams. Effective coaches must have deep knowledge in what they are coaching people in, and that requires proven experience.

Retention

To retain your certification you should be dedicated to continuous learning of agile strategies in general, and in Disciplined Agile (DA) strategies in particular. Once someone is certified there are no direct membership dues. For CDA’s to retain their certification level they must take and pass the CDA test every two years. Having said that, at the two year point a practicing CDA is eligible to apply to become a CDAP anyway. Anyone with a CDAP will need to either pass the CDAP test every two years, or if they are qualified to apply for and become a CDAC. CDACs must provide proof of continuing give back to the DA community.

Further Reading

• Apply for certification
• Certification roadmap
• How to pass the certification test
• ShuHaRi
• The principles behind Disciplined  Agile certification
• The value and limits of certification

One of the age-old debates in the software world is whether software architects need to write code.  We suspect that as an industry we’ll never reach consensus on this topic. Here are our thoughts on the subject.

Short Answer:

Hell yes!

Detailed Answer:

In the following table we list the advantages, disadvantages, and considerations (when does the strategy makes sense) to compare whether a software architect should write code or not.  You may recognize this approach from our book Disciplined Agile Delivery.

In the Disciplined Agile (DA) toolkit we’ve made it very clear that we expect Architecture Owners to be actively involved with the development of the solution.  On Disciplined Agile teams the Architecture Owner is effectively a team member with additional responsibilities around leading the team through architecture decisions, in coaching them on architecture skills, and in working closely with your Enterprise Architecture team (if any) to ensure their development team understands and is working towards your organization’s technical roadmap.

We’re often told that it isn’t realistic to expect architects to write code.  Invariably this is coming from people who are currently working in traditional IT organizations that have very well-defined roles, IT organizations that more often than not are struggling to be effective.  Our response is always the same – Really?  Are development teams following your architectural strategy?  Are they eager to work with you, or are they forced to work with you?  This generally leads to a discussion that reveals that things aren’t going so well for these architects in practice, and sometimes leads to a positive discussion as to how we can move towards a more effective approach for them.  They kind of approach described in the Disciplined Agile (DA) toolkit.

Additional Reading:

• In the mid-1990s James Coplien captured the organizational pattern Architect Also Implements.
• The Architects Don’t Code anti-pattern on Ward Cunningham’s C2 wiki is also a valuable read.

On Tuesday, February 26th I ran a webcast entitled Disciplined Agile Data Warehousing/Business Intelligence.  During the webcast we received several very good questions, some of which we had time for and some of which we didn’t get to.  Regardless, we’ve decided to answer all of them here in this blog.  In a few cases we’ve had to reword the questions to correct spelling or grammar mistakes and in a few cases we combined questions because they were effectively the same.  We have organized the questions and answers into the following categories:

1. Start here
2. Vertical slides of DW/BI functionality
3. Architecture and design
4. Refactoring databases
5. Other development practices
6. Miscellaneous
7. Where can we learn more?

Start Here

Where can we download the slides?

1. A PDF of the slide deck can be found on Slideshare.net.
2. The recording of the presentation can be found on the DAC webinars page.
3. We have also created a Disciplined Agile DW/BI poster that you can download from the DAC posters page.
   

Vertical Slices of DW/BI Functionality

How do we start with high-risk, vertical slices?

Disciplined Agile teams take a risk-value approach to prioritizing their work, an extension to Scrum’s value-driven approach.  The basic idea is that disciplined agile teams will implement the highest-risk requirements first so as to prove the architecture with working code early in the lifecycle.  This strategy works quite well for DW/BI solutions, just like any other type of solution.  To do so, you need to understand the risks that your team faces.  For a DW/BI solution, these risks may include:

• Can you access the data in key operational data sources?
• Can you process the volume of incoming data?
• Can you address data quality issues found in operational data sources?

To address these risks, look for high-value requirements whose implementation would force your to address the risks.  Implement those requirements first.

How can we include all of the stages of DW/BI development into iterations/sprints e.g. Data modeling, staging, profiling, dd, DQ, ETL, reporting, testing into single iteration?

This can be a struggle for any team new to agile, not just a DW/BI team.  The challenge is that the majority of organizations have taken a Tayloristic approach to organizing the way that they work.  They have specialists who each do a portion of the work, handing off their portion to the next person once they’ve completed it.  It is virtually impossible for specialists to get all of the work done to develop a working, vertical slice of the solution within the timeframe of a two-week iteration, let alone one that is shorter.  The overhead of specialists trying to work in a Tayloristic, “software factory” strategy is just too great.  Unfortunately the culture within the data community tends to motivate over specialization and the overhead surrounding it.

What agile teams need are generalizing specialists, T-skilled cross-functional people who work together collaboratively.  Each person has one or more specialties, they need to be able to do something useful, but they also have a general knowledge of the rest of the process and are willing to pick up new skills from one another.  When your team is made up of people like this the wait time between tasks (modelling, development, testing, …) starts to disappear as does all the bureaucracy (reviews, traceability matrices, …) around coordinating such activities.

The fundamental challenge is that you likely don’t have generalizing specialists right now.  As we like to say, you go to war with the army that you’ve got.  You need to build a team of specialists right now because that’s the type of people you have.  Insist that they produce a vertical slice of the solution during the current iteration.  There will very likely be a lot of complaining about this at first, often because the team can’t imagine how they can pull this off.  If possible, colocate them in an agile team room (sometimes called a tiger team room or war room) to get them working side-by-side.  This will help to improve communication between the people involved and provide better ways to collaborate (such as agile modelling at a whiteboard).  Push the idea that they should be doing non-solo work – such as pair programming, mob programming, or modelling with others – so as to share skills and get the work done quickly.  They will need a strong agile coach to help them to learn these new strategies and to break themselves of their ineffective specialist habits.  An important thing to observe is that many other teams have discovered how to work this way – you can too.

Vertical slice is good but if only half of a report is created in a single iteration it may not be useful to the stakeholders.  What should we do?

The idea is to get something that works completely done each iteration.  The stakeholders, often via the Product Owner (PO), will determine whether what you’ve built can be deployed into production.  This is why we use the term “potentially consumable solution“, an improvement over Scrum’s “potentially shippable software” – it’s potentially consumable, but that doesn’t mean it has to be deployed only that the option to deploy is there.

Having said this, if possible find a way to complete the entire report in a single iteration.  Sometimes easier said than done.

How to handle data coming from multiple source during data modeling within a single sprint when we are expected to develop a report in end?

This is a very common occurrence.  The solution is that you only model enough for that report at the time.  Early in the lifecycle during Inception you will have done a bit of high-level modelling to explore the initial scope and to identify your technical strategy (your architecture).  These high-level visions are fleshed out during Construction each iteration via Agile Modeling practices such as just-in-time (JIT) model storming, look-ahead modelling (backlog refinement in Scrum), and even iteration modelling (an aspect of iteration/sprint planning).

Architecture and Design

But many of the companies are not using data vault?  And may companies are reluctant in using this?

The Data Vault 2 method is not required to be agile, but it is an approach that we highly suggest due to its practicality and flexibility.

While using Data Vault how can we overcome challenges with teams supporting and creating data from various companies and geographies. Are there are control risks associated?

Regardless of the architecture and design methodology that you follow, you will have risks associated with using data from multiple sources.  Those risks tend to increase with multiple geographies or multiple companies involved.  The greater the risk, the greater the importance of having a database regression test suite in place that validates your work.

Refactoring Databases

Can we take these database refactorings as technical debt user stories?

Database refactorings are small changes to the design of your database schema (which includes functionality such as triggers or stored procedures) that improves the quality of the design without changing the semantics of the schema in a practical manner.  Examples of database refactoring includes Rename Column, Introduce Cascading Delete, and Replace One-to-Many with Associative Table.  A full catalog of database refactorings can be found here.  Because database refactorings are small you should just do them as a matter of course as you work on the database, they generally aren’t large enough to justify their own work items (such as a technical debt story).  I suggest that you read the Spilled Juice Analogy.  However, if you wanted to fix a collection of refactorings up into a single technical debt story I suppose you could do this.

Most often, data sources sit in the “business” area, that cares very little about how software are works or needs to work. Isn’t this attempt to clear data at the source something that will put strain on the organization?

I suggest that you read the Spilled Juice Analogy.  Does the business want to be able to make decisions based on information that they can trust?  Does the business want to reduce their long-term IT costs?  Does the business want IT to be able to bring solutions to market quicker?  If the answer to any of these questions is yes then they need to start treating data like an asset and invest in concrete quality techniques such as database regression testing and database refactoring.

Other Development Practices

Can we use database virtualization?

Sure, why not?

Automation – How that can be done in DW/BI project?

You can:

• Write regression tests for your database.
• Adopt continuous database integration.
• Write automated deployment scripts for database changes.

How does the practice of spikes apply?

A spike is a bit of code that is used to explore or prove a concept, typically written to pay down a technical risk.  In a DW/BI solution that might be a bit of code to:

• See how a data source is accessed
• Explore a feature of your ETL tool
• Work with a BI tool for the first time
• Explore whether a data source can handle expected volume
• And many more technical risks.

How do you make sure that documentation is updated and yet consumes less time which is accommodatable within the Agile iteration/sprint of 2 weeks? Without documentation the developers who come into the project at a later point of time are in no man’s land without updated documentation?

See:

• Agile/Lean Documentation Strategies
• Practice: Document Continuously
• Practice: Document Late
• When should we create a document on an agile team?

Miscellaneous

How can enterprise data governance fit into an agile DW/BI mindset?

See:

• Lean Data Governance.
• Supporting Agile Delivery Team Governance.

What lifecycle would be most appropriate for projects implementing COTS software solutions (like ERP) within the company so investment would be maximized?

It depends on the situation that you face, including the skills of the people involved.  I would think that your best bet is the Agile/Basic lifecycle that is based on Scrum.

Where Can We Learn More?

At the DAC posters page you can download the Disciplined Agile DW/BI poster (amongst many others).

We have a detailed article entitled Disciplined Agile Data Warehousing that you will find informative.

In IT we are often asked to estimate the expected time/schedule or cost of software development. Sadly, the desire of stakeholders to have “predictable” schedules or costs results in significant dysfunction within a software development team.  When a software team is  forced by their stakeholders to commit to a schedule/cost they must then ensure that the schedule/cost doesn’t slip. For example, to protect themselves from increased time and cost due to scope creep, software development teams will make it difficult for stakeholders to change their requirements during Construction and even go so far as to drop promised scope late in a project. The desire of stakeholders to reduce their financial risk often results in behaviors by the software development team that ensure that stakeholders don’t get what they actually want.  Naturally IT gets blamed for this.

We need to do better.  In this blog we summarize the things that we know to be true about software development estimation. In no particular order, they are:

1. Estimates are guesses. Look up the word in the dictionary – An estimate is a rough approximation or calculation, in other words a guess. Unfortunately, too many people think that estimates are promises, or worse yet guarantees.  In our opinion “guesstimate” is a far more appropriate word that “estimate”.
2. Scope on IT projects is a moving target. Our stakeholders struggle to tell us what they want and even when they do they change their minds anyway.  Any guesstimate based on varying scope must also vary in kind.
3. Guesstimates are probability distributions. Although your stakeholders may ask for a fixed amount guesstimate, for example “this will cost $1 million”, the reality is that there’s a chance the cost will be less than $1 million and a very good chance that it will be more.  There is ample evidence that the initial estimate for a software development project should be given in a range of -25% to +75%, so your million dollar project should be quoted as a range of $750,000 and $1,750,000.  This is shown in the diagram above by the green distribution curve.  In many organizations this can be politically difficult to do, and strangely enough in many cases stakeholders prefer to be lied to (it’s going to be $1,000,000) rather than be told the truth.
4. Guesstimates must reflect the quality of the inputs.  A guesstimate needs to reflect the quality of the information going into it – if your scope is fuzzy your guesstimate based on that scope needs to be equally fuzzy.  Sometimes stakeholders want a guesstimate with a tight range, perhaps +/- 10% (the red curve in the diagram), early in a project.  To provide a tight range such as this you need to have a very good understanding of the requirements and the design.  Early in the software development process this can only be done through more detailed modeling, an expensive and risky proposition which often proves to be a wasted effort because the requirements will evolve over time anyway.
5. Guesstimates anchor perception. The primary danger of providing guesstimates to people is that they believe them.  Tell someone that it’s going to be $1,000,000 and they fixate on that cost even while they are changing their minds.  Tell them that it’s going to be between $750,000 and $1,750,000 and most people will fixate on the cost of $750,000.  Some people will focus on the average cost of $1,250,000 even though the median was $1,000,000 (guesstimates are in effect Weibull probability distributions).
6. It’s easier to guesstimate small things. ‘Nuff said.
7. It’s easier to guesstimate work you’re just about to do instead of work in the distant future. It is much easier to identify the details of work to be done right now, and thus turn a large piece of work into a collection of smaller pieces that are easier to guesstimate.  In part this is because you have a much better understanding of the current situation you are working in and in part because you are more focused on the here and now.
8. The people doing the work will likely give a better guesstimate.   They are more motivated to get the guesstimate right, particularly when they must commit to it, and have a much better idea of their abilities.  Granted, someone may need to coach people through the guesstimation effort.  In Disciplined Agile this is a responsibility of the team lead.
9. Someone who has done the work before will give a better guesstimate than someone who hasn’t. Experience counts.
10. Guesstimates reflect the situation that you face.  Which organizational situation do you think will result in a short schedule and lower cost: Five people co-located in a single room or the same five people working from different locations in difficult time zones?  Or how about a team working under regulatory constraints versus the same team without those constraints?  Context counts.
11. Multiple guesstimates are better than a single guesstimate. Getting guesstimates from several people provides insights from several points of view, hopefully prompting an intelligent conversation that enables you to develop a guesstimate with better confidence.  Similarly, the same person producing guesstimates for the same piece of work using different guesstimation strategies will also provide a range of answers that you can combine.
12. Guesstimates should be updated over time. As your understanding of what stakeholders want improves, and your understanding of how well your team works together, you should update your guesstimates.  As your understanding of the fundamental inputs into your estimate improves you are able to produce a better estimate, thus enabling the stakeholders of that guesstimate to make better decisions.
13. It costs money to produce a guesstimate. The precision of an estimate is driven by the detail and stability of the information going into it. Want a tighter range on your estimate?  Then you’re going to have to have a better handle on the requirements, design, and capabilities of the team doing the work.  This greater precision requires greater cost.  The fundamental question posed by the #NoEstimates community is effectively “Is the value of improved decision making capability from having the guesstimate greater than the cost of creating the guesstimate?” The implication is that you must ensure the cost is much less than the benefit, hence their focus on finding ways to streamline and even eliminate the guesstimation effort.
14. Guesstimation is far more art than science. See point #1 about estimates being guesses.  The best guesstimates are done by the people doing the work, just before they need to do the work, for small pieces of work.
15. Formal software guesstimation schemes are little more than a scientific façade. Function point counting, feature point counting, and COCOMO II are all examples of formal strategies.  They boil down to generating numeric guesses from your detailed requirements and design, plugging these guesses into an algorithm which then produces a guesstimate. These are all expensive strategies (they require detailed requirements and design work to be performed) that prove to be risky (because they often force you into a waterfall approach) in practice.  Yes, they do in fact work to some extent, but in practice there are much less expensive and less risky strategies to choose from.  People like these type of guesstimation strategies because they provide a false sense of security due to their complexity and cost.
16. Past history isn’t as valuable as people hope.  Some formal guesstimation strategies are based on past history, but this proves to be a false foundation from which to build upon for several reasons.  First, people have different levels of capability which change over time as they learn. Capers Jones has shown that developers have productivity ranges of 1 to 25, the implication being that if you don’t know exactly who is on a team and how well they work together your corporate history will be questionable.   Second, technologies evolve quickly so past history from working with older versions of technologies or completely different technologies becomes questionable at best.  Third, people and teams change (hopefully for the better) over time, implying that an input into your guesstimate is fuzzy at best.  Fourth, because every team is unique and faces a unique situation basing estimates on past history from other teams in different situations proves questionable.
17. Beware professional guesstimators. They tend to break many of the rules we’ve described above.

To summarize, when you are required to provide estimates for your software development efforts that you should take a pragmatic, light-weight approach to doing so.  This blog posting has provided many practical insights that should help guide your decisions.  These insights and many more, are built right into the Disciplined Agile (DA) toolkit.

When transitioning to an agile mindset people invariably ask about how documentation is addressed on agile teams.  Some documents, such as system overview documentation or operations manuals, are still a valuable part of the overall solution being delivered by the agile team.  More often than not a document, such as a logical data model (LDM) or detailed requirements model, isn’t needed at all because it can be replaced with a more effective strategy or the document can be greatly simplified compared with what a traditional team would develop.  This of course is a shock to most traditionalists, particularly for those who currently spend most of their time creating such documents.

To help people understand the agile approach to documentation creation, we find that it’s valuable to describe the logic that disciplined agilists follow.  This logic is captured in the following flow chart.

Let’s walk through the decision points one at a time:

1. Do you need a document?  There is a significant difference between needing a document, perhaps to persist important information, and wanting a document.  If you’re creating the document to communicate information to someone else then you really need to question its creation.  Research into media richness communication has found that detailed documentation is the least effective strategy available to you – better options include overview documentation, video conferencing, and of course face-to-face (F2F) discussion. For a second example, many organizations still follow a traditional, documentation-based approach to IT governance and as a result many teams are forced to create documents solely because someone in a governance roles wants it to be created (or at least the team perceives that they want it).  Sometimes people want a document because they fear that the information it would contain would be lost, not realizing that the older documentation becomes the less it is trusted (see Documentation CRUFT), so in effect the information is effectively lost even when it is captured as written documentation.  The point is that many documents aren’t really needed, they are merely wanted – if the latter, isn’t it better to deal with the real people and help people to recognize they don’t really need the document after all?
2. Is there a better option?  In many cases there are significantly better alternatives to writing documentation.  For example, instead of creating a detailed written requirement specification when you follow an acceptance test-driven development (ATDD), also known as a behavior driven development (BDD) approach, you capture requirements in the form of executable specifications.  Granted, this takes a different set of skills and tools to perform, but in the long run it offers far greater value to your team than written specifications.
3. Will you maintain the document? If you’re not willing to maintain a document throughout what should be its useful lifespan then don’t create it.  If this isn’t an option, then at minimum only work on it until the point where you’re still able to viably keep it up to date.  Documents that aren’t maintained aren’t trusted, and therefore not used.
4. Do you understand what the consumer of the document actually needs?  The only way that you’ll be able to create an effective document is if you know what the audience for that document actually needs, and the best way to do that is to work with them closely.  Your goal is to create a document that is just barely good enough (JBGE), or just sufficient, that fulfills their needs and no more.  For example, instead of creating a detailed architecture model using a software-based modelling tool, co-located agile teams will often prefer to create whiteboard sketches which they leave on their workroom walls.  Yes, a pretty architecture model would be nice to have.  But the whiteboard sketch gets the job done, it is much easier to evolve, and much less expensive to create.

When an organization transforms to agile many traditional IT professionals will struggle at first with taking an agile approach to documentation.  In traditional software development, and in particular traditional IT governance, documentation is used as a crutch and worse yet a band-aid over organizational dysfunction.  We can no longer afford this and must instead be smarter about our approach to whether and how we write documentation.

For more information about agile approaches to documentation, we suggest you read the article Agile/Lean Documentation: Strategies for Agile Software Development.