Why Theory Can Solve Problems Empiricism by Itself Can't

From the Manifesting Business Agility Blog
This blog concerns itself with organizations moving to business agility—the quick realization of value predictably and sustainably, and with high quality. It includes all aspects of this—from the business stakeholders through ops and support. Topics will be far-reaching but will mostly discuss FLEX, Flow, Lean-Thinking, Lean-Management, Theory of Constraints, Systems Thinking, Test-First and Agile.

About this Blog


Recent Posts

Are You Working on Your Problems?

Dealing with Complexity by Creating a Bias For Simplicity

Using Inherent Simplicity to Guide Your Actions

Why to use the word capacity instead of resource when referring to people

Why Teams Should Have a Tailored Approach for Adopting Agile

This was originally published in May 2012.

Those who cannot remember the past are condemned to repeat it. George Santayana

The time was 1847. The place was the Vienna General Hospital. New mothers in the doctors’ wards had been dying of puerperal fever with an extremely high mortality rate – three times that in the midwives’ ward. It was a mystery. It could not be explained. But Ignaz Semmelweis had been observing this for years. Had studied the situation and made some interesting connections.

The situation was very puzzling. There were two clinics in the hospital Semmelweis oversaw. Clinic one was the teaching service for medical students; clinic two was where only midwives worked. Why was the presence of doctors apparently killing new mothers? Women were coming to the hospital’s maternity ward for the benefits of the child care they would receive. But the high mortality rates had them try to avoid coming on a day when they would be admitted to the first clinic. In fact, many preferred to have their births in the street, then come to the clinic for the benefits. Surprisingly, the mortality rate for those giving birth in the street was significantly lower than for those giving birth in the doctors’ clinic. It was a mystery, and nothing could explain it.

Until Semmelweis figured out the connection. And proved it – lowering the mortality rate from the 10-35% it had been to 1%. The connection was doctors working on cadavers (it was a teaching hospital) and then going to do their rounds with patients. The solution was Semmelweis instituting the practice of hand disinfection with a chlorinated lime solution he created. The results were dramatic. But his theory was incomplete. He could not explain why it worked. The existence of germs had not been postulated yet, let alone detected.

His theories were scorned. Administrators of hospitals thought the suggested disinfection process would take too much time. Doctors were not eager to admit that they had caused so many deaths. It was not until years after Semmelweis’ death that his theories were accepted – after Pasteur could demonstrate the existence of germs. For more on Semmelweis, see Wikipedia.

How does this relate to us? I would suggest the knowledge of germs to doctors is like the knowledge of flow to software developers. It is not all there is (other things cause disease than germs) but it is pretty important to know. Things that often appear complex and unknowable, are, in fact, complicated but unknown. BTW: I am not suggesting that software development as a whole is not complex, just that not all of it is complex.

I have been doing some form of agile consciously for over 20 years. I have been doing agile practices at one time or another for over 3 decades Unfortunately, that “one time or another” was hit or miss. I did it when I intuited a solution, but that was relatively rare. I am a big believer in understanding why what you are doing works – see an old blog of mine – Smart People, XP and Scrum – Is There a Pattern?

It seems the software industry has hit a crisis in the adoption of Agile. It is almost to be expected that when you hear about a large organization successfully adopting Scrum for several teams working individually, you learn they can’t quite get it to work well across teams. Why is this? Well, it’s a mystery for some. Not for others.

Since 2005 we (Net Objectives) have been helping clients who have been encountering cross-team challenges in their development methods (IT and products). Many of the insights we’ve had have come from looking at the theories of Flow and how they apply to software development. This is one reason I am so passionate about the need to understand that Scrum itself, is a manifestation of Flow and Lean thinking. By not being consciously aware of this, many Scrum practitioners can’t extend it as needed, or abandon it for better methods when available.

I have seen some development groups (75-150 folks) transform themselves almost overnight by attending to flow. I have also been somewhat mystified by much of the Agile consulting community’s resistance to many of these ideas – having once been thrown off a discussion group for insisting they were a better alternative than (still) popular methods of team collaboration.

Posted on: November 15, 2019 08:32 PM | Permalink

Comments (9)

Please login or join to subscribe to this item
A rationalist I see. Theory does have a place here, but so too does empiricism through his experiments and experience in order to prove his theory.

Empiricism by it self is a theory, so there is some level of contadition here, nevertheless like @Sante pointed is nice to have the theory bases but we have to experiment to adapt the theory to reality, in the end I think se need both like @Sante.

Dear Al
Interesting reflection
Thanks for sharing

We agree that much of the knowledge comes from experience

Scientific thinking takes a different approach.

I agree with the comments of Sante and Alexandre

@sante - i don't consider myself a rationalist. I consider myself a scientist. i use a combination of theory (hypotheses) validate by observation (sometimes experiments, sometimes life). I talk about theory like this because much of the Agile community is based on empirical process control resulting from a misunderstanding one one of Scrum's creaters (Ken) had when he talked to some chemical process engineers.

I like this paraphrase of Deming: Theory, by itself is useless. practice by itself is expensive.

@alex - that made me smile. I agree. But science is based on theories that are to be validated. The "theory of empiricism" is self-contains and does not search for validation.

@luis - i did not mean to imply that theory is more important than practice. Just that both equally important.

@luis - i did not mean to imply that theory is more important than practice. Just that both equally important.

@Al, I haven't heard that one from Deming, but I like it.

Please Login/Register to leave a comment.


"Don't let school interfere with your education."

- Mark Twain