Software Engineering (and many other professions) requires regular diagnostics, retrospectives, post-mortems, root cause analysis, etc. We want to reflect on the past and learn something from it, so we can improve, and avoid repeating mistakes.

But often this process of learning from the past doesn’t work. Some teams identify causes and still keep struggling with the same issues. They compose a good story of why things happened, and still fail to improve.

"I have seen far too many people who upon recognizing today's gap try very hard to determine what decision has to be made to close it. But today's gap represents a failure of planning sometime in the past." - Andrew Grove (from "High Output Management")

Many times I have seen fantastic descriptions of past errors, and as soon as the next day, the teams reverted to the usual, almost instinctive behavior.

Why is this so? Take a look a this excerpt form a book about medical diagnostics: “The Sociology of Medical Screening: Critical Perspectives, New Directions” edited by Natalie Armstrong and Helen Eborall

Because we humans don’t like ambiguity, we make stories with “logical” conclusions. It gives a sense of safety that we "figured it out".

When doing a retro, there is no way to know for sure if we took everything that happened into account (unknown unknowns). We may even have a preference where we wish a root cause to be (a specific person, team or a system) — and tailor our story to fit that. We sometimes mix facts and assumptions.

What to do about this and how to learn from mistakes?

The most valuable learning is to find out what was (and what was not) in our minds during the time of the error. Ask this:

Why did it look reasonable for us to do this the way we did? Why the important things were not important to us before the error and at the time of error?