The Dangers of Over-engineering

Let me start with a quote from one of the greatest anthropologists:

“Our students wanted to know everything: but only the newest theory seemed to them worth bothering with. Knowing nothing of the intellectual achievements of the past, they kept fresh and intact their enthusiasm for ‘the latest thing’. Fashion dominated their interest: they valued ideas not for themselves but for the prestige that they could wring from them.”

— Claude Levi-Strauss (1908–2009), French anthropologist and ethnologist

Many failures to deliver profitable software come from two types of planning errors, both related to over-engineering:

1. optimizing systems for a larger scale than the business needs
2. preparing for future needs that never come

A common thing for these two planning errors is that they provide space for complex, fashionable technical solutions. Over-engineered solutions are often disconnected from businesses and rarely deliver value in the projected timelines.

Everyone loves to work on new and modern technologies. It brings prestige to work on the "latest tech", but some teams often lose sight of the actual problem they are trying to solve, at least until they discover that each modern solution brings another set of issues and toil.

Some engineers and leaders rush toward new and shiny things without considering tradeoffs because they are either: a) not aware of what is important for the business or b) they use opportunities for technical acrobatics which they feel proud to work on (and it is good for their resumes).

Business awareness of engineers

All salaries in a company come from the same budget, yet many engineers don’t know what brings the most profit. They work on user stories given by product owners/managers who rarely explain the impact engineering has on business. Some of the product managers are more focused on “velocity” than providing engineers with context. Without clarity of business goals, and limitations, engineers can not optimize the systems for profit.

If engineers were informed and cared about the business goals, they would be able to constantly make micro-decisions aligned with the business, and this would reduce over-engineering.

Prestige and hype in tech

But besides knowing and caring about the company’s goals, there is another reason for over-engineering: professional prestige and media hype.

Internet blogs and bookstores are full of content that is actually marketing material rather than helpful information. Technical authors love to write about the newest trends to create prestige and a name for themselves. Some companies also publish material to tell stories about their technology to attract new talent. Of course, they will skip the parts where they struggle with badly engineered components and attrition in the teams who have to maintain those.

Once a certain technology becomes fashionable, everyone wants to join that elite group of experts who have it. When too many people have a skill, it stops being prestigious, and another fashion shows up. These fashion cycles can last years.

For the teams who followed a fashion blindly and started projects without sufficient insights, a network of technical and organizational dependencies will remain hidden until deep into the project implementation. When these problems surface, predictability of delivery will become impossible. The only fuel that remains to push the project forward is enthusiasm and the ability of the business (budget) to suffer lower productivity until things stabilize. This fuel is limited and burns quickly.

Two pushbacks from engineers

One example is optimizing a system for a larger scale than needed. The famous push back from engineers when challenged about over-scaling is “Yes, but we don’t know how big scale and how many users will we have in the future”. Unless we all are clear about the business hypothesis (the size of the scale we will aim for), optimizing for too large a scale is a waste of time and money. The only meaningful bet of how large to scale comes from the business and sales trend projections. If that bet is too high, the company will lose money because systems start to bring value only when released to production. And each unnecessary requirement (functional or non-functional) delays the release date. The later the release, the later you will realize the profit.

Another thing we hear from engineers is “Yes, but if things change in the future, we will have to rewrite a system, so let’s invest more time now to make it future-proof now”. Unfortunately, you can not make anything 100% future-proof because when that “future” comes, you will likely face new and unpredictable problems. You may get lucky with a feature or two or play a bit smarter, but you cannot prepare a system for all possible futures.

What to do about this?

An organizational culture that favors pragmatism and flexibility instead of fashion and intellectual prestige. Culture is best visible by who gets promoted and who leaves the company, so if pragmatic and insightful people start leaving, over-engineering is unavoidable. If pragmatism is not a criterion for gaining seniority, over-engineering will appear.

Enemies of pragmatism are vanity (chasing personal recognition rather than business objectives), perfectionism, and disconnect from the business. Teams that figure out these three things usually don’t have to worry about “velocity” because they do things directly related to business goals, and they like to do it with as few resources as possible. When an engineering team provides a tangible increase in the company’s profit or valuation, nobody really cares about their “velocity”.

Why is over-engineering difficult to solve?

Because it is about how technical decisions are made in a company. Besides that, a lack of technical experience makes things seem easier than they are, so some teams don’t properly understand the solution’s impact until it is too late.

There is no simple cure to over-engineering. It starts with hiring and training engineers and leaders who require clarity of goals and enjoy accomplishing as much as possible with minimal resources. People who think like this evaluate their investment of time, cost of maintenance, and all other costs against the benefit for the company. For them, technology is a tool to achieve profit.

What to do about over-engineering?

I am not saying that we should never try anything new, far from it. What I am saying is that many engineers and leaders do not optimize technology for business first. What can we do about it? It starts with hiring, and extends to coaching teams on decision-making and having good processes. In the end, none of these efforts can move the needle enough without the involvement and support of top leadership who can visibly reward pragmatism. Improving an organization in this aspect is a process that takes time and attention to shift the culture. The earlier you start the better.

Hiring

If you hire pragmatic people, they will naturally resist over-engineering.

Here are a few ideas for job interview questions that could show if a candidate has a pragmatic approach to engineering:

• How do you recognize overengineered solutions?
• What is the perfect software for you?
• What is the total cost of developing a software feature?
• How do you optimize software systems, and why?

Processes

Processes are not magic bullets that solve everything, but they can help to keep over-engineering under control. Here are a few ideas:

• Introduce a system for making technical decisions based on costs and benefits - like business cases. This will keep over-engineering under control because costs (monetary, time, processes, time to change) will surface.
• Introduce tables of comparisons of multiple solutions. When faced with multiple options, compare them in terms of what kind of investment is necessary (money, time, change of existing systems) for each solution.
• Visibly reward pragmatic decision-making.