Technical Debt as a Product-Engineering Issue

What is technical debt, how does it affect the business, and how product and engineering can work together to handle it?

A search of the current literature reveals many definitions of "technical debt." Here are a few examples:

Technical debt is a concept in programming that reflects the extra development work that arises when code that is easy to implement in the short run is used instead of applying the best overall solution. - From techopedia.com

Technical debt … is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy (limited) solution now instead of using a better approach that would take longer.  -From wikipedia.com

Technical debt … is the result of prioritizing speedy delivery over perfect code.  - From productplan.com

Frankly, it doesn't add up. If we need to change something in the implementation of the code today because we made a "wrong" decision in the past, it doesn't matter why we made that decision. Now we simply need to fix it before it does more damage.

Maybe it was a good, optimal decision at the time. Maybe a future-proof solution would have taken too long to implement, the company wouldn't survive, and we wouldn't have a product today. Maybe a better technology wasn't available at the time or it wasn't as reliable as it is today. Maybe we couldn't assemble a team that could build a future-proof solution.

My point is that technical debt doesn't only happen because the team wanted to speed things up. There can be other things that manifest in the same way. Ultimately, the code written in the past doesn't fit what we need today.

A few realistic questions

• Is the purity of technical design the only relevant aspect when choosing an approach?
• When is technical excellence less relevant, and when it is the most important? We can't predict the future very far, so beyond some future date, there is no point in asking "what if" when building systems. On the other hand, if we implement everything "quick and dirty" the system will fall apart quickly.
• So how much should we design for the future?

These are questions related to technical debt and almost every software engineering team has faced them.

Even excellent code can become a problem

All definitions of technical debt that I found focus on "selecting the easy approach instead of the right one." That's not the only reason people write code that is not future-proof. True, it's a very common reason, but it's not the only one.

There are millions of lines of badly written code that work sufficiently well, deliver value, and will never have any kind of bad effect on anything. There are implementations full of antipatterns and "worst practices" that serve the needed purpose, and nobody will ever deep dive into that code because it doesn't generate any defects. I am convinced that many lines of bad code reside in the devices we happily use every day. Are these lines of code "technical debt" or just "impurities"? How relevant are they and how certain can we be in that answer? How much of our time should we dedicate to improving such "impurities" and what should we trade off to prioritize that cleanup?

I know some readers are now thinking "this author is advocating quick and dirty solutions just to produce features," which is far from true. Code perfectly written in the past can become technical debt today. There is a very good reason why best practices of programming exist - they represent the accumulated experience of generations of engineers and should be studied and used.

But technical excellence and best practices are not a waterproof seal against refactoring in the future.

I am saying that no matter how we implement a system, at one moment in the future that implementation may become unfit. So, in my view, this is a better definition of technical debt:

The reasons that the code doesn't fit anymore are relevant if we are about to do a retrospective and learn from the past. But when we start to notice the effects of technical debt on the business (and our productivity), we need to fix it. At that point, the reasons it's introduced and who's to blame don't matter anymore. What matters is doing what's best for the business. Every team has faced this challenge.

Sometimes technical debt is so big that it hinders or even prevents further development. These clunky components of unfit code are embedded in products whether we like it or not. And this is where product/business people can help a lot.

How technical debt affects the Business/Product

All products need maintenance. Maintenance takes time. Engineering time is a scarce resource in many companies, and we want to use it optimally. Some systems need just a bit of maintenance from time to time, and some systems require daily work just to keep them running.

To be profitable, the value a product generates should be higher than the sum of expenses to run it. Engineering time is one of those expenses. It's the simple economics of any investment, and that's what software development is: an investment. Technical debt increases operational expenses and that negatively affects the overall profit.

The more technical debt we remove, the more time is left for developing the product further. That's how the removal of technical debt directly contributes to the business. It reduces the cost of running products, which increases profit. It also has a lot of less tangible benefits: engineers' satisfaction (retention), cognitive load, motivation, focus, and productivity. Technical debt often causes toil: manual, numbing, and repetitive work. It can also slow down or even stop further product development.

Left unattended for long enough, technical debt often turns into a big hindrance for the business. When technical systems can't catch up with business needs, the company usually starts a "modernization" or "technical transformation" project. These are often years-long, painful efforts that drain the enthusiasm and motivation of everyone involved, but most importantly, drain the budget. Technical transformations can feel like re-implementing the same thing again. It is often very difficult to get the proper recognition and credit for the work done on technical transformation - if the business value is not clear for everyone.

The best way to avoid these expensive and long projects is to continually work to resolve smaller parts of technical debt and never let it accumulate. It's more of a planning problem than anything else. The nature of technical debt is that it grows in time if it is not handled (like any debt).

"The techies will handle it"

In our industry today we often see a strong distinction between "product" and "engineering" departments. Well-known frameworks and methodologies reinforce that silo. Job titles and responsibilities, personal "KPIs" and organizational trends support this division. The dominant thought pattern in the current era is "Product people should make and prioritize the requirements and engineers should implement them." Because of that, we have two things happening:

• engineers are not familiar with building requirements and the business context, and
• product teams don't know what kind of work is needed to build and run their products.

This separation of responsibilities causes a deep separation of knowledge and information. Many times when we talked about the technical debt I heard the following sentence from product and business people: "The techies will handle it, it's their responsibility to make sure the product runs reliably."

Coming up with new features is the most glorified aspect of the product profession, and that glorification casts a shadow on an equally important aspect: controlling the costs of the products they manage. People who make sure that a product is profitable (product managers and such) need to take technical debt into account if they want to have a full picture of how their product makes and spends money. In the end, they are responsible for the profitability of the product, and that equation includes engineers' time.

Sometimes you can make a product more profitable by optimizing the current spend. By far the most common approach product/business people have about technical debt is "The techies will handle it, and we don't understand why it takes them so much time to deliver new features, everything is slower and slower." That is not comprehensive product management. It's necessary to partner up with "techies" to understand what's really going on.

Like it or not, technical debt is a part of any product, and can't be avoided. Every time you write a line of code you create a possible technical debt, not because the engineers are bad, but because today's requirements are different than tomorrow's, and nobody can predict with 100% certainty what will be optimal in the future. Of course, experienced people have more chances to predict it a bit further in the future, but it's impossible to completely avoid technical debt.

Symptoms of technical debt

Here are some frequently seen signals of technical debt:

• More and more bug fixes.
• Increases in the time engineers spend on maintenance.
• Slower and slower releases.
• You hear things like "it's complicated" and "we're late because we discovered new things" more and more often.
• Engineers complain about toil and cumbersome development.
• Engineers push back when you want to introduce new features because "it's complex".
• The team's health and mood changes because of tech debt.
• The product becomes unstable.
• More and more fixes are needed outside usual hours.
• ...and many other symptoms.

Some of these signals are obvious, while others are more subjective and require a measuring/tracking method. For example, lengthened release schedules. If we don't have data in place, we only have subjective opinions which can differ from person to person. That can be sufficient in some cases, but nothing can replace data. It pays to think about and implement a small number of measures for each team or product: some simple "data signal" which you can interpret to detect technical debt. One of these signals can be, for example, the frequency of release. Another idea requires more effort to measure: a relative proportion of time spent on developing new things as compared to the maintenance of an existing system.

How to handle technical debt: Product-Engineering partnership

How do we decide what technical debt is important? We need good cost-benefit analysis and some measure of impact: "how much will it cost us if we don't take care of this?" This estimation is often vague because nobody can predict the future. That makes it even more important to approach the impact with as much business mindset as possible.

Who can make this cost-benefit analysis? The topic is technical debt, so technical people must be involved. But engineers aren't looking forward to doing "boring paperwork" like writing good cost-benefit analysis, investigating the effects of various technical debt, and helping in prioritization. Some engineers are there to code and solve technical puzzles, looking at modern technologies and new shiny things. But senior engineers understand that building software is about running a business and they know how to work with product people to help optimize for the most profit.

It turns out that the line between product managers and engineers has to be blurry if we want to keep technical debt under control. We can observe technical debt as a part of the product that needs changes. Like any feature that brings value, removing tech debt also brings value. That implies prioritization. And for prioritization, we need to know the value that resolving technical debt will create.

It's also important to remember that not all technical debt is relevant at the same time, so we need to re-prioritize and re-estimate technical debt from time to time.

One possible process for handling technical debt

Here is one of the possible solutions for handling technical debt.

1. Have periodic lightweight sessions with engineers to collect information about technical debt.
2. Evaluate the effect and write "lightweight business cases" about the technical issues team finds most relevant.
3. Select the most beneficial technical issues to solve and plan a work breakdown (milestones, architectural diagrams, dependencies, etc.). Treat it like you would treat any other initiative.
4. Compare the benefit of the selected technical issues against the new features. One approach is to determine how much time you want to dedicate to handling the technical debt issues. Sometimes it's 20% of the roadmap time, sometimes 10% and sometimes the best decision is to dedicate more than 50% if there is an approaching deadline or another impactful event.
5. Put it on the roadmap, but focus only on a few topics at the time.

Let's deep dive into some of these steps.

Step 1: periodic lightweight sessions. It's important that the sessions to identify tech debt don't just happen once, or randomly. They should be periodic, depending on how much of a problem it is. For example, once per month, quarterly, or even every 2 weeks. You can keep a document of the technical debt issues and give them scores based on impact, risk, urgency, business value, wasted time per month, etc. Based on that you can prioritize as you see fit and pick the issues that are the most relevant. The next important thing is keeping these sessions light. They should be very simple and easy to attend; otherwise, people won't expose the issues. Step 1 is just about collecting the issues.

Step 2: Once you have the list of technical issues and their priorities, you can easily pick one, or the few most relevant ones. Before proceeding, it's time for a business analysis. The purpose of this step is to make sure that the overall effort to resolve this issue is not bigger than the benefit fixing it would bring. For example, if a particular technical debt issue causes 10 minutes a month of additional work, and fixing it would cost 8 hours, it will take 4 years to break even (4 years is 48 months, which is 480 minutes of wasted time, or 8 hours of fixing). It is very likely that you can invest these 8 hours in something that will return that investment much faster.

Step 3: Once the business cases are done, it is easy to select the most beneficial issues to handle. Some issues are easy to understand and don't need a lot of technical analysis or refinement. But some technical debt issues can be the size of epics or even bigger. In these cases, it is useful to treat them as any other initiative, and make proper architectural decisions and set up a work breakdown structure (milestones), etc.

Step 4: Now it's time to put the technical debt into the big picture - the complete business context of the system you are working on. I have seen the entire spectrum of prioritization in my career. From systems that need only occasional refactoring of some components (5% of the roadmap) to systems that, at that particular moment, required 80% of the roadmap dedicated to fixing the burning issues. If we didn't handle those, the product would cease to work very soon. Regardless of the system you decide to use, resolving the technical debt issues becomes a part of the planned work.

Two more things

Don't work on too many issues at once. It's very rare that technical debt issues are not connected in some way, and if they are, the complexity multiplies. For example, the availability of people who can work on a particular issue, or architectural changes and integrations that make it more complex to work on more issues in parallel, etc. It's helpful if big transformative projects can be broken down into smaller, autonomous chunks. I always ask "Can we break this idea into several milestones, so each milestone delivers some value, and if we decide to stop the big project after some milestone, will the value we provided remain?"

Find a way to celebrate the success of removing technical debt and connect that result with the company goals. This will keep everyone involved motivated and the business happy. It's about making partnerships, and when you are able to present the data about how your team contributed to the overall success, those partnerships will come about naturally. The best way to do this is to keep good documentation about handling technical debt. One trick for keeping the momentum rolling is to start small. Choose a smaller milestone or project to prove to everyone (including yourself) that this system works. And change the system if it doesn't work well for you.

Summary

• Technical debt is the unfitness of code written in the past to meet today's needs. Even high-quality code can become technical debt if the context changes in the future. It is a part of any product and there is no guarantee we can completely avoid it.
• Left unresolved, technical debt will slow development down by consuming more and more engineering time and adding cognitive load. Some technical debt issues can even halt the development of products.
• Some symptoms of technical debt rising are unstable systems, more and more engineering time spent on maintenance, and slower releases.
• To handle technical debt, treat it as a part of the product and involve both engineers and product people in analysis and prioritization. A mindset of "the techies will handle it" doesn't contribute to the effective handling of technical debt.
• Handling technical debt issues should be a part of the planned work.
• Make your own process, but beware that the process should be lightweight. Document it and find a way to celebrate the removal of technical debt by connecting it to the business strategy.