From Agile to Burden. The Impact of Agile on Your Growing Tech Debt!

When we think back to the Waterfall and Iterative development methodologies taught 20 years ago, the terminology of “Tech Debt” didn’t exist. We worked on architectural improvements, but because our engineering process required much more thorough up front design considerations, the final product ( though slow to develop ) usually was much cleaner in terms of its design. 

The Waterfall and Iterative models had a different challenge. The development was not done in Sprints, the best we had was an Iterative process which consisted of small Waterfalls. And both of these models fell short on 2 primary things.

First, building the right solution. By the time everything was built, it was likely no longer what was needed. The way people developed things – features were almost always not available until the entire product was put together and transferred to the Test Phase.

Second, the development was utterly slow and expensive, with rework causing massive destruction in plans and withering budgets.

Agile came out as a savior to address both of these. With Agile, we iterate in very small functional cycles aimed to achieve small specific attainable goals. This allowed us to create and validate what the team built was exactly as desired and eliminate waste of building something unnecessary. Our development is poised to move faster by producing increments that are usable and available for end users to try and validate.

So how did Tech Debt become an Issue?

As the Teams went on to work in an Agile way, the approach changed. The typical Agile Team looks at the goal and has the following conversation:

— Team  – “Sure, we’ll get it done in a short amount of time. We’ll do it quickly so we’ll need to make it work so we can gain feedback, then refactor and tune for performance.” 

— Product – “Yup, understood, we don’t know yet how it will work and if that’s exactly what we need or how we’ll need to tweak this capability, so yes, let’s do a first pass so we can play around with the new capability. Once we have it, we’ll decide on the next steps.”

— The Team finally delivers and says “It’s done, it works, but it’s not finished. It works on scotch tape, glue, rubber bands, and magic dust. We’ll need to go back to tighten it up and make improvements.”

— Product says “sure, I promise, we’ll find the time to get it done.”

Product sees the Demo. Excited, runs to the leadership ( and maybe users ) and shows off the next best thing. Gets awesome reviews, comes back and says: 

— “Great job Team. We discussed it and now that we have what we need,  our next goal is that other thing…”

— “Wait, we’re not done”, says the Team

— “That’s ok, we’ll do it later,” says Product, “This other thing we have to build is much more important.”

In this almost Shakespearean Play, the tragedy triggered the que for Tech Debt to enter.

Enters the Tech Debt

The concept of Tech Debt as “something you have to repay” is an interesting one. Engineers are not finance oriented, so they see this more like a “Tech Burden”. Burden on the system that performs poorly, burden of poor quality leading to large volume of new bugs, burden on engineers who may need to support incomplete functionality, burden to develop new functionality in a hard to understand code base, burden to carry code that is difficult to test. This burden needs to be lifted and to be lifted it needs to be eliminated.

Agile gave the team the speed, but Agile also made the discipline of following proper foundational principles in software engineering so much harder to maintain. Where Waterfall’s Implementation Phase didn’t start until the Design Phase was done and Implementation Phase had to match the designs to be complete – Agile does not have these solid barriers. The way Agile works is by making quick leaps to show what’s possible and then requires the courage to stay the course to maintain proper engineering discipline by taking on work to tighten up the design, architecture, and code base.

This is exactly where Agile’s flexible structure felt short – it left important things up to the people, assuming the people know better. Unfortunately proper human behaviors, especially in bigger companies were quite overestimated. What sounded good on paper and presentations regarding Product and Engineering Agility didn’t account for organizational subordination and pressure.

Organizational Influence

Organizations follow a hierarchical leadership structure and as such, those in higher chains maintain a higher seniority level with the ability to deliver order-like directions to lower levels of the hierarchy. Anyone higher in command sees a working button and the typical reaction is that  it’s perceived as a complete working product. Convincing someone that it is still a work in progress and requires a few more iterations is a test of patience, knowledge and understanding of how Agile works. Leadership on the other hand, often wants quick wins. And the more wins Teams will give them, they will want even more.

Therefore, the effort to convince and maintain quality discipline to eliminate Tech Debt is hard to keep. And if the organization wants “results”, it’s hard to gain recognition of accomplishment that “the button works better” especially because the button has already been built and not been much of an issue. 

And thus, gaining faster speed and establishing an Agile way to experiment to inspect and adapt the product, the focus on the quality and Technical Burden is shunned to the bottom of the backlog – invisible, ignored, and constantly compounding the debt-like interest by slowing down development, impacting quality, and costs.

The way out

Agile is not a mindset, it’s not a framework, it’s not a process. Agile is a discipline – a technique of doing things. 

Why do we need technique? Just like in sports, quality technique improves the chances of achieving goals. Not that you can’t achieve those goals without a technique, but technique makes those chances much higher and more certain. 

Agile is the technique of software development and to do it well and to do it right, there needs to be discipline to handle resistance from leadership to switch priorities to new goals by disregarding incomplete work, to plan addressing technical debt on a continuous basis, and giving the team an opportunity to self-manage.

If the team is struggling to bubble up the Tech Debt work to the top of the backlog, the best way is to address this is to provide the leadership team with a visualization displaying the amount of tech debt accrued, and the burden the unaddressed tech debt has on the engineering team – and a metric that provides evidence of how much the Tech Debt is slowing Team down. And most importantly, continue educating the cost of Tech Debt and its accruing interest.