I came across an article on Firstround called ‘Forget Tech Debt…Here’s How to Build Technical Wealth.’ Andrea Goulet introduces us to some of the practices her consultancy uses to dive headfirst into legacy codebases, improve their design, and make them more maintainable.

The article covers a few practices at a high level: test-driven development, continuous integration, culture changes. It is also important to note the context in which these changes are made.

My work has included a lot of enterprise companies: companies whose technical choices go back 40 or 50 years, companies whose systems must handle millions of customers and billions of dollars of transactions in a year. These software systems handle immense amounts of data. The structure and content of the underlying data is frequently the limiting reagent for nice software.

This is because we want the end product to be nice for end users. The end users want to see the data in a readable, organized format. Our code is the thing that transforms the data as it exists into the data as it appears for the end user. The more transformation that requires, the more complex the code will be regardless of how it is refactored.

What kinds of data make it possible to write simple code? Best case scenario and excluding other limiting factors, one network call provides the exact attributes we need to show the users what they want to see.

In the real world, we can’t exclude other limiting factors. One important limiting factor is speed. The more data you try to send over the internet, the longer it takes. And remember, we could be talking about a LOT of data. So for efficiency, services will try not to send extra things. This is the origin of the list-detail API pattern. A list endpoint will send you some summary info on each of your rows of data, and to get more details about a given row you have to hit a different endpoint to fetch more info on that specific row.

Code has to be more complex if, say, you want to show a list of all your rows, but you want each list item to include a piece of data that doesn’t come back in the list call – only in the detail call. Now the app has to make that detail call for every row item, and it cannot start until the original list call comes back. So now we need to provide some indication on the UI that the data is loading. If it’s going to take a really long time, we employ tricks to try to preload the data or load it in the background while the user does something else. Code doing more stuff = complexity.

That’s a basic example of how data that doesn’t match up with user needs can result in more complex code. But this example just comes from how the data is organized. The content of the data presents further challenges.

Usually legacy systems that handle giant, historied datasets are making up for all kinds of data issues. Data might be incomplete: some attributes in some rows are missing. So we have to take that into account by ignoring those rows, or setting a default value, or trying to guess what goes there based on the values in other fields. Sometimes data is poorly formatted or corrupted, and again we have to either ignore or try to extract. Sometimes data from multiple sources might come in different formats and require standardization. Or it might contain conflicting information. The more of these issues our code must address, the more complex it will be.

These issues come from data entry mistakes, sensor errors, miscommunications, and organizational restructuring. They force apps to be more complex, and they compromise the accuracy of the information provided to the end user.

So what can we do? How can we prevent some of these data issues, and can any of them be fixed once they exist?

That’s a tough question, and the answer is not always inspiring. But there are a few things we can do. More on that in Part 2.