Tithe.ly Engineering

If there was one, key takeaway from the many business and finance courses I took in college, it was an understanding of the difference between good debt and bad debt. Whole accounting techniques exist to codify and quantized various debt structures within a business. Whether it is Depreciation, Depletion, or Amortization, a business has a set of fundamental tools to use for planning and structuring their spend when it comes to debt.

In the technology world, there is a form of debt, Tech Debt, that has very few structured techniques for addressing. If tech debt is left too long, organizations can become overwhelmed and unsure where to start addressing the root cause. Much like financial debt, this can have long-term effects to the financial health, and overall longevity, of a business.

What is Technical Debt

Much like financial debt, by incurring certain amounts of technical debt, a team can achieve long term goals quickly. The intent behind this practice is to go back and clean up sloppy design later. Either by refactoring or redesigning things that were intentionally done poorly in the interest of time, an engineer can design and implement a solution quickly but still, eventually, create a solution that is sustainable in the long term. But this only happens if the engineer makes good on the promise and pays off the debt, actually returning to a hastily coded project to perform refactoring. Broadly, technical debt can be classified into three categories: Deliberate, Accidental, and Mounting Complexity.

Deliberate

Like I mentioned before, deliberate technical debt usually arises from an engineer choosing to do something the “wrong” way, architecting something for a specific use case instead of architecting for future sustainability. This can take many forms but in the infrastructure world it often manifests as deploying single solutions per application instead of taking a holistic approach to the design, considering your technology stack as an ecosystem rather than a set of disconnected islands. Industry backlash to this kind of intentional design philosophy has been growing in recent years. In an article for the Harvard Business Review, Hemant Taneja outlines some key reasons to move away from this methodology as innovations and changes in data-security make this deliberate technical debt dangerous for modern companies.

Accidental

Accidental technical debt is often the easiest debt for teams to incur. There is a whole world of technology and solutions out there and no one person can know it all. What this translates to, especially in the infrastructure world, is engineers having only a base level of knowledge or expertise around their own implementations. Just considering the complexity of hosted solutions like Amazon Web Services or Microsoft Azure, the catalogs on both these solutions are staggering and ever evolving. For businesses and engineers deploying solutions from those catalogs, there are bound to be gaps in methodology or architecting that add to an organization’s technical debt.

The nice thing about accidental tech debt is it is often resolved by time. As an engineer works more with the system, deploying more and more services from a given solution, their experience and knowledge with that solution grow. They might never become an expert, but the more exposure, the more the engineer is able to rapidly adapt that solution to meet the needs of their organization.

Mounting Complexity

The final broad categorization, and the one which confronted the DevOps team here at Tithe.ly, is mounting complexity. The foundational model of OS level virtualization can trace its roots back to the first implementation of ‘chroot’ (literally a ‘change root’ system call) first introduced for V7 Unix in 1983. From these humble beginnings, the complexities of virtualization grew until, In 2013, Docker disrupted the virtualization market by introducing the management engine to host containerized applications at the OS level, removing the overhead incurred by virtual machine hypervisors and individual operating systems.

At Tithe.ly, this meant we could launch a scalable Giving platform that could grow at the speed of our business, oftentimes a dizzying pace. In time, as new products were developed by our engineers, new containers were needed. This necessitated the creation of a new fleet of containers to house the new applications. Another layer of complexity was added. In time, the need for ongoing development, both from a functionality and code perspective necessitated the implementation of a way to dynamically build and push images out to the container ecosystem––another layer of complexity. As time and business needs changed, so too did the stress on the deployment phase of the infrastructure.

Addressing Infrastructure Technical Debt at Tithe.ly

By the time I joined the DevOps team in mid 2020, it was clear that the deployment process was not keeping pace with the changing needs of the infrastructure. Something needed to be done to address the problem. To return to the financial metaphor, interest needed to be paid on the technical debt behind infrastructure deployments. Working through the backlog, it was clear that the DevOps team was in a unique position to address this concern.

The Plan for the Future

By working to vet new methodologies, the team has identified a path forward that will allow for both scalability as well as portability to new services and applications. We’re currently in the process of changing this system to dynamically address deployments to fit a more boiler-plate model, allowing teams to plug in and begin consuming infrastructure much faster than before.

We were lucky, simply adding an additional engineer, allowed the technical debt to come to the forefront. Spreading out the workload meant that mental energy could be spent on the solution, rather than continuing to incur the debt to keep the pace of the rapidly evolving business. The DevOps team was able to determine a path forward that will allow for a solution to scale at pace.

Conclusion

Managing technical debt at an organizational level is key to maintaining a rapid pace of development. Not all debt is bad, and not all technical debt needs to result in lost nights and weekends keeping the business running. There will always be a need to evaluate current practices and methodologies; it’s up to individual engineers to recognize when they’ve reached a break-point and begin addressing their own technical debt. The DevOps team at Tithe.ly is still learning, but we’re excited to push into new grounds; it’s business needs like this that keep an engineer agile.

Further reading on Technical Debt: Technical Debt - Martin Fowler Managing Technical Debt - Allan Stewart The Perfect Process to Manage Technical Debt - Alex Omeyer