Win a copy of Testing JavaScript Applications this week in the HTML Pages with CSS and JavaScript forum!

Adam Tornhill

+ Follow
since Dec 08, 2015
Cows and Likes
Total received
In last 30 days
Total given
Total received
Received in last 30 days
Total given
Given in last 30 days
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Adam Tornhill

Thanks for all questions, and thanks a lot for having me here!
Congratulations to the winners -- I hope you enjoy the book.
Hi Germán,

German Gonzalez-Morris wrote:
From book's point of view Behavioral Code Analysis deliverable are patterns to be implemented (management) to paid the technical debt or have some coding related patterns as well ?

I'm not sure I understand the question, so I might get this wrong. But let me try to clarify what the book is about. Software Design X-Rays is a practical book targeting experienced developers, tech leads, architects, and technical managers. The book provides a set of techniques that gives you insights into different aspects of your codebase. Using the techniques you can uncover and prioritize both technical debt and potential organizational issues. To make the findings actionable, the book also includes a set of patterns and ideas on how to address the issues you might find. Most of these can be addressed by developers while others have a more natural management focus. The key point is that the whole organization (managers, testers, developers, etc) can get a shared understanding of what their system looks like and where the potential pain points might be.

All techniques and analyses are demonstrated on real-world systems of different scales, and we do look at the source code too and come up with specific refactoring suggestions. The case studies are open source so you can interact with them and inspect the results yourself.
Hi Junilu,

One example is the Splinter pattern from chapter 4: "Pay Off Your Technical Debt". The intent of the Splinter pattern is to provide a structured way to break up hotspots into manageable pieces that can be divided among several developers to work on, rather than having a group of developers work on one large piece of code. Breaking up large modules that start to become unmanageable isn't exactly a revolutionary idea; It's something experienced developers do all the time. What sets the Splinter pattern apart is that it approaches the steps of refactorings from a social perspective. Hence, the initial goal of the pattern isn't to solve the technical problems, but rather transform the existing code into a new context where those problems can be solved with a minimized risk of coordination and code conflicts between multiple developers and teams.

Another principle that I think is followed implicitly is to organize code according to its age. Following this principle lets us evolve our systems toward increased development stability. The main reason we don't discuss the age dimension of code is because age is invisible in the code itself. However, our version-control data knows and can provide guidance for the package-level refactorings that I cover in chapter 5, "The Principles of Code Age".

One of the most important principles in the book is that "There Is No Such Thing as Just Test Code". In my day job I analyze lots of codebases from different organizations, and some of the worst hotspots and design issues tend to be in automated tests. This is a dangerous fallacy because from a maintenance perspective our test code is at least as important as our application code.

Finally, there are also several principles around change coupling. My personal favorite is to combine change coupling measures with copy-paste detectors. Most systems contain lots of duplicated code. An important question in that context is whether we should extract shared abstractions to eliminate the duplication or if the code is fine as is. This is a hard problem: just because two pieces of code look similar, that doesn't mean they should share a common abstraction. Experienced developers with deep domain knowledge often know when to draw this distinction. And the change coupling measures can help us further by pointing to true violations of the Don't Repeat Yourself (DRY) principle.

Junilu Lacar wrote:
One topic heading in Chapter 9 caught my attention: Measuring Technical Sprawl

Can you please explain what you mean by "technical sprawl" and how metrics around it can help address technical debt? Thanks.

With "technical sprawl" I'm referring to when a codebase uses different libraries, frameworks, and infrastructures. Sometimes it might be good to chose different tools for different jobs. In other situations, adding more technology variations don't result in any obvious benefit to the business but rather to increased costs and a reduction in consistency.

Chapter 9 in the book discusses technical sprawl from the point of microservice architectures. The reason I chose that context was because one of the early selling points behind microservices was that each team could choose their own programming language for a service. In practice, unrestricted technology adaptation leads to slower onboarding, it might complicate the deployment, and also makes it harder to collaborate and share knowledge between teams. So technical sprawl is always a trade-off and IMO there should be a clear benefit to adopting one more framework/technology/language; All dependencies come at a cost.

But I also use the same measure of technical sprawl for legacy systems. For example, it's fairly common to have an old core that's written in something like Visual Basic and a newer part of the system implemented in C#. The idea is often to re-implement features from the old part in a new language of choice and, thus, reduce the legacy portion. In such situations I use the technical sprawl measure to highlight progress. If the approach is successful, we should expect a reduction in technical sprawl and an increase in implementation consistency over time.
Hi Junilu,

Thanks for your question and your interesting insights. I appreciate it.

Junilu Lacar wrote:Hi Adam,
That is, you go into debt so that you can spend more time getting experience with working software. You pay back that debt by feeding the understanding that you gained from that experience back into the software by spending the time to refactor it to reflect your current understanding. If you don't pay back that debt, then your software will become progressively more incomprehensible and harder to work with, which again costs time.

I do find the original intent and metaphor valuable and I use them myself in my daily work. However, in my experience, the industry in general uses technical debt mostly to describe what Martin Fowler calls Reckless Debt. I find this interesting as it’s hard to distinguish between deliberate technical debt and reckless debt in retrospect. So in my book I wrote that "the resulting context is the same: we face code that isn’t up to par and we need to do something about it".

Junilu Lacar wrote:Your view on the time dimension of technical debt, however, takes a different tack. I think it's a valid perspective and a useful one. I'm just curious about the path that led you to that realization.

My perspective is largely influenced by my psychology background. I spent 6 years on my psychology degree and in parallell to that I worked as a software consultant, which gave me plenty of opportunities to relate the two fields.

As a consultant, I was often hired as a tech lead or architect. One of my recurring tasks was to try to shorten lead times for new features and control potential risks in the project. I found that incredibly hard to do, and more than once I made the wrong decisions.

Back then I relied heavily on static analysis tools and frequent conversations with the developers. While both tools and conversations are valuable, they also have their limitations. For example, static analysis might point out that a certain piece of code is complex. But it doesn't tell us if that complexity matters or not; If that overly complex piece of code hasn't been modified in 3 months time, is that really where we should invest time in improvements? And conversations with our peers help us get insights, but they also tend to get biased by the recent problems that are fresh in our heads. So I wanted to find more objective data.

At the same time as I wrestled with these problems, I was in the middle of a course on forensic psychology. As part of the course I did a geographical offender profile on a series of crimes in my hometown. The basic idea of a geographical offender profile is that each crime scene, their geographic location, contains information about the criminal. The rest is a matter of mathematical weighting combined with our knowledge of human behavior. When I learned about this I kind of though: "Wow, what if we could do this for software?". I then realized that we can: we have behavioral information in our version-control system.

From there it was a matter of digging deeper. There's a lot of good empirical research on repository mining, but none of it has really made it into mainstream software development. So I took it as a mission to try to bridge that gap between academia and industry by making the research findings accessible and actionable while adding my own contributions on top.

Today, I look at each piece of code from the perspective of time. If I know that I have a low interest rate on a piece of code, then I can take on a bit more debt there which lets me move forward fast with a calculated risk. If on the other hand the interest rate is high in a particular module, then I might start by paying off some debt before I extend or modify the functionality. To me, code doesn't have to be perfect. But the best code should be in the parts where we work the most.

Michael Angstadt wrote:I maintain a handful of open source projects in my spare time.  I am the sole developer on these projects (I very rarely get pull requests or patches).  How well do the principles in your book apply to projects that only have a single person working on them?  Are the principles in your book more centered around large teams?
Michael Angstadt

The principles in the book become more important with larger teams and systems. Somewhere around just 5-7 developers, social and organizational factors tend to become just as important as technical aspects. This is clearly not the case in your situation. For example, you're likely to have a high system mastery as you've written all code by yourself, and you don't need to consider aspects like coordination or loss of knowledge due to former contributors.

That said, I do use these techniques on my own side projects too. The value isn't as big as on larger projects, but the analysis techniques might still provide some insights that I use to guide refactorings:

* I use hotspots to get a visual overview of my codebase. This helps by giving me a different perspective as I get to view the code from an evolutionary perspective. It's quite easy to spot modules that start to grow out of control so that we can act on it.
* I'm also interested in long-term complexity trends. If I find a significant hotspot, is it code that continues to accumulate complexity and is likely to become worse in the future? A complexity trend answers that question.
* Finally, and perhaps most important, I look at the temporal dimension to identify change coupling in my code. That is, are there certain components/files/functions that I regularly have to modify together? If yes, did I expect the coupling or is it a surprising finding?

We developers seem to lack an intuition for change coupling and, hence, such an analysis will often provide insights about implicit dependencies that we cannot see in the code itself. I use the technique all the time to identify units to combine, split, or even re-design.

Junilu Lacar wrote:Welcome, Adam!

One thing in your bio blurb stood out for me: your interest in martial arts. Any art in particular?

Several arts, actually. I started out with Karate as a kid and then switched to Taekwondo where I'm a black belt. I love the sport and practiced it for 10 years until my knees gave up on me. After that I did some kickboxing and I'm now considering to start with regular boxing. But this time it's going to be for exercise rather than competition.

Junilu Lacar wrote:
I practice Aikido and a lot of the philosophy has rubbed off on my own Agile journey and practice. I'd bet you have had similar experiences.

Yes, I agree. That's definitely one part.

Thinking of it, I think the most practical skill that translates to what I do in software was that I started to get used to public speaking. I used to train a couple of groups in Taekwondo, so that definitely helped me get used to stand in front of an audience and explain something. Admittedly it's easier in martial arts as there's a clear hierarchy so no one will question you
Hi Darren,

Thanks for your questions! Let me try to clarify what the book is about:

darren hartford wrote:
q1: Is there a bit more you could share, is this about the developers being good at certain 'design patterns' versus others and creating technical debt by keep doing it 'the same way', or is there a different perspective/point of view?

The underlaying theme in the book is that there's more to code than code. Just because a certain module is complex, that doesn't mean its technical debt is high. It's only technical debt if we need to pay interest on it. Since interest is a function of time, we learn how to add a time dimension to our codebase by mining our version-control system, which knows exactly when and how often each piece of code was changed. By using version-control data, we can then identify hotspots that I define as "complex code that we need to work with often". Used that way, I view version-control as a behavioral data source that we can tap into for information that can guide our decisions.

But that's only the starting point. Once we embrace version-control data, we get access to lots of information that is invisible in the code itself. For example, most of the book focuses on how multiple developers and teams impact code quality and feature implementation efficiency. As an example, you might have heard about Conway's Law; Using behavioral code analysis, we can start to measure and visualize how well our current architecture supports the way our system evolves, as well as how well-aligned our architecture and organization are.

darren hartford wrote:
q2: Being able to measure is what makes it easier to see progress.  Are there automated ways to help measure where this approach may be useful (such as using a common software analysis tools such as sonarqube)?

When I started out with these techniques 7-8 years ago, there weren't any tools available that could do the analyses I wanted to do. So I started to write my own tools. I open sourced my first tool suite as Code Maat which might be a good starting point. These days I spend most of my time on CodeScene, which is a commercial product but free for open source projects.

Campbell Ritchie wrote:

Adam Tornhill wrote:. . . Your Code as a Crime Scene . . .

Have you been here before to promote that book? I can't remember, but there is something familiar about the title.

Yes, that's correct. I got a wonderful review by Coderanch and I did a book promotion too, approximately 2-3 years ago.
Thanks for the warm welcome! I'm looking forward to answer any questions you might have around the topics in Software Design X-Rays such as:

- How to identify and prioritize technical debt,
- the social side of code and how multiple developers and teams might affect code quality,
- why we need a time dimension into our code and how it helps us make better decisions,
- and much more.

A little bit about me:
Adam Tornhill is a programmer who combines degrees in engineering and psychology. He's the founder of Empear where he designs tools for software analysis. He's also the author of Software Design X-Rays, Your Code as a Crime Scene, Lisp for the Web, and Patterns in C. Adam's other interests include modern history, music, and martial arts.
Thanks for having me here and thanks to everyone for all the interesting questions! This was fun to do!
4 years ago
Hi Tom,

Yes, I'm quite familiar with that problem myself (I used to be a consultant before I started Empear).

One thing that I've learned is that managers, most of the time, are pretty much like the rest of the population That means, they want an ROI on the things they invest in and they want to minimize risk. I've found that it makes a huge difference when you can support suggested improvements with data. In addition, we also need to show the consequences of ignoring those suggested improvements.

So what I suggest is to present a hotspot map (Chapter 4) together with complexity trends (Chapter 6) of the top hotspots. The nice thing here is that you now have supportive evidence to the suggested improvements, which minimizes risk and has a high probability of leading to real savings since the data is based on how we developers actually work with the code. The complexity trends are important too since it's easy to understand the cost and risk of maintaining code that grows along the infamous hockey stick curve.

I think that's one of the big wins with these kind of technologies; they make deep technical issues accessible to a non-technical audience. It's something I see in my workshops where I occasionally have participants that don't even code.

Hope this helps.
4 years ago
Hi Paul,

My strategy has shifted over the years. These days I tend to start by focusing on understanding the problem domain first. I've found that a basic understanding of the problem the application tries to solve gives me a framework and foundation to explore the solution.

Once I understand the basics of the domain its time to dive into the code. This is where the strategies in Your Code as a Crime Scene help; I typically start with a hotspot analysis followed by a Sum of Coupling analysis. These two analyses are particular good at identifying the central components that I most probably will have to meet one day (often sooner than later).

When I dive into the code I sometimes make small refactorings just to understand the details. Often I just revert those changes since it's the iterative act of working with the code that helps me build knowledge. I've found that this strategy works really well for me.

There's one more technique that I use. If it's a large codebase I also generate a knowledge map and use it as a communication aid to know who to ask about the details.
4 years ago
Hi Arvi,

I'm not sure that I understand your distinction between design and build, so please correct me if I get it wrong.

My take is that it's virtually impossible to separate design from coding. I have a discussion on this topic in the book. To summarize, basically programming is problem solving. And problem solving requires a large degree of experimentation. That means programming is inherently iterative, just like all human problem solving is. We learn by doing and that learning results in better designs.

The only way I know to get early feedback on a proposed design is to prototype it. The analyses in the book will help your learning by pointing out the areas of the system where the design would need to improve. However, they do that by analyzing the evolution of the actual code.
4 years ago
Hi Will,

That's something I see myself quite often in the codebases that I analyze; We find a problem in some module, refactor the code and a few months later the same module has degraded again. The reason is that the root cause of the original problem wasn't addressed with the refactoring (often an incomplete understanding of the problem domain).

What I recommend is to incorporate the techniques from the book in your daily workflow. For example, these days a lot of teams do some variation on short daily meetings or retrospectives. Why not gather around a hotspot map and see how the features you work on right now impact the overall system? That way you get a chance to react early and investigate potential problems deeper.

I discuss the idea of an evolutionary storyboard in the book, but it's only a minor part of it. That said, there's a lot of value in early detection of potential design problems. Since I finished the book I've continued to evolve the analyses and I've focused a lot on automating these kind of early warning mechanisms. I plan to write it up in some form.

Hope this answered your question.

4 years ago