• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Bear Bibeault
  • Ron McLeod
  • Jeanne Boyarsky
  • Paul Clapham
Sheriffs:
  • Tim Cooke
  • Liutauras Vilda
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • fred rosenberger
  • salvin francis
Bartenders:
  • Piet Souris
  • Frits Walraven
  • Carey Brown

Righting Software: the case against unit tests

 
Sheriff
Posts: 15815
264
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Juval,

Let me start of by saying that I found the book sample on Software Design Decomposition very interesting and a number of things resonated with me based on my experience.

However, one paragraph under the heading of "Testability and Design" got me wondering:

Juval Löwy wrote:The sad reality is that unit testing is borderline useless. While unit testing is an essential part of testing, it cannot really test a system. ... The inefficacy of unit testing is not a new observation and has been demonstrated across thousands of well-measured systems


You seem to contradict yourself in these two successive sentences. First, you say unit testing is borderline useless but immediately after that say that it is an essential part of testing. You then conclude that paragraph by repeating the assertion about the inefficacy of unit testing.

I think the paragraph goes into considerable length to explain why unit testing is ineffective (and I don't even disagree with some of the points you made) but in what way or context are unit tests then also an essential part of testing? I am a strident proponent for unit testing by developers and I don't think most developers do it often or well enough. My experience is that unit testing is a valuable and essential part of the detailed design process, especially if you're doing Test-Driven Development (TDD). I wonder if you have a different perspective from experience and if so, what was the context?
 
Author
Posts: 12
10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Junilu,

I am not against unit testing. As you have stated, I consider them absolutely essential.
But essential does not mean it adds value. It could even be essential and important, but still not add much value. Brushing teeth in the morning is essential. But you do not take too much pride in doing that task?
Breathing is essential and important, but where is the added value?
Some activities you just take for granted that they are done, and done well, and you focus on activities that add value.

Even if all components passed unit testing 100% it means nothing to the quality and correctness of the system.
The book explains why unit testing cannot really test a system (the defects are in-between the components), and why developers focus on unit testing, because it is the only kind of testing that is possible in a functionally-decomposed system (searching under the street light).
If you have a giant ball of mud (the result of functional decomposition), then unit testing is the only practically possible testing, so developers focus on that, instead of what they should have tested but cannot.
This does not mean you should not test your units. There is no contradiction. Unit testing is essential, must be done, and also borderline useless (passing it means nothing).

Juval.
 
Junilu Lacar
Sheriff
Posts: 15815
264
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Juval Lowy wrote:Breathing is essential and important, but where is the added value? ... Brushing teeth in the morning is essential. But you do not take too much pride in doing that task?


Uhm... I'd die if I stopped breathing, so I think there's lots of value added by that activity, at least to me there is  By the same token, are you saying then that eating is essential but adds little value? And I don't see the connection between taking pride in something and adding value either. Frankly, I don't quite get your line of reasoning here.

Even if all components passed unit testing 100% it means nothing to the quality and correctness of the system.


I can fully agree with this. I have seen many instances where unit tests were written just to satisfy some requirement to report a high degree of test coverage to a manager and yet the designs are still poor and in fact, the gamed high test coverage numbers gave a false sense of security and often make quality-related issues worse.

 
Junilu Lacar
Sheriff
Posts: 15815
264
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

(The book explains why) unit testing cannot really test a system (the defects are in-between the components)


Again, I can agree with this sentiment.

why developers focus on unit testing, because it is the only kind of testing that is possible in a functionally-decomposed system (searching under the street light).


This is where my mileage seems to vary from yours. My experience is that most developers don't focus too much or put much effort on unit testing, relying instead on a separate team to do (usually manual) end-to-end testing. This is often also dictated by the kind of designs they tend to produce which are not testable at a unit level so they can at least verify the quality and correctness at the component level. In my experience, properly-performed unit testing and integration testing (mostly automated) by developers does add significant value to a product/project.

I think we're diverging on basic assumptions here. When I say "unit testing does add value" it's in the context of tests being used to drive design thinking up and down the system architecture and across different parts of the design. Not only that, good tests can help developers quickly detect regressions, verify integration points, and facilitate the refactoring of designs and architecture to be able to adapt to changing requirements in the future. Astute developers can also use tests to detect "hot spots" in their designs, the places that change the most or are most troublesome to change. Well-designed tests also provide a safety net for refactoring those hot spots so you can more easily isolate and encapsulate them, maybe even in accordance with some of the guidance you give in your book.

To me, developer testing, when done properly and diligently, does add value but it has to be done in concert with other engineering disciplines as well.
 
Greenhorn
Posts: 19
1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think the question is why are unit tests considered essential. What makes them important to write if they provide very little value as actual tests in terms of finding bugs? Why are unit tests preferable to  behavioral tests and integration tests?
I don't know if we should really call unit tests "tests," because it implies that they exercise the code. I've found dozens of bugs by manual testing and reading code, and probably less than ten from unit tests.
I consider unit tests important because they implicitly ask developers to write simple, short code that has a small number of readily understandable dependencies. If it's difficult or cumbersome to write a unit test, it's strong evidence that there's something wrong with the code being tested (or the dependencies of that code or of the test code), and developers should immediately fix that, rather than add layers of complexity to a unit test. Of course, if developers don't have this mentality or don't understand how to improve their code, then unit tests are a waste of time, and conversely (though far more rarely), if code is written well enough to start out with, then unit tests won't promote any changes to the source code, which removes most of their utility.
 
Junilu Lacar
Sheriff
Posts: 15815
264
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Grey Smith wrote:I don't know if we should really call unit tests "tests," because it implies that they exercise the code. I've found dozens of bugs by manual testing and reading code, and probably less than ten from unit tests.


Again, I think we differ in contexts here. I find unit tests that are written before the code they are meant to exercise very valuable. Unit tests written after the production code is far less valuable. I think your context is the latter.
 
Juval Lowy
Author
Posts: 12
10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I was referring to breathing from the business perspective (business in this case is you as a human being).
There is no added value in breathing , even though it is essential and important.
Companies hire people not because they breath well but because they add value somewhere.
Relationships, success, has nothing to do with the essential task of breathing, but everything with some value you add by doing other things.

Same is true for unit testing. You are not hired because of your unit testing. Ever. You are hired to add value to customers. That means addressing their requirements in an impeccable manner.
Composable design founded on volatility-based decomposition and regression testing is where you add the value.

Juval.
 
Junilu Lacar
Sheriff
Posts: 15815
264
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Grey Smith wrote:I consider unit tests important because they implicitly ask developers to write simple, short code that has a small number of readily understandable dependencies. If it's difficult or cumbersome to write a unit test, it's strong evidence that there's something wrong with the code being tested (or the dependencies of that code or of the test code), and developers should immediately fix that, rather than add layers of complexity to a unit test. Of course, if developers don't have this mentality or don't understand how to improve their code, then unit tests are a waste of time, and conversely (though far more rarely), if code is written well enough to start out with, then unit tests won't promote any changes to the source code, which removes most of their utility.


Again, I would agree that all this is true for unit tests that are written after the production code that they exercise. This is seldom true for well-designed tests written before the production code.
 
Grey Smith
Greenhorn
Posts: 19
1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Reading this:

In theory, you could perform regression testing even on a functionally decomposed system. In practice, the complexity of that task would set the bar very high. The sheer number of the functional components would make testing all the interactions impractical. The very large services would be internally so complex that no one could effectively devise a comprehensive strategy that tests all code paths through such services. With functional decomposition, most developers give up and perform just simple unit testing. Therefore, by precluding regression testing, functional decomposition makes the entire system untestable, and untestable systems are always rife with defects.



I don't think I quite get what you're saying. Testing all code paths is always impossible for any fair-sized application. But doing manual regression testing on any program is as simple as using it in its most common ways -- does it start up? Does it do Task A and B? What if I give it bad input? That kind of manual testing is unavoidable and forms the backbone of development. If you're referring to integration (automated) tests -- my experience is remarkably different. I've worked on several projects with minimal bugs and no integration tests, because developers religiously practice manual testing and code reviews.
 
Junilu Lacar
Sheriff
Posts: 15815
264
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Juval Lowy wrote:Same is true for unit testing. You are not hired because of your unit testing. Ever. You are hired to add value to customers.


Good carpenters/builders spend a considerable amount of time preparing and cleaning their tools. They are not paid to do that. Ever. They are paid to build things of value to their customers. However, if their tools are not in tip-top condition, they cannot produce good work with those tools. To me, good tests are part of my toolkit as a developer and designer. If I don't grow and maintain a good suite of tests as I'm evolving my code/design, then I'm not producing very good work and therefore not delivering the value that I'm paid to deliver. There is one difference though: builders don't leave their tools with the customer when the job is over. They bring their tools with them. Good developers leave their tests with the production code. Well-written tests will continue to provide value even when the original developers are gone (assuming of course the new developers who take over keep them in good shape and don't neglect them)
 
Juval Lowy
Author
Posts: 12
10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, of course you can work correctly. But for the most part, the industry is not, hence Righting Software.
Most systems do not have good regression testing, or not at all, but still have unit testing, for the reasons I wrote in the book.

Juval.
 
Grey Smith
Greenhorn
Posts: 19
1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Junilu - quite right. I should totally try out TDD a couple more times, at least when I'm writing behavioral tests...
I think there's a big difference between behavioral tests -- does some chunk of code act correctly given some input -- and what I refer to as "unit" tests, which usually only test individual functions or at most 2-3 functions together. My experience has been largely with the latter, so I can speak to their utility. Behavioral tests require a different mindset and I unfortunately haven't been able to write many of them because the codebase at my job is not designed for it. I'd love to be able to write a mix of behavioral tests and unit tests on a future project. Having test cases in mind -- knowing the right behavior -- is the first thing you need to know. If writing behavioral tests first forces developers to consider thoroughly what they will write and why, that would be wonderful.
 
Junilu Lacar
Sheriff
Posts: 15815
264
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Grey Smith wrote:If you're referring to integration (automated) tests -- my experience is remarkably different. I've worked on several projects with minimal bugs and no integration tests, because developers religiously practice manual testing and code reviews.


Well, that's certainly one way to live. My experience is that manual tests have their place in the development process but when you have to do the same thing more than three or four times, then it's practically your responsibility and obligation to automate it. As for code reviews, it's a hit or miss for me. Most of the time I'm focused on simplicity: do all the tests pass, is the code expressive enough, is there any duplication that I can eliminate, and are things small enough to be manageable/maintainable?
 
Marshal
Posts: 69896
278
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Breathing is an overrated activity.

Any cows are for the whole th‍read rather than individual posts.
    Bookmark Topic Watch Topic
  • New Topic