• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Pragmatic unit testing question

 
Ranch Hand
Posts: 86
2
VI Editor Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,

I'd like to know, pragmatically, what are the positives and downsides of unit testing. I can identify couple of each as a newbie to the topic, but the strongest would be early problems discovery and unbiased testing on the benefits side, and time demanding on the drawbacks side. And from the practical point of view, the last drawback outweighs the benefits for me. Especially at the early stages of a project, when design evolves a lot and the requirements change all the time, it's a double work - write the test and write the code, refactor tests and then the code. Is it really worthy? And is it for everyone and for every kind of project? How to justify the higher costs and slower delivery to the customer (especially when it's internal customer)?

Thanks
Tomas
 
author
Posts: 799
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Tomas,

Short-term--and I mean very short term--unit testing is probably a little more trouble than it's worth. Here's my story on writing code without unit tests:

http://rubylearning.com/blog/2010/12/08/my-ruby-regrets/

I still have the same experiences when I try to slam out code--in short, it's fairly easy to put out a lot of code quickly and then regret not keeping it under control. Like, a day's worth or less. So I almost always write unit tests.

Unit tests are indeed an investment and can become a liability. The book focuses heavily on keeping your tests maintainable. When you get a test passing, you're not done. Your job is to use the tests to help you better craft the code as you go, and to use the tests to document things. That means you have to refactor the tests about as much as the production code.

On the surface, it suggests double the maintenance. However, the more you do this, the more flexibility you ultimately get out of your code. The smaller methods and classes, which should come about partly because you can refactor your code because you have tests, give you more design flexibility (and sometimes reuse). The tests give you the confidence to make changes as new features come. The tests also help you better understand the code, which reduces dramatically the time we otherwise waste wading through code to understand it. And then there's defects, the obvious part of all this. If we were to quantify the cost of a defect well (time to pinpoint, time to fix, lost opportunity time, lost customer satisfaction, etc.), it'd be obvious that the cost of unit testing far underweighs the cost of defects.

Ultimately, doing unit testing (TDD in my case) has allowed me to keep a sustainable pace over time, when for those not doing it, their pace decreases over time.

If you are not careful about unit testing, however, it can become too costly for its own good. Tests must be kept maintainable, lest they not reap all possible benefits. I've seen some cases like this where the investment in unit testing was tossed out the window--the tests were long, difficult, and provided little value.

Regards,
Jeff
 
Tomas Linhart
Ranch Hand
Posts: 86
2
VI Editor Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for your thoughts, Jeff!
From your answer, I have a feeling that tests should actually drive the coding, which I'm not really convinced is the right way to develop software. Maybe it's just the lack of experience in bigger projects or I might have just fallen behind the evolution in this area :-) In my opinion, test should just prove correctness of your code, and not influence the implementation details. But I guess that's not the case of unit tests, right.
 
Jeff Langr
author
Posts: 799
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Tomas,

That's for you to decide! :-) I'm not one to say unit testing or TDD or whatever is essential. It is for me, because I've gotten a lot of value out of it. You can choose to instead write integration tests.

The way I view it (and have experienced it) is that there's a lot of $$ to putting unit tests in place, and the more benefits you can gain from them, the more likely your investment has a positive ROI.

Since I've been challenged in the past: I am currently a full-time programmer (doing Clojure), and write unit tests extensively (predominantly via TDD). I've been in this full-time role for a year and a half. Prior to that, I did consulting for a couple years, prior to that, full-time development, prior to that, consulting, etc. I alternate between the two--which I think helps keep me up to date and "real."

Jeff
 
Jeff Langr
author
Posts: 799
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
In fact I'm convinced there's no right or wrong way to develop software. While "pragmatic" in the book title comes from the name of the publisher, it's absolutely the way to think about things. TDD is a great technique, and yet I don't do it for 100% of my efforts. Do what makes sense.

Unit testing after is a fine technique, but it incurs costs. I've been tempted in the past to say "don't do unit testing" at all if you can't find a way to gain more benefits from the significant cost of investment (which is what the book promotes); maybe your efforts are better invested elsewhere in that case.

Jeff
 
Sheriff
Posts: 17644
300
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Jeff Langr wrote:TDD is a great technique, and yet I don't do it for 100% of my efforts. Do what makes sense.


That's a great point and reminder. Kent Beck said pretty much the same thing during those public on-line discussions with David Heinemeier Hansson about whether TDD was dead.

I think Uncle Bob Martin has said something like that, too, but don't take my word for it -- last thing I need is an indignant tweet from Uncle Bob about false attributions

I don't do TDD all the time either. I do, however, feel better when I am able to do TDD. Whenever I skip TDD for some reason, I usually feel more uneasy and uncertain about my work and sometimes even guilty.
 
Junilu Lacar
Sheriff
Posts: 17644
300
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tomas Linhart wrote:And from the practical point of view, the last drawback outweighs the benefits for me. Especially at the early stages of a project, when design evolves a lot and the requirements change all the time, it's a double work - write the test and write the code, refactor tests and then the code. Is it really worthy? And is it for everyone and for every kind of project? How to justify the higher costs and slower delivery to the customer (especially when it's internal customer)?


It all depends on the kind of tests you write and the level at which the tests are written. At the onset of a project, where many details are still unknown and designs are still very much in flux, it's very costly to try to focus on the detailed design level when writing tests. Don't do that. Rather, focus on writing tests with more granularity, with the purpose of using them to learn more about what the system should do. Even if these are going to be thrown away anyway, they are still a good investment. Anything that helps you gain more understanding about the system is a good investment in time and effort, IMO.

What is the alternative? Not writing tests early deprives you of critical early feedback about what your system is doing or not doing correctly. You are likely accumulating Technical Debt, which is more costly to pay back later. I'm pretty sure there are numerous studies that have found that the return on investment of writing tests along the way and managing technical debt is much higher than the long-term costs of doing things expediently now and addressing bugs and poor design issues later. High levels of unmanaged technical debt have been the downfall of many projects. Just look at the Healthcare.gov fiasco and other project failures, big and small. On the other hand, well-written tests always seem to pay for themselves in terms of savings from reduced effort spent chasing down bugs and dealing with production issues. At least that has been my experience in the past ten or so years of doing automated testing.

Is it for everyone and every kind of project? Probably not, especially for the ones that you want to fail.

Funny story: A few years ago, my boss' boss asked me whether Agile development methods were good for all projects and I replied along the same lines: "Not for all projects, only the ones that you want to succeed." He laughed at my cockiness and "zeal" at that time but two years later he was the executive sponsor and champion for a group-wide agile adoption effort. I was tasked with promoting TDD and other agile technical practices among the engineering teams and that's one of the main things I do at work.

In a keynote he made a few years ago, Uncle Bob Martin declared that "The jury is in: TDD works!" and said that there really should be no argument or doubt about the effectiveness of techniques like TDD at this point in time. I guess we still have a few more years to go before reality aligns with Uncle Bob's world view.
 
Tomas Linhart
Ranch Hand
Posts: 86
2
VI Editor Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I got your points guys. And thanks for that in-depth explanation.

There are still points I don't really understand, i.e. key how to find out when it's worthy to write tests and when it's not (because to the day I could live without them). Also, how do they actually help me in better understanding the system? Shouldn't they just provide information whether the software works correctly or not?

But I guess this comes from lack of my knowledge and experience.
 
Jeff Langr
author
Posts: 799
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Tomas,

If you're writing code that's clear, well-designed, accommodates new features easily, and doesn't exhibit any defects, you might not need unit tests. Unfortunately, most systems I've seen aren't like that. They're bloated, with lots of duplicate code, hard-to-understand code, inflexible design, and so on. They get that way because of the adage "if it ain't broke, don't fix it." We get our code to work, and ship it. When we add new code, we seek to add it in a minimally invasive way, which often isn't the best way to design it (e.g. it's easier to copy a long method and make a small change to get your stuff working, rather than do the right thing and factor the common code to a single spot). We fear touching code already proven to work, because why would we risk breaking something that's not what we're working on? As such, the natural progression of a system is to degrade in quality.

The larger hope for unit testing is that it allows you to make changes while minimizing the fear of breaking things already working. Regular cleanup of the code can help stave off the degradation.

I did no unit testing for ~18 years prior to learning about it. I didn't ship a lot of defects, though of course I did ship some, and they were often dumb mistakes that a simple unit test might've caught.

You can survive without unit tests. But it's likely your systems will degrade faster than mine. I wouldn't have spent another 15 years doing unit testing/TDD if I didn't gain a number of significant benefits.

As far as "better understanding the system," there's a lot of emphasis in the book on creating tests that clearly document the behavior they are verifying. Test names are important, as is the structure and readability of tests. I often will pull up just the list of test names for a class, and quickly get a good understanding of what features it exposes, and what the developers intended with its design. And as long as the tests are passing, I know they honestly depict how the system currently behaves (unlike comments).

Regards,
Jeff
 
Junilu Lacar
Sheriff
Posts: 17644
300
Mac Android IntelliJ IDE Eclipse IDE Spring Debian Java Ubuntu Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Jack Reeves wrote some essays a while back that revolved around the idea that the code is the design. I find a lot of merit in his proposition and actually take the same viewpoint when I do unit testing and TDD. Basically, Reeves submits that coding is primarily a design process. Building of software is done by compilers and linkers.

Thus when I read my JUnit test reports, they read like a list of statements that summarize the behavior that I expect from the system. I have names like

Without knowing anything else about the class these tests are exercising, I bet you can easily figure out what the class is, right? That kind of what good documentation does, so in effect, my tests become part of my detailed design documentation. What's more, if you look at the tests, you will find that they are examples of how the public API of the class is supposed to be used. The the test setup methods show you how the class is initialized and/or configured. When I look at other people's code during reviews, I always ask to start with the unit tests. This is how tests can help you better understand the system.

Another way I look at it is during development, when I find something is difficult to test and/or refactor, I take it as a sign that the code is telling me that something is off in the design. When a test does not clearly describe some piece of functionality to me or if it's trying to do too many things, it's another sign that the code or design could use some improvement. These are all "smells" that you learn to recognize with experience but you need to be in the right mindset to detect them and become sensitized to them. Focusing on testing early and often helps you do that and being sensitive to what your code is telling you is just another way of understanding how it works. I wish this kind of approach would work on my wife but somehow, she remains inscrutable after all these years
 
Tomas Linhart
Ranch Hand
Posts: 86
2
VI Editor Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Those are strong arguments and I have to agree with them. I should try to start with unit testing and see what I can gain from it.

Anyway, thanks a lot for your insights, it's really much appreciated.
 
Don't get me started about those stupid light bulbs.
reply
    Bookmark Topic Watch Topic
  • New Topic