aspose file tools*
The moose likes Testing and the fly likes Journal Article - Introduction to Code Coverage Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Spring in Action this week in the Spring forum!
JavaRanch » Java Forums » Engineering » Testing
Bookmark "Journal Article - Introduction to Code Coverage" Watch "Journal Article - Introduction to Code Coverage" New topic
Author

Journal Article - Introduction to Code Coverage

Dirk Schreckmann
Sheriff

Joined: Dec 10, 2001
Posts: 7023
If you've ever wanted to understand just what code coverage is all about, take a look at Lasse Koskela's most recent JavaRanch Journal article, "Introduction to Code Coverage".
Please post any responses here.


[How To Ask Good Questions] [JavaRanch FAQ Wiki] [JavaRanch Radio]
John Lindwall
Ranch Hand

Joined: May 29, 2002
Posts: 31
Great article! Thanks! Very timely because I had been wondering if any free coverage tools existed.
I knew about clover (not free) but not the other alternatives. I briefly tried the jcoverage GPL version and groboutils.
JCoverage failed if I specified too many classes to be instrumented -- seemingly due to Windows command-line limits. That was a bummer. Another problem is that it complains about inner classes not being compiled w/debug though they are. Gave up and moved to groboutils.
Groboutils was harder to setup but not impossible. It generates a TON of data during capture and takes forever. I expect performance hits from any such tool for the data logging but I terminated a test after 6 minutes that takes 13 secs normally.
I'll take a closer look at such tools later when my need is more pressing. I wonder if the $$$ versions of jcoverage correct the issues I ran across?
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
John, I'm glad you enjoyed the article.
I don't know whether the commercial version solves the problems you encountered, but I would strongly suggest driving instrumentation, test execution, and reporting from an Ant script. At least you'll avoid the command-line length problem


Author of Test Driven (2007) and Effective Unit Testing (2013) [Blog] [HowToAskQuestionsOnJavaRanch]
Vlad Roubtsov
Greenhorn

Joined: Jun 14, 2003
Posts: 9
EMMA (http://emma.sourceforge.net) uses normal classpath-like input (path-like elements when used from ANT) to specify instrumentation input and would not have any command line length limitations (assuming they arise from having to list all input classes on the command line).


<a href="http://emma.sourceforge.net" target="_blank" rel="nofollow">EMMA</a>: an open-source code coverage toolkit
Matt Albrecht
Greenhorn

Joined: Apr 19, 2004
Posts: 3
The third beta of GroboUtils 1.1.0 is out, which simplifies the Ant interface, cleans up the reports, and a few other bits. You can try it out by downloading from here.
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Matt, 'example2' is complaining about not finding the source files.
Matt Albrecht
Greenhorn

Joined: Apr 19, 2004
Posts: 3
That warning is supposed to happen - it shows what happens if the source isn't available. Other reports generated do have the source included.
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Ah. Ok.
John Cogan
Ranch Hand

Joined: Jan 22, 2003
Posts: 32
Hi,
I have being looking for code coverage tool that works with Maven, that seems to leave me with Clover and JCoverage.
Clover costs $1500 for the server edition which includes html reports (essential). This is expensive.
Even more unfortunate is that Clover and JCoverage gave me different results on the same test code. JCoverage seems to be at fault here, it doesn't go into some of the branches I know the unit test exercises.
Are there any other alternatives?
Best regards,
John


----------------<br />SCJP, SCWCD
Peter den Haan
author
Ranch Hand

Joined: Apr 20, 2000
Posts: 3252
The version of JCoverage I tried did seemed inconsistent in the way it counts lines:Appeared to be counted as 3 lines when not tested, and 1 line when tested, which skews the coverage figures. It also didn't properly handle inner classes, but that might be an artifact of the way it is integrated with Maven and/or the way the HTML report is generated.

- Peter
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Note that the actual number isn't that important. It's the trend that counts... You could also take a look at Koalog Code Coverage (I've never used it myself, though, so I can't comment on it).
Pradeep bhatt
Ranch Hand

Joined: Feb 27, 2002
Posts: 8919

Lasse,

What is your opnion on using JProbe for code coverage?


Groovy
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Pradeep Bhat:
Lasse, What is your opnion on using JProbe for code coverage?

JProbe? JProbe the performance profiling product? I've never used JProbe myself -- especially not for code coverage reports -- but it would seem like much more complex procedure compared to Ant-driven coverage tools like Clover and JCoverage.
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Another interesting project is Hansel: http://hansel.sourceforge.net/

With Hansel, you can specify what classes a JUnit TestSuite should cover, which will make the suite fail if they aren't covered by 100% - including detailed reports which lines weren't covered.


The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
matt or
Greenhorn

Joined: Sep 11, 2004
Posts: 5
Could anyone so me what the basic block is.
Thank you so much

Matt
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by matt or:
Could anyone so me what the basic block is.

If you're referring to this sentence in my article:
Basic block coverage considers each sequence of non-branching statements as its unit of code instead of individual statements.

It means that the "block" consists of a group of adjacent statements that do not have the ability to branch execution conditionally. I.e. in the following snippet:

Lines 2..4 make up one block, line 6 makes up one block, and line 8 makes up one block. The if statement on line 5 is a branching statement, which decides whether we're going to execute the block starting from line 6 or the block starting from line 8.
matt or
Greenhorn

Joined: Sep 11, 2004
Posts: 5
Thanks Lasse.
How can I collect the percentage of basic block coverage??
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by matt or:
How can I collect the percentage of basic block coverage??

Manually? You count your basic blocks, you figure out how many blocks aren't executed by your test suite, and do the math from there.

However, this is a really bad idea for anything other than learning purposes. I'd recommend using one of the existing code coverage tools and trusting that whatever they say about your tests' code coverage is reasonably accurate...
matt or
Greenhorn

Joined: Sep 11, 2004
Posts: 5
Thank you Lasse

No, not manually.
How can I automatic this process? any technique should be used?
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by matt or:
How can I automatic this process? any technique should be used?
My article (the original topic of this thread...) describes how to automate code coverage reporting by integrating it into your Ant build script using three different products (two of which are open source).
Vlad Roubtsov
Greenhorn

Joined: Jun 14, 2003
Posts: 9
Another, more recent, review of the coverage tools: http://europetravelogue.com/blog/pivot/entry.php?id=41#body

The winners are EMMA and Clover.
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Looking good. Vlad, will we see those nice green and red bars in EMMA soon?
Terry Mullett
Greenhorn

Joined: Feb 21, 2003
Posts: 26
John,
JCoverage failed if I specified too many classes to be instrumented -- seemingly due to Windows command-line limits. That was a bummer. Another problem is that it complains about inner classes not being compiled w/debug though they are.


I ran into the too-many-classes problem even running from ant. It was a jcoverage bug, but if you go to their bugzilla system you'll find a patch. I applied the patch to the GPL source, and the problem went away.

I haven't encountered the second problem you mentioned, though I do use inner classes, so I'm no help there.
Vlad Roubtsov
Greenhorn

Joined: Jun 14, 2003
Posts: 9
Originally posted by Lasse Koskela:
Looking good. Vlad, will we see those nice green and red bars in EMMA soon?


Clearly, "green and red bars" are the most important feature in a coverage tool Yes, v2.1 development is about to start and hopefully this will make it into the release plan.
[ September 20, 2004: Message edited by: Vlad Roubtsov ]
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Vlad Roubtsov:
Clearly, "green and red bars" are the most important feature in a coverage tool

Of course not. Everyone knows they're only the second most important feature in a coverage tool
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Lasse Koskela:
Of course not. Everyone knows they're only the second most important feature in a coverage tool

Right after Ant support

(OK. No more off-topic babbling. I promise.)
Vlad Roubtsov
Greenhorn

Joined: Jun 14, 2003
Posts: 9
Originally posted by Lasse Koskela:

Right after Ant support


EMMA supports ANT very well. Of course, now everybody wants Maven.
Prashant Jain
Greenhorn

Joined: Aug 19, 2004
Posts: 29
Hi

I am in a situation where I have a tool written in Java, which is going to autmate testing for all APIs without the tests being JUnits. This tool at least caters to my project needs :-)

Now I need to integrate this program with a tool which allows me to measure the code coverage. Most of the code coverage tools drive JUnits it seems.

I read the documentation of EMMA and I know I can use it in my scenario.Are there any other options as well that I could consider?

Any idea/hint regarding this will really help me.

thanks
Prashant Jain
prashantjainn@yahoo.com
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Prashant Jain:
Now I need to integrate this program with a tool which allows me to measure the code coverage. Most of the code coverage tools drive JUnits it seems.

I read the documentation of EMMA and I know I can use it in my scenario.Are there any other options as well that I could consider?
Most of the popular code coverage tools don't care about whether the code is executed by a set of JUnit tests or by a user manually launching the application and doing whatever he wishes. The key is that the code coverage tool has injected necessary logging statements into your .class files so that when that particular code is executed, the event is logged into a log file.

I haven't tried it myself, but I believe all three tools demonstrated in my article can be used with your custom unit test framework.
Prashant Jain
Greenhorn

Joined: Aug 19, 2004
Posts: 29
Lasse , Thanks for the response.

Could you and all the others also tell me which tool to choose.
Clover is a commercial tool, I dont think it will be feasible for me unless the value addition is really substantial.

I have read about Grobo utils that it gets really slow for a large codebase.And it is also known to have a few problems.

I have looked at EMMA and till now it seems to be a good option.Would any one of you have anything to say about EMMA or any other good freeware?

thanks
Prashant Jain
prashantjainn@yahoo.com
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
If Clover is not an option, I would probably recommend EMMA or JCoverage. However, I've heard that a lot of people have had problems with JCoverage on big projects...
Sathya Srinivasan
Ranch Hand

Joined: Jan 29, 2002
Posts: 379
I started using JCoverage after reading this thread and I really like it. It has a very simple interface, integrates with Ant without any problem, and gives a nice visual coverage on what code was not covered by my test cases, which I think is the best feature.

I would suggest that you use JCoverage first and see if it meets your needs. If it does not, maybe you can go for other options.


Cheers, Sathya Srinivasan - SCJP 1.2, SCWCD 1.2, SCMAD 1.0
Co-Author of Whizlabs SCMAD Certification Exam Simulator and SCMAD Exam Guide Book
peter wooster
Ranch Hand

Joined: Jun 13, 2004
Posts: 1033
Vlad,

I just downloaded and tried Emma, its very nice. Keep up the good work.
Bryan Long
Greenhorn

Joined: Dec 06, 2004
Posts: 1
With Emma or any of the other coverage tools, do the ant tasks provide any mechansim to have the build fail if a certain coverage level is not met? From the doc on emma, it looks like you can specify failure metrics but they only apply to the generated report.
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Bryan Long:
With Emma or any of the other coverage tools, do the ant tasks provide any mechansim to have the build fail if a certain coverage level is not met? From the doc on emma, it looks like you can specify failure metrics but they only apply to the generated report.

At least Clover and GroboUtils have the option of failing a build if the required coverage levels are not achieved. I don't remember off-hand whether JCoverage has such a feature and I haven't looked into EMMA that closely either.
Rajkumar Chandrasekaran
Greenhorn

Joined: Nov 23, 2004
Posts: 11
Hi,
The article was really useful. But i have a situation here where i must find out the code coverage when my application is running. In more detailed terms, i compile the classes and instrument them and if i build a war or a ear file with the instrumented classes and deploy it on the server like tomcat or jboss and i hit the url the classes work fine, but where will the .ser (trying to use jcoverage). files will be created so that i can generate reports.
Please let me know if there is some other option to the above mentioned problem

Thanks
Rajkumar
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Rajkumar, I moved your question into a new thread because it's not so much about the article as it is about using a specific coverage tool (Cobertura).
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Journal Article - Introduction to Code Coverage