• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Journal Article - Introduction to Code Coverage

 
Sheriff
Posts: 7023
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
If you've ever wanted to understand just what code coverage is all about, take a look at Lasse Koskela's most recent JavaRanch Journal article, "Introduction to Code Coverage".
Please post any responses here.
 
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Great article! Thanks! Very timely because I had been wondering if any free coverage tools existed.
I knew about clover (not free) but not the other alternatives. I briefly tried the jcoverage GPL version and groboutils.
JCoverage failed if I specified too many classes to be instrumented -- seemingly due to Windows command-line limits. That was a bummer. Another problem is that it complains about inner classes not being compiled w/debug though they are. Gave up and moved to groboutils.
Groboutils was harder to setup but not impossible. It generates a TON of data during capture and takes forever. I expect performance hits from any such tool for the data logging but I terminated a test after 6 minutes that takes 13 secs normally.
I'll take a closer look at such tools later when my need is more pressing. I wonder if the $$$ versions of jcoverage correct the issues I ran across?
 
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
John, I'm glad you enjoyed the article.
I don't know whether the commercial version solves the problems you encountered, but I would strongly suggest driving instrumentation, test execution, and reporting from an Ant script. At least you'll avoid the command-line length problem
 
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
EMMA (http://emma.sourceforge.net) uses normal classpath-like input (path-like elements when used from ANT) to specify instrumentation input and would not have any command line length limitations (assuming they arise from having to list all input classes on the command line).
 
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The third beta of GroboUtils 1.1.0 is out, which simplifies the Ant interface, cleans up the reports, and a few other bits. You can try it out by downloading from here.
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Matt, 'example2' is complaining about not finding the source files.
 
Matt Albrecht
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
That warning is supposed to happen - it shows what happens if the source isn't available. Other reports generated do have the source included.
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Ah. Ok.
 
Ranch Hand
Posts: 32
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,
I have being looking for code coverage tool that works with Maven, that seems to leave me with Clover and JCoverage.
Clover costs $1500 for the server edition which includes html reports (essential). This is expensive.
Even more unfortunate is that Clover and JCoverage gave me different results on the same test code. JCoverage seems to be at fault here, it doesn't go into some of the branches I know the unit test exercises.
Are there any other alternatives?
Best regards,
John
 
author
Posts: 3252
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The version of JCoverage I tried did seemed inconsistent in the way it counts lines:Appeared to be counted as 3 lines when not tested, and 1 line when tested, which skews the coverage figures. It also didn't properly handle inner classes, but that might be an artifact of the way it is integrated with Maven and/or the way the HTML report is generated.

- Peter
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Note that the actual number isn't that important. It's the trend that counts... You could also take a look at Koalog Code Coverage (I've never used it myself, though, so I can't comment on it).
 
Ranch Hand
Posts: 8945
Firefox Browser Spring Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Lasse,

What is your opnion on using JProbe for code coverage?
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Pradeep Bhat:
Lasse, What is your opnion on using JProbe for code coverage?


JProbe? JProbe the performance profiling product? I've never used JProbe myself -- especially not for code coverage reports -- but it would seem like much more complex procedure compared to Ant-driven coverage tools like Clover and JCoverage.
 
author
Posts: 14112
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Another interesting project is Hansel: http://hansel.sourceforge.net/

With Hansel, you can specify what classes a JUnit TestSuite should cover, which will make the suite fail if they aren't covered by 100% - including detailed reports which lines weren't covered.
 
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Could anyone so me what the basic block is.
Thank you so much

Matt
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by matt or:
Could anyone so me what the basic block is.


If you're referring to this sentence in my article:
Basic block coverage considers each sequence of non-branching statements as its unit of code instead of individual statements.

It means that the "block" consists of a group of adjacent statements that do not have the ability to branch execution conditionally. I.e. in the following snippet:

Lines 2..4 make up one block, line 6 makes up one block, and line 8 makes up one block. The if statement on line 5 is a branching statement, which decides whether we're going to execute the block starting from line 6 or the block starting from line 8.
 
matt or
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Lasse.
How can I collect the percentage of basic block coverage??
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by matt or:
How can I collect the percentage of basic block coverage??


Manually? You count your basic blocks, you figure out how many blocks aren't executed by your test suite, and do the math from there.

However, this is a really bad idea for anything other than learning purposes. I'd recommend using one of the existing code coverage tools and trusting that whatever they say about your tests' code coverage is reasonably accurate...
 
matt or
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thank you Lasse

No, not manually.
How can I automatic this process? any technique should be used?
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by matt or:
How can I automatic this process? any technique should be used?

My article (the original topic of this thread...) describes how to automate code coverage reporting by integrating it into your Ant build script using three different products (two of which are open source).
 
Vlad Roubtsov
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Another, more recent, review of the coverage tools: http://europetravelogue.com/blog/pivot/entry.php?id=41#body

The winners are EMMA and Clover.
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Looking good. Vlad, will we see those nice green and red bars in EMMA soon?
 
Greenhorn
Posts: 26
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
John,

JCoverage failed if I specified too many classes to be instrumented -- seemingly due to Windows command-line limits. That was a bummer. Another problem is that it complains about inner classes not being compiled w/debug though they are.



I ran into the too-many-classes problem even running from ant. It was a jcoverage bug, but if you go to their bugzilla system you'll find a patch. I applied the patch to the GPL source, and the problem went away.

I haven't encountered the second problem you mentioned, though I do use inner classes, so I'm no help there.
 
Vlad Roubtsov
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Lasse Koskela:
Looking good. Vlad, will we see those nice green and red bars in EMMA soon?



Clearly, "green and red bars" are the most important feature in a coverage tool Yes, v2.1 development is about to start and hopefully this will make it into the release plan.
[ September 20, 2004: Message edited by: Vlad Roubtsov ]
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Vlad Roubtsov:
Clearly, "green and red bars" are the most important feature in a coverage tool


Of course not. Everyone knows they're only the second most important feature in a coverage tool
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Lasse Koskela:
Of course not. Everyone knows they're only the second most important feature in a coverage tool


Right after Ant support

(OK. No more off-topic babbling. I promise.)
 
Vlad Roubtsov
Greenhorn
Posts: 9
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Lasse Koskela:

Right after Ant support



EMMA supports ANT very well. Of course, now everybody wants Maven.
 
Greenhorn
Posts: 29
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi

I am in a situation where I have a tool written in Java, which is going to autmate testing for all APIs without the tests being JUnits. This tool at least caters to my project needs :-)

Now I need to integrate this program with a tool which allows me to measure the code coverage. Most of the code coverage tools drive JUnits it seems.

I read the documentation of EMMA and I know I can use it in my scenario.Are there any other options as well that I could consider?

Any idea/hint regarding this will really help me.

thanks
Prashant Jain
prashantjainn@yahoo.com
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Prashant Jain:
Now I need to integrate this program with a tool which allows me to measure the code coverage. Most of the code coverage tools drive JUnits it seems.

I read the documentation of EMMA and I know I can use it in my scenario.Are there any other options as well that I could consider?

Most of the popular code coverage tools don't care about whether the code is executed by a set of JUnit tests or by a user manually launching the application and doing whatever he wishes. The key is that the code coverage tool has injected necessary logging statements into your .class files so that when that particular code is executed, the event is logged into a log file.

I haven't tried it myself, but I believe all three tools demonstrated in my article can be used with your custom unit test framework.
 
Prashant Jain
Greenhorn
Posts: 29
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Lasse , Thanks for the response.

Could you and all the others also tell me which tool to choose.
Clover is a commercial tool, I dont think it will be feasible for me unless the value addition is really substantial.

I have read about Grobo utils that it gets really slow for a large codebase.And it is also known to have a few problems.

I have looked at EMMA and till now it seems to be a good option.Would any one of you have anything to say about EMMA or any other good freeware?

thanks
Prashant Jain
prashantjainn@yahoo.com
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
If Clover is not an option, I would probably recommend EMMA or JCoverage. However, I've heard that a lot of people have had problems with JCoverage on big projects...
 
Ranch Hand
Posts: 379
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I started using JCoverage after reading this thread and I really like it. It has a very simple interface, integrates with Ant without any problem, and gives a nice visual coverage on what code was not covered by my test cases, which I think is the best feature.

I would suggest that you use JCoverage first and see if it meets your needs. If it does not, maybe you can go for other options.
 
Ranch Hand
Posts: 1033
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Vlad,

I just downloaded and tried Emma, its very nice. Keep up the good work.
 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
With Emma or any of the other coverage tools, do the ant tasks provide any mechansim to have the build fail if a certain coverage level is not met? From the doc on emma, it looks like you can specify failure metrics but they only apply to the generated report.
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Bryan Long:
With Emma or any of the other coverage tools, do the ant tasks provide any mechansim to have the build fail if a certain coverage level is not met? From the doc on emma, it looks like you can specify failure metrics but they only apply to the generated report.


At least Clover and GroboUtils have the option of failing a build if the required coverage levels are not achieved. I don't remember off-hand whether JCoverage has such a feature and I haven't looked into EMMA that closely either.
 
Greenhorn
Posts: 11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi,
The article was really useful. But i have a situation here where i must find out the code coverage when my application is running. In more detailed terms, i compile the classes and instrument them and if i build a war or a ear file with the instrumented classes and deploy it on the server like tomcat or jboss and i hit the url the classes work fine, but where will the .ser (trying to use jcoverage). files will be created so that i can generate reports.
Please let me know if there is some other option to the above mentioned problem

Thanks
Rajkumar
 
Lasse Koskela
author
Posts: 11962
5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Rajkumar, I moved your question into a new thread because it's not so much about the article as it is about using a specific coverage tool (Cobertura).
 
reply
    Bookmark Topic Watch Topic
  • New Topic