wood burning stoves 2.0*
The moose likes Testing and the fly likes TestNG vs JUnit, mockito + Powermock vs jmockit Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Engineering » Testing
Bookmark "TestNG vs JUnit, mockito + Powermock vs jmockit" Watch "TestNG vs JUnit, mockito + Powermock vs jmockit" New topic
Author

TestNG vs JUnit, mockito + Powermock vs jmockit

Le Fre
Greenhorn

Joined: Aug 23, 2010
Posts: 6

I should choose between some testing frameworks including mock libraries for testing legacy code and new development.
The possible candidates I have identified so far are JUnit vs TestNG, and (Mockito or EasyMock in combination with powermock) vs jmockit.
Which combination would you pick and why?

TestNG and jmockit seems to be the best solutions:
  • TestNG offers more features than JUnit: dataprovider, nonstatic @beforeclass annotation, grouping annotation.
  • Jmockit is an integrated mock framework with the same capabilities as easymock or mockito combined with powermock while being a bit simpler (at least for its powermock-like abilities).


  • On the other side,
  • there are incompatibilities between TestNG and jmockit: jmockit is only compatible with some versions of testng, and I did not get it to run it with any...
  • jmockit is on a version 0.9.*.*, does this mean that it is still unstable?


  • So, should I stick with jmockit + junit, or take testng + powermock + mockito (or easymock)
    Jeanne Boyarsky
    author & internet detective
    Marshal

    Joined: May 26, 2003
    Posts: 31062
        
    232

    Le,
    Welcome to CodeRanch!

    Personally, I'd go with JUnit over TestNG. I think TestNG gives rope to hang yourself (aka features I don't think encourage good unit tests.) TestNG shines in the complex tests with dependencies arena. A situation I don't want to find myself in.

    For unit testing, I'd go with Mockito or jMock. I'm not familiar with Jmockit though. I don't need PowerMock because Mockito and jMock are powerful enough for my needs.

    The nice thing is there isn't a right or wrong answer here. The important thing is you are looking at writing unit tests.


    [Blog] [JavaRanch FAQ] [How To Ask Questions The Smart Way] [Book Promos]
    Blogging on Certs: SCEA Part 1, Part 2 & 3, Core Spring 3, OCAJP, OCPJP beta, TOGAF part 1 and part 2
    Le Fre
    Greenhorn

    Joined: Aug 23, 2010
    Posts: 6

    Thanks for your reply Jeanne,
    which feature do you think testng do not encourage good unit test?
    Actually the main advantage I see in testng as for now is to be able to prepare database settings in a super class in a non-static @beforeclass annotated method and still be able to hack this preparation in the subclass.

    I will need to have something like powermock for testing legacy code without refactoring it, so for example to mock newed dependency. The nice thing with jmockit is that it is a framework which can do mock in the style in mockito and has the abilities of powermock(mocking newed dependency, static, private methods).
    But maybe it is even too powerful, I mean having the possibility to do all these powermock things in the base framework could encourage too have a bad design in new development?
    Jeanne Boyarsky
    author & internet detective
    Marshal

    Joined: May 26, 2003
    Posts: 31062
        
    232

    Le Fre wrote:which feature do you think testng do not encourage good unit test?
    Actually the main advantage I see in testng as for now is to be able to prepare database settings in a super class in a non-static @beforeclass annotated method and still be able to hack this preparation in the subclass.

    1) That's not a unit test. A unit test doesn't access the database.
    2) The word "hack" doesn't scream great design to me
    3) I prefer doing the setup of the common database outside JUnit. (Run a main that does the common setup, calls your JUnit suite and does the common tear down.)

    The reason I don't like the approach you've described is that it creates dependencies between tests. While TestNG "solves" this problem, it encourages scenarios for the problem to exist that don't need to.
    Cedric Beust
    author
    Ranch Hand

    Joined: Oct 12, 2004
    Posts: 46
    Jeanne,

    Just because this feature exists doesn't mean you have to use it.

    Besides, when you embark on picking a testing framework, it's very likely that you will be using it to do both unit *and* functional testing. For the latter, dependencies can be useful, especially if you're using Selenium or doing some web testing. Check out the Selenium forums to understand how people use TestNG's dependencies to simplify their tests.

    As for your 3), it does sound like a hack to me. You created a workaround for the fact that JUnit doesn't support @BeforeSuite and @AfterSuite. It doesn't scream "great design" to me :-)

    Le: happy to answer your questions about TestNG, feel free to ask here or on the testng-users mailing-list.

    --
    Cédric

    Sagar Rohankar
    Ranch Hand

    Joined: Feb 19, 2008
    Posts: 2902
        
        1

    Jeanne Boyarsky wrote:TestNG shines in the complex tests with dependencies arena. A situation I don't want to find myself in.

    Great said, if you're finding unit testing difficult, time to refactor your design.

    Cedric Beust wrote:Besides, when you embark on picking a testing framework, it's very likely that you will be using it to do both unit *and* functional testing. For the latter, dependencies can be useful, especially if you're using Selenium or doing some web testing. Check out the Selenium forums to understand how people use TestNG's dependencies to simplify their tests.

    Yeah, I worked on selenium test where TestNG found to be a life saver, thanks for such great framework Cedric.


    [LEARNING bLOG] | [Freelance Web Designer] | [and "Rohan" is part of my surname]
    Jeanne Boyarsky
    author & internet detective
    Marshal

    Joined: May 26, 2003
    Posts: 31062
        
    232

    Cedric Beust wrote:Just because this feature exists doesn't mean you have to use it.

    I agree.

    Cedric Beust wrote:Besides, when you embark on picking a testing framework, it's very likely that you will be using it to do both unit *and* functional testing.

    The question was about unit testing so I answered that.

    Cedric Beust wrote:As for your 3), it does sound like a hack to me. You created a workaround for the fact that JUnit doesn't support @BeforeSuite and @AfterSuite. It doesn't scream "great design" to me :-)

    It is a hack. But it is an isolated hack in one place. I don't like my tests depending on each other because I have to look in multiple places for the database setup. Le talks about superclasses/subclasses for database setup which seems worse to me. if he had proposed actually using @BeforeSuite/@AfterSuite, it would have sounded better to me.
    Rogerio Liesenfeld
    Greenhorn

    Joined: Jan 24, 2011
    Posts: 3
    Jeanne, Le,

    I would like to address the concern that more powerful tools could somehow encourage "bad design".

    First of all, any tool, language, or API can be (and usually is) abused or misused. It's up to the people using them to learn how to do it properly. Otherwise, how would progress happen?

    In the specific case of mocking tools, consider what the ultimate goal is: to *enable* the testing of classes/components in isolation from "external" dependencies. (You choose which dependencies are internal or external to the unit under consideration, based on what exaclty you are trying to test.) A mocking tool/API, by itself, is not about encouraging or discouraging good/bad design. For that, we have development processes such as TDD, where a mocking tool can be used or not.

    Now, the simple fact of the matter is that "conventional" mocking tools (EasyMock, jMock, Mockito, etc.) do *not* enable unit testing to the point that is generally needed. A tool like JMockit does, and this was precisely the reason why I created it. I had a need to write unit tests which could not be written with those tools. And it wasn't a case of "poorly designed legacy code", but of certain coding/design practices which I happen to use a lot, and which I *want* to use because they proved their value (to me, at the very least).

    For instance, have a look at the example tests in the JMockit Tutorial (http://code.google.com/p/jmockit/source/browse/trunk/samples/tutorial). Can you really improve on those design choices? I don't think so. (That said, I am open to different ideas, but they would have to come with solid justifications.)

    Or if you want an independent opinion, also with a concrete example, read http://www.jakubkorab.net/2009/07/testing-the-untestable.html.


    Cheers,

    Rogerio
    Le Fre
    Greenhorn

    Joined: Aug 23, 2010
    Posts: 6

    Thanks for your replies Cédric and Rogerio. I am indeed convinced that the way to go would be TestNG and JMockit.

    TestNG, mainly because it supports a workflow which is much more close from conventional java and that tests can be organized with the full power of inheritance (hacking beforeclass or beforesuite). I do understand that other features of TestNG have advantages, in particular the possibility to test for prallelism. For other, I think their values have to be fully discovered by practicing it. For example, I was not so much convinced about dataproviders but having played with them a little, I find them handy, now. But I have still a lot to experiment with to have an idea of the different feature and their usefullness.

    JMockit, because as you underlined Rogerio, new or static methodd are not anti-patterns per se, and your framework brings the advantage of conventional mock frameworks without their limitations. I like as well the possibility to have easymock-style (StrictExpectations), and mockito style (NonStrictExpectations) available.

    I indeed presented your frameworks to my team together with JUnit, easymock and mockito, and tried to have an objective viewpoint. And they largely agreed with the conclusions I presented above.

    Now there is still some open questions.
    First as JMockit and TestNG are not compatible, is there some plans to bring the gap (is this for JMockit 1.0)? JMockit should be compatible with TestNG < 5.11. I did not even succeed to have this working, but I will try harder
    Could you give me some arguments for or against the use of JUnit in parallel with TestNG: the only one I see for now is for reporting, in particular a question is if code coverage can cope with both TestNG and JUnit for a same project.
    Rogerio Liesenfeld
    Greenhorn

    Joined: Jan 24, 2011
    Posts: 3
    JMockit supports both JUnit and TestNG, but not all versions of each test framework are covered. For JUnit, it must be version 4.5 or newer. For TestNG, currently only the 5.9, 5.10 and 5.11 versions are supported (provided "-javaagent:jmockit.jar" is used, or TestNG is configured with the JMockit initialization listener). Running tests with the Eclipse or IDEA plugin for TestNG fails if the plugin uses a newer version of TestNG internally.

    I will try to add support for TestNG 5.14 today.
    Le Fre
    Greenhorn

    Joined: Aug 23, 2010
    Posts: 6

    That would be great!
    Le Fre
    Greenhorn

    Joined: Aug 23, 2010
    Posts: 6

    Thank you Rogerio! That was fast!

    I have tested JMockit with TestNG 5.14 and it is working.

    To have it work under eclipse, I used the listener argument to my test runner (javaagent:jmockit.jar did not work for me)
    -listener mockit.integration.testng.Initializer
    The testng jar to compile the test and to run it have to be the same, so I replaced the testng.jar used by the testng eclipse plugin (on my box it is at eclipseDir/plugins/org.testng.eclipse_5.14.6/lib) by the one I use in my project.
    Cedric Beust
    author
    Ranch Hand

    Joined: Oct 12, 2004
    Posts: 46
    I am not aware of any incompatibility between JMockIt and TestNG, please let me know if you are having any problem and I'll be happy to take a look.

    Le Fre: I'm not sure why you need to replace the testng.jar that comes with the Eclipse plug-in, this should definitely not be necessary. Feel free to email me directly (cedric@beust.com) if you'd like to give me more details.
    Le Fre
    Greenhorn

    Joined: Aug 23, 2010
    Posts: 6

    By updating both the testng plugin for eclipse and jmockit to 0.999.6 (both update were necessary), running testng with jmockit imports works out of the box from within eclipse.
    tom pine
    Greenhorn

    Joined: Mar 10, 2011
    Posts: 3
    thanks for information...
     
    I agree. Here's the link: http://aspose.com/file-tools
     
    subject: TestNG vs JUnit, mockito + Powermock vs jmockit