aspose file tools*
The moose likes Testing and the fly likes junit test cases which fail Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Spring in Action this week in the Spring forum!
JavaRanch » Java Forums » Engineering » Testing
Bookmark "junit test cases which fail" Watch "junit test cases which fail" New topic
Author

junit test cases which fail

Kalpesh Soni
Ranch Hand

Joined: Jan 02, 2001
Posts: 312
We are using junit for writing test cases
I am writing some test cases which i intentionally want to fail (I am passing bad values to functions)
If I use fail () then the report shows them as failures
How do i distinguish between these failures and the ACTUAL failures ???
:roll:
[ March 12, 2002: Message edited by: kalpesh soni ]

Test 094, IBM WID 6.0 cert
SCJP 1.2
SCBCD 1.3 Beta
SCWCD 1.4 Beta
SCMAD Beta
SCDJWS Beta

KS
Wilfried LAURENT
Ranch Hand

Joined: Jul 13, 2001
Posts: 269
Write your test as if you want it to succeed.
Put the code in a try catch (AssertionFailed) block. In the catch statement, put a assertTrue(true);
Embody the above code in a testMethod. If your test fails, then, well , your test is succesful.
That's it.
W.
[ March 12, 2002: Message edited by: Wilfried LAURENT ]
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Of course you can ommit the assertTrue(true). For example:


The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
timothy zimmerman
Ranch Hand

Joined: Jun 26, 2001
Posts: 149
Let me start by saying that I have only a little experience with JUnit but a reasonable amount of experience testing code.
Both of the prevoius suggestions will allow your test to pass when it 'fails'. But here's my $.02:
You did not say how your test would fail when provided bad input. Just like a positive test in which you give valid input and expect a known ouptut, with a negative test you provide invalid input and you should have a known output. Whether that output is an error message or whatever you have told the application to do when receiving such bad input.
The concern that I would have with the previous posts is that you are catching exceptions and assuming that they were caused by the invalid input and that your test has passed (meaning it should have thrown an exception). But, what if something else caused that exception to be thrown? You might consider writing your own, more specific, exception. One which would only result from your invalid input and then, if you catch that exception you can consider the test a success.
Craig Demyanovich
Ranch Hand

Joined: Sep 25, 2000
Posts: 173
Since unit testing is, by definition, white-box testing, some people validate the contents of the caught exception. For example:

Consider in what other ways you could accomplish verifying that the exception you catch is caused for the reason that you expect.
Craig
[ March 12, 2002: Message edited by: Craig Demyanovich ]
timothy zimmerman
Ranch Hand

Joined: Jun 26, 2001
Posts: 149
Thank you Craig. That is the point I was trying to make but you made it much more succinclty than I did.
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by timothy zimmerman:
The concern that I would have with the previous posts is that you are catching exceptions and assuming that they were caused by the invalid input and that your test has passed (meaning it should have thrown an exception). But, what if something else caused that exception to be thrown?

Well, if this "something else" will always cause an exception when providing invalid input, I wouldn't care. How do I know wether it will always throw an exception?
I do know because I develop "test-first". That is, I will first write the test and throw it against the code before adding any code to handle this case. Of course I would expect the test to fail. When I added the code that throws the exception, I do know that it is thrown because of the invalid input.
Of course sometimes it happens that the code already throws the exception. In this case I take a closer look at my code to decide if I am either already satisfied with the cause of the exception, or should change my test so that it fails and forces me to do the right thing to the production code.
Tavia Young
Ranch Hand

Joined: Mar 15, 2004
Posts: 31
With regards to testing for null values that are passed into a method, would the logical step be to write a test to initially catch the NullPointerException (i.e. as Craig mentions above) and then test for the case in which such parameters are not null? How would these two tests be able to co-exist in the TestCase?

Thanks much!
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Tavia Young:
With regards to testing for null values that are passed into a method, would the logical step be to write a test to initially catch the NullPointerException (i.e. as Craig mentions above) and then test for the case in which such parameters are not null? How would these two tests be able to co-exist in the TestCase?

They would co-exist in the TestCase as separate test methods:



Author of Test Driven (2007) and Effective Unit Testing (2013) [Blog] [HowToAskQuestionsOnJavaRanch]
Tavia Young
Ranch Hand

Joined: Mar 15, 2004
Posts: 31
Right, I see.

In the example below, if division by zero occurs, the test will fail. Is this failure a sign of "success" that the code accounts for null input? Should both tests be written to pass?
Tavia Young
Ranch Hand

Joined: Mar 15, 2004
Posts: 31
Oops - I misinterpreted the example that Lasse provided above. If division by zero occurs, it should catch the InvalidArgumentException and the test will pass.

But, what happens in the case if the Exception you're looking for is nested?
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by Tavia Young:
But, what happens in the case if the Exception you're looking for is nested?


Than you catch the outer exception and in the catch block assert that it contains the expected nested exception.
Tavia Young
Ranch Hand

Joined: Mar 15, 2004
Posts: 31
Thank you, Ilja.

I'm still somewhat confused. Using Lasse's example above, in the case where the test passes, this would mean that the code does not check for invalid input (i.e. the InvalidArgumentException). If the code is changed to check for invalid input, the test fails.
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Ilja Preuss:
Than you catch the outer exception and in the catch block assert that it contains the expected nested exception.

In other words (or language;):
Tavia Young
Ranch Hand

Joined: Mar 15, 2004
Posts: 31
Thanks, Lasse. You're so quick! =)

If the testInvalidInput() passes, this would mean that the code does not check for invalid input?! My confusion is that if I alter the code to check for null values, this test would fail.

I hope you could shed more light on this for me. Thanks again!
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Tavia Young:
If the testInvalidInput() passes, this would mean that the code does not check for invalid input?! My confusion is that if I alter the code to check for null values, this test would fail.

No, actually.

In this case, getting the specific exception we're catching means that the code is working as expected -- our test specifies that we want our code to throw an IllegalArgumentException if trying to divide by zero. If the code would not check for invalid values (in our case, a zero, in some other cases, perhaps a null or an empty string, etc.), the method call would throw a java.lang.ArithmeticException. Our test would not catch that particular exception and JUnit would flag the method as a failed one (a test method throwing anything is considered to fail (to be more specific, an error, while throwing AssertionFailedError leads to a fail)).

I'm sorry if that paragraph became a bit confusing
Tavia Young
Ranch Hand

Joined: Mar 15, 2004
Posts: 31
I see your point, Lasse. Thanks again for the explanation. By testing for the specific exception, (e.g. a NullPointerException in the case where a parameter passed to a method results in a null) we are verifying the code does throw this exception. If code were written to include a check (e.g. parameter != null), this would lead to a fail. So, would it be correct to then check for the expected exception thrown when the parameter is not null?

I am testing this within a J2EE environment and I also get an error from the EJB even though the unit tests pass?
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Tavia Young:
By testing for the specific exception, (e.g. a NullPointerException in the case where a parameter passed to a method results in a null) we are verifying the code does throw this exception. If code were written to include a check (e.g. parameter != null), this would lead to a fail. So, would it be correct to then check for the expected exception thrown when the parameter is not null?

If the parameter is not null, the code shouldn't throw an exception. You could surround the method call with an "empty" try-catch clause but that wouldn't buy you anything extra. I'm trying to illustrate with a couple of examples:


Both of these methods raise a red flag if the 'doSomething()' method throws a NullPointerException. The only differences are that 1) testNumberOne is more verbose and 2) testNumberTwo would fail as an "error" while testNumberOne would fail as a "failed test". I would prefer the approach in testNumberTwo.

Originally posted by Tavia Young:
I am testing this within a J2EE environment and I also get an error from the EJB even though the unit tests pass?

What kind of error?
Tavia Young
Ranch Hand

Joined: Mar 15, 2004
Posts: 31
The examples are helpful, Lasse. The confusing bit for me still is if the code included a line such as if(param != null), the test fails.

The additional error generated is due to the org.mockejb.ExceptionHandler intercept as I am testing outside the container with mockEJB's. Is there any reason why this error is raised?

Thanks again!
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by Tavia Young:
The examples are helpful, Lasse. The confusing bit for me still is if the code included a line such as if(param != null), the test fails.


Well, either you expect the method to throw an exception, than including the "if" line certainly *is* an error - or your test simply is wrong.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: junit test cases which fail