junit test cases which fail

[Kalpesh Soni]

We are using junit for writing test cases
I am writing some test cases which i intentionally want to fail (I am passing bad values to functions)
If I use fail () then the report shows them as failures
How do i distinguish between these failures and the ACTUAL failures ???
:roll:
[ March 12, 2002: Message edited by: kalpesh soni ]

[Wilfried LAURENT]

Write your test as if you want it to succeed.
Put the code in a try catch (AssertionFailed) block. In the catch statement, put a assertTrue(true);
Embody the above code in a testMethod. If your test fails, then, well , your test is succesful.
That's it.
W.
[ March 12, 2002: Message edited by: Wilfried LAURENT ]

[Ilja Preuss]

Of course you can ommit the assertTrue(true). For example:

public void testNPE() {
  try {
    myObject.myMethod(null);
    fail("NPE expected");
  } catch(NullPointerException expected) {
  }
}

[timothy zimmerman]

Let me start by saying that I have only a little experience with JUnit but a reasonable amount of experience testing code.
Both of the prevoius suggestions will allow your test to pass when it 'fails'. But here's my $.02:
You did not say how your test would fail when provided bad input. Just like a positive test in which you give valid input and expect a known ouptut, with a negative test you provide invalid input and you should have a known output. Whether that output is an error message or whatever you have told the application to do when receiving such bad input.
The concern that I would have with the previous posts is that you are catching exceptions and assuming that they were caused by the invalid input and that your test has passed (meaning it should have thrown an exception). But, what if something else caused that exception to be thrown? You might consider writing your own, more specific, exception. One which would only result from your invalid input and then, if you catch that exception you can consider the test a success.

[Craig Demyanovich]

Since unit testing is, by definition, white-box testing, some people validate the contents of the caught exception. For example:

try {
    new Person(null);
    fail("Created Person with null name");
} catch(NullPointerException success) {
    assertTrue(success.getMessage().indexOf("null name") > -1)
}

Consider in what other ways you could accomplish verifying that the exception you catch is caused for the reason that you expect.
Craig
[ March 12, 2002: Message edited by: Craig Demyanovich ]

[timothy zimmerman]

Thank you Craig. That is the point I was trying to make but you made it much more succinclty than I did.

[Ilja Preuss]

Originally posted by timothy zimmerman:
The concern that I would have with the previous posts is that you are catching exceptions and assuming that they were caused by the invalid input and that your test has passed (meaning it should have thrown an exception). But, what if something else caused that exception to be thrown?

Well, if this "something else" will always cause an exception when providing invalid input, I wouldn't care. How do I know wether it will always throw an exception?
I do know because I develop "test-first". That is, I will first write the test and throw it against the code before adding any code to handle this case. Of course I would expect the test to fail. When I added the code that throws the exception, I do know that it is thrown because of the invalid input.
Of course sometimes it happens that the code already throws the exception. In this case I take a closer look at my code to decide if I am either already satisfied with the cause of the exception, or should change my test so that it fails and forces me to do the right thing to the production code.

[Tavia Young]

With regards to testing for null values that are passed into a method, would the logical step be to write a test to initially catch the NullPointerException (i.e. as Craig mentions above) and then test for the case in which such parameters are not null? How would these two tests be able to co-exist in the TestCase?

Thanks much!

[Lasse Koskela]

Originally posted by Tavia Young:
With regards to testing for null values that are passed into a method, would the logical step be to write a test to initially catch the NullPointerException (i.e. as Craig mentions above) and then test for the case in which such parameters are not null? How would these two tests be able to co-exist in the TestCase?

They would co-exist in the TestCase as separate test methods:

public class MyTest extends TestCase {
 
    public void testInvalidInput() throws Exception {
        try {
            MyCalculator.divide(10, 0);
            fail("Dividing with zero should cause an exception!");
        } catch (InvalidArgumentException expected) {
        }
    }
 
    public void testValidInput() throws Exception {
        assertEquals(5.0d, MyCalculator.divide(10, 2), 0.000001d);
    }
}

[Tavia Young]

Right, I see.

In the example below, if division by zero occurs, the test will fail. Is this failure a sign of "success" that the code accounts for null input? Should both tests be written to pass?

[Tavia Young]

Oops - I misinterpreted the example that Lasse provided above. If division by zero occurs, it should catch the InvalidArgumentException and the test will pass.

But, what happens in the case if the Exception you're looking for is nested?

[Ilja Preuss]

Originally posted by Tavia Young:
But, what happens in the case if the Exception you're looking for is nested?

Than you catch the outer exception and in the catch block assert that it contains the expected nested exception.

[Tavia Young]

Thank you, Ilja.

I'm still somewhat confused. Using Lasse's example above, in the case where the test passes, this would mean that the code does not check for invalid input (i.e. the InvalidArgumentException). If the code is changed to check for invalid input, the test fails.

[Lasse Koskela]

Originally posted by Ilja Preuss:
Than you catch the outer exception and in the catch block assert that it contains the expected nested exception.

In other words (or language;):

public class MyTest extends TestCase {
 
    public void testInvalidInput() throws Exception {
        try {
            MyCalculator.divide(10, 0);
            fail("Dividing with zero should cause an exception!");
        } catch (NestedException nested) {
            // asserting that the root cause was what we expect
            assertEquals(RootException.class, nested.getRootCause().getClass());
        }
    }
 
}

[Tavia Young]

Thanks, Lasse. You're so quick! =)

If the testInvalidInput() passes, this would mean that the code does not check for invalid input?! My confusion is that if I alter the code to check for null values, this test would fail.

I hope you could shed more light on this for me. Thanks again!

[Lasse Koskela]

Originally posted by Tavia Young:
If the testInvalidInput() passes, this would mean that the code does not check for invalid input?! My confusion is that if I alter the code to check for null values, this test would fail.

No, actually.

In this case, getting the specific exception we're catching means that the code is working as expected -- our test specifies that we want our code to throw an IllegalArgumentException if trying to divide by zero. If the code would not check for invalid values (in our case, a zero, in some other cases, perhaps a null or an empty string, etc.), the method call would throw a java.lang.ArithmeticException. Our test would not catch that particular exception and JUnit would flag the method as a failed one (a test method throwing anything is considered to fail (to be more specific, an error, while throwing AssertionFailedError leads to a fail)).

I'm sorry if that paragraph became a bit confusing

[Tavia Young]

I see your point, Lasse. Thanks again for the explanation. By testing for the specific exception, (e.g. a NullPointerException in the case where a parameter passed to a method results in a null) we are verifying the code does throw this exception. If code were written to include a check (e.g. parameter != null), this would lead to a fail. So, would it be correct to then check for the expected exception thrown when the parameter is not null?

I am testing this within a J2EE environment and I also get an error from the EJB even though the unit tests pass?

[Lasse Koskela]

Originally posted by Tavia Young:
By testing for the specific exception, (e.g. a NullPointerException in the case where a parameter passed to a method results in a null) we are verifying the code does throw this exception. If code were written to include a check (e.g. parameter != null), this would lead to a fail. So, would it be correct to then check for the expected exception thrown when the parameter is not null?

If the parameter is not null, the code shouldn't throw an exception. You could surround the method call with an "empty" try-catch clause but that wouldn't buy you anything extra. I'm trying to illustrate with a couple of examples:

public void testNumberOne() throws Exception {
    try {
        objectBeingTested.doSomething("a non-null input");
    } catch (NullPointerException e) {
        fail("Shouldn't have thrown a NPE!");
    }
}
 
public void testNumberTwo() throws Exception {
    objectBeingTested.doSomething("a non-null input");
}

Both of these methods raise a red flag if the 'doSomething()' method throws a NullPointerException. The only differences are that 1) testNumberOne is more verbose and 2) testNumberTwo would fail as an "error" while testNumberOne would fail as a "failed test". I would prefer the approach in testNumberTwo.

Originally posted by Tavia Young:
I am testing this within a J2EE environment and I also get an error from the EJB even though the unit tests pass?

What kind of error?

[Tavia Young]

The examples are helpful, Lasse. The confusing bit for me still is if the code included a line such as if(param != null), the test fails.

The additional error generated is due to the org.mockejb.ExceptionHandler intercept as I am testing outside the container with mockEJB's. Is there any reason why this error is raised?

Thanks again!

[Ilja Preuss]

Originally posted by Tavia Young:
The examples are helpful, Lasse. The confusing bit for me still is if the code included a line such as if(param != null), the test fails.

Well, either you expect the method to throw an exception, than including the "if" line certainly *is* an error - or your test simply is wrong.