aspose file tools*
The moose likes Testing and the fly likes Parameterizing test cases Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Java 8 in Action this week in the Java 8 forum!
JavaRanch » Java Forums » Engineering » Testing
Bookmark "Parameterizing test cases" Watch "Parameterizing test cases" New topic
Author

Parameterizing test cases

Layne Lund
Ranch Hand

Joined: Dec 06, 2001
Posts: 3061
I am using JUnit to write tests for a project I am currently working on. I am currently in a situation where I want to run basically the same test several times with different values for some of the variables (basically). I have several ideas how to approach this and am wondering if there are any "common practices".

Perhaps I can illustrate with some examples and pseudocode without getting into the details of my project. Let's say I have paramListA and paramListB that list the different values for two parameters for the test case I wish to run. Here are two different approaches that I can think of to implement this (in pseudocode):

1) Create individual instances of my TestCase for each pair of parameters. This would probably look something like


Using this approach, testMyMethod() would just run the test once on the parameters that are now stored as instance variables.

2) Another approach is to create a single instance of my test case and loop over the possible parameter values in a testXxx() method:


These are the two extremes. It also seems possible to split it up where I create instances for each value in just one of the parameter lists and iterate over the other one in testMyMethod().

Are there other ways to do this? And are there any guidelines that can help me decide which option to choose? I sure would appreciate any comments on this.

Layne


Java API Documentation
The Java Tutorial
Jeanne Boyarsky
internet detective
Marshal

Joined: May 26, 2003
Posts: 29287
    
140

Layne,
The first choice is preferred. In fact, JB Rainsberger named it the "Parameterized test case pattern". The advantage is that you can easily see which tests pass/fail and the tests are independent of each other.

The first choice also has the advantage of running all the tests in that test case using the provided values. I use this for checking a bunch of things (running a bunch of tests) on each of many different files.


[Blog] [JavaRanch FAQ] [How To Ask Questions The Smart Way] [Book Promos]
Blogging on Certs: SCEA Part 1, Part 2 & 3, Core Spring 3, OCAJP, OCPJP beta, TOGAF part 1 and part 2
Layne Lund
Ranch Hand

Joined: Dec 06, 2001
Posts: 3061
Thanks I think I will go with that. My main concern is that the first option can potentially use a lot of memory to create all those instances. If paramListA has m values and paramListB has n values then there will be m*n instances of the test case. In this particular project, both m and n have the potential to be rather large. Of course, I probably should gather some more precise statistics before I worry about it too much. But for the sake of argument, how should I mitigate the potential for large amounts of memory usage?

Layne
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Layne Lund:
My main concern is that the first option can potentially use a lot of memory to create all those instances. If paramListA has m values and paramListB has n values then there will be m*n instances of the test case. In this particular project, both m and n have the potential to be rather large. Of course, I probably should gather some more precise statistics before I worry about it too much. But for the sake of argument, how should I mitigate the potential for large amounts of memory usage?

Where is the contents of "paramListA" and "paramListB" coming from? Would it be possible to partition the test data and split them into two test case classes (extending an abstract test case class)?


Author of Test Driven (2007) and Effective Unit Testing (2013) [Blog] [HowToAskQuestionsOnJavaRanch]
Layne Lund
Ranch Hand

Joined: Dec 06, 2001
Posts: 3061
One of the lists of parameters is a list of files in a particular directory, so it comes from Files.listFiles(). I could potentiall process a lot of files for different variations on my test. For now I have about 80 files I am going to use, but I have many more that I could. Perhaps I'm overdoing it a bit.

I haven't decided about the other list of parameters yet. Some of the values are randomly generated. I also need some static values that are repeatable. Most likely these values will be read from a single file. (The other option is to hard code them into the test case, which is what I'm trying to get away from.)

I think I understand what you mean by "partitioning," but I don't see how this will save memory usage. I guess as described above, I could partition the random values for "list B" from the static ones. However, "list A" will remain the same for both. It seems like the same number of objects will be created. The only difference I see is that it is possible to limit how many objects are created at a given time. Is that what your suggesting or am I missing something?

Layne
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Layne Lund:
I think I understand what you mean by "partitioning," but I don't see how this will save memory usage. I guess as described above, I could partition the random values for "list B" from the static ones. However, "list A" will remain the same for both. It seems like the same number of objects will be created. The only difference I see is that it is possible to limit how many objects are created at a given time. Is that what your suggesting or am I missing something?

Exactly. By splitting the data to 2 separate TestCase instances, you're effectively making it possible for the JVM to garbage collect the data held by one test case while running the tests for another test case.

Another approach could be to process the data piece by piece, if your tests only ever need "one line of data" (or something like that) to execute. Instead of reading whole files into memory first and only then run the tests, you could read the necessary portions of the files into local variables which are ready to be garbage collected as soon as the test method returns.
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
There recently was a lot of discussion on support for parameterized test cases in JUnit 4.

If you are interested, take a look at http://groups.yahoo.com/group/junit/messagesearch?query=parameterized


The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
Jeanne Boyarsky
internet detective
Marshal

Joined: May 26, 2003
Posts: 29287
    
140

We have 1600 Test case objects in one of our projects without any memory issues. And yes, that is test case objects rather than tests.

So I suggest trying it and seeing what happens. If there are memory problems, the easiest workaround is to initialize your larger attributes to null in the tear downs. Or all the attributes in the parameterized test case could be set to null since there are more of them.
Layne Lund
Ranch Hand

Joined: Dec 06, 2001
Posts: 3061
Originally posted by Jeanne Boyarsky:
We have 1600 Test case objects in one of our projects without any memory issues. And yes, that is test case objects rather than tests.

So I suggest trying it and seeing what happens. If there are memory problems, the easiest workaround is to initialize your larger attributes to null in the tear downs. Or all the attributes in the parameterized test case could be set to null since there are more of them.


That's good to know. For now, I won't worry about it too much. One reason I'm slightly worried is that my main home computer is down and I'm delegated to development on an old P166 with 64MB RAM (if that much) until my new mobo comes in. Even without those limitations I try to be aware of these issues. Hopefully I will have a new machine running in the next week or so.

Layne
[ January 30, 2006: Message edited by: Layne Lund ]
Ernest Friedman-Hill
author and iconoclast
Marshal

Joined: Jul 08, 2003
Posts: 24168
    
  30

Originally posted by Jeanne Boyarsky:
We have 1600 Test case objects in one of our projects without any memory issues.


One of ours has over 17 thousand We run that test suite with a 600MB heap.


[Jess in Action][AskingGoodQuestions]
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Parameterizing test cases
 
Similar Threads
Looking for a testing tool that's "one level higher than JUnit"
Synchronization
Question on statistical analysis
Problem with static variables on a specific server
How to call Stored Procedure using jdbcTemplate and store result in object DTO/String/List/Result se