Win a copy of Re-engineering Legacy Software this week in the Refactoring forum
or Docker in Action in the Agile forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Parameterizing test cases

 
Layne Lund
Ranch Hand
Posts: 3061
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am using JUnit to write tests for a project I am currently working on. I am currently in a situation where I want to run basically the same test several times with different values for some of the variables (basically). I have several ideas how to approach this and am wondering if there are any "common practices".

Perhaps I can illustrate with some examples and pseudocode without getting into the details of my project. Let's say I have paramListA and paramListB that list the different values for two parameters for the test case I wish to run. Here are two different approaches that I can think of to implement this (in pseudocode):

1) Create individual instances of my TestCase for each pair of parameters. This would probably look something like


Using this approach, testMyMethod() would just run the test once on the parameters that are now stored as instance variables.

2) Another approach is to create a single instance of my test case and loop over the possible parameter values in a testXxx() method:


These are the two extremes. It also seems possible to split it up where I create instances for each value in just one of the parameter lists and iterate over the other one in testMyMethod().

Are there other ways to do this? And are there any guidelines that can help me decide which option to choose? I sure would appreciate any comments on this.

Layne
 
Jeanne Boyarsky
author & internet detective
Marshal
Posts: 34084
337
Eclipse IDE Java VI Editor
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Layne,
The first choice is preferred. In fact, JB Rainsberger named it the "Parameterized test case pattern". The advantage is that you can easily see which tests pass/fail and the tests are independent of each other.

The first choice also has the advantage of running all the tests in that test case using the provided values. I use this for checking a bunch of things (running a bunch of tests) on each of many different files.
 
Layne Lund
Ranch Hand
Posts: 3061
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks I think I will go with that. My main concern is that the first option can potentially use a lot of memory to create all those instances. If paramListA has m values and paramListB has n values then there will be m*n instances of the test case. In this particular project, both m and n have the potential to be rather large. Of course, I probably should gather some more precise statistics before I worry about it too much. But for the sake of argument, how should I mitigate the potential for large amounts of memory usage?

Layne
 
Lasse Koskela
author
Sheriff
Posts: 11962
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Layne Lund:
My main concern is that the first option can potentially use a lot of memory to create all those instances. If paramListA has m values and paramListB has n values then there will be m*n instances of the test case. In this particular project, both m and n have the potential to be rather large. Of course, I probably should gather some more precise statistics before I worry about it too much. But for the sake of argument, how should I mitigate the potential for large amounts of memory usage?

Where is the contents of "paramListA" and "paramListB" coming from? Would it be possible to partition the test data and split them into two test case classes (extending an abstract test case class)?
 
Layne Lund
Ranch Hand
Posts: 3061
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
One of the lists of parameters is a list of files in a particular directory, so it comes from Files.listFiles(). I could potentiall process a lot of files for different variations on my test. For now I have about 80 files I am going to use, but I have many more that I could. Perhaps I'm overdoing it a bit.

I haven't decided about the other list of parameters yet. Some of the values are randomly generated. I also need some static values that are repeatable. Most likely these values will be read from a single file. (The other option is to hard code them into the test case, which is what I'm trying to get away from.)

I think I understand what you mean by "partitioning," but I don't see how this will save memory usage. I guess as described above, I could partition the random values for "list B" from the static ones. However, "list A" will remain the same for both. It seems like the same number of objects will be created. The only difference I see is that it is possible to limit how many objects are created at a given time. Is that what your suggesting or am I missing something?

Layne
 
Lasse Koskela
author
Sheriff
Posts: 11962
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Layne Lund:
I think I understand what you mean by "partitioning," but I don't see how this will save memory usage. I guess as described above, I could partition the random values for "list B" from the static ones. However, "list A" will remain the same for both. It seems like the same number of objects will be created. The only difference I see is that it is possible to limit how many objects are created at a given time. Is that what your suggesting or am I missing something?

Exactly. By splitting the data to 2 separate TestCase instances, you're effectively making it possible for the JVM to garbage collect the data held by one test case while running the tests for another test case.

Another approach could be to process the data piece by piece, if your tests only ever need "one line of data" (or something like that) to execute. Instead of reading whole files into memory first and only then run the tests, you could read the necessary portions of the files into local variables which are ready to be garbage collected as soon as the test method returns.
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
There recently was a lot of discussion on support for parameterized test cases in JUnit 4.

If you are interested, take a look at http://groups.yahoo.com/group/junit/messagesearch?query=parameterized
 
Jeanne Boyarsky
author & internet detective
Marshal
Posts: 34084
337
Eclipse IDE Java VI Editor
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
We have 1600 Test case objects in one of our projects without any memory issues. And yes, that is test case objects rather than tests.

So I suggest trying it and seeing what happens. If there are memory problems, the easiest workaround is to initialize your larger attributes to null in the tear downs. Or all the attributes in the parameterized test case could be set to null since there are more of them.
 
Layne Lund
Ranch Hand
Posts: 3061
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Jeanne Boyarsky:
We have 1600 Test case objects in one of our projects without any memory issues. And yes, that is test case objects rather than tests.

So I suggest trying it and seeing what happens. If there are memory problems, the easiest workaround is to initialize your larger attributes to null in the tear downs. Or all the attributes in the parameterized test case could be set to null since there are more of them.


That's good to know. For now, I won't worry about it too much. One reason I'm slightly worried is that my main home computer is down and I'm delegated to development on an old P166 with 64MB RAM (if that much) until my new mobo comes in. Even without those limitations I try to be aware of these issues. Hopefully I will have a new machine running in the next week or so.

Layne
[ January 30, 2006: Message edited by: Layne Lund ]
 
Ernest Friedman-Hill
author and iconoclast
Marshal
Pie
Posts: 24208
35
Chrome Eclipse IDE Mac OS X
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Jeanne Boyarsky:
We have 1600 Test case objects in one of our projects without any memory issues.


One of ours has over 17 thousand We run that test suite with a 600MB heap.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic