• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Automatically repeating tests with different configurations

 
author
Posts: 5856
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I have a situation where I would like to repeat a set of tests but with a different configuration. For example, assume a test class with 4 test cases, and where I have three different possible configurations (all valid at the same time). What I would like to do is run the test class three times, once with each configuration and have the junit test report show a total of 12 test cases being run.

I've looked at using TestSuite and the JUnit4TestAdapter class to define the tests to run, and using a for loop to run the tests multiple times changing the configuration between runs. And that works fairly well, but reporting is lacking (shows up as only a single test in the report). I have also looked into using the @Suite.SuiteClasses annotation, but that allows running the test cases only once; I guess I could declare 3 test suite classes with the @Suite.SuiteCLasses annotation and in each one set up the configuration but that isn't much better than the workaround I am using right now (more on the workaround later).

Of course, what I have given here is a simple example. I really have about a dozen classes with well over a hundred test cases that I need to run in 5 different configurations. To make things even more interesting, I have about another 50 test classes with perhaps 6-700 test cases that are configuration independent. My goal is, in a single integration test run, to run the configuration independent tests once and the configuration dependent test cases 5 time each, each time with a different configuration, and to end up with a single report showing the total number of individual test cases run (approx 1200 test cases total).

My current workaround is for the configuration dependent test classes, I have an abstract class with the test cases defined, then 5 subclasses each with only an @BeforeClass methods that sets the configuration. This works. But it is a miantenance headache if I have to add a new configuration (need to create a lot of new subclasses) and is also a pain when in a particular test environment only a subset of the configurations are available (right now I have the @BeforeClass methods checking if the configuration is present). I do have an API where I can determine the available configurations, so ideally I'd like to do a loop that runs the configuration specific tests once for each available configuration. And as I mentioned I can run the tests in a loop, but the final report doesn't include the individual tests. (If I knew how to grab the TestResult object being used by Maven to report on the tests, that would probably work for me as I could pass it off to TestSuite.run().)

Any ideas, thoughts or suggestions would be appreciated.

By the way, I'm using JUnit 4 and running the tests within Maven.
 
Sheriff
Posts: 5555
326
IntelliJ IDE Python Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I'm not sure about the configuration part of your question, but if you're looking for something that will feed a test method with lots of data then junitparams may be of interest. Or if you like a bit of groovy in your tests then Spock is even better. Both of those will report individual results if you ask it.
 
Peter Johnson
author
Posts: 5856
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
junitparams looks interesting, now if I could only place the @Parameters annotation of the @BeforeClass or @Before method, and that in turn would cause all the test cases to be rerun, that would be ideal. But I can't see annotating over a hundred test cases with the exact same @Parameters annotation, that would again be a maintenance headache if another configuration gets added.

Spock, on the use hand, would require some more investigation because I din't see anything obvious that would would do what I need.

But thanks for the suggestions, it's appreciated!
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic