Application Testing Techniques

[Mark Herschberg]

The other day, I was looking at using JUnit to do some testing of the code at my company. I knew what JUnit did, and, in general, it makes a lot of sense. But when I tried to make use of it, I encountered difficulties. This leads me to two questions. I'm particularly interested in the latter, because I think it is more difficult, but I suspect more people will be interested in the former. Hopefully we can discuss both here.

1) When and where do you use unit testing?
Well, the answer is obvious, for the units of your system, i.e. the methods. But the actual implementation of this can be tricky. Most software isn't as simple as a couple basic functions which can be tested for correctness. Often software, even that which is reasonably refactored, can be rather intertwined. Many functions involve complex objects. In some cases, to test a relatively basic function requires not only creating the object with the function, and the objects which the function uses as parameters, but also the factories or other objetcs which create the parameter objects, as well as other objetcs referenced in the paramter objects and the istance of the class being tested. These secondary objects may not be used in the test, but are needed during the contruction of the objects which are. Was that sentence complex and confusing? If yes, then you see my issue. If testing is that complex, then developers aren't going to write tests.
Can anyone suggest guidelines, tips, tricks, hints, or magical incantations which help people decide when unit testing makes sense, and how to apply it effectly, in these complex cases?
2) Unit Testing Application Frameworks
For the last 4 years, I've been doing framework design. This type of code tends to be more complex, because it involves misdirection. Often things aren't as clean as they should be once you get under the covers, because, well, you needed to organize them this way in order to get the cover on. So often times even the contrite and unhelpful response to question 1 of, "well, if you can't easily unit test your code, then you haven't refactored enough", is even less useful here. Making matters worse, application frameworks typically have an inversion of control. It expects objects to be plugged in and invoke the framework. The framework itself doesn't really do anything, rather it just organizes objects, and moves them around. Is unit testing even appropriate? If so, again I am looking for guidelines, tips, tricks, hints, or magical incantations.
Thanks.
--Mark
[e-mail]

[Cindy Glass]

What! You are going to stop there? Where is the stuff about interface testing, system testing, regression testing? These things I could really RANT about.
But unit testing is the basic groundwork of programming. IMHO unit testing must ALWAYS be done if code is touched, and it should be done extensively, each and every painful path that the touched code could follow, and the testing is only as good as the test writter can identify each and every problem situation that you are showing the code can handle. In a perfect world the coder should NOT be writting those tests.
Yes I know this sounds painful - it is. But it is also what separates the professionals from the garage hacks that sprout code overnight and put it out for the users to find the bugs.
I don't know much about your framework design stuff, but if it is run by code, and you have a set of requirements that the framework must adhere to, then you need to test. It just might be easier to do your "unit" testing in a system test context. You execute only enough tests to shake down the one class touched but execute the whole system/application/framework to validate them.

[Mark Herschberg]

Let me rephrase. Testing is important and necessary. Testing at different levels is important. I am unclear where to draw the lines between those levels. Specifically, what is appropriate for unit testing? There are really two questions.
A) What can and should be unit tested, in an ideal world?
B) What is easy to unit test?
I've become a believer lately in the XP/Refactoring-like philosophies o:, programers are lazy, and if there's something they don't like to do which is hard, they won't do it. If unit testing is difficult, coders won't write unit tests.

So what can I do to make my unit test writing easy? Specifically, how do I identify what to unit test, and how do I write the tests given the complexities described in my first post.

--Mark
[e-mail]

[Frank Carver]

OK, I'll take a stab at this one.
In your first point (complex interrelations), I feel the confusion comes because you don't mention the creation of mockups to help you with the testing. The feeling that you need to implement the whole system just to test a small part of it is one of the biggest hold-ups to implementing a unit test strategy. The ice-breaker for this is to grok that unit testing is not just about writing a series of method calls and asserts, but about producing some test code which will test the internals of your real code for you, however that needs to be done.
In the case you describe where a chunk of the complete system will have, for example: a class which we want to test; a method of that class requires a factory which generates objects of classes which implement an interface, and maybe a few other misc objects.
A factory which doesn't use an interface, but generates objects descended from a (maybe abstract) base class is usually a bad design decision, becaue it is much harder to test, so I'll assume that you use an interface. If you don't, then converting shouldn't be too hard.
Step one in unit testing this scenario is to write some basic unit tests for the interface used by the factory and its methods then create a mock class which implements the interface and passes the tests. Then write some tests for the factory class, and produce a mock factory which passes them by producing the mock objects written earlier. You can now write tests for classes and methods which require a factory, and give them the mock factory to work with. Similarly for any other objects needed by the method under test.
For your second point, the use of what I call "crash-test dummies" rather than just simple mockups is usually more appropriate. Consider a modern crash-test dummy. It sits in the vehicle under test, packed full of sensors and connected to an array of data-logging and decision making gear. When building and testing a framework, these are vital pieces of equipment. I'll take a very simple container framework as an example, but most dynamic frameworks are usually amenable to the same approach.
Stage one, of course, is to write some tests for the operation of your crash-test dummy. You are going to rely on its results later, so it's vital to trust it. Imagine all contained classes implement the following simple interface:

public interface Action
{
  public void init(ActionContext context);
  public boolean process(String command);
  public void destroy();
}
 
with
 
public interface ActionContext
{
  public String getRoot();
  public String getProperty(String name);
  public void setProperty(String name);
}

Following my points above, the first step is to create a mock ActionContext (I'll skip the tests for it, it's pretty self-contained, so they are not hard to do):

public class MockActionContext implements ActionContext
{
  public String root;
  public Properties properties;
  public MockActionContext(String root)
  {
    properties = new Properties();
  }
 
  public String getRoot()
  {
    return root;
  }
 
  public String getProperty(String name)
  {
    return properties.getProperty(name);
  }
 
  public void setProperty(String name, String value)
  {
    properties.put(name, value);
  }
}

Not that contrary to my usual practice, the member variables are all public. This class is designed solely for testing, so I deliberately make its internals visible to the test harness. Note also that the "real" Action may end up very similar to this (although probably with private members) but I am deliberately ignoring thinking about that now. That's a different story, for later. One thing at a time.
Now we can write a very simple dummy Action:

public class DummyAction implements action
{
  public void init(ActionContext context)
  {
  }
 
  public boolean process(String command)
  {
    return true;
  }
  
  public void destroy()
  {
  }
}

At the moment, this is still just a "Mock", rather than a "Dummy". In order to make it more useful, we need to decide on the kind of "instrumentation" we need. In this case I'll cut to the chase and add a few methods, but in the real world you'd need to spend some thought and develop these as you go along writing your tests.
The tests might include (I've omitted the JUnit preamble)

ActionContext c = new MockActionContext();
DummyAction a = new DummyAction("A");
assert(null == a.context);
assertEquals(DummyAction.NEW, a.status); 
 
a.init(c);
assert(c == a.context);
assertEquals(DummyAction.INITED, a.status); 
 
boolean b = a.process("Hello");
assert(!b);
assertEquals(null, a.lastArg);
assertEquals(1, a.callCount);
b = a.process("Act Now!");
assert(b);
assertEquals("Act Now!", a.lastArg);
assertEquals(2, a.callCount);
 
boolean b = a.process("Hello");
assert(!b);
assertEquals("Act Now!", a.lastArg);
assertEquals(3, a.callCount);
 
a.destroy();
assertEquals(DummyAction.DESTROYED, a.status); 

and a class which passes these tests might look like

public class DummyAction implements Action
{
  public static final int NEW = 0;
  public static final int INITED = 1;
  public static final int DESTROYED = 2;
 
  public int status = NEW;
  public ActionContext context = null;
  public int callCount = 0;
  public String lastArg = null;
  public String prefix = null;
  
  public DummyAction(String prefix)
  {
    this.prefix = prefix;
  }
 
  public void init(ActionContext context)
  {
    this.context = context;
    status = INITED;
  }
 
  public boolean process(String command)
  {
    ++callCount;
    boolean ret = command.startsWith(prefix);
    if (ret)
    {
      lastArg = command;
    }
    return ret;
  }
  
  public void destroy()
  {
    status = DESTROYED;
  }
}

Why is all this useful? Well, we can now test the framework, and look "inside" the dummy at each step to make sure things are working OK. Imagine a very simple framework which has a few operations - 1. add an Action and call its init method with the current context, 2. process a String, passing it to each action in turn and returning true if any of the actions returned true, 3. remove a specified action, 4. remove all actions. I won't go into great detail here, as your framework will no doubt be different, but consider the following unit test examples:

DummyAction a1 = new DummyAction("H");
DummyAction a2 = new DummyAction("A");
Framework f = new Framework();
assertEquals(false, f.process("Act Now!");
assertEquals(false, f.process("Hello");
 
f.addAction(a1);
assertEquals(DummyAction.INITED, a1.status);
assertEquals(0, a1.callCount);
 
assertEquals(false, f.process("Act Now!");
assertEquals(1, a1.callCount);
assertEquals(null, a1.lastArg);
 
assertEquals(true, f.process("Hello");
assertEquals(2, a1.callCount);
assertEquals("Hello", a1.lastArg);
 
f.addAction(a2);
assertEquals(DummyAction.INITED, a2.status);
assertEquals(0, a2.callCount);
 
assertEquals(true, f.process("Achtung");
assertEquals(3, a1.callCount);
assertEquals(1, a2.callCount);
assertEquals("Hello", a1.lastArg);
assertEquals("Achtung", a2.lastArg);
 
assertEquals(true, f.process("Happiness");
assertEquals(4, a1.callCount);
assertEquals(2, a2.callCount);
assertEquals("Happiness", a1.lastArg);
assertEquals("Achtung", a2.lastArg);
 
f.remove(a1);
assertEquals(DummyAction.DESTROYED, a1.status);
 
assertEquals(false, f.process("Hit");
assertEquals(4, a1.callCount);
assertEquals(3, a2.callCount);
assertEquals("Happiness", a1.lastArg);
assertEquals("Achtung", a2.lastArg);
 
f.removeAll();
assertEquals(DummyAction.DESTROYED, a2.status);
 
assertEquals(false, f.process("Aargh");
assertEquals(4, a1.callCount);
assertEquals(3, a2.callCount);
assertEquals("Happiness", a1.lastArg);
assertEquals("Achtung", a2.lastArg);

This may have been a bit rambling, but I hope you get the idea of unit testing frameworks using mock classes and instrumented dummy objects.
Has it helped at all?