Black and white box unit testing techniques

If you have ever written a unit test, chance has it that you used either black or white box testing techniques to determine what that unit test was testing. In this post I want to explore both these techniques and look at when we should be using them.

Black-box techniques

Black-box testing is were we use the contract (api) and/or a functional specification of a method to determine how to test it. We can think of this as testing the method's inputs and outputs. It requires us to look at the various possible input values that exists, split into equivalence partitions and test the efforts that they have on the method's output. Let's take this example of a functional specification for a method testing that someone's username is valid:

“This method must ensure that the username passed is greater than 12 characters in length and less than 27 characters (leading or trailing whitespaces will be stripped before checking length). If the string is valid return a boolean value of TRUE else return FALSE”

There is a lot happening in the above specification requirement for our method but what caught my attention was:

"greater than 12 characters in length and less than 27 characters"

This tells us that we have 3 possible input groups:

1. 12 or less characters is invalid
2. Between 12 and 27 characters is valid
3. 27 or more characters is invalid

So we know that we at least 3 unit tests to cover these possible input groups. Ok so armed with this knowledge let's break down the rest of the requirement into individual unit tests. As I’m optimistic I will start with the happy paths (A happy path is where the method executes and a non-error based outcome is the results):

1. We pass a username that is in the valid character range and check that the method outcome is TRUE
2. We pass a username that is one the edge of the valid lower boundary and check that the method outcome is TRUE
3. We pass a username that is one the edge of the valid upper boundary and check that the method outcome is TRUE
4. We pass a username that contains leading and trailing whitespaces that exceeds the valid upper boundary with the whitespace (but is under it without the white space) and check that the outcome is TRUE

So we have 4 valid test cases that can be transformed into unit test - all without actually seeing a single line of this method.

Now for the unhappy paths (An unhappy path is where the method executes and an expected error based outcome is the result):

1. We pass a username that is below the lower boundary value and check that the method outcome is FALSE
2. We pass a username that is above the upper boundary value and check that the method outcome is FALSE
3. We pass a username parameter that only contains whitespace and check that the outcome is FALSE

As we can see from that simple specification requirement we have produced a number of happy and unhappy path test cases to ensure that we are producing a method that will fulfil the functional specification.

Ok, so let's see the method's signature (contract) and work out if we have anymore test cases

- (BOOL)isUsernameValid:(NSString *)username

The only additional test case that I can think of is an unhappy path:

1. We pass a nil username and check that the method outcome is FALSE

White-box techniques

White-box testing is were we use the implementation of a method to ensure that we exercise every unique branch through that method even if two different input groups give us the same output value. Let's take the example above and look at a potential method implementation.

- (BOOL)isUsernameValid:(NSString *)username
{
    if (!username) 
    {
        return NO;
    }
    
    NSString *trimmedUsername = [username stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceCharacterSet]];
    
    if ([trimmedUsername length] > 12 && [trimmedUsername length] < 27) 
    {
        return YES;
    }
    
    return NO; 
}

In the above code snippet we can reinforce the need for the one additional test case that we identified in the signature black-box analysis:

1. We pass a nil username and check that the method outcome is FALSE

So using a combination of black and white box testing techniques we have identified the test cases required to determine that when a username is supplied if it is valid or invalid.

To white-box test or not that is the question

In the simple example above white-box testing only reinforces the test cases already identified during analysis using the black-box testing technique however what if that method had a side effect and set the value of another private (internal) property:

should that be tested?

This is the danger with white-box testing, it's possible to very tightly couple your unit tests to the implementation of the method and make every change to that method a very expensive process (as you need to update a whole raft of unit tests). Because of this I tend not to use white-box testing to define tests cases but rather only to confirm the validity of any test cases that I felt were questionable when discovered using black-box analysis techniques.

What do you think? Let me know by getting in touch on Twitter - @wibosco