There are a huge number of possible test cases for even a relatively simple telephony feature when multiple interop combinations are involved.
Look at a simple two-way voice call: just pick up the phone (“A”), dial a number, waiting for ringing, answer (“B”), and make sure you can talk. Easy, right?
The Baseline Test
First we have different possibilities for the identity of A and B:
- A and B are both “onnet”
- A is “offnet”, B is onnet.
- B is offnet, A is onnet.
Then we have a few different possible call flows:
- B answers after one ring (the “happy path”).
- B is busy.
- B does not answer. (A hangs up after a few rings.)
- B picks up after half a ring (mid-ring).
- B picks up after two rings.
There are probably more conditions we can come up with, but lets look at these. The first grouping gives three different tests to run, which isn’t too bad. The five call flows in the second grouping make for 15 total test combinations. At about 30 seconds per call, that’s 7.5 minutes to run a suite for this feature.
Mix it Up With Different Gear
It gets worse quickly if you’re an interop tester working with various models of gear.
- Cisco ATA
- Netopia ATA
- Asterisk
- Avaya
- 3Com
The 15 tests in the matrix mentioned above become 30 when there are two different adapters to test, and that’s assuming a homogenous network. If the Cisco and Netopia are on the same net, there are four combinations of “A – B” to test, meaning 60 tests to run. Three possible switches would mean 180 tests. Now we’re talking about an hour and a half to run the test suite, longer if you’ve got to change physical connections to your call generator. (VoIP gateway testers have an advantage that the connections are virtual and “changing” them is software controlled.)
Overload!
Whoa! That’s too much to think about. We haven’t even tested anything interesting yet. No feature interactions, caller id/name, call waiting, various flavors of forwarding, etc.
Three-line features build bigger test matrices even faster: a three-way-call test with just five flows and the same set of equipment above gives 840 combinations to test — seven hours of testing for a single feature. And TWC will definitely have more than five flows! A dozen flows would take over a dozen hours to test.
How do we manage all of this? The typical technique I’ve seen is to ignore most of the matrix and focus on the most common flows with the most common combinations of features. This makes a lot of sense from one perspective: if something is wrong with the common case, then there’s a real problem. Also, this probably represents most of your customer base.
Looking at it another way, though, it doesn’t make much sense. If the point of testing is to find problems, then we should define a “successful” test case as one that finds an error. Through this lens, the usual technique looks like a lot of wasted time repeatedly running the same batch of passing (“unsuccessful”) tests. It would be more profitable to instead run the odd, boundary probing, edge cases. Here, we can have a high comfort level that if these are passing the common cases are passing too.
Simplify and Find More Bugs
This implies that we can whittle down the matrix to something reasonable:
- Happy path, Cisco to Netopia, 2 rings, Asterisk.
- Happy, offnet to Netopia, 1 ring, Avaya.
- Happy, Cisco to offnet, half ring, 3Com.
- Happy, Netopia to Cisco onnet, 2 rings, Asterisk.
- Happy, offnet to Cisco, 1 ring, Avaya.
- Happy, Netopia to offnet, half ring, 3Com.
- Busy, Cisco to Netopia, Asterisk.
- Busy, Netopia to Cisco, Avaya.
- Busy, Cisco to offnet, 3Com.
- Busy, Netopia to offnet, Asterisk.
- No Answer, Cisco to offnet, Avaya.
- No Answer, Netopia to Cisco, 3Com
This reduces the set to 1/15 of the original cases. It is not comprehensive but it does touch all the devices and all the flows. When the situation calls for comprehensive testing — maybe for interop against a new switch — you’d want to run the full matrix. I chose the tests for the reduced test suite on an intuitive basis, just picking combinations that touched each matrix dimension. If you know through experience that certain combinations are troublesome, make sure they get a spot in the reduced test suite — possibly bumping another less likely test case.
Lastly, if you’re running tests daily, say on new software builds, you can achieve full matrix coverage over time by running different reduced suites on your call generator every day. Partition the full matrix into N distinct subsets, where N is the number of days to spread out the tests. Each subset should touch all the flows, but it may not use all of the equipment — you can connect new equipment to the call generator each day to achieve the full set of combinations.
Related posts:
