We know that call generators are handy for finding call flows that break with odd combinations of features.
Sometimes a feature is broken in a way that the call generator can’t detect right away. For instance, let’s say the device under test (DUT) leaks memory with a given call flow. The call will work hundreds or thousands of times, but at some point the entire DUT may stop working — and the cause won’t be obvious.
You want the DUT to melt in your lab, not on your customer. Let’s look at three types of robustness testing.
Load Testing
Load testing is typically performed by bulk call generators with many ports. They specialize in generating high call volume to place a high load on the DUT and verify that it is robust under heavy load. For lab testing, a call volume beyond the rated capacity should be used to verify that the DUT degrades gracefully (i.e. does not crash) under stress. If you are testing a small CPE like a one or two port ATA, then a small port-count call generator can do the job. 
Much of the load testing I’ve seen at various companies consists of simple call flows and does not probe the behavior of the DUT with more “interesting” calls under load. This is a missed opportunity for finding bugs. Load tests should include a variety of call flows.
Longevity Testing
With longevity testing we want to find out if the DUT can survive a large number of calls that occur at a reasonable rate. For general longevity testing you may want to randomize the test suite so that — as described above — “interesting” sets of call flows occur throughout the test.
The point of longevity testing is not to overload the DUT but rather to subject the DUT to a reasonable load over a long period of time. (Where “reasonable” is defined for the DUT according to the device’s performance goals.)
The end goal is to find defects that occur after the DUT has processed a huge number of calls over a period of days or even weeks. Again, too often longevity testing uses a single, simple call flow. It makes sense to perform longevity testing of happy-path calls, but every once in a while it’s good to mix it up and make sure all of the boundary cases are handled well.
Repeated Forced Errors
As someone who has written and read thousands of lines of code I know that the best place to find bugs is in the dark corners of the error handling code. It doesn’t get exercised often (if at all), so it’s likely that it is not quite right.
As mentioned above, these failures may not be apparent right away. It may take large numbers of iterations before they rear their head, and then the symptoms may not point to the defect. You can make it easier to diagnose the failure by repeating a test that forces a particular error to occur.
“Errors” too can include multiple classes of errors, including some that can’t be induced at the FXO interface. You may have to create some clever test scripts.
For example, let’s say the DUT is an ATA. Connect the ATA’s analog line to the RCG-4001, and provision the ATA to use the RCG as a switch. On the RCG, your script is the SIP proxy, but it does not always behave properly. Perhaps it drops contact with the ATA, or maybe it returns a 4xx or 5xx response at some point in the flow. If you run this test repeatedly you may find that your production, commercial-quality stack is not as robust to these errors as it should be.
Related posts:
