Step Outside the Call Generator to Force More Failures

You’ve taught your call generator all kinds of tricks, stacking and chaining call features left and right. In the process you’ve found (and fixed!) tons of bugs. This has built up a huge regression test suite.

But running this test suite regularly means that you’re not finding many bugs any more. This is good news (the product is better) and bad news (there are probably still more bugs to find). The really bad news is that the bugs that are left can be really hard to find, and often require stepping outside your current test framework.

We’ve talked about forcing errors from the call generator, and making sure that the DUT can handle them — repeatedly. This builds on that practice and expands it beyond the call generator and into your test network.

What’s the Worst Thing That Could Happen?

This is the question to ask.

1. What is the worst thing that could happen internally? Perhaps a memory leak that uses up all available memory. Or the internal storage (disk, flash) fills up. What if a fan fails and the heat builds up?
2. What is the worst thing that could happen externally? Loss of connection to the network. Power failure. Network storm.
3. What could a malicious person do? Send a flood. Send specially crafted packets.

Now take the answers to those questions, and make the worst thing happen.

1. If you’re testing a VoIP gateway, disconnect it from the network mid-call.
2. Take that VoIP gateway and run a longevity test. Now run a script that toggles the port on the network switch on-off every 2 minutes. (Don’t have a managed switch? Get a managed power supply and just toggle power to your hub.)
3. Using the same gateway and longevity test, this time use your managed power supply to kill power to the gateway at random intervals.
4. Use a network traffic generator to simulate a network storm.
5. Write scripts to generate some of those “specially crafted packets” that might do bad things.
6. Cheat if you can: examine the code and look for error paths. Do whatever it takes to hit those error paths. (Hint: this is what malicious people do.)

Go do your worst.

Get free email updates or follow our feed in your reader.

Related posts:

1. Use a Call Generator for Robustness Testing
2. Why Do Experts Plan For Failure?
3. Induce Failures by Varying the DTMF Pulse Width
4. The Secret to Managing the Voice Test Matrix Explosion
5. Troubleshooting Telephony Failures