The Product Development Agenda Blog

I recently reviewed an article "Test SOA for the unexpected" by Rich Seeley that got me thinking about SOA testing frameworks.

The design of experimentation (DoE) is probably one of the most significant testing issues one faces in SOA and is more complicated that most other archetypes. Take, for example, three simple services that can be choreographed in any order, each capable of interacting with each other. How many tests do you need to ensure 100% functional coverage? You're right if you said 6 (3x2x1). This is a classic factorial problem, in this case with 3 service. Now, let's say your system has 100 services, so how many test would you need? Again, your right if you said 100!, or 9.33x10^157, and it would take you over 3x10^150 yrs to test completely, if you ran continuously a test every second of every day.

Traditional testing in these kinds of systems are impractical given the number of total tests needed to understand/certify the functional/non-functional behavior of the system. So what do you do in these circumstances? Well, that is where a good , well thought out, SOA design of experiment (DoE) comes into play. DoEs are all about reducing the number of experiments without unnecessarily diminishing the value of the test. Such DoE are call Partial Factorial DoE. A good SOA DoE for the 100 services, for example, could reduce it down to 10! or 5040 tests, a significant reduction in effort and cost. 

So, the next time you are thinking about testing your complete SOA environment, ask whether or not you have the right type of testing framework (DoE) and whether or not you really need to do all those tests in order to show that your systems is actually working correctly or not. 

Dr. Jerry A. Smith