There are many tools available for testing web services. One of the ones we use that's easy and has a fairly rich set of features is soapUI (did I mention that it's also free and open source). With this tool you can easily test SOAP or RESTful services either by manually adding the links and messages, or via a WSDL or WADL utilizing their import feature. In either case, entering a request and receiving a response is a fairly trivial task and for purposes of time, will not be discussed here. You can find more information on how to get started with soapUI on their website.
Setting up a Test Suite
Another capability of soapUI that you might not know about is the ability to setup Test Suites for automated Unit Testing and a step beyond that, actually running simulated Load Tests for your web services. Setting up a Test Suite with a Test Case is as simple as right-clicking on an existing request and selecting "Add to Test Case". When the dialog appears, you can either pick an existing Test Suite or add a new one. The Test Case editor will then appear and let you configure the Test Case with the request details (input message, headers, etc.) and the Assertions you want to perform. soapUI allows quite a few options for how to build the assertions such as XQuery, XPath, contains, not contains, schema compliance, and SLA Response. More details on how to setup test cases can be found here.
For example, a simple Test Case should have some assertion(s) that check the response message for correctness and another for the SLA Response Time. So perhaps one of your assertions could use an XPath expression to test the number of XML nodes returned or maybe the content of the JSON response. Setting up an SLA Response Time assertion is pretty straight-forward but is significant if you are running load tests and it requires some thought as to how long each Test Case should run. If you have a service request that is hitting the database using a primary key search, this should be much faster than one that uses wildcards. These types of scenarios should be considered when determining the desired response time. Once you setup your assertions and request details, you can run your Test Case to make sure it passes. Run it several times and check out the "Request Log" tab to get an idea of the SLA Response Time and adjust the assertion if necessary.
Load Testing
Once you have your Test Suite setup with at least one Test Case, you can perform a Load Test; this is where you find out how your service is handling a lot of messages all at once. To create a Load Test, you simply right click on the Load Tests option under the Test Suite and then select "New Load Test". Once you get to the Load Test Editor screen you will see that it was nice enough to add all of your Test Cases (or Steps) to the new Load Test. Now you can configure the Load Test to setup how long it will run, how many threads, the delay between requests, and the strategy (this drives more test options depending on which you choose). soapUI offers several load testing strategies which simulate different scenarios and types of load. Here are a few of the ones you might want to try (you can find a detailed list here):
- Simple - Runs a specified number of threads using a randomized delay between requests. This is good for setting a baseline for performance of your service. Run this test first to make sure you're service is performing well under normal circumstances.
- Variance - Runs a varied number of threads in a "sawtooth" (randomly increasing/decreasing number of threads; think of an ECG line of your heartbeat). This is good for stress testing and potentially baseline testing based upon what you believe will be "normal" load on your service.
- Burst - Runs a specified number of threads all at once for a specified amount of time, then waits until the next "burst" to begin based off the set delay. This is good for recovery testing, i.e. if you service is completely slammed, how does it react.
Putting your Service to the Test
Once you have a load test setup, the challenge becomes setting the thresholds for your service that will help to meet the desired SLA Response Time. This means different things for different applications. In a Java or .NET application, it might mean adjusting the number of threads running in parallel. It could also mean you need more hardware (i.e. Memory, CPU) to handle the expected load. Whatever the case may be, it is always fun (and useful) to get a before and after snapshot of the performance of the service. This way you can make sure your changes actually worked!
In the following example, we have hosted HTTP RESTful Services using IBM's Message Broker (version 7). Message Broker makes it very easy to scale the number of threads by adjusting the number of additional instances of a flow that is deployed into the runtime. Each instance of the flow is essentially an additional thread that can be utilized for handling requests. The default is 0 additional instances which means, in this case, only one (1) HTTP request can be handled at a time. When you hit the service with multiple threads, the subsequent threads will wait for the previous one to finish. As you can see, this is not a good setup for a production web service:
You can see in the image above that during a 60 second time period, 51 runs of the Search_ById service resulted in an assertion error which, in this case, is because it did not meet the SLA response Time. This is simply using the default settings of the Simple Test strategy which this should not be a problem for a typical service to handle. At this point, running any other testing strategy is pointless until we figure out how to make some pretty drastic changes to the service. So we will increase the number of instances of the flow in Message Broker to 5 and re-run our test:
After this change, you can plainly see that the service handled the load much better. There was only 1 error and the SLA set on the "by Id" search is pretty aggressive, so even the max response of 2,347 milliseconds is still acceptable. You can also see that it executed 236 more tests in the same 60 second time period. This is because the requests did not have to wait for the previous request to finish. This service is now acceptable to run in a production environment. Just for fun, let's try the service with a more aggressive test strategy and see how it does:
In this scenario, we started with 8 threads, set the interval to 60, and the variance to 0.8 which means the number of threads will increase from 8 to 14 within the first 15 seconds, then decrease back to 8 and continue down to 2 threads after 45 seconds, and finally go back up 8 after 60 seconds. You can see that the service performed fairly well overall during this test. We might want to raise the expected SLA on the "by Id" search service to a little less aggressive number, but again, 1,968ms for the max response is acceptable.
Taking it a Step Further
With the latest release of soapUI, you have the ability to run the load tests out of another tool called, loadUI. This tool uses the Test Suite and Load Tests you have setup in soapUI to run the tests but provides much more detail and better graphs and charts to analyze your results. I have not used this tool, but the soapUI website recommends using it for benefits such as distributed tests in real-time, better UI, scalable testing, live analytics, and more. It is free and open source and integrates seamlessly with soapUI, so why not give it a shot? If you have already used loadUI or decide to try it out, leave some feedback below.
