Posted by George Pirocanac , Test Engineering ManagerAPI calls that the application makes through the use of various constructs called mocks, fakes, dummy routines, etc. Depending on how the application is designed and written, this can lead to smaller, simpler tests that cover more, execute more quickly, and lead to quicker diagnosis of problems than the larger end-to-end or system tests. On the other hand, they are not a complete replacement for end-to-end testing. By their very nature, the small tests don't test the assumptions and interactions between the application and the APIs that it calls. As a result, a diversified application testing strategy includes small, medium, and large tests. (See Copeland’s GTAC Video , fast forward about 5 minutes in to hear a brief description of developer testing and small, medium, large)
What about testing the APIs themselves? What if anything is different? The first approach mirrors the small test approach. Each of the API calls is exercised with a variety of inputs and the outputs that are verified according to the specification. For isolated, stateless APIs (math library functions come to mind), this can be very effective by itself. However, many APIs are not isolated or stateless, and their results can vary according to the *combinations* of calls that were made. One way to deal with this is to analyze the dependencies between the calls and create mini-applications to exercise and verify these combinations of calls. Often, this falls into the so-called typical usage patterns or user scenarios. While good, this first approach only gives us limited confidence. We also need to test what happens when not-so-typical sets of calls are made as well. Often application writers introduce usage patterns that the spec didn't anticipate.
Another approach is to capture the API calls made by real applications under controlled situations and then replay only the calls under the same controlled situations. These types of tests fall into the medium category. Again, the idea is to test series and combinations of calls, but the difficulty can lie in recreating the exact environment. In addition, this approach is vulnerable to building tests that traverse the same paths in the code. Adding fuzz to the parameters and call patterns can help, but not eliminate, this problem.
The third approach is to pull out the big hammer. Does it make sense to test the APIs with large applications? After all, if something goes wrong, you have to have specific knowledge about the application to triage the problem. You also have to be familiar with the techniques in testing the application. Testing a map-based application can be quite different from a calendar-based one, even if they share a common subset of APIs . The strongest case for testing APIs with large applications is compatibility testing. APIs not only have to return correct results, but they have to do it in the same manner from revision to revision. It's a sort of contract between the API writer and the application writer. When the API is private, then only a relative small number of parties have to agree on a change to the contract, but when it is public, even a small change can break a lot of applications.
So when it comes to API testing, it seems we are back to small, medium, and large approaches after all. Just as in application testing where you can't completely divorce the API from the application, we cannot completely divorce the application from API testing.
Hi George,
ReplyDeleteI am new to API testing.
I am hired as a QA engineer for health industry.
They works on applications which provide communications between Doctors, hospitals, paitients, pharmacy.
They use APIs widely.
What you will recommend to read on API testing from QA point of view?
Some free online tutorials?
I like your article very much.
Thank you.
Alex Lazovskiy
altora@comcast.net
Its a nice article on Testing APIs. I would also like to add the challenges involved in testing APIs
ReplyDelete1.Parameter selection
2.Parameter combinations
3.Call sequencing
and combination of all above three points
Also traditional tes techniquest to evaluate the API such boundary value analysis, Forced error detection etc.
Hi George,
ReplyDeleteI'm a mobility Automation test engineer.I'm new to API testing .As a value added to my project my company decides to provide API testing too without using any automation tool.I heard API can be tested by using core Java without using any SOAPUI or REST or something else API tools, is it so?. If it is possible please share materials to nagaraju482@gmail.com email id.
Thanks in Advance,
Nagaraj