A technique to test apis specified in natural language.

Abstract

Developers of widely used Application Programming Interfaces (APIs) implement and test APIs based on a document, which is commonly specified using natural language. How- ever, there is limited knowledge on whether API developers are able to systematically reveal i) underdetermined specifications; and ii) non-conformances between their implementation and the specification. To better understand the problem, we analyze test suites of Java Reflection API, and we conduct two surveys. A survey with 130 developers who use the Java Reflection API, and a survey with 128 C# developers who use and implement the .NET Reflection API to see whether the specification impacts on their understanding. We also propose a technique to detect underdetermined specifications and non-conformances between the specification and the implementations of the APIs. It automatically creates test cases, and executes them using different implementations. It saves objects yielded by methods to be used to create more test cases. If results differ, it detects an underdetermined specification or a non-conformance candidate between the specification and at least one implementation of the API. We evaluate our technique using the Java Reflection API in 446 input programs. Our technique identifies underdetermined specification and non-conformance candidates in 32 Java Reflection API public methods of 7 classes. We report underdetermined specification candidates in 12 Java Reflection API methods. Java Reflection API specifiers accept 3 underdetermined specification candidates (25%). We also report 24 non-conformance candidates to Eclipse OpenJ9 JVM, and 7 to Oracle JVM. Eclipse OpenJ9 JVM developers accept and fix 21 candidates (87.5%), and Oracle JVM developers accept 5 and fix 4 non-conformance candidates. Twelve test cases are now part of the Eclipse OpenJ9 JVM test suite. We also evaluate our technique using the Java Collections API. Even being a very popular Java API, our technique identifies 29 underdetermined specification and non-conformance candidates. Our technique identifies 17 candidates that cannot be detected by popular automatic test suite generators. We report 5 underdetermined specification candidates to the Java Collections API specifiers. We also report 9 non-conformance candidates to Eclipse OpenJ9 JVM, and 4 to Oracle JVM. Oracle JVM developers accept and fix 3 non-conformance candidates. Eclipse OpenJ9 JVM developers accept and fix 1 non-conformance candidate.