Cytoscape 3.0 Functional and Acceptance Testing

• For 3.0 we would like to develop a suite of automated acceptance tests that check the basic functionality of Cytoscape.

• The purpose of the tests is twofold:
  1. Provide a simple way to validate that expected functionality is present.
  2. Serve as regression test to verify that no functionality has been lost or broken.
• The tests would be delivered as one or more separate, standalone modules. These would not be unit tests verifying small parts of the system, but rather integration tests than span many modules and require the GUI to run.
• Ideally the tests should be specifiable in English-like prose, so that non-technical users can write the specifications.

Decision:

Which selection of tools should we use to accomplish this goal.

Options

JUnit Only

We could simply use the JUnit framework and express our specifications as a suite of tests. We can use tools like the fest toolkit to execute GUI widgets.

• Benefits - Most developers are familiar with JUnit and it would therefore be relatively easy for them to write tests.

• Costs - The tests would require knowledge of Java to understand.

JBehave

JBehave is a Behavior Driven Development framework that allows test cases (a.k.a. "stories") to be written in English text, which are then mapped onto JUnit-like test cases.

• Benefits - Tests are written in English, so anyone can write a test case, which developers can then implement. Tooling appears fairly robust and support for Maven is good.

• Costs - Mapping the English text to test cases could certainly become time consuming.

JDave

JDave is another BDD framework that uses a "fluent" Java API for expressing test cases. You might see code like this: specify(stack, must.be.full());.

• Benefits - Fluent APIs can be easier to use than plain Java. Specifications are easier to write since they don't require separation of spec and code that runs the test.

• Costs - Writing the test specifications is more complicated than JBehave since it involves a Java-like syntax. Whether an API is "fluent" or not is debatable.

BDoc

BDoc is another BDD framework that rather than focusing on a fluent or readable specification, focuses on producing prose report that describes in English sentences the behavior that passed or failed. This works by translating long method names like aLoadedNetworkProducesANetworkVisualization() into sentences like "A loaded network produces a network visualization" in the report.

• Benefits - This results in a nice document describing the functionality of the system.

• Costs - Configuration appears slightly non-obvious. Writing the specification involves writing Java code.

easyb

easyb is yet another BDD framework. This framework combines English sentences with snippets of Groovy code to define a specification and set of tests.

• Benefits - The English specification is fairly clear and the embedded tests make it fairly obvious which parts of the spec have and have not been met.

• Costs - We'd need to figure out how to specify everything in Groovy.