Cytoscape_3/CoreDevelopment/AutomatedTesting

Purpose

This page provides information useful for developing automated tests for Cytoscape 3.0.

Unit Testing

• All 3.0 modules are built using maven, which means the maven-surefire-plugin executes whatever JUnit test cases are found in the src/test/java directory.

• While there are a few special cases where we use JUnit 3, JUnit 4 should be the preferred target when writing tests.
• Follow unit testing best practices:
  • Each test should only test ONE (1) condition. Don't write long tests that test many conditions because these are much harder to debug and the purpose of the test becomes obscured.
  • You should only be testing YOUR code and you shouldn't be testing external code. Keep your tests focused on the functionality YOU provide and not that of others. If your test requires elaborate external configuration to execute, then you might be writing an integration test (or you might need to refactor).

  • Never rely on external resources (e.g. websites, databases, etc.). Try mocking the external resource to test your code. If your code needs some external resource and won't work with a mock, then this is a sign that your code may not be sufficiently modular. In this case try refactoring your code to isolate those parts that need an external resource. While not ideal (because it slows tests down), it is acceptable for test cases to depend on files provided with the project.

  • Your tests should be as fast as possible. Long tests or tests with delays are OK in integration tests, but slow things down substantially when part of unit tests.
  • Your code is almost never "too simple to test". When in doubt, write a test.
  • Be aware of both simple code coverage AND branch coverage!

• We tend to use Mockito as a mocking framework, but in some places we use EasyMock instead. If you don't have a good reason for using EasyMock (like your test needs to run in an OSGi container), prefer Mockito to be consistent.

• We use Sonar to evaluate all sorts of design and quality metrics, including test coverage. Spend some time at the site and evaluate your code. Several of the metrics can also be generated using the maven-site-plugin (run: mvn site).

• Just because your code is a Swing GUI doesn't mean it can't be tested! You can often construct Swing objects that don't get displayed, but are nonetheless able to be tested. It is often fairly simple to create mock events that can be used to test listener interface implementations.

Testing API Bundles

• API bundles contain both interfaces and actual code implementations. Our goal is provide tests for both.
• For actual code implementation (e.g. static classes, final classes, enums that have additional methods), just write normal unit tests that cover as much functionality of the code as possible. You should be able to get pretty close to 100% test coverage.

• For interfaces we instead provide abstract unit tests distributed in an API-test jar. These tests should go in the normal src/test/java directory, but be declared abstract.

• The purpose of abstract unit tests for interfaces is to define the expected behavior of methods. For example given a List<CyNode> getNodeList() method, should this method ever be allowed to return null? Or should it just return an empty list? This sort of policy decision can be defined in the abstract unit test so that any implementation of the interface will be more likely of working seamlessly with the rest of the system.

• Abstract unit tests are not meant to provide 100% coverage for all implementations and implementations will almost certainly have special cases that will need to be tested separately. However, these tests should be as thorough as possible.
• Abstract unit tests can only be written to the interface, making it hard to cheat using implementation specific details. The ability to write clear simple code in your tests is a good way of validating the quality of your interfaces!

Testing Implementation Bundles

• If you are implementing an API, first check to see if a test jar is provided along with the API and whether the test jar includes abstract test cases. Always extend whatever abstract test cases are available!
• Whenever you add additional tests to an implementation of an API, consider whether those tests should be provided by the API as an abstract test instead. In general, if the test relies strictly on the interface and is testing a policy (e.g. never return null), then it should probably be part of the API.

Integration Testing

Performance Testing