Oscar very much relies on its automated test suite, and we've developed a number of test types as part of our approach.
Given the breadth of the project these are broadly true, but there may be small deviations in individual projects (which we're trying to iron out).
The tests are classified via package rather than filename, with the un-prefixed package tending to be used for shared test code. Some projects do have tests in the un-prefixed package, however, and as such the generic test task runs everything that's not in a known prefix.
Dependencies: apps which the app-in-question is dependent onDependents: apps which depend on the app-in-question
These are specified via package prefixes, e.g. cdcs.com.springer.oscar.
Unit tests are our most basic test element. They test a single object in isolation, with dependencies stubbed out in some fashion. We discourage mocking, especially for data objects. They should not require external dependencies, and they must be isolated - i.e. you can run them concurrently and in any order without unintended effects.
Functional tests are where we start the entire application with stubbed-out external dependencies (or "stubbed-out apps that this app is dependent on"). This allows us to send requests into the application without using HTTP, and test the application in a more complete sense, while retaining isolation from the environment/external dependencies. Like unit tests, these should be safe for concurrent execution in any order.
Integration tests ensure that the app-in-question integrates as expected with any external apps it is dependent on ("dependencies"). As such they may be limited in what they can do regarding mutation or flow. They use the application configuration (e.g. the appropriate manifest in CloudFoundry apps) to determine the remote endpoint to use.
Local integration tests are tests of integrations against external dependencies where we create a dummy instance of the dependency, which is disposed of after the test. For instance, we may use this to test mutations against a MySQL database, but use Docker Compose to provide the DB. These tests must not be run against real dependencies.
Consumer-driven Contract tests are tests to ensure that our changes have not broken our contracts with external services that are dependent on us ("dependents"). Every time we deploy changes we give dependents an opportunity to run their CDCs against our candidate service. If they fail, we will terminate the deployment.
We don't quite implement CDCs in the pattern linked above, for historical reasons. Originally CDC support came via Springer's "Autopipeline" system. This had three drawbacks, for us. Firstly, it would invoke a single script for both integration and CDC tests, in the latter case supplying the name of the dependency that was being deployed and the location of the candidate service. This meant we needed a single execution point for these, which is our case is the CDC Test Suite. We then created LocalCdcTestFinder to manage the difference modes of execution and aid in gathering integration tests. Secondly, it runs the CDCs against live systems. This, like integration tests, limits the testing we can do as the tests may have side-effects. Thirdly, the tests are run anew each time, rather than storing success or failure for version combinations already tested.
We've now moved to Halfpipe, which doesn't impose the single-script constraint, so one improvement would be to split the execution (and hence make the execution much more readable).
On the other points, Content Acquisition is currently working on a complete re-invention of a CDC execution system, which we should investigate as soon as it is stable.
Smoke tests are our final line of defence. These are run against the candidate system just before it is promoted to the live routes, and hence should perform sanity checks to increase confidence that the application is correctly deployed and ready to serve production traffic.
The root build.gradle defines a number of test targets to ensure consistency across the various projects. As such you can be pretty confident that these work the same way in all projects.
This is the generic Gradle test task, and is used for unit-testing and functional tests. In particular, any tests that are run by this runner should not require network access.
The tasks is run by exclusion - any types that are not covered by one of the runners below will be run by this runner.
This tasks runs the consumer-driven-contact tests and integration tests, depending on how it is triggered. Please see the cdc section above for a discussion of why this madness exists.
It runs anything matching cdcs/**/*CdcTestSuite*. If the environment variables DEPENDENCY_NAME and <dependency>_DEPLOYED_HOST are supplied then we will run in dependent CDC mode; otherwise we will run integration tests. This logic is codified in com.springer.oscar.shared.cdcs.LocalCdcTestFinder.
This tasks runs smoke tests against the candidate system, the hostname of which is passed via the environment variable TEST_ROUTE.
It runs anything matching smoke/**.
This task runs integration tests against a local instance, generally provided via Docker Compose. These are differentiated from other integration tests as we assume that we can throw the local instance away - hence in any tests under package localintegration we can mutate data to our heart's content.
In runs anything matching localintegration/** and, for completeness, integration/**.
Run ./gradlew clean test in the main oscar directory
To run the integration tests of oscar-sites-springer against a locally deployed oscar-components
- Make sure the dependency is running.
E.g. Run
UberLauncherQAwithoscar-componentsuncommented out inuber.yml - Run the integration tests against the dependencies
E.g.
DEPENDENCY_NAME=oscar-components OSCAR_COMPONENTS_DEPLOYED_HOST=localhost:14000 bin/run-integration-tests qa