Test Runner

The Test Runner is a feature that was introduced with Orchestra 4.16. It offers the possibility to validate Scenario Elements in an Orchestra Scenario (e.g. Mapping, Process model) via different testing levels.

Unit, integration and system tests can thus be implemented. With this feature, quality assurance and error detection in your application can be extensively optimized. It enhances the Testing feature, allowing multiple scenarios to be tested within a single pipeline.

Functionality

• Repeatable tests: The Test Runner enables the automated and repeatable execution of recorded tests. This saves time and resources, especially for regular regression tests.

• Difference tracking: The Test Runner can compare the current behavior of the application with the recorded behavior and makes differences visible in a clear and traceable way. This makes it easier to identify and rectify errors.

• Defining landscapes: Different test environments can be defined, which enable tests to be tested on different systems. Here you can overwrite existing environment entries or define which Docker containers should be started before the test is executed.

• CI/CD integration: The Test Runner can be integrated into a pipeline, capable of testing multiple scenarios and providing test results for each scenario.

Use Cases

• Automated regression tests: By recording and automating test cases, regular tests can be carried out to ensure that new developments do not have any undesirable effects on existing functions.
• Troubleshooting and error tracking: In the event of failed tests, the assert functionality enables the rapid identification of problems and their precise localization.
• Realistic tests: Even if external systems are not available, realistic tests can be carried out by mocking steps.

Advantages

• Improved quality assurance and error detection.
• Time savings through automated tests and simple test definition.
• User-friendly description language for tests.
• Traceability of differences in behavior.

Configuration

To use the Test Runner, you must ensure that the correct Java path is set. Open startscripts/Unix/orchestra_env.sh and verify that ORC_JAVA_HOME is set correctly.

###############################################################
# Please adjust these values that they fit your system
###############################################################
ORC_JAVA_HOME=$JAVA_HOME

note

Java 21 is the minimum requirement.

Usage

The Test Runner is a command line interface that can be started via a script. To do this, run the following script:

Operating System Scripts

Operating System	Script Path
Linux	startscripts/Unix/test-runner.sh
Windows	startscripts/Windows/test-runner.cmd

To start the application, you can use the following commands and options with the script:

Commands

Command(s)	Description
test, execute	Executes tests defined in a scenario. Both commands are equivalent.

Options

Option	Description
-d, --scenarioDir	Path to the directory containing scenarios to be tested. br/ Can be used multiple times. br/ Can be combined with -s.
-f, --format	Defines the output format of the test results. br/ Options: json, junit. Default: json.
--failNever	If set, the command exits with 0 even if tests fail. Otherwise, exits with -1. br/ Useful in pipelines.
-h, --help	Shows help message and exits.
-o, --output	Path to the output directory for test results. br/ Default: current working directory.
-p, --testProfile	Name of the test profile to be executed. br/ Scenarios without the profile will be skipped.
-s, --scenario	Path to a scenario file to be tested. br/ Can be used multiple times and combined with -d.
-V, --version	Prints version information and exits.

Linux

On Linux, an autocompletion script can be executed, allowing you to run the test runner directly from the command line without executing the script manually. Additionally, this provides code completion capabilities. To enable this, simply run the test-runner-autocompletion-install.sh script. This will add an entry to your .bashrc file and creates an test-runner_completion.sh script.

Docker

To use the Test Runner inside a Docker container, you can simply use the image soffico/test-runner:latest. After running the docker run command, you can append any commands you want the test runner to execute.

Example:

docker run --rm -v <local.scenario.dir>:/home/scenarien -v <local.output.dir>:/home/output soffico/test-runner:latest \
test-runner test -d /home/scenarien -o /home/output -c -f json