Run tests in CI
Use the devscenario command-line tool to run a suite in a CI job. It reads the project folder from your repository and runs it on the emulators and simulators connected to the runner. The job fails when a test fails.
Before you start
- The project folder is in your repository, for example in
e2e/. CI runs what is on the branch. - The project has a suite. In the app, click New suite on Home, give it a name and tags, and tag the scenarios and chains it should run. Commit the change.
- The runner has Java 17 or later.
- An emulator or simulator is running and your app is installed on it. The CLI doesn't start devices or install apps.
1. Install
Add the Dev Scenario action to your workflow. It installs the CLI and puts devscenario on the PATH.
- uses: devscenario/e2e-cli@v1
with:
version: 1.6.1 # optional; the latest release when left out
Other CI systems: download the zip from the releases page and add its bin/ folder to the PATH.
2. Run a suite
Point the CLI at the project folder and name the suite. It runs every scenario and chain that carries one of the suite's tags and writes the results to reports/.
- uses: devscenario/e2e-cli@v1
# … start the emulator and install your app
- run: devscenario run e2e/ --suite Regression --artifact-root reports
In the app, the terminal icon on a suite's card shows this command for that suite.
| Option | Runs |
|---|---|
--suite NAME | That suite. |
| No option | The project's only suite. With several suites the CLI stops and lists their names. |
--scenario NAME | One scenario or chain, also one that is in no suite. |
A new scenario joins a suite when it gets one of the suite's tags. To catch untagged ones on a pull request, run devscenario suites check e2e/: it lists every scenario and chain that no suite runs and exits with 1 when there is one. It needs no device.
3. Choose devices
Without options the CLI uses the first connected Android device.
| Option | Runs on |
|---|---|
--platform ios | The first booted iOS simulator. |
--platform both | The first Android device and the first iOS simulator, at the same time. |
--device emulator-5554 | That device only. |
--devices emulator-5554,emulator-5556 | All listed devices at once. They share one queue: a free device takes the next test, so every test runs once. |
--devices … --shard-all | Every test on every listed device, for example to cover several Android versions. |
--preset NAME | A run preset saved with the project in the app's Run suite dialog: its devices, how tests are split, retries and stop on first failure. The suite still decides what runs, so add --suite when the project has several. Manual presets run only in the app. |
4. Failures and retries
By default every test runs, even after one fails. --fail-fast stops the run after the first failed test; the tests that didn't start are reported as skipped, and tests already running on other devices finish. --retries N runs a failed test again, up to N more times, on the same device, and the last attempt counts. Retries set on steps inside a scenario still work as before.
- run: devscenario run e2e/ --suite Regression --artifact-root reports --retries 1 --fail-fast
5. Results
| Exit code | Meaning |
|---|---|
0 | All tests passed. |
1 | A test failed, or the run stopped on an error such as no connected device. |
2 | The command is wrong: an unknown option, a suite name that doesn't exist, several suites and no --suite, a named device that isn't online. |
--artifact-root reports writes:
| File | Contents |
|---|---|
junit.xml | One test case per flow. Most CI systems can show it on the pull request. |
run.json | Each flow's status, duration, device and error. |
events.jsonl | Every step as it ran. |
screenshots/ | The screen at each failed step. |
Use a new folder for each run; the next run overwrites these files.
6. API mocks
A scenario that uses mock groups gets the same mocked responses in CI. The command stays the same: when a scenario of the run selects a mock group, the CLI serves the mocks by itself.
- In the app's Network tab, open the group's menu and choose Track in Git. The group and its rules are saved in the project's
mocks/folder. Commit it. - In the scenario's header, select the mock groups it uses. Single rules picked there with Pick rules… are saved in
mocks/by the app when they are in no group; commit them too. A picked rule that is in a group needs Track in Git on its group. - Your app includes the network interceptor library.
Each scenario gets only its own groups while it runs, also when several devices run different scenarios at the same time. Mocks are served to Android devices and emulators and to iOS simulators.
| Option | What it does |
|---|---|
--no-mocks | Serves no mocks, so the app talks to the real backend. |
--with-mocks | Serves the mocks also when no scenario selects a group, for rules that are in no group. |
7. Report to Zephyr Scale
Each scenario's result can go to Zephyr Scale as an execution of its test case. Reporting from CI works on every plan.
- The scenario is linked to its case: the case key, for example
SHOP-T12, is saved in the scenario file asexternalKey. The Zephyr plugin links it for you. - A Zephyr Scale API token is saved as a secret in your CI and given to the step as
DS_ZEPHYR_TOKEN. The CLI reads the token from this variable only. It has no option for it, so the token never shows in the command or the log.
- run: devscenario run e2e/ --suite Regression --artifact-root reports
- name: Report to Zephyr
if: always()
env:
DS_ZEPHYR_TOKEN: ${{ secrets.ZEPHYR_TOKEN }}
run: devscenario zephyr push reports
The separate step reports failed runs too, and only this step sees the token. To report in the run step, add --zephyr to the run command and give that step the token.
| Option | What it does |
|---|---|
--zephyr-cycle KEY | Send the results to this test cycle. PROJ=KEY picks a cycle per Jira project; repeat it for more. |
--zephyr-cycle-name TEMPLATE | Name of the cycles the CLI creates. Takes {project}, {module}, {date}, {source} and {branch}. |
--zephyr-dry-run | Print what would be sent and send nothing. Needs no token. |
--zephyr-strict | Exit with 1 when results can't be sent. Without it a problem is a warning and the exit code stays as the tests set it. |
Without a cycle option the results go to the cycles set in the plugin's settings, which are saved in the project as test-management.json. Without that file the CLI uses Zephyr Scale Cloud and creates the cycles itself. Reporting the same run again sends nothing twice. Before the first run, devscenario zephyr check with --project e2e/ checks the token and the settings.
All options
| Option | What it does |
|---|---|
--suite NAME | Run this suite. |
--scenario NAME | Run one scenario or chain instead of a suite. |
--platform android|ios|both | Platform to run on. Default android. |
--device ID | Run on one device (serial or simulator UDID). |
--devices A,B,… | Run on several devices with one shared queue. |
--shard-split N / --shard-all N | Share the tests across N devices, or run every test on each of them. |
--preset NAME | Use a run preset saved in the project: devices, split, retries, stop on first failure. |
--ios-physical | The iOS device is a physical iPhone. Needs --platform ios and --device or --devices. |
--profile NAME | Param profile that plain @key values read. Not needed when the project has one profile. |
--env KEY=VALUE | Value for ${KEY} in flows. Repeat for more. |
--artifact-root DIR | Where to write the reports. |
--filter-tag TAG | Run only flows with this tag in their meta(tags = […]) line. Suites go by scenario tags; use --suite for those. |
--timeout MS | Time limit per flow. |
--wait-timeout-ms MS | How long a step waits for its element when it sets no timeout. Default 10 s on Android, 5 s on iOS. |
--android-driver-mode in-app-sdk | Drive the app through the in-app SDK. Default is UIAutomator. |
--ios-driver-mode in-app-sdk | The same for iOS. |
--android-cross-app-fallback off | Fail instead of using UIAutomator when the SDK doesn't answer. |
--ios-auto-fallback off | Fail instead of using the XCUITest runner when the iOS SDK doesn't answer. |
--app-id ID | Package or bundle id to use when a flow doesn't name one. |
--with-mocks / --no-mocks | Serve the project's API mocks always, or never. See API mocks. |
--zephyr | Report the results to Zephyr Scale after the run. See Report to Zephyr Scale. |
--fail-fast | Stop after the first failed test and skip the rest. |
--retries N | Run a failed test up to N more times on the same device. Default 0. |
--coverage | Also record which elements each flow touched, in coverage.json. Slower. Needs --artifact-root. |
--mutate | Measure how strong the tests are instead of running them once. See UI mutation testing. |
--pretty | Readable progress lines instead of JSON on stdout. |
--timing | Print where the time went, per flow and per step. |
--run-id ID | Name of the run in the reports. |
Troubleshooting
No device found
The emulator wasn't running when the CLI started, or the CLI ran in a different step from the one that started it. Run the CLI in the same step, after the device has booted.
Pick a suite with --suite NAME
The project has several suites and the command names none. Add --suite with one of the names the error lists.
The project has no suites
The project on this branch has no suites.json. Create a suite on Home in the app and commit the file.
Unknown suite, profile or preset
The name doesn't match one saved in the project. The error lists the names that exist.
A mock group isn't in the project's mocks/ folder
A scenario selects a group that exists only on someone's computer. In the Network tab, choose Track in Git in the group's menu and commit the mocks/ folder.
DS_ZEPHYR_TOKEN is not set
The step that reports has no DS_ZEPHYR_TOKEN, or the secret is missing. Secrets aren't passed to pull requests from forks.
The run uses UIAutomator although you asked for the in-app SDK
The installed app doesn't include the SDK, for example a release build. Install the debug build, or add --android-cross-app-fallback off to fail the job instead.
Steps time out in CI but pass on your machine
CI emulators are slower. Raise the default wait with --wait-timeout-ms. The first iOS step of a run also takes longer while the simulator's test driver starts.