Dev Scenario Docs
Automate / Run in CI

Run tests in CI

Use the devscenario command-line tool to run a suite in a CI job. It reads the project folder from your repository and runs it on the emulators and simulators connected to the runner. The job fails when a test fails.

Before you start

  • The project folder is in your repository, for example in e2e/. CI runs what is on the branch.
  • The project has a suite. In the app, click New suite on Home, give it a name and tags, and tag the scenarios and chains it should run. Commit the change.
  • The runner has Java 17 or later.
  • An emulator or simulator is running and your app is installed on it. The CLI doesn't start devices or install apps.

1. Install

Add the Dev Scenario action to your workflow. It installs the CLI and puts devscenario on the PATH.

.github/workflows/regression.yml
- uses: devscenario/e2e-cli@v1
  with:
    version: 1.6.1   # optional; the latest release when left out

Other CI systems: download the zip from the releases page and add its bin/ folder to the PATH.

2. Run a suite

Point the CLI at the project folder and name the suite. It runs every scenario and chain that carries one of the suite's tags and writes the results to reports/.

.github/workflows/regression.yml
- uses: devscenario/e2e-cli@v1

# … start the emulator and install your app
- run: devscenario run e2e/ --suite Regression --artifact-root reports

In the app, the terminal icon on a suite's card shows this command for that suite.

OptionRuns
--suite NAMEThat suite.
No optionThe project's only suite. With several suites the CLI stops and lists their names.
--scenario NAMEOne scenario or chain, also one that is in no suite.

A new scenario joins a suite when it gets one of the suite's tags. To catch untagged ones on a pull request, run devscenario suites check e2e/: it lists every scenario and chain that no suite runs and exits with 1 when there is one. It needs no device.

3. Choose devices

Without options the CLI uses the first connected Android device.

OptionRuns on
--platform iosThe first booted iOS simulator.
--platform bothThe first Android device and the first iOS simulator, at the same time.
--device emulator-5554That device only.
--devices emulator-5554,emulator-5556All listed devices at once. They share one queue: a free device takes the next test, so every test runs once.
--devices … --shard-allEvery test on every listed device, for example to cover several Android versions.
--preset NAMEA run preset saved with the project in the app's Run suite dialog: its devices, how tests are split, retries and stop on first failure. The suite still decides what runs, so add --suite when the project has several. Manual presets run only in the app.

4. Failures and retries

By default every test runs, even after one fails. --fail-fast stops the run after the first failed test; the tests that didn't start are reported as skipped, and tests already running on other devices finish. --retries N runs a failed test again, up to N more times, on the same device, and the last attempt counts. Retries set on steps inside a scenario still work as before.

.github/workflows/regression.yml
- run: devscenario run e2e/ --suite Regression --artifact-root reports --retries 1 --fail-fast

5. Results

Exit codeMeaning
0All tests passed.
1A test failed, or the run stopped on an error such as no connected device.
2The command is wrong: an unknown option, a suite name that doesn't exist, several suites and no --suite, a named device that isn't online.

--artifact-root reports writes:

FileContents
junit.xmlOne test case per flow. Most CI systems can show it on the pull request.
run.jsonEach flow's status, duration, device and error.
events.jsonlEvery step as it ran.
screenshots/The screen at each failed step.

Use a new folder for each run; the next run overwrites these files.

6. API mocks

A scenario that uses mock groups gets the same mocked responses in CI. The command stays the same: when a scenario of the run selects a mock group, the CLI serves the mocks by itself.

  • In the app's Network tab, open the group's menu and choose Track in Git. The group and its rules are saved in the project's mocks/ folder. Commit it.
  • In the scenario's header, select the mock groups it uses. Single rules picked there with Pick rules… are saved in mocks/ by the app when they are in no group; commit them too. A picked rule that is in a group needs Track in Git on its group.
  • Your app includes the network interceptor library.

Each scenario gets only its own groups while it runs, also when several devices run different scenarios at the same time. Mocks are served to Android devices and emulators and to iOS simulators.

OptionWhat it does
--no-mocksServes no mocks, so the app talks to the real backend.
--with-mocksServes the mocks also when no scenario selects a group, for rules that are in no group.

7. Report to Zephyr Scale

Each scenario's result can go to Zephyr Scale as an execution of its test case. Reporting from CI works on every plan.

  • The scenario is linked to its case: the case key, for example SHOP-T12, is saved in the scenario file as externalKey. The Zephyr plugin links it for you.
  • A Zephyr Scale API token is saved as a secret in your CI and given to the step as DS_ZEPHYR_TOKEN. The CLI reads the token from this variable only. It has no option for it, so the token never shows in the command or the log.
.github/workflows/regression.yml
- run: devscenario run e2e/ --suite Regression --artifact-root reports

- name: Report to Zephyr
  if: always()
  env:
    DS_ZEPHYR_TOKEN: ${{ secrets.ZEPHYR_TOKEN }}
  run: devscenario zephyr push reports

The separate step reports failed runs too, and only this step sees the token. To report in the run step, add --zephyr to the run command and give that step the token.

OptionWhat it does
--zephyr-cycle KEYSend the results to this test cycle. PROJ=KEY picks a cycle per Jira project; repeat it for more.
--zephyr-cycle-name TEMPLATEName of the cycles the CLI creates. Takes {project}, {module}, {date}, {source} and {branch}.
--zephyr-dry-runPrint what would be sent and send nothing. Needs no token.
--zephyr-strictExit with 1 when results can't be sent. Without it a problem is a warning and the exit code stays as the tests set it.

Without a cycle option the results go to the cycles set in the plugin's settings, which are saved in the project as test-management.json. Without that file the CLI uses Zephyr Scale Cloud and creates the cycles itself. Reporting the same run again sends nothing twice. Before the first run, devscenario zephyr check with --project e2e/ checks the token and the settings.

All options

OptionWhat it does
--suite NAMERun this suite.
--scenario NAMERun one scenario or chain instead of a suite.
--platform android|ios|bothPlatform to run on. Default android.
--device IDRun on one device (serial or simulator UDID).
--devices A,B,…Run on several devices with one shared queue.
--shard-split N / --shard-all NShare the tests across N devices, or run every test on each of them.
--preset NAMEUse a run preset saved in the project: devices, split, retries, stop on first failure.
--ios-physicalThe iOS device is a physical iPhone. Needs --platform ios and --device or --devices.
--profile NAMEParam profile that plain @key values read. Not needed when the project has one profile.
--env KEY=VALUEValue for ${KEY} in flows. Repeat for more.
--artifact-root DIRWhere to write the reports.
--filter-tag TAGRun only flows with this tag in their meta(tags = […]) line. Suites go by scenario tags; use --suite for those.
--timeout MSTime limit per flow.
--wait-timeout-ms MSHow long a step waits for its element when it sets no timeout. Default 10 s on Android, 5 s on iOS.
--android-driver-mode in-app-sdkDrive the app through the in-app SDK. Default is UIAutomator.
--ios-driver-mode in-app-sdkThe same for iOS.
--android-cross-app-fallback offFail instead of using UIAutomator when the SDK doesn't answer.
--ios-auto-fallback offFail instead of using the XCUITest runner when the iOS SDK doesn't answer.
--app-id IDPackage or bundle id to use when a flow doesn't name one.
--with-mocks / --no-mocksServe the project's API mocks always, or never. See API mocks.
--zephyrReport the results to Zephyr Scale after the run. See Report to Zephyr Scale.
--fail-fastStop after the first failed test and skip the rest.
--retries NRun a failed test up to N more times on the same device. Default 0.
--coverageAlso record which elements each flow touched, in coverage.json. Slower. Needs --artifact-root.
--mutateMeasure how strong the tests are instead of running them once. See UI mutation testing.
--prettyReadable progress lines instead of JSON on stdout.
--timingPrint where the time went, per flow and per step.
--run-id IDName of the run in the reports.

Troubleshooting

No device found

The emulator wasn't running when the CLI started, or the CLI ran in a different step from the one that started it. Run the CLI in the same step, after the device has booted.

Pick a suite with --suite NAME

The project has several suites and the command names none. Add --suite with one of the names the error lists.

The project has no suites

The project on this branch has no suites.json. Create a suite on Home in the app and commit the file.

Unknown suite, profile or preset

The name doesn't match one saved in the project. The error lists the names that exist.

A mock group isn't in the project's mocks/ folder

A scenario selects a group that exists only on someone's computer. In the Network tab, choose Track in Git in the group's menu and commit the mocks/ folder.

DS_ZEPHYR_TOKEN is not set

The step that reports has no DS_ZEPHYR_TOKEN, or the secret is missing. Secrets aren't passed to pull requests from forks.

The run uses UIAutomator although you asked for the in-app SDK

The installed app doesn't include the SDK, for example a release build. Install the debug build, or add --android-cross-app-fallback off to fail the job instead.

Steps time out in CI but pass on your machine

CI emulators are slower. Raise the default wait with --wait-timeout-ms. The first iOS step of a run also takes longer while the simulator's test driver starts.