Roverly in your CI pipeline

Run Roverly's UI tests on every pull request and gate the merge on the result. Your pipeline builds the app, uploads it, asks Roverly to test it, and turns the verdict into a passing or failing check.

Works the same on GitHub Actions, GitLab CI, Jenkins, Bitbucket, CircleCI — the integration is an HTTP API, so there is nothing platform-specific to install.

Before you start

Two things are set up once, by a person. CI cannot do either, and a pipeline wired before they are done will fail every time.

1. A paired desktop must be online. Runs execute on your own machine, not on Roverly's servers — that is what lets us drive a real device, a simulator, or a browser you control. Install the desktop app, pair it, and make sure the device or simulator you want to test is connected and configured. For iOS on real hardware, provisioning must already be in place.

A pull request opened while every paired machine is asleep produces a run that never starts. Your CI step times out and says so.

2. Regression runs need a pinned production run. A regression run works by comparing what it sees against a known-good baseline. From the desktop app, do a learn run against your app, then mark it as the Production run on the runs page.

This one is genuinely manual: pinning requires a learn run, and CI is not allowed to trigger learn runs. If you skip it, POST /v1/runs refuses with a message saying so.

Getting a key

Dashboard → Settings → API. A CI key carries two scopes:

Scope	What it allows
`trigger`	Starting runs, and uploading builds
`read`	Polling a run and reading its findings

Give your pipeline both. Store it as ROVERLY_API_KEY in your CI's secret store — it is a bearer token, so anyone holding it can spend your run allowance.

The key your desktop uses is a different kind and cannot call this API.

The quickest version

curl -fsSL https://roverly.ai/cli/roverly -o roverly && chmod +x roverly

ROVERLY_API_KEY="$ROVERLY_KEY" ./roverly test \
  --project-id "$ROVERLY_PROJECT" \
  --run-type feature \
  --pr-body "$PR_BODY" --pr-title "$PR_TITLE" --pr-number "$PR_NUMBER" \
  --branch "$BRANCH" --commit "$COMMIT_SHA" \
  --build ./app/build/outputs/apk/release/app-release.apk \
  --summary-file roverly-comment.md

Exit code 0 passes the check, 1 fails it, 2 means the tool was invoked wrong. --summary-file writes a ready-to-post pull-request comment.

Prefer to paste rather than download? The same gate is about fifteen lines of curl and jq: roverly.ai/cli/snippet.sh. It skips the build upload and the PR comment, and gates on the same verdict.

The API

Base URL https://api.roverly.ai. Every call takes Authorization: Bearer <your key>.

Upload a build

Mobile only. Web projects test a URL and install nothing.

POST /v1/artifacts        → { id, upload_url, upload_content_length, expires_at_iso }
PUT  <upload_url>         → the bytes, straight to storage
POST /v1/artifacts/{id}/complete

The bytes never pass through the API — you PUT them directly to storage using the signed URL. Send exactly the Content-Length returned as upload_content_length; the signature binds it, and a different value is rejected.

Declare the file's size_bytes, its sha256, and the package_id (Android package or iOS bundle id) that the build contains. The package id is checked against your project's — a mismatch means the agent would install one app and launch another, which would report a verdict about the wrong software.

Builds are kept 7 days and capped at 200 MB. Long enough to re-run a pull request; not a place to store artifacts.

For an iOS simulator build, zip the .app bundle and upload the zip — a .app is a directory, so it is not a file you can upload. The archive must contain exactly one .app.

Start a run

POST /v1/runs

Field	Notes
`project_id`	required
`run_type`	`regression` or `feature`. `learn` is refused.
`goal`	required for `feature` — what this change should be exercised against. Map your PR description to it.
`artifact_id`	the build to install, from the upload above
`target_url`	web only — the preview deployment to test instead of your project's URL
`branch`, `commit_sha`, `environment`	free-text traceability
`pr_number`, `pr_url`, `pr_title`	pull-request context, entirely optional

Which type to use:

regression — diff the whole app against your pinned baseline. Catches visual changes, navigation changes, structural changes and crashes.
feature — a goal-scoped run over the area your change touches. It has no baseline to diff against, so it catches crashes and logged errors in that area. It does not tell you whether anything looks different. That is narrower than "test what this PR changed" may sound, and worth knowing before you rely on it.

Testing a preview deployment. For web, target_url points the run at your PR's preview build. Because previews live off your own domain (*.vercel.app, *.netlify.app), add those hosts to the project's preview-domain allowlist in project settings first — otherwise the run is refused, deliberately, so a stray URL cannot aim a run at somebody else's site.

Wait for it, then read the verdict

GET /v1/runs/{id}          → { status, ... }
GET /v1/runs/{id}/summary  → { status, reasons, blocking, markdown, superseded }

Poll GET /v1/runs/{id} until status is one of succeeded, failed, cancelled.

processing is not finished. The analysis that produces the findings runs during processing. A client that stops as soon as the status is no longer running reads the verdict before the findings exist, and reports a confident pass on a broken change. Check for membership of the three terminal statuses — never "not running".

Then GET /v1/runs/{id}/summary. Branch on status (pass / fail) and post markdown as a comment. It is always HTTP 200 — an HTTP error means auth or infrastructure, never a verdict.

Give up before an hour. A run nobody has picked up is cleaned up after 60 minutes. Waiting past that means polling a run that has already been abandoned, and reporting that as your app's verdict. 45 minutes is a sensible ceiling.

Pushing again while a run is in flight

Push six times and you get the in-flight run plus one run for the sixth commit — not six runs, and not the second commit's result.

A run an agent has already started always finishes. A queued run that no agent has picked up yet is replaced by the newer one. Replaced runs do not count against your plan.

When your job is the one that got replaced, summary returns superseded: true with status: "pass". Exit 0. Being replaced is a skip, not a failure — a newer run is about to answer the same question, and a red check here would just be noise on a busy pull request.

When things go wrong

What happened	What you see
No paired desktop online	The run stays queued; your step times out saying so
Another run in progress	Queued behind it; same timeout applies
No production run pinned	`400` at trigger time, naming the two-step fix
Over your plan's allowance	`402` at trigger time
Build missing, expired, or the wrong package	`400` at trigger time — no run starts
Run itself errored	`summary` fails, naming the error
Replaced by a newer commit	`superseded: true`, exit 0

In this version, everything that reaches your CI step and is not a pass renders as a failed check. The summary text tells you which of the above it was.

Fork pull requests are not special-cased — gate them yourself. Roverly's API is CI-agnostic and has no way to know a pull request came from a fork, so a fork PR starts a real run and spends a real slot from your monthly pool. On a public repository that is usually not what you want, both for cost and because a contributor's token generally cannot post the comment back. Skip the step in your pipeline's own condition — on GitHub Actions, for example, if: github.event.pull_request.head.repo.full_name == github.repository.

Plans

No plan gates this API, and there is no separate CI quota. A CI-triggered run draws on the same monthly pool of agentic runs as a run you start from the desktop, and every plan can call /v1.

What differs is how many runs the pool holds, and what happens when you exhaust it:

Plan	Runs included per month	Past the pool
Starter	8	`402` — runs are refused until the month rolls over
Pro	40	Metered as pay-as-you-go; runs keep going
Enterprise	Unlimited	—

The practical consequence for Starter: eight gated pull requests a month, and the ninth POST /v1/runs returns 402 rather than queueing. If your team merges more often than that, Pro is the plan that does not stop.

Questions, or an integration we do not cover? Get in touch.