119 lines
4.3 KiB
Markdown
119 lines
4.3 KiB
Markdown
# test-preflight matrix schema
|
||
|
||
The input is YAML. `mode` is supplied by the CLI, so a file may be reused for
|
||
`verify` and `prepare`.
|
||
|
||
```yaml
|
||
schema_version: 1
|
||
project_root: /absolute/project/root
|
||
host: https://test.example.invalid
|
||
public_host: https://public-test.example.invalid # optional
|
||
service_probes:
|
||
- id: api-health
|
||
url: "{HOST}/health"
|
||
expected_status: [200]
|
||
fixtures:
|
||
required: true
|
||
references: [isolated-test-account]
|
||
cleanup:
|
||
required: true
|
||
steps: [delete-run-data]
|
||
capabilities:
|
||
# These baseline checks are mandatory for mobile rows and cannot be disabled
|
||
# with AUTHORIZE_ALBUM or IME_PINYIN.
|
||
required: [album_picker, photo_permission, ime_pinyin]
|
||
matrix:
|
||
- platform: android
|
||
tester:
|
||
binding: tester.android
|
||
node: local
|
||
runner: {kind: local}
|
||
artifact:
|
||
path: /absolute/dist/writes.apk
|
||
sha256: 012345...
|
||
package: page.writes
|
||
source_commit: 40-character-commit-sha
|
||
identity_command: [apkanalyzer, manifest, application-id, "{artifact}"]
|
||
```
|
||
|
||
Required matrix fields are `platform` (`android` or `ios`), `tester.binding`,
|
||
`tester.node`, and `artifact.path`. Artifact identity is required as
|
||
`package` for Android or `bundle_id` for iOS. `sha256` and `source_commit` are
|
||
optional but recommended; when present they are checked exactly.
|
||
|
||
`service_probes` must contain at least one URL. `{HOST}` and `{PUBLIC_HOST}`
|
||
are expanded from the top-level values. Each probe is checked from the actual
|
||
tester node as DNS, TCP, and HTTP. A status outside 2xx–4xx fails transport
|
||
readiness; `expected_status` can require an exact allow-list. HTTP 401/403 can
|
||
therefore be a valid reachability result when explicitly expected.
|
||
|
||
`fixtures` and `cleanup` are declarations, not commands. A required fixture
|
||
needs non-empty `references`; a required cleanup needs non-empty `steps` or
|
||
`run`. For a read-only run, use `required: false` with a non-empty `reason`.
|
||
The skill never executes these values.
|
||
|
||
## Mobile capability contract
|
||
|
||
Every Android/iOS matrix row checks these capabilities on the actual tester node:
|
||
|
||
| capability | required evidence |
|
||
|---|---|
|
||
| `album_picker` | `checks.picker`: the real platform album/Photos picker opened and returned a selectable asset |
|
||
| `photo_permission` | `checks.permission` and `checks.test_image`: permission state was exercised and the declared test image was available |
|
||
| `ime_pinyin` | `checks.composition` and `checks.candidates`: real Chinese pinyin composition and candidate selection succeeded |
|
||
|
||
The default list is always applied; `capabilities.required` can add project-specific
|
||
checks but cannot remove these three. A contract runner receives one request per
|
||
capability:
|
||
|
||
```json
|
||
{
|
||
"operation": "capability_probe",
|
||
"platform": "android",
|
||
"node": "tester.android",
|
||
"capability": "ime_pinyin",
|
||
"required_checks": ["composition", "candidates"]
|
||
}
|
||
```
|
||
|
||
It must return every requested check with `status`, `observed`, and (when useful)
|
||
`expected`/`remediation`. Missing evidence is `blocked`; a real probe failure is
|
||
`environment-fail`. Environment variables such as `AUTHORIZE_ALBUM` and
|
||
`IME_PINYIN` are not capability evidence.
|
||
|
||
## Tester-node contract
|
||
|
||
`tester.runner.kind: local` executes platform and network probes on the current
|
||
node. It is only valid for `node: local`/`localhost`. A non-local node must use
|
||
`kind: contract` and a non-empty argv `command`. The command receives one JSON
|
||
request on stdin and returns one JSON object on stdout; no shell string is
|
||
accepted.
|
||
|
||
Requests have `operation` (`platform_probe`, `network_probe`, `install`, or
|
||
`start`), `platform`, `node`, and operation-specific fields. The response is:
|
||
|
||
```json
|
||
{
|
||
"status": "ready",
|
||
"observed": "short evidence",
|
||
"checks": {
|
||
"macos": {"status": "ready", "observed": "Darwin 24", "expected": "Darwin"}
|
||
}
|
||
}
|
||
```
|
||
|
||
`checks` is optional; each key becomes a separate evidence row. A non-zero,
|
||
invalid, or unavailable runner is `environment-fail`. The adapter is expected
|
||
to execute on the resolved tester node, not on the coordinator. See
|
||
[boundaries.md](boundaries.md).
|
||
|
||
## Evidence
|
||
|
||
YAML contains `mode`, overall `status`, timestamps, a summary, and `checks`.
|
||
CSV uses these stable columns:
|
||
|
||
`check_id,mode,platform,node,target,observed,expected,status,remediation,timestamp`
|
||
|
||
Every check must use one of `ready`, `blocked`, or `environment-fail`; the
|
||
report does not invent `fail` or `not-run` testcase results.
|