feat(test-preflight): add verify and prepare skill

This commit is contained in:
2026-10-15 18:04:59 +08:00
parent ffa7cbd63e
commit c2eb207bd2
9 changed files with 1014 additions and 0 deletions
@@ -0,0 +1,4 @@
# Reference moved
Read [schema.md](schema.md) for the matrix and evidence contract, and
[boundaries.md](boundaries.md) for agent-runtime/deployer/builder integration.
@@ -0,0 +1,44 @@
# Integration boundaries
`test-preflight` is an orchestration and evidence layer. It consumes existing
skills instead of implementing their policy or storage formats.
| Boundary | Caller-supplied contract | What preflight does | What it must not do |
| --- | --- | --- | --- |
| agent-runtime | resolved `tester.<name>` binding: binding id, node, environment, project/path, and a node runner | validate that the binding is explicit and route node probes through the runner | parse Orca state, invent a host, or create/reuse a terminal |
| deployer | `prepare` only: an argv command returning a JSON deployment receipt for `environment: test` | call it once and record its receipt/status | copy Compose/Argo logic, SSH directly, or accept production |
| builder/obtain | caller-provided local artifact path (and optional remote path), hash, source commit, package/bundle identity | hash and inspect the exact file; pass it to install contract | build, guess an old artifact, or silently download/upload |
## Deployer command contract
`prepare` requires:
```yaml
deployment:
environment: test
command: [python3, /path/to/existing-deployer-adapter.py]
```
The command receives JSON on stdin:
```json
{"operation":"prepare","environment":"test","project_root":"/abs/project","host":"...","public_host":"..."}
```
It must return `{ "status": "ready", "observed": "..." }` (an optional
`checks` mapping is recorded). Non-zero/invalid output is an
`environment-fail`; absent command or a non-test environment is `blocked`.
The adapter owns deployer-specific configuration, receipts, cleanup, and
authorization.
## Device contract
The node runner owns remote execution. For a local node the implementation uses
read-only `adb`/`xcrun simctl` checks in `verify`; `prepare` alone invokes
`adb install`/launch or `simctl install`/launch. For a remote node the same
operations are JSON requests to the runner. Missing xcrun on Linux, Android
`unauthorized`/`offline`, a missing device, or a failed install/start is an
`environment-fail`, not a product failure.
No client session is created by this skill. The runner's `install` and `start`
operations are the only device mutations, and are never called by `verify`.
@@ -0,0 +1,85 @@
# test-preflight matrix schema
The input is YAML. `mode` is supplied by the CLI, so a file may be reused for
`verify` and `prepare`.
```yaml
schema_version: 1
project_root: /absolute/project/root
host: https://test.example.invalid
public_host: https://public-test.example.invalid # optional
service_probes:
- id: api-health
url: "{HOST}/health"
expected_status: [200]
fixtures:
required: true
references: [isolated-test-account]
cleanup:
required: true
steps: [delete-run-data]
matrix:
- platform: android
tester:
binding: tester.android
node: local
runner: {kind: local}
artifact:
path: /absolute/dist/writes.apk
sha256: 012345...
package: page.writes
source_commit: 40-character-commit-sha
identity_command: [apkanalyzer, manifest, application-id, "{artifact}"]
```
Required matrix fields are `platform` (`android` or `ios`), `tester.binding`,
`tester.node`, and `artifact.path`. Artifact identity is required as
`package` for Android or `bundle_id` for iOS. `sha256` and `source_commit` are
optional but recommended; when present they are checked exactly.
`service_probes` must contain at least one URL. `{HOST}` and `{PUBLIC_HOST}`
are expanded from the top-level values. Each probe is checked from the actual
tester node as DNS, TCP, and HTTP. A status outside 2xx–4xx fails transport
readiness; `expected_status` can require an exact allow-list. HTTP 401/403 can
therefore be a valid reachability result when explicitly expected.
`fixtures` and `cleanup` are declarations, not commands. A required fixture
needs non-empty `references`; a required cleanup needs non-empty `steps` or
`run`. For a read-only run, use `required: false` with a non-empty `reason`.
The skill never executes these values.
## Tester-node contract
`tester.runner.kind: local` executes platform and network probes on the current
node. It is only valid for `node: local`/`localhost`. A non-local node must use
`kind: contract` and a non-empty argv `command`. The command receives one JSON
request on stdin and returns one JSON object on stdout; no shell string is
accepted.
Requests have `operation` (`platform_probe`, `network_probe`, `install`, or
`start`), `platform`, `node`, and operation-specific fields. The response is:
```json
{
"status": "ready",
"observed": "short evidence",
"checks": {
"macos": {"status": "ready", "observed": "Darwin 24", "expected": "Darwin"}
}
}
```
`checks` is optional; each key becomes a separate evidence row. A non-zero,
invalid, or unavailable runner is `environment-fail`. The adapter is expected
to execute on the resolved tester node, not on the coordinator. See
[boundaries.md](boundaries.md).
## Evidence
YAML contains `mode`, overall `status`, timestamps, a summary, and `checks`.
CSV uses these stable columns:
`check_id,mode,platform,node,target,observed,expected,status,remediation,timestamp`
Every check must use one of `ready`, `blocked`, or `environment-fail`; the
report does not invent `fail` or `not-run` testcase results.