feat(test-preflight): add verify and prepare skill
This commit is contained in:
@@ -0,0 +1,4 @@
|
||||
# Reference moved
|
||||
|
||||
Read [schema.md](schema.md) for the matrix and evidence contract, and
|
||||
[boundaries.md](boundaries.md) for agent-runtime/deployer/builder integration.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Integration boundaries
|
||||
|
||||
`test-preflight` is an orchestration and evidence layer. It consumes existing
|
||||
skills instead of implementing their policy or storage formats.
|
||||
|
||||
| Boundary | Caller-supplied contract | What preflight does | What it must not do |
|
||||
| --- | --- | --- | --- |
|
||||
| agent-runtime | resolved `tester.<name>` binding: binding id, node, environment, project/path, and a node runner | validate that the binding is explicit and route node probes through the runner | parse Orca state, invent a host, or create/reuse a terminal |
|
||||
| deployer | `prepare` only: an argv command returning a JSON deployment receipt for `environment: test` | call it once and record its receipt/status | copy Compose/Argo logic, SSH directly, or accept production |
|
||||
| builder/obtain | caller-provided local artifact path (and optional remote path), hash, source commit, package/bundle identity | hash and inspect the exact file; pass it to install contract | build, guess an old artifact, or silently download/upload |
|
||||
|
||||
## Deployer command contract
|
||||
|
||||
`prepare` requires:
|
||||
|
||||
```yaml
|
||||
deployment:
|
||||
environment: test
|
||||
command: [python3, /path/to/existing-deployer-adapter.py]
|
||||
```
|
||||
|
||||
The command receives JSON on stdin:
|
||||
|
||||
```json
|
||||
{"operation":"prepare","environment":"test","project_root":"/abs/project","host":"...","public_host":"..."}
|
||||
```
|
||||
|
||||
It must return `{ "status": "ready", "observed": "..." }` (an optional
|
||||
`checks` mapping is recorded). Non-zero/invalid output is an
|
||||
`environment-fail`; absent command or a non-test environment is `blocked`.
|
||||
The adapter owns deployer-specific configuration, receipts, cleanup, and
|
||||
authorization.
|
||||
|
||||
## Device contract
|
||||
|
||||
The node runner owns remote execution. For a local node the implementation uses
|
||||
read-only `adb`/`xcrun simctl` checks in `verify`; `prepare` alone invokes
|
||||
`adb install`/launch or `simctl install`/launch. For a remote node the same
|
||||
operations are JSON requests to the runner. Missing xcrun on Linux, Android
|
||||
`unauthorized`/`offline`, a missing device, or a failed install/start is an
|
||||
`environment-fail`, not a product failure.
|
||||
|
||||
No client session is created by this skill. The runner's `install` and `start`
|
||||
operations are the only device mutations, and are never called by `verify`.
|
||||
@@ -0,0 +1,85 @@
|
||||
# test-preflight matrix schema
|
||||
|
||||
The input is YAML. `mode` is supplied by the CLI, so a file may be reused for
|
||||
`verify` and `prepare`.
|
||||
|
||||
```yaml
|
||||
schema_version: 1
|
||||
project_root: /absolute/project/root
|
||||
host: https://test.example.invalid
|
||||
public_host: https://public-test.example.invalid # optional
|
||||
service_probes:
|
||||
- id: api-health
|
||||
url: "{HOST}/health"
|
||||
expected_status: [200]
|
||||
fixtures:
|
||||
required: true
|
||||
references: [isolated-test-account]
|
||||
cleanup:
|
||||
required: true
|
||||
steps: [delete-run-data]
|
||||
matrix:
|
||||
- platform: android
|
||||
tester:
|
||||
binding: tester.android
|
||||
node: local
|
||||
runner: {kind: local}
|
||||
artifact:
|
||||
path: /absolute/dist/writes.apk
|
||||
sha256: 012345...
|
||||
package: page.writes
|
||||
source_commit: 40-character-commit-sha
|
||||
identity_command: [apkanalyzer, manifest, application-id, "{artifact}"]
|
||||
```
|
||||
|
||||
Required matrix fields are `platform` (`android` or `ios`), `tester.binding`,
|
||||
`tester.node`, and `artifact.path`. Artifact identity is required as
|
||||
`package` for Android or `bundle_id` for iOS. `sha256` and `source_commit` are
|
||||
optional but recommended; when present they are checked exactly.
|
||||
|
||||
`service_probes` must contain at least one URL. `{HOST}` and `{PUBLIC_HOST}`
|
||||
are expanded from the top-level values. Each probe is checked from the actual
|
||||
tester node as DNS, TCP, and HTTP. A status outside 2xx–4xx fails transport
|
||||
readiness; `expected_status` can require an exact allow-list. HTTP 401/403 can
|
||||
therefore be a valid reachability result when explicitly expected.
|
||||
|
||||
`fixtures` and `cleanup` are declarations, not commands. A required fixture
|
||||
needs non-empty `references`; a required cleanup needs non-empty `steps` or
|
||||
`run`. For a read-only run, use `required: false` with a non-empty `reason`.
|
||||
The skill never executes these values.
|
||||
|
||||
## Tester-node contract
|
||||
|
||||
`tester.runner.kind: local` executes platform and network probes on the current
|
||||
node. It is only valid for `node: local`/`localhost`. A non-local node must use
|
||||
`kind: contract` and a non-empty argv `command`. The command receives one JSON
|
||||
request on stdin and returns one JSON object on stdout; no shell string is
|
||||
accepted.
|
||||
|
||||
Requests have `operation` (`platform_probe`, `network_probe`, `install`, or
|
||||
`start`), `platform`, `node`, and operation-specific fields. The response is:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ready",
|
||||
"observed": "short evidence",
|
||||
"checks": {
|
||||
"macos": {"status": "ready", "observed": "Darwin 24", "expected": "Darwin"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`checks` is optional; each key becomes a separate evidence row. A non-zero,
|
||||
invalid, or unavailable runner is `environment-fail`. The adapter is expected
|
||||
to execute on the resolved tester node, not on the coordinator. See
|
||||
[boundaries.md](boundaries.md).
|
||||
|
||||
## Evidence
|
||||
|
||||
YAML contains `mode`, overall `status`, timestamps, a summary, and `checks`.
|
||||
CSV uses these stable columns:
|
||||
|
||||
`check_id,mode,platform,node,target,observed,expected,status,remediation,timestamp`
|
||||
|
||||
Every check must use one of `ready`, `blocked`, or `environment-fail`; the
|
||||
report does not invent `fail` or `not-run` testcase results.
|
||||
Reference in New Issue
Block a user