11 Commits

34 changed files with 3585 additions and 45 deletions
+14
View File
@@ -0,0 +1,14 @@
# test-preflight
Reusable preflight for tester matrices. It supports read-only `verify` and
test-environment-only `prepare`, and writes per-check YAML/CSV evidence for
artifacts, tester nodes, devices, network, services, and fixture/cleanup contracts.
This project assumes mobile album/photo/IME capabilities; journey cases exercise them
and preflight does not gate them or read authorization environment variables.
Run the self-tests with:
```bash
python3 -m unittest discover -s skills/test-preflight/tests
python3 /home/ace/.codex/skills/.system/skill-creator/scripts/quick_validate.py skills/test-preflight
```
+58
View File
@@ -0,0 +1,58 @@
---
name: test-preflight
description: >-
Verify or prepare a mobile/integration test environment from a declared matrix:
artifacts, tester bindings, HOST/PUBLIC_HOST, service probes, and fixture/cleanup
contracts. Use before tester runs; it is not a deployment,
build, or business-data skill.
---
# test-preflight
Use this skill before `integration` or `journey` tests. It answers whether the
declared environment can start a run and emits one YAML and one CSV evidence
file. It never logs in, creates business data, builds/downloads an artifact, or
deploys production.
## Modes
- `verify` is read-only: validate the matrix, artifact identity, binding, actual
tester-node platform/device state, DNS/TCP/HTTP probes, backend probes, and
fixture/cleanup declarations. It must not install, launch, or deploy.
- `prepare` is side-effecting only within the test environment: call the
caller-supplied deployer command contract, run the same checks, then use the
tester-node command contract to install and launch the already supplied APK or
IPA. It never calls a builder and refuses `environment: prod`.
The command is:
```bash
python3 <skill-dir>/scripts/preflight.py verify \
--config <matrix.yaml> --yaml <preflight.yaml> --csv <preflight.csv>
python3 <skill-dir>/scripts/preflight.py prepare \
--config <matrix.yaml> --yaml <preflight.yaml> --csv <preflight.csv>
```
Read [references/schema.md](references/schema.md) for the input contract and
[references/boundaries.md](references/boundaries.md) before wiring a real
agent-runtime/deployer adapter. The included command-contract fixture is the
safe way to test remote-node behavior without pretending that this machine is a
Mac or that a deployment happened.
Overall status is `ready`, `blocked`, or `environment-fail`:
- `blocked` means a required declaration, artifact, binding, runner, fixture,
cleanup contract, or authorization is absent/unsafe.
- `environment-fail` means a declared environment exists but DNS/TCP/HTTP,
device/toolchain, installation, launch, or deployment execution failed.
- `ready` means every recorded check passed. A ready preflight permits the
caller to continue with `integration`; this skill does not run test cases.
This project assumes the client has the required album picker, photo permission, and Chinese
IME capabilities. Preflight does not probe or gate these capabilities, and does not read
`AUTHORIZE_ALBUM` or `IME_PINYIN`. Their real behavior is exercised by journey cases; a failure
there is a product test failure, not an environment block.
Do not treat a preflight report as a product result. A preflight failure means
no client session may be created; the tester owns `not-run` propagation and must write
one result row per selected testcase with `blocked_by: [test-preflight]`.
+3
View File
@@ -0,0 +1,3 @@
interface:
display_name: "Test Preflight"
short_description: "Verify or prepare tester environments"
@@ -0,0 +1,4 @@
# Reference moved
Read [schema.md](schema.md) for the matrix and evidence contract, and
[boundaries.md](boundaries.md) for agent-runtime/deployer/builder integration.
@@ -0,0 +1,44 @@
# Integration boundaries
`test-preflight` is an orchestration and evidence layer. It consumes existing
skills instead of implementing their policy or storage formats.
| Boundary | Caller-supplied contract | What preflight does | What it must not do |
| --- | --- | --- | --- |
| agent-runtime | resolved `tester.<name>` binding: binding id, node, environment, project/path, and a node runner | validate that the binding is explicit and route node probes through the runner | parse Orca state, invent a host, or create/reuse a terminal |
| deployer | `prepare` only: an argv command returning a JSON deployment receipt for `environment: test` | call it once and record its receipt/status | copy Compose/Argo logic, SSH directly, or accept production |
| builder/obtain | caller-provided local artifact path (and optional remote path), hash, source commit, package/bundle identity | hash and inspect the exact file; pass it to install contract | build, guess an old artifact, or silently download/upload |
## Deployer command contract
`prepare` requires:
```yaml
deployment:
environment: test
command: [python3, /path/to/existing-deployer-adapter.py]
```
The command receives JSON on stdin:
```json
{"operation":"prepare","environment":"test","project_root":"/abs/project","host":"...","public_host":"..."}
```
It must return `{ "status": "ready", "observed": "..." }` (an optional
`checks` mapping is recorded). Non-zero/invalid output is an
`environment-fail`; absent command or a non-test environment is `blocked`.
The adapter owns deployer-specific configuration, receipts, cleanup, and
authorization.
## Device contract
The node runner owns remote execution. For a local node the implementation uses
read-only `adb`/`xcrun simctl` checks in `verify`; `prepare` alone invokes
`adb install`/launch or `simctl install`/launch. For a remote node the same
operations are JSON requests to the runner. Missing xcrun on Linux, Android
`unauthorized`/`offline`, a missing device, or a failed install/start is an
`environment-fail`, not a product failure.
No client session is created by this skill. The runner's `install` and `start`
operations are the only device mutations, and are never called by `verify`.
@@ -0,0 +1,95 @@
# test-preflight matrix schema
The input is YAML. `mode` is supplied by the CLI, so a file may be reused for
`verify` and `prepare`.
```yaml
schema_version: 1
project_root: /absolute/project/root
host: https://test.example.invalid
public_host: https://public-test.example.invalid # optional
service_probes:
- id: api-health
url: "{HOST}/health"
expected_status: [200]
fixtures:
required: true
references: [isolated-test-account]
cleanup:
required: true
steps: [delete-run-data]
# Optional project metadata. It is not a preflight gate.
capabilities:
assumed: [album_picker, photo_permission, ime_pinyin]
matrix:
- platform: android
tester:
binding: tester.android
node: local
runner: {kind: local}
artifact:
path: /absolute/dist/writes.apk
sha256: 012345...
package: page.writes
source_commit: 40-character-commit-sha
identity_command: [apkanalyzer, manifest, application-id, "{artifact}"]
```
Required matrix fields are `platform` (`android` or `ios`), `tester.binding`,
`tester.node`, and `artifact.path`. Artifact identity is required as
`package` for Android or `bundle_id` for iOS. `sha256` and `source_commit` are
optional but recommended; when present they are checked exactly.
`service_probes` must contain at least one URL. `{HOST}` and `{PUBLIC_HOST}`
are expanded from the top-level values. Each probe is checked from the actual
tester node as DNS, TCP, and HTTP. A status outside 2xx–4xx fails transport
readiness; `expected_status` can require an exact allow-list. HTTP 401/403 can
therefore be a valid reachability result when explicitly expected.
`fixtures` and `cleanup` are declarations, not commands. A required fixture
needs non-empty `references`; a required cleanup needs non-empty `steps` or
`run`. For a read-only run, use `required: false` with a non-empty `reason`.
The skill never executes these values.
## Client capabilities
The project assumes the mobile test nodes provide the album picker, photo permission,
test image, and Chinese IME. These are not preflight declarations or gates. Journey
cases exercise the real UI; failures are recorded as product failures. `AUTHORIZE_ALBUM`
and `IME_PINYIN` are not read.
## Tester-node contract
`tester.runner.kind: local` executes platform and network probes on the current
node. It is only valid for `node: local`/`localhost`. A non-local node must use
`kind: contract` and a non-empty argv `command`. The command receives one JSON
request on stdin and returns one JSON object on stdout; no shell string is
accepted.
Requests have `operation` (`platform_probe`, `network_probe`, `install`, or
`start`), `platform`, `node`, and operation-specific fields. The response is:
```json
{
"status": "ready",
"observed": "short evidence",
"checks": {
"macos": {"status": "ready", "observed": "Darwin 24", "expected": "Darwin"}
}
}
```
`checks` is optional; each key becomes a separate evidence row. A non-zero,
invalid, or unavailable runner is `environment-fail`. The adapter is expected
to execute on the resolved tester node, not on the coordinator. See
[boundaries.md](boundaries.md).
## Evidence
YAML contains `mode`, overall `status`, timestamps, a summary, and `checks`.
CSV uses these stable columns:
`check_id,mode,platform,node,target,observed,expected,status,remediation,timestamp`
Every check must use one of `ready`, `blocked`, or `environment-fail`; the
report does not invent `fail` or `not-run` testcase results.
+553
View File
@@ -0,0 +1,553 @@
#!/usr/bin/env python3
"""Run the test-preflight verify/prepare contract."""
from __future__ import annotations
import argparse
import csv
import hashlib
import json
import os
import platform as host_platform
import plistlib
import re
import shutil
import socket
import subprocess
import sys
import urllib.error
import urllib.parse
import urllib.request
import zipfile
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Mapping
try:
import yaml
except ImportError as exc: # pragma: no cover
raise SystemExit("test-preflight 需要 PyYAML;请安装运行依赖后重试") from exc
PLATFORMS = {"android", "ios"}
STATUSES = {"ready", "blocked", "environment-fail"}
EVIDENCE_FIELDS = [
"check_id", "mode", "platform", "node", "target", "observed", "expected",
"status", "remediation", "timestamp",
]
SHA256 = re.compile(r"^[0-9a-fA-F]{64}$")
COMMIT = re.compile(r"^[0-9a-fA-F]{40}$")
def now() -> str:
return datetime.now(timezone.utc).isoformat()
def text(value: Any) -> str:
if isinstance(value, (dict, list)):
return json.dumps(value, ensure_ascii=False, sort_keys=True)
return "" if value is None else str(value)
def status_from(value: Any, default: str = "environment-fail") -> str:
return value if value in STATUSES else default
def sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for block in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(block)
return digest.hexdigest()
def command_text(command: list[str], *, payload: Mapping[str, Any] | None = None) -> tuple[int, str]:
try:
process = subprocess.run(
command,
input=(json.dumps(payload, ensure_ascii=False) if payload is not None else None),
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
check=False,
)
except OSError as exc:
return 127, str(exc)
return process.returncode, process.stdout.strip()
class Evidence:
def __init__(self, mode: str) -> None:
self.mode = mode
self.started_at = now()
self.rows: list[dict[str, str]] = []
def add(
self,
check_id: str,
*,
platform: str = "all",
node: str = "coordinator",
target: str = "",
observed: Any = "",
expected: Any = "",
status: str,
remediation: str = "",
) -> None:
if status not in STATUSES:
raise ValueError(f"invalid evidence status: {status}")
self.rows.append({
"check_id": check_id,
"mode": self.mode,
"platform": platform,
"node": node,
"target": target,
"observed": text(observed),
"expected": text(expected),
"status": status,
"remediation": remediation,
"timestamp": now(),
})
def overall(self) -> str:
statuses = {row["status"] for row in self.rows}
if "environment-fail" in statuses:
return "environment-fail"
if "blocked" in statuses:
return "blocked"
return "ready"
def payload(self) -> dict[str, Any]:
return {
"schema_version": 1,
"mode": self.mode,
"status": self.overall(),
"started_at": self.started_at,
"finished_at": now(),
"summary": {state: sum(row["status"] == state for row in self.rows) for state in sorted(STATUSES)},
"checks": self.rows,
}
def load_config(path: Path) -> dict[str, Any]:
data = yaml.safe_load(path.read_text(encoding="utf-8"))
if not isinstance(data, dict):
raise ValueError("配置根必须是 mapping")
return data
def expand(value: str, config: Mapping[str, Any]) -> str:
return value.replace("{HOST}", text(config.get("host"))).replace("{PUBLIC_HOST}", text(config.get("public_host")))
def normalize_url(value: str) -> str:
return value if urllib.parse.urlsplit(value).scheme else "http://" + value.lstrip("/")
def argv(value: Any) -> list[str] | None:
if not isinstance(value, list) or not value or not all(isinstance(item, str) and item for item in value):
return None
return value
def validate_shape(config: Mapping[str, Any], evidence: Evidence) -> bool:
ok = True
if config.get("schema_version") != 1:
evidence.add("config-schema", target="config", observed=config.get("schema_version"), expected="1", status="blocked", remediation="将 schema_version 设为 1")
ok = False
if not isinstance(config.get("project_root"), str) or not Path(text(config.get("project_root"))).is_absolute():
evidence.add("config-project-root", target="project_root", observed=config.get("project_root"), expected="absolute project_root", status="blocked", remediation="提供项目根绝对路径")
ok = False
if not isinstance(config.get("host"), str) or not config["host"].strip():
evidence.add("config-host", target="HOST", expected="non-empty host URL", status="blocked", remediation="提供 HOST")
ok = False
probes = config.get("service_probes")
if not isinstance(probes, list) or not probes:
evidence.add("config-service-probes", target="service_probes", expected="at least one probe", status="blocked", remediation="声明服务 URL 探针")
ok = False
else:
for index, probe in enumerate(probes):
if not isinstance(probe, dict) or not isinstance(probe.get("url"), str) or not probe["url"].strip():
evidence.add(f"config-service-probe-{index}", target=f"service_probes[{index}]", expected="url", status="blocked", remediation="为每个服务探针提供 URL")
ok = False
matrix = config.get("matrix")
if not isinstance(matrix, list) or not matrix:
evidence.add("config-matrix", target="matrix", expected="non-empty list", status="blocked", remediation="声明至少一个平台矩阵行")
return False
seen: set[str] = set()
for index, row in enumerate(matrix):
prefix = f"matrix-{index}"
if not isinstance(row, dict):
evidence.add(prefix, target=f"matrix[{index}]", expected="mapping", status="blocked", remediation="修正矩阵行")
ok = False
continue
platform_name = text(row.get("platform")).lower()
if platform_name not in PLATFORMS or platform_name in seen:
evidence.add(f"{prefix}-platform", platform=platform_name or "unknown", target=f"matrix[{index}].platform", expected="unique android or ios", status="blocked", remediation="为每个平台保留唯一矩阵行")
ok = False
seen.add(platform_name)
tester = row.get("tester")
if not isinstance(tester, dict) or not isinstance(tester.get("binding"), str) or not tester["binding"].startswith("tester."):
evidence.add(f"{prefix}-binding", platform=platform_name, target=f"matrix[{index}].tester.binding", expected="tester.<name>", status="blocked", remediation="由 agent-runtime 解析并传入 tester binding")
ok = False
if not isinstance(tester, dict) or not isinstance(tester.get("node"), str) or not tester["node"].strip():
evidence.add(f"{prefix}-node", platform=platform_name, target=f"matrix[{index}].tester.node", expected="actual tester node", status="blocked", remediation="提供实际 tester 节点名")
ok = False
artifact = row.get("artifact")
identity_key = "package" if platform_name == "android" else "bundle_id"
if not isinstance(artifact, dict) or not isinstance(artifact.get("path"), str) or not artifact["path"].strip():
evidence.add(f"{prefix}-artifact", platform=platform_name, target=f"matrix[{index}].artifact.path", expected="caller-provided artifact path", status="blocked", remediation="提供本轮 APK/IPA,不要让 preflight 猜产物")
ok = False
if not isinstance(artifact, dict) or not isinstance(artifact.get(identity_key), str) or not artifact[identity_key].strip():
evidence.add(f"{prefix}-identity", platform=platform_name, target=f"matrix[{index}].artifact.{identity_key}", expected="declared artifact identity", status="blocked", remediation=f"声明 artifact.{identity_key}")
ok = False
if isinstance(artifact, dict) and artifact.get("sha256") is not None and not SHA256.fullmatch(text(artifact.get("sha256"))):
evidence.add(f"{prefix}-sha256", platform=platform_name, target=f"matrix[{index}].artifact.sha256", expected="64 hex characters", status="blocked", remediation="提供正确 SHA-256 或删除字段")
ok = False
if isinstance(artifact, dict) and artifact.get("source_commit") is not None and not COMMIT.fullmatch(text(artifact.get("source_commit"))):
evidence.add(f"{prefix}-commit", platform=platform_name, target=f"matrix[{index}].artifact.source_commit", expected="40-character commit SHA", status="blocked", remediation="提供产物来源 commit SHA")
ok = False
return ok
def artifact_identity(path: Path, platform_name: str, artifact: Mapping[str, Any]) -> tuple[str | None, str]:
if platform_name == "ios":
try:
with zipfile.ZipFile(path) as archive:
names = [name for name in archive.namelist() if re.fullmatch(r"Payload/[^/]+\.app/Info\.plist", name)]
if len(names) != 1:
return None, "IPA 中没有唯一 Payload/*.app/Info.plist"
plist = plistlib.loads(archive.read(names[0]))
value = plist.get("CFBundleIdentifier")
return (value, "Info.plist CFBundleIdentifier") if isinstance(value, str) else (None, "Info.plist 缺少 CFBundleIdentifier")
except (OSError, zipfile.BadZipFile, plistlib.InvalidFileException) as exc:
return None, f"IPA 读取失败: {exc}"
command = argv(artifact.get("identity_command"))
if command:
command = [item.replace("{artifact}", str(path)) for item in command]
else:
tool = shutil.which("apkanalyzer") or shutil.which("aapt2") or shutil.which("aapt")
if not tool:
return None, "缺少 apkanalyzer/aapt2/aapt;无法读取 APK package identity"
command = [tool, "manifest", "application-id", str(path)] if tool.endswith("apkanalyzer") else [tool, "dump", "badging", str(path)]
code, output = command_text(command)
if code != 0:
return None, output[-500:] or f"identity command exited {code}"
match = re.search(r"(?:application-id|name)\s*[:=]\s*['\"]?([A-Za-z0-9_.]+)", output)
if not match:
match = re.search(r"package:\s+name='([^']+)'", output)
if not match:
match = re.fullmatch(r"\s*([A-Za-z0-9_.]+)\s*", output)
return (match.group(1), output[-500:]) if match else (None, "identity command output did not contain package")
def record_artifact(evidence: Evidence, row: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
artifact = row.get("artifact") or {}
path = Path(text(artifact.get("path"))).expanduser()
prefix = f"{platform_name}-{index}-artifact"
if not path.is_file():
evidence.add(prefix, platform=platform_name, node=node, target=str(path), expected="existing caller-provided artifact", status="blocked", remediation="补齐 APK/IPA 路径;不要默认构建")
return False
actual_hash = sha256(path)
expected_hash = artifact.get("sha256")
if expected_hash and actual_hash.lower() != text(expected_hash).lower():
evidence.add(prefix + "-hash", platform=platform_name, node=node, target=str(path), observed=actual_hash, expected=expected_hash, status="blocked", remediation="确认产物路径或 SHA-256 与本轮构建身份一致")
return False
evidence.add(prefix + "-hash", platform=platform_name, node=node, target=str(path), observed=actual_hash, expected=expected_hash or "file SHA-256 recorded", status="ready")
identity, observed = artifact_identity(path, platform_name, artifact)
expected_identity = text(artifact.get("package" if platform_name == "android" else "bundle_id"))
if identity != expected_identity:
evidence.add(prefix + "-identity", platform=platform_name, node=node, target=str(path), observed=identity or observed, expected=expected_identity, status="environment-fail", remediation="使用与 catalog 身份一致的 APK/IPA,并重新校验产物")
return False
evidence.add(prefix + "-identity", platform=platform_name, node=node, target=str(path), observed=identity, expected=expected_identity, status="ready")
if artifact.get("source_commit"):
evidence.add(prefix + "-commit", platform=platform_name, node=node, target=str(path), observed=artifact.get("source_commit"), expected=artifact.get("source_commit"), status="ready")
return True
def local_platform_probe(platform_name: str) -> tuple[str, dict[str, dict[str, Any]]]:
if platform_name == "android":
adb = shutil.which("adb")
if not adb:
return "environment-fail", {"adb": {"status": "environment-fail", "observed": "missing", "expected": "Android SDK platform-tools", "remediation": "在 tester.android 节点安装并配置 adb"}}
code, output = command_text([adb, "devices", "-l"])
states = [line.split() for line in output.splitlines()[1:] if line.strip()]
device = next((parts[0] for parts in states if len(parts) > 1 and parts[1] == "device"), "")
bad = next((parts for parts in states if len(parts) > 1 and parts[1] in {"unauthorized", "offline"}), None)
if bad:
return "environment-fail", {"adb": {"status": "ready", "observed": output, "expected": "adb responds"}, "device": {"status": "environment-fail", "observed": " ".join(bad), "expected": "authorized device", "remediation": "解锁 Android 设备并接受 RSA 授权;offline 时重启 adb/设备"}}
if code != 0 or not device:
return "environment-fail", {"adb": {"status": "environment-fail", "observed": output, "expected": "one authorized device", "remediation": "连接并授权 Android device/emulator"}}
boot_code, boot = command_text([adb, "-s", device, "shell", "getprop", "sys.boot_completed"])
booted = boot_code == 0 and boot.strip() == "1"
return ("ready" if booted else "environment-fail", {
"adb": {"status": "ready", "observed": adb, "expected": "adb available"},
"device": {"status": "ready", "observed": device, "expected": "authorized device"},
"boot": {"status": "ready" if booted else "environment-fail", "observed": boot, "expected": "sys.boot_completed=1", "remediation": "等待 emulator boot complete 后重跑"},
})
if host_platform.system() != "Darwin":
return "environment-fail", {"macos": {"status": "environment-fail", "observed": host_platform.system(), "expected": "Darwin/macOS tester.ios node", "remediation": "将 tester.ios 指向可用 macOS,不要在 Linux 伪装 iOS"}}
xcrun = shutil.which("xcrun")
if not xcrun:
return "environment-fail", {"xcrun": {"status": "environment-fail", "observed": "missing", "expected": "xcrun and simctl", "remediation": "在 Mac 安装 Xcode Command Line Tools/Xcode"}}
code, output = command_text([xcrun, "simctl", "list", "devices", "available"])
booted = any("(Booted)" in line for line in output.splitlines())
return ("ready" if code == 0 and booted else "environment-fail", {
"macos": {"status": "ready", "observed": "Darwin", "expected": "Darwin/macOS"},
"xcrun": {"status": "ready" if code == 0 else "environment-fail", "observed": output[-500:], "expected": "simctl available", "remediation": "安装 Xcode 并接受许可"},
"boot": {"status": "ready" if booted else "environment-fail", "observed": "Booted" if booted else output[-300:], "expected": "Booted simulator", "remediation": "启动目标 Simulator 后重跑"},
})
def runner_call(runner: Mapping[str, Any], request: Mapping[str, Any]) -> tuple[str, str, dict[str, Any]]:
command = argv(runner.get("command"))
if not command:
return "blocked", "runner command missing", {}
code, output = command_text(command, payload=request)
if code != 0:
return "environment-fail", output[-800:], {}
try:
result = json.loads(output)
except json.JSONDecodeError:
return "environment-fail", output[-800:], {}
if not isinstance(result, dict):
return "environment-fail", "runner returned non-object", {}
return status_from(result.get("status")), text(result.get("observed", "")), result
def add_runner_checks(evidence: Evidence, result: Mapping[str, Any], *, prefix: str, platform_name: str, node: str, default_expected: str) -> str:
checks = result.get("checks")
if isinstance(checks, dict) and checks:
statuses: list[str] = []
for name, detail in checks.items():
detail = detail if isinstance(detail, dict) else {"observed": detail}
status = status_from(detail.get("status"), status_from(result.get("status")))
statuses.append(status)
evidence.add(f"{prefix}-{name}", platform=platform_name, node=node, target=text(detail.get("target", name)), observed=detail.get("observed", result.get("observed", "")), expected=detail.get("expected", default_expected), status=status, remediation=text(detail.get("remediation", "")))
return "environment-fail" if "environment-fail" in statuses else ("blocked" if "blocked" in statuses else "ready")
status = status_from(result.get("status"))
evidence.add(prefix, platform=platform_name, node=node, target=prefix, observed=result.get("observed", ""), expected=default_expected, status=status, remediation=text(result.get("remediation", "")))
return status
def require_runner_checks(evidence: Evidence, result: Mapping[str, Any], *, prefix: str, platform_name: str, node: str, required: set[str]) -> bool:
checks = result.get("checks")
missing = sorted(required - set(checks)) if isinstance(checks, dict) else sorted(required)
if not missing:
return True
evidence.add(prefix + "-contract", platform=platform_name, node=node, target="runner response", observed={"missing": missing}, expected=sorted(required), status="blocked", remediation="让 tester 节点 adapter 返回每个必需的 adb/boot、macOS/xcrun/simctl 或 DNS/TCP/HTTP 证据")
return False
def network_local(url: str) -> list[dict[str, Any]]:
parsed = urllib.parse.urlsplit(url)
host = parsed.hostname or ""
port = parsed.port or (443 if parsed.scheme == "https" else 80)
try:
addresses = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
except OSError as exc:
return [{"name": "dns", "status": "environment-fail", "observed": str(exc), "expected": "resolvable hostname", "remediation": "检查 tester 节点 DNS/VPN/网络"}]
rows: list[dict[str, Any]] = [{"name": "dns", "status": "ready", "observed": sorted({item[4][0] for item in addresses}), "expected": "resolvable hostname"}]
try:
with socket.create_connection((host, port), timeout=5):
rows.append({"name": "tcp", "status": "ready", "observed": f"{host}:{port}", "expected": "TCP connection"})
except OSError as exc:
rows.append({"name": "tcp", "status": "environment-fail", "observed": str(exc), "expected": f"TCP {host}:{port}", "remediation": "检查端口、防火墙和测试节点到 HOST 的路由"})
return rows
try:
with urllib.request.urlopen(urllib.request.Request(url, method="GET"), timeout=5) as response:
code = response.status
rows.append({"name": "http", "status": "ready" if 200 <= code < 500 else "environment-fail", "observed": code, "expected": "HTTP 2xx-4xx", "remediation": "检查后端健康状态" if code >= 500 else ""})
except urllib.error.HTTPError as exc:
rows.append({"name": "http", "status": "ready" if 200 <= exc.code < 500 else "environment-fail", "observed": exc.code, "expected": "HTTP 2xx-4xx", "remediation": "检查后端服务,不把 5xx 当成可用" if exc.code >= 500 else ""})
except (OSError, urllib.error.URLError) as exc:
rows.append({"name": "http", "status": "environment-fail", "observed": str(exc), "expected": "HTTP response", "remediation": "检查服务监听、代理、TLS 和节点网络"})
return rows
def check_network(evidence: Evidence, config: Mapping[str, Any], row: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
tester = row.get("tester") or {}
runner = tester.get("runner") if isinstance(tester, dict) else None
is_local = node in {"local", "localhost", "127.0.0.1"}
statuses: list[str] = []
for probe_index, probe in enumerate(config.get("service_probes") or []):
raw_url = text(probe.get("url"))
if "{PUBLIC_HOST}" in raw_url and not text(config.get("public_host")).strip():
evidence.add(f"{platform_name}-{index}-network-{probe_index}-public-host", platform=platform_name, node=node, target=raw_url, expected="PUBLIC_HOST configured", status="blocked", remediation="提供 PUBLIC_HOST 或改用 HOST 探针")
statuses.append("blocked")
continue
url = normalize_url(expand(raw_url, config))
if not urllib.parse.urlsplit(url).hostname:
evidence.add(f"{platform_name}-{index}-network-{probe_index}-url", platform=platform_name, node=node, target=url, expected="valid probe URL", status="blocked", remediation="修正 service_probes URL 或 HOST/PUBLIC_HOST")
statuses.append("blocked")
continue
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
details = network_local(url)
expected_statuses = probe.get("expected_status")
for detail in details:
if detail["name"] == "http" and isinstance(expected_statuses, list) and detail["status"] == "ready" and detail["observed"] not in expected_statuses:
detail = {**detail, "status": "environment-fail", "expected": expected_statuses, "remediation": "检查服务探针的 expected_status 或后端响应"}
status = status_from(detail["status"])
statuses.append(status)
evidence.add(f"{platform_name}-{index}-network-{probe_index}-{detail['name']}", platform=platform_name, node=node, target=url, observed=detail["observed"], expected=detail["expected"], status=status, remediation=detail.get("remediation", ""))
elif isinstance(runner, dict) and runner.get("kind") == "contract":
binding = text(tester.get("binding")) if isinstance(tester, dict) else ""
status, observed, result = runner_call(runner, {"operation": "network_probe", "platform": platform_name, "node": node, "binding": binding, "url": url, "probe": probe})
result = result or {"status": status, "observed": observed}
complete = require_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-network-{probe_index}", platform_name=platform_name, node=node, required={"dns", "tcp", "http"})
network_status = add_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-network-{probe_index}", platform_name=platform_name, node=node, default_expected="DNS/TCP/HTTP reachable from actual tester node")
statuses.append(network_status if complete else "blocked")
else:
evidence.add(f"{platform_name}-{index}-network-{probe_index}", platform=platform_name, node=node, target=url, expected="node runner contract", status="blocked", remediation="远端 tester 必须提供 agent-runtime adapter runner")
statuses.append("blocked")
return "environment-fail" not in statuses and "blocked" not in statuses
def check_fixture_cleanup(evidence: Evidence, config: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
ok = True
fixture = config.get("fixtures")
valid_fixture = isinstance(fixture, dict) and (not fixture.get("required", True) or bool(fixture.get("references")))
if not valid_fixture:
evidence.add(f"{platform_name}-{index}-fixture", platform=platform_name, node=node, target="fixtures", expected="required fixture references", status="blocked", remediation="声明隔离测试账号/fixture;不要在运行中临时创建")
ok = False
else:
evidence.add(f"{platform_name}-{index}-fixture", platform=platform_name, node=node, target="fixtures", observed=fixture.get("references"), expected="declared fixture references", status="ready")
cleanup = config.get("cleanup")
valid_noop = isinstance(cleanup, dict) and cleanup.get("required") is False and bool(text(cleanup.get("reason")).strip())
valid_steps = isinstance(cleanup, dict) and (bool(cleanup.get("steps")) or bool(text(cleanup.get("run")).strip()))
if not (valid_noop or valid_steps):
evidence.add(f"{platform_name}-{index}-cleanup", platform=platform_name, node=node, target="cleanup", expected="steps/run or explicit read-only reason", status="blocked", remediation="声明 teardown/rollback;清理缺失时禁止开始测试")
ok = False
else:
evidence.add(f"{platform_name}-{index}-cleanup", platform=platform_name, node=node, target="cleanup", observed=cleanup, expected="cleanup contract declared", status="ready")
return ok
def run(config: Mapping[str, Any], mode: str) -> Evidence:
evidence = Evidence(mode)
shape_ok = validate_shape(config, evidence)
deployment_ok = True
if mode == "prepare":
deployment = config.get("deployment")
environment = deployment.get("environment") if isinstance(deployment, dict) else None
command = argv(deployment.get("command")) if isinstance(deployment, dict) else None
if environment != "test":
evidence.add("deployment-environment", target="deployment.environment", observed=environment, expected="test", status="blocked", remediation="prepare 只允许 deployer 的 test 环境")
if not command:
evidence.add("deployment-command", target="deployment.command", expected="existing deployer adapter command", status="blocked", remediation="提供既有 deployer 的 JSON command contract;不要在 skill 内实现部署")
deployment_ok = False
elif not command:
evidence.add("deployment-command", target="deployment.command", expected="existing deployer adapter command", status="blocked", remediation="提供既有 deployer 的 JSON command contract;不要在 skill 内实现部署")
deployment_ok = False
else:
code, output = command_text(command, payload={"operation": "prepare", "environment": "test", "project_root": config.get("project_root"), "host": config.get("host"), "public_host": config.get("public_host")})
if code != 0:
evidence.add("deployment", target="test", observed=output[-800:], expected="deployer receipt status=ready", status="environment-fail", remediation="按 deployer 返回证据修复测试后端,再重跑 prepare")
deployment_ok = False
else:
try:
receipt = json.loads(output)
except json.JSONDecodeError:
receipt = {}
status = status_from(receipt.get("status"), "environment-fail") if isinstance(receipt, dict) else "environment-fail"
evidence.add("deployment", target="test", observed=receipt.get("observed", output[-800:]) if isinstance(receipt, dict) else output[-800:], expected="deployer receipt status=ready", status=status, remediation="检查既有 deployer 的测试环境 receipt" if status != "ready" else "")
deployment_ok = status == "ready"
if not shape_ok:
return evidence
for index, row in enumerate(config["matrix"]):
platform_name = text(row.get("platform")).lower()
tester = row.get("tester") or {}
node = text(tester.get("node"))
binding = text(tester.get("binding"))
evidence.add(f"{platform_name}-{index}-binding", platform=platform_name, node=node, target=binding, observed=binding, expected="resolved tester.<name> binding", status="ready")
artifact_ok = record_artifact(evidence, row, platform_name, node, index)
fixture_ok = check_fixture_cleanup(evidence, config, platform_name, node, index)
runner = tester.get("runner") if isinstance(tester, dict) else None
is_local = node in {"local", "localhost", "127.0.0.1"}
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
platform_status, details = local_platform_probe(platform_name)
for name, detail in details.items():
evidence.add(f"{platform_name}-{index}-platform-{name}", platform=platform_name, node=node, target=name, observed=detail.get("observed", ""), expected=detail.get("expected", "platform ready"), status=status_from(detail.get("status"), platform_status), remediation=detail.get("remediation", ""))
platform_ok = platform_status == "ready"
elif isinstance(runner, dict) and runner.get("kind") == "contract":
status, observed, result = runner_call(runner, {"operation": "platform_probe", "platform": platform_name, "node": node, "binding": binding})
result = result or {"status": status, "observed": observed}
required = {"adb", "device", "boot"} if platform_name == "android" else {"macos", "xcrun", "simctl", "boot"}
complete = require_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-platform", platform_name=platform_name, node=node, required=required)
platform_status = add_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-platform", platform_name=platform_name, node=node, default_expected="platform/device ready")
platform_ok = complete and platform_status == "ready"
else:
evidence.add(f"{platform_name}-{index}-platform-runner", platform=platform_name, node=node, target=node, expected="node runner contract", status="blocked", remediation="远端 tester 必须由 agent-runtime 提供可执行 runner")
platform_ok = False
network_ok = check_network(evidence, config, row, platform_name, node, index)
# This isolated project assumes the client capabilities used by its
# journeys. Their real behavior is a product assertion, not a preflight
# environment gate.
row_ok = artifact_ok and fixture_ok and platform_ok and network_ok
if mode == "prepare":
artifact = row.get("artifact") or {}
identity = artifact.get("package" if platform_name == "android" else "bundle_id")
target_path = text(artifact.get("remote_path") or artifact.get("path"))
if not row_ok or not deployment_ok:
evidence.add(f"{platform_name}-{index}-install", platform=platform_name, node=node, target=target_path, observed="not-run", expected="verify and deployment ready", status="blocked", remediation="先修复前置检查;prepare 不伪造安装成功")
evidence.add(f"{platform_name}-{index}-start", platform=platform_name, node=node, target=text(identity), observed="not-run", expected="install ready", status="blocked", remediation="安装前置失败,未启动 app")
else:
install_request = {"operation": "install", "platform": platform_name, "node": node, "artifact": target_path, "identity": identity}
start_request = {"operation": "start", "platform": platform_name, "node": node, "identity": identity}
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
if platform_name == "android":
adb = shutil.which("adb") or "adb"
serial = os.environ.get("ANDROID_SERIAL", "")
install_code, install_output = command_text([adb, "-s", serial, "install", "-r", target_path] if serial else [adb, "install", "-r", target_path])
install_result = {"status": "ready" if install_code == 0 else "environment-fail", "observed": install_output}
launch = [adb, "-s", serial, "shell", "monkey", "-p", text(identity), "1"] if serial else [adb, "shell", "monkey", "-p", text(identity), "1"]
start_code, start_output = command_text(launch) if install_code == 0 else (1, "install failed")
start_result = {"status": "ready" if start_code == 0 else "environment-fail", "observed": start_output}
else:
install_result = {"status": "environment-fail", "observed": "local iOS install requires a macOS runner", "remediation": "use tester.ios contract runner on Mac"}
start_result = {"status": "blocked", "observed": "not-run", "remediation": "install failed"}
add_runner_checks(evidence, install_result, prefix=f"{platform_name}-{index}-install", platform_name=platform_name, node=node, default_expected="artifact installed")
add_runner_checks(evidence, start_result, prefix=f"{platform_name}-{index}-start", platform_name=platform_name, node=node, default_expected="app launched")
elif isinstance(runner, dict) and runner.get("kind") == "contract":
install_status, install_observed, install_result = runner_call(runner, install_request)
add_runner_checks(evidence, install_result or {"status": install_status, "observed": install_observed}, prefix=f"{platform_name}-{index}-install", platform_name=platform_name, node=node, default_expected="artifact installed")
if install_status == "ready":
start_status, start_observed, start_result = runner_call(runner, start_request)
add_runner_checks(evidence, start_result or {"status": start_status, "observed": start_observed}, prefix=f"{platform_name}-{index}-start", platform_name=platform_name, node=node, default_expected="app launched")
else:
evidence.add(f"{platform_name}-{index}-start", platform=platform_name, node=node, target=text(identity), observed="not-run", expected="install ready", status="blocked", remediation="安装失败,先修复 tester 节点")
return evidence
def write_evidence(evidence: Evidence, yaml_path: Path, csv_path: Path) -> dict[str, Any]:
payload = evidence.payload()
yaml_path.parent.mkdir(parents=True, exist_ok=True)
csv_path.parent.mkdir(parents=True, exist_ok=True)
yaml_path.write_text(yaml.safe_dump(payload, allow_unicode=True, sort_keys=False), encoding="utf-8")
with csv_path.open("w", newline="", encoding="utf-8") as handle:
writer = csv.DictWriter(handle, fieldnames=EVIDENCE_FIELDS)
writer.writeheader()
writer.writerows(evidence.rows)
return {"status": payload["status"], "yaml": str(yaml_path), "csv": str(csv_path), "checks": len(evidence.rows), "summary": payload["summary"]}
def main() -> int:
parser = argparse.ArgumentParser(description="Verify or prepare a test environment and emit YAML/CSV evidence")
parser.add_argument("mode", choices=("verify", "prepare"))
parser.add_argument("--config", required=True, type=Path)
parser.add_argument("--yaml", dest="yaml_path", type=Path, default=Path("preflight.yaml"))
parser.add_argument("--csv", dest="csv_path", type=Path, default=Path("preflight.csv"))
args = parser.parse_args()
try:
config = load_config(args.config)
evidence = run(config, args.mode)
result = write_evidence(evidence, args.yaml_path, args.csv_path)
except (OSError, ValueError, yaml.YAMLError) as exc:
print(json.dumps({"status": "blocked", "error": str(exc)}, ensure_ascii=False), file=sys.stderr)
return 2
print(json.dumps(result, ensure_ascii=False, indent=2))
return 0 if result["status"] == "ready" else 1
if __name__ == "__main__":
raise SystemExit(main())
+75
View File
@@ -0,0 +1,75 @@
#!/usr/bin/env python3
"""Deterministic runner fixture for test-preflight contract tests."""
import json
import os
import sys
request = json.load(sys.stdin)
scenario = os.environ.get("PREFLIGHT_RUNNER_SCENARIO", "ready")
operation = request.get("operation")
if operation in {"deployment", "prepare"}:
print(json.dumps({"status": "ready", "observed": "mock test deploy receipt"}))
elif operation == "platform_probe":
platform = request.get("platform")
if scenario == "android-unauthorized" and platform == "android":
print(json.dumps({"status": "environment-fail", "checks": {
"adb": {"status": "ready", "observed": "adb 35", "expected": "adb responds"},
"device": {"status": "environment-fail", "observed": "emulator-5554 unauthorized", "expected": "authorized device", "remediation": "accept RSA authorization"},
"boot": {"status": "environment-fail", "observed": "unknown", "expected": "boot complete"},
}}))
elif scenario == "mac-unavailable" and platform == "ios":
print(json.dumps({"status": "environment-fail", "checks": {
"macos": {"status": "environment-fail", "observed": "Mac unavailable", "expected": "Darwin tester node", "remediation": "restore tester.ios Mac"},
}}))
else:
print(json.dumps({"status": "ready", "checks": {
"adb": {"status": "ready", "observed": "adb 35", "expected": "adb responds"},
"device": {"status": "ready", "observed": "authorized", "expected": "authorized device"},
"boot": {"status": "ready", "observed": "boot complete", "expected": "boot complete"},
"macos": {"status": "ready", "observed": "Darwin", "expected": "Darwin"},
"xcrun": {"status": "ready", "observed": "xcrun", "expected": "xcrun"},
"simctl": {"status": "ready", "observed": "available", "expected": "simctl"},
}}))
elif operation == "network_probe":
if scenario == "host-unreachable":
print(json.dumps({"status": "environment-fail", "checks": {
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
"tcp": {"status": "environment-fail", "observed": "connection refused", "expected": "TCP reachable", "remediation": "check HOST route"},
}}))
elif scenario == "http-failure":
print(json.dumps({"status": "environment-fail", "checks": {
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
"tcp": {"status": "ready", "observed": "443 open", "expected": "TCP reachable"},
"http": {"status": "environment-fail", "observed": 503, "expected": "HTTP 2xx-4xx", "remediation": "check service health"},
}}))
else:
print(json.dumps({"status": "ready", "checks": {
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
"tcp": {"status": "ready", "observed": "443 open", "expected": "TCP reachable"},
"http": {"status": "ready", "observed": 200, "expected": "HTTP 2xx-4xx"},
}}))
elif operation == "capability_probe":
capability = request.get("capability")
checks = {
"album_picker": {
"picker": {"status": "ready", "observed": "real picker opened and returned image-001"},
},
"photo_permission": {
"permission": {"status": "ready", "observed": "permission state checked on device"},
"test_image": {"status": "ready", "observed": "image-001 available in test album"},
},
"ime_pinyin": {
"composition": {"status": "ready", "observed": "nihao composed in Chinese IME"},
"candidates": {"status": "ready", "observed": "你好 candidate selected"},
},
}
if scenario == "capability-missing" and capability == "album_picker":
print(json.dumps({"status": "ready", "checks": {}}))
else:
print(json.dumps({"status": "ready", "checks": checks.get(capability, {})}))
elif operation in {"install", "start"}:
print(json.dumps({"status": "ready", "observed": operation + "ed mock artifact"}))
else:
print(json.dumps({"status": "environment-fail", "observed": "unknown operation"}))
@@ -0,0 +1,225 @@
#!/usr/bin/env python3
from __future__ import annotations
import json
import os
import plistlib
import subprocess
import sys
import tempfile
import unittest
import zipfile
from pathlib import Path
from unittest import mock
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "scripts"))
import preflight # noqa: E402
FIXTURE_RUNNER = Path(__file__).resolve().parent / "fixtures" / "contract_runner.py"
class PreflightTests(unittest.TestCase):
def setUp(self) -> None:
self.temp = tempfile.TemporaryDirectory()
self.root = Path(self.temp.name)
self.apk = self.root / "writes.apk"
self.apk.write_bytes(b"mock apk")
self.ipa = self.root / "writes.ipa"
with zipfile.ZipFile(self.ipa, "w") as archive:
archive.writestr("Payload/Writes.app/Info.plist", plistlib.dumps({"CFBundleIdentifier": "page.writes"}))
self.identity = self.root / "identity.py"
self.identity.write_text("print('page.writes')\n", encoding="utf-8")
def tearDown(self) -> None:
self.temp.cleanup()
def runner(self, scenario: str = "ready") -> dict[str, object]:
return {"kind": "contract", "command": [sys.executable, str(FIXTURE_RUNNER)]}
def config(self, *, platform: str = "android", artifact: Path | None = None, node: str = "tester.android", scenario: str = "ready") -> dict[str, object]:
artifact = artifact or self.apk
artifact_row: dict[str, object] = {
"path": str(artifact),
"package": "page.writes" if platform == "android" else None,
"bundle_id": "page.writes" if platform == "ios" else None,
}
if platform == "android":
artifact_row["identity_command"] = [sys.executable, str(self.identity)]
artifact_row.pop("bundle_id")
else:
artifact_row.pop("package")
return {
"schema_version": 1,
"project_root": str(self.root),
"host": "https://test.example.invalid",
"public_host": "https://public.example.invalid",
"service_probes": [{"id": "health", "url": "{HOST}/health", "expected_status": [200]}],
"fixtures": {"required": True, "references": ["isolated-account"]},
"cleanup": {"required": True, "steps": ["cleanup-run-data"]},
"matrix": [{
"platform": platform,
"tester": {"binding": "tester.android" if platform == "android" else "tester.ios", "node": node, "runner": self.runner(scenario)},
"artifact": artifact_row,
}],
}
def write_config(self, data: dict[str, object]) -> Path:
path = self.root / "matrix.yaml"
import yaml
path.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8")
return path
def invoke(self, mode: str, data: dict[str, object]) -> tuple[int, dict[str, object], dict[str, object]]:
config = self.write_config(data)
yaml_path, csv_path = self.root / f"{mode}.yaml", self.root / f"{mode}.csv"
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "ready"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), mode,
"--config", str(config), "--yaml", str(yaml_path), "--csv", str(csv_path),
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
import yaml
return process.returncode, yaml.safe_load(yaml_path.read_text(encoding="utf-8")), {"stdout": process.stdout, "stderr": process.stderr}
def test_missing_artifact_is_blocked_and_report_has_contract_columns(self):
data = self.config(artifact=self.root / "missing.apk")
code, report, _ = self.invoke("verify", data)
self.assertEqual(code, 1)
self.assertEqual(report["status"], "blocked")
row = next(row for row in report["checks"] if row["check_id"] == "android-0-artifact")
self.assertEqual(row["status"], "blocked")
self.assertEqual(set(row), set(preflight.EVIDENCE_FIELDS))
def test_android_unauthorized_is_environment_fail(self):
data = self.config()
config = self.write_config(data)
out = self.root / "unauthorized.yaml"
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "android-unauthorized"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config), "--yaml", str(out), "--csv", str(self.root / "u.csv")
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
report = json.loads(json.dumps(__import__("yaml").safe_load(out.read_text(encoding="utf-8"))))
self.assertEqual(process.returncode, 1)
self.assertEqual(report["status"], "environment-fail")
self.assertEqual(next(row for row in report["checks"] if row["check_id"].endswith("-device"))["status"], "environment-fail")
def test_linux_without_xcrun_is_environment_fail(self):
data = self.config(platform="ios", artifact=self.ipa, node="local")
data["matrix"][0]["tester"]["runner"] = {"kind": "local"}
with mock.patch.object(preflight.host_platform, "system", return_value="Linux"), mock.patch.object(preflight, "network_local", return_value=[{"name": "dns", "status": "ready", "observed": "ok", "expected": "DNS"}, {"name": "tcp", "status": "ready", "observed": "ok", "expected": "TCP"}, {"name": "http", "status": "ready", "observed": 200, "expected": "HTTP"}]):
result = preflight.run(data, "verify")
self.assertEqual(result.overall(), "environment-fail")
row = next(row for row in result.rows if row["check_id"].endswith("-platform-macos"))
self.assertEqual(row["status"], "environment-fail")
def test_mac_unavailable_contract_is_environment_fail(self):
data = self.config(platform="ios", artifact=self.ipa)
config_path = self.write_config(data)
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "mac-unavailable"
output = self.root / "mac.yaml"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "mac.csv")
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
import yaml
report = yaml.safe_load(output.read_text(encoding="utf-8"))
self.assertEqual(process.returncode, 1)
self.assertEqual(report["status"], "environment-fail")
self.assertTrue(any(row["check_id"].endswith("-macos") and row["status"] == "environment-fail" for row in report["checks"]))
def test_missing_fixture_or_cleanup_is_blocked(self):
data = self.config()
data["fixtures"] = {"required": True, "references": []}
data["cleanup"] = {"required": True, "steps": []}
result = preflight.run(data, "verify")
self.assertEqual(result.overall(), "blocked")
self.assertTrue(any(row["check_id"].endswith("-fixture") and row["status"] == "blocked" for row in result.rows))
self.assertTrue(any(row["check_id"].endswith("-cleanup") and row["status"] == "blocked" for row in result.rows))
def test_mobile_capabilities_are_assumed_and_not_a_preflight_gate(self):
data = self.config()
config_path = self.write_config(data)
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "capability-missing"
output = self.root / "capabilities.yaml"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify",
"--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "capabilities.csv"),
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
report = __import__("yaml").safe_load(output.read_text(encoding="utf-8"))
self.assertEqual(process.returncode, 0)
self.assertEqual(report["status"], "ready")
self.assertFalse(any("capability-" in row["check_id"] for row in report["checks"]))
def test_host_unreachable_from_actual_runner_is_environment_fail(self):
data = self.config()
config_path = self.write_config(data)
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "host-unreachable"
output = self.root / "host.yaml"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "host.csv")
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
import yaml
report = yaml.safe_load(output.read_text(encoding="utf-8"))
self.assertEqual(process.returncode, 1)
self.assertEqual(report["status"], "environment-fail")
self.assertTrue(any("network" in row["check_id"] and row["status"] == "environment-fail" for row in report["checks"]))
def test_http_500_is_environment_fail_not_product_fail(self):
data = self.config()
config_path = self.write_config(data)
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "http-failure"
output = self.root / "http.yaml"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "http.csv")
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
import yaml
report = yaml.safe_load(output.read_text(encoding="utf-8"))
self.assertEqual(process.returncode, 1)
http_row = next(row for row in report["checks"] if row["check_id"].endswith("-http"))
self.assertEqual(http_row["observed"], "503")
self.assertEqual(http_row["status"], "environment-fail")
def test_prepare_without_deployer_is_blocked_and_never_installs(self):
data = self.config()
code, report, _ = self.invoke("prepare", data)
self.assertEqual(code, 1)
self.assertEqual(report["status"], "blocked")
self.assertFalse(any(row["check_id"].endswith("-install") and row["status"] == "ready" for row in report["checks"]))
self.assertTrue(any(row["check_id"] == "deployment-command" for row in report["checks"]))
def test_prepare_mock_contract_calls_install_and_start(self):
data = self.config()
data["deployment"] = {"environment": "test", "command": [sys.executable, str(FIXTURE_RUNNER)]}
code, report, _ = self.invoke("prepare", data)
self.assertEqual(code, 0)
self.assertEqual(report["status"], "ready")
ids = {row["check_id"] for row in report["checks"]}
self.assertIn("android-0-install", ids)
self.assertIn("android-0-start", ids)
def test_verify_never_calls_install_or_start(self):
data = self.config()
calls: list[str] = []
original = preflight.runner_call
def record(runner, request):
calls.append(str(request["operation"]))
return original(runner, request)
with mock.patch.object(preflight, "runner_call", side_effect=record):
result = preflight.run(data, "verify")
self.assertEqual(result.overall(), "ready")
self.assertNotIn("install", calls)
self.assertNotIn("start", calls)
self.assertNotIn("capability_probe", calls)
if __name__ == "__main__":
unittest.main()
+63 -2
View File
@@ -27,7 +27,12 @@
每轮测试会创建新的 terminal,结束后自动关闭;不会复用或关闭已有 terminal。
- 测试环境部署由 deployer 在本机 Compose 中触发;正式环境部署由本机 `~/code/app00`
项目的 fresh Agent terminal 触发,tester 不直接 SSH 或执行正式 Compose。
- 会动硬件、会改数据的用例带风险级别(`readonly` / `mutating` / `destructive`)。默认只跑最低档 `readonly`。要跑更高风险,需要你明确说,例如「风险到 mutating」;碰哪个子系统另由 `gate`(如 hardware / weld)控制。
- 会动硬件、会改数据的用例带风险级别(`readonly` / `mutating` / `destructive`)。默认
`standard` 只跑最低档 `readonly`。隔离、可重置的 Writes 类环境在 catalog 设置
`execution_profile: isolated-full` 后,所有 active 自动化用例都进入执行,不因
`AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或
`AUTHORIZE_PASSWORD` skipped;risk 原值仍保留。设备、产物、服务和 cleanup/rollback
由 test-preflight 保证;客户端相册、权限和中文输入能力默认具备,由 journey 用例验证。
## 使用示例
@@ -76,7 +81,8 @@ APK/IPA,为本轮创建隔离 worktree,在 Android 和 iOS tester 节点安
不执行可用平台;只有全部可用才按目标环境走 deployer 路由、解析 Agent、输出生效
列表、准备 Orca 项目并创建临时 terminal。脚本类、浏览器和移动模拟器都在对应
Agent 环境中执行。`builder.<platform>` 不能替代 `tester.<platform>`。
4. 超出你允许的风险级别会跳过,不会偷偷执行。
4. `standard` 下超出你允许的风险级别会跳过;`isolated-full` 由 catalog profile 明确
全量 active 执行,不把风险悄悄改成 readonly。
5. 无论测试成功、失败或异常,都会关闭本轮创建的 terminal;跑完给出按分类计数的
结果,以及 `pass / fail / environment-fail / skipped / not-run`。
@@ -85,3 +91,58 @@ APK/IPA,为本轮创建隔离 worktree,在 Android 和 iOS tester 节点安
- **列出**:能看到分类表和总条数;`.pouch/tester/cases.csv` 已由 skill 里的 `generate_cases_csv.py` 从 `cases/` 重新生成。每次运行的结果保存在独立的 `.pouch/tester/runs/<run-id>.*` 文件中。
- **维护**:目标 id 能 `查看` 到,且检查通过。
- **运行**:有本次 `runs/*.yaml` 快照和同名 `.csv` 成绩单(`executed` 列区分已跑 / 未跑);没填 HOST 时会停,而不是编造失败。
### 客户端用户旅程
测试用例支持两种顶层类型:`test_type: integration` 和 `test_type: journey`。集成套件
至少包括 `api-contract`、`authorization`、`build-contract`;客户端套件仍是
`client-smoke`、`client-regression`、`safety`。没有显式范围时默认不选择 integration。
执行顺序固定为 `test-preflight -> integration -> journey`。preflight 状态为 blocked 或
environment-fail 时不创建客户端 session;integration 失败时依赖 journey 为 `not-run`。
服务不可达是 environment-fail,契约断言错误是 integration fail,正确账号登录失败仍是
journey fail。`tester run`/bundle 必须传入 caller 提供的 matrix:
```bash
<skill-dir>/tester run --preflight-config /abs/preflight.yaml \
--artifact android=/abs/app.apk --artifact ios=/abs/app.ipa
```
没有真实 deployer、Mac 或设备时,只能使用 test-preflight 的 command-contract mock fixture
验证协议,不能把 mock 结果写成真实运行成功。
客户端 E2E 用例使用 `client_suite: client-smoke | client-regression | safety`,并明确
`journey`、`fixture`、`session`、`cleanup`、`platform`。标准生命周期是:
```text
preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告
```
正确测试账号登录失败记 `fail`;设备、产物或服务不可达记 `environment-fail`。fixture 或
cleanup 缺失会在开始前阻断;前置失败后的剩余用例是 `not-run`,明确排除才是 `skipped`。
`safety` 必须显式选择和授权,不进入默认 smoke/full。它与普通 tester 用例共用原有
CSV/YAML 运行记录和汇总校验。
移动 preflight 还必须在实际 tester 节点验证相册 picker、照片权限与测试图片,以及中文
拼音的真实 composition 和候选词选择;缺失时所有 testcase 都保留 `not-run` +
`blocked_by`,不会静默 `skipped`。
失败回流使用 run YAML 顶层 `defects`/`test_issues` 与 `feedback`:PR 触发先更新 PR/关联
issue、通知 dispatcher 重新分配开发;需求/bug 触发直接回开发流程。可用
`scripts/record_issues.py` 生成同名 CSV 的结构化 issue 列。
### 汇总已有运行结果
对已经生成的同一运行 CSV/YAML,使用显式路径汇总:
```bash
python3 <skill-dir>/scripts/summarize_run.py \
--csv .pouch/tester/runs/<run-id>.csv \
--yaml .pouch/tester/runs/<run-id>.yaml
```
脚本只从 CSV 的 `result` 列计算 `pass`、`fail`、`environment-fail`、`skipped`、
`not-run` 五类状态,并要求五类计数和等于 CSV 数据行数。YAML 只用于校验
`results` 数量和 testcase id 集合;数量、集合、重复/缺失 id 或 CSV 缺失/未知状态
时返回非零并输出诊断。校验成功才表示这份汇总可作为运行报告;非零表示报告不一致,
不能据此判定产品失败。
+56 -2
View File
@@ -74,10 +74,10 @@ tester agent 配置。
- `local.yaml.example` 的 HOST 默认是 `127.0.0.1`;如果目标不是本机,必须在
`local.yaml` 或环境变量中显式覆盖。`local.yaml` 不存在且没有环境变量时仍不猜
HOST,停下并记 environment-fail。
- 默认风险上限是 catalog 里的 `default_max_risk`。超出上限的条目标 `skipped`,禁止执行。
- 默认执行 profile 是 `standard`:风险上限来自 catalog 的 `default_max_risk`,超出上限的条目标 `skipped`,禁止执行。若 catalog 声明 `execution_profile: isolated-full`,只允许用于隔离且可重置的测试环境:所有 active 自动化用例都进入执行矩阵,不因 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或 `AUTHORIZE_PASSWORD` skipped;原有 `risk` 值必须保留并进入报告。设备、产物、服务和清理契约由 test-preflight 检查;相册、照片权限/测试图片和中文拼音能力在本项目中默认具备,由 journey 用例真实验证。
- `risk` 只能是 `readonly` / `mutating` / `destructive`,默认只跑 `readonly`。`tag=smoke` 只能配 `risk=readonly`。规则见 [levels.md](references/levels.md)。
- `driver=browser` 必须开真实页面;`driver=mobile` 必须挂模拟器 session。二者都不得改成只打 API。
- `automation=assisted|manual` 需要人在场:无人值守时记 `not-run`,不要自己代跑人工步骤。见 [concepts.md](references/concepts.md)。
- `automation=assisted|manual` 需要人在场:无人值守时记 `not-run`,不要自己代跑人工步骤。isolated-full 取消的是风险/授权过滤,不会伪造人工操作。见 [concepts.md](references/concepts.md)。
- 浏览器、移动模拟器按 catalog `sessions` 初始化一次,同一次运行内复用。
- Agent、环境、项目目录和 Orca terminal 生命周期见 [execution.md](references/execution.md):可以从 project id/名称或 terminal id 自动解析,更新后输出配置列表;每轮都新建 terminal,结束后关闭本轮创建的 terminal,不复用或关闭已有 terminal。
- 环境准备见 [deployment.md](references/deployment.md):测试环境只调用 deployer 的本机 Compose 路径;正式环境只委派给本机 `~/code/app00` 的 fresh Agent terminal,tester 不直接 SSH 或执行生产 Compose。
@@ -145,3 +145,57 @@ python3 <skill-dir>/scripts/catalog.py show --root <project-root> --id <CASE_ID>
4. 只有矩阵所有行都可用,才创建本轮测试 terminal/session,按 `select` 结果执行并记录成绩。
测试环境由本机 deployer 触发,正式环境由 app00 Agent terminal 委派。随后按 agent 解析目标项目、目录和 Agent,更新配置并输出生效列表。移动端每个平台的产物位置必须由调用方给出,值为本地文件或 URL;缺了就停,不要猜 builder、CI release 或本机旧包。矩阵行记产物位置、身份和 `tester.<platform>` binding,并用 `<skill-dir>/tester run --artifact android=<path-or-url> --artifact ios=<path-or-url> --platform android,ios` 下载或取用、安装并准备本轮隔离 worktree。Android 在 `tester.android`、iOS 在 `tester.ios` 节点执行。未指定范围时等同「tag=smoke 且风险上限=默认且 automation=automated」。`select` 的 `run_together` 组按 [run.md](references/run.md) 整组执行。无论部署或测试成功、失败、超时还是中断,都要清理各自本轮创建的 terminal。
### 客户端用户旅程标准
### 双层测试与执行顺序
用例顶层类型是 `test_type: integration | journey`。旧目录没有该字段时保持兼容:已有
`client_suite`/`journey` 视为 journey,已有 `integration_suite` 视为 integration。集成
套件至少包括 `api-contract`、`authorization`、`build-contract`;旅程套件仍固定为
`client-smoke`、`client-regression`、`safety`。集成用例不需要客户端 session、platform
或 journey 字段,不能被客户端套件选中。
一次运行严格遵守:
```text
test-preflight (blocked/environment-fail -> 不创建 session)
-> integration
-> journey (integration fail -> not-run,不创建 session)
```
`test-preflight` 的报告是环境证据,不计入 testcase 总数。服务不可达记
`environment-fail`;服务已达但契约断言错误记 integration `fail`;正确测试账号登录失败
仍记 journey `fail`。tester 的 `run`/bundle 入口必须接收 caller 提供的 preflight matrix,
没有 matrix 不得绕过闸门或伪造成功。
声明 `client_suite`(迁移期也接受 `suite`)的用例采用固定的三套客户端集合:
`client-smoke`、`client-regression`、`safety`。前两者分别覆盖最短只读主路径和隔离测试
环境中的普通用户回归;`safety` 覆盖危险/权限边界场景,必须显式选择和授权,不能被
默认 smoke/full 扩入。选择某套客户端旅程可用:
```bash
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-smoke
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-regression --max-risk mutating
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite safety --max-risk destructive
```
客户端用例必须声明 `journey`、`fixture`、`session`、`cleanup` 和 `platform`。旅程包含
`goal`、`entry_state`、`actions`、`assertions`、`final_state`、`failure_policy`;fixture
至少声明测试账号;session 指向 catalog 中可复用的 browser/mobile session;cleanup 要有
步骤或命令(只读也要显式说明无需清理);platform 只能是 Android/iOS。具体字段见
`references/schema.md`。平台适配只能改操作层,不能把真实客户端操作换成 API 调用。
客户端运行器统一遵守 `preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告`。正确测试
账号登录失败是产品 `fail`;设备、产物、服务或 session 不可达才是 `environment-fail`。
fixture 或 cleanup 缺失会在 `catalog.py select` 的 preflight 阶段阻断,不产生虚假逐条结果。
设备、产物、服务或 session 能力缺失时 preflight 必须明确 blocked/environment-fail,并为所有选中的 testcase 生成
`not-run` 结果行;本项目默认具备相册、照片权限、测试图片或中文拼音 composition/candidate
能力,不在 preflight 中校验或静默变成 `skipped`,由 journey 真实验证并记录产品失败。独立用例继续执行,依赖失败的用例写 `not-run` 和 `blocked_by`;
修复后按 targeted rerun,再执行 full integration → journey。成绩仍写现有
`runs/<run-id>.yaml` 并用 `generate_run_csv.py` 生成同名 CSV;YAML 可带 `defects`/
`test_issues` 与 `feedback`,记录 testcase、product/test issue 类型、expected/observed、
artifact hash、environment、证据路径、复现和下一步,不添加新的 result 值。
PR 触发的失败回流必须先更新 PR/关联 issue,再通知 dispatcher 重新分配开发;需求/bug
触发的失败直接返回开发流程。tester 只生成结构化回流计划,不在本地偷偷修改远端。
+25 -1
View File
@@ -35,11 +35,19 @@ catalog(整个测试集)
| `session` | 可复用会话(浏览器 / 模拟器) | pytest fixture;Robot suite variable | 不是一条用例 |
| `gate` | 项目细分闸门名 → 环境变量(`motion` / `hardware` / `weld`) | 授权开关 | 不是 `risk`,不要拿它代替风险等级 |
| `authorize` | 风险档 → 环境变量。该档执行前必须为 `true` | pytest `skipif` 条件 | 不是认证(auth),是**授权** |
| `execution_profile` | 执行环境策略:`standard` 或隔离可重置的 `isolated-full` | pytest marker/fixture policy | 不是风险等级;isolated-full 不会抹掉 `risk` |
| `assume` | 额外跳过条件(gate / authorize 之外的环境表达式) | JUnit `assumeTrue`;pytest `skipif` | 不是授权开关的重复写法 |
| `status_probe` | 分类级前置条件:请求一个 URL,命中即视为前置不满足 | pytest session fixture;Robot Suite Setup | 不是产品失败,不是整轮硬闸 |
| `how` | 怎么测:`steps` / `run` / `url` / `timeout` | test steps | 不是判定标准 |
| `expected` | 怎么算过:`{kind, value}` 列表 | pytest `assert`;Robot `Should …` | 不是步骤 |
| `run_together` | 必须成组执行的用例(有序、可选 continue_on_fail) | 场景组(scenario) | 不是 suite,不是 tag |
| `client_suite` | 客户端用户旅程套件:`client-smoke` / `client-regression` / `safety` | mobile/browser E2E suite | 不是 `category`,也不是通用 tag;迁移期可读 `suite` 别名 |
| `test_type` | 顶层测试类型:`journey` 或 `integration` | E2E vs service/contract test | 不是 driver;旧目录可由既有字段推断 |
| `integration_suite` | 集成套件:`api-contract` / `authorization` / `build-contract` | API/contract test suite | 不是客户端 `client_suite`;迁移期可读 `suite` 别名 |
| `journey` | 用户目标、入口、操作、UI 断言、最终状态和失败继续策略 | Cucumber scenario / Robot task | 不是 API contract |
| `fixture` | 旅程开始前声明的测试账号与隔离数据 | pytest fixture | 不是运行中临时猜出的数据 |
| `cleanup` | 本轮数据的 teardown 契约;缺失会在 preflight 阻断 | pytest teardown / Robot teardown | 不是测试通过后的可选备注 |
| `platform` | 旅程适用的 `android`/`ios` 平台列表 | device matrix | 不是 tester Agent binding;binding 仍在命令参数/运行环境 |
### 结果词
@@ -48,7 +56,7 @@ catalog(整个测试集)
| `pass` | 通过 | passed |
| `fail` | 产品缺陷 | failed |
| `environment-fail` | 前置/环境导致无法执行,**永远不是产品失败** | pytest error;JUnit aborted |
| `skipped` | **主动**不跑:`retired` / `pending` / 超风险上限 / 未授权 / `assume` 不成立 | pytest skipped;JUnit disabled |
| `skipped` | **主动**不跑:standard 下的 `retired` / `pending` / 超风险上限 / 未授权 / `assume` 不成立 | pytest skipped;JUnit disabled |
| `not-run` | **被动**没轮到:失败即停后剩下的;分类被 `status_probe` 挡住整组没执行;`automation` 需人工而无人值守 | pytest not run;deselected |
## 2. 命名迁移对照(v1 → v2)
@@ -75,6 +83,8 @@ catalog(整个测试集)
- 三级**单调**:`readonly ⊂ mutating ⊂ destructive`,所以只需要一个上限 `default_max_risk` 和一个 `--max-risk`,不用组合矩阵。
- 与 `gate` 正交:`risk` 说「影响多大」,`gate` 说「碰哪个子系统」。点动 vs 焊接这种细分授权用 `gate`,不要再造 risk 名。
- `tag=smoke` 只能配 `risk=readonly`:冒烟测试不许留下变化。
- 隔离可重置环境可设置 `execution_profile: isolated-full` 跑所有 active 用例;安全前置由
test-preflight 的设备、服务与 cleanup/rollback 契约保证,原始 risk 仍必须报告;客户端能力由 journey 用例验证。
### 3.2 `automation`:为什么是 `automated / assisted / manual`
@@ -86,6 +96,7 @@ catalog(整个测试集)
- `assisted`:Agent 驱动,但至少一步需要人在场;该步在 `how.steps` 里以 `[人工]` 开头(可被 `validate` 机器判定)。
- `manual`:只能人工执行;禁止 `driver`、禁止 `how.run`,必须有 `how.steps`。
- 默认只跑 `automated`。无人值守时 `assisted` / `manual` 记 `not-run`;有人在场时 `assisted` 暂停等人确认,`manual` 由人工回报结果。
- isolated-full 只取消风险/授权过滤,不伪造 assisted/manual 的人工步骤。
### 3.3 结果为什么拆出 `not-run`
@@ -99,6 +110,19 @@ catalog(整个测试集)
- 因此它是**分类级前置条件**:`catalog.status_probe` 是全局默认,`categories[].status_probe` 可覆盖(缺省继承 / `false` 关闭 / mapping 整体替换)。对应 pytest 的 session fixture、Robot 的 Suite Setup。
- 命中或不可达只影响用它的分类;只有所有选中分类都被挡住,整轮结论才是 `environment-fail`。`select` 会按分类返回解析后的生效值,Agent 不要自己读 catalog 猜。
### 3.5 客户端套件和生命周期
客户端套件固定为:
- `client-smoke`:启动、正确账号登录和少量只读主路径,通常配 `tag=smoke`、`risk=readonly`。
- `client-regression`:完整的普通用户旅程,可包含隔离环境中的 `mutating` 操作;不等于所有 full 用例。
- `safety`:删除、权限边界、改密、停用或故障注入等危险旅程;必须显式选择和授权,不进入默认运行。
每条旅程按 `preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告` 执行。登录失败在
客户端和正确测试账号都已就绪时记 `fail`;设备、产物、服务或 session 不可达才记
`environment-fail`。fixture/cleanup 缺失是 contract block,不伪装成逐条结果。前置失败
后的剩余条目是 `not-run`,主动排除仍是 `skipped`。
## 4. 不采用的词
| 词 | 不用它的原因 |
+10
View File
@@ -25,6 +25,16 @@ bindings:
上执行测试。`tester.test` 只能用于明确的单平台或非移动测试;同一轮包含 Android
和 iOS 时必须分别解析两个 tester 绑定。
移动 bundle 入口必须把平台绑定显式传入:
```bash
--binding android=tester.android --binding ios=tester.ios
```
入口会校验 preflight matrix 的每个平台 binding、node 和 runner。远程 iOS 必须是
`contract` runner;多平台运行不会读取旧的单一 `local.yaml.agent` 来决定执行节点,
也不会在协调节点直接执行 Android/iOS 平台命令。
`locations` 和 `profiles` 的定义见共享 `agent-runtime`。tester adapter 只把它 materialize
到本轮兼容配置;每轮 terminal 仍然新建并在结束时关闭,不能因为 binding 是 persistent
就复用测试 terminal。
+5 -1
View File
@@ -36,7 +36,11 @@
| `destructive` | 难恢复或伤人/伤钱/伤环境 | 删库、真实扣费 | 清数据、内购 | 急停、断电、真焊 |
默认只跑 `readonly`。项目细分闸(点动 vs 焊接)用可选 `gate`,不要再发明 risk 名。
隔离且可重置的项目可在 catalog 设置 `execution_profile: isolated-full`,此时所有 active
用例按原始 risk 执行;`risk` 仍用于报告和 cleanup 设计,不能改写为 readonly。
`tag=smoke` 且 `risk!=readonly` 为非法。
不想跑某一档时,`authorize.<risk>` 指向的环境变量保持 false 即可——该档用例记 `skipped`,不是 `not-run`。
standard 下不想跑某一档时,`authorize.<risk>` 指向的环境变量保持 false 即可——该档用例记
`skipped`,不是 `not-run`。isolated-full 不使用这些授权开关,改由 preflight 能力探针和
cleanup/rollback 契约阻断不安全环境。
+70 -13
View File
@@ -9,6 +9,11 @@ Agent、执行目标和临时 terminal 的完整协议见 [execution.md](executi
## 0. 建立测试矩阵并完成全量 preflight
`test-preflight` 是 integration 和 journey 的共同硬闸。tester 必须先调用其 `verify`
或 `prepare`,保存独立的 preflight YAML/CSV;状态为 `blocked` 或 `environment-fail`
时不创建 browser/mobile session,所有尚未执行的 testcase 记 `not-run`。preflight 证据
不算 testcase,也不进入 testcase CSV/YAML 的总数。
测试不是“找到一个能跑的设备就开始”。每轮先建立一份测试矩阵;移动端每个目标产物
必须单独占一行:
@@ -67,7 +72,11 @@ iOS | … | … | page.writes | tester.ios | remote-ios
## 1. 选范围
未指定范围 → `--tag smoke`,风险上限 = `default_max_risk`,`automation` 默认只跑 `automated`。
未指定范围 → `--tag smoke`,执行 profile = catalog 的 `execution_profile`(默认 `standard`),
`standard` 的风险上限 = `default_max_risk`,`automation` 默认只跑 `automated`。
`execution_profile=isolated-full` 只用于隔离可重置环境:不按风险上限或 `AUTHORIZE_*`/gate
授权变量过滤 active 用例,保留每条用例原始 `risk`;安全门禁由 test-preflight 能力探针和
cleanup/rollback 契约承担。
用户说分类、id、tag、风险上限时,原样传给 select:
@@ -75,33 +84,65 @@ iOS | … | … | page.writes | tester.ios | remote-ios
python3 <skill-dir>/scripts/catalog.py select --root <root> \
[--category CAT] [--id ID]... [--tag smoke|full|slow] \
[--max-risk <risk>] [--driver script|browser|mobile] \
[--execution-profile standard|isolated-full] \
[--automation automated|assisted|manual|any] \
[--test-type integration|journey] [--suite <client-or-integration-suite>] \
[--platform android|ios]... \
[--artifact android=<path-or-url>] [--artifact ios=<path-or-url>] \
[--binding android=tester.android] [--binding ios=tester.ios]
```
客户端旅程还可按标准套件选择:
```bash
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-smoke
```
`client-smoke`、`client-regression`、`safety` 是 `client_suite` 的固定值,不替换既有
`tag=smoke|full|slow`。`safety` 不参与未指定范围的默认选择。
集成测试显式选择:
```bash
python3 <skill-dir>/scripts/catalog.py select --root <root> --test-type integration
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite api-contract
```
未指定 `--test-type`/`--suite` 时,默认选择不会包含 `test_type=integration`。
选中任一 `driver=mobile` 用例时,`--platform`、每个平台的 `--artifact` 和对应
`tester` binding 是硬性参数;`catalog.py select` 会检查产物文件/下载地址和
`.pouch/agent.yaml` 中的 binding。缺任何一项直接返回非零并打印 `preflight: blocked`,
不能拿到 cases 后绕过闸门开始测试。默认 binding 是 `tester.<platform>`,所以 iOS
会强制检查 `tester.ios`,不会回退到 `builder.ios` 或通用 `tester.test`。
`--automation` 默认 `automated`(需人工的条目会被跳过,并在 JSON `skipped` 里说明);用户明确要求现场配合时用 `assisted`,`any` 表示不按人工参与度筛。
`--automation` 默认 `automated`(需人工的条目会被记为 `not-run`,不会伪造人工步骤);用户明确要求现场配合时用 `assisted`,`any` 表示不按人工参与度筛。
单根 `select` 的 JSON 字段:`max_risk` / `automation` / `count` / `vars`(已展开的变量,用来替换 `how` 里的 `${…}`)/ `artifacts` / `preflight` / `skipped` / `run_together` / `status_probe`(按分类,`url` 已展开)/ `cases`。从父仓库 fan-out 时返回 `mode: fan-out` 和每个根的独立结果,不能把多个根的 cases 拼成一个列表。
`select` 已排除 `retired` 和风险高于上限的条目(后者若用户点了具体 id,会在 JSON 的 `skipped` 里说明,不执行)。
`standard` 的 `select` 已排除 `retired` 和风险高于上限的条目(后者若用户点了具体 id,会在 JSON 的 `skipped` 里说明);`isolated-full` 保留所有 active 条目,不因风险或授权变量 skipped。
若 catalog 有 `run_together`,命中组内任一条就把整组按 `ids` 顺序扩进列表(JSON 带 `run_together`)。这不是擅自扩大范围。
无命中:停下说明筛选条件,不要扩大范围。
## 2. 运行前闸
## 2. 双层执行
preflight 通过后,先按 catalog 顺序执行 `integration`,再按客户端套件顺序执行
`journey`。集成服务不可达是 `environment-fail`;服务已达但 API/权限/数据契约断言错误
是 `fail`。任何 integration `fail` 都阻断依赖的 journey,并把它们记为 `not-run`;此时
不得创建客户端 session。正确测试账号登录失败发生在客户端已启动后,仍是产品 `fail`。
运行器可复用 `scripts/two_tier.py` 的 `run_integration_first`,它只接收外部 driver
回调,不实现 API、浏览器或移动执行器;mock fixture 只能验证调用顺序和失败传播,不能
冒充真实部署、Mac 或设备成功。
## 3. 运行前闸
- `HOST` 未设(环境变量 / `local.yaml` 都没有):全部停,environment-fail。
- 命中条的 `risk` 在 catalog `authorize` 里有对应变量:变量不是 `true` 则该条 `skipped`。有 `gate` 时还要 `catalog.gates` 对应变量为 `true`;`assume` 不成立同样 `skipped`。即使用户点了 id 也不执行。
- 授权/假设不成立是**主动不跑**(`skipped`),不是 `not-run`:这是我们决定不跑,不是没轮到。
- `standard`:命中条的 `risk` 在 catalog `authorize` 里有对应变量时,变量不是 `true` 则该条 `skipped`;有 `gate` 时还要对应变量为 `true`,`assume` 不成立同样 `skipped`。即使用户点了 id 也不执行。
- `isolated-full`:不读取 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM`、`AUTHORIZE_PASSWORD` 等人工开关;相册、照片权限/测试图片和中文拼音能力默认具备,由 journey 用例真实验证,失败记为产品失败。
- standard 中授权/假设不成立是**主动不跑**(`skipped`);isolated-full 不因客户端能力授权变量停止,设备、产物、服务或清理前置失败时才为全部 testcase 写 `not-run` + `blocked_by: [test-preflight]`。
- `status_probe` 是**分类级前置条件**(业界对应 pytest 的 session fixture / Robot 的 Suite Setup)。用 `select` 返回的 `status_probe` 解析生效值,不要自己读 catalog 猜。返回的 `probe.url` 已用 `vars` 展开,直接请求即可;`select` 同批返回 `vars`:
| `categories[].status_probe` | 生效探测 |
@@ -114,7 +155,7 @@ python3 <skill-dir>/scripts/catalog.py select --root <root> \
- 探测不可达或命中 `stop_if`:在 preflight 阶段记录对应分类/平台的不可用项并停止整轮;
先返回用户处理,不创建后续测试 session。只有所有本轮目标环境通过,才进入用例执行。
## 2.1 移动端平台环境闸门与用户引导
## 3.1 移动端平台环境闸门与用户引导
`driver=mobile` 不是一个可以互换的平台。运行前必须根据 catalog session、用例变量或用户指定,明确本轮是 Android 还是 iOS;无法判断时先停下询问平台,不得猜测。两类平台分别执行以下闸门:
@@ -175,7 +216,7 @@ Android+iOS 默认是全量 preflight 闸门:任一项不满足,另一平台
环境引导只负责让用户补齐前置条件;除 catalog 已明确授权的 session 初始化外,不自动安装 SDK、配对真机、修改系统安全设置或上传产物。
## 3. Session
## 4. Session
看命中条用到的 `session` 键:
@@ -184,9 +225,9 @@ Android+iOS 默认是全量 preflight 闸门:任一项不满足,另一平台
- `reuse: run`:整轮结束再关。`persist`:不关。
- 中途 session 挂了:后续同 driver 标 environment-fail,不要标产品失败。
## 4. 执行
## 5. 执行
按 select 返回的顺序,一条一条来。select 按 `catalog.yaml` 的 `categories` 顺序,同一分类内按该 yaml 列表顺序(不是文件名字母序)。默认失败即停(产品 `fail` 或探测到的不安全状态)。用户说继续 / keep-going 才往后跑。
按 select 返回的顺序,一条一条来。select 按 `catalog.yaml` 的 `categories` 顺序,同一分类内按该 yaml 列表顺序(不是文件名字母序)。同一 phase 的独立用例继续执行;只有显式 `depends_on` 的失败依赖才写 `not-run` + `blocked_by`。用户说 targeted rerun 时只重跑受影响用例,修复后再 full integration → journey。
例外:当前条和下一条同属 `run_together` 且该组 `continue_on_fail: true` 时,前一条产品 `fail` 仍跑后一条。`environment-fail` 仍整轮停。
@@ -216,9 +257,9 @@ python3 <skill-dir>/scripts/catalog.py run-script --root <root> --id <id>
挂到已有 session,执行 `how.steps`,对照 `expected`。必须留下可见文本、URL 或交互证据。
## 5. 记录与报告
## 6. 记录与报告
每条:`id, result, evidence, notes`。`result` 只能是 `pass` / `fail` / `environment-fail` / `skipped` / `not-run`。`skipped` 是主动不跑(retired / pending / 超风险上限 / 未授权 / `assume` 不成立),`not-run` 是没轮到(失败即停后剩下的 / 被前置整组挡住的 / 需人工而无人值守的)。
每条:`id, result, evidence, notes`,依赖阻断时另有 `blocked_by`。`result` 只能是 `pass` / `fail` / `environment-fail` / `skipped` / `not-run`。`skipped` 是 standard 中主动不跑(retired / pending / 超风险上限 / 未授权 / `assume` 不成立);`not-run` 是没轮到或 preflight/依赖阻断。失败用例可在 YAML 顶层 `defects`(兼容 `test_issues`)记录 testcase、`issue_type=product|test`、expected/observed、artifact hash、environment、evidence path、reproduction、next step。
写入 `.pouch/tester/runs/<UTC时间>-<范围>-<随机后缀>.yaml`,可用
`generate_run_csv.new_run_id(<范围>)` 生成同名 `runId`。记录包含唯一 `runId`、筛选条件、HOST 是否已设(可以写 HOST 值)、session 是否复用、每个分类的 `status_probe` 生效值与观测结果(`source` / `disabled` 原因 / `hit` / `observed`)、逐条结果。记录文件只创建一次,不覆盖已有文件。立刻用同一快照生成 CSV,不要手改:
@@ -227,7 +268,23 @@ python3 <skill-dir>/scripts/catalog.py run-script --root <root> --id <id>
python3 <skill-dir>/scripts/generate_run_csv.py --root <root> --run <yaml>
```
这会写出与快照同名的 `.pouch/tester/runs/<run-id>.csv`(`executed=yes|no`),不会修改 `cases.csv`。对话里两个路径都报。
这会写出与快照同名的 `.pouch/tester/runs/<run-id>.csv`(`executed=yes|no`),不会修改
`cases.csv`。对话里两个路径都报。
需要复核已有成绩单时,显式同时传入同一次运行的 CSV 和 YAML:
```bash
python3 <skill-dir>/scripts/summarize_run.py \
--csv .pouch/tester/runs/<run-id>.csv \
--yaml .pouch/tester/runs/<run-id>.yaml
```
汇总状态只读取 CSV `result` 列,必须输出五类 `pass / fail / environment-fail /
skipped / not-run`,且计数和等于 CSV 数据行数。汇总前会检查 CSV 数据行数、CSV 唯一
testcase id 数、YAML `results` 数量和 CSV/YAML testcase id 集合;重复或缺失 id、缺失或
未知 CSV 状态、任一数量或集合不一致都会非零退出并给出诊断。YAML 的 `counts` 或其它
历史汇总字段不会参与状态计数。退出码 0 表示汇总校验成功,非零表示报告不一致或输入
无效,不等同于产品测试失败。
对话报告:
+90
View File
@@ -14,6 +14,10 @@ runs/<run-id>.yaml 每次运行快照(执行源);run-id 必须唯一
runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对应 yaml 生成
```
两类用例共用同一套 cases/runs 和五态报告,不另造成绩单。新目录显式声明顶层
`test_type: journey | integration`;旧目录没有该字段时按已有 `client_suite`/`journey`
或 `integration_suite` 推断。`category` 仍是文件/执行顺序容器,不被套件字段替代。
## catalog.yaml
| 字段 | 含义 |
@@ -21,6 +25,7 @@ runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对
| `categories` | `{id, title, status_probe?}` 列表,id 与 `cases/<id小写>.yaml` 对应;列表顺序即 `select` / 执行顺序。`status_probe` 见下 |
| `risks` | 从低到高,只能是 `readonly` → `mutating` → `destructive` 的一段,且以 `readonly` 开头。含义见 [levels.md](levels.md) |
| `default_max_risk` | 未指定时的运行上限,必须是 `risks` 里的值 |
| `execution_profile` | `standard`(默认)或 `isolated-full`。后者只用于隔离可重置环境,保留 risk 语义但不按 risk/authorize/gate 过滤 active 用例;设备、服务和 cleanup/rollback 契约仍由 preflight 检查,客户端能力由 journey 真实验证 |
| `vars` | 可替换变量,如 `API_BASE: http://${HOST}:8095` |
| `artifacts` | 可选的移动产物身份契约。按 `android` / `ios` 声明 `required` 和 `package` / `bundle_id`;每轮文件路径或下载地址由调用方提供,不提交临时产物路径。 |
| `authorize` | 风险名 → 环境变量名。该档执行前变量必须为 `true`(`readonly` 不需要) |
@@ -29,6 +34,12 @@ runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对
| `id_pattern` | 用例 id 正则 |
| `run_together` | 可选。成组执行:`[{ids: [TC-…, TC-…], continue_on_fail: true}]`。点到组内任一条,`select` 会按 `ids` 顺序把整组扩进命中列表。`continue_on_fail: true` 时组内前一条产品 fail 仍跑后一条(返回/收尾)。一个 id 只能属于一组。 |
| `status_probe` | 可选。顶层默认值,一个**分类级前置条件**:`{url, stop_if: {字段: [命中值]}}`。运行前请求 `url`,响应命中任一 `stop_if` 条件即视为前置不满足。分类项可用同名键覆盖:缺省继承顶层、`false` 关闭、mapping 整体替换。命中或不可达只影响用它的分类,不是产品失败。业界对应 pytest 的 session fixture / Robot 的 Suite Setup。`select` 会按分类返回解析后的生效值,`url` 已用 `vars` 展开 |
| `client_suites` | 可选的标准套件声明;出现时必须完整包含 `client-smoke`、`client-regression`、`safety`。缺省运行仍按 `tag` 兼容旧目录 |
| `integration_suites` | 可选的集成套件声明;标准值为 `api-contract`、`authorization`、`build-contract` |
`isolated-full` 不得把风险字段改成 `readonly`,也不得把 `AUTHORIZE_ALBUM`、
`AUTHORIZE_PASSWORD`、`IME_PINYIN` 等变量当成门禁。移动用例必须通过 test-preflight
的设备、产物、服务和网络检查;相册、权限和输入法能力默认具备,交由 journey 用例验证。
`local.yaml` 与环境变量覆盖 `HOST` 和授权开关。模板中的 `HOST` 默认是
`127.0.0.1`;远程目标必须显式覆盖。查找顺序:环境变量 > `local.yaml` >
@@ -70,6 +81,7 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
| 字段 | 必填 | 含义 |
|------|------|------|
| `id` | 是 | 匹配 `id_pattern` |
| `test_type` | 新用例是 | `journey` 或 `integration`;缺省兼容旧目录并按字段推断 |
| `category` | 是 | 必须等于文件声明的 category |
| `title` | 是 | 短名 |
| `what` | 是 | 一句可观察的「测什么」 |
@@ -84,10 +96,67 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
| `expected` | 是 | 怎么过,至少一条 `{kind, value}` |
| `maps_to` | 否 | 对人文档的功能项 id |
| `assume` | 否 | 额外跳过条件(环境变量表达式)。不要重复 `risk` / `gate` 已覆盖的授权变量,否则 `validate` 报错 |
| `depends_on` | 否 | 依赖的 testcase id 列表;依赖 fail/environment-fail/not-run 时本条写 `not-run` 和 `blocked_by`,独立用例继续 |
| `setup` | 否 | 人话前置 |
| `teardown` | 否 | 跑完回退 |
| `notes` | 否 | 备注 |
### 集成测试用例契约
`test_type: integration` 的用例不创建 browser/mobile session,也不能声明 `platform` 或
客户端 `journey`。必须有 `integration_suite`(迁移期可用 `suite`)且值为
`api-contract`、`authorization` 或 `build-contract`;`fixture` 必须是非空 mapping;
`request`、`input`、`call`、`integration` 之一或旧式 `how` 用于描述调用;`expected` 是
协议/数据断言。`mutating`/`destructive` 集成用例必须提供 `cleanup.steps` 或
`cleanup.run`(只读探针可用 `required: false` + `reason`)。
选择 `--suite api-contract` 或 `--test-type integration` 才会进入集成阶段;无显式类型
时的默认选择不会包含 integration。
### 客户端旅程用例契约
声明 `client_suite` 后,以下字段都必须存在;`catalog.py validate` 和 `select` 使用同一
套检查。契约缺失是 `preflight: blocked`,不生成逐条产品失败。
| 字段 | 必填 | 约束 |
|------|------|------|
| `client_suite` | 是 | `client-smoke`、`client-regression` 或 `safety`;安全套件必须显式选择,不属于默认 smoke/full |
| `journey.goal` | 是 | 用户目标,不写接口目标 |
| `journey.entry_state` | 是 | 启动/登录前的可观察入口状态 |
| `journey.actions` | 是 | 非空用户操作列表;真实 App/browser 驱动从这里开始 |
| `journey.assertions` | 是 | 非空可观察 UI 断言列表;最终业务状态可另由 `expected` 的 api-field/api-status 佐证 |
| `journey.final_state` | 是 | 旅程完成后的业务状态 |
| `journey.failure_policy` | 是 | `stop` 或 `continue`;默认不隐式吞掉前置失败 |
| `fixture` | 是 | mapping,至少有 `account`;账号和数据由 app catalog 提供,skill 不写具体账号 |
| `session` | 是 | 复用的 browser/mobile session 名,必须在 catalog.sessions 声明 |
| `cleanup` | 是 | mapping,必须有非空 `steps`/`run`;只读旅程也要用 `required: false` + `reason` 显式说明无需清理 |
| `platform` | 是 | `android`/`ios` 的字符串或列表,不能重复;平台差异只能进入执行适配层 |
一个最小形状如下(值是占位符,不是 app 规则):
```yaml
client_suite: client-smoke
platform: [android, ios]
journey:
goal: "用户完成一个可观察的客户端目标"
entry_state: logged_out
actions: ["启动客户端", "输入测试账号并登录", "完成目标操作"]
assertions: ["看到目标页面", "看到成功提示"]
final_state: "目标状态已保存"
failure_policy: stop
fixture:
account: test-role
data: []
session: mobile
cleanup:
steps: ["删除本轮 run-id 创建的数据"]
```
账号登录属于 session 初始化:正确测试账号登录失败是产品 `fail`;设备、产物或服务
不可达才是 `environment-fail`。fixture/cleanup 缺失必须在启动、登录和任何旅程操作前
阻断。旅程失败后,尚未轮到的用例记 `not-run`;风险、生命周期或明确范围排除记
`skipped`。清理失败单独写入运行快照的 cleanup 证据,不改造为一个新的 result 值。
## how
按 `driver` 分派;`automation=manual` 时没有 `driver`,只有 `how.steps`(人工步骤)。
@@ -107,3 +176,24 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
## 判定
`pass` / `fail` / `environment-fail` / `skipped` / `not-run`
## 运行失败回流
run YAML 的 `results` 必须保留每条 testcase 行。失败可在顶层 `defects`(兼容别名
`test_issues`)追加:
```yaml
defects:
- testcase: TC-CLIENT-001
issue_type: product # product | test
expected: "候选词可选择"
observed: "composition 成功但候选列表为空"
artifact_hash: sha256:...
environment: "Android emulator API 35 / tester.android"
evidence_path: .pouch/tester/evidence/TC-CLIENT-001.png
reproduction: "打开相册后输入 nihao"
next_step: "开发修复候选词渲染后 targeted rerun"
```
PR 触发的 run 可带 `feedback` 计划:先更新 PR/关联 issue,再通知 dispatcher 重新分配开发;
需求/bug 触发则 route 直接为 development。tester 只记录计划,不隐式调用远端写操作。
+7 -3
View File
@@ -7,11 +7,15 @@ CI release、builder 绑定或本机旧文件猜测产物。
<skill-dir>/tester run \
--artifact android=<path-or-url> \
--artifact ios=<path-or-url> \
--platform android,ios
--platform android,ios \
--binding android=tester.android \
--binding ios=tester.ios
```
URL 就下载,本地文件就直接用。每个平台都在自己的 `tester.<platform>` 节点校验身份并安装,
随后由现有 tester catalog 驱动真实移动 session。默认先完成 Android+iOS 双平台
URL 就下载,本地文件就直接用。每个平台都通过自己的 `tester.<platform>` binding
校验、探测和安装;`agent-runtime` 负责把 Android 请求送到 Android 节点、把 iOS
请求送到 Mac 节点。旧的单一 `local.yaml.agent` 只能作为兼容配置,不能作为双平台
运行目标。随后由现有 tester catalog 驱动真实移动 session。默认先完成 Android+iOS 双平台
preflight;任一平台无法准备时整轮停止。只有显式加 `--partial` 才允许运行可用平台。
每轮另外创建隔离 Git worktree,不切用户当前分支。`--commit` 缺省为当前 HEAD。
+149 -10
View File
@@ -18,6 +18,16 @@ from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from journey import (
CLIENT_SUITES,
INTEGRATION_SUITES,
TEST_TYPES,
contract_errors,
preflight as journey_preflight,
suite as case_suite,
test_type,
)
try:
import yaml
except ImportError: # pragma: no cover
@@ -32,6 +42,7 @@ LIFECYCLE_SKIP_REASON = {
}
TAGS = ("smoke", "full", "slow")
CANONICAL_RISKS = ("readonly", "mutating", "destructive")
EXECUTION_PROFILES = ("standard", "isolated-full")
AUTOMATIONS = ("automated", "assisted", "manual")
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
EXPECTED_KINDS = (
@@ -341,6 +352,21 @@ def risk_index(meta: dict[str, Any], name: str) -> int:
return risks.index(name)
def resolve_execution_profile(meta: dict[str, Any], requested: str | None = None) -> str:
"""Resolve the run profile without changing the meaning of ``risk``.
``isolated-full`` is an execution policy for a resettable test environment:
it disables risk/authorization filtering, but leaves the catalog risk value
on every case so reports still describe the side effect that was exercised.
"""
profile = requested or meta.get("execution_profile") or "standard"
if profile not in EXECUTION_PROFILES:
raise SystemExit(
f"未知 execution_profile: {profile!r}(可选: {', '.join(EXECUTION_PROFILES)})"
)
return str(profile)
def host_set(vars_map: dict[str, str]) -> bool:
host = (vars_map.get("HOST") or "").strip()
return bool(host) and "${HOST}" not in host
@@ -368,7 +394,7 @@ def cmd_list(root: Path) -> int:
pending = [c for c in cases if c.get("lifecycle") == "pending"]
print(f"## 测试目录 active {len(active)} / pending {len(pending)} / retired {len(retired)}")
print()
headers = ["分类", "标题", "条数", "smoke"] + risks + list(DRIVERS)
headers = ["分类", "标题", "条数", "journey", "integration", "smoke"] + risks + list(DRIVERS)
print("| " + " | ".join(headers) + " |")
print("| " + " | ".join("---" for _ in headers) + " |")
@@ -386,7 +412,14 @@ def cmd_list(root: Path) -> int:
title = (cats_meta.get(cat) or {}).get("title") or (
group[0].get("_file_title") if group else ""
)
row = [cat, str(title), str(len(group)), str(sum(1 for c in group if c.get("tag") == "smoke"))]
row = [
cat,
str(title),
str(len(group)),
str(sum(1 for c in group if test_type(c) == "journey")),
str(sum(1 for c in group if test_type(c) == "integration")),
str(sum(1 for c in group if c.get("tag") == "smoke")),
]
for risk in risks:
row.append(str(sum(1 for c in group if c.get("risk") == risk)))
for driver in DRIVERS:
@@ -553,6 +586,7 @@ def expand_run_together(
skipped: list[dict[str, str]],
max_i: int,
automation_filter: str = "automated",
bypass_risk: bool = False,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
groups = run_together_groups(meta)
if not groups:
@@ -590,7 +624,7 @@ def expand_run_together(
except SystemExit:
skipped.append({"id": cid, "reason": f"未知风险 {risk}"})
continue
if r_i > max_i:
if not bypass_risk and r_i > max_i:
skipped.append({"id": cid, "reason": f"risk={risk} 超过上限"})
continue
members.append(public_case(case))
@@ -753,9 +787,11 @@ def mobile_preflight(
def cmd_select(args: argparse.Namespace, root: Path) -> int:
meta = load_catalog_meta(root)
cases = load_all_cases(root)
execution_profile = resolve_execution_profile(meta, args.execution_profile)
bypass_risk = execution_profile == "isolated-full"
max_risk = args.max_risk or meta.get("default_max_risk") or "readonly"
max_i = risk_index(meta, max_risk)
automation_filter = args.automation or "automated"
automation_filter = args.automation or ("any" if bypass_risk else "automated")
wanted = set(parse_ids(args.ids or []))
selected: list[dict[str, Any]] = []
skipped: list[dict[str, str]] = []
@@ -781,6 +817,20 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
continue
if args.tag and case.get("tag") != args.tag:
continue
if args.test_type and test_type(case) != args.test_type:
continue
if args.suite and case_suite(case) != args.suite:
continue
# A mixed catalog keeps legacy cases runnable. Standard keeps the
# historical client-only implicit default; isolated-full means
# exactly what it says and includes every active test type.
if (
not bypass_risk
and not args.test_type
and not args.suite
and test_type(case) == "integration"
):
continue
if args.driver and case.get("driver") != args.driver:
continue
automation = str(case.get("automation") or "automated")
@@ -796,7 +846,7 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
except SystemExit:
skipped.append({"id": cid, "reason": f"未知风险 {risk}"})
continue
if r_i > max_i:
if not bypass_risk and r_i > max_i:
skipped.append({"id": cid, "reason": f"risk={risk} 超过上限 {max_risk}"})
continue
selected.append(public_case(case))
@@ -808,7 +858,22 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
)
return 2
selected, together = expand_run_together(meta, cases, selected, skipped, max_i, automation_filter)
selected, together = expand_run_together(
meta,
cases,
selected,
skipped,
max_i,
automation_filter,
bypass_risk=bypass_risk,
)
# The historical key is kept in the JSON, but the gate now validates both
# test types before any caller can start a phase.
journey_gate = journey_preflight(selected, sessions=meta.get("sessions") or {})
if journey_gate["status"] == "blocked":
print(json.dumps({"preflight": journey_gate, "journey_preflight": journey_gate}, ensure_ascii=False, indent=2))
return 2
preflight = mobile_preflight(
root,
@@ -835,11 +900,18 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
json.dumps(
{
"max_risk": max_risk,
"execution_profile": execution_profile,
"risk_filter": "disabled" if bypass_risk else "enforced",
"automation": automation_filter,
"count": len(selected),
"vars": vars_map,
"artifacts": meta.get("artifacts") or {},
"preflight": preflight,
"journey_preflight": journey_gate,
"test_types": {
"journey": sum(1 for case in selected if test_type(case) == "journey"),
"integration": sum(1 for case in selected if test_type(case) == "integration"),
},
"skipped": skipped,
"run_together": together,
"status_probe": status_probes,
@@ -910,6 +982,12 @@ def validate(root: Path) -> list[str]:
default_max = meta.get("default_max_risk")
if default_max not in risks:
errors.append(f"default_max_risk={default_max!r} 不在 risks 中")
profile = meta.get("execution_profile", "standard")
if profile not in EXECUTION_PROFILES:
errors.append(
f"execution_profile={profile!r} 非法,只能是 {', '.join(EXECUTION_PROFILES)}"
)
isolated_full = profile == "isolated-full"
gates = meta.get("gates") or {}
if gates and not isinstance(gates, dict):
errors.append("catalog.gates 必须是 mapping")
@@ -922,6 +1000,28 @@ def validate(root: Path) -> list[str]:
if not isinstance(sessions, dict):
errors.append("catalog.sessions 必须是 mapping")
sessions = {}
client_suites = meta.get("client_suites")
if client_suites is not None:
if not isinstance(client_suites, dict):
errors.append("catalog.client_suites 必须是 mapping")
else:
unknown = sorted(set(client_suites) - set(CLIENT_SUITES))
if unknown:
errors.append("catalog.client_suites 含非法套件: " + ", ".join(unknown))
missing_suites = sorted(set(CLIENT_SUITES) - set(client_suites))
if missing_suites:
errors.append("catalog.client_suites 缺少标准套件: " + ", ".join(missing_suites))
integration_suites = meta.get("integration_suites")
if integration_suites is not None:
if not isinstance(integration_suites, dict):
errors.append("catalog.integration_suites 必须是 mapping")
else:
unknown = sorted(set(integration_suites) - set(INTEGRATION_SUITES))
if unknown:
errors.append("catalog.integration_suites 含非法套件: " + ", ".join(unknown))
missing_suites = sorted(set(INTEGRATION_SUITES) - set(integration_suites))
if missing_suites:
errors.append("catalog.integration_suites 缺少标准套件: " + ", ".join(missing_suites))
artifacts = meta.get("artifacts")
if artifacts is not None:
if not isinstance(artifacts, dict):
@@ -998,6 +1098,7 @@ def validate(root: Path) -> list[str]:
errors.append(f"{where}: category={cat} 与文件 category={file_cat} 不一致")
if cat and cat_ids and cat not in cat_ids:
errors.append(f"{where}: category={cat} 不在 catalog.categories")
errors.extend(contract_errors(case, sessions=sessions))
if case.get("lifecycle", "active") not in LIFECYCLES:
errors.append(f"{where}: 非法 lifecycle")
if case.get("tag") not in TAGS:
@@ -1013,8 +1114,17 @@ def validate(root: Path) -> list[str]:
if case.get("risk") in (None, "readonly"):
errors.append(f"{where}: risk=readonly 不要填 gate")
risk = case.get("risk")
if risk and risk != "readonly" and risk not in authorize and not gate:
if risk and risk != "readonly" and risk not in authorize and not gate and not isolated_full:
errors.append(f"{where}: {risk} 需要 authorize.{risk} 或 gate")
if isolated_full and case.get("lifecycle", "active") == "active" and risk != "readonly":
cleanup = case.get("cleanup")
cleanup_ok = isinstance(cleanup, dict) and (
bool(cleanup.get("steps"))
or bool(str(cleanup.get("run") or "").strip())
or (cleanup.get("required") is False and bool(str(cleanup.get("reason") or "").strip()))
)
if not cleanup_ok:
errors.append(f"{where}: isolated-full 的写入型用例必须声明 cleanup/rollback steps/run")
automation = str(case.get("automation") or "automated")
if automation not in AUTOMATIONS:
errors.append(f"{where}: 非法 automation={case.get('automation')}")
@@ -1146,7 +1256,19 @@ def cmd_next_id(root: Path, category: str) -> int:
return 0
def authorized(vars_map: dict[str, str], meta: dict[str, Any], case: dict[str, Any]) -> bool:
def authorized(
vars_map: dict[str, str],
meta: dict[str, Any],
case: dict[str, Any],
*,
execution_profile: str | None = None,
) -> bool:
profile = resolve_execution_profile(meta, execution_profile)
if profile == "isolated-full":
# AUTHORIZE_* and gate variables are intentionally not run-time
# switches for an isolated resettable environment. Capability and
# cleanup preflight are the safety boundary for this profile.
return True
risk = str(case.get("risk") or "readonly")
risks = list(meta.get("risks") or ["readonly"])
lowest = risks[0] if risks else "readonly"
@@ -1166,6 +1288,7 @@ def authorized(vars_map: dict[str, str], meta: dict[str, Any], case: dict[str, A
def cmd_run_script(root: Path, case_id: str) -> int:
meta = load_catalog_meta(root)
execution_profile = resolve_execution_profile(meta)
vars_map = resolve_vars(root, meta)
case = next((c for c in load_all_cases(root) if c.get("id") == case_id), None)
if not case:
@@ -1203,10 +1326,10 @@ def cmd_run_script(root: Path, case_id: str) -> int:
)
return 2
risk = str(case.get("risk") or "readonly")
if not authorized(vars_map, meta, case):
if not authorized(vars_map, meta, case, execution_profile=execution_profile):
print(
json.dumps(
{"id": public_id, "result": "skipped", "evidence": f"风险 {risk} 未授权"},
{"id": public_id, "result": "skipped", "evidence": f"风险 {risk} 未授权", "execution_profile": execution_profile},
ensure_ascii=False,
)
)
@@ -1255,6 +1378,7 @@ def cmd_run_script(root: Path, case_id: str) -> int:
{
"id": public_id,
"result": "executed",
"execution_profile": execution_profile,
"exit": proc.returncode,
"evidence": evidence,
},
@@ -1275,7 +1399,22 @@ def build_parser() -> argparse.ArgumentParser:
p_sel.add_argument("--category")
p_sel.add_argument("--id", action="append", dest="ids", default=[])
p_sel.add_argument("--tag", choices=TAGS)
p_sel.add_argument(
"--suite", "--client-suite", dest="suite",
choices=[*CLIENT_SUITES, *INTEGRATION_SUITES],
help="客户端或集成测试套件;不传时默认不选 integration",
)
p_sel.add_argument(
"--test-type", choices=TEST_TYPES, default=None,
help="按顶层 test_type 筛选;integration 必须显式选择",
)
p_sel.add_argument("--max-risk")
p_sel.add_argument(
"--execution-profile",
choices=EXECUTION_PROFILES,
default=None,
help="执行策略;isolated-full 在隔离可重置环境中不按 risk/authorize 筛选",
)
p_sel.add_argument("--driver", choices=DRIVERS)
p_sel.add_argument(
"--automation",
+150
View File
@@ -0,0 +1,150 @@
#!/usr/bin/env python3
"""Build structured tester issues and the development feedback route.
This module only writes the run contract. Updating a remote PR/issue or
notifying a dispatcher remains an explicit caller-side integration step; no
network mutation is hidden in the tester.
"""
from __future__ import annotations
from copy import deepcopy
from datetime import datetime, timezone
from typing import Any, Iterable, Mapping
ISSUE_TYPES = ("product", "test")
TRIGGERS = ("pr", "requirement", "bug")
ISSUE_RESULTS = frozenset({"fail", "environment-fail"})
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def issue_type_for(result: str) -> str:
if result == "fail":
return "product"
if result == "environment-fail":
return "test"
raise ValueError(f"只有 fail/environment-fail 可以生成 issue: {result}")
def _value(row: Mapping[str, Any], key: str, defaults: Mapping[str, Any]) -> Any:
value = row.get(key)
return value if value not in (None, "") else defaults.get(key, "")
def build_issue(
row: Mapping[str, Any],
*,
defaults: Mapping[str, Any] | None = None,
issue_type: str | None = None,
) -> dict[str, Any]:
"""Create the stable issue shape required by the run report."""
defaults = defaults or {}
result = str(row.get("result") or "")
if result not in ISSUE_RESULTS:
raise ValueError(f"不能为 {result!r} 结果创建 defect: {row.get('id')}")
selected_type = issue_type or row.get("issue_type") or issue_type_for(result)
if selected_type not in ISSUE_TYPES:
raise ValueError(f"issue_type 必须是 product/test: {selected_type}")
testcase = str(row.get("testcase") or row.get("id") or "")
if not testcase:
raise ValueError("issue 缺少 testcase")
return {
"id": str(row.get("issue_id") or f"TEST-{testcase}-{result}"),
"testcase": testcase,
"issue_type": selected_type,
"result": result,
"phase": _value(row, "phase", defaults),
"expected": _value(row, "expected", defaults),
"observed": _value(row, "observed", defaults),
"artifact_hash": _value(row, "artifact_hash", defaults),
"environment": _value(row, "environment", defaults),
"evidence_path": _value(row, "evidence_path", defaults),
"reproduction": _value(row, "reproduction", defaults),
"next_step": _value(row, "next_step", defaults),
"created_at": str(row.get("created_at") or _now()),
}
def collect_issues(
results: Iterable[Mapping[str, Any]],
*,
defaults: Mapping[str, Any] | None = None,
) -> list[dict[str, Any]]:
issues: list[dict[str, Any]] = []
for row in results:
if not isinstance(row, Mapping) or str(row.get("result") or "") not in ISSUE_RESULTS:
continue
explicit = row.get("issue")
source = dict(row)
if isinstance(explicit, Mapping):
source.update(explicit)
issues.append(build_issue(source, defaults=defaults))
return issues
def feedback_plan(
trigger: str,
*,
target: str = "",
related_issue: str = "",
dispatcher: str = "dispatcher",
) -> dict[str, Any]:
"""Describe the required handoff without performing remote mutations."""
if trigger not in TRIGGERS:
raise ValueError(f"trigger 必须是 {', '.join(TRIGGERS)}")
if trigger == "pr":
actions = [
{"action": "update_pr", "target": target, "status": "required"},
{"action": "update_related_issue", "target": related_issue, "status": "required"},
{"action": "notify_dispatcher", "target": dispatcher, "status": "required"},
{"action": "reassign_development", "target": dispatcher, "status": "pending"},
]
route = "pr-update-issue-dispatcher-reassign"
else:
actions = [
{"action": "return_to_development", "target": target or trigger, "status": "required"},
]
route = "development"
return {
"trigger": trigger,
"route": route,
"target": target,
"related_issue": related_issue,
"actions": actions,
"remote_mutation": "caller-required",
"created_at": _now(),
}
def attach_feedback(
run: Mapping[str, Any],
*,
trigger: str | None = None,
target: str = "",
related_issue: str = "",
dispatcher: str = "dispatcher",
defaults: Mapping[str, Any] | None = None,
) -> dict[str, Any]:
"""Return a new run mapping with results, defects and routing preserved."""
output = deepcopy(dict(run))
results = output.get("results")
if not isinstance(results, list):
raise ValueError("run 缺少 results 列表")
issues = collect_issues(results, defaults=defaults)
output["defects"] = issues
# Compatibility name for consumers that call the same structure
# ``test-issues``. It is deliberately a copy, not a second source of truth.
output["test_issues"] = deepcopy(issues)
if trigger:
output["feedback"] = feedback_plan(
trigger,
target=target,
related_issue=related_issue,
dispatcher=dispatcher,
)
return output
@@ -13,6 +13,8 @@ import sys
from pathlib import Path
from typing import Any
from journey import suite as case_suite, test_type
try:
import yaml
except ImportError: # pragma: no cover
@@ -22,6 +24,10 @@ except ImportError: # pragma: no cover
FIELDS = (
"category",
"id",
"test_type",
"suite",
"client_suite",
"platform",
"maps_to",
"title",
"what",
@@ -156,10 +162,17 @@ def export_cases_csv(root: Path, out: Path | None = None) -> Path:
writer.writeheader()
for case in cases:
cid = str(case.get("id") or "")
client_suite_value = case.get("client_suite") or (
case.get("suite") if test_type(case) == "journey" else ""
)
writer.writerow(
{
"category": case.get("category") or "",
"id": cid,
"test_type": test_type(case),
"suite": case_suite(case),
"client_suite": client_suite_value or "",
"platform": one_line(case.get("platform")),
"maps_to": case.get("maps_to") or "",
"title": case.get("title") or "",
"what": one_line(case.get("what")),
+53
View File
@@ -18,6 +18,7 @@ except ImportError: # pragma: no cover
raise SystemExit(2)
from generate_cases_csv import load_cases, one_line, resolve_root, tester_dir
from journey import suite as case_suite, test_type
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
RUN_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
@@ -25,6 +26,10 @@ EXECUTED_YES = frozenset({"pass", "fail", "environment-fail"})
EXECUTED_NO = frozenset({"skipped", "not-run"})
FIELDS = (
"id",
"test_type",
"suite",
"client_suite",
"platform",
"result",
"executed",
"evidence",
@@ -35,6 +40,16 @@ FIELDS = (
"risk",
"driver",
"maps_to",
"blocked_by",
"issue_id",
"issue_type",
"expected",
"observed",
"artifact_hash",
"environment",
"evidence_path",
"reproduction",
"next_step",
)
@@ -107,6 +122,23 @@ def case_index(root: Path) -> dict[str, dict[str, Any]]:
return out
def issue_index(run: dict[str, Any]) -> dict[str, dict[str, Any]]:
"""Index canonical defects and compatibility test_issues by testcase."""
indexed: dict[str, dict[str, Any]] = {}
raw = run.get("defects")
if not isinstance(raw, list):
raw = run.get("test_issues")
if not isinstance(raw, list):
return indexed
for issue in raw:
if not isinstance(issue, dict):
continue
testcase = str(issue.get("testcase") or issue.get("id") or "")
if testcase:
indexed.setdefault(testcase, issue)
return indexed
def export_run_csv(
root: Path,
run_path: Path | None = None,
@@ -118,6 +150,7 @@ def export_run_csv(
path = path.resolve()
run = load_run(path)
cases = case_index(root)
issues = issue_index(run)
dest = out or path.with_suffix(".csv")
dest.parent.mkdir(parents=True, exist_ok=True)
with dest.open("w", encoding="utf-8-sig", newline="") as handle:
@@ -129,9 +162,19 @@ def export_run_csv(
cid = str(item["id"])
result = str(item.get("result") or "")
case = cases.get(cid) or {}
issue = dict(issues.get(cid, {}))
if isinstance(item.get("issue"), dict):
issue.update(item["issue"])
client_suite_value = case.get("client_suite") or (
case.get("suite") if test_type(case) == "journey" else ""
)
writer.writerow(
{
"id": cid,
"test_type": test_type(case),
"suite": case_suite(case),
"client_suite": client_suite_value or "",
"platform": one_line(case.get("platform")),
"result": result,
"executed": executed_cell(result),
"evidence": one_line(item.get("evidence")),
@@ -142,6 +185,16 @@ def export_run_csv(
"risk": case.get("risk") or "",
"driver": case.get("driver") or "",
"maps_to": case.get("maps_to") or "",
"blocked_by": one_line(item.get("blocked_by")),
"issue_id": issue.get("id") or item.get("issue_id") or "",
"issue_type": issue.get("issue_type") or item.get("issue_type") or "",
"expected": one_line(issue.get("expected") or item.get("expected")),
"observed": one_line(issue.get("observed") or item.get("observed")),
"artifact_hash": issue.get("artifact_hash") or item.get("artifact_hash") or "",
"environment": one_line(issue.get("environment") or item.get("environment")),
"evidence_path": one_line(issue.get("evidence_path") or item.get("evidence_path")),
"reproduction": one_line(issue.get("reproduction") or item.get("reproduction")),
"next_step": one_line(issue.get("next_step") or item.get("next_step")),
}
)
return dest
+290
View File
@@ -0,0 +1,290 @@
#!/usr/bin/env python3
"""Shared client-journey contract and result lifecycle helpers.
The mobile/browser driver still owns the real UI interaction. This module
owns the parts that must be identical for every client catalog: contract
validation, preflight blocking, and result propagation. It deliberately
returns the existing tester result vocabulary instead of introducing a second
report format.
"""
from __future__ import annotations
from typing import Any, Iterable, Mapping
TEST_TYPES = ("journey", "integration")
CLIENT_SUITES = ("client-smoke", "client-regression", "safety")
INTEGRATION_SUITES = ("api-contract", "authorization", "build-contract")
PLATFORMS = ("android", "ios")
FAILURE_POLICIES = ("stop", "continue")
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
def client_suite(case: Mapping[str, Any]) -> str:
"""Return the canonical client suite, accepting ``suite`` as a migration alias."""
value = case.get("client_suite")
alias = case.get("suite")
if value and alias and str(value) != str(alias):
return ""
return str(value or alias or "")
def integration_suite(case: Mapping[str, Any]) -> str:
"""Return the integration suite, accepting ``suite`` during migration."""
value = case.get("integration_suite")
alias = case.get("suite")
if value and alias and str(value) != str(alias):
return ""
return str(value or alias or "")
def test_type(case: Mapping[str, Any]) -> str:
"""Return an explicit type or infer one for an old catalog entry."""
declared = str(case.get("test_type") or "").strip().lower()
if declared:
return declared
if client_suite(case) or case.get("journey") is not None:
return "journey"
if integration_suite(case):
return "integration"
return ""
def suite(case: Mapping[str, Any]) -> str:
"""Return the report/selection suite for either test type."""
kind = test_type(case)
return client_suite(case) if kind == "journey" else integration_suite(case)
def normalize_platforms(value: Any) -> list[str]:
if isinstance(value, str):
values = [value]
elif isinstance(value, (list, tuple)):
values = list(value)
else:
return []
return [str(item).strip().lower() for item in values if str(item).strip()]
def _non_empty_list(value: Any) -> bool:
return isinstance(value, list) and bool(value) and all(str(item).strip() for item in value)
def _has_account(fixture: Mapping[str, Any]) -> bool:
account = fixture.get("account")
if isinstance(account, Mapping):
return bool(str(account.get("role") or account.get("username") or "").strip())
return bool(str(account or "").strip())
def contract_errors(case: Mapping[str, Any], sessions: Mapping[str, Any] | None = None) -> list[str]:
"""Validate a client journey without requiring app-specific field names.
A case becomes a client journey when it declares ``client_suite`` (or the
temporary ``suite`` alias). Legacy non-client cases keep the existing
schema. Once opted in, all lifecycle inputs are explicit so a missing
fixture/cleanup can stop the run before any case result is fabricated.
"""
kind = test_type(case)
declared_kind = str(case.get("test_type") or "").strip().lower()
if declared_kind and declared_kind not in TEST_TYPES:
return [f"{case.get('id') or '?'}: test_type 必须是 journey 或 integration"]
if not kind:
return []
if kind == "integration":
return integration_contract_errors(case)
suite_name = client_suite(case)
journey = case.get("journey")
errors: list[str] = []
cid = str(case.get("id") or "?")
where = f"{cid}: client journey"
if case.get("client_suite") and case.get("suite") and str(case["client_suite"]) != str(case["suite"]):
errors.append(f"{where}: client_suite 与 suite 不一致")
if suite_name not in CLIENT_SUITES:
errors.append(f"{where}: client_suite 必须是 {', '.join(CLIENT_SUITES)}")
if not isinstance(journey, Mapping):
errors.append(f"{where}: journey 必须是 mapping")
journey = {}
for key in ("goal", "entry_state", "final_state"):
if not str(journey.get(key) or "").strip():
errors.append(f"{where}: journey.{key} 必填")
if not _non_empty_list(journey.get("actions")):
errors.append(f"{where}: journey.actions 必须是非空列表")
if not _non_empty_list(journey.get("assertions")):
errors.append(f"{where}: journey.assertions 必须是非空列表")
policy = str(journey.get("failure_policy") or "")
if policy not in FAILURE_POLICIES:
errors.append(f"{where}: journey.failure_policy 必须是 stop 或 continue")
fixture = case.get("fixture")
if not isinstance(fixture, Mapping):
errors.append(f"{where}: fixture 缺失,必须在 preflight 前声明")
elif not _has_account(fixture):
errors.append(f"{where}: fixture.account 缺失,必须声明测试账号或角色")
cleanup = case.get("cleanup")
if not isinstance(cleanup, Mapping):
errors.append(f"{where}: cleanup 缺失,必须在 preflight 前声明")
else:
steps = cleanup.get("steps")
command = cleanup.get("run")
explicitly_noop = cleanup.get("required") is False and str(cleanup.get("reason") or "").strip()
if not explicitly_noop and not _non_empty_list(steps) and not str(command or "").strip():
errors.append(f"{where}: cleanup 必须有非空 steps/run;只读用例也要显式声明 reason")
platforms = normalize_platforms(case.get("platform"))
if not platforms:
errors.append(f"{where}: platform 必须声明 android/ios")
elif any(platform not in PLATFORMS for platform in platforms):
errors.append(f"{where}: platform 只能是 {', '.join(PLATFORMS)}")
elif len(platforms) != len(set(platforms)):
errors.append(f"{where}: platform 不能重复")
driver = case.get("driver")
if driver not in ("mobile", "browser"):
errors.append(f"{where}: 客户端旅程 driver 必须是 mobile 或 browser")
if driver and driver in ("mobile", "browser"):
session = case.get("session")
if not str(session or "").strip():
errors.append(f"{where}: session 必须声明可复用客户端会话")
elif sessions is not None and session not in sessions:
errors.append(f"{where}: session={session} 未在 catalog.sessions 声明")
if suite_name == "safety" and str(case.get("tag") or "") == "smoke":
errors.append(f"{where}: safety 不得标记为 smoke")
return errors
def integration_contract_errors(case: Mapping[str, Any]) -> list[str]:
"""Validate the non-client half of the tester catalog.
Integration cases deliberately do not require a browser/mobile session. A
``how`` block is the legacy call declaration; newer catalogs may use
``request``/``input``/``call`` or an ``integration`` mapping as well.
``expected`` remains the common protocol/data assertion list.
"""
cid = str(case.get("id") or "?")
where = f"{cid}: integration"
errors: list[str] = []
value = case.get("integration_suite")
alias = case.get("suite")
if value and alias and str(value) != str(alias):
errors.append(f"{where}: integration_suite 与 suite 不一致")
suite_name = integration_suite(case)
if suite_name not in INTEGRATION_SUITES:
errors.append(
f"{where}: integration_suite 必须是 {', '.join(INTEGRATION_SUITES)}"
)
fixture = case.get("fixture")
if not isinstance(fixture, Mapping) or not fixture:
errors.append(f"{where}: fixture 缺失,必须在 preflight 前声明")
declared_call = any(
key in case and case.get(key) not in (None, "", [], {})
for key in ("request", "input", "call", "integration")
)
how = case.get("how")
if not declared_call and not isinstance(how, Mapping):
errors.append(f"{where}: 必须声明 request/input/call/integration 或 how")
# Integration writes need an explicit rollback/cleanup contract. Read-only
# probes may state why cleanup is unnecessary, matching test-preflight.
risk = str(case.get("risk") or "")
cleanup = case.get("cleanup")
cleanup_steps = isinstance(cleanup, Mapping) and (
bool(cleanup.get("steps")) or bool(str(cleanup.get("run") or "").strip())
)
cleanup_noop = isinstance(cleanup, Mapping) and cleanup.get("required") is False and bool(
str(cleanup.get("reason") or "").strip()
)
if risk != "readonly" and not (cleanup_steps or cleanup_noop):
errors.append(f"{where}: 写入型 integration 必须声明 cleanup/rollback steps/run")
elif cleanup is not None and not (cleanup_steps or cleanup_noop):
errors.append(f"{where}: cleanup 必须有 steps/run 或 required=false + reason")
forbidden = []
if case.get("client_suite") or case.get("journey") is not None:
forbidden.append("client_suite/journey")
if case.get("session") or case.get("platform"):
forbidden.append("session/platform")
if forbidden:
errors.append(f"{where}: integration 不应声明 " + ", ".join(forbidden))
return errors
def preflight(cases: Iterable[Mapping[str, Any]], sessions: Mapping[str, Any] | None = None) -> dict[str, Any]:
"""Return a contract preflight result; no per-case result is created on block."""
cases = list(cases)
errors: list[str] = []
for case in cases:
errors.extend(contract_errors(case, sessions=sessions))
return {
"required": bool(errors or any(client_suite(case) for case in cases)),
"status": "blocked" if errors else "ready",
"failure_class": "contract" if errors else None,
"errors": errors,
}
def classify_outcome(
*,
ok: bool,
environment_ready: bool = True,
phase: str = "journey",
) -> str:
"""Classify one observed phase using the tester's five result states.
A valid test account is part of the client journey: a failed login is a
product ``fail``. Only an unavailable environment is ``environment-fail``.
"""
if not environment_ready:
return "environment-fail"
if not ok:
return "fail"
return "pass"
def propagate_results(
case_ids: Iterable[str],
observed: Mapping[str, str],
*,
skipped: Mapping[str, str] | None = None,
continue_on_fail: bool = False,
policies: Mapping[str, str] | None = None,
blocked_by: Mapping[str, Iterable[str]] | None = None,
) -> list[dict[str, str]]:
"""Build ordered result rows and mark cases after a blocking failure not-run.
``skipped`` is intentional selection (risk/lifecycle/assumption); it is
never converted to ``not-run``. A product or environment failure stops
subsequent cases unless the caller explicitly opts into continuation.
"""
skipped = skipped or {}
policies = policies or {}
blocked_by = blocked_by or {}
rows: list[dict[str, str]] = []
blocking_case = ""
for raw_id in case_ids:
case_id = str(raw_id)
if case_id in skipped:
rows.append({"id": case_id, "result": "skipped", "notes": skipped[case_id]})
continue
if blocking_case:
rows.append({
"id": case_id,
"result": "not-run",
"notes": "前置用例失败,未轮到执行",
"blocked_by": ",".join(blocked_by.get(case_id) or (blocking_case,)),
})
continue
result = observed.get(case_id, "not-run")
if result not in RESULTS:
raise ValueError(f"未知结果: {case_id}={result}")
rows.append({"id": case_id, "result": result})
policy = policies.get(case_id)
if result in ("fail", "environment-fail") and not (continue_on_fail or policy == "continue"):
blocking_case = case_id
return rows
+302 -10
View File
@@ -11,6 +11,7 @@ from __future__ import annotations
import argparse
import hashlib
import importlib.util
import json
import os
import plistlib
@@ -33,6 +34,7 @@ from typing import Any
COMMIT = re.compile(r"^[0-9a-fA-F]{40}$")
PLATFORMS = {"android", "ios"}
PREFLIGHT_ROOT = Path(__file__).resolve().parents[2] / "test-preflight" / "scripts" / "preflight.py"
class EnvironmentFailure(RuntimeError):
@@ -52,6 +54,7 @@ class CommandResult:
@dataclass
class RuntimeContext:
root: Path
bindings: dict[str, str] = field(default_factory=dict)
retries: dict[str, int] = field(default_factory=lambda: {"download": 3, "start": 3, "install": 3})
evidence: list[dict[str, Any]] = field(default_factory=list)
@@ -124,6 +127,120 @@ def parse_artifacts(values: list[str]) -> dict[str, str]:
return pairs
def parse_bindings(values: list[str], requested: list[str]) -> dict[str, str]:
"""Resolve one tester binding per requested platform.
A single legacy ``agent`` mapping cannot safely represent a multi-platform
run. Keep the platform distinction in the command contract all the way to
agent-runtime instead of treating binding as report metadata.
"""
pairs: dict[str, str] = {}
for raw in values:
platform, sep, value = raw.partition("=")
platform = platform.strip().lower()
value = value.strip()
if not sep or platform not in PLATFORMS or not value or not re.fullmatch(r"[a-z][a-z0-9-]*(?:\.[a-z][a-z0-9-]*)+", value):
raise ArtifactFailure(f"--binding 格式必须是 android=tester.<name> 或 ios=tester.<name>: {raw}")
if platform in pairs:
raise ArtifactFailure(f"--binding 重复平台: {platform}")
pairs[platform] = value
resolved = {platform: pairs.get(platform, f"tester.{platform}") for platform in requested}
if len(set(resolved.values())) != len(resolved) and len(resolved) > 1:
raise ArtifactFailure("多平台运行不能复用同一个 tester binding;请分别配置 tester.android/tester.ios")
return resolved
def load_agent_runtime():
candidates = [
Path.home() / ".codex" / "skills" / "agent-runtime" / "scripts" / "agent.py",
Path.home() / ".pouch" / "skills" / "agent-runtime" / "scripts" / "agent.py",
Path(__file__).resolve().parents[2] / "agent-runtime" / "scripts" / "agent.py",
]
runtime_path = next((item for item in candidates if item.is_file()), None)
if runtime_path is None:
raise EnvironmentFailure("找不到共享 agent-runtime: " + ", ".join(str(item) for item in candidates))
spec = importlib.util.spec_from_file_location("pouch_agent_runtime_for_mobile_bundle", runtime_path)
if spec is None or spec.loader is None:
raise EnvironmentFailure(f"无法加载共享 agent-runtime: {runtime_path}")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def binding_location(root: Path, binding_id: str) -> dict[str, Any]:
runtime = load_agent_runtime()
try:
value = runtime.binding(runtime.load_config(root), binding_id)
except (ValueError, OSError) as exc:
raise EnvironmentFailure(f"无法解析 {binding_id}: {exc}") from exc
if value.get("adapter") != "tester":
raise EnvironmentFailure(f"{binding_id} 不是 tester binding")
return value
def agent_runner(root: Path, request: dict[str, Any]) -> dict[str, Any]:
"""Dispatch a platform operation through the resolved tester binding."""
runtime = load_agent_runtime()
return runtime.execute_runner_request(root, request)
def platform_node(root: Path, binding_id: str) -> str:
location = binding_location(root, binding_id)["location"]
return "local" if location.get("environment") == "local" else str(location.get("environment"))
def validate_preflight_matrix(root: Path, config_path: Path, bindings: dict[str, str], requested: list[str]) -> None:
"""Fail closed when a matrix routes a platform to the wrong node/runner."""
try:
import yaml
raw = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
except (OSError, ValueError, ImportError) as exc:
raise EnvironmentFailure(f"无法读取 preflight matrix: {config_path}: {exc}") from exc
rows = raw.get("matrix") if isinstance(raw, dict) else None
if not isinstance(rows, list):
raise EnvironmentFailure("preflight matrix 缺少 matrix;不能推断平台执行节点")
by_platform = {str(row.get("platform", "")).lower(): row for row in rows if isinstance(row, dict)}
for platform in requested:
expected_binding = bindings[platform]
row = by_platform.get(platform)
if not row:
raise EnvironmentFailure(f"preflight matrix 缺少 {platform} 行;不能创建该平台 session")
tester = row.get("tester") if isinstance(row, dict) else None
actual_binding = tester.get("binding") if isinstance(tester, dict) else None
if actual_binding != expected_binding:
raise EnvironmentFailure(f"{platform} preflight binding={actual_binding!r},期望 {expected_binding!r}")
expected_node = platform_node(root, expected_binding)
actual_node = tester.get("node") if isinstance(tester, dict) else None
if expected_node == "local" and actual_node not in {"local", "localhost", "127.0.0.1"}:
raise EnvironmentFailure(f"{platform} binding 是 local,但 preflight node={actual_node!r}")
runner = tester.get("runner") if isinstance(tester, dict) else None
if expected_node != "local" and (not isinstance(runner, dict) or runner.get("kind") != "contract"):
raise EnvironmentFailure(f"{platform} 绑定到 {expected_node},必须使用 contract runner;禁止在协调节点执行")
def preflight_remote_artifacts(config_path: Path, requested: list[str]) -> dict[str, str]:
"""Return caller-declared artifact paths readable by remote tester nodes."""
try:
import yaml
raw = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
except (OSError, ValueError, ImportError) as exc:
raise EnvironmentFailure(f"无法读取 preflight matrix: {config_path}: {exc}") from exc
rows = raw.get("matrix") if isinstance(raw, dict) else None
if not isinstance(rows, list):
raise EnvironmentFailure("preflight matrix 缺少 matrix;不能解析远程产物路径")
by_platform = {str(row.get("platform", "")).lower(): row for row in rows if isinstance(row, dict)}
paths: dict[str, str] = {}
for platform in requested:
row = by_platform.get(platform) or {}
artifact = row.get("artifact") if isinstance(row, dict) else None
remote_path = artifact.get("remote_path") if isinstance(artifact, dict) else None
if isinstance(remote_path, str) and remote_path.strip():
paths[platform] = remote_path.strip()
return paths
def local_path(locator: str) -> Path | None:
parsed = urllib.parse.urlsplit(locator)
if parsed.scheme in ("http", "https"):
@@ -209,6 +326,10 @@ def android_identity(path: Path, context: RuntimeContext) -> str:
result = run_command(command, context=context)
if result.code == 0:
match = re.search(r"(?:name|application-id)[=:]['\"]?([A-Za-z0-9_.]+)", result.output)
if not match:
# apkanalyzer manifest application-id prints the bare package
# name, while aapt/aapt2 print a package: name=... record.
match = re.fullmatch(r"\s*([A-Za-z0-9_.]+)\s*", result.output)
if match:
return match.group(1)
raise EnvironmentFailure("APK 身份读取失败;请检查 Android SDK build-tools")
@@ -388,20 +509,71 @@ def write_run(root: Path, record: dict[str, Any]) -> Path:
return path
def all_active_result_rows(root: Path, *, blocked_by: str) -> list[dict[str, Any]]:
"""Keep a testcase row for every active catalog entry when preflight blocks."""
try:
from generate_cases_csv import load_cases
cases = load_cases(root)
except (ImportError, OSError, ValueError, SystemExit):
cases = []
rows: list[dict[str, Any]] = []
for case in cases:
if case.get("lifecycle", "active") != "active":
continue
rows.append({
"id": str(case.get("id") or ""),
"result": "not-run",
"notes": "test-preflight 未通过,未创建移动 session",
"blocked_by": [blocked_by],
})
return rows
def run_test_preflight(
config_path: Path,
*,
mode: str = "verify",
yaml_path: Path | None = None,
csv_path: Path | None = None,
) -> dict[str, Any]:
"""Run the shared preflight contract before any client session work."""
if mode not in {"verify", "prepare"}:
raise ArtifactFailure("preflight mode 只能是 verify 或 prepare")
if not PREFLIGHT_ROOT.is_file():
raise EnvironmentFailure(f"找不到 test-preflight skill: {PREFLIGHT_ROOT}")
spec = importlib.util.spec_from_file_location("pouch_test_preflight", PREFLIGHT_ROOT)
if spec is None or spec.loader is None:
raise EnvironmentFailure("无法加载 test-preflight skill")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
try:
config = module.load_config(config_path)
evidence = module.run(config, mode)
yaml_path = yaml_path or config_path.with_name("preflight.yaml")
csv_path = csv_path or config_path.with_name("preflight.csv")
return module.write_evidence(evidence, yaml_path, csv_path)
except (OSError, ValueError, module.yaml.YAMLError) as exc:
return {"status": "blocked", "error": str(exc), "yaml": str(yaml_path or ""), "csv": str(csv_path or "")}
def execute(args: argparse.Namespace) -> int:
requested = platforms(args.platform)
context = RuntimeContext(Path(args.root).resolve())
root = Path(args.root).resolve()
bindings = parse_bindings(args.bindings, requested)
context = RuntimeContext(root, bindings=bindings)
record: dict[str, Any] = {
"schema_version": 1,
"started_at": datetime.now(timezone.utc).isoformat(),
"platforms_requested": requested,
"partial": bool(args.partial),
"binding": args.binding,
"bindings": bindings,
"node": os.environ.get("TESTER_NODE", "local"),
"terminal": {"handle": os.environ.get("ORCA_TERMINAL_HANDLE"), "created": os.environ.get("TESTER_TERMINAL_CREATED", "false").lower() == "true", "closed": os.environ.get("TESTER_TERMINAL_CLOSED", "unknown")},
"product_test": {"status": "not-run", "failure_class": "automatic-repair-failed", "note": "准备完成后由 tester catalog 驱动真实移动 session;本工具不以 API 替代移动测试"},
"platforms": {},
}
remote_artifacts: dict[str, str] = {}
try:
locators = parse_artifacts(args.artifacts)
missing = [platform for platform in requested if platform not in locators]
@@ -417,7 +589,7 @@ def execute(args: argparse.Namespace) -> int:
artifact_dir = Path(tempfile.mkdtemp(prefix="pouch-tester-artifacts-"))
record["artifact_dir"] = str(artifact_dir)
expected = catalog_identities(context.root)
prepared: dict[str, tuple[dict[str, Any], dict[str, str], str]] = {}
prepared: dict[str, tuple[dict[str, Any], dict[str, str] | None, str]] = {}
failures: dict[str, str] = {}
for platform in requested:
result: dict[str, Any] = {"node": "tester.android" if platform == "android" else "tester.ios", "status": "preparing"}
@@ -429,22 +601,120 @@ def execute(args: argparse.Namespace) -> int:
wanted = expected.get(platform)
if wanted and identity != wanted:
raise EnvironmentFailure(f"{platform} 身份不匹配: expected={wanted} actual={identity}")
node = android_preflight(context) if platform == "android" else ios_preflight(context)
check_network(context, node)
result.update({"identity": identity, "node_preflight": node, "status": "preflight-passed"})
prepared[platform] = (result, node, identity)
result.update({"identity": identity, "status": "artifact-ready"})
prepared[platform] = (result, None, identity)
except (EnvironmentFailure, ArtifactFailure) as exc:
failures[platform] = str(exc)
result.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc)})
# This is the hard gate shared by integration and journey callers.
# There is no safe implicit matrix: without a caller-supplied config we
# cannot prove HOST, fixtures, tester bindings, or node reachability.
preflight_config = Path(args.preflight_config).expanduser() if args.preflight_config else None
if not failures:
if preflight_config is None:
preflight_result = {
"status": "blocked",
"error": "缺少 --preflight-config;不能跳过 test-preflight",
}
else:
validate_preflight_matrix(context.root, preflight_config, bindings, requested)
remote_artifacts = preflight_remote_artifacts(preflight_config, requested)
preflight_result = run_test_preflight(
preflight_config,
mode=args.preflight_mode,
yaml_path=Path(args.preflight_yaml).expanduser() if args.preflight_yaml else None,
csv_path=Path(args.preflight_csv).expanduser() if args.preflight_csv else None,
)
record["preflight"] = preflight_result
if preflight_result.get("status") != "ready":
failures["preflight"] = str(preflight_result.get("error") or preflight_result.get("status"))
if not failures and args.preflight_mode == "prepare":
# test-preflight prepare owns platform probing, installation and
# launch; do not install/start the same artifact a second time.
for result, _, _ in prepared.values():
result.update({"status": "preflight-prepared"})
elif not failures:
for platform in requested:
result, _, identity = prepared[platform]
try:
binding_id = bindings[platform]
node_name = platform_node(context.root, binding_id)
probe = agent_runner(context.root, {
"operation": "platform_probe",
"platform": platform,
"node": node_name,
"binding": binding_id,
})
if probe.get("status") != "ready":
raise EnvironmentFailure(str(probe.get("observed") or probe))
host = os.environ.get("HOST", "").strip()
network = agent_runner(context.root, {
"operation": "network_probe",
"platform": platform,
"node": node_name,
"binding": binding_id,
"url": f"http://{host}/v1/me",
}) if host else {"status": "blocked", "observed": "HOST 未设置"}
if network.get("status") != "ready":
raise EnvironmentFailure(str(network.get("observed") or network))
result.update({"node_preflight": probe, "network_preflight": network, "status": "preflight-passed"})
prepared[platform] = (result, None, identity)
except EnvironmentFailure as exc:
failures[platform] = str(exc)
result.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc)})
if failures and not args.partial:
for platform in requested:
if platform not in failures:
record["platforms"][platform].update({"status": "not-run", "reason": "全量 preflight 闸门被另一平台阻断"})
record.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "next_action": "修复列出的平台环境后重跑;默认不会静默降级"})
overall = record.get("preflight", {}).get("status") if "preflight" in record else "environment-fail"
record.update({"status": overall if overall in {"blocked", "environment-fail"} else "environment-fail", "failure_class": "test-preflight" if "preflight" in record else "automatic-repair-failed", "next_action": "修复列出的平台环境后重跑;默认不会静默降级"})
record["results"] = all_active_result_rows(context.root, blocked_by="test-preflight")
else:
for platform, (result, node, identity) in prepared.items():
try:
installed = install_android(node, Path(result["artifact"]["path"]), identity, context) if platform == "android" else install_ios(node, Path(result["artifact"]["path"]), identity, context)
if args.preflight_mode == "prepare":
# test-preflight already ran platform_probe, network_probe,
# install and start through each matrix binding.
installed = {"delegated": "test-preflight.prepare", "binding": bindings[platform]}
else:
binding_id = bindings[platform]
node_name = platform_node(context.root, binding_id)
probe = agent_runner(context.root, {
"operation": "platform_probe",
"platform": platform,
"node": node_name,
"binding": binding_id,
})
if probe.get("status") != "ready":
raise EnvironmentFailure(str(probe.get("observed") or probe))
host = os.environ.get("HOST", "").strip()
if host:
network = agent_runner(context.root, {
"operation": "network_probe",
"platform": platform,
"node": node_name,
"binding": binding_id,
"url": f"http://{host}/v1/me",
})
if network.get("status") != "ready":
raise EnvironmentFailure(str(network.get("observed") or network))
artifact_path = result["artifact"]["path"]
if node_name != "local":
artifact_path = remote_artifacts.get(platform, "")
if not artifact_path:
raise EnvironmentFailure(f"远程 {platform} 安装必须在 matrix.artifact.remote_path 声明节点可读产物")
installed = agent_runner(context.root, {
"operation": "install",
"platform": platform,
"node": node_name,
"binding": binding_id,
"artifact": artifact_path,
"identity": identity,
})
if installed.get("status") != "ready":
raise EnvironmentFailure(str(installed.get("observed") or installed))
result.update({"status": "ready", "install": installed, "failure_class": None})
except EnvironmentFailure as exc:
failures[platform] = str(exc)
@@ -455,7 +725,18 @@ def execute(args: argparse.Namespace) -> int:
record["evidence"] = context.evidence
except (EnvironmentFailure, ArtifactFailure) as exc:
record.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc), "evidence": context.evidence})
if record.get("status") != "ready" and "results" not in record:
record["results"] = all_active_result_rows(context.root, blocked_by="test-preflight")
path = write_run(context.root, record)
try:
from generate_run_csv import export_run_csv
record["csv"] = str(export_run_csv(context.root, run_path=path))
except (OSError, SystemExit, ValueError):
# The YAML run remains the source of truth when a partial/provisioning
# root has no catalog cases yet.
record["csv"] = ""
path.write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
output = {"status": record["status"], "run": str(path), "platforms": record["platforms"], "worktree": record.get("worktree")}
if record.get("error"):
output["error"] = record["error"]
@@ -475,7 +756,18 @@ def parser() -> argparse.ArgumentParser:
run.add_argument("--partial", action="store_true")
run.add_argument("--root", default=".")
run.add_argument("--worktree-root", default="")
run.add_argument("--binding", default="")
run.add_argument(
"--binding",
action="append",
dest="bindings",
default=[],
metavar="PLATFORM=TESTER_BINDING",
help="每个平台的 tester binding,可重复;默认 tester.<platform>",
)
run.add_argument("--preflight-config", default="", help="test-preflight matrix YAML(必填,不得跳过)")
run.add_argument("--preflight-mode", choices=("verify", "prepare"), default="verify")
run.add_argument("--preflight-yaml", default="", help="preflight YAML 证据输出路径")
run.add_argument("--preflight-csv", default="", help="preflight CSV 证据输出路径")
run.set_defaults(func=execute)
return root
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env python3
"""Append structured tester issue evidence to an existing run snapshot."""
from __future__ import annotations
import argparse
from pathlib import Path
from typing import Any
import yaml
from failure_feedback import attach_feedback
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="将 tester 失败结构化回流到 run YAML/CSV")
parser.add_argument("--run", required=True, type=Path)
parser.add_argument("--trigger", choices=("pr", "requirement", "bug"))
parser.add_argument("--target", default="")
parser.add_argument("--related-issue", default="")
parser.add_argument("--dispatcher", default="dispatcher")
parser.add_argument("--testcase", default="", help="为一条失败结果补充 testcase issue 字段")
parser.add_argument("--issue-type", choices=("product", "test"))
parser.add_argument("--expected", default="")
parser.add_argument("--observed", default="")
parser.add_argument("--artifact-hash", default="")
parser.add_argument("--environment", default="")
parser.add_argument("--evidence-path", default="")
parser.add_argument("--reproduction", default="")
parser.add_argument("--next-step", default="")
args = parser.parse_args(argv)
data: dict[str, Any] = yaml.safe_load(args.run.read_text(encoding="utf-8")) or {}
if args.testcase:
rows = data.get("results")
if not isinstance(rows, list):
raise SystemExit("run 缺少 results 列表")
row = next((item for item in rows if isinstance(item, dict) and str(item.get("id")) == args.testcase), None)
if row is None:
raise SystemExit(f"找不到 testcase: {args.testcase}")
row["issue"] = {
key: value
for key, value in {
"issue_type": args.issue_type,
"expected": args.expected,
"observed": args.observed,
"artifact_hash": args.artifact_hash,
"environment": args.environment,
"evidence_path": args.evidence_path,
"reproduction": args.reproduction,
"next_step": args.next_step,
}.items()
if value
}
output = attach_feedback(
data,
trigger=args.trigger,
target=args.target,
related_issue=args.related_issue,
dispatcher=args.dispatcher,
)
args.run.write_text(yaml.safe_dump(output, allow_unicode=True, sort_keys=False), encoding="utf-8")
csv_path = ""
try:
from generate_run_csv import export_run_csv
csv_path = str(export_run_csv(args.run.parents[3], run_path=args.run))
except (ImportError, OSError, SystemExit, ValueError):
pass
print(f"updated {args.run} defects={len(output.get('defects') or [])} csv={csv_path}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+8 -3
View File
@@ -180,9 +180,14 @@ def clean_config(config: dict[str, Any]) -> dict[str, Any]:
def config_from_binding(root: Path, selector: str) -> dict[str, Any]:
"""Materialize a common agent-runtime tester binding for legacy local.yaml."""
runtime_path = Path(__file__).resolve().parents[3] / "skills" / "agent-runtime" / "scripts" / "agent.py"
if not runtime_path.is_file():
raise RuntimeError(f"找不到共享 agent-runtime: {runtime_path}")
candidates = [
Path.home() / ".codex" / "skills" / "agent-runtime" / "scripts" / "agent.py",
Path.home() / ".pouch" / "skills" / "agent-runtime" / "scripts" / "agent.py",
Path(__file__).resolve().parents[3] / "skills" / "agent-runtime" / "scripts" / "agent.py",
]
runtime_path = next((item for item in candidates if item.is_file()), None)
if runtime_path is None:
raise RuntimeError("找不到共享 agent-runtime: " + ", ".join(str(item) for item in candidates))
spec = importlib.util.spec_from_file_location("pouch_agent_runtime", runtime_path)
if spec is None or spec.loader is None:
raise RuntimeError(f"无法加载共享 agent-runtime: {runtime_path}")
+196
View File
@@ -0,0 +1,196 @@
#!/usr/bin/env python3
"""Validate a tester CSV/YAML pair and summarize statuses from CSV only."""
from __future__ import annotations
import argparse
import csv
import sys
from collections import Counter
from pathlib import Path
from typing import Any
try:
import yaml
except ImportError: # pragma: no cover
print("需要 PyYAML: python3 -m pip install pyyaml", file=sys.stderr)
raise SystemExit(2)
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
RESULT_SET = frozenset(RESULTS)
class SummaryError(ValueError):
"""Raised when the CSV/YAML pair cannot be safely summarized."""
def _format_ids(ids: set[str]) -> str:
return ", ".join(sorted(ids)) or "(none)"
def _duplicate_ids(ids: list[str]) -> list[str]:
counts = Counter(ids)
return sorted(case_id for case_id, count in counts.items() if count > 1)
def load_csv(path: Path) -> list[dict[str, str]]:
"""Load CSV rows and reject malformed IDs/results with diagnostics."""
try:
handle = path.open(encoding="utf-8-sig", newline="")
except OSError as exc:
raise SummaryError(f"无法读取 CSV {path}: {exc}") from exc
with handle:
reader = csv.DictReader(handle)
if not reader.fieldnames:
raise SummaryError(f"CSV {path} 缺少表头")
missing_fields = [field for field in ("id", "result") if field not in reader.fieldnames]
if missing_fields:
raise SummaryError(f"CSV {path} 缺少必需列: {', '.join(missing_fields)}")
rows = list(reader)
ids: list[str] = []
errors: list[str] = []
for line_number, row in enumerate(rows, start=2):
case_id = (row.get("id") or "").strip()
result = (row.get("result") or "").strip()
ids.append(case_id)
if not case_id:
errors.append(f"CSV 第 {line_number} 行 testcase id 缺失")
if not result:
errors.append(f"CSV 第 {line_number} 行 result 缺失 (testcase id={case_id or '(missing)'})")
elif result not in RESULT_SET:
errors.append(
f"CSV 第 {line_number} 行 result 未知: {result!r} "
f"(testcase id={case_id or '(missing)'}; 允许值={', '.join(RESULTS)})"
)
duplicates = _duplicate_ids([case_id for case_id in ids if case_id])
if duplicates:
errors.append(f"CSV testcase id 重复: {_format_ids(set(duplicates))}")
if errors:
raise SummaryError("CSV 校验失败:\n- " + "\n- ".join(errors))
return rows
def load_yaml_results(path: Path) -> list[dict[str, Any]]:
"""Load only YAML results structure needed for count/ID validation."""
try:
data = yaml.safe_load(path.read_text(encoding="utf-8"))
except OSError as exc:
raise SummaryError(f"无法读取 YAML {path}: {exc}") from exc
except yaml.YAMLError as exc:
raise SummaryError(f"YAML 解析失败 {path}: {exc}") from exc
if not isinstance(data, dict) or not isinstance(data.get("results"), list):
raise SummaryError(f"YAML {path} 缺少 results 列表")
results = data["results"]
ids: list[str] = []
errors: list[str] = []
for index, item in enumerate(results, start=1):
if not isinstance(item, dict):
errors.append(f"YAML results[{index}] 不是 mapping")
continue
case_id = str(item.get("id") or "").strip()
ids.append(case_id)
if not case_id:
errors.append(f"YAML results[{index}] testcase id 缺失")
duplicates = _duplicate_ids([case_id for case_id in ids if case_id])
if duplicates:
errors.append(f"YAML testcase id 重复: {_format_ids(set(duplicates))}")
if errors:
raise SummaryError("YAML 校验失败:\n- " + "\n- ".join(errors))
issues = data.get("defects", data.get("test_issues"))
if issues is not None:
if not isinstance(issues, list):
raise SummaryError("YAML defects/test_issues 必须是列表")
result_ids = set(ids)
issue_errors: list[str] = []
for index, issue in enumerate(issues, start=1):
if not isinstance(issue, dict):
issue_errors.append(f"YAML defects[{index}] 不是 mapping")
continue
testcase = str(issue.get("testcase") or "")
issue_type = str(issue.get("issue_type") or "")
if not testcase or testcase not in result_ids:
issue_errors.append(f"YAML defects[{index}] testcase 无效: {testcase or '(missing)'}")
if issue_type not in {"product", "test"}:
issue_errors.append(f"YAML defects[{index}] issue_type 无效: {issue_type or '(missing)'}")
for field in ("expected", "observed", "artifact_hash", "environment", "evidence_path", "reproduction", "next_step"):
if field not in issue:
issue_errors.append(f"YAML defects[{index}] 缺少 {field}")
if issue_errors:
raise SummaryError("YAML defects 校验失败:\n- " + "\n- ".join(issue_errors))
return results
def summarize(csv_path: Path, yaml_path: Path) -> tuple[Counter[str], dict[str, int]]:
"""Validate the pair and return counts calculated from CSV result values."""
rows = load_csv(csv_path)
yaml_results = load_yaml_results(yaml_path)
csv_ids = [(row.get("id") or "").strip() for row in rows]
yaml_ids = [str(item.get("id") or "").strip() for item in yaml_results]
csv_id_set = set(csv_ids)
yaml_id_set = set(yaml_ids)
diagnostics: list[str] = []
if len(rows) != len(csv_id_set):
diagnostics.append(
f"CSV 数据行总数与唯一 testcase id 数不一致: rows={len(rows)} unique_ids={len(csv_id_set)}"
)
if len(rows) != len(yaml_results):
diagnostics.append(
f"CSV 数据行总数与 YAML results 数量不一致: csv_rows={len(rows)} yaml_results={len(yaml_results)}"
)
if csv_id_set != yaml_id_set:
diagnostics.append(
"CSV/YAML testcase id 集合不一致: "
f"仅 CSV 有 [{_format_ids(csv_id_set - yaml_id_set)}]; "
f"仅 YAML 有 [{_format_ids(yaml_id_set - csv_id_set)}]"
)
if diagnostics:
raise SummaryError("CSV/YAML 汇总校验失败:\n- " + "\n- ".join(diagnostics))
counts: Counter[str] = Counter((row.get("result") or "").strip() for row in rows)
total = sum(counts[name] for name in RESULTS)
if total != len(rows):
raise SummaryError(
"CSV 状态计数和不等于数据行总数: "
f"count_sum={total} csv_rows={len(rows)}"
)
metadata = {
"csv_rows": len(rows),
"csv_unique_ids": len(csv_id_set),
"yaml_results": len(yaml_results),
"count_sum": total,
}
return counts, metadata
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="校验 tester CSV/YAML 并从 CSV result 汇总状态")
parser.add_argument("--csv", required=True, type=Path, help="运行成绩单 CSV")
parser.add_argument("--yaml", required=True, type=Path, help="对应运行快照 YAML")
args = parser.parse_args(argv)
try:
counts, metadata = summarize(args.csv, args.yaml)
except SummaryError as exc:
print(f"汇总失败: {exc}", file=sys.stderr)
return 1
bits = " ".join(f"{name}={counts[name]}" for name in RESULTS)
print(f"summary source=CSV.result {bits} total={metadata['count_sum']}")
print(
"validated "
f"csv_rows={metadata['csv_rows']} "
f"csv_unique_ids={metadata['csv_unique_ids']} "
f"yaml_results={metadata['yaml_results']}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+185
View File
@@ -0,0 +1,185 @@
#!/usr/bin/env python3
"""Shared execution policy for tester's integration-first runs.
This module intentionally has no browser, mobile, deployment, or API driver.
Those adapters belong to the caller. It only controls the phase gate and the
stable five-state testcase result propagation so a caller cannot create a
journey session before preflight and integration have passed.
"""
from __future__ import annotations
from collections.abc import Callable, Iterable, Mapping
from typing import Any
from failure_feedback import collect_issues
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
PHASES = ("preflight", "integration", "journey")
def classify_integration(*, ok: bool, service_reachable: bool = True) -> str:
"""Separate a contract assertion failure from an unavailable service."""
if not service_reachable:
return "environment-fail"
return "pass" if ok else "fail"
def _case_id(case: Mapping[str, Any] | str) -> str:
if isinstance(case, Mapping):
return str(case.get("id") or "")
return str(case)
def _not_run(
cases: Iterable[Mapping[str, Any] | str],
reason: str,
*,
blocked_by: Iterable[str] = (),
) -> list[dict[str, Any]]:
rows: list[dict[str, str]] = []
blockers = [str(item) for item in blocked_by if str(item)]
for case in cases:
row: dict[str, Any] = {"id": _case_id(case), "result": "not-run", "notes": reason}
if blockers:
row["blocked_by"] = blockers
if isinstance(case, Mapping):
row["test_type"] = str(case.get("test_type") or "")
row["suite"] = str(
case.get("client_suite")
or case.get("integration_suite")
or case.get("suite")
or ""
)
rows.append(row)
return rows
def _dependencies(
case: Mapping[str, Any] | str,
*,
default: Iterable[str] = (),
) -> list[str]:
if not isinstance(case, Mapping):
return [str(item) for item in default]
declared = case.get("depends_on", case.get("dependencies"))
if declared is None:
return [str(item) for item in default]
if isinstance(declared, str):
return [declared]
if isinstance(declared, Iterable):
return [str(item) for item in declared if str(item)]
raise ValueError(f"depends_on 必须是字符串或列表: {_case_id(case)}")
def _failed_dependencies(dependencies: Iterable[str], results: Mapping[str, Mapping[str, Any]]) -> list[str]:
return [
dependency
for dependency in dependencies
if str((results.get(dependency) or {}).get("result"))
in {"fail", "environment-fail", "not-run"}
]
def _issue_structures(results: list[dict[str, Any]]) -> dict[str, list[dict[str, Any]]]:
issues = collect_issues(results)
return {"defects": issues, "test_issues": [dict(issue) for issue in issues]}
def run_integration_first(
*,
preflight_status: str,
integration_cases: Iterable[Mapping[str, Any] | str],
journey_cases: Iterable[Mapping[str, Any] | str],
execute_integration: Callable[[Mapping[str, Any] | str], str | Mapping[str, Any]],
create_journey_session: Callable[[], Any],
execute_journey: Callable[[Any, Mapping[str, Any] | str], str | Mapping[str, Any]],
) -> dict[str, Any]:
"""Execute integration cases before creating any journey session.
``preflight_status`` must be the status emitted by test-preflight. A
blocked/environment-fail gate records only ``not-run`` testcase rows and
never invokes either adapter. An integration failure stops the phase and
propagates ``not-run`` to all journey cases.
"""
integration = list(integration_cases)
journey = list(journey_cases)
if preflight_status != "ready":
reason = f"test-preflight={preflight_status},未创建客户端 session"
result_rows = _not_run(integration, reason, blocked_by=("test-preflight",)) + _not_run(journey, reason, blocked_by=("test-preflight",))
return {
"status": preflight_status,
"phase": "preflight",
"session_created": False,
"results": result_rows,
**_issue_structures(result_rows),
}
results: list[dict[str, Any]] = []
results_by_id: dict[str, dict[str, Any]] = {}
for case in integration:
case_id = _case_id(case)
dependencies = _dependencies(case)
blocked_by = _failed_dependencies(dependencies, results_by_id)
if blocked_by:
rows = _not_run(
[case],
"依赖的 integration 失败,未执行",
blocked_by=blocked_by,
)
results.extend(rows)
results_by_id[case_id] = rows[0]
continue
outcome = execute_integration(case)
row = dict(outcome) if isinstance(outcome, Mapping) else {"result": outcome}
row["id"] = _case_id(case)
if isinstance(case, Mapping):
row.setdefault("test_type", str(case.get("test_type") or "integration"))
row.setdefault("suite", str(case.get("integration_suite") or case.get("suite") or ""))
result = str(row.get("result") or "not-run")
if result not in RESULTS:
raise ValueError(f"未知 integration 结果: {row['id']}={result}")
row["result"] = result
results.append(row)
results_by_id[case_id] = row
integration_ids = [_case_id(case) for case in integration]
has_runnable_journey = any(
not _failed_dependencies(_dependencies(case, default=integration_ids), results_by_id)
for case in journey
)
session = create_journey_session() if has_runnable_journey else None
for case in journey:
dependencies = _dependencies(case, default=integration_ids)
blocked_by = _failed_dependencies(dependencies, results_by_id)
if blocked_by:
row = _not_run(
[case],
"依赖的用例失败,journey 未执行",
blocked_by=blocked_by,
)[0]
results.append(row)
results_by_id[_case_id(case)] = row
continue
outcome = execute_journey(session, case)
row = dict(outcome) if isinstance(outcome, Mapping) else {"result": outcome}
row["id"] = _case_id(case)
if isinstance(case, Mapping):
row.setdefault("test_type", str(case.get("test_type") or "journey"))
row.setdefault("suite", str(case.get("client_suite") or case.get("suite") or ""))
result = str(row.get("result") or "not-run")
if result not in RESULTS:
raise ValueError(f"未知 journey 结果: {row['id']}={result}")
row["result"] = result
results.append(row)
results_by_id[_case_id(case)] = row
status = "fail" if any(row["result"] == "fail" for row in results) else (
"environment-fail" if any(row["result"] == "environment-fail" for row in results) else "pass"
)
return {
"status": status,
"phase": "journey" if journey else "integration",
"session_created": has_runnable_journey,
"results": results,
**_issue_structures(results),
}
+18
View File
@@ -22,6 +22,24 @@ risks:
- mutating
- destructive
default_max_risk: readonly
# standard enforces the default_max_risk/authorize policy. Writes-style isolated
# resettable environments may set this to isolated-full; risk values remain on
# cases and are still reported.
execution_profile: standard
client_suites:
client-smoke:
default: true
client-regression:
default: false
safety:
default: false
integration_suites:
api-contract:
default: false
authorization:
default: false
build-contract:
default: false
vars: {}
authorize:
mutating: AUTHORIZE_MUTATING
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env python3
import json
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
import yaml
ROOT = Path(__file__).resolve().parents[1]
CATALOG = ROOT / "scripts" / "catalog.py"
def write_fixture(root: Path, *, complete: bool = True) -> None:
tester = root / ".pouch" / "tester"
(tester / "cases").mkdir(parents=True)
(tester / "runs").mkdir()
(tester / "catalog.yaml").write_text(
yaml.safe_dump(
{
"version": 2,
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
"categories": [{"id": "CLIENT"}],
"risks": ["readonly", "mutating", "destructive"],
"default_max_risk": "readonly",
"client_suites": {
"client-smoke": {"default": True},
"client-regression": {"default": False},
"safety": {"default": False},
},
"sessions": {"browser": {"kind": "browser", "reuse": "run"}},
},
sort_keys=False,
),
encoding="utf-8",
)
case = {
"id": "TC-CLIENT-001",
"category": "CLIENT",
"title": "journey",
"what": "client journey",
"lifecycle": "active",
"tag": "smoke",
"risk": "readonly",
"driver": "browser",
"session": "browser",
"client_suite": "client-smoke",
"platform": ["android", "ios"],
"journey": {
"goal": "goal",
"entry_state": "logged_out",
"actions": ["start", "login"],
"assertions": ["visible"],
"final_state": "done",
"failure_policy": "stop",
},
"fixture": {"account": "test-role"},
"cleanup": {"steps": ["cleanup"]},
"how": {"steps": ["start", "login"]},
"expected": [{"kind": "visible-text", "value": "done"}],
}
if not complete:
case.pop("fixture")
case.pop("cleanup")
(tester / "cases" / "client.yaml").write_text(
yaml.safe_dump({"category": "CLIENT", "cases": [case]}, sort_keys=False),
encoding="utf-8",
)
class CatalogJourneyTests(unittest.TestCase):
def run_catalog(self, root: Path, *args: str):
return subprocess.run(
[sys.executable, str(CATALOG), *args, "--root", str(root)],
text=True,
capture_output=True,
)
def test_valid_client_catalog_and_suite_filter(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
write_fixture(root)
valid = self.run_catalog(root, "validate")
self.assertEqual(valid.returncode, 0, valid.stderr)
selected = self.run_catalog(root, "select", "--suite", "client-smoke")
self.assertEqual(selected.returncode, 0, selected.stderr)
self.assertEqual(json.loads(selected.stdout)["count"], 1)
def test_missing_fixture_or_cleanup_blocks_before_results(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
write_fixture(root, complete=False)
selected = self.run_catalog(root, "select", "--suite", "client-smoke")
self.assertNotEqual(selected.returncode, 0)
payload = json.loads(selected.stdout)
self.assertEqual(payload["preflight"]["failure_class"], "contract")
self.assertTrue(any("fixture 缺失" in error for error in payload["preflight"]["errors"]))
self.assertEqual(payload["preflight"]["status"], "blocked")
if __name__ == "__main__":
unittest.main()
+69
View File
@@ -0,0 +1,69 @@
#!/usr/bin/env python3
import sys
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
from journey import classify_outcome, contract_errors, propagate_results
def valid_case(**overrides):
case = {
"id": "TC-CLIENT-001",
"client_suite": "client-smoke",
"driver": "browser",
"session": "browser",
"platform": ["android", "ios"],
"journey": {
"goal": "完成客户端目标",
"entry_state": "logged_out",
"actions": ["启动", "登录", "完成目标"],
"assertions": ["看到目标页面"],
"final_state": "目标已完成",
"failure_policy": "stop",
},
"fixture": {"account": "test-role", "data": []},
"cleanup": {"steps": ["清理本轮数据"]},
}
case.update(overrides)
return case
class JourneyContractTests(unittest.TestCase):
def test_contract_accepts_shared_client_shape(self):
self.assertEqual(contract_errors(valid_case(), {"browser": {}}), [])
def test_missing_fixture_and_cleanup_are_contract_errors(self):
errors = contract_errors(
valid_case(fixture=None, cleanup=None),
{"browser": {}},
)
self.assertTrue(any("fixture 缺失" in error for error in errors))
self.assertTrue(any("cleanup 缺失" in error for error in errors))
def test_login_failure_is_product_failure_but_unavailable_environment_is_not(self):
self.assertEqual(classify_outcome(ok=False, phase="login"), "fail")
self.assertEqual(
classify_outcome(ok=False, phase="login", environment_ready=False),
"environment-fail",
)
def test_failed_case_blocks_remaining_cases_and_keeps_explicit_skip(self):
rows = propagate_results(
["TC-1", "TC-2", "TC-3"],
{"TC-1": "fail", "TC-2": "pass", "TC-3": "pass"},
skipped={"TC-3": "未授权"},
)
self.assertEqual([row["result"] for row in rows], ["fail", "not-run", "skipped"])
def test_continue_policy_allows_next_case(self):
rows = propagate_results(
["TC-1", "TC-2"],
{"TC-1": "fail", "TC-2": "pass"},
policies={"TC-1": "continue"},
)
self.assertEqual([row["result"] for row in rows], ["fail", "pass"])
if __name__ == "__main__":
unittest.main()
+88
View File
@@ -50,6 +50,84 @@ class MobileBundleTests(unittest.TestCase):
with self.assertRaises(bundle.ArtifactFailure):
bundle.parse_artifacts(["android=/tmp/a.apk", "android=/tmp/b.apk"])
def test_parse_bindings_keeps_platforms_separate(self):
self.assertEqual(
bundle.parse_bindings(["android=tester.android", "ios=tester.ios"], ["android", "ios"]),
{"android": "tester.android", "ios": "tester.ios"},
)
self.assertEqual(
bundle.parse_bindings([], ["android"]),
{"android": "tester.android"},
)
with self.assertRaises(bundle.ArtifactFailure):
bundle.parse_bindings(["android=tester.test", "ios=tester.test"], ["android", "ios"])
def test_preflight_matrix_rejects_remote_platform_using_local_runner(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
agent_dir = root / ".pouch"
agent_dir.mkdir()
(agent_dir / "agent.yaml").write_text(
"""
schemaVersion: 1
locations:
local-app:
environment: local
host: local
project: test
path: /tmp/test
remote-ios:
environment: macmini
host: runtime:12345678-1234-1234-1234-123456789012
project: test
path: /tmp/test-ios
profiles:
test-agent:
role: test
cli: codex
tier: standard
model: gpt-5
reasoningEffort: medium
permissionMode: workspace-write
bindings:
tester.android: {adapter: tester, location: local-app, profile: test-agent}
tester.ios: {adapter: tester, location: remote-ios, profile: test-agent}
""".strip()
+ "\n",
encoding="utf-8",
)
config = root / "matrix.yaml"
config.write_text(
"""
matrix:
- platform: android
tester: {binding: tester.android, node: local, runner: {kind: local}}
- platform: ios
tester: {binding: tester.ios, node: local, runner: {kind: local}}
""".strip()
+ "\n",
encoding="utf-8",
)
with self.assertRaisesRegex(bundle.EnvironmentFailure, "ios.*contract runner"):
bundle.validate_preflight_matrix(
root,
config,
{"android": "tester.android", "ios": "tester.ios"},
["android", "ios"],
)
def test_remote_artifact_path_comes_from_preflight_matrix(self):
with tempfile.TemporaryDirectory() as temp:
config = Path(temp) / "matrix.yaml"
config.write_text(
"matrix:\n - platform: ios\n artifact:\n remote_path: /Users/tester/app.ipa\n",
encoding="utf-8",
)
self.assertEqual(
bundle.preflight_remote_artifacts(config, ["ios"]),
{"ios": "/Users/tester/app.ipa"},
)
def test_new_worktree_each_run(self):
with tempfile.TemporaryDirectory() as temp:
repo, worktrees = Path(temp) / "repo", Path(temp) / "worktrees"
@@ -80,6 +158,16 @@ class MobileBundleTests(unittest.TestCase):
self.assertTrue(result["verified"])
self.assertTrue(any("install" in call for call in calls))
def test_android_identity_accepts_bare_apkanalyzer_output(self):
with tempfile.TemporaryDirectory() as temp:
apk = Path(temp) / "app.apk"
apk.write_bytes(b"apk")
context = bundle.RuntimeContext(Path(temp))
with mock.patch.object(bundle.shutil, "which", side_effect=lambda name: "apkanalyzer" if name == "apkanalyzer" else None), mock.patch.object(
bundle, "run_command", return_value=bundle.CommandResult(0, "page.writes\n")
):
self.assertEqual(bundle.android_identity(apk, context), "page.writes")
def test_android_missing_device_is_environment_failure(self):
context = bundle.RuntimeContext(Path("."))
with mock.patch.object(bundle.shutil, "which", return_value="/sdk/adb"), mock.patch.object(
+129
View File
@@ -0,0 +1,129 @@
#!/usr/bin/env python3
import csv
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
import summarize_run
class RunSummaryTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.root = Path(self.temp.name)
self.csv_path = self.root / "run.csv"
self.yaml_path = self.root / "run.yaml"
self.rows = [
{"id": "TC-001", "result": "pass"},
{"id": "TC-002", "result": "fail"},
{"id": "TC-003", "result": "environment-fail"},
{"id": "TC-004", "result": "skipped"},
{"id": "TC-005", "result": "not-run"},
]
self.write_csv(self.rows)
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows])
def tearDown(self):
self.temp.cleanup()
def write_csv(self, rows):
with self.csv_path.open("w", encoding="utf-8", newline="") as handle:
writer = csv.DictWriter(handle, fieldnames=["id", "result"])
writer.writeheader()
writer.writerows(rows)
def write_yaml(self, results):
import yaml
self.yaml_path.write_text(
yaml.safe_dump(
{"counts": {"pass": 999}, "results": results},
sort_keys=False,
),
encoding="utf-8",
)
def run_cli(self):
return subprocess.run(
[
sys.executable,
str(Path(summarize_run.__file__).resolve()),
"--csv",
str(self.csv_path),
"--yaml",
str(self.yaml_path),
],
text=True,
capture_output=True,
)
def test_valid_summary_uses_csv_results(self):
result = self.run_cli()
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("source=CSV.result", result.stdout)
self.assertIn("pass=1 fail=1 environment-fail=1 skipped=1 not-run=1 total=5", result.stdout)
def test_yaml_counts_do_not_drive_status_summary(self):
self.write_csv([{"id": "TC-001", "result": "fail"}])
self.write_yaml([{"id": "TC-001", "result": "fail"}])
result = self.run_cli()
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("pass=0 fail=1", result.stdout)
self.assertNotIn("pass=999", result.stdout)
def test_yaml_result_count_mismatch_is_nonzero(self):
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows[:-1]])
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn("YAML results 数量不一致", result.stderr)
def test_id_set_mismatch_is_nonzero(self):
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows[:-1]] + [{"id": "TC-999"}])
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn("testcase id 集合不一致", result.stderr)
def test_duplicate_csv_id_is_nonzero(self):
rows = self.rows.copy()
rows[-1] = {"id": "TC-001", "result": "not-run"}
self.write_csv(rows)
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn("CSV testcase id 重复", result.stderr)
def test_missing_csv_id_is_nonzero(self):
rows = self.rows.copy()
rows[0] = {"id": "", "result": "pass"}
self.write_csv(rows)
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn("CSV 第 2 行 testcase id 缺失", result.stderr)
def test_duplicate_or_missing_yaml_id_is_nonzero(self):
cases = [
([{"id": "TC-001"}, {"id": "TC-001"}] + [{"id": row["id"]} for row in self.rows[2:]], "YAML testcase id 重复"),
([{"id": ""}] + [{"id": row["id"]} for row in self.rows[1:]], "YAML results[1] testcase id 缺失"),
]
for results, expected in cases:
with self.subTest(expected=expected):
self.write_yaml(results)
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn(expected, result.stderr)
def test_unknown_or_missing_csv_result_is_nonzero(self):
for bad_result, expected in [("unexpected", "result 未知"), ("", "result 缺失")]:
with self.subTest(bad_result=bad_result):
rows = self.rows.copy()
rows[0] = {"id": "TC-001", "result": bad_result}
self.write_csv(rows)
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn(expected, result.stderr)
if __name__ == "__main__":
unittest.main()
+299
View File
@@ -0,0 +1,299 @@
import csv
import json
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
import yaml
SCRIPTS = Path(__file__).resolve().parents[1] / "scripts"
CATALOG = SCRIPTS / "catalog.py"
CASES_CSV = SCRIPTS / "generate_cases_csv.py"
RUN_CSV = SCRIPTS / "generate_run_csv.py"
sys.path.insert(0, str(SCRIPTS))
from two_tier import classify_integration, run_integration_first
def write_mixed_catalog(root: Path) -> None:
tester = root / ".pouch" / "tester"
(tester / "cases").mkdir(parents=True)
(tester / "runs").mkdir()
(tester / "catalog.yaml").write_text(
yaml.safe_dump(
{
"version": 2,
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
"categories": [{"id": "MIXED"}],
"risks": ["readonly", "mutating", "destructive"],
"default_max_risk": "readonly",
"client_suites": {
"client-smoke": {"default": True},
"client-regression": {"default": False},
"safety": {"default": False},
},
"sessions": {"browser": {"kind": "browser", "reuse": "run"}},
},
sort_keys=False,
),
encoding="utf-8",
)
journey = {
"id": "TC-JOURNEY-001",
"test_type": "journey",
"category": "MIXED",
"title": "journey",
"what": "client path",
"lifecycle": "active",
"tag": "smoke",
"risk": "readonly",
"driver": "browser",
"automation": "automated",
"session": "browser",
"client_suite": "client-smoke",
"platform": "android",
"journey": {
"goal": "goal",
"entry_state": "logged_out",
"actions": ["login"],
"assertions": ["visible"],
"final_state": "done",
"failure_policy": "stop",
},
"fixture": {"account": "test-role"},
"cleanup": {"required": False, "reason": "read-only"},
"how": {"steps": ["open"]},
"expected": [{"kind": "visible-text", "value": "done"}],
}
integration = {
"id": "TC-INTEGRATION-001",
"test_type": "integration",
"category": "MIXED",
"title": "contract",
"what": "API contract",
"lifecycle": "active",
"tag": "slow",
"risk": "readonly",
"driver": "script",
"automation": "automated",
"integration_suite": "api-contract",
"fixture": {"references": ["isolated-account"]},
"request": {"method": "GET", "path": "/health"},
"how": {"run": "true"},
"cleanup": {"required": False, "reason": "read-only"},
"expected": [{"kind": "api-status", "value": 200}],
}
(tester / "cases" / "mixed.yaml").write_text(
yaml.safe_dump({"category": "MIXED", "cases": [journey, integration]}, sort_keys=False),
encoding="utf-8",
)
class TwoTierCatalogTests(unittest.TestCase):
def run_catalog(self, root: Path, *args: str) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, str(CATALOG), *args, "--root", str(root)],
text=True,
capture_output=True,
)
def test_default_selection_excludes_integration_and_explicit_selection_includes_it(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
write_mixed_catalog(root)
valid = self.run_catalog(root, "validate")
self.assertEqual(valid.returncode, 0, valid.stderr)
default = self.run_catalog(root, "select")
self.assertEqual(default.returncode, 0, default.stderr)
default_payload = json.loads(default.stdout)
self.assertEqual(default_payload["count"], 1)
self.assertEqual(default_payload["test_types"], {"journey": 1, "integration": 0})
integration = self.run_catalog(root, "select", "--test-type", "integration")
self.assertEqual(integration.returncode, 0, integration.stderr)
integration_payload = json.loads(integration.stdout)
self.assertEqual(integration_payload["count"], 1)
self.assertEqual(integration_payload["cases"][0]["test_type"], "integration")
csv_result = subprocess.run(
[sys.executable, str(CASES_CSV), "--root", str(root)],
text=True,
capture_output=True,
)
self.assertEqual(csv_result.returncode, 0, csv_result.stderr)
with (root / ".pouch/tester/cases.csv").open(encoding="utf-8-sig", newline="") as handle:
rows = list(csv.DictReader(handle))
self.assertEqual({row["test_type"] for row in rows}, {"journey", "integration"})
self.assertEqual({row["suite"] for row in rows}, {"client-smoke", "api-contract"})
run_path = root / ".pouch/tester/runs/two-tier.yaml"
run_path.write_text(
yaml.safe_dump(
{"runId": "two-tier", "results": [
{"id": "TC-INTEGRATION-001", "result": "pass"},
{"id": "TC-JOURNEY-001", "result": "not-run"},
]},
sort_keys=False,
),
encoding="utf-8",
)
run_csv = subprocess.run(
[sys.executable, str(RUN_CSV), "--root", str(root), "--run", str(run_path)],
text=True,
capture_output=True,
)
self.assertEqual(run_csv.returncode, 0, run_csv.stderr)
with run_path.with_suffix(".csv").open(encoding="utf-8-sig", newline="") as handle:
report_rows = list(csv.DictReader(handle))
self.assertEqual(report_rows[0]["test_type"], "integration")
self.assertEqual(report_rows[0]["suite"], "api-contract")
def test_isolated_full_keeps_risk_and_ignores_authorize_filter(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
write_mixed_catalog(root)
catalog_path = root / ".pouch/tester/catalog.yaml"
catalog = yaml.safe_load(catalog_path.read_text(encoding="utf-8"))
catalog["authorize"] = {"mutating": "AUTHORIZE_MUTATING"}
catalog["execution_profile"] = "isolated-full"
catalog_path.write_text(yaml.safe_dump(catalog, sort_keys=False), encoding="utf-8")
case_path = root / ".pouch/tester/cases/mixed.yaml"
cases = yaml.safe_load(case_path.read_text(encoding="utf-8"))
journey = cases["cases"][0]
journey.update({"risk": "mutating", "tag": "full", "client_suite": "client-regression"})
case_path.write_text(yaml.safe_dump(cases, sort_keys=False), encoding="utf-8")
selected = self.run_catalog(root, "select", "--suite", "client-regression")
self.assertEqual(selected.returncode, 0, selected.stderr)
payload = json.loads(selected.stdout)
self.assertEqual(payload["execution_profile"], "isolated-full")
self.assertEqual(payload["risk_filter"], "disabled")
self.assertEqual(payload["count"], 1)
self.assertEqual(payload["cases"][0]["risk"], "mutating")
self.assertEqual(payload["skipped"], [])
all_active = self.run_catalog(root, "select")
self.assertEqual(all_active.returncode, 0, all_active.stderr)
all_payload = json.loads(all_active.stdout)
self.assertEqual(all_payload["count"], 2)
self.assertEqual(all_payload["test_types"], {"journey": 1, "integration": 1})
def test_isolated_full_run_script_does_not_use_authorize_switch(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
tester = root / ".pouch" / "tester"
(tester / "cases").mkdir(parents=True)
(tester / "local.yaml").write_text("HOST: http://127.0.0.1\nAUTHORIZE_MUTATING: false\n", encoding="utf-8")
(tester / "catalog.yaml").write_text(
yaml.safe_dump(
{
"version": 2,
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
"categories": [{"id": "SCRIPT"}],
"risks": ["readonly", "mutating", "destructive"],
"default_max_risk": "readonly",
"execution_profile": "isolated-full",
"authorize": {"mutating": "AUTHORIZE_MUTATING"},
},
sort_keys=False,
),
encoding="utf-8",
)
(tester / "cases" / "script.yaml").write_text(
yaml.safe_dump(
{
"category": "SCRIPT",
"cases": [{
"id": "TC-SCRIPT-001",
"title": "mutating script",
"what": "run isolated script",
"lifecycle": "active",
"tag": "full",
"risk": "mutating",
"driver": "script",
"how": {"run": "exit 0"},
"cleanup": {"steps": ["reset"]},
"expected": [{"kind": "script-exit", "value": 0}],
}],
},
sort_keys=False,
),
encoding="utf-8",
)
result = subprocess.run(
[sys.executable, str(CATALOG), "run-script", "--root", str(root), "--id", "TC-SCRIPT-001"],
text=True,
capture_output=True,
)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(json.loads(result.stdout)["result"], "executed")
class IntegrationFirstTests(unittest.TestCase):
def test_preflight_block_does_not_create_session(self):
calls: list[str] = []
result = run_integration_first(
preflight_status="blocked",
integration_cases=["INT-1"],
journey_cases=["J-1"],
execute_integration=lambda case: calls.append("integration") or "pass",
create_journey_session=lambda: calls.append("session"),
execute_journey=lambda session, case: "pass",
)
self.assertEqual(result["results"][0]["result"], "not-run")
self.assertFalse(result["session_created"])
self.assertEqual(calls, [])
def test_integration_failure_propagates_journey_not_run(self):
calls: list[str] = []
result = run_integration_first(
preflight_status="ready",
integration_cases=["INT-1", "INT-2"],
journey_cases=["J-1", "J-2"],
execute_integration=lambda case: calls.append(case) or ("fail" if case == "INT-1" else "pass"),
create_journey_session=lambda: calls.append("session"),
execute_journey=lambda session, case: "pass",
)
self.assertEqual(result["status"], "fail")
self.assertEqual(calls, ["INT-1", "INT-2"])
self.assertFalse(result["session_created"])
self.assertEqual([row["result"] for row in result["results"]], ["fail", "pass", "not-run", "not-run"])
self.assertEqual(result["results"][2]["blocked_by"], ["INT-1"])
def test_independent_integration_continues_and_dependency_is_blocked(self):
result = run_integration_first(
preflight_status="ready",
integration_cases=[
{"id": "INT-1"},
{"id": "INT-2"},
{"id": "INT-3", "depends_on": ["INT-1"]},
],
journey_cases=[],
execute_integration=lambda case: "fail" if case["id"] == "INT-1" else "pass",
create_journey_session=lambda: object(),
execute_journey=lambda session, case: "pass",
)
self.assertEqual([row["result"] for row in result["results"]], ["fail", "pass", "not-run"])
self.assertEqual(result["results"][2]["blocked_by"], ["INT-1"])
def test_preflight_block_creates_every_result_row(self):
result = run_integration_first(
preflight_status="blocked",
integration_cases=[{"id": "INT-1"}],
journey_cases=[{"id": "J-1"}],
execute_integration=lambda case: "pass",
create_journey_session=lambda: object(),
execute_journey=lambda session, case: "pass",
)
self.assertEqual([row["result"] for row in result["results"]], ["not-run", "not-run"])
self.assertEqual(result["results"][0]["blocked_by"], ["test-preflight"])
def test_service_unreachable_is_environment_failure(self):
self.assertEqual(classify_integration(ok=False, service_reachable=False), "environment-fail")
self.assertEqual(classify_integration(ok=False, service_reachable=True), "fail")
if __name__ == "__main__":
unittest.main()
+59
View File
@@ -12,6 +12,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1]
CASES = REPO_ROOT / "skills" / "tester" / "scripts" / "generate_cases_csv.py"
RUNS = REPO_ROOT / "skills" / "tester" / "scripts" / "generate_run_csv.py"
sys.path.insert(0, str(RUNS.parent))
from failure_feedback import attach_feedback
def load_module(name: str, path: Path):
@@ -29,6 +30,7 @@ class TesterRunRecordTests(unittest.TestCase):
root = Path(temp)
cases_dir = root / ".pouch" / "tester" / "cases"
cases_dir.mkdir(parents=True)
(root / ".pouch" / "tester" / "runs").mkdir()
(root / ".pouch" / "tester" / "catalog.yaml").write_text(
"categories:\n - id: smoke\n title: Smoke\n",
encoding="utf-8",
@@ -51,6 +53,63 @@ class TesterRunRecordTests(unittest.TestCase):
self.assertNotEqual(first, second)
self.assertRegex(first, r"^\d{8}T\d{6}Z-smoke-[0-9a-f]{6}$")
def test_run_csv_preserves_all_results_and_structured_issue_fields(self) -> None:
runs = load_module("tester_run_csv_issues", RUNS)
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
cases_dir = root / ".pouch" / "tester" / "cases"
cases_dir.mkdir(parents=True)
(root / ".pouch" / "tester" / "runs").mkdir()
(root / ".pouch" / "tester" / "catalog.yaml").write_text(
"categories:\n - id: smoke\n title: Smoke\n", encoding="utf-8"
)
(cases_dir / "smoke.yaml").write_text(
"category: smoke\ncases:\n"
" - id: TC-1\n title: One\n what: one\n lifecycle: active\n"
" tag: smoke\n risk: readonly\n how: {steps: [open]}\n"
" expected: [{kind: visible-text, value: ready}]\n"
" - id: TC-2\n title: Two\n what: two\n lifecycle: active\n"
" tag: smoke\n risk: readonly\n how: {steps: [open]}\n"
" expected: [{kind: visible-text, value: ready}]\n",
encoding="utf-8",
)
run = attach_feedback(
{
"runId": "issues",
"results": [
{
"id": "TC-1",
"result": "fail",
"issue": {
"expected": "ready",
"observed": "blank",
"artifact_hash": "sha256:abc",
"environment": "Android tester",
"evidence_path": "evidence/1.png",
"reproduction": "open",
"next_step": "targeted rerun",
},
},
{"id": "TC-2", "result": "not-run", "blocked_by": ["TC-1"]},
],
},
trigger="pr",
target="PR-1",
related_issue="ISSUE-1",
)
run_path = root / ".pouch" / "tester" / "runs" / "issues.yaml"
import yaml
run_path.write_text(yaml.safe_dump(run, sort_keys=False), encoding="utf-8")
out = runs.export_run_csv(root, run_path=run_path)
with out.open(encoding="utf-8-sig", newline="") as handle:
rows = list(csv.DictReader(handle))
self.assertEqual([row["id"] for row in rows], ["TC-1", "TC-2"])
self.assertEqual(rows[0]["issue_type"], "product")
self.assertEqual(rows[0]["expected"], "ready")
self.assertEqual(rows[0]["artifact_hash"], "sha256:abc")
self.assertEqual(rows[1]["blocked_by"], "TC-1")
self.assertEqual(run["feedback"]["route"], "pr-update-issue-dispatcher-reassign")
if __name__ == "__main__":
unittest.main()