Compare commits
11 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| b94aa597ca | |||
| 841a2e0ae6 | |||
| 4d31d074ad | |||
| 29f725fb95 | |||
| 3a757e3376 | |||
| 5b9a693fda | |||
| 9b2d0a28d8 | |||
| 3a18a27dc6 | |||
| c2eb207bd2 | |||
| ffa7cbd63e | |||
| baed23d02a |
@@ -0,0 +1,14 @@
|
||||
# test-preflight
|
||||
|
||||
Reusable preflight for tester matrices. It supports read-only `verify` and
|
||||
test-environment-only `prepare`, and writes per-check YAML/CSV evidence for
|
||||
artifacts, tester nodes, devices, network, services, and fixture/cleanup contracts.
|
||||
This project assumes mobile album/photo/IME capabilities; journey cases exercise them
|
||||
and preflight does not gate them or read authorization environment variables.
|
||||
|
||||
Run the self-tests with:
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s skills/test-preflight/tests
|
||||
python3 /home/ace/.codex/skills/.system/skill-creator/scripts/quick_validate.py skills/test-preflight
|
||||
```
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
name: test-preflight
|
||||
description: >-
|
||||
Verify or prepare a mobile/integration test environment from a declared matrix:
|
||||
artifacts, tester bindings, HOST/PUBLIC_HOST, service probes, and fixture/cleanup
|
||||
contracts. Use before tester runs; it is not a deployment,
|
||||
build, or business-data skill.
|
||||
---
|
||||
|
||||
# test-preflight
|
||||
|
||||
Use this skill before `integration` or `journey` tests. It answers whether the
|
||||
declared environment can start a run and emits one YAML and one CSV evidence
|
||||
file. It never logs in, creates business data, builds/downloads an artifact, or
|
||||
deploys production.
|
||||
|
||||
## Modes
|
||||
|
||||
- `verify` is read-only: validate the matrix, artifact identity, binding, actual
|
||||
tester-node platform/device state, DNS/TCP/HTTP probes, backend probes, and
|
||||
fixture/cleanup declarations. It must not install, launch, or deploy.
|
||||
- `prepare` is side-effecting only within the test environment: call the
|
||||
caller-supplied deployer command contract, run the same checks, then use the
|
||||
tester-node command contract to install and launch the already supplied APK or
|
||||
IPA. It never calls a builder and refuses `environment: prod`.
|
||||
|
||||
The command is:
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/preflight.py verify \
|
||||
--config <matrix.yaml> --yaml <preflight.yaml> --csv <preflight.csv>
|
||||
python3 <skill-dir>/scripts/preflight.py prepare \
|
||||
--config <matrix.yaml> --yaml <preflight.yaml> --csv <preflight.csv>
|
||||
```
|
||||
|
||||
Read [references/schema.md](references/schema.md) for the input contract and
|
||||
[references/boundaries.md](references/boundaries.md) before wiring a real
|
||||
agent-runtime/deployer adapter. The included command-contract fixture is the
|
||||
safe way to test remote-node behavior without pretending that this machine is a
|
||||
Mac or that a deployment happened.
|
||||
|
||||
Overall status is `ready`, `blocked`, or `environment-fail`:
|
||||
|
||||
- `blocked` means a required declaration, artifact, binding, runner, fixture,
|
||||
cleanup contract, or authorization is absent/unsafe.
|
||||
- `environment-fail` means a declared environment exists but DNS/TCP/HTTP,
|
||||
device/toolchain, installation, launch, or deployment execution failed.
|
||||
- `ready` means every recorded check passed. A ready preflight permits the
|
||||
caller to continue with `integration`; this skill does not run test cases.
|
||||
|
||||
This project assumes the client has the required album picker, photo permission, and Chinese
|
||||
IME capabilities. Preflight does not probe or gate these capabilities, and does not read
|
||||
`AUTHORIZE_ALBUM` or `IME_PINYIN`. Their real behavior is exercised by journey cases; a failure
|
||||
there is a product test failure, not an environment block.
|
||||
|
||||
Do not treat a preflight report as a product result. A preflight failure means
|
||||
no client session may be created; the tester owns `not-run` propagation and must write
|
||||
one result row per selected testcase with `blocked_by: [test-preflight]`.
|
||||
@@ -0,0 +1,3 @@
|
||||
interface:
|
||||
display_name: "Test Preflight"
|
||||
short_description: "Verify or prepare tester environments"
|
||||
@@ -0,0 +1,4 @@
|
||||
# Reference moved
|
||||
|
||||
Read [schema.md](schema.md) for the matrix and evidence contract, and
|
||||
[boundaries.md](boundaries.md) for agent-runtime/deployer/builder integration.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Integration boundaries
|
||||
|
||||
`test-preflight` is an orchestration and evidence layer. It consumes existing
|
||||
skills instead of implementing their policy or storage formats.
|
||||
|
||||
| Boundary | Caller-supplied contract | What preflight does | What it must not do |
|
||||
| --- | --- | --- | --- |
|
||||
| agent-runtime | resolved `tester.<name>` binding: binding id, node, environment, project/path, and a node runner | validate that the binding is explicit and route node probes through the runner | parse Orca state, invent a host, or create/reuse a terminal |
|
||||
| deployer | `prepare` only: an argv command returning a JSON deployment receipt for `environment: test` | call it once and record its receipt/status | copy Compose/Argo logic, SSH directly, or accept production |
|
||||
| builder/obtain | caller-provided local artifact path (and optional remote path), hash, source commit, package/bundle identity | hash and inspect the exact file; pass it to install contract | build, guess an old artifact, or silently download/upload |
|
||||
|
||||
## Deployer command contract
|
||||
|
||||
`prepare` requires:
|
||||
|
||||
```yaml
|
||||
deployment:
|
||||
environment: test
|
||||
command: [python3, /path/to/existing-deployer-adapter.py]
|
||||
```
|
||||
|
||||
The command receives JSON on stdin:
|
||||
|
||||
```json
|
||||
{"operation":"prepare","environment":"test","project_root":"/abs/project","host":"...","public_host":"..."}
|
||||
```
|
||||
|
||||
It must return `{ "status": "ready", "observed": "..." }` (an optional
|
||||
`checks` mapping is recorded). Non-zero/invalid output is an
|
||||
`environment-fail`; absent command or a non-test environment is `blocked`.
|
||||
The adapter owns deployer-specific configuration, receipts, cleanup, and
|
||||
authorization.
|
||||
|
||||
## Device contract
|
||||
|
||||
The node runner owns remote execution. For a local node the implementation uses
|
||||
read-only `adb`/`xcrun simctl` checks in `verify`; `prepare` alone invokes
|
||||
`adb install`/launch or `simctl install`/launch. For a remote node the same
|
||||
operations are JSON requests to the runner. Missing xcrun on Linux, Android
|
||||
`unauthorized`/`offline`, a missing device, or a failed install/start is an
|
||||
`environment-fail`, not a product failure.
|
||||
|
||||
No client session is created by this skill. The runner's `install` and `start`
|
||||
operations are the only device mutations, and are never called by `verify`.
|
||||
@@ -0,0 +1,95 @@
|
||||
# test-preflight matrix schema
|
||||
|
||||
The input is YAML. `mode` is supplied by the CLI, so a file may be reused for
|
||||
`verify` and `prepare`.
|
||||
|
||||
```yaml
|
||||
schema_version: 1
|
||||
project_root: /absolute/project/root
|
||||
host: https://test.example.invalid
|
||||
public_host: https://public-test.example.invalid # optional
|
||||
service_probes:
|
||||
- id: api-health
|
||||
url: "{HOST}/health"
|
||||
expected_status: [200]
|
||||
fixtures:
|
||||
required: true
|
||||
references: [isolated-test-account]
|
||||
cleanup:
|
||||
required: true
|
||||
steps: [delete-run-data]
|
||||
# Optional project metadata. It is not a preflight gate.
|
||||
capabilities:
|
||||
assumed: [album_picker, photo_permission, ime_pinyin]
|
||||
matrix:
|
||||
- platform: android
|
||||
tester:
|
||||
binding: tester.android
|
||||
node: local
|
||||
runner: {kind: local}
|
||||
artifact:
|
||||
path: /absolute/dist/writes.apk
|
||||
sha256: 012345...
|
||||
package: page.writes
|
||||
source_commit: 40-character-commit-sha
|
||||
identity_command: [apkanalyzer, manifest, application-id, "{artifact}"]
|
||||
```
|
||||
|
||||
Required matrix fields are `platform` (`android` or `ios`), `tester.binding`,
|
||||
`tester.node`, and `artifact.path`. Artifact identity is required as
|
||||
`package` for Android or `bundle_id` for iOS. `sha256` and `source_commit` are
|
||||
optional but recommended; when present they are checked exactly.
|
||||
|
||||
`service_probes` must contain at least one URL. `{HOST}` and `{PUBLIC_HOST}`
|
||||
are expanded from the top-level values. Each probe is checked from the actual
|
||||
tester node as DNS, TCP, and HTTP. A status outside 2xx–4xx fails transport
|
||||
readiness; `expected_status` can require an exact allow-list. HTTP 401/403 can
|
||||
therefore be a valid reachability result when explicitly expected.
|
||||
|
||||
`fixtures` and `cleanup` are declarations, not commands. A required fixture
|
||||
needs non-empty `references`; a required cleanup needs non-empty `steps` or
|
||||
`run`. For a read-only run, use `required: false` with a non-empty `reason`.
|
||||
The skill never executes these values.
|
||||
|
||||
## Client capabilities
|
||||
|
||||
The project assumes the mobile test nodes provide the album picker, photo permission,
|
||||
test image, and Chinese IME. These are not preflight declarations or gates. Journey
|
||||
cases exercise the real UI; failures are recorded as product failures. `AUTHORIZE_ALBUM`
|
||||
and `IME_PINYIN` are not read.
|
||||
|
||||
## Tester-node contract
|
||||
|
||||
`tester.runner.kind: local` executes platform and network probes on the current
|
||||
node. It is only valid for `node: local`/`localhost`. A non-local node must use
|
||||
`kind: contract` and a non-empty argv `command`. The command receives one JSON
|
||||
request on stdin and returns one JSON object on stdout; no shell string is
|
||||
accepted.
|
||||
|
||||
Requests have `operation` (`platform_probe`, `network_probe`, `install`, or
|
||||
`start`), `platform`, `node`, and operation-specific fields. The response is:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ready",
|
||||
"observed": "short evidence",
|
||||
"checks": {
|
||||
"macos": {"status": "ready", "observed": "Darwin 24", "expected": "Darwin"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`checks` is optional; each key becomes a separate evidence row. A non-zero,
|
||||
invalid, or unavailable runner is `environment-fail`. The adapter is expected
|
||||
to execute on the resolved tester node, not on the coordinator. See
|
||||
[boundaries.md](boundaries.md).
|
||||
|
||||
## Evidence
|
||||
|
||||
YAML contains `mode`, overall `status`, timestamps, a summary, and `checks`.
|
||||
CSV uses these stable columns:
|
||||
|
||||
`check_id,mode,platform,node,target,observed,expected,status,remediation,timestamp`
|
||||
|
||||
Every check must use one of `ready`, `blocked`, or `environment-fail`; the
|
||||
report does not invent `fail` or `not-run` testcase results.
|
||||
@@ -0,0 +1,553 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the test-preflight verify/prepare contract."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import platform as host_platform
|
||||
import plistlib
|
||||
import re
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import zipfile
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError as exc: # pragma: no cover
|
||||
raise SystemExit("test-preflight 需要 PyYAML;请安装运行依赖后重试") from exc
|
||||
|
||||
|
||||
PLATFORMS = {"android", "ios"}
|
||||
STATUSES = {"ready", "blocked", "environment-fail"}
|
||||
EVIDENCE_FIELDS = [
|
||||
"check_id", "mode", "platform", "node", "target", "observed", "expected",
|
||||
"status", "remediation", "timestamp",
|
||||
]
|
||||
SHA256 = re.compile(r"^[0-9a-fA-F]{64}$")
|
||||
COMMIT = re.compile(r"^[0-9a-fA-F]{40}$")
|
||||
|
||||
|
||||
def now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def text(value: Any) -> str:
|
||||
if isinstance(value, (dict, list)):
|
||||
return json.dumps(value, ensure_ascii=False, sort_keys=True)
|
||||
return "" if value is None else str(value)
|
||||
|
||||
|
||||
def status_from(value: Any, default: str = "environment-fail") -> str:
|
||||
return value if value in STATUSES else default
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def command_text(command: list[str], *, payload: Mapping[str, Any] | None = None) -> tuple[int, str]:
|
||||
try:
|
||||
process = subprocess.run(
|
||||
command,
|
||||
input=(json.dumps(payload, ensure_ascii=False) if payload is not None else None),
|
||||
text=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
check=False,
|
||||
)
|
||||
except OSError as exc:
|
||||
return 127, str(exc)
|
||||
return process.returncode, process.stdout.strip()
|
||||
|
||||
|
||||
class Evidence:
|
||||
def __init__(self, mode: str) -> None:
|
||||
self.mode = mode
|
||||
self.started_at = now()
|
||||
self.rows: list[dict[str, str]] = []
|
||||
|
||||
def add(
|
||||
self,
|
||||
check_id: str,
|
||||
*,
|
||||
platform: str = "all",
|
||||
node: str = "coordinator",
|
||||
target: str = "",
|
||||
observed: Any = "",
|
||||
expected: Any = "",
|
||||
status: str,
|
||||
remediation: str = "",
|
||||
) -> None:
|
||||
if status not in STATUSES:
|
||||
raise ValueError(f"invalid evidence status: {status}")
|
||||
self.rows.append({
|
||||
"check_id": check_id,
|
||||
"mode": self.mode,
|
||||
"platform": platform,
|
||||
"node": node,
|
||||
"target": target,
|
||||
"observed": text(observed),
|
||||
"expected": text(expected),
|
||||
"status": status,
|
||||
"remediation": remediation,
|
||||
"timestamp": now(),
|
||||
})
|
||||
|
||||
def overall(self) -> str:
|
||||
statuses = {row["status"] for row in self.rows}
|
||||
if "environment-fail" in statuses:
|
||||
return "environment-fail"
|
||||
if "blocked" in statuses:
|
||||
return "blocked"
|
||||
return "ready"
|
||||
|
||||
def payload(self) -> dict[str, Any]:
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"mode": self.mode,
|
||||
"status": self.overall(),
|
||||
"started_at": self.started_at,
|
||||
"finished_at": now(),
|
||||
"summary": {state: sum(row["status"] == state for row in self.rows) for state in sorted(STATUSES)},
|
||||
"checks": self.rows,
|
||||
}
|
||||
|
||||
|
||||
def load_config(path: Path) -> dict[str, Any]:
|
||||
data = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("配置根必须是 mapping")
|
||||
return data
|
||||
|
||||
|
||||
def expand(value: str, config: Mapping[str, Any]) -> str:
|
||||
return value.replace("{HOST}", text(config.get("host"))).replace("{PUBLIC_HOST}", text(config.get("public_host")))
|
||||
|
||||
|
||||
def normalize_url(value: str) -> str:
|
||||
return value if urllib.parse.urlsplit(value).scheme else "http://" + value.lstrip("/")
|
||||
|
||||
|
||||
def argv(value: Any) -> list[str] | None:
|
||||
if not isinstance(value, list) or not value or not all(isinstance(item, str) and item for item in value):
|
||||
return None
|
||||
return value
|
||||
|
||||
|
||||
def validate_shape(config: Mapping[str, Any], evidence: Evidence) -> bool:
|
||||
ok = True
|
||||
if config.get("schema_version") != 1:
|
||||
evidence.add("config-schema", target="config", observed=config.get("schema_version"), expected="1", status="blocked", remediation="将 schema_version 设为 1")
|
||||
ok = False
|
||||
if not isinstance(config.get("project_root"), str) or not Path(text(config.get("project_root"))).is_absolute():
|
||||
evidence.add("config-project-root", target="project_root", observed=config.get("project_root"), expected="absolute project_root", status="blocked", remediation="提供项目根绝对路径")
|
||||
ok = False
|
||||
if not isinstance(config.get("host"), str) or not config["host"].strip():
|
||||
evidence.add("config-host", target="HOST", expected="non-empty host URL", status="blocked", remediation="提供 HOST")
|
||||
ok = False
|
||||
probes = config.get("service_probes")
|
||||
if not isinstance(probes, list) or not probes:
|
||||
evidence.add("config-service-probes", target="service_probes", expected="at least one probe", status="blocked", remediation="声明服务 URL 探针")
|
||||
ok = False
|
||||
else:
|
||||
for index, probe in enumerate(probes):
|
||||
if not isinstance(probe, dict) or not isinstance(probe.get("url"), str) or not probe["url"].strip():
|
||||
evidence.add(f"config-service-probe-{index}", target=f"service_probes[{index}]", expected="url", status="blocked", remediation="为每个服务探针提供 URL")
|
||||
ok = False
|
||||
matrix = config.get("matrix")
|
||||
if not isinstance(matrix, list) or not matrix:
|
||||
evidence.add("config-matrix", target="matrix", expected="non-empty list", status="blocked", remediation="声明至少一个平台矩阵行")
|
||||
return False
|
||||
seen: set[str] = set()
|
||||
for index, row in enumerate(matrix):
|
||||
prefix = f"matrix-{index}"
|
||||
if not isinstance(row, dict):
|
||||
evidence.add(prefix, target=f"matrix[{index}]", expected="mapping", status="blocked", remediation="修正矩阵行")
|
||||
ok = False
|
||||
continue
|
||||
platform_name = text(row.get("platform")).lower()
|
||||
if platform_name not in PLATFORMS or platform_name in seen:
|
||||
evidence.add(f"{prefix}-platform", platform=platform_name or "unknown", target=f"matrix[{index}].platform", expected="unique android or ios", status="blocked", remediation="为每个平台保留唯一矩阵行")
|
||||
ok = False
|
||||
seen.add(platform_name)
|
||||
tester = row.get("tester")
|
||||
if not isinstance(tester, dict) or not isinstance(tester.get("binding"), str) or not tester["binding"].startswith("tester."):
|
||||
evidence.add(f"{prefix}-binding", platform=platform_name, target=f"matrix[{index}].tester.binding", expected="tester.<name>", status="blocked", remediation="由 agent-runtime 解析并传入 tester binding")
|
||||
ok = False
|
||||
if not isinstance(tester, dict) or not isinstance(tester.get("node"), str) or not tester["node"].strip():
|
||||
evidence.add(f"{prefix}-node", platform=platform_name, target=f"matrix[{index}].tester.node", expected="actual tester node", status="blocked", remediation="提供实际 tester 节点名")
|
||||
ok = False
|
||||
artifact = row.get("artifact")
|
||||
identity_key = "package" if platform_name == "android" else "bundle_id"
|
||||
if not isinstance(artifact, dict) or not isinstance(artifact.get("path"), str) or not artifact["path"].strip():
|
||||
evidence.add(f"{prefix}-artifact", platform=platform_name, target=f"matrix[{index}].artifact.path", expected="caller-provided artifact path", status="blocked", remediation="提供本轮 APK/IPA,不要让 preflight 猜产物")
|
||||
ok = False
|
||||
if not isinstance(artifact, dict) or not isinstance(artifact.get(identity_key), str) or not artifact[identity_key].strip():
|
||||
evidence.add(f"{prefix}-identity", platform=platform_name, target=f"matrix[{index}].artifact.{identity_key}", expected="declared artifact identity", status="blocked", remediation=f"声明 artifact.{identity_key}")
|
||||
ok = False
|
||||
if isinstance(artifact, dict) and artifact.get("sha256") is not None and not SHA256.fullmatch(text(artifact.get("sha256"))):
|
||||
evidence.add(f"{prefix}-sha256", platform=platform_name, target=f"matrix[{index}].artifact.sha256", expected="64 hex characters", status="blocked", remediation="提供正确 SHA-256 或删除字段")
|
||||
ok = False
|
||||
if isinstance(artifact, dict) and artifact.get("source_commit") is not None and not COMMIT.fullmatch(text(artifact.get("source_commit"))):
|
||||
evidence.add(f"{prefix}-commit", platform=platform_name, target=f"matrix[{index}].artifact.source_commit", expected="40-character commit SHA", status="blocked", remediation="提供产物来源 commit SHA")
|
||||
ok = False
|
||||
return ok
|
||||
|
||||
|
||||
def artifact_identity(path: Path, platform_name: str, artifact: Mapping[str, Any]) -> tuple[str | None, str]:
|
||||
if platform_name == "ios":
|
||||
try:
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
names = [name for name in archive.namelist() if re.fullmatch(r"Payload/[^/]+\.app/Info\.plist", name)]
|
||||
if len(names) != 1:
|
||||
return None, "IPA 中没有唯一 Payload/*.app/Info.plist"
|
||||
plist = plistlib.loads(archive.read(names[0]))
|
||||
value = plist.get("CFBundleIdentifier")
|
||||
return (value, "Info.plist CFBundleIdentifier") if isinstance(value, str) else (None, "Info.plist 缺少 CFBundleIdentifier")
|
||||
except (OSError, zipfile.BadZipFile, plistlib.InvalidFileException) as exc:
|
||||
return None, f"IPA 读取失败: {exc}"
|
||||
command = argv(artifact.get("identity_command"))
|
||||
if command:
|
||||
command = [item.replace("{artifact}", str(path)) for item in command]
|
||||
else:
|
||||
tool = shutil.which("apkanalyzer") or shutil.which("aapt2") or shutil.which("aapt")
|
||||
if not tool:
|
||||
return None, "缺少 apkanalyzer/aapt2/aapt;无法读取 APK package identity"
|
||||
command = [tool, "manifest", "application-id", str(path)] if tool.endswith("apkanalyzer") else [tool, "dump", "badging", str(path)]
|
||||
code, output = command_text(command)
|
||||
if code != 0:
|
||||
return None, output[-500:] or f"identity command exited {code}"
|
||||
match = re.search(r"(?:application-id|name)\s*[:=]\s*['\"]?([A-Za-z0-9_.]+)", output)
|
||||
if not match:
|
||||
match = re.search(r"package:\s+name='([^']+)'", output)
|
||||
if not match:
|
||||
match = re.fullmatch(r"\s*([A-Za-z0-9_.]+)\s*", output)
|
||||
return (match.group(1), output[-500:]) if match else (None, "identity command output did not contain package")
|
||||
|
||||
|
||||
def record_artifact(evidence: Evidence, row: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
|
||||
artifact = row.get("artifact") or {}
|
||||
path = Path(text(artifact.get("path"))).expanduser()
|
||||
prefix = f"{platform_name}-{index}-artifact"
|
||||
if not path.is_file():
|
||||
evidence.add(prefix, platform=platform_name, node=node, target=str(path), expected="existing caller-provided artifact", status="blocked", remediation="补齐 APK/IPA 路径;不要默认构建")
|
||||
return False
|
||||
actual_hash = sha256(path)
|
||||
expected_hash = artifact.get("sha256")
|
||||
if expected_hash and actual_hash.lower() != text(expected_hash).lower():
|
||||
evidence.add(prefix + "-hash", platform=platform_name, node=node, target=str(path), observed=actual_hash, expected=expected_hash, status="blocked", remediation="确认产物路径或 SHA-256 与本轮构建身份一致")
|
||||
return False
|
||||
evidence.add(prefix + "-hash", platform=platform_name, node=node, target=str(path), observed=actual_hash, expected=expected_hash or "file SHA-256 recorded", status="ready")
|
||||
identity, observed = artifact_identity(path, platform_name, artifact)
|
||||
expected_identity = text(artifact.get("package" if platform_name == "android" else "bundle_id"))
|
||||
if identity != expected_identity:
|
||||
evidence.add(prefix + "-identity", platform=platform_name, node=node, target=str(path), observed=identity or observed, expected=expected_identity, status="environment-fail", remediation="使用与 catalog 身份一致的 APK/IPA,并重新校验产物")
|
||||
return False
|
||||
evidence.add(prefix + "-identity", platform=platform_name, node=node, target=str(path), observed=identity, expected=expected_identity, status="ready")
|
||||
if artifact.get("source_commit"):
|
||||
evidence.add(prefix + "-commit", platform=platform_name, node=node, target=str(path), observed=artifact.get("source_commit"), expected=artifact.get("source_commit"), status="ready")
|
||||
return True
|
||||
|
||||
|
||||
def local_platform_probe(platform_name: str) -> tuple[str, dict[str, dict[str, Any]]]:
|
||||
if platform_name == "android":
|
||||
adb = shutil.which("adb")
|
||||
if not adb:
|
||||
return "environment-fail", {"adb": {"status": "environment-fail", "observed": "missing", "expected": "Android SDK platform-tools", "remediation": "在 tester.android 节点安装并配置 adb"}}
|
||||
code, output = command_text([adb, "devices", "-l"])
|
||||
states = [line.split() for line in output.splitlines()[1:] if line.strip()]
|
||||
device = next((parts[0] for parts in states if len(parts) > 1 and parts[1] == "device"), "")
|
||||
bad = next((parts for parts in states if len(parts) > 1 and parts[1] in {"unauthorized", "offline"}), None)
|
||||
if bad:
|
||||
return "environment-fail", {"adb": {"status": "ready", "observed": output, "expected": "adb responds"}, "device": {"status": "environment-fail", "observed": " ".join(bad), "expected": "authorized device", "remediation": "解锁 Android 设备并接受 RSA 授权;offline 时重启 adb/设备"}}
|
||||
if code != 0 or not device:
|
||||
return "environment-fail", {"adb": {"status": "environment-fail", "observed": output, "expected": "one authorized device", "remediation": "连接并授权 Android device/emulator"}}
|
||||
boot_code, boot = command_text([adb, "-s", device, "shell", "getprop", "sys.boot_completed"])
|
||||
booted = boot_code == 0 and boot.strip() == "1"
|
||||
return ("ready" if booted else "environment-fail", {
|
||||
"adb": {"status": "ready", "observed": adb, "expected": "adb available"},
|
||||
"device": {"status": "ready", "observed": device, "expected": "authorized device"},
|
||||
"boot": {"status": "ready" if booted else "environment-fail", "observed": boot, "expected": "sys.boot_completed=1", "remediation": "等待 emulator boot complete 后重跑"},
|
||||
})
|
||||
if host_platform.system() != "Darwin":
|
||||
return "environment-fail", {"macos": {"status": "environment-fail", "observed": host_platform.system(), "expected": "Darwin/macOS tester.ios node", "remediation": "将 tester.ios 指向可用 macOS,不要在 Linux 伪装 iOS"}}
|
||||
xcrun = shutil.which("xcrun")
|
||||
if not xcrun:
|
||||
return "environment-fail", {"xcrun": {"status": "environment-fail", "observed": "missing", "expected": "xcrun and simctl", "remediation": "在 Mac 安装 Xcode Command Line Tools/Xcode"}}
|
||||
code, output = command_text([xcrun, "simctl", "list", "devices", "available"])
|
||||
booted = any("(Booted)" in line for line in output.splitlines())
|
||||
return ("ready" if code == 0 and booted else "environment-fail", {
|
||||
"macos": {"status": "ready", "observed": "Darwin", "expected": "Darwin/macOS"},
|
||||
"xcrun": {"status": "ready" if code == 0 else "environment-fail", "observed": output[-500:], "expected": "simctl available", "remediation": "安装 Xcode 并接受许可"},
|
||||
"boot": {"status": "ready" if booted else "environment-fail", "observed": "Booted" if booted else output[-300:], "expected": "Booted simulator", "remediation": "启动目标 Simulator 后重跑"},
|
||||
})
|
||||
|
||||
|
||||
def runner_call(runner: Mapping[str, Any], request: Mapping[str, Any]) -> tuple[str, str, dict[str, Any]]:
|
||||
command = argv(runner.get("command"))
|
||||
if not command:
|
||||
return "blocked", "runner command missing", {}
|
||||
code, output = command_text(command, payload=request)
|
||||
if code != 0:
|
||||
return "environment-fail", output[-800:], {}
|
||||
try:
|
||||
result = json.loads(output)
|
||||
except json.JSONDecodeError:
|
||||
return "environment-fail", output[-800:], {}
|
||||
if not isinstance(result, dict):
|
||||
return "environment-fail", "runner returned non-object", {}
|
||||
return status_from(result.get("status")), text(result.get("observed", "")), result
|
||||
|
||||
|
||||
def add_runner_checks(evidence: Evidence, result: Mapping[str, Any], *, prefix: str, platform_name: str, node: str, default_expected: str) -> str:
|
||||
checks = result.get("checks")
|
||||
if isinstance(checks, dict) and checks:
|
||||
statuses: list[str] = []
|
||||
for name, detail in checks.items():
|
||||
detail = detail if isinstance(detail, dict) else {"observed": detail}
|
||||
status = status_from(detail.get("status"), status_from(result.get("status")))
|
||||
statuses.append(status)
|
||||
evidence.add(f"{prefix}-{name}", platform=platform_name, node=node, target=text(detail.get("target", name)), observed=detail.get("observed", result.get("observed", "")), expected=detail.get("expected", default_expected), status=status, remediation=text(detail.get("remediation", "")))
|
||||
return "environment-fail" if "environment-fail" in statuses else ("blocked" if "blocked" in statuses else "ready")
|
||||
status = status_from(result.get("status"))
|
||||
evidence.add(prefix, platform=platform_name, node=node, target=prefix, observed=result.get("observed", ""), expected=default_expected, status=status, remediation=text(result.get("remediation", "")))
|
||||
return status
|
||||
|
||||
|
||||
def require_runner_checks(evidence: Evidence, result: Mapping[str, Any], *, prefix: str, platform_name: str, node: str, required: set[str]) -> bool:
|
||||
checks = result.get("checks")
|
||||
missing = sorted(required - set(checks)) if isinstance(checks, dict) else sorted(required)
|
||||
if not missing:
|
||||
return True
|
||||
evidence.add(prefix + "-contract", platform=platform_name, node=node, target="runner response", observed={"missing": missing}, expected=sorted(required), status="blocked", remediation="让 tester 节点 adapter 返回每个必需的 adb/boot、macOS/xcrun/simctl 或 DNS/TCP/HTTP 证据")
|
||||
return False
|
||||
|
||||
|
||||
def network_local(url: str) -> list[dict[str, Any]]:
|
||||
parsed = urllib.parse.urlsplit(url)
|
||||
host = parsed.hostname or ""
|
||||
port = parsed.port or (443 if parsed.scheme == "https" else 80)
|
||||
try:
|
||||
addresses = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
|
||||
except OSError as exc:
|
||||
return [{"name": "dns", "status": "environment-fail", "observed": str(exc), "expected": "resolvable hostname", "remediation": "检查 tester 节点 DNS/VPN/网络"}]
|
||||
rows: list[dict[str, Any]] = [{"name": "dns", "status": "ready", "observed": sorted({item[4][0] for item in addresses}), "expected": "resolvable hostname"}]
|
||||
try:
|
||||
with socket.create_connection((host, port), timeout=5):
|
||||
rows.append({"name": "tcp", "status": "ready", "observed": f"{host}:{port}", "expected": "TCP connection"})
|
||||
except OSError as exc:
|
||||
rows.append({"name": "tcp", "status": "environment-fail", "observed": str(exc), "expected": f"TCP {host}:{port}", "remediation": "检查端口、防火墙和测试节点到 HOST 的路由"})
|
||||
return rows
|
||||
try:
|
||||
with urllib.request.urlopen(urllib.request.Request(url, method="GET"), timeout=5) as response:
|
||||
code = response.status
|
||||
rows.append({"name": "http", "status": "ready" if 200 <= code < 500 else "environment-fail", "observed": code, "expected": "HTTP 2xx-4xx", "remediation": "检查后端健康状态" if code >= 500 else ""})
|
||||
except urllib.error.HTTPError as exc:
|
||||
rows.append({"name": "http", "status": "ready" if 200 <= exc.code < 500 else "environment-fail", "observed": exc.code, "expected": "HTTP 2xx-4xx", "remediation": "检查后端服务,不把 5xx 当成可用" if exc.code >= 500 else ""})
|
||||
except (OSError, urllib.error.URLError) as exc:
|
||||
rows.append({"name": "http", "status": "environment-fail", "observed": str(exc), "expected": "HTTP response", "remediation": "检查服务监听、代理、TLS 和节点网络"})
|
||||
return rows
|
||||
|
||||
|
||||
def check_network(evidence: Evidence, config: Mapping[str, Any], row: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
|
||||
tester = row.get("tester") or {}
|
||||
runner = tester.get("runner") if isinstance(tester, dict) else None
|
||||
is_local = node in {"local", "localhost", "127.0.0.1"}
|
||||
statuses: list[str] = []
|
||||
for probe_index, probe in enumerate(config.get("service_probes") or []):
|
||||
raw_url = text(probe.get("url"))
|
||||
if "{PUBLIC_HOST}" in raw_url and not text(config.get("public_host")).strip():
|
||||
evidence.add(f"{platform_name}-{index}-network-{probe_index}-public-host", platform=platform_name, node=node, target=raw_url, expected="PUBLIC_HOST configured", status="blocked", remediation="提供 PUBLIC_HOST 或改用 HOST 探针")
|
||||
statuses.append("blocked")
|
||||
continue
|
||||
url = normalize_url(expand(raw_url, config))
|
||||
if not urllib.parse.urlsplit(url).hostname:
|
||||
evidence.add(f"{platform_name}-{index}-network-{probe_index}-url", platform=platform_name, node=node, target=url, expected="valid probe URL", status="blocked", remediation="修正 service_probes URL 或 HOST/PUBLIC_HOST")
|
||||
statuses.append("blocked")
|
||||
continue
|
||||
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
|
||||
details = network_local(url)
|
||||
expected_statuses = probe.get("expected_status")
|
||||
for detail in details:
|
||||
if detail["name"] == "http" and isinstance(expected_statuses, list) and detail["status"] == "ready" and detail["observed"] not in expected_statuses:
|
||||
detail = {**detail, "status": "environment-fail", "expected": expected_statuses, "remediation": "检查服务探针的 expected_status 或后端响应"}
|
||||
status = status_from(detail["status"])
|
||||
statuses.append(status)
|
||||
evidence.add(f"{platform_name}-{index}-network-{probe_index}-{detail['name']}", platform=platform_name, node=node, target=url, observed=detail["observed"], expected=detail["expected"], status=status, remediation=detail.get("remediation", ""))
|
||||
elif isinstance(runner, dict) and runner.get("kind") == "contract":
|
||||
binding = text(tester.get("binding")) if isinstance(tester, dict) else ""
|
||||
status, observed, result = runner_call(runner, {"operation": "network_probe", "platform": platform_name, "node": node, "binding": binding, "url": url, "probe": probe})
|
||||
result = result or {"status": status, "observed": observed}
|
||||
complete = require_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-network-{probe_index}", platform_name=platform_name, node=node, required={"dns", "tcp", "http"})
|
||||
network_status = add_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-network-{probe_index}", platform_name=platform_name, node=node, default_expected="DNS/TCP/HTTP reachable from actual tester node")
|
||||
statuses.append(network_status if complete else "blocked")
|
||||
else:
|
||||
evidence.add(f"{platform_name}-{index}-network-{probe_index}", platform=platform_name, node=node, target=url, expected="node runner contract", status="blocked", remediation="远端 tester 必须提供 agent-runtime adapter runner")
|
||||
statuses.append("blocked")
|
||||
return "environment-fail" not in statuses and "blocked" not in statuses
|
||||
|
||||
|
||||
def check_fixture_cleanup(evidence: Evidence, config: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
|
||||
ok = True
|
||||
fixture = config.get("fixtures")
|
||||
valid_fixture = isinstance(fixture, dict) and (not fixture.get("required", True) or bool(fixture.get("references")))
|
||||
if not valid_fixture:
|
||||
evidence.add(f"{platform_name}-{index}-fixture", platform=platform_name, node=node, target="fixtures", expected="required fixture references", status="blocked", remediation="声明隔离测试账号/fixture;不要在运行中临时创建")
|
||||
ok = False
|
||||
else:
|
||||
evidence.add(f"{platform_name}-{index}-fixture", platform=platform_name, node=node, target="fixtures", observed=fixture.get("references"), expected="declared fixture references", status="ready")
|
||||
cleanup = config.get("cleanup")
|
||||
valid_noop = isinstance(cleanup, dict) and cleanup.get("required") is False and bool(text(cleanup.get("reason")).strip())
|
||||
valid_steps = isinstance(cleanup, dict) and (bool(cleanup.get("steps")) or bool(text(cleanup.get("run")).strip()))
|
||||
if not (valid_noop or valid_steps):
|
||||
evidence.add(f"{platform_name}-{index}-cleanup", platform=platform_name, node=node, target="cleanup", expected="steps/run or explicit read-only reason", status="blocked", remediation="声明 teardown/rollback;清理缺失时禁止开始测试")
|
||||
ok = False
|
||||
else:
|
||||
evidence.add(f"{platform_name}-{index}-cleanup", platform=platform_name, node=node, target="cleanup", observed=cleanup, expected="cleanup contract declared", status="ready")
|
||||
return ok
|
||||
|
||||
|
||||
def run(config: Mapping[str, Any], mode: str) -> Evidence:
|
||||
evidence = Evidence(mode)
|
||||
shape_ok = validate_shape(config, evidence)
|
||||
deployment_ok = True
|
||||
if mode == "prepare":
|
||||
deployment = config.get("deployment")
|
||||
environment = deployment.get("environment") if isinstance(deployment, dict) else None
|
||||
command = argv(deployment.get("command")) if isinstance(deployment, dict) else None
|
||||
if environment != "test":
|
||||
evidence.add("deployment-environment", target="deployment.environment", observed=environment, expected="test", status="blocked", remediation="prepare 只允许 deployer 的 test 环境")
|
||||
if not command:
|
||||
evidence.add("deployment-command", target="deployment.command", expected="existing deployer adapter command", status="blocked", remediation="提供既有 deployer 的 JSON command contract;不要在 skill 内实现部署")
|
||||
deployment_ok = False
|
||||
elif not command:
|
||||
evidence.add("deployment-command", target="deployment.command", expected="existing deployer adapter command", status="blocked", remediation="提供既有 deployer 的 JSON command contract;不要在 skill 内实现部署")
|
||||
deployment_ok = False
|
||||
else:
|
||||
code, output = command_text(command, payload={"operation": "prepare", "environment": "test", "project_root": config.get("project_root"), "host": config.get("host"), "public_host": config.get("public_host")})
|
||||
if code != 0:
|
||||
evidence.add("deployment", target="test", observed=output[-800:], expected="deployer receipt status=ready", status="environment-fail", remediation="按 deployer 返回证据修复测试后端,再重跑 prepare")
|
||||
deployment_ok = False
|
||||
else:
|
||||
try:
|
||||
receipt = json.loads(output)
|
||||
except json.JSONDecodeError:
|
||||
receipt = {}
|
||||
status = status_from(receipt.get("status"), "environment-fail") if isinstance(receipt, dict) else "environment-fail"
|
||||
evidence.add("deployment", target="test", observed=receipt.get("observed", output[-800:]) if isinstance(receipt, dict) else output[-800:], expected="deployer receipt status=ready", status=status, remediation="检查既有 deployer 的测试环境 receipt" if status != "ready" else "")
|
||||
deployment_ok = status == "ready"
|
||||
if not shape_ok:
|
||||
return evidence
|
||||
for index, row in enumerate(config["matrix"]):
|
||||
platform_name = text(row.get("platform")).lower()
|
||||
tester = row.get("tester") or {}
|
||||
node = text(tester.get("node"))
|
||||
binding = text(tester.get("binding"))
|
||||
evidence.add(f"{platform_name}-{index}-binding", platform=platform_name, node=node, target=binding, observed=binding, expected="resolved tester.<name> binding", status="ready")
|
||||
artifact_ok = record_artifact(evidence, row, platform_name, node, index)
|
||||
fixture_ok = check_fixture_cleanup(evidence, config, platform_name, node, index)
|
||||
runner = tester.get("runner") if isinstance(tester, dict) else None
|
||||
is_local = node in {"local", "localhost", "127.0.0.1"}
|
||||
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
|
||||
platform_status, details = local_platform_probe(platform_name)
|
||||
for name, detail in details.items():
|
||||
evidence.add(f"{platform_name}-{index}-platform-{name}", platform=platform_name, node=node, target=name, observed=detail.get("observed", ""), expected=detail.get("expected", "platform ready"), status=status_from(detail.get("status"), platform_status), remediation=detail.get("remediation", ""))
|
||||
platform_ok = platform_status == "ready"
|
||||
elif isinstance(runner, dict) and runner.get("kind") == "contract":
|
||||
status, observed, result = runner_call(runner, {"operation": "platform_probe", "platform": platform_name, "node": node, "binding": binding})
|
||||
result = result or {"status": status, "observed": observed}
|
||||
required = {"adb", "device", "boot"} if platform_name == "android" else {"macos", "xcrun", "simctl", "boot"}
|
||||
complete = require_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-platform", platform_name=platform_name, node=node, required=required)
|
||||
platform_status = add_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-platform", platform_name=platform_name, node=node, default_expected="platform/device ready")
|
||||
platform_ok = complete and platform_status == "ready"
|
||||
else:
|
||||
evidence.add(f"{platform_name}-{index}-platform-runner", platform=platform_name, node=node, target=node, expected="node runner contract", status="blocked", remediation="远端 tester 必须由 agent-runtime 提供可执行 runner")
|
||||
platform_ok = False
|
||||
network_ok = check_network(evidence, config, row, platform_name, node, index)
|
||||
# This isolated project assumes the client capabilities used by its
|
||||
# journeys. Their real behavior is a product assertion, not a preflight
|
||||
# environment gate.
|
||||
row_ok = artifact_ok and fixture_ok and platform_ok and network_ok
|
||||
if mode == "prepare":
|
||||
artifact = row.get("artifact") or {}
|
||||
identity = artifact.get("package" if platform_name == "android" else "bundle_id")
|
||||
target_path = text(artifact.get("remote_path") or artifact.get("path"))
|
||||
if not row_ok or not deployment_ok:
|
||||
evidence.add(f"{platform_name}-{index}-install", platform=platform_name, node=node, target=target_path, observed="not-run", expected="verify and deployment ready", status="blocked", remediation="先修复前置检查;prepare 不伪造安装成功")
|
||||
evidence.add(f"{platform_name}-{index}-start", platform=platform_name, node=node, target=text(identity), observed="not-run", expected="install ready", status="blocked", remediation="安装前置失败,未启动 app")
|
||||
else:
|
||||
install_request = {"operation": "install", "platform": platform_name, "node": node, "artifact": target_path, "identity": identity}
|
||||
start_request = {"operation": "start", "platform": platform_name, "node": node, "identity": identity}
|
||||
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
|
||||
if platform_name == "android":
|
||||
adb = shutil.which("adb") or "adb"
|
||||
serial = os.environ.get("ANDROID_SERIAL", "")
|
||||
install_code, install_output = command_text([adb, "-s", serial, "install", "-r", target_path] if serial else [adb, "install", "-r", target_path])
|
||||
install_result = {"status": "ready" if install_code == 0 else "environment-fail", "observed": install_output}
|
||||
launch = [adb, "-s", serial, "shell", "monkey", "-p", text(identity), "1"] if serial else [adb, "shell", "monkey", "-p", text(identity), "1"]
|
||||
start_code, start_output = command_text(launch) if install_code == 0 else (1, "install failed")
|
||||
start_result = {"status": "ready" if start_code == 0 else "environment-fail", "observed": start_output}
|
||||
else:
|
||||
install_result = {"status": "environment-fail", "observed": "local iOS install requires a macOS runner", "remediation": "use tester.ios contract runner on Mac"}
|
||||
start_result = {"status": "blocked", "observed": "not-run", "remediation": "install failed"}
|
||||
add_runner_checks(evidence, install_result, prefix=f"{platform_name}-{index}-install", platform_name=platform_name, node=node, default_expected="artifact installed")
|
||||
add_runner_checks(evidence, start_result, prefix=f"{platform_name}-{index}-start", platform_name=platform_name, node=node, default_expected="app launched")
|
||||
elif isinstance(runner, dict) and runner.get("kind") == "contract":
|
||||
install_status, install_observed, install_result = runner_call(runner, install_request)
|
||||
add_runner_checks(evidence, install_result or {"status": install_status, "observed": install_observed}, prefix=f"{platform_name}-{index}-install", platform_name=platform_name, node=node, default_expected="artifact installed")
|
||||
if install_status == "ready":
|
||||
start_status, start_observed, start_result = runner_call(runner, start_request)
|
||||
add_runner_checks(evidence, start_result or {"status": start_status, "observed": start_observed}, prefix=f"{platform_name}-{index}-start", platform_name=platform_name, node=node, default_expected="app launched")
|
||||
else:
|
||||
evidence.add(f"{platform_name}-{index}-start", platform=platform_name, node=node, target=text(identity), observed="not-run", expected="install ready", status="blocked", remediation="安装失败,先修复 tester 节点")
|
||||
return evidence
|
||||
|
||||
|
||||
def write_evidence(evidence: Evidence, yaml_path: Path, csv_path: Path) -> dict[str, Any]:
|
||||
payload = evidence.payload()
|
||||
yaml_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
csv_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
yaml_path.write_text(yaml.safe_dump(payload, allow_unicode=True, sort_keys=False), encoding="utf-8")
|
||||
with csv_path.open("w", newline="", encoding="utf-8") as handle:
|
||||
writer = csv.DictWriter(handle, fieldnames=EVIDENCE_FIELDS)
|
||||
writer.writeheader()
|
||||
writer.writerows(evidence.rows)
|
||||
return {"status": payload["status"], "yaml": str(yaml_path), "csv": str(csv_path), "checks": len(evidence.rows), "summary": payload["summary"]}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Verify or prepare a test environment and emit YAML/CSV evidence")
|
||||
parser.add_argument("mode", choices=("verify", "prepare"))
|
||||
parser.add_argument("--config", required=True, type=Path)
|
||||
parser.add_argument("--yaml", dest="yaml_path", type=Path, default=Path("preflight.yaml"))
|
||||
parser.add_argument("--csv", dest="csv_path", type=Path, default=Path("preflight.csv"))
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
config = load_config(args.config)
|
||||
evidence = run(config, args.mode)
|
||||
result = write_evidence(evidence, args.yaml_path, args.csv_path)
|
||||
except (OSError, ValueError, yaml.YAMLError) as exc:
|
||||
print(json.dumps({"status": "blocked", "error": str(exc)}, ensure_ascii=False), file=sys.stderr)
|
||||
return 2
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
return 0 if result["status"] == "ready" else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,75 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic runner fixture for test-preflight contract tests."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
request = json.load(sys.stdin)
|
||||
scenario = os.environ.get("PREFLIGHT_RUNNER_SCENARIO", "ready")
|
||||
operation = request.get("operation")
|
||||
|
||||
if operation in {"deployment", "prepare"}:
|
||||
print(json.dumps({"status": "ready", "observed": "mock test deploy receipt"}))
|
||||
elif operation == "platform_probe":
|
||||
platform = request.get("platform")
|
||||
if scenario == "android-unauthorized" and platform == "android":
|
||||
print(json.dumps({"status": "environment-fail", "checks": {
|
||||
"adb": {"status": "ready", "observed": "adb 35", "expected": "adb responds"},
|
||||
"device": {"status": "environment-fail", "observed": "emulator-5554 unauthorized", "expected": "authorized device", "remediation": "accept RSA authorization"},
|
||||
"boot": {"status": "environment-fail", "observed": "unknown", "expected": "boot complete"},
|
||||
}}))
|
||||
elif scenario == "mac-unavailable" and platform == "ios":
|
||||
print(json.dumps({"status": "environment-fail", "checks": {
|
||||
"macos": {"status": "environment-fail", "observed": "Mac unavailable", "expected": "Darwin tester node", "remediation": "restore tester.ios Mac"},
|
||||
}}))
|
||||
else:
|
||||
print(json.dumps({"status": "ready", "checks": {
|
||||
"adb": {"status": "ready", "observed": "adb 35", "expected": "adb responds"},
|
||||
"device": {"status": "ready", "observed": "authorized", "expected": "authorized device"},
|
||||
"boot": {"status": "ready", "observed": "boot complete", "expected": "boot complete"},
|
||||
"macos": {"status": "ready", "observed": "Darwin", "expected": "Darwin"},
|
||||
"xcrun": {"status": "ready", "observed": "xcrun", "expected": "xcrun"},
|
||||
"simctl": {"status": "ready", "observed": "available", "expected": "simctl"},
|
||||
}}))
|
||||
elif operation == "network_probe":
|
||||
if scenario == "host-unreachable":
|
||||
print(json.dumps({"status": "environment-fail", "checks": {
|
||||
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
|
||||
"tcp": {"status": "environment-fail", "observed": "connection refused", "expected": "TCP reachable", "remediation": "check HOST route"},
|
||||
}}))
|
||||
elif scenario == "http-failure":
|
||||
print(json.dumps({"status": "environment-fail", "checks": {
|
||||
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
|
||||
"tcp": {"status": "ready", "observed": "443 open", "expected": "TCP reachable"},
|
||||
"http": {"status": "environment-fail", "observed": 503, "expected": "HTTP 2xx-4xx", "remediation": "check service health"},
|
||||
}}))
|
||||
else:
|
||||
print(json.dumps({"status": "ready", "checks": {
|
||||
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
|
||||
"tcp": {"status": "ready", "observed": "443 open", "expected": "TCP reachable"},
|
||||
"http": {"status": "ready", "observed": 200, "expected": "HTTP 2xx-4xx"},
|
||||
}}))
|
||||
elif operation == "capability_probe":
|
||||
capability = request.get("capability")
|
||||
checks = {
|
||||
"album_picker": {
|
||||
"picker": {"status": "ready", "observed": "real picker opened and returned image-001"},
|
||||
},
|
||||
"photo_permission": {
|
||||
"permission": {"status": "ready", "observed": "permission state checked on device"},
|
||||
"test_image": {"status": "ready", "observed": "image-001 available in test album"},
|
||||
},
|
||||
"ime_pinyin": {
|
||||
"composition": {"status": "ready", "observed": "nihao composed in Chinese IME"},
|
||||
"candidates": {"status": "ready", "observed": "你好 candidate selected"},
|
||||
},
|
||||
}
|
||||
if scenario == "capability-missing" and capability == "album_picker":
|
||||
print(json.dumps({"status": "ready", "checks": {}}))
|
||||
else:
|
||||
print(json.dumps({"status": "ready", "checks": checks.get(capability, {})}))
|
||||
elif operation in {"install", "start"}:
|
||||
print(json.dumps({"status": "ready", "observed": operation + "ed mock artifact"}))
|
||||
else:
|
||||
print(json.dumps({"status": "environment-fail", "observed": "unknown operation"}))
|
||||
@@ -0,0 +1,225 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import plistlib
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "scripts"))
|
||||
import preflight # noqa: E402
|
||||
|
||||
|
||||
FIXTURE_RUNNER = Path(__file__).resolve().parent / "fixtures" / "contract_runner.py"
|
||||
|
||||
|
||||
class PreflightTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.temp.name)
|
||||
self.apk = self.root / "writes.apk"
|
||||
self.apk.write_bytes(b"mock apk")
|
||||
self.ipa = self.root / "writes.ipa"
|
||||
with zipfile.ZipFile(self.ipa, "w") as archive:
|
||||
archive.writestr("Payload/Writes.app/Info.plist", plistlib.dumps({"CFBundleIdentifier": "page.writes"}))
|
||||
self.identity = self.root / "identity.py"
|
||||
self.identity.write_text("print('page.writes')\n", encoding="utf-8")
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.temp.cleanup()
|
||||
|
||||
def runner(self, scenario: str = "ready") -> dict[str, object]:
|
||||
return {"kind": "contract", "command": [sys.executable, str(FIXTURE_RUNNER)]}
|
||||
|
||||
def config(self, *, platform: str = "android", artifact: Path | None = None, node: str = "tester.android", scenario: str = "ready") -> dict[str, object]:
|
||||
artifact = artifact or self.apk
|
||||
artifact_row: dict[str, object] = {
|
||||
"path": str(artifact),
|
||||
"package": "page.writes" if platform == "android" else None,
|
||||
"bundle_id": "page.writes" if platform == "ios" else None,
|
||||
}
|
||||
if platform == "android":
|
||||
artifact_row["identity_command"] = [sys.executable, str(self.identity)]
|
||||
artifact_row.pop("bundle_id")
|
||||
else:
|
||||
artifact_row.pop("package")
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"project_root": str(self.root),
|
||||
"host": "https://test.example.invalid",
|
||||
"public_host": "https://public.example.invalid",
|
||||
"service_probes": [{"id": "health", "url": "{HOST}/health", "expected_status": [200]}],
|
||||
"fixtures": {"required": True, "references": ["isolated-account"]},
|
||||
"cleanup": {"required": True, "steps": ["cleanup-run-data"]},
|
||||
"matrix": [{
|
||||
"platform": platform,
|
||||
"tester": {"binding": "tester.android" if platform == "android" else "tester.ios", "node": node, "runner": self.runner(scenario)},
|
||||
"artifact": artifact_row,
|
||||
}],
|
||||
}
|
||||
|
||||
def write_config(self, data: dict[str, object]) -> Path:
|
||||
path = self.root / "matrix.yaml"
|
||||
import yaml
|
||||
path.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8")
|
||||
return path
|
||||
|
||||
def invoke(self, mode: str, data: dict[str, object]) -> tuple[int, dict[str, object], dict[str, object]]:
|
||||
config = self.write_config(data)
|
||||
yaml_path, csv_path = self.root / f"{mode}.yaml", self.root / f"{mode}.csv"
|
||||
env = os.environ.copy()
|
||||
env["PREFLIGHT_RUNNER_SCENARIO"] = "ready"
|
||||
process = subprocess.run([
|
||||
sys.executable, str(ROOT / "scripts" / "preflight.py"), mode,
|
||||
"--config", str(config), "--yaml", str(yaml_path), "--csv", str(csv_path),
|
||||
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||
import yaml
|
||||
return process.returncode, yaml.safe_load(yaml_path.read_text(encoding="utf-8")), {"stdout": process.stdout, "stderr": process.stderr}
|
||||
|
||||
def test_missing_artifact_is_blocked_and_report_has_contract_columns(self):
|
||||
data = self.config(artifact=self.root / "missing.apk")
|
||||
code, report, _ = self.invoke("verify", data)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertEqual(report["status"], "blocked")
|
||||
row = next(row for row in report["checks"] if row["check_id"] == "android-0-artifact")
|
||||
self.assertEqual(row["status"], "blocked")
|
||||
self.assertEqual(set(row), set(preflight.EVIDENCE_FIELDS))
|
||||
|
||||
def test_android_unauthorized_is_environment_fail(self):
|
||||
data = self.config()
|
||||
config = self.write_config(data)
|
||||
out = self.root / "unauthorized.yaml"
|
||||
env = os.environ.copy()
|
||||
env["PREFLIGHT_RUNNER_SCENARIO"] = "android-unauthorized"
|
||||
process = subprocess.run([
|
||||
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config), "--yaml", str(out), "--csv", str(self.root / "u.csv")
|
||||
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||
report = json.loads(json.dumps(__import__("yaml").safe_load(out.read_text(encoding="utf-8"))))
|
||||
self.assertEqual(process.returncode, 1)
|
||||
self.assertEqual(report["status"], "environment-fail")
|
||||
self.assertEqual(next(row for row in report["checks"] if row["check_id"].endswith("-device"))["status"], "environment-fail")
|
||||
|
||||
def test_linux_without_xcrun_is_environment_fail(self):
|
||||
data = self.config(platform="ios", artifact=self.ipa, node="local")
|
||||
data["matrix"][0]["tester"]["runner"] = {"kind": "local"}
|
||||
with mock.patch.object(preflight.host_platform, "system", return_value="Linux"), mock.patch.object(preflight, "network_local", return_value=[{"name": "dns", "status": "ready", "observed": "ok", "expected": "DNS"}, {"name": "tcp", "status": "ready", "observed": "ok", "expected": "TCP"}, {"name": "http", "status": "ready", "observed": 200, "expected": "HTTP"}]):
|
||||
result = preflight.run(data, "verify")
|
||||
self.assertEqual(result.overall(), "environment-fail")
|
||||
row = next(row for row in result.rows if row["check_id"].endswith("-platform-macos"))
|
||||
self.assertEqual(row["status"], "environment-fail")
|
||||
|
||||
def test_mac_unavailable_contract_is_environment_fail(self):
|
||||
data = self.config(platform="ios", artifact=self.ipa)
|
||||
config_path = self.write_config(data)
|
||||
env = os.environ.copy()
|
||||
env["PREFLIGHT_RUNNER_SCENARIO"] = "mac-unavailable"
|
||||
output = self.root / "mac.yaml"
|
||||
process = subprocess.run([
|
||||
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "mac.csv")
|
||||
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||
import yaml
|
||||
report = yaml.safe_load(output.read_text(encoding="utf-8"))
|
||||
self.assertEqual(process.returncode, 1)
|
||||
self.assertEqual(report["status"], "environment-fail")
|
||||
self.assertTrue(any(row["check_id"].endswith("-macos") and row["status"] == "environment-fail" for row in report["checks"]))
|
||||
|
||||
def test_missing_fixture_or_cleanup_is_blocked(self):
|
||||
data = self.config()
|
||||
data["fixtures"] = {"required": True, "references": []}
|
||||
data["cleanup"] = {"required": True, "steps": []}
|
||||
result = preflight.run(data, "verify")
|
||||
self.assertEqual(result.overall(), "blocked")
|
||||
self.assertTrue(any(row["check_id"].endswith("-fixture") and row["status"] == "blocked" for row in result.rows))
|
||||
self.assertTrue(any(row["check_id"].endswith("-cleanup") and row["status"] == "blocked" for row in result.rows))
|
||||
|
||||
def test_mobile_capabilities_are_assumed_and_not_a_preflight_gate(self):
|
||||
data = self.config()
|
||||
config_path = self.write_config(data)
|
||||
env = os.environ.copy()
|
||||
env["PREFLIGHT_RUNNER_SCENARIO"] = "capability-missing"
|
||||
output = self.root / "capabilities.yaml"
|
||||
process = subprocess.run([
|
||||
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify",
|
||||
"--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "capabilities.csv"),
|
||||
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||
report = __import__("yaml").safe_load(output.read_text(encoding="utf-8"))
|
||||
self.assertEqual(process.returncode, 0)
|
||||
self.assertEqual(report["status"], "ready")
|
||||
self.assertFalse(any("capability-" in row["check_id"] for row in report["checks"]))
|
||||
|
||||
def test_host_unreachable_from_actual_runner_is_environment_fail(self):
|
||||
data = self.config()
|
||||
config_path = self.write_config(data)
|
||||
env = os.environ.copy()
|
||||
env["PREFLIGHT_RUNNER_SCENARIO"] = "host-unreachable"
|
||||
output = self.root / "host.yaml"
|
||||
process = subprocess.run([
|
||||
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "host.csv")
|
||||
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||
import yaml
|
||||
report = yaml.safe_load(output.read_text(encoding="utf-8"))
|
||||
self.assertEqual(process.returncode, 1)
|
||||
self.assertEqual(report["status"], "environment-fail")
|
||||
self.assertTrue(any("network" in row["check_id"] and row["status"] == "environment-fail" for row in report["checks"]))
|
||||
|
||||
def test_http_500_is_environment_fail_not_product_fail(self):
|
||||
data = self.config()
|
||||
config_path = self.write_config(data)
|
||||
env = os.environ.copy()
|
||||
env["PREFLIGHT_RUNNER_SCENARIO"] = "http-failure"
|
||||
output = self.root / "http.yaml"
|
||||
process = subprocess.run([
|
||||
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "http.csv")
|
||||
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||
import yaml
|
||||
report = yaml.safe_load(output.read_text(encoding="utf-8"))
|
||||
self.assertEqual(process.returncode, 1)
|
||||
http_row = next(row for row in report["checks"] if row["check_id"].endswith("-http"))
|
||||
self.assertEqual(http_row["observed"], "503")
|
||||
self.assertEqual(http_row["status"], "environment-fail")
|
||||
|
||||
def test_prepare_without_deployer_is_blocked_and_never_installs(self):
|
||||
data = self.config()
|
||||
code, report, _ = self.invoke("prepare", data)
|
||||
self.assertEqual(code, 1)
|
||||
self.assertEqual(report["status"], "blocked")
|
||||
self.assertFalse(any(row["check_id"].endswith("-install") and row["status"] == "ready" for row in report["checks"]))
|
||||
self.assertTrue(any(row["check_id"] == "deployment-command" for row in report["checks"]))
|
||||
|
||||
def test_prepare_mock_contract_calls_install_and_start(self):
|
||||
data = self.config()
|
||||
data["deployment"] = {"environment": "test", "command": [sys.executable, str(FIXTURE_RUNNER)]}
|
||||
code, report, _ = self.invoke("prepare", data)
|
||||
self.assertEqual(code, 0)
|
||||
self.assertEqual(report["status"], "ready")
|
||||
ids = {row["check_id"] for row in report["checks"]}
|
||||
self.assertIn("android-0-install", ids)
|
||||
self.assertIn("android-0-start", ids)
|
||||
|
||||
def test_verify_never_calls_install_or_start(self):
|
||||
data = self.config()
|
||||
calls: list[str] = []
|
||||
original = preflight.runner_call
|
||||
|
||||
def record(runner, request):
|
||||
calls.append(str(request["operation"]))
|
||||
return original(runner, request)
|
||||
|
||||
with mock.patch.object(preflight, "runner_call", side_effect=record):
|
||||
result = preflight.run(data, "verify")
|
||||
self.assertEqual(result.overall(), "ready")
|
||||
self.assertNotIn("install", calls)
|
||||
self.assertNotIn("start", calls)
|
||||
self.assertNotIn("capability_probe", calls)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+63
-2
@@ -27,7 +27,12 @@
|
||||
每轮测试会创建新的 terminal,结束后自动关闭;不会复用或关闭已有 terminal。
|
||||
- 测试环境部署由 deployer 在本机 Compose 中触发;正式环境部署由本机 `~/code/app00`
|
||||
项目的 fresh Agent terminal 触发,tester 不直接 SSH 或执行正式 Compose。
|
||||
- 会动硬件、会改数据的用例带风险级别(`readonly` / `mutating` / `destructive`)。默认只跑最低档 `readonly`。要跑更高风险,需要你明确说,例如「风险到 mutating」;碰哪个子系统另由 `gate`(如 hardware / weld)控制。
|
||||
- 会动硬件、会改数据的用例带风险级别(`readonly` / `mutating` / `destructive`)。默认
|
||||
`standard` 只跑最低档 `readonly`。隔离、可重置的 Writes 类环境在 catalog 设置
|
||||
`execution_profile: isolated-full` 后,所有 active 自动化用例都进入执行,不因
|
||||
`AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或
|
||||
`AUTHORIZE_PASSWORD` skipped;risk 原值仍保留。设备、产物、服务和 cleanup/rollback
|
||||
由 test-preflight 保证;客户端相册、权限和中文输入能力默认具备,由 journey 用例验证。
|
||||
|
||||
## 使用示例
|
||||
|
||||
@@ -76,7 +81,8 @@ APK/IPA,为本轮创建隔离 worktree,在 Android 和 iOS tester 节点安
|
||||
不执行可用平台;只有全部可用才按目标环境走 deployer 路由、解析 Agent、输出生效
|
||||
列表、准备 Orca 项目并创建临时 terminal。脚本类、浏览器和移动模拟器都在对应
|
||||
Agent 环境中执行。`builder.<platform>` 不能替代 `tester.<platform>`。
|
||||
4. 超出你允许的风险级别会跳过,不会偷偷执行。
|
||||
4. `standard` 下超出你允许的风险级别会跳过;`isolated-full` 由 catalog profile 明确
|
||||
全量 active 执行,不把风险悄悄改成 readonly。
|
||||
5. 无论测试成功、失败或异常,都会关闭本轮创建的 terminal;跑完给出按分类计数的
|
||||
结果,以及 `pass / fail / environment-fail / skipped / not-run`。
|
||||
|
||||
@@ -85,3 +91,58 @@ APK/IPA,为本轮创建隔离 worktree,在 Android 和 iOS tester 节点安
|
||||
- **列出**:能看到分类表和总条数;`.pouch/tester/cases.csv` 已由 skill 里的 `generate_cases_csv.py` 从 `cases/` 重新生成。每次运行的结果保存在独立的 `.pouch/tester/runs/<run-id>.*` 文件中。
|
||||
- **维护**:目标 id 能 `查看` 到,且检查通过。
|
||||
- **运行**:有本次 `runs/*.yaml` 快照和同名 `.csv` 成绩单(`executed` 列区分已跑 / 未跑);没填 HOST 时会停,而不是编造失败。
|
||||
|
||||
### 客户端用户旅程
|
||||
|
||||
测试用例支持两种顶层类型:`test_type: integration` 和 `test_type: journey`。集成套件
|
||||
至少包括 `api-contract`、`authorization`、`build-contract`;客户端套件仍是
|
||||
`client-smoke`、`client-regression`、`safety`。没有显式范围时默认不选择 integration。
|
||||
|
||||
执行顺序固定为 `test-preflight -> integration -> journey`。preflight 状态为 blocked 或
|
||||
environment-fail 时不创建客户端 session;integration 失败时依赖 journey 为 `not-run`。
|
||||
服务不可达是 environment-fail,契约断言错误是 integration fail,正确账号登录失败仍是
|
||||
journey fail。`tester run`/bundle 必须传入 caller 提供的 matrix:
|
||||
|
||||
```bash
|
||||
<skill-dir>/tester run --preflight-config /abs/preflight.yaml \
|
||||
--artifact android=/abs/app.apk --artifact ios=/abs/app.ipa
|
||||
```
|
||||
|
||||
没有真实 deployer、Mac 或设备时,只能使用 test-preflight 的 command-contract mock fixture
|
||||
验证协议,不能把 mock 结果写成真实运行成功。
|
||||
|
||||
客户端 E2E 用例使用 `client_suite: client-smoke | client-regression | safety`,并明确
|
||||
`journey`、`fixture`、`session`、`cleanup`、`platform`。标准生命周期是:
|
||||
|
||||
```text
|
||||
preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告
|
||||
```
|
||||
|
||||
正确测试账号登录失败记 `fail`;设备、产物或服务不可达记 `environment-fail`。fixture 或
|
||||
cleanup 缺失会在开始前阻断;前置失败后的剩余用例是 `not-run`,明确排除才是 `skipped`。
|
||||
`safety` 必须显式选择和授权,不进入默认 smoke/full。它与普通 tester 用例共用原有
|
||||
CSV/YAML 运行记录和汇总校验。
|
||||
|
||||
移动 preflight 还必须在实际 tester 节点验证相册 picker、照片权限与测试图片,以及中文
|
||||
拼音的真实 composition 和候选词选择;缺失时所有 testcase 都保留 `not-run` +
|
||||
`blocked_by`,不会静默 `skipped`。
|
||||
|
||||
失败回流使用 run YAML 顶层 `defects`/`test_issues` 与 `feedback`:PR 触发先更新 PR/关联
|
||||
issue、通知 dispatcher 重新分配开发;需求/bug 触发直接回开发流程。可用
|
||||
`scripts/record_issues.py` 生成同名 CSV 的结构化 issue 列。
|
||||
|
||||
### 汇总已有运行结果
|
||||
|
||||
对已经生成的同一运行 CSV/YAML,使用显式路径汇总:
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/summarize_run.py \
|
||||
--csv .pouch/tester/runs/<run-id>.csv \
|
||||
--yaml .pouch/tester/runs/<run-id>.yaml
|
||||
```
|
||||
|
||||
脚本只从 CSV 的 `result` 列计算 `pass`、`fail`、`environment-fail`、`skipped`、
|
||||
`not-run` 五类状态,并要求五类计数和等于 CSV 数据行数。YAML 只用于校验
|
||||
`results` 数量和 testcase id 集合;数量、集合、重复/缺失 id 或 CSV 缺失/未知状态
|
||||
时返回非零并输出诊断。校验成功才表示这份汇总可作为运行报告;非零表示报告不一致,
|
||||
不能据此判定产品失败。
|
||||
|
||||
+56
-2
@@ -74,10 +74,10 @@ tester agent 配置。
|
||||
- `local.yaml.example` 的 HOST 默认是 `127.0.0.1`;如果目标不是本机,必须在
|
||||
`local.yaml` 或环境变量中显式覆盖。`local.yaml` 不存在且没有环境变量时仍不猜
|
||||
HOST,停下并记 environment-fail。
|
||||
- 默认风险上限是 catalog 里的 `default_max_risk`。超出上限的条目标 `skipped`,禁止执行。
|
||||
- 默认执行 profile 是 `standard`:风险上限来自 catalog 的 `default_max_risk`,超出上限的条目标 `skipped`,禁止执行。若 catalog 声明 `execution_profile: isolated-full`,只允许用于隔离且可重置的测试环境:所有 active 自动化用例都进入执行矩阵,不因 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或 `AUTHORIZE_PASSWORD` skipped;原有 `risk` 值必须保留并进入报告。设备、产物、服务和清理契约由 test-preflight 检查;相册、照片权限/测试图片和中文拼音能力在本项目中默认具备,由 journey 用例真实验证。
|
||||
- `risk` 只能是 `readonly` / `mutating` / `destructive`,默认只跑 `readonly`。`tag=smoke` 只能配 `risk=readonly`。规则见 [levels.md](references/levels.md)。
|
||||
- `driver=browser` 必须开真实页面;`driver=mobile` 必须挂模拟器 session。二者都不得改成只打 API。
|
||||
- `automation=assisted|manual` 需要人在场:无人值守时记 `not-run`,不要自己代跑人工步骤。见 [concepts.md](references/concepts.md)。
|
||||
- `automation=assisted|manual` 需要人在场:无人值守时记 `not-run`,不要自己代跑人工步骤。isolated-full 取消的是风险/授权过滤,不会伪造人工操作。见 [concepts.md](references/concepts.md)。
|
||||
- 浏览器、移动模拟器按 catalog `sessions` 初始化一次,同一次运行内复用。
|
||||
- Agent、环境、项目目录和 Orca terminal 生命周期见 [execution.md](references/execution.md):可以从 project id/名称或 terminal id 自动解析,更新后输出配置列表;每轮都新建 terminal,结束后关闭本轮创建的 terminal,不复用或关闭已有 terminal。
|
||||
- 环境准备见 [deployment.md](references/deployment.md):测试环境只调用 deployer 的本机 Compose 路径;正式环境只委派给本机 `~/code/app00` 的 fresh Agent terminal,tester 不直接 SSH 或执行生产 Compose。
|
||||
@@ -145,3 +145,57 @@ python3 <skill-dir>/scripts/catalog.py show --root <project-root> --id <CASE_ID>
|
||||
4. 只有矩阵所有行都可用,才创建本轮测试 terminal/session,按 `select` 结果执行并记录成绩。
|
||||
|
||||
测试环境由本机 deployer 触发,正式环境由 app00 Agent terminal 委派。随后按 agent 解析目标项目、目录和 Agent,更新配置并输出生效列表。移动端每个平台的产物位置必须由调用方给出,值为本地文件或 URL;缺了就停,不要猜 builder、CI release 或本机旧包。矩阵行记产物位置、身份和 `tester.<platform>` binding,并用 `<skill-dir>/tester run --artifact android=<path-or-url> --artifact ios=<path-or-url> --platform android,ios` 下载或取用、安装并准备本轮隔离 worktree。Android 在 `tester.android`、iOS 在 `tester.ios` 节点执行。未指定范围时等同「tag=smoke 且风险上限=默认且 automation=automated」。`select` 的 `run_together` 组按 [run.md](references/run.md) 整组执行。无论部署或测试成功、失败、超时还是中断,都要清理各自本轮创建的 terminal。
|
||||
|
||||
### 客户端用户旅程标准
|
||||
|
||||
### 双层测试与执行顺序
|
||||
|
||||
用例顶层类型是 `test_type: integration | journey`。旧目录没有该字段时保持兼容:已有
|
||||
`client_suite`/`journey` 视为 journey,已有 `integration_suite` 视为 integration。集成
|
||||
套件至少包括 `api-contract`、`authorization`、`build-contract`;旅程套件仍固定为
|
||||
`client-smoke`、`client-regression`、`safety`。集成用例不需要客户端 session、platform
|
||||
或 journey 字段,不能被客户端套件选中。
|
||||
|
||||
一次运行严格遵守:
|
||||
|
||||
```text
|
||||
test-preflight (blocked/environment-fail -> 不创建 session)
|
||||
-> integration
|
||||
-> journey (integration fail -> not-run,不创建 session)
|
||||
```
|
||||
|
||||
`test-preflight` 的报告是环境证据,不计入 testcase 总数。服务不可达记
|
||||
`environment-fail`;服务已达但契约断言错误记 integration `fail`;正确测试账号登录失败
|
||||
仍记 journey `fail`。tester 的 `run`/bundle 入口必须接收 caller 提供的 preflight matrix,
|
||||
没有 matrix 不得绕过闸门或伪造成功。
|
||||
|
||||
声明 `client_suite`(迁移期也接受 `suite`)的用例采用固定的三套客户端集合:
|
||||
`client-smoke`、`client-regression`、`safety`。前两者分别覆盖最短只读主路径和隔离测试
|
||||
环境中的普通用户回归;`safety` 覆盖危险/权限边界场景,必须显式选择和授权,不能被
|
||||
默认 smoke/full 扩入。选择某套客户端旅程可用:
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-smoke
|
||||
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-regression --max-risk mutating
|
||||
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite safety --max-risk destructive
|
||||
```
|
||||
|
||||
客户端用例必须声明 `journey`、`fixture`、`session`、`cleanup` 和 `platform`。旅程包含
|
||||
`goal`、`entry_state`、`actions`、`assertions`、`final_state`、`failure_policy`;fixture
|
||||
至少声明测试账号;session 指向 catalog 中可复用的 browser/mobile session;cleanup 要有
|
||||
步骤或命令(只读也要显式说明无需清理);platform 只能是 Android/iOS。具体字段见
|
||||
`references/schema.md`。平台适配只能改操作层,不能把真实客户端操作换成 API 调用。
|
||||
|
||||
客户端运行器统一遵守 `preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告`。正确测试
|
||||
账号登录失败是产品 `fail`;设备、产物、服务或 session 不可达才是 `environment-fail`。
|
||||
fixture 或 cleanup 缺失会在 `catalog.py select` 的 preflight 阶段阻断,不产生虚假逐条结果。
|
||||
设备、产物、服务或 session 能力缺失时 preflight 必须明确 blocked/environment-fail,并为所有选中的 testcase 生成
|
||||
`not-run` 结果行;本项目默认具备相册、照片权限、测试图片或中文拼音 composition/candidate
|
||||
能力,不在 preflight 中校验或静默变成 `skipped`,由 journey 真实验证并记录产品失败。独立用例继续执行,依赖失败的用例写 `not-run` 和 `blocked_by`;
|
||||
修复后按 targeted rerun,再执行 full integration → journey。成绩仍写现有
|
||||
`runs/<run-id>.yaml` 并用 `generate_run_csv.py` 生成同名 CSV;YAML 可带 `defects`/
|
||||
`test_issues` 与 `feedback`,记录 testcase、product/test issue 类型、expected/observed、
|
||||
artifact hash、environment、证据路径、复现和下一步,不添加新的 result 值。
|
||||
|
||||
PR 触发的失败回流必须先更新 PR/关联 issue,再通知 dispatcher 重新分配开发;需求/bug
|
||||
触发的失败直接返回开发流程。tester 只生成结构化回流计划,不在本地偷偷修改远端。
|
||||
|
||||
@@ -35,11 +35,19 @@ catalog(整个测试集)
|
||||
| `session` | 可复用会话(浏览器 / 模拟器) | pytest fixture;Robot suite variable | 不是一条用例 |
|
||||
| `gate` | 项目细分闸门名 → 环境变量(`motion` / `hardware` / `weld`) | 授权开关 | 不是 `risk`,不要拿它代替风险等级 |
|
||||
| `authorize` | 风险档 → 环境变量。该档执行前必须为 `true` | pytest `skipif` 条件 | 不是认证(auth),是**授权** |
|
||||
| `execution_profile` | 执行环境策略:`standard` 或隔离可重置的 `isolated-full` | pytest marker/fixture policy | 不是风险等级;isolated-full 不会抹掉 `risk` |
|
||||
| `assume` | 额外跳过条件(gate / authorize 之外的环境表达式) | JUnit `assumeTrue`;pytest `skipif` | 不是授权开关的重复写法 |
|
||||
| `status_probe` | 分类级前置条件:请求一个 URL,命中即视为前置不满足 | pytest session fixture;Robot Suite Setup | 不是产品失败,不是整轮硬闸 |
|
||||
| `how` | 怎么测:`steps` / `run` / `url` / `timeout` | test steps | 不是判定标准 |
|
||||
| `expected` | 怎么算过:`{kind, value}` 列表 | pytest `assert`;Robot `Should …` | 不是步骤 |
|
||||
| `run_together` | 必须成组执行的用例(有序、可选 continue_on_fail) | 场景组(scenario) | 不是 suite,不是 tag |
|
||||
| `client_suite` | 客户端用户旅程套件:`client-smoke` / `client-regression` / `safety` | mobile/browser E2E suite | 不是 `category`,也不是通用 tag;迁移期可读 `suite` 别名 |
|
||||
| `test_type` | 顶层测试类型:`journey` 或 `integration` | E2E vs service/contract test | 不是 driver;旧目录可由既有字段推断 |
|
||||
| `integration_suite` | 集成套件:`api-contract` / `authorization` / `build-contract` | API/contract test suite | 不是客户端 `client_suite`;迁移期可读 `suite` 别名 |
|
||||
| `journey` | 用户目标、入口、操作、UI 断言、最终状态和失败继续策略 | Cucumber scenario / Robot task | 不是 API contract |
|
||||
| `fixture` | 旅程开始前声明的测试账号与隔离数据 | pytest fixture | 不是运行中临时猜出的数据 |
|
||||
| `cleanup` | 本轮数据的 teardown 契约;缺失会在 preflight 阻断 | pytest teardown / Robot teardown | 不是测试通过后的可选备注 |
|
||||
| `platform` | 旅程适用的 `android`/`ios` 平台列表 | device matrix | 不是 tester Agent binding;binding 仍在命令参数/运行环境 |
|
||||
|
||||
### 结果词
|
||||
|
||||
@@ -48,7 +56,7 @@ catalog(整个测试集)
|
||||
| `pass` | 通过 | passed |
|
||||
| `fail` | 产品缺陷 | failed |
|
||||
| `environment-fail` | 前置/环境导致无法执行,**永远不是产品失败** | pytest error;JUnit aborted |
|
||||
| `skipped` | **主动**不跑:`retired` / `pending` / 超风险上限 / 未授权 / `assume` 不成立 | pytest skipped;JUnit disabled |
|
||||
| `skipped` | **主动**不跑:standard 下的 `retired` / `pending` / 超风险上限 / 未授权 / `assume` 不成立 | pytest skipped;JUnit disabled |
|
||||
| `not-run` | **被动**没轮到:失败即停后剩下的;分类被 `status_probe` 挡住整组没执行;`automation` 需人工而无人值守 | pytest not run;deselected |
|
||||
|
||||
## 2. 命名迁移对照(v1 → v2)
|
||||
@@ -75,6 +83,8 @@ catalog(整个测试集)
|
||||
- 三级**单调**:`readonly ⊂ mutating ⊂ destructive`,所以只需要一个上限 `default_max_risk` 和一个 `--max-risk`,不用组合矩阵。
|
||||
- 与 `gate` 正交:`risk` 说「影响多大」,`gate` 说「碰哪个子系统」。点动 vs 焊接这种细分授权用 `gate`,不要再造 risk 名。
|
||||
- `tag=smoke` 只能配 `risk=readonly`:冒烟测试不许留下变化。
|
||||
- 隔离可重置环境可设置 `execution_profile: isolated-full` 跑所有 active 用例;安全前置由
|
||||
test-preflight 的设备、服务与 cleanup/rollback 契约保证,原始 risk 仍必须报告;客户端能力由 journey 用例验证。
|
||||
|
||||
### 3.2 `automation`:为什么是 `automated / assisted / manual`
|
||||
|
||||
@@ -86,6 +96,7 @@ catalog(整个测试集)
|
||||
- `assisted`:Agent 驱动,但至少一步需要人在场;该步在 `how.steps` 里以 `[人工]` 开头(可被 `validate` 机器判定)。
|
||||
- `manual`:只能人工执行;禁止 `driver`、禁止 `how.run`,必须有 `how.steps`。
|
||||
- 默认只跑 `automated`。无人值守时 `assisted` / `manual` 记 `not-run`;有人在场时 `assisted` 暂停等人确认,`manual` 由人工回报结果。
|
||||
- isolated-full 只取消风险/授权过滤,不伪造 assisted/manual 的人工步骤。
|
||||
|
||||
### 3.3 结果为什么拆出 `not-run`
|
||||
|
||||
@@ -99,6 +110,19 @@ catalog(整个测试集)
|
||||
- 因此它是**分类级前置条件**:`catalog.status_probe` 是全局默认,`categories[].status_probe` 可覆盖(缺省继承 / `false` 关闭 / mapping 整体替换)。对应 pytest 的 session fixture、Robot 的 Suite Setup。
|
||||
- 命中或不可达只影响用它的分类;只有所有选中分类都被挡住,整轮结论才是 `environment-fail`。`select` 会按分类返回解析后的生效值,Agent 不要自己读 catalog 猜。
|
||||
|
||||
### 3.5 客户端套件和生命周期
|
||||
|
||||
客户端套件固定为:
|
||||
|
||||
- `client-smoke`:启动、正确账号登录和少量只读主路径,通常配 `tag=smoke`、`risk=readonly`。
|
||||
- `client-regression`:完整的普通用户旅程,可包含隔离环境中的 `mutating` 操作;不等于所有 full 用例。
|
||||
- `safety`:删除、权限边界、改密、停用或故障注入等危险旅程;必须显式选择和授权,不进入默认运行。
|
||||
|
||||
每条旅程按 `preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告` 执行。登录失败在
|
||||
客户端和正确测试账号都已就绪时记 `fail`;设备、产物、服务或 session 不可达才记
|
||||
`environment-fail`。fixture/cleanup 缺失是 contract block,不伪装成逐条结果。前置失败
|
||||
后的剩余条目是 `not-run`,主动排除仍是 `skipped`。
|
||||
|
||||
## 4. 不采用的词
|
||||
|
||||
| 词 | 不用它的原因 |
|
||||
|
||||
@@ -25,6 +25,16 @@ bindings:
|
||||
上执行测试。`tester.test` 只能用于明确的单平台或非移动测试;同一轮包含 Android
|
||||
和 iOS 时必须分别解析两个 tester 绑定。
|
||||
|
||||
移动 bundle 入口必须把平台绑定显式传入:
|
||||
|
||||
```bash
|
||||
--binding android=tester.android --binding ios=tester.ios
|
||||
```
|
||||
|
||||
入口会校验 preflight matrix 的每个平台 binding、node 和 runner。远程 iOS 必须是
|
||||
`contract` runner;多平台运行不会读取旧的单一 `local.yaml.agent` 来决定执行节点,
|
||||
也不会在协调节点直接执行 Android/iOS 平台命令。
|
||||
|
||||
`locations` 和 `profiles` 的定义见共享 `agent-runtime`。tester adapter 只把它 materialize
|
||||
到本轮兼容配置;每轮 terminal 仍然新建并在结束时关闭,不能因为 binding 是 persistent
|
||||
就复用测试 terminal。
|
||||
|
||||
@@ -36,7 +36,11 @@
|
||||
| `destructive` | 难恢复或伤人/伤钱/伤环境 | 删库、真实扣费 | 清数据、内购 | 急停、断电、真焊 |
|
||||
|
||||
默认只跑 `readonly`。项目细分闸(点动 vs 焊接)用可选 `gate`,不要再发明 risk 名。
|
||||
隔离且可重置的项目可在 catalog 设置 `execution_profile: isolated-full`,此时所有 active
|
||||
用例按原始 risk 执行;`risk` 仍用于报告和 cleanup 设计,不能改写为 readonly。
|
||||
|
||||
`tag=smoke` 且 `risk!=readonly` 为非法。
|
||||
|
||||
不想跑某一档时,`authorize.<risk>` 指向的环境变量保持 false 即可——该档用例记 `skipped`,不是 `not-run`。
|
||||
standard 下不想跑某一档时,`authorize.<risk>` 指向的环境变量保持 false 即可——该档用例记
|
||||
`skipped`,不是 `not-run`。isolated-full 不使用这些授权开关,改由 preflight 能力探针和
|
||||
cleanup/rollback 契约阻断不安全环境。
|
||||
|
||||
@@ -9,6 +9,11 @@ Agent、执行目标和临时 terminal 的完整协议见 [execution.md](executi
|
||||
|
||||
## 0. 建立测试矩阵并完成全量 preflight
|
||||
|
||||
`test-preflight` 是 integration 和 journey 的共同硬闸。tester 必须先调用其 `verify`
|
||||
或 `prepare`,保存独立的 preflight YAML/CSV;状态为 `blocked` 或 `environment-fail`
|
||||
时不创建 browser/mobile session,所有尚未执行的 testcase 记 `not-run`。preflight 证据
|
||||
不算 testcase,也不进入 testcase CSV/YAML 的总数。
|
||||
|
||||
测试不是“找到一个能跑的设备就开始”。每轮先建立一份测试矩阵;移动端每个目标产物
|
||||
必须单独占一行:
|
||||
|
||||
@@ -67,7 +72,11 @@ iOS | … | … | page.writes | tester.ios | remote-ios
|
||||
|
||||
## 1. 选范围
|
||||
|
||||
未指定范围 → `--tag smoke`,风险上限 = `default_max_risk`,`automation` 默认只跑 `automated`。
|
||||
未指定范围 → `--tag smoke`,执行 profile = catalog 的 `execution_profile`(默认 `standard`),
|
||||
`standard` 的风险上限 = `default_max_risk`,`automation` 默认只跑 `automated`。
|
||||
`execution_profile=isolated-full` 只用于隔离可重置环境:不按风险上限或 `AUTHORIZE_*`/gate
|
||||
授权变量过滤 active 用例,保留每条用例原始 `risk`;安全门禁由 test-preflight 能力探针和
|
||||
cleanup/rollback 契约承担。
|
||||
|
||||
用户说分类、id、tag、风险上限时,原样传给 select:
|
||||
|
||||
@@ -75,33 +84,65 @@ iOS | … | … | page.writes | tester.ios | remote-ios
|
||||
python3 <skill-dir>/scripts/catalog.py select --root <root> \
|
||||
[--category CAT] [--id ID]... [--tag smoke|full|slow] \
|
||||
[--max-risk <risk>] [--driver script|browser|mobile] \
|
||||
[--execution-profile standard|isolated-full] \
|
||||
[--automation automated|assisted|manual|any] \
|
||||
[--test-type integration|journey] [--suite <client-or-integration-suite>] \
|
||||
[--platform android|ios]... \
|
||||
[--artifact android=<path-or-url>] [--artifact ios=<path-or-url>] \
|
||||
[--binding android=tester.android] [--binding ios=tester.ios]
|
||||
```
|
||||
|
||||
客户端旅程还可按标准套件选择:
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-smoke
|
||||
```
|
||||
|
||||
`client-smoke`、`client-regression`、`safety` 是 `client_suite` 的固定值,不替换既有
|
||||
`tag=smoke|full|slow`。`safety` 不参与未指定范围的默认选择。
|
||||
|
||||
集成测试显式选择:
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/catalog.py select --root <root> --test-type integration
|
||||
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite api-contract
|
||||
```
|
||||
|
||||
未指定 `--test-type`/`--suite` 时,默认选择不会包含 `test_type=integration`。
|
||||
|
||||
选中任一 `driver=mobile` 用例时,`--platform`、每个平台的 `--artifact` 和对应
|
||||
`tester` binding 是硬性参数;`catalog.py select` 会检查产物文件/下载地址和
|
||||
`.pouch/agent.yaml` 中的 binding。缺任何一项直接返回非零并打印 `preflight: blocked`,
|
||||
不能拿到 cases 后绕过闸门开始测试。默认 binding 是 `tester.<platform>`,所以 iOS
|
||||
会强制检查 `tester.ios`,不会回退到 `builder.ios` 或通用 `tester.test`。
|
||||
|
||||
`--automation` 默认 `automated`(需人工的条目会被跳过,并在 JSON `skipped` 里说明);用户明确要求现场配合时用 `assisted`,`any` 表示不按人工参与度筛。
|
||||
`--automation` 默认 `automated`(需人工的条目会被记为 `not-run`,不会伪造人工步骤);用户明确要求现场配合时用 `assisted`,`any` 表示不按人工参与度筛。
|
||||
|
||||
单根 `select` 的 JSON 字段:`max_risk` / `automation` / `count` / `vars`(已展开的变量,用来替换 `how` 里的 `${…}`)/ `artifacts` / `preflight` / `skipped` / `run_together` / `status_probe`(按分类,`url` 已展开)/ `cases`。从父仓库 fan-out 时返回 `mode: fan-out` 和每个根的独立结果,不能把多个根的 cases 拼成一个列表。
|
||||
|
||||
`select` 已排除 `retired` 和风险高于上限的条目(后者若用户点了具体 id,会在 JSON 的 `skipped` 里说明,不执行)。
|
||||
`standard` 的 `select` 已排除 `retired` 和风险高于上限的条目(后者若用户点了具体 id,会在 JSON 的 `skipped` 里说明);`isolated-full` 保留所有 active 条目,不因风险或授权变量 skipped。
|
||||
|
||||
若 catalog 有 `run_together`,命中组内任一条就把整组按 `ids` 顺序扩进列表(JSON 带 `run_together`)。这不是擅自扩大范围。
|
||||
|
||||
无命中:停下说明筛选条件,不要扩大范围。
|
||||
|
||||
## 2. 运行前闸
|
||||
## 2. 双层执行
|
||||
|
||||
preflight 通过后,先按 catalog 顺序执行 `integration`,再按客户端套件顺序执行
|
||||
`journey`。集成服务不可达是 `environment-fail`;服务已达但 API/权限/数据契约断言错误
|
||||
是 `fail`。任何 integration `fail` 都阻断依赖的 journey,并把它们记为 `not-run`;此时
|
||||
不得创建客户端 session。正确测试账号登录失败发生在客户端已启动后,仍是产品 `fail`。
|
||||
|
||||
运行器可复用 `scripts/two_tier.py` 的 `run_integration_first`,它只接收外部 driver
|
||||
回调,不实现 API、浏览器或移动执行器;mock fixture 只能验证调用顺序和失败传播,不能
|
||||
冒充真实部署、Mac 或设备成功。
|
||||
|
||||
## 3. 运行前闸
|
||||
|
||||
- `HOST` 未设(环境变量 / `local.yaml` 都没有):全部停,environment-fail。
|
||||
- 命中条的 `risk` 在 catalog `authorize` 里有对应变量:变量不是 `true` 则该条 `skipped`。有 `gate` 时还要 `catalog.gates` 对应变量为 `true`;`assume` 不成立同样 `skipped`。即使用户点了 id 也不执行。
|
||||
- 授权/假设不成立是**主动不跑**(`skipped`),不是 `not-run`:这是我们决定不跑,不是没轮到。
|
||||
- `standard`:命中条的 `risk` 在 catalog `authorize` 里有对应变量时,变量不是 `true` 则该条 `skipped`;有 `gate` 时还要对应变量为 `true`,`assume` 不成立同样 `skipped`。即使用户点了 id 也不执行。
|
||||
- `isolated-full`:不读取 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM`、`AUTHORIZE_PASSWORD` 等人工开关;相册、照片权限/测试图片和中文拼音能力默认具备,由 journey 用例真实验证,失败记为产品失败。
|
||||
- standard 中授权/假设不成立是**主动不跑**(`skipped`);isolated-full 不因客户端能力授权变量停止,设备、产物、服务或清理前置失败时才为全部 testcase 写 `not-run` + `blocked_by: [test-preflight]`。
|
||||
- `status_probe` 是**分类级前置条件**(业界对应 pytest 的 session fixture / Robot 的 Suite Setup)。用 `select` 返回的 `status_probe` 解析生效值,不要自己读 catalog 猜。返回的 `probe.url` 已用 `vars` 展开,直接请求即可;`select` 同批返回 `vars`:
|
||||
|
||||
| `categories[].status_probe` | 生效探测 |
|
||||
@@ -114,7 +155,7 @@ python3 <skill-dir>/scripts/catalog.py select --root <root> \
|
||||
- 探测不可达或命中 `stop_if`:在 preflight 阶段记录对应分类/平台的不可用项并停止整轮;
|
||||
先返回用户处理,不创建后续测试 session。只有所有本轮目标环境通过,才进入用例执行。
|
||||
|
||||
## 2.1 移动端平台环境闸门与用户引导
|
||||
## 3.1 移动端平台环境闸门与用户引导
|
||||
|
||||
`driver=mobile` 不是一个可以互换的平台。运行前必须根据 catalog session、用例变量或用户指定,明确本轮是 Android 还是 iOS;无法判断时先停下询问平台,不得猜测。两类平台分别执行以下闸门:
|
||||
|
||||
@@ -175,7 +216,7 @@ Android+iOS 默认是全量 preflight 闸门:任一项不满足,另一平台
|
||||
|
||||
环境引导只负责让用户补齐前置条件;除 catalog 已明确授权的 session 初始化外,不自动安装 SDK、配对真机、修改系统安全设置或上传产物。
|
||||
|
||||
## 3. Session
|
||||
## 4. Session
|
||||
|
||||
看命中条用到的 `session` 键:
|
||||
|
||||
@@ -184,9 +225,9 @@ Android+iOS 默认是全量 preflight 闸门:任一项不满足,另一平台
|
||||
- `reuse: run`:整轮结束再关。`persist`:不关。
|
||||
- 中途 session 挂了:后续同 driver 标 environment-fail,不要标产品失败。
|
||||
|
||||
## 4. 执行
|
||||
## 5. 执行
|
||||
|
||||
按 select 返回的顺序,一条一条来。select 按 `catalog.yaml` 的 `categories` 顺序,同一分类内按该 yaml 列表顺序(不是文件名字母序)。默认失败即停(产品 `fail` 或探测到的不安全状态)。用户说继续 / keep-going 才往后跑。
|
||||
按 select 返回的顺序,一条一条来。select 按 `catalog.yaml` 的 `categories` 顺序,同一分类内按该 yaml 列表顺序(不是文件名字母序)。同一 phase 的独立用例继续执行;只有显式 `depends_on` 的失败依赖才写 `not-run` + `blocked_by`。用户说 targeted rerun 时只重跑受影响用例,修复后再 full integration → journey。
|
||||
|
||||
例外:当前条和下一条同属 `run_together` 且该组 `continue_on_fail: true` 时,前一条产品 `fail` 仍跑后一条。`environment-fail` 仍整轮停。
|
||||
|
||||
@@ -216,9 +257,9 @@ python3 <skill-dir>/scripts/catalog.py run-script --root <root> --id <id>
|
||||
|
||||
挂到已有 session,执行 `how.steps`,对照 `expected`。必须留下可见文本、URL 或交互证据。
|
||||
|
||||
## 5. 记录与报告
|
||||
## 6. 记录与报告
|
||||
|
||||
每条:`id, result, evidence, notes`。`result` 只能是 `pass` / `fail` / `environment-fail` / `skipped` / `not-run`。`skipped` 是主动不跑(retired / pending / 超风险上限 / 未授权 / `assume` 不成立),`not-run` 是没轮到(失败即停后剩下的 / 被前置整组挡住的 / 需人工而无人值守的)。
|
||||
每条:`id, result, evidence, notes`,依赖阻断时另有 `blocked_by`。`result` 只能是 `pass` / `fail` / `environment-fail` / `skipped` / `not-run`。`skipped` 是 standard 中主动不跑(retired / pending / 超风险上限 / 未授权 / `assume` 不成立);`not-run` 是没轮到或 preflight/依赖阻断。失败用例可在 YAML 顶层 `defects`(兼容 `test_issues`)记录 testcase、`issue_type=product|test`、expected/observed、artifact hash、environment、evidence path、reproduction、next step。
|
||||
|
||||
写入 `.pouch/tester/runs/<UTC时间>-<范围>-<随机后缀>.yaml`,可用
|
||||
`generate_run_csv.new_run_id(<范围>)` 生成同名 `runId`。记录包含唯一 `runId`、筛选条件、HOST 是否已设(可以写 HOST 值)、session 是否复用、每个分类的 `status_probe` 生效值与观测结果(`source` / `disabled` 原因 / `hit` / `observed`)、逐条结果。记录文件只创建一次,不覆盖已有文件。立刻用同一快照生成 CSV,不要手改:
|
||||
@@ -227,7 +268,23 @@ python3 <skill-dir>/scripts/catalog.py run-script --root <root> --id <id>
|
||||
python3 <skill-dir>/scripts/generate_run_csv.py --root <root> --run <yaml>
|
||||
```
|
||||
|
||||
这会写出与快照同名的 `.pouch/tester/runs/<run-id>.csv`(`executed=yes|no`),不会修改 `cases.csv`。对话里两个路径都报。
|
||||
这会写出与快照同名的 `.pouch/tester/runs/<run-id>.csv`(`executed=yes|no`),不会修改
|
||||
`cases.csv`。对话里两个路径都报。
|
||||
|
||||
需要复核已有成绩单时,显式同时传入同一次运行的 CSV 和 YAML:
|
||||
|
||||
```bash
|
||||
python3 <skill-dir>/scripts/summarize_run.py \
|
||||
--csv .pouch/tester/runs/<run-id>.csv \
|
||||
--yaml .pouch/tester/runs/<run-id>.yaml
|
||||
```
|
||||
|
||||
汇总状态只读取 CSV `result` 列,必须输出五类 `pass / fail / environment-fail /
|
||||
skipped / not-run`,且计数和等于 CSV 数据行数。汇总前会检查 CSV 数据行数、CSV 唯一
|
||||
testcase id 数、YAML `results` 数量和 CSV/YAML testcase id 集合;重复或缺失 id、缺失或
|
||||
未知 CSV 状态、任一数量或集合不一致都会非零退出并给出诊断。YAML 的 `counts` 或其它
|
||||
历史汇总字段不会参与状态计数。退出码 0 表示汇总校验成功,非零表示报告不一致或输入
|
||||
无效,不等同于产品测试失败。
|
||||
|
||||
对话报告:
|
||||
|
||||
|
||||
@@ -14,6 +14,10 @@ runs/<run-id>.yaml 每次运行快照(执行源);run-id 必须唯一
|
||||
runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对应 yaml 生成
|
||||
```
|
||||
|
||||
两类用例共用同一套 cases/runs 和五态报告,不另造成绩单。新目录显式声明顶层
|
||||
`test_type: journey | integration`;旧目录没有该字段时按已有 `client_suite`/`journey`
|
||||
或 `integration_suite` 推断。`category` 仍是文件/执行顺序容器,不被套件字段替代。
|
||||
|
||||
## catalog.yaml
|
||||
|
||||
| 字段 | 含义 |
|
||||
@@ -21,6 +25,7 @@ runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对
|
||||
| `categories` | `{id, title, status_probe?}` 列表,id 与 `cases/<id小写>.yaml` 对应;列表顺序即 `select` / 执行顺序。`status_probe` 见下 |
|
||||
| `risks` | 从低到高,只能是 `readonly` → `mutating` → `destructive` 的一段,且以 `readonly` 开头。含义见 [levels.md](levels.md) |
|
||||
| `default_max_risk` | 未指定时的运行上限,必须是 `risks` 里的值 |
|
||||
| `execution_profile` | `standard`(默认)或 `isolated-full`。后者只用于隔离可重置环境,保留 risk 语义但不按 risk/authorize/gate 过滤 active 用例;设备、服务和 cleanup/rollback 契约仍由 preflight 检查,客户端能力由 journey 真实验证 |
|
||||
| `vars` | 可替换变量,如 `API_BASE: http://${HOST}:8095` |
|
||||
| `artifacts` | 可选的移动产物身份契约。按 `android` / `ios` 声明 `required` 和 `package` / `bundle_id`;每轮文件路径或下载地址由调用方提供,不提交临时产物路径。 |
|
||||
| `authorize` | 风险名 → 环境变量名。该档执行前变量必须为 `true`(`readonly` 不需要) |
|
||||
@@ -29,6 +34,12 @@ runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对
|
||||
| `id_pattern` | 用例 id 正则 |
|
||||
| `run_together` | 可选。成组执行:`[{ids: [TC-…, TC-…], continue_on_fail: true}]`。点到组内任一条,`select` 会按 `ids` 顺序把整组扩进命中列表。`continue_on_fail: true` 时组内前一条产品 fail 仍跑后一条(返回/收尾)。一个 id 只能属于一组。 |
|
||||
| `status_probe` | 可选。顶层默认值,一个**分类级前置条件**:`{url, stop_if: {字段: [命中值]}}`。运行前请求 `url`,响应命中任一 `stop_if` 条件即视为前置不满足。分类项可用同名键覆盖:缺省继承顶层、`false` 关闭、mapping 整体替换。命中或不可达只影响用它的分类,不是产品失败。业界对应 pytest 的 session fixture / Robot 的 Suite Setup。`select` 会按分类返回解析后的生效值,`url` 已用 `vars` 展开 |
|
||||
| `client_suites` | 可选的标准套件声明;出现时必须完整包含 `client-smoke`、`client-regression`、`safety`。缺省运行仍按 `tag` 兼容旧目录 |
|
||||
| `integration_suites` | 可选的集成套件声明;标准值为 `api-contract`、`authorization`、`build-contract` |
|
||||
|
||||
`isolated-full` 不得把风险字段改成 `readonly`,也不得把 `AUTHORIZE_ALBUM`、
|
||||
`AUTHORIZE_PASSWORD`、`IME_PINYIN` 等变量当成门禁。移动用例必须通过 test-preflight
|
||||
的设备、产物、服务和网络检查;相册、权限和输入法能力默认具备,交由 journey 用例验证。
|
||||
|
||||
`local.yaml` 与环境变量覆盖 `HOST` 和授权开关。模板中的 `HOST` 默认是
|
||||
`127.0.0.1`;远程目标必须显式覆盖。查找顺序:环境变量 > `local.yaml` >
|
||||
@@ -70,6 +81,7 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
|
||||
| 字段 | 必填 | 含义 |
|
||||
|------|------|------|
|
||||
| `id` | 是 | 匹配 `id_pattern` |
|
||||
| `test_type` | 新用例是 | `journey` 或 `integration`;缺省兼容旧目录并按字段推断 |
|
||||
| `category` | 是 | 必须等于文件声明的 category |
|
||||
| `title` | 是 | 短名 |
|
||||
| `what` | 是 | 一句可观察的「测什么」 |
|
||||
@@ -84,10 +96,67 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
|
||||
| `expected` | 是 | 怎么过,至少一条 `{kind, value}` |
|
||||
| `maps_to` | 否 | 对人文档的功能项 id |
|
||||
| `assume` | 否 | 额外跳过条件(环境变量表达式)。不要重复 `risk` / `gate` 已覆盖的授权变量,否则 `validate` 报错 |
|
||||
| `depends_on` | 否 | 依赖的 testcase id 列表;依赖 fail/environment-fail/not-run 时本条写 `not-run` 和 `blocked_by`,独立用例继续 |
|
||||
| `setup` | 否 | 人话前置 |
|
||||
| `teardown` | 否 | 跑完回退 |
|
||||
| `notes` | 否 | 备注 |
|
||||
|
||||
### 集成测试用例契约
|
||||
|
||||
`test_type: integration` 的用例不创建 browser/mobile session,也不能声明 `platform` 或
|
||||
客户端 `journey`。必须有 `integration_suite`(迁移期可用 `suite`)且值为
|
||||
`api-contract`、`authorization` 或 `build-contract`;`fixture` 必须是非空 mapping;
|
||||
`request`、`input`、`call`、`integration` 之一或旧式 `how` 用于描述调用;`expected` 是
|
||||
协议/数据断言。`mutating`/`destructive` 集成用例必须提供 `cleanup.steps` 或
|
||||
`cleanup.run`(只读探针可用 `required: false` + `reason`)。
|
||||
|
||||
选择 `--suite api-contract` 或 `--test-type integration` 才会进入集成阶段;无显式类型
|
||||
时的默认选择不会包含 integration。
|
||||
|
||||
### 客户端旅程用例契约
|
||||
|
||||
声明 `client_suite` 后,以下字段都必须存在;`catalog.py validate` 和 `select` 使用同一
|
||||
套检查。契约缺失是 `preflight: blocked`,不生成逐条产品失败。
|
||||
|
||||
| 字段 | 必填 | 约束 |
|
||||
|------|------|------|
|
||||
| `client_suite` | 是 | `client-smoke`、`client-regression` 或 `safety`;安全套件必须显式选择,不属于默认 smoke/full |
|
||||
| `journey.goal` | 是 | 用户目标,不写接口目标 |
|
||||
| `journey.entry_state` | 是 | 启动/登录前的可观察入口状态 |
|
||||
| `journey.actions` | 是 | 非空用户操作列表;真实 App/browser 驱动从这里开始 |
|
||||
| `journey.assertions` | 是 | 非空可观察 UI 断言列表;最终业务状态可另由 `expected` 的 api-field/api-status 佐证 |
|
||||
| `journey.final_state` | 是 | 旅程完成后的业务状态 |
|
||||
| `journey.failure_policy` | 是 | `stop` 或 `continue`;默认不隐式吞掉前置失败 |
|
||||
| `fixture` | 是 | mapping,至少有 `account`;账号和数据由 app catalog 提供,skill 不写具体账号 |
|
||||
| `session` | 是 | 复用的 browser/mobile session 名,必须在 catalog.sessions 声明 |
|
||||
| `cleanup` | 是 | mapping,必须有非空 `steps`/`run`;只读旅程也要用 `required: false` + `reason` 显式说明无需清理 |
|
||||
| `platform` | 是 | `android`/`ios` 的字符串或列表,不能重复;平台差异只能进入执行适配层 |
|
||||
|
||||
一个最小形状如下(值是占位符,不是 app 规则):
|
||||
|
||||
```yaml
|
||||
client_suite: client-smoke
|
||||
platform: [android, ios]
|
||||
journey:
|
||||
goal: "用户完成一个可观察的客户端目标"
|
||||
entry_state: logged_out
|
||||
actions: ["启动客户端", "输入测试账号并登录", "完成目标操作"]
|
||||
assertions: ["看到目标页面", "看到成功提示"]
|
||||
final_state: "目标状态已保存"
|
||||
failure_policy: stop
|
||||
fixture:
|
||||
account: test-role
|
||||
data: []
|
||||
session: mobile
|
||||
cleanup:
|
||||
steps: ["删除本轮 run-id 创建的数据"]
|
||||
```
|
||||
|
||||
账号登录属于 session 初始化:正确测试账号登录失败是产品 `fail`;设备、产物或服务
|
||||
不可达才是 `environment-fail`。fixture/cleanup 缺失必须在启动、登录和任何旅程操作前
|
||||
阻断。旅程失败后,尚未轮到的用例记 `not-run`;风险、生命周期或明确范围排除记
|
||||
`skipped`。清理失败单独写入运行快照的 cleanup 证据,不改造为一个新的 result 值。
|
||||
|
||||
## how
|
||||
|
||||
按 `driver` 分派;`automation=manual` 时没有 `driver`,只有 `how.steps`(人工步骤)。
|
||||
@@ -107,3 +176,24 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
|
||||
## 判定
|
||||
|
||||
`pass` / `fail` / `environment-fail` / `skipped` / `not-run`
|
||||
|
||||
## 运行失败回流
|
||||
|
||||
run YAML 的 `results` 必须保留每条 testcase 行。失败可在顶层 `defects`(兼容别名
|
||||
`test_issues`)追加:
|
||||
|
||||
```yaml
|
||||
defects:
|
||||
- testcase: TC-CLIENT-001
|
||||
issue_type: product # product | test
|
||||
expected: "候选词可选择"
|
||||
observed: "composition 成功但候选列表为空"
|
||||
artifact_hash: sha256:...
|
||||
environment: "Android emulator API 35 / tester.android"
|
||||
evidence_path: .pouch/tester/evidence/TC-CLIENT-001.png
|
||||
reproduction: "打开相册后输入 nihao"
|
||||
next_step: "开发修复候选词渲染后 targeted rerun"
|
||||
```
|
||||
|
||||
PR 触发的 run 可带 `feedback` 计划:先更新 PR/关联 issue,再通知 dispatcher 重新分配开发;
|
||||
需求/bug 触发则 route 直接为 development。tester 只记录计划,不隐式调用远端写操作。
|
||||
|
||||
@@ -7,11 +7,15 @@ CI release、builder 绑定或本机旧文件猜测产物。
|
||||
<skill-dir>/tester run \
|
||||
--artifact android=<path-or-url> \
|
||||
--artifact ios=<path-or-url> \
|
||||
--platform android,ios
|
||||
--platform android,ios \
|
||||
--binding android=tester.android \
|
||||
--binding ios=tester.ios
|
||||
```
|
||||
|
||||
URL 就下载,本地文件就直接用。每个平台都在自己的 `tester.<platform>` 节点校验身份并安装,
|
||||
随后由现有 tester catalog 驱动真实移动 session。默认先完成 Android+iOS 双平台
|
||||
URL 就下载,本地文件就直接用。每个平台都通过自己的 `tester.<platform>` binding
|
||||
校验、探测和安装;`agent-runtime` 负责把 Android 请求送到 Android 节点、把 iOS
|
||||
请求送到 Mac 节点。旧的单一 `local.yaml.agent` 只能作为兼容配置,不能作为双平台
|
||||
运行目标。随后由现有 tester catalog 驱动真实移动 session。默认先完成 Android+iOS 双平台
|
||||
preflight;任一平台无法准备时整轮停止。只有显式加 `--partial` 才允许运行可用平台。
|
||||
|
||||
每轮另外创建隔离 Git worktree,不切用户当前分支。`--commit` 缺省为当前 HEAD。
|
||||
|
||||
@@ -18,6 +18,16 @@ from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from journey import (
|
||||
CLIENT_SUITES,
|
||||
INTEGRATION_SUITES,
|
||||
TEST_TYPES,
|
||||
contract_errors,
|
||||
preflight as journey_preflight,
|
||||
suite as case_suite,
|
||||
test_type,
|
||||
)
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
@@ -32,6 +42,7 @@ LIFECYCLE_SKIP_REASON = {
|
||||
}
|
||||
TAGS = ("smoke", "full", "slow")
|
||||
CANONICAL_RISKS = ("readonly", "mutating", "destructive")
|
||||
EXECUTION_PROFILES = ("standard", "isolated-full")
|
||||
AUTOMATIONS = ("automated", "assisted", "manual")
|
||||
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
||||
EXPECTED_KINDS = (
|
||||
@@ -341,6 +352,21 @@ def risk_index(meta: dict[str, Any], name: str) -> int:
|
||||
return risks.index(name)
|
||||
|
||||
|
||||
def resolve_execution_profile(meta: dict[str, Any], requested: str | None = None) -> str:
|
||||
"""Resolve the run profile without changing the meaning of ``risk``.
|
||||
|
||||
``isolated-full`` is an execution policy for a resettable test environment:
|
||||
it disables risk/authorization filtering, but leaves the catalog risk value
|
||||
on every case so reports still describe the side effect that was exercised.
|
||||
"""
|
||||
profile = requested or meta.get("execution_profile") or "standard"
|
||||
if profile not in EXECUTION_PROFILES:
|
||||
raise SystemExit(
|
||||
f"未知 execution_profile: {profile!r}(可选: {', '.join(EXECUTION_PROFILES)})"
|
||||
)
|
||||
return str(profile)
|
||||
|
||||
|
||||
def host_set(vars_map: dict[str, str]) -> bool:
|
||||
host = (vars_map.get("HOST") or "").strip()
|
||||
return bool(host) and "${HOST}" not in host
|
||||
@@ -368,7 +394,7 @@ def cmd_list(root: Path) -> int:
|
||||
pending = [c for c in cases if c.get("lifecycle") == "pending"]
|
||||
print(f"## 测试目录 active {len(active)} / pending {len(pending)} / retired {len(retired)}")
|
||||
print()
|
||||
headers = ["分类", "标题", "条数", "smoke"] + risks + list(DRIVERS)
|
||||
headers = ["分类", "标题", "条数", "journey", "integration", "smoke"] + risks + list(DRIVERS)
|
||||
print("| " + " | ".join(headers) + " |")
|
||||
print("| " + " | ".join("---" for _ in headers) + " |")
|
||||
|
||||
@@ -386,7 +412,14 @@ def cmd_list(root: Path) -> int:
|
||||
title = (cats_meta.get(cat) or {}).get("title") or (
|
||||
group[0].get("_file_title") if group else ""
|
||||
)
|
||||
row = [cat, str(title), str(len(group)), str(sum(1 for c in group if c.get("tag") == "smoke"))]
|
||||
row = [
|
||||
cat,
|
||||
str(title),
|
||||
str(len(group)),
|
||||
str(sum(1 for c in group if test_type(c) == "journey")),
|
||||
str(sum(1 for c in group if test_type(c) == "integration")),
|
||||
str(sum(1 for c in group if c.get("tag") == "smoke")),
|
||||
]
|
||||
for risk in risks:
|
||||
row.append(str(sum(1 for c in group if c.get("risk") == risk)))
|
||||
for driver in DRIVERS:
|
||||
@@ -553,6 +586,7 @@ def expand_run_together(
|
||||
skipped: list[dict[str, str]],
|
||||
max_i: int,
|
||||
automation_filter: str = "automated",
|
||||
bypass_risk: bool = False,
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
groups = run_together_groups(meta)
|
||||
if not groups:
|
||||
@@ -590,7 +624,7 @@ def expand_run_together(
|
||||
except SystemExit:
|
||||
skipped.append({"id": cid, "reason": f"未知风险 {risk}"})
|
||||
continue
|
||||
if r_i > max_i:
|
||||
if not bypass_risk and r_i > max_i:
|
||||
skipped.append({"id": cid, "reason": f"risk={risk} 超过上限"})
|
||||
continue
|
||||
members.append(public_case(case))
|
||||
@@ -753,9 +787,11 @@ def mobile_preflight(
|
||||
def cmd_select(args: argparse.Namespace, root: Path) -> int:
|
||||
meta = load_catalog_meta(root)
|
||||
cases = load_all_cases(root)
|
||||
execution_profile = resolve_execution_profile(meta, args.execution_profile)
|
||||
bypass_risk = execution_profile == "isolated-full"
|
||||
max_risk = args.max_risk or meta.get("default_max_risk") or "readonly"
|
||||
max_i = risk_index(meta, max_risk)
|
||||
automation_filter = args.automation or "automated"
|
||||
automation_filter = args.automation or ("any" if bypass_risk else "automated")
|
||||
wanted = set(parse_ids(args.ids or []))
|
||||
selected: list[dict[str, Any]] = []
|
||||
skipped: list[dict[str, str]] = []
|
||||
@@ -781,6 +817,20 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
|
||||
continue
|
||||
if args.tag and case.get("tag") != args.tag:
|
||||
continue
|
||||
if args.test_type and test_type(case) != args.test_type:
|
||||
continue
|
||||
if args.suite and case_suite(case) != args.suite:
|
||||
continue
|
||||
# A mixed catalog keeps legacy cases runnable. Standard keeps the
|
||||
# historical client-only implicit default; isolated-full means
|
||||
# exactly what it says and includes every active test type.
|
||||
if (
|
||||
not bypass_risk
|
||||
and not args.test_type
|
||||
and not args.suite
|
||||
and test_type(case) == "integration"
|
||||
):
|
||||
continue
|
||||
if args.driver and case.get("driver") != args.driver:
|
||||
continue
|
||||
automation = str(case.get("automation") or "automated")
|
||||
@@ -796,7 +846,7 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
|
||||
except SystemExit:
|
||||
skipped.append({"id": cid, "reason": f"未知风险 {risk}"})
|
||||
continue
|
||||
if r_i > max_i:
|
||||
if not bypass_risk and r_i > max_i:
|
||||
skipped.append({"id": cid, "reason": f"risk={risk} 超过上限 {max_risk}"})
|
||||
continue
|
||||
selected.append(public_case(case))
|
||||
@@ -808,7 +858,22 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
|
||||
)
|
||||
return 2
|
||||
|
||||
selected, together = expand_run_together(meta, cases, selected, skipped, max_i, automation_filter)
|
||||
selected, together = expand_run_together(
|
||||
meta,
|
||||
cases,
|
||||
selected,
|
||||
skipped,
|
||||
max_i,
|
||||
automation_filter,
|
||||
bypass_risk=bypass_risk,
|
||||
)
|
||||
|
||||
# The historical key is kept in the JSON, but the gate now validates both
|
||||
# test types before any caller can start a phase.
|
||||
journey_gate = journey_preflight(selected, sessions=meta.get("sessions") or {})
|
||||
if journey_gate["status"] == "blocked":
|
||||
print(json.dumps({"preflight": journey_gate, "journey_preflight": journey_gate}, ensure_ascii=False, indent=2))
|
||||
return 2
|
||||
|
||||
preflight = mobile_preflight(
|
||||
root,
|
||||
@@ -835,11 +900,18 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
|
||||
json.dumps(
|
||||
{
|
||||
"max_risk": max_risk,
|
||||
"execution_profile": execution_profile,
|
||||
"risk_filter": "disabled" if bypass_risk else "enforced",
|
||||
"automation": automation_filter,
|
||||
"count": len(selected),
|
||||
"vars": vars_map,
|
||||
"artifacts": meta.get("artifacts") or {},
|
||||
"preflight": preflight,
|
||||
"journey_preflight": journey_gate,
|
||||
"test_types": {
|
||||
"journey": sum(1 for case in selected if test_type(case) == "journey"),
|
||||
"integration": sum(1 for case in selected if test_type(case) == "integration"),
|
||||
},
|
||||
"skipped": skipped,
|
||||
"run_together": together,
|
||||
"status_probe": status_probes,
|
||||
@@ -910,6 +982,12 @@ def validate(root: Path) -> list[str]:
|
||||
default_max = meta.get("default_max_risk")
|
||||
if default_max not in risks:
|
||||
errors.append(f"default_max_risk={default_max!r} 不在 risks 中")
|
||||
profile = meta.get("execution_profile", "standard")
|
||||
if profile not in EXECUTION_PROFILES:
|
||||
errors.append(
|
||||
f"execution_profile={profile!r} 非法,只能是 {', '.join(EXECUTION_PROFILES)}"
|
||||
)
|
||||
isolated_full = profile == "isolated-full"
|
||||
gates = meta.get("gates") or {}
|
||||
if gates and not isinstance(gates, dict):
|
||||
errors.append("catalog.gates 必须是 mapping")
|
||||
@@ -922,6 +1000,28 @@ def validate(root: Path) -> list[str]:
|
||||
if not isinstance(sessions, dict):
|
||||
errors.append("catalog.sessions 必须是 mapping")
|
||||
sessions = {}
|
||||
client_suites = meta.get("client_suites")
|
||||
if client_suites is not None:
|
||||
if not isinstance(client_suites, dict):
|
||||
errors.append("catalog.client_suites 必须是 mapping")
|
||||
else:
|
||||
unknown = sorted(set(client_suites) - set(CLIENT_SUITES))
|
||||
if unknown:
|
||||
errors.append("catalog.client_suites 含非法套件: " + ", ".join(unknown))
|
||||
missing_suites = sorted(set(CLIENT_SUITES) - set(client_suites))
|
||||
if missing_suites:
|
||||
errors.append("catalog.client_suites 缺少标准套件: " + ", ".join(missing_suites))
|
||||
integration_suites = meta.get("integration_suites")
|
||||
if integration_suites is not None:
|
||||
if not isinstance(integration_suites, dict):
|
||||
errors.append("catalog.integration_suites 必须是 mapping")
|
||||
else:
|
||||
unknown = sorted(set(integration_suites) - set(INTEGRATION_SUITES))
|
||||
if unknown:
|
||||
errors.append("catalog.integration_suites 含非法套件: " + ", ".join(unknown))
|
||||
missing_suites = sorted(set(INTEGRATION_SUITES) - set(integration_suites))
|
||||
if missing_suites:
|
||||
errors.append("catalog.integration_suites 缺少标准套件: " + ", ".join(missing_suites))
|
||||
artifacts = meta.get("artifacts")
|
||||
if artifacts is not None:
|
||||
if not isinstance(artifacts, dict):
|
||||
@@ -998,6 +1098,7 @@ def validate(root: Path) -> list[str]:
|
||||
errors.append(f"{where}: category={cat} 与文件 category={file_cat} 不一致")
|
||||
if cat and cat_ids and cat not in cat_ids:
|
||||
errors.append(f"{where}: category={cat} 不在 catalog.categories")
|
||||
errors.extend(contract_errors(case, sessions=sessions))
|
||||
if case.get("lifecycle", "active") not in LIFECYCLES:
|
||||
errors.append(f"{where}: 非法 lifecycle")
|
||||
if case.get("tag") not in TAGS:
|
||||
@@ -1013,8 +1114,17 @@ def validate(root: Path) -> list[str]:
|
||||
if case.get("risk") in (None, "readonly"):
|
||||
errors.append(f"{where}: risk=readonly 不要填 gate")
|
||||
risk = case.get("risk")
|
||||
if risk and risk != "readonly" and risk not in authorize and not gate:
|
||||
if risk and risk != "readonly" and risk not in authorize and not gate and not isolated_full:
|
||||
errors.append(f"{where}: {risk} 需要 authorize.{risk} 或 gate")
|
||||
if isolated_full and case.get("lifecycle", "active") == "active" and risk != "readonly":
|
||||
cleanup = case.get("cleanup")
|
||||
cleanup_ok = isinstance(cleanup, dict) and (
|
||||
bool(cleanup.get("steps"))
|
||||
or bool(str(cleanup.get("run") or "").strip())
|
||||
or (cleanup.get("required") is False and bool(str(cleanup.get("reason") or "").strip()))
|
||||
)
|
||||
if not cleanup_ok:
|
||||
errors.append(f"{where}: isolated-full 的写入型用例必须声明 cleanup/rollback steps/run")
|
||||
automation = str(case.get("automation") or "automated")
|
||||
if automation not in AUTOMATIONS:
|
||||
errors.append(f"{where}: 非法 automation={case.get('automation')}")
|
||||
@@ -1146,7 +1256,19 @@ def cmd_next_id(root: Path, category: str) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def authorized(vars_map: dict[str, str], meta: dict[str, Any], case: dict[str, Any]) -> bool:
|
||||
def authorized(
|
||||
vars_map: dict[str, str],
|
||||
meta: dict[str, Any],
|
||||
case: dict[str, Any],
|
||||
*,
|
||||
execution_profile: str | None = None,
|
||||
) -> bool:
|
||||
profile = resolve_execution_profile(meta, execution_profile)
|
||||
if profile == "isolated-full":
|
||||
# AUTHORIZE_* and gate variables are intentionally not run-time
|
||||
# switches for an isolated resettable environment. Capability and
|
||||
# cleanup preflight are the safety boundary for this profile.
|
||||
return True
|
||||
risk = str(case.get("risk") or "readonly")
|
||||
risks = list(meta.get("risks") or ["readonly"])
|
||||
lowest = risks[0] if risks else "readonly"
|
||||
@@ -1166,6 +1288,7 @@ def authorized(vars_map: dict[str, str], meta: dict[str, Any], case: dict[str, A
|
||||
|
||||
def cmd_run_script(root: Path, case_id: str) -> int:
|
||||
meta = load_catalog_meta(root)
|
||||
execution_profile = resolve_execution_profile(meta)
|
||||
vars_map = resolve_vars(root, meta)
|
||||
case = next((c for c in load_all_cases(root) if c.get("id") == case_id), None)
|
||||
if not case:
|
||||
@@ -1203,10 +1326,10 @@ def cmd_run_script(root: Path, case_id: str) -> int:
|
||||
)
|
||||
return 2
|
||||
risk = str(case.get("risk") or "readonly")
|
||||
if not authorized(vars_map, meta, case):
|
||||
if not authorized(vars_map, meta, case, execution_profile=execution_profile):
|
||||
print(
|
||||
json.dumps(
|
||||
{"id": public_id, "result": "skipped", "evidence": f"风险 {risk} 未授权"},
|
||||
{"id": public_id, "result": "skipped", "evidence": f"风险 {risk} 未授权", "execution_profile": execution_profile},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
)
|
||||
@@ -1255,6 +1378,7 @@ def cmd_run_script(root: Path, case_id: str) -> int:
|
||||
{
|
||||
"id": public_id,
|
||||
"result": "executed",
|
||||
"execution_profile": execution_profile,
|
||||
"exit": proc.returncode,
|
||||
"evidence": evidence,
|
||||
},
|
||||
@@ -1275,7 +1399,22 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
p_sel.add_argument("--category")
|
||||
p_sel.add_argument("--id", action="append", dest="ids", default=[])
|
||||
p_sel.add_argument("--tag", choices=TAGS)
|
||||
p_sel.add_argument(
|
||||
"--suite", "--client-suite", dest="suite",
|
||||
choices=[*CLIENT_SUITES, *INTEGRATION_SUITES],
|
||||
help="客户端或集成测试套件;不传时默认不选 integration",
|
||||
)
|
||||
p_sel.add_argument(
|
||||
"--test-type", choices=TEST_TYPES, default=None,
|
||||
help="按顶层 test_type 筛选;integration 必须显式选择",
|
||||
)
|
||||
p_sel.add_argument("--max-risk")
|
||||
p_sel.add_argument(
|
||||
"--execution-profile",
|
||||
choices=EXECUTION_PROFILES,
|
||||
default=None,
|
||||
help="执行策略;isolated-full 在隔离可重置环境中不按 risk/authorize 筛选",
|
||||
)
|
||||
p_sel.add_argument("--driver", choices=DRIVERS)
|
||||
p_sel.add_argument(
|
||||
"--automation",
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build structured tester issues and the development feedback route.
|
||||
|
||||
This module only writes the run contract. Updating a remote PR/issue or
|
||||
notifying a dispatcher remains an explicit caller-side integration step; no
|
||||
network mutation is hidden in the tester.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from copy import deepcopy
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Iterable, Mapping
|
||||
|
||||
|
||||
ISSUE_TYPES = ("product", "test")
|
||||
TRIGGERS = ("pr", "requirement", "bug")
|
||||
ISSUE_RESULTS = frozenset({"fail", "environment-fail"})
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def issue_type_for(result: str) -> str:
|
||||
if result == "fail":
|
||||
return "product"
|
||||
if result == "environment-fail":
|
||||
return "test"
|
||||
raise ValueError(f"只有 fail/environment-fail 可以生成 issue: {result}")
|
||||
|
||||
|
||||
def _value(row: Mapping[str, Any], key: str, defaults: Mapping[str, Any]) -> Any:
|
||||
value = row.get(key)
|
||||
return value if value not in (None, "") else defaults.get(key, "")
|
||||
|
||||
|
||||
def build_issue(
|
||||
row: Mapping[str, Any],
|
||||
*,
|
||||
defaults: Mapping[str, Any] | None = None,
|
||||
issue_type: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Create the stable issue shape required by the run report."""
|
||||
defaults = defaults or {}
|
||||
result = str(row.get("result") or "")
|
||||
if result not in ISSUE_RESULTS:
|
||||
raise ValueError(f"不能为 {result!r} 结果创建 defect: {row.get('id')}")
|
||||
selected_type = issue_type or row.get("issue_type") or issue_type_for(result)
|
||||
if selected_type not in ISSUE_TYPES:
|
||||
raise ValueError(f"issue_type 必须是 product/test: {selected_type}")
|
||||
testcase = str(row.get("testcase") or row.get("id") or "")
|
||||
if not testcase:
|
||||
raise ValueError("issue 缺少 testcase")
|
||||
return {
|
||||
"id": str(row.get("issue_id") or f"TEST-{testcase}-{result}"),
|
||||
"testcase": testcase,
|
||||
"issue_type": selected_type,
|
||||
"result": result,
|
||||
"phase": _value(row, "phase", defaults),
|
||||
"expected": _value(row, "expected", defaults),
|
||||
"observed": _value(row, "observed", defaults),
|
||||
"artifact_hash": _value(row, "artifact_hash", defaults),
|
||||
"environment": _value(row, "environment", defaults),
|
||||
"evidence_path": _value(row, "evidence_path", defaults),
|
||||
"reproduction": _value(row, "reproduction", defaults),
|
||||
"next_step": _value(row, "next_step", defaults),
|
||||
"created_at": str(row.get("created_at") or _now()),
|
||||
}
|
||||
|
||||
|
||||
def collect_issues(
|
||||
results: Iterable[Mapping[str, Any]],
|
||||
*,
|
||||
defaults: Mapping[str, Any] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
issues: list[dict[str, Any]] = []
|
||||
for row in results:
|
||||
if not isinstance(row, Mapping) or str(row.get("result") or "") not in ISSUE_RESULTS:
|
||||
continue
|
||||
explicit = row.get("issue")
|
||||
source = dict(row)
|
||||
if isinstance(explicit, Mapping):
|
||||
source.update(explicit)
|
||||
issues.append(build_issue(source, defaults=defaults))
|
||||
return issues
|
||||
|
||||
|
||||
def feedback_plan(
|
||||
trigger: str,
|
||||
*,
|
||||
target: str = "",
|
||||
related_issue: str = "",
|
||||
dispatcher: str = "dispatcher",
|
||||
) -> dict[str, Any]:
|
||||
"""Describe the required handoff without performing remote mutations."""
|
||||
if trigger not in TRIGGERS:
|
||||
raise ValueError(f"trigger 必须是 {', '.join(TRIGGERS)}")
|
||||
if trigger == "pr":
|
||||
actions = [
|
||||
{"action": "update_pr", "target": target, "status": "required"},
|
||||
{"action": "update_related_issue", "target": related_issue, "status": "required"},
|
||||
{"action": "notify_dispatcher", "target": dispatcher, "status": "required"},
|
||||
{"action": "reassign_development", "target": dispatcher, "status": "pending"},
|
||||
]
|
||||
route = "pr-update-issue-dispatcher-reassign"
|
||||
else:
|
||||
actions = [
|
||||
{"action": "return_to_development", "target": target or trigger, "status": "required"},
|
||||
]
|
||||
route = "development"
|
||||
return {
|
||||
"trigger": trigger,
|
||||
"route": route,
|
||||
"target": target,
|
||||
"related_issue": related_issue,
|
||||
"actions": actions,
|
||||
"remote_mutation": "caller-required",
|
||||
"created_at": _now(),
|
||||
}
|
||||
|
||||
|
||||
def attach_feedback(
|
||||
run: Mapping[str, Any],
|
||||
*,
|
||||
trigger: str | None = None,
|
||||
target: str = "",
|
||||
related_issue: str = "",
|
||||
dispatcher: str = "dispatcher",
|
||||
defaults: Mapping[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Return a new run mapping with results, defects and routing preserved."""
|
||||
output = deepcopy(dict(run))
|
||||
results = output.get("results")
|
||||
if not isinstance(results, list):
|
||||
raise ValueError("run 缺少 results 列表")
|
||||
issues = collect_issues(results, defaults=defaults)
|
||||
output["defects"] = issues
|
||||
# Compatibility name for consumers that call the same structure
|
||||
# ``test-issues``. It is deliberately a copy, not a second source of truth.
|
||||
output["test_issues"] = deepcopy(issues)
|
||||
if trigger:
|
||||
output["feedback"] = feedback_plan(
|
||||
trigger,
|
||||
target=target,
|
||||
related_issue=related_issue,
|
||||
dispatcher=dispatcher,
|
||||
)
|
||||
return output
|
||||
|
||||
@@ -13,6 +13,8 @@ import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from journey import suite as case_suite, test_type
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
@@ -22,6 +24,10 @@ except ImportError: # pragma: no cover
|
||||
FIELDS = (
|
||||
"category",
|
||||
"id",
|
||||
"test_type",
|
||||
"suite",
|
||||
"client_suite",
|
||||
"platform",
|
||||
"maps_to",
|
||||
"title",
|
||||
"what",
|
||||
@@ -156,10 +162,17 @@ def export_cases_csv(root: Path, out: Path | None = None) -> Path:
|
||||
writer.writeheader()
|
||||
for case in cases:
|
||||
cid = str(case.get("id") or "")
|
||||
client_suite_value = case.get("client_suite") or (
|
||||
case.get("suite") if test_type(case) == "journey" else ""
|
||||
)
|
||||
writer.writerow(
|
||||
{
|
||||
"category": case.get("category") or "",
|
||||
"id": cid,
|
||||
"test_type": test_type(case),
|
||||
"suite": case_suite(case),
|
||||
"client_suite": client_suite_value or "",
|
||||
"platform": one_line(case.get("platform")),
|
||||
"maps_to": case.get("maps_to") or "",
|
||||
"title": case.get("title") or "",
|
||||
"what": one_line(case.get("what")),
|
||||
|
||||
@@ -18,6 +18,7 @@ except ImportError: # pragma: no cover
|
||||
raise SystemExit(2)
|
||||
|
||||
from generate_cases_csv import load_cases, one_line, resolve_root, tester_dir
|
||||
from journey import suite as case_suite, test_type
|
||||
|
||||
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
||||
RUN_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
|
||||
@@ -25,6 +26,10 @@ EXECUTED_YES = frozenset({"pass", "fail", "environment-fail"})
|
||||
EXECUTED_NO = frozenset({"skipped", "not-run"})
|
||||
FIELDS = (
|
||||
"id",
|
||||
"test_type",
|
||||
"suite",
|
||||
"client_suite",
|
||||
"platform",
|
||||
"result",
|
||||
"executed",
|
||||
"evidence",
|
||||
@@ -35,6 +40,16 @@ FIELDS = (
|
||||
"risk",
|
||||
"driver",
|
||||
"maps_to",
|
||||
"blocked_by",
|
||||
"issue_id",
|
||||
"issue_type",
|
||||
"expected",
|
||||
"observed",
|
||||
"artifact_hash",
|
||||
"environment",
|
||||
"evidence_path",
|
||||
"reproduction",
|
||||
"next_step",
|
||||
)
|
||||
|
||||
|
||||
@@ -107,6 +122,23 @@ def case_index(root: Path) -> dict[str, dict[str, Any]]:
|
||||
return out
|
||||
|
||||
|
||||
def issue_index(run: dict[str, Any]) -> dict[str, dict[str, Any]]:
|
||||
"""Index canonical defects and compatibility test_issues by testcase."""
|
||||
indexed: dict[str, dict[str, Any]] = {}
|
||||
raw = run.get("defects")
|
||||
if not isinstance(raw, list):
|
||||
raw = run.get("test_issues")
|
||||
if not isinstance(raw, list):
|
||||
return indexed
|
||||
for issue in raw:
|
||||
if not isinstance(issue, dict):
|
||||
continue
|
||||
testcase = str(issue.get("testcase") or issue.get("id") or "")
|
||||
if testcase:
|
||||
indexed.setdefault(testcase, issue)
|
||||
return indexed
|
||||
|
||||
|
||||
def export_run_csv(
|
||||
root: Path,
|
||||
run_path: Path | None = None,
|
||||
@@ -118,6 +150,7 @@ def export_run_csv(
|
||||
path = path.resolve()
|
||||
run = load_run(path)
|
||||
cases = case_index(root)
|
||||
issues = issue_index(run)
|
||||
dest = out or path.with_suffix(".csv")
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
with dest.open("w", encoding="utf-8-sig", newline="") as handle:
|
||||
@@ -129,9 +162,19 @@ def export_run_csv(
|
||||
cid = str(item["id"])
|
||||
result = str(item.get("result") or "")
|
||||
case = cases.get(cid) or {}
|
||||
issue = dict(issues.get(cid, {}))
|
||||
if isinstance(item.get("issue"), dict):
|
||||
issue.update(item["issue"])
|
||||
client_suite_value = case.get("client_suite") or (
|
||||
case.get("suite") if test_type(case) == "journey" else ""
|
||||
)
|
||||
writer.writerow(
|
||||
{
|
||||
"id": cid,
|
||||
"test_type": test_type(case),
|
||||
"suite": case_suite(case),
|
||||
"client_suite": client_suite_value or "",
|
||||
"platform": one_line(case.get("platform")),
|
||||
"result": result,
|
||||
"executed": executed_cell(result),
|
||||
"evidence": one_line(item.get("evidence")),
|
||||
@@ -142,6 +185,16 @@ def export_run_csv(
|
||||
"risk": case.get("risk") or "",
|
||||
"driver": case.get("driver") or "",
|
||||
"maps_to": case.get("maps_to") or "",
|
||||
"blocked_by": one_line(item.get("blocked_by")),
|
||||
"issue_id": issue.get("id") or item.get("issue_id") or "",
|
||||
"issue_type": issue.get("issue_type") or item.get("issue_type") or "",
|
||||
"expected": one_line(issue.get("expected") or item.get("expected")),
|
||||
"observed": one_line(issue.get("observed") or item.get("observed")),
|
||||
"artifact_hash": issue.get("artifact_hash") or item.get("artifact_hash") or "",
|
||||
"environment": one_line(issue.get("environment") or item.get("environment")),
|
||||
"evidence_path": one_line(issue.get("evidence_path") or item.get("evidence_path")),
|
||||
"reproduction": one_line(issue.get("reproduction") or item.get("reproduction")),
|
||||
"next_step": one_line(issue.get("next_step") or item.get("next_step")),
|
||||
}
|
||||
)
|
||||
return dest
|
||||
|
||||
@@ -0,0 +1,290 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Shared client-journey contract and result lifecycle helpers.
|
||||
|
||||
The mobile/browser driver still owns the real UI interaction. This module
|
||||
owns the parts that must be identical for every client catalog: contract
|
||||
validation, preflight blocking, and result propagation. It deliberately
|
||||
returns the existing tester result vocabulary instead of introducing a second
|
||||
report format.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Iterable, Mapping
|
||||
|
||||
|
||||
TEST_TYPES = ("journey", "integration")
|
||||
CLIENT_SUITES = ("client-smoke", "client-regression", "safety")
|
||||
INTEGRATION_SUITES = ("api-contract", "authorization", "build-contract")
|
||||
PLATFORMS = ("android", "ios")
|
||||
FAILURE_POLICIES = ("stop", "continue")
|
||||
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
||||
|
||||
|
||||
def client_suite(case: Mapping[str, Any]) -> str:
|
||||
"""Return the canonical client suite, accepting ``suite`` as a migration alias."""
|
||||
value = case.get("client_suite")
|
||||
alias = case.get("suite")
|
||||
if value and alias and str(value) != str(alias):
|
||||
return ""
|
||||
return str(value or alias or "")
|
||||
|
||||
|
||||
def integration_suite(case: Mapping[str, Any]) -> str:
|
||||
"""Return the integration suite, accepting ``suite`` during migration."""
|
||||
value = case.get("integration_suite")
|
||||
alias = case.get("suite")
|
||||
if value and alias and str(value) != str(alias):
|
||||
return ""
|
||||
return str(value or alias or "")
|
||||
|
||||
|
||||
def test_type(case: Mapping[str, Any]) -> str:
|
||||
"""Return an explicit type or infer one for an old catalog entry."""
|
||||
declared = str(case.get("test_type") or "").strip().lower()
|
||||
if declared:
|
||||
return declared
|
||||
if client_suite(case) or case.get("journey") is not None:
|
||||
return "journey"
|
||||
if integration_suite(case):
|
||||
return "integration"
|
||||
return ""
|
||||
|
||||
|
||||
def suite(case: Mapping[str, Any]) -> str:
|
||||
"""Return the report/selection suite for either test type."""
|
||||
kind = test_type(case)
|
||||
return client_suite(case) if kind == "journey" else integration_suite(case)
|
||||
|
||||
|
||||
def normalize_platforms(value: Any) -> list[str]:
|
||||
if isinstance(value, str):
|
||||
values = [value]
|
||||
elif isinstance(value, (list, tuple)):
|
||||
values = list(value)
|
||||
else:
|
||||
return []
|
||||
return [str(item).strip().lower() for item in values if str(item).strip()]
|
||||
|
||||
|
||||
def _non_empty_list(value: Any) -> bool:
|
||||
return isinstance(value, list) and bool(value) and all(str(item).strip() for item in value)
|
||||
|
||||
|
||||
def _has_account(fixture: Mapping[str, Any]) -> bool:
|
||||
account = fixture.get("account")
|
||||
if isinstance(account, Mapping):
|
||||
return bool(str(account.get("role") or account.get("username") or "").strip())
|
||||
return bool(str(account or "").strip())
|
||||
|
||||
|
||||
def contract_errors(case: Mapping[str, Any], sessions: Mapping[str, Any] | None = None) -> list[str]:
|
||||
"""Validate a client journey without requiring app-specific field names.
|
||||
|
||||
A case becomes a client journey when it declares ``client_suite`` (or the
|
||||
temporary ``suite`` alias). Legacy non-client cases keep the existing
|
||||
schema. Once opted in, all lifecycle inputs are explicit so a missing
|
||||
fixture/cleanup can stop the run before any case result is fabricated.
|
||||
"""
|
||||
kind = test_type(case)
|
||||
declared_kind = str(case.get("test_type") or "").strip().lower()
|
||||
if declared_kind and declared_kind not in TEST_TYPES:
|
||||
return [f"{case.get('id') or '?'}: test_type 必须是 journey 或 integration"]
|
||||
if not kind:
|
||||
return []
|
||||
if kind == "integration":
|
||||
return integration_contract_errors(case)
|
||||
|
||||
suite_name = client_suite(case)
|
||||
journey = case.get("journey")
|
||||
errors: list[str] = []
|
||||
cid = str(case.get("id") or "?")
|
||||
where = f"{cid}: client journey"
|
||||
if case.get("client_suite") and case.get("suite") and str(case["client_suite"]) != str(case["suite"]):
|
||||
errors.append(f"{where}: client_suite 与 suite 不一致")
|
||||
if suite_name not in CLIENT_SUITES:
|
||||
errors.append(f"{where}: client_suite 必须是 {', '.join(CLIENT_SUITES)}")
|
||||
|
||||
if not isinstance(journey, Mapping):
|
||||
errors.append(f"{where}: journey 必须是 mapping")
|
||||
journey = {}
|
||||
for key in ("goal", "entry_state", "final_state"):
|
||||
if not str(journey.get(key) or "").strip():
|
||||
errors.append(f"{where}: journey.{key} 必填")
|
||||
if not _non_empty_list(journey.get("actions")):
|
||||
errors.append(f"{where}: journey.actions 必须是非空列表")
|
||||
if not _non_empty_list(journey.get("assertions")):
|
||||
errors.append(f"{where}: journey.assertions 必须是非空列表")
|
||||
policy = str(journey.get("failure_policy") or "")
|
||||
if policy not in FAILURE_POLICIES:
|
||||
errors.append(f"{where}: journey.failure_policy 必须是 stop 或 continue")
|
||||
|
||||
fixture = case.get("fixture")
|
||||
if not isinstance(fixture, Mapping):
|
||||
errors.append(f"{where}: fixture 缺失,必须在 preflight 前声明")
|
||||
elif not _has_account(fixture):
|
||||
errors.append(f"{where}: fixture.account 缺失,必须声明测试账号或角色")
|
||||
|
||||
cleanup = case.get("cleanup")
|
||||
if not isinstance(cleanup, Mapping):
|
||||
errors.append(f"{where}: cleanup 缺失,必须在 preflight 前声明")
|
||||
else:
|
||||
steps = cleanup.get("steps")
|
||||
command = cleanup.get("run")
|
||||
explicitly_noop = cleanup.get("required") is False and str(cleanup.get("reason") or "").strip()
|
||||
if not explicitly_noop and not _non_empty_list(steps) and not str(command or "").strip():
|
||||
errors.append(f"{where}: cleanup 必须有非空 steps/run;只读用例也要显式声明 reason")
|
||||
|
||||
platforms = normalize_platforms(case.get("platform"))
|
||||
if not platforms:
|
||||
errors.append(f"{where}: platform 必须声明 android/ios")
|
||||
elif any(platform not in PLATFORMS for platform in platforms):
|
||||
errors.append(f"{where}: platform 只能是 {', '.join(PLATFORMS)}")
|
||||
elif len(platforms) != len(set(platforms)):
|
||||
errors.append(f"{where}: platform 不能重复")
|
||||
|
||||
driver = case.get("driver")
|
||||
if driver not in ("mobile", "browser"):
|
||||
errors.append(f"{where}: 客户端旅程 driver 必须是 mobile 或 browser")
|
||||
if driver and driver in ("mobile", "browser"):
|
||||
session = case.get("session")
|
||||
if not str(session or "").strip():
|
||||
errors.append(f"{where}: session 必须声明可复用客户端会话")
|
||||
elif sessions is not None and session not in sessions:
|
||||
errors.append(f"{where}: session={session} 未在 catalog.sessions 声明")
|
||||
if suite_name == "safety" and str(case.get("tag") or "") == "smoke":
|
||||
errors.append(f"{where}: safety 不得标记为 smoke")
|
||||
return errors
|
||||
|
||||
|
||||
def integration_contract_errors(case: Mapping[str, Any]) -> list[str]:
|
||||
"""Validate the non-client half of the tester catalog.
|
||||
|
||||
Integration cases deliberately do not require a browser/mobile session. A
|
||||
``how`` block is the legacy call declaration; newer catalogs may use
|
||||
``request``/``input``/``call`` or an ``integration`` mapping as well.
|
||||
``expected`` remains the common protocol/data assertion list.
|
||||
"""
|
||||
cid = str(case.get("id") or "?")
|
||||
where = f"{cid}: integration"
|
||||
errors: list[str] = []
|
||||
value = case.get("integration_suite")
|
||||
alias = case.get("suite")
|
||||
if value and alias and str(value) != str(alias):
|
||||
errors.append(f"{where}: integration_suite 与 suite 不一致")
|
||||
suite_name = integration_suite(case)
|
||||
if suite_name not in INTEGRATION_SUITES:
|
||||
errors.append(
|
||||
f"{where}: integration_suite 必须是 {', '.join(INTEGRATION_SUITES)}"
|
||||
)
|
||||
|
||||
fixture = case.get("fixture")
|
||||
if not isinstance(fixture, Mapping) or not fixture:
|
||||
errors.append(f"{where}: fixture 缺失,必须在 preflight 前声明")
|
||||
|
||||
declared_call = any(
|
||||
key in case and case.get(key) not in (None, "", [], {})
|
||||
for key in ("request", "input", "call", "integration")
|
||||
)
|
||||
how = case.get("how")
|
||||
if not declared_call and not isinstance(how, Mapping):
|
||||
errors.append(f"{where}: 必须声明 request/input/call/integration 或 how")
|
||||
|
||||
# Integration writes need an explicit rollback/cleanup contract. Read-only
|
||||
# probes may state why cleanup is unnecessary, matching test-preflight.
|
||||
risk = str(case.get("risk") or "")
|
||||
cleanup = case.get("cleanup")
|
||||
cleanup_steps = isinstance(cleanup, Mapping) and (
|
||||
bool(cleanup.get("steps")) or bool(str(cleanup.get("run") or "").strip())
|
||||
)
|
||||
cleanup_noop = isinstance(cleanup, Mapping) and cleanup.get("required") is False and bool(
|
||||
str(cleanup.get("reason") or "").strip()
|
||||
)
|
||||
if risk != "readonly" and not (cleanup_steps or cleanup_noop):
|
||||
errors.append(f"{where}: 写入型 integration 必须声明 cleanup/rollback steps/run")
|
||||
elif cleanup is not None and not (cleanup_steps or cleanup_noop):
|
||||
errors.append(f"{where}: cleanup 必须有 steps/run 或 required=false + reason")
|
||||
|
||||
forbidden = []
|
||||
if case.get("client_suite") or case.get("journey") is not None:
|
||||
forbidden.append("client_suite/journey")
|
||||
if case.get("session") or case.get("platform"):
|
||||
forbidden.append("session/platform")
|
||||
if forbidden:
|
||||
errors.append(f"{where}: integration 不应声明 " + ", ".join(forbidden))
|
||||
return errors
|
||||
|
||||
|
||||
def preflight(cases: Iterable[Mapping[str, Any]], sessions: Mapping[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""Return a contract preflight result; no per-case result is created on block."""
|
||||
cases = list(cases)
|
||||
errors: list[str] = []
|
||||
for case in cases:
|
||||
errors.extend(contract_errors(case, sessions=sessions))
|
||||
return {
|
||||
"required": bool(errors or any(client_suite(case) for case in cases)),
|
||||
"status": "blocked" if errors else "ready",
|
||||
"failure_class": "contract" if errors else None,
|
||||
"errors": errors,
|
||||
}
|
||||
|
||||
|
||||
def classify_outcome(
|
||||
*,
|
||||
ok: bool,
|
||||
environment_ready: bool = True,
|
||||
phase: str = "journey",
|
||||
) -> str:
|
||||
"""Classify one observed phase using the tester's five result states.
|
||||
|
||||
A valid test account is part of the client journey: a failed login is a
|
||||
product ``fail``. Only an unavailable environment is ``environment-fail``.
|
||||
"""
|
||||
if not environment_ready:
|
||||
return "environment-fail"
|
||||
if not ok:
|
||||
return "fail"
|
||||
return "pass"
|
||||
|
||||
|
||||
def propagate_results(
|
||||
case_ids: Iterable[str],
|
||||
observed: Mapping[str, str],
|
||||
*,
|
||||
skipped: Mapping[str, str] | None = None,
|
||||
continue_on_fail: bool = False,
|
||||
policies: Mapping[str, str] | None = None,
|
||||
blocked_by: Mapping[str, Iterable[str]] | None = None,
|
||||
) -> list[dict[str, str]]:
|
||||
"""Build ordered result rows and mark cases after a blocking failure not-run.
|
||||
|
||||
``skipped`` is intentional selection (risk/lifecycle/assumption); it is
|
||||
never converted to ``not-run``. A product or environment failure stops
|
||||
subsequent cases unless the caller explicitly opts into continuation.
|
||||
"""
|
||||
skipped = skipped or {}
|
||||
policies = policies or {}
|
||||
blocked_by = blocked_by or {}
|
||||
rows: list[dict[str, str]] = []
|
||||
blocking_case = ""
|
||||
for raw_id in case_ids:
|
||||
case_id = str(raw_id)
|
||||
if case_id in skipped:
|
||||
rows.append({"id": case_id, "result": "skipped", "notes": skipped[case_id]})
|
||||
continue
|
||||
if blocking_case:
|
||||
rows.append({
|
||||
"id": case_id,
|
||||
"result": "not-run",
|
||||
"notes": "前置用例失败,未轮到执行",
|
||||
"blocked_by": ",".join(blocked_by.get(case_id) or (blocking_case,)),
|
||||
})
|
||||
continue
|
||||
result = observed.get(case_id, "not-run")
|
||||
if result not in RESULTS:
|
||||
raise ValueError(f"未知结果: {case_id}={result}")
|
||||
rows.append({"id": case_id, "result": result})
|
||||
policy = policies.get(case_id)
|
||||
if result in ("fail", "environment-fail") and not (continue_on_fail or policy == "continue"):
|
||||
blocking_case = case_id
|
||||
return rows
|
||||
@@ -11,6 +11,7 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import plistlib
|
||||
@@ -33,6 +34,7 @@ from typing import Any
|
||||
|
||||
COMMIT = re.compile(r"^[0-9a-fA-F]{40}$")
|
||||
PLATFORMS = {"android", "ios"}
|
||||
PREFLIGHT_ROOT = Path(__file__).resolve().parents[2] / "test-preflight" / "scripts" / "preflight.py"
|
||||
|
||||
|
||||
class EnvironmentFailure(RuntimeError):
|
||||
@@ -52,6 +54,7 @@ class CommandResult:
|
||||
@dataclass
|
||||
class RuntimeContext:
|
||||
root: Path
|
||||
bindings: dict[str, str] = field(default_factory=dict)
|
||||
retries: dict[str, int] = field(default_factory=lambda: {"download": 3, "start": 3, "install": 3})
|
||||
evidence: list[dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
@@ -124,6 +127,120 @@ def parse_artifacts(values: list[str]) -> dict[str, str]:
|
||||
return pairs
|
||||
|
||||
|
||||
def parse_bindings(values: list[str], requested: list[str]) -> dict[str, str]:
|
||||
"""Resolve one tester binding per requested platform.
|
||||
|
||||
A single legacy ``agent`` mapping cannot safely represent a multi-platform
|
||||
run. Keep the platform distinction in the command contract all the way to
|
||||
agent-runtime instead of treating binding as report metadata.
|
||||
"""
|
||||
pairs: dict[str, str] = {}
|
||||
for raw in values:
|
||||
platform, sep, value = raw.partition("=")
|
||||
platform = platform.strip().lower()
|
||||
value = value.strip()
|
||||
if not sep or platform not in PLATFORMS or not value or not re.fullmatch(r"[a-z][a-z0-9-]*(?:\.[a-z][a-z0-9-]*)+", value):
|
||||
raise ArtifactFailure(f"--binding 格式必须是 android=tester.<name> 或 ios=tester.<name>: {raw}")
|
||||
if platform in pairs:
|
||||
raise ArtifactFailure(f"--binding 重复平台: {platform}")
|
||||
pairs[platform] = value
|
||||
resolved = {platform: pairs.get(platform, f"tester.{platform}") for platform in requested}
|
||||
if len(set(resolved.values())) != len(resolved) and len(resolved) > 1:
|
||||
raise ArtifactFailure("多平台运行不能复用同一个 tester binding;请分别配置 tester.android/tester.ios")
|
||||
return resolved
|
||||
|
||||
|
||||
def load_agent_runtime():
|
||||
candidates = [
|
||||
Path.home() / ".codex" / "skills" / "agent-runtime" / "scripts" / "agent.py",
|
||||
Path.home() / ".pouch" / "skills" / "agent-runtime" / "scripts" / "agent.py",
|
||||
Path(__file__).resolve().parents[2] / "agent-runtime" / "scripts" / "agent.py",
|
||||
]
|
||||
runtime_path = next((item for item in candidates if item.is_file()), None)
|
||||
if runtime_path is None:
|
||||
raise EnvironmentFailure("找不到共享 agent-runtime: " + ", ".join(str(item) for item in candidates))
|
||||
spec = importlib.util.spec_from_file_location("pouch_agent_runtime_for_mobile_bundle", runtime_path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise EnvironmentFailure(f"无法加载共享 agent-runtime: {runtime_path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def binding_location(root: Path, binding_id: str) -> dict[str, Any]:
|
||||
runtime = load_agent_runtime()
|
||||
try:
|
||||
value = runtime.binding(runtime.load_config(root), binding_id)
|
||||
except (ValueError, OSError) as exc:
|
||||
raise EnvironmentFailure(f"无法解析 {binding_id}: {exc}") from exc
|
||||
if value.get("adapter") != "tester":
|
||||
raise EnvironmentFailure(f"{binding_id} 不是 tester binding")
|
||||
return value
|
||||
|
||||
|
||||
def agent_runner(root: Path, request: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Dispatch a platform operation through the resolved tester binding."""
|
||||
runtime = load_agent_runtime()
|
||||
return runtime.execute_runner_request(root, request)
|
||||
|
||||
|
||||
def platform_node(root: Path, binding_id: str) -> str:
|
||||
location = binding_location(root, binding_id)["location"]
|
||||
return "local" if location.get("environment") == "local" else str(location.get("environment"))
|
||||
|
||||
|
||||
def validate_preflight_matrix(root: Path, config_path: Path, bindings: dict[str, str], requested: list[str]) -> None:
|
||||
"""Fail closed when a matrix routes a platform to the wrong node/runner."""
|
||||
try:
|
||||
import yaml
|
||||
|
||||
raw = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
|
||||
except (OSError, ValueError, ImportError) as exc:
|
||||
raise EnvironmentFailure(f"无法读取 preflight matrix: {config_path}: {exc}") from exc
|
||||
rows = raw.get("matrix") if isinstance(raw, dict) else None
|
||||
if not isinstance(rows, list):
|
||||
raise EnvironmentFailure("preflight matrix 缺少 matrix;不能推断平台执行节点")
|
||||
by_platform = {str(row.get("platform", "")).lower(): row for row in rows if isinstance(row, dict)}
|
||||
for platform in requested:
|
||||
expected_binding = bindings[platform]
|
||||
row = by_platform.get(platform)
|
||||
if not row:
|
||||
raise EnvironmentFailure(f"preflight matrix 缺少 {platform} 行;不能创建该平台 session")
|
||||
tester = row.get("tester") if isinstance(row, dict) else None
|
||||
actual_binding = tester.get("binding") if isinstance(tester, dict) else None
|
||||
if actual_binding != expected_binding:
|
||||
raise EnvironmentFailure(f"{platform} preflight binding={actual_binding!r},期望 {expected_binding!r}")
|
||||
expected_node = platform_node(root, expected_binding)
|
||||
actual_node = tester.get("node") if isinstance(tester, dict) else None
|
||||
if expected_node == "local" and actual_node not in {"local", "localhost", "127.0.0.1"}:
|
||||
raise EnvironmentFailure(f"{platform} binding 是 local,但 preflight node={actual_node!r}")
|
||||
runner = tester.get("runner") if isinstance(tester, dict) else None
|
||||
if expected_node != "local" and (not isinstance(runner, dict) or runner.get("kind") != "contract"):
|
||||
raise EnvironmentFailure(f"{platform} 绑定到 {expected_node},必须使用 contract runner;禁止在协调节点执行")
|
||||
|
||||
|
||||
def preflight_remote_artifacts(config_path: Path, requested: list[str]) -> dict[str, str]:
|
||||
"""Return caller-declared artifact paths readable by remote tester nodes."""
|
||||
try:
|
||||
import yaml
|
||||
|
||||
raw = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
|
||||
except (OSError, ValueError, ImportError) as exc:
|
||||
raise EnvironmentFailure(f"无法读取 preflight matrix: {config_path}: {exc}") from exc
|
||||
rows = raw.get("matrix") if isinstance(raw, dict) else None
|
||||
if not isinstance(rows, list):
|
||||
raise EnvironmentFailure("preflight matrix 缺少 matrix;不能解析远程产物路径")
|
||||
by_platform = {str(row.get("platform", "")).lower(): row for row in rows if isinstance(row, dict)}
|
||||
paths: dict[str, str] = {}
|
||||
for platform in requested:
|
||||
row = by_platform.get(platform) or {}
|
||||
artifact = row.get("artifact") if isinstance(row, dict) else None
|
||||
remote_path = artifact.get("remote_path") if isinstance(artifact, dict) else None
|
||||
if isinstance(remote_path, str) and remote_path.strip():
|
||||
paths[platform] = remote_path.strip()
|
||||
return paths
|
||||
|
||||
|
||||
def local_path(locator: str) -> Path | None:
|
||||
parsed = urllib.parse.urlsplit(locator)
|
||||
if parsed.scheme in ("http", "https"):
|
||||
@@ -209,6 +326,10 @@ def android_identity(path: Path, context: RuntimeContext) -> str:
|
||||
result = run_command(command, context=context)
|
||||
if result.code == 0:
|
||||
match = re.search(r"(?:name|application-id)[=:]['\"]?([A-Za-z0-9_.]+)", result.output)
|
||||
if not match:
|
||||
# apkanalyzer manifest application-id prints the bare package
|
||||
# name, while aapt/aapt2 print a package: name=... record.
|
||||
match = re.fullmatch(r"\s*([A-Za-z0-9_.]+)\s*", result.output)
|
||||
if match:
|
||||
return match.group(1)
|
||||
raise EnvironmentFailure("APK 身份读取失败;请检查 Android SDK build-tools")
|
||||
@@ -388,20 +509,71 @@ def write_run(root: Path, record: dict[str, Any]) -> Path:
|
||||
return path
|
||||
|
||||
|
||||
def all_active_result_rows(root: Path, *, blocked_by: str) -> list[dict[str, Any]]:
|
||||
"""Keep a testcase row for every active catalog entry when preflight blocks."""
|
||||
try:
|
||||
from generate_cases_csv import load_cases
|
||||
|
||||
cases = load_cases(root)
|
||||
except (ImportError, OSError, ValueError, SystemExit):
|
||||
cases = []
|
||||
rows: list[dict[str, Any]] = []
|
||||
for case in cases:
|
||||
if case.get("lifecycle", "active") != "active":
|
||||
continue
|
||||
rows.append({
|
||||
"id": str(case.get("id") or ""),
|
||||
"result": "not-run",
|
||||
"notes": "test-preflight 未通过,未创建移动 session",
|
||||
"blocked_by": [blocked_by],
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
def run_test_preflight(
|
||||
config_path: Path,
|
||||
*,
|
||||
mode: str = "verify",
|
||||
yaml_path: Path | None = None,
|
||||
csv_path: Path | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Run the shared preflight contract before any client session work."""
|
||||
if mode not in {"verify", "prepare"}:
|
||||
raise ArtifactFailure("preflight mode 只能是 verify 或 prepare")
|
||||
if not PREFLIGHT_ROOT.is_file():
|
||||
raise EnvironmentFailure(f"找不到 test-preflight skill: {PREFLIGHT_ROOT}")
|
||||
spec = importlib.util.spec_from_file_location("pouch_test_preflight", PREFLIGHT_ROOT)
|
||||
if spec is None or spec.loader is None:
|
||||
raise EnvironmentFailure("无法加载 test-preflight skill")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
try:
|
||||
config = module.load_config(config_path)
|
||||
evidence = module.run(config, mode)
|
||||
yaml_path = yaml_path or config_path.with_name("preflight.yaml")
|
||||
csv_path = csv_path or config_path.with_name("preflight.csv")
|
||||
return module.write_evidence(evidence, yaml_path, csv_path)
|
||||
except (OSError, ValueError, module.yaml.YAMLError) as exc:
|
||||
return {"status": "blocked", "error": str(exc), "yaml": str(yaml_path or ""), "csv": str(csv_path or "")}
|
||||
|
||||
|
||||
def execute(args: argparse.Namespace) -> int:
|
||||
requested = platforms(args.platform)
|
||||
context = RuntimeContext(Path(args.root).resolve())
|
||||
root = Path(args.root).resolve()
|
||||
bindings = parse_bindings(args.bindings, requested)
|
||||
context = RuntimeContext(root, bindings=bindings)
|
||||
record: dict[str, Any] = {
|
||||
"schema_version": 1,
|
||||
"started_at": datetime.now(timezone.utc).isoformat(),
|
||||
"platforms_requested": requested,
|
||||
"partial": bool(args.partial),
|
||||
"binding": args.binding,
|
||||
"bindings": bindings,
|
||||
"node": os.environ.get("TESTER_NODE", "local"),
|
||||
"terminal": {"handle": os.environ.get("ORCA_TERMINAL_HANDLE"), "created": os.environ.get("TESTER_TERMINAL_CREATED", "false").lower() == "true", "closed": os.environ.get("TESTER_TERMINAL_CLOSED", "unknown")},
|
||||
"product_test": {"status": "not-run", "failure_class": "automatic-repair-failed", "note": "准备完成后由 tester catalog 驱动真实移动 session;本工具不以 API 替代移动测试"},
|
||||
"platforms": {},
|
||||
}
|
||||
remote_artifacts: dict[str, str] = {}
|
||||
try:
|
||||
locators = parse_artifacts(args.artifacts)
|
||||
missing = [platform for platform in requested if platform not in locators]
|
||||
@@ -417,7 +589,7 @@ def execute(args: argparse.Namespace) -> int:
|
||||
artifact_dir = Path(tempfile.mkdtemp(prefix="pouch-tester-artifacts-"))
|
||||
record["artifact_dir"] = str(artifact_dir)
|
||||
expected = catalog_identities(context.root)
|
||||
prepared: dict[str, tuple[dict[str, Any], dict[str, str], str]] = {}
|
||||
prepared: dict[str, tuple[dict[str, Any], dict[str, str] | None, str]] = {}
|
||||
failures: dict[str, str] = {}
|
||||
for platform in requested:
|
||||
result: dict[str, Any] = {"node": "tester.android" if platform == "android" else "tester.ios", "status": "preparing"}
|
||||
@@ -429,22 +601,120 @@ def execute(args: argparse.Namespace) -> int:
|
||||
wanted = expected.get(platform)
|
||||
if wanted and identity != wanted:
|
||||
raise EnvironmentFailure(f"{platform} 身份不匹配: expected={wanted} actual={identity}")
|
||||
node = android_preflight(context) if platform == "android" else ios_preflight(context)
|
||||
check_network(context, node)
|
||||
result.update({"identity": identity, "node_preflight": node, "status": "preflight-passed"})
|
||||
prepared[platform] = (result, node, identity)
|
||||
result.update({"identity": identity, "status": "artifact-ready"})
|
||||
prepared[platform] = (result, None, identity)
|
||||
except (EnvironmentFailure, ArtifactFailure) as exc:
|
||||
failures[platform] = str(exc)
|
||||
result.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc)})
|
||||
|
||||
# This is the hard gate shared by integration and journey callers.
|
||||
# There is no safe implicit matrix: without a caller-supplied config we
|
||||
# cannot prove HOST, fixtures, tester bindings, or node reachability.
|
||||
preflight_config = Path(args.preflight_config).expanduser() if args.preflight_config else None
|
||||
if not failures:
|
||||
if preflight_config is None:
|
||||
preflight_result = {
|
||||
"status": "blocked",
|
||||
"error": "缺少 --preflight-config;不能跳过 test-preflight",
|
||||
}
|
||||
else:
|
||||
validate_preflight_matrix(context.root, preflight_config, bindings, requested)
|
||||
remote_artifacts = preflight_remote_artifacts(preflight_config, requested)
|
||||
preflight_result = run_test_preflight(
|
||||
preflight_config,
|
||||
mode=args.preflight_mode,
|
||||
yaml_path=Path(args.preflight_yaml).expanduser() if args.preflight_yaml else None,
|
||||
csv_path=Path(args.preflight_csv).expanduser() if args.preflight_csv else None,
|
||||
)
|
||||
record["preflight"] = preflight_result
|
||||
if preflight_result.get("status") != "ready":
|
||||
failures["preflight"] = str(preflight_result.get("error") or preflight_result.get("status"))
|
||||
|
||||
if not failures and args.preflight_mode == "prepare":
|
||||
# test-preflight prepare owns platform probing, installation and
|
||||
# launch; do not install/start the same artifact a second time.
|
||||
for result, _, _ in prepared.values():
|
||||
result.update({"status": "preflight-prepared"})
|
||||
elif not failures:
|
||||
for platform in requested:
|
||||
result, _, identity = prepared[platform]
|
||||
try:
|
||||
binding_id = bindings[platform]
|
||||
node_name = platform_node(context.root, binding_id)
|
||||
probe = agent_runner(context.root, {
|
||||
"operation": "platform_probe",
|
||||
"platform": platform,
|
||||
"node": node_name,
|
||||
"binding": binding_id,
|
||||
})
|
||||
if probe.get("status") != "ready":
|
||||
raise EnvironmentFailure(str(probe.get("observed") or probe))
|
||||
host = os.environ.get("HOST", "").strip()
|
||||
network = agent_runner(context.root, {
|
||||
"operation": "network_probe",
|
||||
"platform": platform,
|
||||
"node": node_name,
|
||||
"binding": binding_id,
|
||||
"url": f"http://{host}/v1/me",
|
||||
}) if host else {"status": "blocked", "observed": "HOST 未设置"}
|
||||
if network.get("status") != "ready":
|
||||
raise EnvironmentFailure(str(network.get("observed") or network))
|
||||
result.update({"node_preflight": probe, "network_preflight": network, "status": "preflight-passed"})
|
||||
prepared[platform] = (result, None, identity)
|
||||
except EnvironmentFailure as exc:
|
||||
failures[platform] = str(exc)
|
||||
result.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc)})
|
||||
if failures and not args.partial:
|
||||
for platform in requested:
|
||||
if platform not in failures:
|
||||
record["platforms"][platform].update({"status": "not-run", "reason": "全量 preflight 闸门被另一平台阻断"})
|
||||
record.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "next_action": "修复列出的平台环境后重跑;默认不会静默降级"})
|
||||
overall = record.get("preflight", {}).get("status") if "preflight" in record else "environment-fail"
|
||||
record.update({"status": overall if overall in {"blocked", "environment-fail"} else "environment-fail", "failure_class": "test-preflight" if "preflight" in record else "automatic-repair-failed", "next_action": "修复列出的平台环境后重跑;默认不会静默降级"})
|
||||
record["results"] = all_active_result_rows(context.root, blocked_by="test-preflight")
|
||||
else:
|
||||
for platform, (result, node, identity) in prepared.items():
|
||||
try:
|
||||
installed = install_android(node, Path(result["artifact"]["path"]), identity, context) if platform == "android" else install_ios(node, Path(result["artifact"]["path"]), identity, context)
|
||||
if args.preflight_mode == "prepare":
|
||||
# test-preflight already ran platform_probe, network_probe,
|
||||
# install and start through each matrix binding.
|
||||
installed = {"delegated": "test-preflight.prepare", "binding": bindings[platform]}
|
||||
else:
|
||||
binding_id = bindings[platform]
|
||||
node_name = platform_node(context.root, binding_id)
|
||||
probe = agent_runner(context.root, {
|
||||
"operation": "platform_probe",
|
||||
"platform": platform,
|
||||
"node": node_name,
|
||||
"binding": binding_id,
|
||||
})
|
||||
if probe.get("status") != "ready":
|
||||
raise EnvironmentFailure(str(probe.get("observed") or probe))
|
||||
host = os.environ.get("HOST", "").strip()
|
||||
if host:
|
||||
network = agent_runner(context.root, {
|
||||
"operation": "network_probe",
|
||||
"platform": platform,
|
||||
"node": node_name,
|
||||
"binding": binding_id,
|
||||
"url": f"http://{host}/v1/me",
|
||||
})
|
||||
if network.get("status") != "ready":
|
||||
raise EnvironmentFailure(str(network.get("observed") or network))
|
||||
artifact_path = result["artifact"]["path"]
|
||||
if node_name != "local":
|
||||
artifact_path = remote_artifacts.get(platform, "")
|
||||
if not artifact_path:
|
||||
raise EnvironmentFailure(f"远程 {platform} 安装必须在 matrix.artifact.remote_path 声明节点可读产物")
|
||||
installed = agent_runner(context.root, {
|
||||
"operation": "install",
|
||||
"platform": platform,
|
||||
"node": node_name,
|
||||
"binding": binding_id,
|
||||
"artifact": artifact_path,
|
||||
"identity": identity,
|
||||
})
|
||||
if installed.get("status") != "ready":
|
||||
raise EnvironmentFailure(str(installed.get("observed") or installed))
|
||||
result.update({"status": "ready", "install": installed, "failure_class": None})
|
||||
except EnvironmentFailure as exc:
|
||||
failures[platform] = str(exc)
|
||||
@@ -455,7 +725,18 @@ def execute(args: argparse.Namespace) -> int:
|
||||
record["evidence"] = context.evidence
|
||||
except (EnvironmentFailure, ArtifactFailure) as exc:
|
||||
record.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc), "evidence": context.evidence})
|
||||
if record.get("status") != "ready" and "results" not in record:
|
||||
record["results"] = all_active_result_rows(context.root, blocked_by="test-preflight")
|
||||
path = write_run(context.root, record)
|
||||
try:
|
||||
from generate_run_csv import export_run_csv
|
||||
|
||||
record["csv"] = str(export_run_csv(context.root, run_path=path))
|
||||
except (OSError, SystemExit, ValueError):
|
||||
# The YAML run remains the source of truth when a partial/provisioning
|
||||
# root has no catalog cases yet.
|
||||
record["csv"] = ""
|
||||
path.write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
output = {"status": record["status"], "run": str(path), "platforms": record["platforms"], "worktree": record.get("worktree")}
|
||||
if record.get("error"):
|
||||
output["error"] = record["error"]
|
||||
@@ -475,7 +756,18 @@ def parser() -> argparse.ArgumentParser:
|
||||
run.add_argument("--partial", action="store_true")
|
||||
run.add_argument("--root", default=".")
|
||||
run.add_argument("--worktree-root", default="")
|
||||
run.add_argument("--binding", default="")
|
||||
run.add_argument(
|
||||
"--binding",
|
||||
action="append",
|
||||
dest="bindings",
|
||||
default=[],
|
||||
metavar="PLATFORM=TESTER_BINDING",
|
||||
help="每个平台的 tester binding,可重复;默认 tester.<platform>",
|
||||
)
|
||||
run.add_argument("--preflight-config", default="", help="test-preflight matrix YAML(必填,不得跳过)")
|
||||
run.add_argument("--preflight-mode", choices=("verify", "prepare"), default="verify")
|
||||
run.add_argument("--preflight-yaml", default="", help="preflight YAML 证据输出路径")
|
||||
run.add_argument("--preflight-csv", default="", help="preflight CSV 证据输出路径")
|
||||
run.set_defaults(func=execute)
|
||||
return root
|
||||
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Append structured tester issue evidence to an existing run snapshot."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from failure_feedback import attach_feedback
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="将 tester 失败结构化回流到 run YAML/CSV")
|
||||
parser.add_argument("--run", required=True, type=Path)
|
||||
parser.add_argument("--trigger", choices=("pr", "requirement", "bug"))
|
||||
parser.add_argument("--target", default="")
|
||||
parser.add_argument("--related-issue", default="")
|
||||
parser.add_argument("--dispatcher", default="dispatcher")
|
||||
parser.add_argument("--testcase", default="", help="为一条失败结果补充 testcase issue 字段")
|
||||
parser.add_argument("--issue-type", choices=("product", "test"))
|
||||
parser.add_argument("--expected", default="")
|
||||
parser.add_argument("--observed", default="")
|
||||
parser.add_argument("--artifact-hash", default="")
|
||||
parser.add_argument("--environment", default="")
|
||||
parser.add_argument("--evidence-path", default="")
|
||||
parser.add_argument("--reproduction", default="")
|
||||
parser.add_argument("--next-step", default="")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
data: dict[str, Any] = yaml.safe_load(args.run.read_text(encoding="utf-8")) or {}
|
||||
if args.testcase:
|
||||
rows = data.get("results")
|
||||
if not isinstance(rows, list):
|
||||
raise SystemExit("run 缺少 results 列表")
|
||||
row = next((item for item in rows if isinstance(item, dict) and str(item.get("id")) == args.testcase), None)
|
||||
if row is None:
|
||||
raise SystemExit(f"找不到 testcase: {args.testcase}")
|
||||
row["issue"] = {
|
||||
key: value
|
||||
for key, value in {
|
||||
"issue_type": args.issue_type,
|
||||
"expected": args.expected,
|
||||
"observed": args.observed,
|
||||
"artifact_hash": args.artifact_hash,
|
||||
"environment": args.environment,
|
||||
"evidence_path": args.evidence_path,
|
||||
"reproduction": args.reproduction,
|
||||
"next_step": args.next_step,
|
||||
}.items()
|
||||
if value
|
||||
}
|
||||
output = attach_feedback(
|
||||
data,
|
||||
trigger=args.trigger,
|
||||
target=args.target,
|
||||
related_issue=args.related_issue,
|
||||
dispatcher=args.dispatcher,
|
||||
)
|
||||
args.run.write_text(yaml.safe_dump(output, allow_unicode=True, sort_keys=False), encoding="utf-8")
|
||||
csv_path = ""
|
||||
try:
|
||||
from generate_run_csv import export_run_csv
|
||||
|
||||
csv_path = str(export_run_csv(args.run.parents[3], run_path=args.run))
|
||||
except (ImportError, OSError, SystemExit, ValueError):
|
||||
pass
|
||||
print(f"updated {args.run} defects={len(output.get('defects') or [])} csv={csv_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
||||
@@ -180,9 +180,14 @@ def clean_config(config: dict[str, Any]) -> dict[str, Any]:
|
||||
def config_from_binding(root: Path, selector: str) -> dict[str, Any]:
|
||||
"""Materialize a common agent-runtime tester binding for legacy local.yaml."""
|
||||
|
||||
runtime_path = Path(__file__).resolve().parents[3] / "skills" / "agent-runtime" / "scripts" / "agent.py"
|
||||
if not runtime_path.is_file():
|
||||
raise RuntimeError(f"找不到共享 agent-runtime: {runtime_path}")
|
||||
candidates = [
|
||||
Path.home() / ".codex" / "skills" / "agent-runtime" / "scripts" / "agent.py",
|
||||
Path.home() / ".pouch" / "skills" / "agent-runtime" / "scripts" / "agent.py",
|
||||
Path(__file__).resolve().parents[3] / "skills" / "agent-runtime" / "scripts" / "agent.py",
|
||||
]
|
||||
runtime_path = next((item for item in candidates if item.is_file()), None)
|
||||
if runtime_path is None:
|
||||
raise RuntimeError("找不到共享 agent-runtime: " + ", ".join(str(item) for item in candidates))
|
||||
spec = importlib.util.spec_from_file_location("pouch_agent_runtime", runtime_path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError(f"无法加载共享 agent-runtime: {runtime_path}")
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate a tester CSV/YAML pair and summarize statuses from CSV only."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("需要 PyYAML: python3 -m pip install pyyaml", file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
||||
RESULT_SET = frozenset(RESULTS)
|
||||
|
||||
|
||||
class SummaryError(ValueError):
|
||||
"""Raised when the CSV/YAML pair cannot be safely summarized."""
|
||||
|
||||
|
||||
def _format_ids(ids: set[str]) -> str:
|
||||
return ", ".join(sorted(ids)) or "(none)"
|
||||
|
||||
|
||||
def _duplicate_ids(ids: list[str]) -> list[str]:
|
||||
counts = Counter(ids)
|
||||
return sorted(case_id for case_id, count in counts.items() if count > 1)
|
||||
|
||||
|
||||
def load_csv(path: Path) -> list[dict[str, str]]:
|
||||
"""Load CSV rows and reject malformed IDs/results with diagnostics."""
|
||||
try:
|
||||
handle = path.open(encoding="utf-8-sig", newline="")
|
||||
except OSError as exc:
|
||||
raise SummaryError(f"无法读取 CSV {path}: {exc}") from exc
|
||||
|
||||
with handle:
|
||||
reader = csv.DictReader(handle)
|
||||
if not reader.fieldnames:
|
||||
raise SummaryError(f"CSV {path} 缺少表头")
|
||||
missing_fields = [field for field in ("id", "result") if field not in reader.fieldnames]
|
||||
if missing_fields:
|
||||
raise SummaryError(f"CSV {path} 缺少必需列: {', '.join(missing_fields)}")
|
||||
rows = list(reader)
|
||||
|
||||
ids: list[str] = []
|
||||
errors: list[str] = []
|
||||
for line_number, row in enumerate(rows, start=2):
|
||||
case_id = (row.get("id") or "").strip()
|
||||
result = (row.get("result") or "").strip()
|
||||
ids.append(case_id)
|
||||
if not case_id:
|
||||
errors.append(f"CSV 第 {line_number} 行 testcase id 缺失")
|
||||
if not result:
|
||||
errors.append(f"CSV 第 {line_number} 行 result 缺失 (testcase id={case_id or '(missing)'})")
|
||||
elif result not in RESULT_SET:
|
||||
errors.append(
|
||||
f"CSV 第 {line_number} 行 result 未知: {result!r} "
|
||||
f"(testcase id={case_id or '(missing)'}; 允许值={', '.join(RESULTS)})"
|
||||
)
|
||||
|
||||
duplicates = _duplicate_ids([case_id for case_id in ids if case_id])
|
||||
if duplicates:
|
||||
errors.append(f"CSV testcase id 重复: {_format_ids(set(duplicates))}")
|
||||
if errors:
|
||||
raise SummaryError("CSV 校验失败:\n- " + "\n- ".join(errors))
|
||||
return rows
|
||||
|
||||
|
||||
def load_yaml_results(path: Path) -> list[dict[str, Any]]:
|
||||
"""Load only YAML results structure needed for count/ID validation."""
|
||||
try:
|
||||
data = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
except OSError as exc:
|
||||
raise SummaryError(f"无法读取 YAML {path}: {exc}") from exc
|
||||
except yaml.YAMLError as exc:
|
||||
raise SummaryError(f"YAML 解析失败 {path}: {exc}") from exc
|
||||
|
||||
if not isinstance(data, dict) or not isinstance(data.get("results"), list):
|
||||
raise SummaryError(f"YAML {path} 缺少 results 列表")
|
||||
|
||||
results = data["results"]
|
||||
ids: list[str] = []
|
||||
errors: list[str] = []
|
||||
for index, item in enumerate(results, start=1):
|
||||
if not isinstance(item, dict):
|
||||
errors.append(f"YAML results[{index}] 不是 mapping")
|
||||
continue
|
||||
case_id = str(item.get("id") or "").strip()
|
||||
ids.append(case_id)
|
||||
if not case_id:
|
||||
errors.append(f"YAML results[{index}] testcase id 缺失")
|
||||
|
||||
duplicates = _duplicate_ids([case_id for case_id in ids if case_id])
|
||||
if duplicates:
|
||||
errors.append(f"YAML testcase id 重复: {_format_ids(set(duplicates))}")
|
||||
if errors:
|
||||
raise SummaryError("YAML 校验失败:\n- " + "\n- ".join(errors))
|
||||
issues = data.get("defects", data.get("test_issues"))
|
||||
if issues is not None:
|
||||
if not isinstance(issues, list):
|
||||
raise SummaryError("YAML defects/test_issues 必须是列表")
|
||||
result_ids = set(ids)
|
||||
issue_errors: list[str] = []
|
||||
for index, issue in enumerate(issues, start=1):
|
||||
if not isinstance(issue, dict):
|
||||
issue_errors.append(f"YAML defects[{index}] 不是 mapping")
|
||||
continue
|
||||
testcase = str(issue.get("testcase") or "")
|
||||
issue_type = str(issue.get("issue_type") or "")
|
||||
if not testcase or testcase not in result_ids:
|
||||
issue_errors.append(f"YAML defects[{index}] testcase 无效: {testcase or '(missing)'}")
|
||||
if issue_type not in {"product", "test"}:
|
||||
issue_errors.append(f"YAML defects[{index}] issue_type 无效: {issue_type or '(missing)'}")
|
||||
for field in ("expected", "observed", "artifact_hash", "environment", "evidence_path", "reproduction", "next_step"):
|
||||
if field not in issue:
|
||||
issue_errors.append(f"YAML defects[{index}] 缺少 {field}")
|
||||
if issue_errors:
|
||||
raise SummaryError("YAML defects 校验失败:\n- " + "\n- ".join(issue_errors))
|
||||
return results
|
||||
|
||||
|
||||
def summarize(csv_path: Path, yaml_path: Path) -> tuple[Counter[str], dict[str, int]]:
|
||||
"""Validate the pair and return counts calculated from CSV result values."""
|
||||
rows = load_csv(csv_path)
|
||||
yaml_results = load_yaml_results(yaml_path)
|
||||
csv_ids = [(row.get("id") or "").strip() for row in rows]
|
||||
yaml_ids = [str(item.get("id") or "").strip() for item in yaml_results]
|
||||
csv_id_set = set(csv_ids)
|
||||
yaml_id_set = set(yaml_ids)
|
||||
diagnostics: list[str] = []
|
||||
|
||||
if len(rows) != len(csv_id_set):
|
||||
diagnostics.append(
|
||||
f"CSV 数据行总数与唯一 testcase id 数不一致: rows={len(rows)} unique_ids={len(csv_id_set)}"
|
||||
)
|
||||
if len(rows) != len(yaml_results):
|
||||
diagnostics.append(
|
||||
f"CSV 数据行总数与 YAML results 数量不一致: csv_rows={len(rows)} yaml_results={len(yaml_results)}"
|
||||
)
|
||||
if csv_id_set != yaml_id_set:
|
||||
diagnostics.append(
|
||||
"CSV/YAML testcase id 集合不一致: "
|
||||
f"仅 CSV 有 [{_format_ids(csv_id_set - yaml_id_set)}]; "
|
||||
f"仅 YAML 有 [{_format_ids(yaml_id_set - csv_id_set)}]"
|
||||
)
|
||||
if diagnostics:
|
||||
raise SummaryError("CSV/YAML 汇总校验失败:\n- " + "\n- ".join(diagnostics))
|
||||
|
||||
counts: Counter[str] = Counter((row.get("result") or "").strip() for row in rows)
|
||||
total = sum(counts[name] for name in RESULTS)
|
||||
if total != len(rows):
|
||||
raise SummaryError(
|
||||
"CSV 状态计数和不等于数据行总数: "
|
||||
f"count_sum={total} csv_rows={len(rows)}"
|
||||
)
|
||||
metadata = {
|
||||
"csv_rows": len(rows),
|
||||
"csv_unique_ids": len(csv_id_set),
|
||||
"yaml_results": len(yaml_results),
|
||||
"count_sum": total,
|
||||
}
|
||||
return counts, metadata
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="校验 tester CSV/YAML 并从 CSV result 汇总状态")
|
||||
parser.add_argument("--csv", required=True, type=Path, help="运行成绩单 CSV")
|
||||
parser.add_argument("--yaml", required=True, type=Path, help="对应运行快照 YAML")
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
counts, metadata = summarize(args.csv, args.yaml)
|
||||
except SummaryError as exc:
|
||||
print(f"汇总失败: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
bits = " ".join(f"{name}={counts[name]}" for name in RESULTS)
|
||||
print(f"summary source=CSV.result {bits} total={metadata['count_sum']}")
|
||||
print(
|
||||
"validated "
|
||||
f"csv_rows={metadata['csv_rows']} "
|
||||
f"csv_unique_ids={metadata['csv_unique_ids']} "
|
||||
f"yaml_results={metadata['yaml_results']}"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,185 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Shared execution policy for tester's integration-first runs.
|
||||
|
||||
This module intentionally has no browser, mobile, deployment, or API driver.
|
||||
Those adapters belong to the caller. It only controls the phase gate and the
|
||||
stable five-state testcase result propagation so a caller cannot create a
|
||||
journey session before preflight and integration have passed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable, Iterable, Mapping
|
||||
from typing import Any
|
||||
|
||||
from failure_feedback import collect_issues
|
||||
|
||||
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
||||
PHASES = ("preflight", "integration", "journey")
|
||||
|
||||
|
||||
def classify_integration(*, ok: bool, service_reachable: bool = True) -> str:
|
||||
"""Separate a contract assertion failure from an unavailable service."""
|
||||
if not service_reachable:
|
||||
return "environment-fail"
|
||||
return "pass" if ok else "fail"
|
||||
|
||||
|
||||
def _case_id(case: Mapping[str, Any] | str) -> str:
|
||||
if isinstance(case, Mapping):
|
||||
return str(case.get("id") or "")
|
||||
return str(case)
|
||||
|
||||
|
||||
def _not_run(
|
||||
cases: Iterable[Mapping[str, Any] | str],
|
||||
reason: str,
|
||||
*,
|
||||
blocked_by: Iterable[str] = (),
|
||||
) -> list[dict[str, Any]]:
|
||||
rows: list[dict[str, str]] = []
|
||||
blockers = [str(item) for item in blocked_by if str(item)]
|
||||
for case in cases:
|
||||
row: dict[str, Any] = {"id": _case_id(case), "result": "not-run", "notes": reason}
|
||||
if blockers:
|
||||
row["blocked_by"] = blockers
|
||||
if isinstance(case, Mapping):
|
||||
row["test_type"] = str(case.get("test_type") or "")
|
||||
row["suite"] = str(
|
||||
case.get("client_suite")
|
||||
or case.get("integration_suite")
|
||||
or case.get("suite")
|
||||
or ""
|
||||
)
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def _dependencies(
|
||||
case: Mapping[str, Any] | str,
|
||||
*,
|
||||
default: Iterable[str] = (),
|
||||
) -> list[str]:
|
||||
if not isinstance(case, Mapping):
|
||||
return [str(item) for item in default]
|
||||
declared = case.get("depends_on", case.get("dependencies"))
|
||||
if declared is None:
|
||||
return [str(item) for item in default]
|
||||
if isinstance(declared, str):
|
||||
return [declared]
|
||||
if isinstance(declared, Iterable):
|
||||
return [str(item) for item in declared if str(item)]
|
||||
raise ValueError(f"depends_on 必须是字符串或列表: {_case_id(case)}")
|
||||
|
||||
|
||||
def _failed_dependencies(dependencies: Iterable[str], results: Mapping[str, Mapping[str, Any]]) -> list[str]:
|
||||
return [
|
||||
dependency
|
||||
for dependency in dependencies
|
||||
if str((results.get(dependency) or {}).get("result"))
|
||||
in {"fail", "environment-fail", "not-run"}
|
||||
]
|
||||
|
||||
|
||||
def _issue_structures(results: list[dict[str, Any]]) -> dict[str, list[dict[str, Any]]]:
|
||||
issues = collect_issues(results)
|
||||
return {"defects": issues, "test_issues": [dict(issue) for issue in issues]}
|
||||
|
||||
|
||||
def run_integration_first(
|
||||
*,
|
||||
preflight_status: str,
|
||||
integration_cases: Iterable[Mapping[str, Any] | str],
|
||||
journey_cases: Iterable[Mapping[str, Any] | str],
|
||||
execute_integration: Callable[[Mapping[str, Any] | str], str | Mapping[str, Any]],
|
||||
create_journey_session: Callable[[], Any],
|
||||
execute_journey: Callable[[Any, Mapping[str, Any] | str], str | Mapping[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
"""Execute integration cases before creating any journey session.
|
||||
|
||||
``preflight_status`` must be the status emitted by test-preflight. A
|
||||
blocked/environment-fail gate records only ``not-run`` testcase rows and
|
||||
never invokes either adapter. An integration failure stops the phase and
|
||||
propagates ``not-run`` to all journey cases.
|
||||
"""
|
||||
integration = list(integration_cases)
|
||||
journey = list(journey_cases)
|
||||
if preflight_status != "ready":
|
||||
reason = f"test-preflight={preflight_status},未创建客户端 session"
|
||||
result_rows = _not_run(integration, reason, blocked_by=("test-preflight",)) + _not_run(journey, reason, blocked_by=("test-preflight",))
|
||||
return {
|
||||
"status": preflight_status,
|
||||
"phase": "preflight",
|
||||
"session_created": False,
|
||||
"results": result_rows,
|
||||
**_issue_structures(result_rows),
|
||||
}
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
results_by_id: dict[str, dict[str, Any]] = {}
|
||||
for case in integration:
|
||||
case_id = _case_id(case)
|
||||
dependencies = _dependencies(case)
|
||||
blocked_by = _failed_dependencies(dependencies, results_by_id)
|
||||
if blocked_by:
|
||||
rows = _not_run(
|
||||
[case],
|
||||
"依赖的 integration 失败,未执行",
|
||||
blocked_by=blocked_by,
|
||||
)
|
||||
results.extend(rows)
|
||||
results_by_id[case_id] = rows[0]
|
||||
continue
|
||||
outcome = execute_integration(case)
|
||||
row = dict(outcome) if isinstance(outcome, Mapping) else {"result": outcome}
|
||||
row["id"] = _case_id(case)
|
||||
if isinstance(case, Mapping):
|
||||
row.setdefault("test_type", str(case.get("test_type") or "integration"))
|
||||
row.setdefault("suite", str(case.get("integration_suite") or case.get("suite") or ""))
|
||||
result = str(row.get("result") or "not-run")
|
||||
if result not in RESULTS:
|
||||
raise ValueError(f"未知 integration 结果: {row['id']}={result}")
|
||||
row["result"] = result
|
||||
results.append(row)
|
||||
results_by_id[case_id] = row
|
||||
|
||||
integration_ids = [_case_id(case) for case in integration]
|
||||
has_runnable_journey = any(
|
||||
not _failed_dependencies(_dependencies(case, default=integration_ids), results_by_id)
|
||||
for case in journey
|
||||
)
|
||||
session = create_journey_session() if has_runnable_journey else None
|
||||
for case in journey:
|
||||
dependencies = _dependencies(case, default=integration_ids)
|
||||
blocked_by = _failed_dependencies(dependencies, results_by_id)
|
||||
if blocked_by:
|
||||
row = _not_run(
|
||||
[case],
|
||||
"依赖的用例失败,journey 未执行",
|
||||
blocked_by=blocked_by,
|
||||
)[0]
|
||||
results.append(row)
|
||||
results_by_id[_case_id(case)] = row
|
||||
continue
|
||||
outcome = execute_journey(session, case)
|
||||
row = dict(outcome) if isinstance(outcome, Mapping) else {"result": outcome}
|
||||
row["id"] = _case_id(case)
|
||||
if isinstance(case, Mapping):
|
||||
row.setdefault("test_type", str(case.get("test_type") or "journey"))
|
||||
row.setdefault("suite", str(case.get("client_suite") or case.get("suite") or ""))
|
||||
result = str(row.get("result") or "not-run")
|
||||
if result not in RESULTS:
|
||||
raise ValueError(f"未知 journey 结果: {row['id']}={result}")
|
||||
row["result"] = result
|
||||
results.append(row)
|
||||
results_by_id[_case_id(case)] = row
|
||||
status = "fail" if any(row["result"] == "fail" for row in results) else (
|
||||
"environment-fail" if any(row["result"] == "environment-fail" for row in results) else "pass"
|
||||
)
|
||||
return {
|
||||
"status": status,
|
||||
"phase": "journey" if journey else "integration",
|
||||
"session_created": has_runnable_journey,
|
||||
"results": results,
|
||||
**_issue_structures(results),
|
||||
}
|
||||
@@ -22,6 +22,24 @@ risks:
|
||||
- mutating
|
||||
- destructive
|
||||
default_max_risk: readonly
|
||||
# standard enforces the default_max_risk/authorize policy. Writes-style isolated
|
||||
# resettable environments may set this to isolated-full; risk values remain on
|
||||
# cases and are still reported.
|
||||
execution_profile: standard
|
||||
client_suites:
|
||||
client-smoke:
|
||||
default: true
|
||||
client-regression:
|
||||
default: false
|
||||
safety:
|
||||
default: false
|
||||
integration_suites:
|
||||
api-contract:
|
||||
default: false
|
||||
authorization:
|
||||
default: false
|
||||
build-contract:
|
||||
default: false
|
||||
vars: {}
|
||||
authorize:
|
||||
mutating: AUTHORIZE_MUTATING
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
#!/usr/bin/env python3
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
CATALOG = ROOT / "scripts" / "catalog.py"
|
||||
|
||||
|
||||
def write_fixture(root: Path, *, complete: bool = True) -> None:
|
||||
tester = root / ".pouch" / "tester"
|
||||
(tester / "cases").mkdir(parents=True)
|
||||
(tester / "runs").mkdir()
|
||||
(tester / "catalog.yaml").write_text(
|
||||
yaml.safe_dump(
|
||||
{
|
||||
"version": 2,
|
||||
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
|
||||
"categories": [{"id": "CLIENT"}],
|
||||
"risks": ["readonly", "mutating", "destructive"],
|
||||
"default_max_risk": "readonly",
|
||||
"client_suites": {
|
||||
"client-smoke": {"default": True},
|
||||
"client-regression": {"default": False},
|
||||
"safety": {"default": False},
|
||||
},
|
||||
"sessions": {"browser": {"kind": "browser", "reuse": "run"}},
|
||||
},
|
||||
sort_keys=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
case = {
|
||||
"id": "TC-CLIENT-001",
|
||||
"category": "CLIENT",
|
||||
"title": "journey",
|
||||
"what": "client journey",
|
||||
"lifecycle": "active",
|
||||
"tag": "smoke",
|
||||
"risk": "readonly",
|
||||
"driver": "browser",
|
||||
"session": "browser",
|
||||
"client_suite": "client-smoke",
|
||||
"platform": ["android", "ios"],
|
||||
"journey": {
|
||||
"goal": "goal",
|
||||
"entry_state": "logged_out",
|
||||
"actions": ["start", "login"],
|
||||
"assertions": ["visible"],
|
||||
"final_state": "done",
|
||||
"failure_policy": "stop",
|
||||
},
|
||||
"fixture": {"account": "test-role"},
|
||||
"cleanup": {"steps": ["cleanup"]},
|
||||
"how": {"steps": ["start", "login"]},
|
||||
"expected": [{"kind": "visible-text", "value": "done"}],
|
||||
}
|
||||
if not complete:
|
||||
case.pop("fixture")
|
||||
case.pop("cleanup")
|
||||
(tester / "cases" / "client.yaml").write_text(
|
||||
yaml.safe_dump({"category": "CLIENT", "cases": [case]}, sort_keys=False),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
class CatalogJourneyTests(unittest.TestCase):
|
||||
def run_catalog(self, root: Path, *args: str):
|
||||
return subprocess.run(
|
||||
[sys.executable, str(CATALOG), *args, "--root", str(root)],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
def test_valid_client_catalog_and_suite_filter(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
write_fixture(root)
|
||||
valid = self.run_catalog(root, "validate")
|
||||
self.assertEqual(valid.returncode, 0, valid.stderr)
|
||||
selected = self.run_catalog(root, "select", "--suite", "client-smoke")
|
||||
self.assertEqual(selected.returncode, 0, selected.stderr)
|
||||
self.assertEqual(json.loads(selected.stdout)["count"], 1)
|
||||
|
||||
def test_missing_fixture_or_cleanup_blocks_before_results(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
write_fixture(root, complete=False)
|
||||
selected = self.run_catalog(root, "select", "--suite", "client-smoke")
|
||||
self.assertNotEqual(selected.returncode, 0)
|
||||
payload = json.loads(selected.stdout)
|
||||
self.assertEqual(payload["preflight"]["failure_class"], "contract")
|
||||
self.assertTrue(any("fixture 缺失" in error for error in payload["preflight"]["errors"]))
|
||||
self.assertEqual(payload["preflight"]["status"], "blocked")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env python3
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
|
||||
from journey import classify_outcome, contract_errors, propagate_results
|
||||
|
||||
|
||||
def valid_case(**overrides):
|
||||
case = {
|
||||
"id": "TC-CLIENT-001",
|
||||
"client_suite": "client-smoke",
|
||||
"driver": "browser",
|
||||
"session": "browser",
|
||||
"platform": ["android", "ios"],
|
||||
"journey": {
|
||||
"goal": "完成客户端目标",
|
||||
"entry_state": "logged_out",
|
||||
"actions": ["启动", "登录", "完成目标"],
|
||||
"assertions": ["看到目标页面"],
|
||||
"final_state": "目标已完成",
|
||||
"failure_policy": "stop",
|
||||
},
|
||||
"fixture": {"account": "test-role", "data": []},
|
||||
"cleanup": {"steps": ["清理本轮数据"]},
|
||||
}
|
||||
case.update(overrides)
|
||||
return case
|
||||
|
||||
|
||||
class JourneyContractTests(unittest.TestCase):
|
||||
def test_contract_accepts_shared_client_shape(self):
|
||||
self.assertEqual(contract_errors(valid_case(), {"browser": {}}), [])
|
||||
|
||||
def test_missing_fixture_and_cleanup_are_contract_errors(self):
|
||||
errors = contract_errors(
|
||||
valid_case(fixture=None, cleanup=None),
|
||||
{"browser": {}},
|
||||
)
|
||||
self.assertTrue(any("fixture 缺失" in error for error in errors))
|
||||
self.assertTrue(any("cleanup 缺失" in error for error in errors))
|
||||
|
||||
def test_login_failure_is_product_failure_but_unavailable_environment_is_not(self):
|
||||
self.assertEqual(classify_outcome(ok=False, phase="login"), "fail")
|
||||
self.assertEqual(
|
||||
classify_outcome(ok=False, phase="login", environment_ready=False),
|
||||
"environment-fail",
|
||||
)
|
||||
|
||||
def test_failed_case_blocks_remaining_cases_and_keeps_explicit_skip(self):
|
||||
rows = propagate_results(
|
||||
["TC-1", "TC-2", "TC-3"],
|
||||
{"TC-1": "fail", "TC-2": "pass", "TC-3": "pass"},
|
||||
skipped={"TC-3": "未授权"},
|
||||
)
|
||||
self.assertEqual([row["result"] for row in rows], ["fail", "not-run", "skipped"])
|
||||
|
||||
def test_continue_policy_allows_next_case(self):
|
||||
rows = propagate_results(
|
||||
["TC-1", "TC-2"],
|
||||
{"TC-1": "fail", "TC-2": "pass"},
|
||||
policies={"TC-1": "continue"},
|
||||
)
|
||||
self.assertEqual([row["result"] for row in rows], ["fail", "pass"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -50,6 +50,84 @@ class MobileBundleTests(unittest.TestCase):
|
||||
with self.assertRaises(bundle.ArtifactFailure):
|
||||
bundle.parse_artifacts(["android=/tmp/a.apk", "android=/tmp/b.apk"])
|
||||
|
||||
def test_parse_bindings_keeps_platforms_separate(self):
|
||||
self.assertEqual(
|
||||
bundle.parse_bindings(["android=tester.android", "ios=tester.ios"], ["android", "ios"]),
|
||||
{"android": "tester.android", "ios": "tester.ios"},
|
||||
)
|
||||
self.assertEqual(
|
||||
bundle.parse_bindings([], ["android"]),
|
||||
{"android": "tester.android"},
|
||||
)
|
||||
with self.assertRaises(bundle.ArtifactFailure):
|
||||
bundle.parse_bindings(["android=tester.test", "ios=tester.test"], ["android", "ios"])
|
||||
|
||||
def test_preflight_matrix_rejects_remote_platform_using_local_runner(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
agent_dir = root / ".pouch"
|
||||
agent_dir.mkdir()
|
||||
(agent_dir / "agent.yaml").write_text(
|
||||
"""
|
||||
schemaVersion: 1
|
||||
locations:
|
||||
local-app:
|
||||
environment: local
|
||||
host: local
|
||||
project: test
|
||||
path: /tmp/test
|
||||
remote-ios:
|
||||
environment: macmini
|
||||
host: runtime:12345678-1234-1234-1234-123456789012
|
||||
project: test
|
||||
path: /tmp/test-ios
|
||||
profiles:
|
||||
test-agent:
|
||||
role: test
|
||||
cli: codex
|
||||
tier: standard
|
||||
model: gpt-5
|
||||
reasoningEffort: medium
|
||||
permissionMode: workspace-write
|
||||
bindings:
|
||||
tester.android: {adapter: tester, location: local-app, profile: test-agent}
|
||||
tester.ios: {adapter: tester, location: remote-ios, profile: test-agent}
|
||||
""".strip()
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
config = root / "matrix.yaml"
|
||||
config.write_text(
|
||||
"""
|
||||
matrix:
|
||||
- platform: android
|
||||
tester: {binding: tester.android, node: local, runner: {kind: local}}
|
||||
- platform: ios
|
||||
tester: {binding: tester.ios, node: local, runner: {kind: local}}
|
||||
""".strip()
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
with self.assertRaisesRegex(bundle.EnvironmentFailure, "ios.*contract runner"):
|
||||
bundle.validate_preflight_matrix(
|
||||
root,
|
||||
config,
|
||||
{"android": "tester.android", "ios": "tester.ios"},
|
||||
["android", "ios"],
|
||||
)
|
||||
|
||||
def test_remote_artifact_path_comes_from_preflight_matrix(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
config = Path(temp) / "matrix.yaml"
|
||||
config.write_text(
|
||||
"matrix:\n - platform: ios\n artifact:\n remote_path: /Users/tester/app.ipa\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
self.assertEqual(
|
||||
bundle.preflight_remote_artifacts(config, ["ios"]),
|
||||
{"ios": "/Users/tester/app.ipa"},
|
||||
)
|
||||
|
||||
def test_new_worktree_each_run(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
repo, worktrees = Path(temp) / "repo", Path(temp) / "worktrees"
|
||||
@@ -80,6 +158,16 @@ class MobileBundleTests(unittest.TestCase):
|
||||
self.assertTrue(result["verified"])
|
||||
self.assertTrue(any("install" in call for call in calls))
|
||||
|
||||
def test_android_identity_accepts_bare_apkanalyzer_output(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
apk = Path(temp) / "app.apk"
|
||||
apk.write_bytes(b"apk")
|
||||
context = bundle.RuntimeContext(Path(temp))
|
||||
with mock.patch.object(bundle.shutil, "which", side_effect=lambda name: "apkanalyzer" if name == "apkanalyzer" else None), mock.patch.object(
|
||||
bundle, "run_command", return_value=bundle.CommandResult(0, "page.writes\n")
|
||||
):
|
||||
self.assertEqual(bundle.android_identity(apk, context), "page.writes")
|
||||
|
||||
def test_android_missing_device_is_environment_failure(self):
|
||||
context = bundle.RuntimeContext(Path("."))
|
||||
with mock.patch.object(bundle.shutil, "which", return_value="/sdk/adb"), mock.patch.object(
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
#!/usr/bin/env python3
|
||||
import csv
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
|
||||
import summarize_run
|
||||
|
||||
|
||||
class RunSummaryTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.temp.name)
|
||||
self.csv_path = self.root / "run.csv"
|
||||
self.yaml_path = self.root / "run.yaml"
|
||||
self.rows = [
|
||||
{"id": "TC-001", "result": "pass"},
|
||||
{"id": "TC-002", "result": "fail"},
|
||||
{"id": "TC-003", "result": "environment-fail"},
|
||||
{"id": "TC-004", "result": "skipped"},
|
||||
{"id": "TC-005", "result": "not-run"},
|
||||
]
|
||||
self.write_csv(self.rows)
|
||||
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows])
|
||||
|
||||
def tearDown(self):
|
||||
self.temp.cleanup()
|
||||
|
||||
def write_csv(self, rows):
|
||||
with self.csv_path.open("w", encoding="utf-8", newline="") as handle:
|
||||
writer = csv.DictWriter(handle, fieldnames=["id", "result"])
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
|
||||
def write_yaml(self, results):
|
||||
import yaml
|
||||
|
||||
self.yaml_path.write_text(
|
||||
yaml.safe_dump(
|
||||
{"counts": {"pass": 999}, "results": results},
|
||||
sort_keys=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
def run_cli(self):
|
||||
return subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
str(Path(summarize_run.__file__).resolve()),
|
||||
"--csv",
|
||||
str(self.csv_path),
|
||||
"--yaml",
|
||||
str(self.yaml_path),
|
||||
],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
def test_valid_summary_uses_csv_results(self):
|
||||
result = self.run_cli()
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn("source=CSV.result", result.stdout)
|
||||
self.assertIn("pass=1 fail=1 environment-fail=1 skipped=1 not-run=1 total=5", result.stdout)
|
||||
|
||||
def test_yaml_counts_do_not_drive_status_summary(self):
|
||||
self.write_csv([{"id": "TC-001", "result": "fail"}])
|
||||
self.write_yaml([{"id": "TC-001", "result": "fail"}])
|
||||
result = self.run_cli()
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn("pass=0 fail=1", result.stdout)
|
||||
self.assertNotIn("pass=999", result.stdout)
|
||||
|
||||
def test_yaml_result_count_mismatch_is_nonzero(self):
|
||||
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows[:-1]])
|
||||
result = self.run_cli()
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("YAML results 数量不一致", result.stderr)
|
||||
|
||||
def test_id_set_mismatch_is_nonzero(self):
|
||||
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows[:-1]] + [{"id": "TC-999"}])
|
||||
result = self.run_cli()
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("testcase id 集合不一致", result.stderr)
|
||||
|
||||
def test_duplicate_csv_id_is_nonzero(self):
|
||||
rows = self.rows.copy()
|
||||
rows[-1] = {"id": "TC-001", "result": "not-run"}
|
||||
self.write_csv(rows)
|
||||
result = self.run_cli()
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("CSV testcase id 重复", result.stderr)
|
||||
|
||||
def test_missing_csv_id_is_nonzero(self):
|
||||
rows = self.rows.copy()
|
||||
rows[0] = {"id": "", "result": "pass"}
|
||||
self.write_csv(rows)
|
||||
result = self.run_cli()
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("CSV 第 2 行 testcase id 缺失", result.stderr)
|
||||
|
||||
def test_duplicate_or_missing_yaml_id_is_nonzero(self):
|
||||
cases = [
|
||||
([{"id": "TC-001"}, {"id": "TC-001"}] + [{"id": row["id"]} for row in self.rows[2:]], "YAML testcase id 重复"),
|
||||
([{"id": ""}] + [{"id": row["id"]} for row in self.rows[1:]], "YAML results[1] testcase id 缺失"),
|
||||
]
|
||||
for results, expected in cases:
|
||||
with self.subTest(expected=expected):
|
||||
self.write_yaml(results)
|
||||
result = self.run_cli()
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn(expected, result.stderr)
|
||||
|
||||
def test_unknown_or_missing_csv_result_is_nonzero(self):
|
||||
for bad_result, expected in [("unexpected", "result 未知"), ("", "result 缺失")]:
|
||||
with self.subTest(bad_result=bad_result):
|
||||
rows = self.rows.copy()
|
||||
rows[0] = {"id": "TC-001", "result": bad_result}
|
||||
self.write_csv(rows)
|
||||
result = self.run_cli()
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn(expected, result.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,299 @@
|
||||
import csv
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
SCRIPTS = Path(__file__).resolve().parents[1] / "scripts"
|
||||
CATALOG = SCRIPTS / "catalog.py"
|
||||
CASES_CSV = SCRIPTS / "generate_cases_csv.py"
|
||||
RUN_CSV = SCRIPTS / "generate_run_csv.py"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
from two_tier import classify_integration, run_integration_first
|
||||
|
||||
|
||||
def write_mixed_catalog(root: Path) -> None:
|
||||
tester = root / ".pouch" / "tester"
|
||||
(tester / "cases").mkdir(parents=True)
|
||||
(tester / "runs").mkdir()
|
||||
(tester / "catalog.yaml").write_text(
|
||||
yaml.safe_dump(
|
||||
{
|
||||
"version": 2,
|
||||
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
|
||||
"categories": [{"id": "MIXED"}],
|
||||
"risks": ["readonly", "mutating", "destructive"],
|
||||
"default_max_risk": "readonly",
|
||||
"client_suites": {
|
||||
"client-smoke": {"default": True},
|
||||
"client-regression": {"default": False},
|
||||
"safety": {"default": False},
|
||||
},
|
||||
"sessions": {"browser": {"kind": "browser", "reuse": "run"}},
|
||||
},
|
||||
sort_keys=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
journey = {
|
||||
"id": "TC-JOURNEY-001",
|
||||
"test_type": "journey",
|
||||
"category": "MIXED",
|
||||
"title": "journey",
|
||||
"what": "client path",
|
||||
"lifecycle": "active",
|
||||
"tag": "smoke",
|
||||
"risk": "readonly",
|
||||
"driver": "browser",
|
||||
"automation": "automated",
|
||||
"session": "browser",
|
||||
"client_suite": "client-smoke",
|
||||
"platform": "android",
|
||||
"journey": {
|
||||
"goal": "goal",
|
||||
"entry_state": "logged_out",
|
||||
"actions": ["login"],
|
||||
"assertions": ["visible"],
|
||||
"final_state": "done",
|
||||
"failure_policy": "stop",
|
||||
},
|
||||
"fixture": {"account": "test-role"},
|
||||
"cleanup": {"required": False, "reason": "read-only"},
|
||||
"how": {"steps": ["open"]},
|
||||
"expected": [{"kind": "visible-text", "value": "done"}],
|
||||
}
|
||||
integration = {
|
||||
"id": "TC-INTEGRATION-001",
|
||||
"test_type": "integration",
|
||||
"category": "MIXED",
|
||||
"title": "contract",
|
||||
"what": "API contract",
|
||||
"lifecycle": "active",
|
||||
"tag": "slow",
|
||||
"risk": "readonly",
|
||||
"driver": "script",
|
||||
"automation": "automated",
|
||||
"integration_suite": "api-contract",
|
||||
"fixture": {"references": ["isolated-account"]},
|
||||
"request": {"method": "GET", "path": "/health"},
|
||||
"how": {"run": "true"},
|
||||
"cleanup": {"required": False, "reason": "read-only"},
|
||||
"expected": [{"kind": "api-status", "value": 200}],
|
||||
}
|
||||
(tester / "cases" / "mixed.yaml").write_text(
|
||||
yaml.safe_dump({"category": "MIXED", "cases": [journey, integration]}, sort_keys=False),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
class TwoTierCatalogTests(unittest.TestCase):
|
||||
def run_catalog(self, root: Path, *args: str) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(
|
||||
[sys.executable, str(CATALOG), *args, "--root", str(root)],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
def test_default_selection_excludes_integration_and_explicit_selection_includes_it(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
write_mixed_catalog(root)
|
||||
valid = self.run_catalog(root, "validate")
|
||||
self.assertEqual(valid.returncode, 0, valid.stderr)
|
||||
|
||||
default = self.run_catalog(root, "select")
|
||||
self.assertEqual(default.returncode, 0, default.stderr)
|
||||
default_payload = json.loads(default.stdout)
|
||||
self.assertEqual(default_payload["count"], 1)
|
||||
self.assertEqual(default_payload["test_types"], {"journey": 1, "integration": 0})
|
||||
|
||||
integration = self.run_catalog(root, "select", "--test-type", "integration")
|
||||
self.assertEqual(integration.returncode, 0, integration.stderr)
|
||||
integration_payload = json.loads(integration.stdout)
|
||||
self.assertEqual(integration_payload["count"], 1)
|
||||
self.assertEqual(integration_payload["cases"][0]["test_type"], "integration")
|
||||
|
||||
csv_result = subprocess.run(
|
||||
[sys.executable, str(CASES_CSV), "--root", str(root)],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
self.assertEqual(csv_result.returncode, 0, csv_result.stderr)
|
||||
with (root / ".pouch/tester/cases.csv").open(encoding="utf-8-sig", newline="") as handle:
|
||||
rows = list(csv.DictReader(handle))
|
||||
self.assertEqual({row["test_type"] for row in rows}, {"journey", "integration"})
|
||||
self.assertEqual({row["suite"] for row in rows}, {"client-smoke", "api-contract"})
|
||||
|
||||
run_path = root / ".pouch/tester/runs/two-tier.yaml"
|
||||
run_path.write_text(
|
||||
yaml.safe_dump(
|
||||
{"runId": "two-tier", "results": [
|
||||
{"id": "TC-INTEGRATION-001", "result": "pass"},
|
||||
{"id": "TC-JOURNEY-001", "result": "not-run"},
|
||||
]},
|
||||
sort_keys=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
run_csv = subprocess.run(
|
||||
[sys.executable, str(RUN_CSV), "--root", str(root), "--run", str(run_path)],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
self.assertEqual(run_csv.returncode, 0, run_csv.stderr)
|
||||
with run_path.with_suffix(".csv").open(encoding="utf-8-sig", newline="") as handle:
|
||||
report_rows = list(csv.DictReader(handle))
|
||||
self.assertEqual(report_rows[0]["test_type"], "integration")
|
||||
self.assertEqual(report_rows[0]["suite"], "api-contract")
|
||||
|
||||
def test_isolated_full_keeps_risk_and_ignores_authorize_filter(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
write_mixed_catalog(root)
|
||||
catalog_path = root / ".pouch/tester/catalog.yaml"
|
||||
catalog = yaml.safe_load(catalog_path.read_text(encoding="utf-8"))
|
||||
catalog["authorize"] = {"mutating": "AUTHORIZE_MUTATING"}
|
||||
catalog["execution_profile"] = "isolated-full"
|
||||
catalog_path.write_text(yaml.safe_dump(catalog, sort_keys=False), encoding="utf-8")
|
||||
case_path = root / ".pouch/tester/cases/mixed.yaml"
|
||||
cases = yaml.safe_load(case_path.read_text(encoding="utf-8"))
|
||||
journey = cases["cases"][0]
|
||||
journey.update({"risk": "mutating", "tag": "full", "client_suite": "client-regression"})
|
||||
case_path.write_text(yaml.safe_dump(cases, sort_keys=False), encoding="utf-8")
|
||||
|
||||
selected = self.run_catalog(root, "select", "--suite", "client-regression")
|
||||
self.assertEqual(selected.returncode, 0, selected.stderr)
|
||||
payload = json.loads(selected.stdout)
|
||||
self.assertEqual(payload["execution_profile"], "isolated-full")
|
||||
self.assertEqual(payload["risk_filter"], "disabled")
|
||||
self.assertEqual(payload["count"], 1)
|
||||
self.assertEqual(payload["cases"][0]["risk"], "mutating")
|
||||
self.assertEqual(payload["skipped"], [])
|
||||
|
||||
all_active = self.run_catalog(root, "select")
|
||||
self.assertEqual(all_active.returncode, 0, all_active.stderr)
|
||||
all_payload = json.loads(all_active.stdout)
|
||||
self.assertEqual(all_payload["count"], 2)
|
||||
self.assertEqual(all_payload["test_types"], {"journey": 1, "integration": 1})
|
||||
|
||||
def test_isolated_full_run_script_does_not_use_authorize_switch(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
tester = root / ".pouch" / "tester"
|
||||
(tester / "cases").mkdir(parents=True)
|
||||
(tester / "local.yaml").write_text("HOST: http://127.0.0.1\nAUTHORIZE_MUTATING: false\n", encoding="utf-8")
|
||||
(tester / "catalog.yaml").write_text(
|
||||
yaml.safe_dump(
|
||||
{
|
||||
"version": 2,
|
||||
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
|
||||
"categories": [{"id": "SCRIPT"}],
|
||||
"risks": ["readonly", "mutating", "destructive"],
|
||||
"default_max_risk": "readonly",
|
||||
"execution_profile": "isolated-full",
|
||||
"authorize": {"mutating": "AUTHORIZE_MUTATING"},
|
||||
},
|
||||
sort_keys=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(tester / "cases" / "script.yaml").write_text(
|
||||
yaml.safe_dump(
|
||||
{
|
||||
"category": "SCRIPT",
|
||||
"cases": [{
|
||||
"id": "TC-SCRIPT-001",
|
||||
"title": "mutating script",
|
||||
"what": "run isolated script",
|
||||
"lifecycle": "active",
|
||||
"tag": "full",
|
||||
"risk": "mutating",
|
||||
"driver": "script",
|
||||
"how": {"run": "exit 0"},
|
||||
"cleanup": {"steps": ["reset"]},
|
||||
"expected": [{"kind": "script-exit", "value": 0}],
|
||||
}],
|
||||
},
|
||||
sort_keys=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(CATALOG), "run-script", "--root", str(root), "--id", "TC-SCRIPT-001"],
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(json.loads(result.stdout)["result"], "executed")
|
||||
|
||||
|
||||
class IntegrationFirstTests(unittest.TestCase):
|
||||
def test_preflight_block_does_not_create_session(self):
|
||||
calls: list[str] = []
|
||||
result = run_integration_first(
|
||||
preflight_status="blocked",
|
||||
integration_cases=["INT-1"],
|
||||
journey_cases=["J-1"],
|
||||
execute_integration=lambda case: calls.append("integration") or "pass",
|
||||
create_journey_session=lambda: calls.append("session"),
|
||||
execute_journey=lambda session, case: "pass",
|
||||
)
|
||||
self.assertEqual(result["results"][0]["result"], "not-run")
|
||||
self.assertFalse(result["session_created"])
|
||||
self.assertEqual(calls, [])
|
||||
|
||||
def test_integration_failure_propagates_journey_not_run(self):
|
||||
calls: list[str] = []
|
||||
result = run_integration_first(
|
||||
preflight_status="ready",
|
||||
integration_cases=["INT-1", "INT-2"],
|
||||
journey_cases=["J-1", "J-2"],
|
||||
execute_integration=lambda case: calls.append(case) or ("fail" if case == "INT-1" else "pass"),
|
||||
create_journey_session=lambda: calls.append("session"),
|
||||
execute_journey=lambda session, case: "pass",
|
||||
)
|
||||
self.assertEqual(result["status"], "fail")
|
||||
self.assertEqual(calls, ["INT-1", "INT-2"])
|
||||
self.assertFalse(result["session_created"])
|
||||
self.assertEqual([row["result"] for row in result["results"]], ["fail", "pass", "not-run", "not-run"])
|
||||
self.assertEqual(result["results"][2]["blocked_by"], ["INT-1"])
|
||||
|
||||
def test_independent_integration_continues_and_dependency_is_blocked(self):
|
||||
result = run_integration_first(
|
||||
preflight_status="ready",
|
||||
integration_cases=[
|
||||
{"id": "INT-1"},
|
||||
{"id": "INT-2"},
|
||||
{"id": "INT-3", "depends_on": ["INT-1"]},
|
||||
],
|
||||
journey_cases=[],
|
||||
execute_integration=lambda case: "fail" if case["id"] == "INT-1" else "pass",
|
||||
create_journey_session=lambda: object(),
|
||||
execute_journey=lambda session, case: "pass",
|
||||
)
|
||||
self.assertEqual([row["result"] for row in result["results"]], ["fail", "pass", "not-run"])
|
||||
self.assertEqual(result["results"][2]["blocked_by"], ["INT-1"])
|
||||
|
||||
def test_preflight_block_creates_every_result_row(self):
|
||||
result = run_integration_first(
|
||||
preflight_status="blocked",
|
||||
integration_cases=[{"id": "INT-1"}],
|
||||
journey_cases=[{"id": "J-1"}],
|
||||
execute_integration=lambda case: "pass",
|
||||
create_journey_session=lambda: object(),
|
||||
execute_journey=lambda session, case: "pass",
|
||||
)
|
||||
self.assertEqual([row["result"] for row in result["results"]], ["not-run", "not-run"])
|
||||
self.assertEqual(result["results"][0]["blocked_by"], ["test-preflight"])
|
||||
|
||||
def test_service_unreachable_is_environment_failure(self):
|
||||
self.assertEqual(classify_integration(ok=False, service_reachable=False), "environment-fail")
|
||||
self.assertEqual(classify_integration(ok=False, service_reachable=True), "fail")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -12,6 +12,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
CASES = REPO_ROOT / "skills" / "tester" / "scripts" / "generate_cases_csv.py"
|
||||
RUNS = REPO_ROOT / "skills" / "tester" / "scripts" / "generate_run_csv.py"
|
||||
sys.path.insert(0, str(RUNS.parent))
|
||||
from failure_feedback import attach_feedback
|
||||
|
||||
|
||||
def load_module(name: str, path: Path):
|
||||
@@ -29,6 +30,7 @@ class TesterRunRecordTests(unittest.TestCase):
|
||||
root = Path(temp)
|
||||
cases_dir = root / ".pouch" / "tester" / "cases"
|
||||
cases_dir.mkdir(parents=True)
|
||||
(root / ".pouch" / "tester" / "runs").mkdir()
|
||||
(root / ".pouch" / "tester" / "catalog.yaml").write_text(
|
||||
"categories:\n - id: smoke\n title: Smoke\n",
|
||||
encoding="utf-8",
|
||||
@@ -51,6 +53,63 @@ class TesterRunRecordTests(unittest.TestCase):
|
||||
self.assertNotEqual(first, second)
|
||||
self.assertRegex(first, r"^\d{8}T\d{6}Z-smoke-[0-9a-f]{6}$")
|
||||
|
||||
def test_run_csv_preserves_all_results_and_structured_issue_fields(self) -> None:
|
||||
runs = load_module("tester_run_csv_issues", RUNS)
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
cases_dir = root / ".pouch" / "tester" / "cases"
|
||||
cases_dir.mkdir(parents=True)
|
||||
(root / ".pouch" / "tester" / "runs").mkdir()
|
||||
(root / ".pouch" / "tester" / "catalog.yaml").write_text(
|
||||
"categories:\n - id: smoke\n title: Smoke\n", encoding="utf-8"
|
||||
)
|
||||
(cases_dir / "smoke.yaml").write_text(
|
||||
"category: smoke\ncases:\n"
|
||||
" - id: TC-1\n title: One\n what: one\n lifecycle: active\n"
|
||||
" tag: smoke\n risk: readonly\n how: {steps: [open]}\n"
|
||||
" expected: [{kind: visible-text, value: ready}]\n"
|
||||
" - id: TC-2\n title: Two\n what: two\n lifecycle: active\n"
|
||||
" tag: smoke\n risk: readonly\n how: {steps: [open]}\n"
|
||||
" expected: [{kind: visible-text, value: ready}]\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
run = attach_feedback(
|
||||
{
|
||||
"runId": "issues",
|
||||
"results": [
|
||||
{
|
||||
"id": "TC-1",
|
||||
"result": "fail",
|
||||
"issue": {
|
||||
"expected": "ready",
|
||||
"observed": "blank",
|
||||
"artifact_hash": "sha256:abc",
|
||||
"environment": "Android tester",
|
||||
"evidence_path": "evidence/1.png",
|
||||
"reproduction": "open",
|
||||
"next_step": "targeted rerun",
|
||||
},
|
||||
},
|
||||
{"id": "TC-2", "result": "not-run", "blocked_by": ["TC-1"]},
|
||||
],
|
||||
},
|
||||
trigger="pr",
|
||||
target="PR-1",
|
||||
related_issue="ISSUE-1",
|
||||
)
|
||||
run_path = root / ".pouch" / "tester" / "runs" / "issues.yaml"
|
||||
import yaml
|
||||
run_path.write_text(yaml.safe_dump(run, sort_keys=False), encoding="utf-8")
|
||||
out = runs.export_run_csv(root, run_path=run_path)
|
||||
with out.open(encoding="utf-8-sig", newline="") as handle:
|
||||
rows = list(csv.DictReader(handle))
|
||||
self.assertEqual([row["id"] for row in rows], ["TC-1", "TC-2"])
|
||||
self.assertEqual(rows[0]["issue_type"], "product")
|
||||
self.assertEqual(rows[0]["expected"], "ready")
|
||||
self.assertEqual(rows[0]["artifact_hash"], "sha256:abc")
|
||||
self.assertEqual(rows[1]["blocked_by"], "TC-1")
|
||||
self.assertEqual(run["feedback"]["route"], "pr-update-issue-dispatcher-reassign")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Reference in New Issue
Block a user