19 Commits

Author SHA1 Message Date
laily b94aa597ca fix(tester): use remote artifact path for installs 2026-10-16 15:33:12 +08:00
laily 841a2e0ae6 fix(tester): parse bare Android package identity 2026-10-16 15:27:09 +08:00
laily 4d31d074ad docs(tester): align capability gate policy 2026-10-16 15:26:23 +08:00
laily 29f725fb95 fix(test-preflight): route network probes by binding 2026-10-16 14:38:48 +08:00
laily 3a757e3376 fix(tester): assume mobile capabilities in preflight 2026-10-16 14:33:55 +08:00
laily 5b9a693fda fix(tester): route mobile checks by platform binding 2026-10-16 12:38:11 +08:00
laily 9b2d0a28d8 feat(tester): support isolated full regression workflow 2026-10-16 12:24:49 +08:00
laily 3a18a27dc6 feat(tester): add integration-first journey execution 2026-10-15 18:11:28 +08:00
laily c2eb207bd2 feat(test-preflight): add verify and prepare skill 2026-10-15 18:04:59 +08:00
laily ffa7cbd63e feat(tester): add client journey testing standard 2026-10-15 16:48:54 +08:00
laily baed23d02a test(tester): validate CSV regression summary totals 2026-10-15 11:08:33 +08:00
laily 9ba9b3da0f feat(builder): query CI via project status script
When BUILD_VENUE=ci, init confirms tea/glab/gh/curl (or a custom
command) and writes .pouch/builder/ci-status.sh. wait_ci.sh only
polls that script and no longer guesses forge CLIs.
2026-10-15 09:56:51 +08:00
laily 0c9a530390 chore(tester): drop HX_TEST_HOST alias
HOST is the only accepted test-service address from env, local.yaml, or catalog vars.
2026-10-15 09:44:23 +08:00
laily 5fef8247fc feat(tester): require caller artifact locators instead of --release
Mobile runs now take --artifact android|ios=<path-or-url> from the caller
and create a fresh isolated worktree each time. Drop CI release/manifest
lookup.
2026-10-15 01:05:09 +08:00
laily 2fcf16627d fix(tester): expose bundle provisioning failure action 2026-10-14 17:30:11 +08:00
laily 706c553778 feat(tester): provision CI mobile bundles in shared skill 2026-10-14 17:29:45 +08:00
laily db92baf3a7 fix tester multi-root fan-out execution 2026-10-14 15:35:13 +08:00
laily fbf0ca8cd8 fix(deployer): align stale test contracts 2026-10-14 12:21:09 +08:00
laily 8d48d2d335 feat(ship): enforce test manifest handoff 2026-10-14 12:16:55 +08:00
59 changed files with 5521 additions and 662 deletions
+6 -4
View File
@@ -47,7 +47,7 @@ pouch add builder -g # 全局
并负责上层编排。
1. `makefile.builder` 目标:`help/build/clean/version` + 条件 `deb/docker/push*`,
以及 `obtain` / `venue`。变量 `ARCHES/ARCH/VERSION/DIST_DIR/PROJECT_NAME/BUILD_VENUE`。
以及 `obtain` / `venue`。变量 `ARCHES/ARCH/VERSION/DIST_DIR/PROJECT_NAME/BUILD_VENUE/CI_STATUS_SCRIPT`。
`VERSION` 通过 `include <builder>/scripts/version.mk` 从 Git 推导。
多架构清单是 `ARCHES`,见下方「多架构」。
2. 运行 `python3 -I -S <builder>/scripts/check.py . --ready` 直到构建项 PASS。
@@ -66,7 +66,7 @@ COPY_AS=latest.apk
# COPY_SCRIPT= # 仅外部/临时脚本覆盖;项目内脚本写 makefile.builder
```
`BUILD_VENUE=ci` 写在 `makefile.builder`:产物由 CI 构建。`make -f makefile.builder obtain`(或 `push-docker`)等待该 commit 的流水线并 inspect registry,不在本机 `docker push`。
`BUILD_VENUE=ci` 写在 `makefile.builder`:产物由 CI 构建。初始化时确认查询命令(tea / glab / gh / curl),写成 `.pouch/builder/ci-status.sh`。`make -f makefile.builder obtain`(或 `push-docker`)执行该脚本等待流水线并 inspect registry,不在本机 `docker push`。
4. 日常发布就是两条命令:
`make -f makefile.builder deb && make -f makefile.builder push-deb`、
@@ -195,7 +195,7 @@ detached HEAD 用 `detached`,CI 可注入 `BUILD_BRANCH` / `CI_COMMIT_BRANCH`
## Agent 会做什么
1. 初始化:探测 DEB / Docker / Android / iOS / copy 轨道,写出 `makefile.builder` 和 `.env.builder` 骨架,按仓库证据填 `ARCHES`(不默认双架构),不改你已有的 `Makefile` 或 `.env`。
1. 初始化:探测 DEB / Docker / Android / iOS / copy 轨道,写出 `makefile.builder` 和 `.env.builder` 骨架,确认 `BUILD_VENUE`;选 ci 时生成 `.pouch/builder/ci-status.sh`。按仓库证据填 `ARCHES`(不默认双架构),不改你已有的 `Makefile` 或 `.env`。
2. 检查:只跑 `scripts/check.py --ready`,不改文件。
3. 构建:走 `make -f makefile.builder`;只获构建授权时不走会自动上传的目标。
4. 发布:先校验产物,再上传。缺发布键、未授权、或可能覆盖同版本时停下问你。
@@ -223,7 +223,9 @@ Docker 只有你点名镜像时才走。token 不会写进输出、提交或回
| `scripts/verify_deb.sh` | 核对包元数据、内容与 SHA-256 |
| `scripts/copy_artifact.sh` | 按 `COPY_SCRIPT` 分发产物,未配置时拷到 `COPY_DEST` |
| `scripts/obtain.sh` | 对精确 SHA 拿到产物;`BUILD_VENUE=ci` 时等流水线再 inspect |
| `scripts/wait_ci.sh` | 只等该 SHA 的 GitLab/GitHub/Gitea 流水线,不触发;Gitea 读取 workflow jobs |
| `scripts/wait_ci.sh` | 轮询项目 `CI_STATUS_SCRIPT`,只等该 SHA,不触发 |
| `scripts/init_ci_status.sh` | 初始化时把 tea/glab/gh/curl 模板拷到 `.pouch/builder/ci-status.sh` |
| `templates/ci-status/` | tea / glab / gh / curl 查询脚本模板;契约见 [ci-status.md](references/ci-status.md) |
| `templates/makefile.android.builder` | Android APK + copy 轨道骨架 |
| `templates/makefile.ios.builder` | iOS IPA + copy 轨道骨架 |
+3 -2
View File
@@ -47,7 +47,8 @@ worktree,也可以指向已配对的远程 Mac/其它构建机;不要把远
3. 没有 `makefile.builder`:按轨道拷模板。deb/docker 用 `templates/makefile.builder`;仅 android 用 `templates/makefile.android.builder`;仅 ios 用 `templates/makefile.ios.builder`。按轨道删掉未使用的 deb/docker/push* 段,把 `build` 的 TODO 换成仓库里已有的真实编译命令(可从用户 Makefile 抄配方,但不要 `include` 或递归调用它),且配方必须使用 `$(ARCH)`。Android/iOS 的 `ARCHES` 是构建机 ISA(fat APK/IPA 仍只留本机一项)。按 [contract.md](references/contract.md) 填 `ARCHES`:`debian/control` / CI matrix / 用户点名的子集;否则只写本机(`x86_64`→`amd64`,`aarch64`→`arm64`)。不要默认双架构;看不出交叉编法时只留一项并说明。`include` builder 的 `scripts/version.mk`。多产物把 `push` 改成 `push: push-deb push-docker push-copy` 的子集。不要改用户 Makefile。
4. 已有 `makefile.builder`:跑检查;按 FAIL 给出修补说明。不覆盖 `build`。用户明确要求按契约改时再加 `ARCHES`,并确认配方里有 `$(ARCH)`。不要调用 create-makefile(版本规则冲突)。
5. 没有 `.env.builder`:把 `<skill-dir>/templates/env.builder` 拷到项目根(注释键,不含值)。缺发布键时在报告里给出可粘贴示例,并把 `.env.builder` 加入 `.gitignore`,不要提交。
6. 运行 `python3 -I -S <skill-dir>/scripts/check.py <project-dir> --ready`。结构校验通过且当前轨道能构建才称「完成」;只缺发布键是「部分完成」。契约 FAIL 或轨道工具缺失是「阻塞」。除非用户明确要求,不提交、不推送、不上传。
6. 确认 `BUILD_VENUE`:`local` 或 `ci`,不要猜。选 `ci` 时读 [ci-status.md](references/ci-status.md),确认查询命令(`tea` / `glab` / `gh` / `curl`),执行 `<skill-dir>/scripts/init_ci_status.sh --root <project-dir> --kind <kind>`,在 `makefile.builder` 写入 `BUILD_VENUE=ci` 和 `CI_STATUS_SCRIPT=.pouch/builder/ci-status.sh`。用户给了自定义查询命令时,把命令写进同一份脚本并保持 `STATUS=` 输出契约。
7. 运行 `python3 -I -S <skill-dir>/scripts/check.py <project-dir> --ready`。结构校验通过且当前轨道能构建才称「完成」;只缺发布键是「部分完成」。契约 FAIL 或轨道工具缺失是「阻塞」。除非用户明确要求,不提交、不推送、不上传。
```text
## builder 初始化:完成 | 部分完成 | 阻塞
@@ -90,6 +91,6 @@ COPY_AS=latest.apk
按 [obtain.md](references/obtain.md) 执行 `scripts/obtain.sh --sha <commit>`。
- `BUILD_VENUE=local`:当前 HEAD 必须是该 SHA,跑 makefile 的 docker/deb。
- `BUILD_VENUE=ci`:等该 SHA 的 GitLab/GitHub/Gitea 流水线,再 inspect `version.sh --docker` 的镜像 tag。不触发流水线,不本机 push。
- `BUILD_VENUE=ci`:`wait_ci.sh` 轮询项目的 `CI_STATUS_SCRIPT`,再 inspect `version.sh --docker` 的镜像 tag。不触发流水线,不本机 push。缺脚本则停下,回到初始化。
- 失败(无流水线、job 失败、digest 不对)必须原样返回给调用方,不得标 `artifact_built`。
- 合进默认分支若项目不跑 CI,这是正确失败;预览用 `dev/*`,正式用 tag。
+55
View File
@@ -0,0 +1,55 @@
# CI 状态查询脚本
`BUILD_VENUE=ci` 时,流水线状态由**项目脚本**查询。builder 只轮询这份脚本,
不在 skill 里猜 gh / glab / tea / curl。
## 初始化
选 `ci` 后确认查询方式,再生成脚本:
```bash
<skill-dir>/scripts/init_ci_status.sh --root <project> --kind tea|glab|gh|curl
```
写出 `.pouch/builder/ci-status.sh`(可执行、进 git)。`makefile.builder` 写:
```makefile
BUILD_VENUE ?= ci
CI_STATUS_SCRIPT ?= .pouch/builder/ci-status.sh
```
相对路径只能在 `.pouch/builder/` 下。凭据仍放 `.env.builder`(如 `GITEA_TOKEN`),
不要写进脚本参数或日志。
用户给了自定义命令时,把该命令写进同一份脚本,输出必须符合下方契约。
## 调用
```bash
.pouch/builder/ci-status.sh --sha <40-character-sha>
```
只查询、不触发流水线。`BUILDER_PROJECT_ROOT` 是项目根。
## 输出
退出 0 且 stdout 含 `STATUS=` 键值(可多行或同一行):
| STATUS | 含义 |
| --- | --- |
| `empty` | 该 SHA 还没有流水线 |
| `running` | 已出现、尚未结束 |
| `success` | 该 SHA 的流水线成功 |
| `failed` | 失败、取消或超时 |
| `error` | 查询本身失败(鉴权、CLI 缺失) |
可选:`ID=`、`PIPELINE=`(URL)、`JOBS=`。
| 退出码 | 含义 |
| --- | --- |
| 0 | 已查出状态(含 empty/running/failed) |
| 2 | 用法错误 |
| 3 | 无法查询(缺 CLI、未登录、被拒绝) |
`wait_ci.sh` 按这份输出轮询:`empty` 超过出现窗口记 `no_pipeline`;
`success` 才继续 obtain;`failed` 原样失败。
+4 -3
View File
@@ -52,7 +52,7 @@ make -f makefile.builder <target>
| `push-docker` | 同时有多种发布轨道的项目 | `BUILD_VENUE=local` 时仅调 `publish_docker.sh`;`ci` 时仅调 `obtain.sh` |
| `push-copy` | 同时有多种发布轨道的项目 | 仅调 builder 的 `copy_artifact.sh`;由它按配置选择内置拷贝或自定义脚本 |
| `push` | 单一产物类型时必备;多产物项目为聚合 | 依序调用对应 push-* 或直接调脚本;是发布的唯一 make 入口 |
| `obtain` | 有产物的项目 | 调用 `obtain.sh`,对精确 commit SHA 拿到已存在的产物。`BUILD_VENUE=ci` 时等待该 SHA 的 forge 流水线再 inspect,不在本机 `docker push` |
| `obtain` | 有产物的项目 | 调用 `obtain.sh`,对精确 commit SHA 拿到已存在的产物。`BUILD_VENUE=ci` 时轮询 `CI_STATUS_SCRIPT` 再 inspect,不在本机 `docker push` |
| `venue` | 可选(新模板必备) | 打印一行 `BUILD_VENUE`(`local` 或 `ci`) |
规则:
@@ -82,7 +82,8 @@ make -f makefile.builder <target>
| 变量 | 默认 | 说明 |
|------|------|------|
| `BUILD_VENUE` | `local` | 产物在哪构建:`local` 本机 make;`ci` 由 GitLab/GitHub/Gitea 流水线构建。这不是新轨道。写在 `makefile.builder` 并提交。`ci` 时 `push-docker` 委托 `obtain.sh`(等流水线 + inspect),不跑 `publish_docker.sh` |
| `BUILD_VENUE` | `local` | 产物在哪构建:`local` 本机 make;`ci` 由项目 CI 流水线构建。这不是新轨道。写在 `makefile.builder` 并提交。`ci` 时 `push-docker` 委托 `obtain.sh`(等流水线 + inspect),不跑 `publish_docker.sh` |
| `CI_STATUS_SCRIPT` | `.pouch/builder/ci-status.sh` | `BUILD_VENUE=ci` 时必填。项目内查询该 SHA 流水线状态的可执行脚本,相对路径只能在 `.pouch/builder/`。初始化时按 tea / glab / gh / curl 生成,契约见 [ci-status.md](ci-status.md) |
| `ARCHES` | `amd64` | 本项目要发布的架构清单,空格分隔,子集只能是 `amd64` `arm64`。写在 `makefile.builder` 里并提交;不要放进 `.env.builder`。缺省或无 `arches` 目标时视为单架构 `amd64`。不要默认写成双架构 |
| `ARCH` | `$(firstword $(ARCHES))` | 这一次构建的架构。必须落在 `ARCHES` 里;非法值必须 `$(error)` 并提示合法值。命令行/环境传入时只构建/上传这一项 |
| `VERSION` | `` (空) | 为空时按「版本号」节从 Git 祖先稳定 tag 推导;禁止 `sort -V` 取全局最新 |
@@ -189,7 +190,7 @@ include $(HOME)/.pouch/skills/builder/scripts/version.mk
`PLATFORMS` 优先级(高 → 低):用户这轮 `--platform` 或已 export 的 `PLATFORMS` → 命令行 `ARCH=` 映射的 `linux/$(ARCH)`(由 `push-docker` 传入)→ `.env.builder` 的 `PLATFORMS` → `makefile.builder` 的 `ARCHES` → `linux/amd64`。
`BUILD_VENUE=ci` 时 Docker 发布路径是 `obtain.sh`:只认 **该 commit SHA** 的流水线。没有流水线、job 失败、超时、或 Harbor inspect 不到 `version.sh --docker` 算出的 tag,一律非零退出。不触发流水线。GitHub 用 `gh`、GitLab 用 `glab`、Gitea 用 `curl` 读取 Actions API;Gitea 会再次校验 `head_sha` 并读取 workflow jobs。合进默认分支若项目声明不构建,这是正确失败;调用方应打 tag 或推送到项目会构建的 ref(例如 `dev/*`)。CI job 若已经 `docker push`,obtain 成功即视为产物已在 registry,不要再本机 push 一次。
`BUILD_VENUE=ci` 时 Docker 发布路径是 `obtain.sh`:`wait_ci.sh` 只轮询 `CI_STATUS_SCRIPT`,只认 **该 commit SHA**。没有流水线、job 失败、超时、或 Harbor inspect 不到 `version.sh --docker` 算出的 tag,一律非零退出。不触发流水线。查询命令在初始化时写成项目脚本,运行时不再猜测 gh / glab / tea / curl。合进默认分支若项目声明不构建,这是正确失败;调用方应打 tag 或推送到项目会构建的 ref(例如 `dev/*`)。CI job 若已经 `docker push`,obtain 成功即视为产物已在 registry,不要再本机 push 一次。
配置来源优先级:shell 已显式设置的值 > 项目根 `.env.builder` > 失败并询问用户。
`.env.builder` 由 builder 脚本加载(不回显任何值);当前 shell 已设置的值优先。
+2 -2
View File
@@ -36,13 +36,13 @@ TAG=1.4.2
| 0 | 产物已核验 | `artifact_built` 完成 |
| 1 | 该 SHA 无流水线、job 失败、超时、inspect 失败 | 标 `failed`,不得 publish/deploy |
| 2 | 用法或配置错误 | 标 `blocked` |
| 3 | glab/gh/curl 缺失或读不到该项目 | 停;可用**已打开且已登录**的同 host Orca tab 查流水线。禁止新开未登录页 |
| 3 | 项目 CI 状态脚本无法查询 | 停;回到初始化补 `CI_STATUS_SCRIPT`,或用**已打开且已登录**的同 host Orca tab 查流水线。禁止新开未登录页 |
## `ci` 规则
- 只认该 SHA 的 pipeline,不认默认分支「最新一次绿」。
- 不触发流水线。
- Gitea 使用 Actions REST API 的 `head_sha` 精确过滤,并读取每个 workflow run 的 jobs;私有仓库可在 `.env.builder` 配置 `GITEA_TOKEN`。自定义 Gitea host 可设置 `BUILDER_CI_FORGE=gitea`,API 前缀默认是 `https://<origin-host>/api/v1`,也可用 `GITEA_API_URL` 覆盖。
- 查询命令是项目 `.pouch/builder/ci-status.sh`(`CI_STATUS_SCRIPT`),初始化时按 tea / glab / gh / curl 写好。凭据仍放 `.env.builder`。
- job 变绿不够,docker 轨道必须 inspect 到 digest。
- 合进 `main` 若项目不构建,`no_pipeline` 是正确失败。预览走项目会构建的 ref(如 `dev/*`),正式走 tag。
+49 -52
View File
@@ -551,8 +551,8 @@ def configured_value(
return os.environ.get(key, "") or env_values.get(key, "") or (make_values or {}).get(key, "")
def resolve_copy_script(project: Path, script: str) -> tuple[Path | None, str | None]:
"""Resolve COPY_SCRIPT without allowing arbitrary project-relative paths."""
def resolve_builder_local_script(project: Path, script: str) -> tuple[Path | None, str | None]:
"""Resolve a project script; relative paths must stay under .pouch/builder/."""
configured = Path(script).expanduser()
if configured.is_absolute():
return configured, None
@@ -561,10 +561,17 @@ def resolve_copy_script(project: Path, script: str) -> tuple[Path | None, str |
allowed_root = (project / ".pouch" / "builder").resolve()
resolved.relative_to(allowed_root)
except ValueError:
return None, "relative COPY_SCRIPT must be under .pouch/builder/"
return None, "relative script must be under .pouch/builder/"
return resolved, None
def resolve_copy_script(project: Path, script: str) -> tuple[Path | None, str | None]:
path, error = resolve_builder_local_script(project, script)
if error:
return None, "relative COPY_SCRIPT must be under .pouch/builder/"
return path, None
def check_copy_script(
report: Report,
project: Path,
@@ -591,6 +598,9 @@ def check_copy_script(
def read_venue(project: Path) -> str:
env = (os.environ.get("BUILD_VENUE") or "").strip()
if env in ("local", "ci"):
return env
result = run_make(project, "venue")
if not has_no_rule(result):
echoed = parse_dry_echo(result)
@@ -604,34 +614,35 @@ def read_venue(project: Path) -> str:
return raw or "local"
def origin_forge(project: Path) -> str:
result = subprocess.run(
["git", "-C", str(project), "remote", "get-url", "origin"],
capture_output=True,
text=True,
check=False,
def check_ci_status_script(
report: Report,
project: Path,
env_values: dict[str, str],
make_values: dict[str, str],
) -> None:
if read_venue(project) != "ci":
report.add(SKIP, 15, "CI_STATUS_SCRIPT 可执行", "BUILD_VENUE is not ci")
return
script = configured_value("CI_STATUS_SCRIPT", env_values, make_values)
problems = []
if not script:
problems.append("BUILD_VENUE=ci requires CI_STATUS_SCRIPT")
path = None
resolve_error = None
else:
path, resolve_error = resolve_builder_local_script(project, script)
if resolve_error:
problems.append(resolve_error)
elif path is not None and not path.is_file():
problems.append("CI_STATUS_SCRIPT does not point to a file")
elif path is not None and not os.access(path, os.X_OK):
problems.append("CI_STATUS_SCRIPT is not executable")
report.add(
FAIL if problems else PASS,
15,
"CI_STATUS_SCRIPT 可执行",
"\n".join(problems) if problems else f"ci status script: {script}",
)
url = (result.stdout or "").strip()
configured = env_file_values(project).get("BUILDER_CI_FORGE", "").strip().lower()
forced = (os.environ.get("BUILDER_CI_FORGE") or configured).strip().lower()
if forced in ("github", "gitlab", "gitea"):
return forced
if "github.com" in url:
return "github"
host_match = re.search(r"(?:https?://|ssh://git@|git@)([^/:]+)", url)
host = host_match.group(1).lower() if host_match else ""
builder_keys = env_file_keys(project)
if (
"gitea" in host
or "GITEA_TOKEN" in builder_keys
or "GITEA_API_URL" in builder_keys
or os.environ.get("GITEA_TOKEN")
or os.environ.get("GITEA_API_URL")
):
return "gitea"
if url:
return "gitlab"
return ""
def check_ready_toolchain(
@@ -655,28 +666,7 @@ def check_ready_toolchain(
lines.append(
f"docker: {'found' if docker_ok else 'optional for inspect; obtain fails without it'}"
)
forge = origin_forge(project) if project is not None else ""
if forge == "github":
gh_ok = shutil.which("gh") is not None
lines.append(f"gh: {'found' if gh_ok else 'MISSING (blocks ci venue)'}")
if not gh_ok:
problems.append("install gh and authenticate for this GitHub repo")
elif forge == "gitlab":
glab_ok = shutil.which("glab") is not None
lines.append(f"glab: {'found' if glab_ok else 'MISSING (blocks ci venue)'}")
if not glab_ok:
problems.append("install glab and authenticate for this GitLab project")
elif forge == "gitea":
curl_ok = shutil.which("curl") is not None
lines.append(f"curl: {'found' if curl_ok else 'MISSING (blocks ci venue)'}")
lines.append("Gitea API: /api/v1; GITEA_TOKEN optional for public repositories")
if not curl_ok:
problems.append("install curl to read Gitea Actions for this repository")
else:
glab_ok = shutil.which("glab") is not None
lines.append(f"glab: {'found' if glab_ok else 'MISSING (blocks ci venue)'}")
if not glab_ok:
problems.append("install glab and authenticate for this GitLab project")
lines.append("CI status: project CI_STATUS_SCRIPT (tea/glab/gh/curl)")
elif docker_project:
docker_ok = shutil.which("docker") is not None
lines.append(f"docker: {'found' if docker_ok else 'MISSING (blocks docker track)'}")
@@ -867,6 +857,7 @@ def main(argv: list[str] | None = None) -> int:
check_ready_env_keys(
report, project, deb_project=deb_project, docker_project=docker_project
)
check_ci_status_script(report, project, env_file_values(project), makefile_values(project) if (project / BUILDER_MAKEFILE).is_file() else {})
print()
print(f"RESULT: FAILED ({report.failures} check(s) failed)")
return 1
@@ -963,6 +954,12 @@ def main(argv: list[str] | None = None) -> int:
docker_project=docker_project,
copy_project=copy_project,
)
check_ci_status_script(
report,
project,
env_file_values(project),
makefile_values(project),
)
total_fail = report.failures
print()
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env bash
set -euo pipefail
# Copy a CI status template into the project.
# Usage: init_ci_status.sh --root DIR --kind tea|glab|gh|curl
usage() {
echo "Usage: init_ci_status.sh --root DIR --kind tea|glab|gh|curl" >&2
}
root=
kind=
while (($#)); do
case "$1" in
--root) [[ $# -ge 2 ]] || { usage; exit 2; }; root=$2; shift 2 ;;
--kind) [[ $# -ge 2 ]] || { usage; exit 2; }; kind=$2; shift 2 ;;
-h|--help) usage; exit 0 ;;
*) usage; exit 2 ;;
esac
done
if [[ -z "$root" || -z "$kind" ]]; then
usage
exit 2
fi
case "$kind" in
tea|glab|gh|curl) ;;
*) echo "Error: --kind must be tea, glab, gh, or curl" >&2; exit 2 ;;
esac
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
src=$script_dir/../templates/ci-status/${kind}.sh
if [[ ! -f "$src" ]]; then
echo "Error: missing template: $src" >&2
exit 2
fi
dest=$root/.pouch/builder/ci-status.sh
mkdir -p "$(dirname "$dest")"
cp "$src" "$dest"
chmod +x "$dest"
echo "$dest"
+2 -3
View File
@@ -10,14 +10,13 @@ set -euo pipefail
#
# BUILD_VENUE (makefile.builder, default local):
# local HEAD must be SHA; run `make -f makefile.builder docker` or `deb`
# ci wait for the forge pipeline of this SHA, then inspect the image
# (GitHub Actions, GitLab CI, or Gitea Actions)
# ci wait for this SHA via the project CI_STATUS_SCRIPT, then inspect the image
#
# Exit:
# 0 artifact verified; prints ARTIFACT= / DIGEST= / SHA= / VENUE=
# 1 no pipeline, job failed, inspect failed
# 2 usage / config
# 3 forge CLI cannot read the project (ci venue)
# 3 CI status script cannot query (ci venue)
usage() {
sed -n '2,20p' "$0"
+84 -358
View File
@@ -1,25 +1,17 @@
#!/usr/bin/env bash
set -euo pipefail
# Wait until the forge pipeline for an exact commit SHA finishes.
# Poll the project CI status script until the exact SHA finishes.
# Does not trigger a pipeline. Does not print tokens.
#
# Usage:
# wait_ci.sh --sha SHA [--root DIR] [--timeout SEC]
#
# Supported forges:
# GitHub Actions (gh), GitLab CI (glab), and Gitea Actions (curl + REST API).
# Set BUILDER_CI_FORGE=gitea for a custom Gitea host that does not contain
# "gitea" in its hostname. GITEA_API_URL may override the /api/v1 endpoint.
#
# Exit:
# 0 pipeline succeeded; prints "OK pipeline=<url> sha=<sha>"
# 1 no pipeline for this SHA, or a job failed/canceled
# 2 usage / bad arguments
# 3 forge CLI missing, or it cannot read this project
# The project script is CI_STATUS_SCRIPT (makefile.builder, default
# .pouch/builder/ci-status.sh). See references/ci-status.md.
usage() {
sed -n '2,18p' "$0"
sed -n '2,14p' "$0"
}
sha=
@@ -51,7 +43,6 @@ if [[ -z "$sha" ]]; then
fi
sha=$(git -C "$root" rev-parse "$sha")
# Load only builder configuration. Shell values win and values are never echoed.
if [[ -f "$root/.env.builder" ]]; then
while IFS='=' read -r key value; do
key=${key%%[[:space:]]*}
@@ -67,391 +58,126 @@ if [[ -f "$root/.env.builder" ]]; then
done < <(grep -v '^[[:space:]]*$' "$root/.env.builder")
fi
python_json() {
python3 -c "$1" "${@:2}"
}
parse_origin() {
python_json '
import os, re
url = os.environ["WAIT_CI_ORIGIN"].strip()
url = re.sub(r"^git@([^:]+):", r"ssh://git@\1/", url)
url = re.sub(r"^ssh://git@([^/]+):", r"ssh://git@\1/", url)
if re.match(r"^https?://", url):
m = re.match(r"^(https?)://([^/]+)(?:/(.*))?$", url)
scheme, host, path = (m.group(1), m.group(2), m.group(3) or "") if m else ("", "", "")
else:
m = re.match(r"^ssh://git@([^/]+)/(.+)$", url)
scheme, host, path = ("https", m.group(1), m.group(2)) if m else ("", "", "")
path = re.sub(r"\.git$", "", path)
override = os.environ.get("BUILDER_CI_FORGE", "").strip().lower()
if override in ("github", "gitlab", "gitea"):
kind = override
elif host == "github.com" or host.endswith(".github.com"):
kind = "github"
elif ("gitea" in host.lower() or os.environ.get("GITEA_TOKEN") or
os.environ.get("GITEA_API_URL")):
kind = "gitea"
else:
kind = "gitlab"
print(kind, host, path, (scheme or "https") + "://" + host)
'
}
origin=$(git -C "$root" remote get-url origin 2>/dev/null || true)
if [[ -z "$origin" ]]; then
echo "Error: no origin remote" >&2
exit 2
fi
export WAIT_CI_ORIGIN=$origin
read -r forge_kind forge_host forge_path forge_base <<<"$(parse_origin)"
if [[ -z "$forge_kind" || -z "$forge_path" ]]; then
echo "Error: cannot parse origin: $origin" >&2
exit 2
fi
export WAIT_CI_PATH=$forge_path
export WAIT_CI_SHA=$sha
gitlab_pipelines() {
local encoded
encoded=$(python_json 'import os,urllib.parse; print(urllib.parse.quote(os.environ["WAIT_CI_PATH"], safe=""))')
glab api "projects/${encoded}/pipelines?sha=${sha}&per_page=20"
}
gitlab_jobs() {
local encoded pid=$1
encoded=$(python_json 'import os,urllib.parse; print(urllib.parse.quote(os.environ["WAIT_CI_PATH"], safe=""))')
glab api "projects/${encoded}/pipelines/${pid}/jobs"
}
github_runs() {
gh run list --repo "$forge_path" --commit "$sha" --limit 20 \
--json databaseId,status,conclusion,url,headSha
}
gitea_api_base=${GITEA_API_URL:-$forge_base/api/v1}
gitea_api_base=${gitea_api_base%/}
if [[ "$gitea_api_base" != */api/v1 ]]; then
gitea_api_base+="/api/v1"
fi
gitea_repo_path=$(python_json 'import os,urllib.parse; print(urllib.parse.quote(os.environ["WAIT_CI_PATH"], safe="/"))')
gitea_curl() {
local url=$1
if [[ -n "${GITEA_TOKEN:-}" ]]; then
# Process substitution keeps the token out of curl's argument list and logs.
curl -fsS --retry 2 --connect-timeout 10 --max-time 30 \
--config <(printf 'header = "Authorization: token %s"\n' "$GITEA_TOKEN") "$url"
resolve_status_script() {
local configured=${CI_STATUS_SCRIPT:-}
if [[ -z "$configured" && -f "$root/makefile.builder" ]]; then
configured=$(make -s --no-print-directory -C "$root" -f makefile.builder print-CI_STATUS_SCRIPT 2>/dev/null | tail -n 1 || true)
fi
configured=${configured:-.pouch/builder/ci-status.sh}
local path
if [[ "$configured" == /* ]]; then
path=$configured
else
curl -fsS --retry 2 --connect-timeout 10 --max-time 30 "$url"
path=$root/$configured
case "$configured" in
.pouch/builder/*) ;;
*)
echo "Error: relative CI_STATUS_SCRIPT must be under .pouch/builder/: $configured" >&2
exit 2
;;
esac
fi
if [[ ! -f "$path" ]]; then
echo "Error: CI status script missing: $path" >&2
echo "Run builder init with BUILD_VENUE=ci to write .pouch/builder/ci-status.sh" >&2
exit 2
fi
if [[ ! -x "$path" ]]; then
echo "Error: CI status script is not executable: $path" >&2
exit 2
fi
printf '%s' "$path"
}
gitea_runs() {
gitea_curl "${gitea_api_base}/repos/${gitea_repo_path}/actions/runs?head_sha=${sha}&limit=20&page=1"
}
gitea_jobs() {
gitea_curl "${gitea_api_base}/repos/${gitea_repo_path}/actions/runs/$1/jobs?limit=100&page=1"
}
summarize_gitlab() {
python_json '
import json, sys
try:
data = json.loads(sys.stdin.read())
except json.JSONDecodeError:
print("error")
raise SystemExit(0)
if isinstance(data, dict) and data.get("message"):
print("auth", data.get("message", "")[:80].replace(" ", "_"))
raise SystemExit(0)
if not isinstance(data, list):
print("error")
raise SystemExit(0)
if not data:
print("empty")
raise SystemExit(0)
p = data[0]
print(p.get("id"), p.get("status"), p.get("web_url", ""))
parse_status() {
python3 -c '
import sys
text = sys.stdin.read()
values = {}
for raw in text.replace("\n", " ").split():
if "=" in raw:
key, value = raw.split("=", 1)
values[key.upper()] = value
print(values.get("STATUS", "error").lower())
print(values.get("ID", "-"))
print(values.get("PIPELINE", "-"))
print(values.get("JOBS", ""))
'
}
summarize_jobs() {
python_json '
import json, sys
data = json.loads(sys.stdin.read())
parts = []
for j in data if isinstance(data, list) else []:
parts.append("%s=%s" % (j.get("name"), j.get("status")))
print(",".join(parts))
'
}
status_script=$(resolve_status_script)
export BUILDER_PROJECT_ROOT=$root
summarize_github() {
python_json '
import json, sys
data = json.loads(sys.stdin.read())
if not data:
print("empty")
raise SystemExit(0)
r = data[0]
status = r.get("status")
conc = r.get("conclusion")
if status == "completed" and conc == "success":
st = "success"
elif status == "completed":
st = conc or "failed"
else:
st = status or "running"
print(r.get("databaseId"), st, r.get("url", ""))
'
}
summarize_gitea_runs() {
python_json '
import json, sys
try:
data = json.loads(sys.stdin.read())
except json.JSONDecodeError:
print("error")
raise SystemExit(0)
if isinstance(data, dict) and data.get("message") and not (data.get("workflow_runs") or data.get("runs")):
print("auth", data.get("message", "")[:80].replace(" ", "_"))
raise SystemExit(0)
if isinstance(data, dict):
data = data.get("workflow_runs", data.get("runs", []))
if not isinstance(data, list):
print("error")
raise SystemExit(0)
want = sys.argv[1] if len(sys.argv) > 1 else ""
for run in data:
head = run.get("head_sha") or run.get("headSha") or (run.get("head_commit") or {}).get("id")
if not head or head.lower() != want.lower():
continue
run_id = run.get("id") or run.get("database_id")
if run_id is None:
continue
status = str(run.get("status") or "").lower()
conclusion = str(run.get("conclusion") or run.get("result") or "").lower()
url = run.get("html_url") or run.get("web_url") or run.get("url") or "-"
print("%s\t%s\t%s\t%s" % (run_id, status, conclusion, url))
' "$sha"
}
summarize_gitea_jobs() {
python_json '
import json, sys
try:
data = json.loads(sys.stdin.read())
except json.JSONDecodeError:
print("error")
raise SystemExit(0)
if isinstance(data, dict) and data.get("message") and not data.get("jobs"):
print("auth", data.get("message", "")[:80].replace(" ", "_"))
raise SystemExit(0)
if isinstance(data, dict):
data = data.get("jobs", [])
if not isinstance(data, list):
print("error")
raise SystemExit(0)
want = sys.argv[1] if len(sys.argv) > 1 else ""
parts = []
state = "success" if data else "empty"
for job in data:
head = job.get("head_sha") or job.get("headSha")
if head and want and head.lower() != want.lower():
continue
name = str(job.get("name") or job.get("id") or "job").replace(" ", "_")
status = str(job.get("status") or "").lower()
result = str(job.get("conclusion") or job.get("result") or "").lower()
value = result or status or "unknown"
parts.append("%s=%s" % (name, value))
if value in ("failure", "failed", "cancelled", "canceled", "skipped", "timed_out", "action_required"):
state = "failed"
elif value not in ("success", "succeeded", "completed"):
state = "running"
if not parts:
state = "empty"
print("%s\t%s" % (state, ",".join(parts)))
' "$sha"
}
gitea_status_line() {
local raw runs
if ! raw=$(gitea_runs); then
echo "error"
return 0
query() {
local out err rc
err=$(mktemp)
set +e
out=$(BUILDER_PROJECT_ROOT="$root" "$status_script" --sha "$sha" 2>"$err")
rc=$?
set -e
if [[ $rc -ne 0 ]]; then
echo "Error: CI status script exited $rc" >&2
sed '/token/Id; /Bearer/d; /Authorization/Id' "$err" >&2
rm -f "$err"
if [[ $rc -eq 2 ]]; then
exit 2
fi
runs=$(printf '%s' "$raw" | summarize_gitea_runs)
if [[ "$runs" == auth* || "$runs" == error ]]; then
printf '%s\n' "$runs"
return 0
fi
if [[ -z "$runs" ]]; then
echo "empty"
return 0
fi
local overall=success ids= urls= jobs_all= first=1
while IFS=$'\t' read -r id run_status conclusion url; do
[[ -z "$id" ]] && continue
local jobs job_line job_state job_summary
if ! jobs=$(gitea_jobs "$id"); then
echo "error"
return 0
fi
job_line=$(printf '%s' "$jobs" | summarize_gitea_jobs "$sha")
job_state=${job_line%%$'\t'*}
job_summary=${job_line#*$'\t'}
if [[ "$job_state" == auth* || "$job_state" == error ]]; then
printf '%s\n' "$job_state"
return 0
fi
if [[ "$first" -eq 0 ]]; then
ids+=","; urls+=","; jobs_all+=","
fi
ids+="$id"; urls+="$url"; jobs_all+="$job_summary"; first=0
if [[ "$run_status" =~ ^(failure|failed|cancelled|canceled|skipped|timed_out|action_required)$ || "$conclusion" =~ ^(failure|failed|cancelled|canceled|skipped|timed_out|action_required)$ || "$job_state" == failed ]]; then
overall=failed
elif [[ "$overall" == success && ( "$job_state" == running || "$job_state" == empty || ( "$run_status" != completed && "$run_status" != success ) ) ]]; then
overall=running
fi
done <<<"$runs"
printf '%s %s %s %s\n' "$ids" "$overall" "$urls" "$jobs_all"
}
if [[ "$forge_kind" == github ]]; then
if ! command -v gh >/dev/null 2>&1; then
echo "Error: gh is required to watch GitHub Actions for $forge_host" >&2
exit 3
fi
elif [[ "$forge_kind" == gitlab ]]; then
if ! command -v glab >/dev/null 2>&1; then
echo "Error: glab is required to watch GitLab CI for $forge_host" >&2
exit 3
fi
else
if ! command -v curl >/dev/null 2>&1; then
echo "Error: curl is required to watch Gitea Actions for $forge_host" >&2
exit 3
fi
fi
rm -f "$err"
printf '%s' "$out"
}
started=$(date +%s)
first_seen=0
empty_since=$started
probe_auth() {
local out rc kind
if [[ "$forge_kind" == github ]]; then
set +e
out=$(github_runs 2>&1)
rc=$?
set -e
if [[ $rc -ne 0 ]]; then
echo "Error: gh cannot list runs for $forge_path" >&2
echo "$out" | sed '/token/Id; /Bearer/d' >&2
exit 3
fi
return
fi
if [[ "$forge_kind" == gitlab ]]; then
set +e
out=$(gitlab_pipelines 2>&1)
rc=$?
set -e
if [[ $rc -ne 0 ]]; then
echo "Error: glab cannot list pipelines for $forge_path on $forge_host" >&2
echo "$out" | sed '/token/Id; /Bearer/d' >&2
exit 3
fi
kind=$(printf '%s' "$out" | summarize_gitlab | awk '{print $1}')
if [[ "$kind" == auth ]]; then
echo "Error: glab is authenticated but cannot read $forge_path" >&2
exit 3
fi
return
fi
set +e
out=$(gitea_runs 2>&1)
rc=$?
set -e
if [[ $rc -ne 0 ]]; then
echo "Error: curl cannot read Gitea Actions for $forge_path on $forge_host" >&2
echo "$out" | sed '/token/Id; /Bearer/d; /Authorization/Id' >&2
exit 3
fi
kind=$(printf '%s' "$out" | summarize_gitea_runs | awk '{print $1}')
if [[ "$kind" == auth ]]; then
echo "Error: Gitea API rejected access to $forge_path" >&2
exit 3
fi
}
probe_auth
status_line() {
if [[ "$forge_kind" == github ]]; then
github_runs | summarize_github
elif [[ "$forge_kind" == gitlab ]]; then
gitlab_pipelines | summarize_gitlab
else
gitea_status_line
fi
}
while true; do
now=$(date +%s)
if (( now - started > timeout_sec )); then
echo "FAIL reason=timeout sha=$sha" >&2
exit 1
fi
line=$(status_line)
set -- $line
kind=${1:-empty}
raw=$(query)
mapfile -t fields < <(printf '%s' "$raw" | parse_status)
st=${fields[0]:-error}
pid=${fields[1]:--}
url=${fields[2]:--}
jobs_s=${fields[3]:-}
if [[ "$kind" == empty ]]; then
case "$st" in
empty)
if (( now - empty_since >= appear_sec )); then
echo "FAIL reason=no_pipeline sha=$sha origin=$forge_host/$forge_path" >&2
echo "FAIL reason=no_pipeline sha=$sha" >&2
echo "This SHA has no CI pipeline. Tag or push a ref the project actually builds." >&2
exit 1
fi
sleep "$poll_sec"
continue
fi
if [[ "$kind" == error || "$kind" == auth ]]; then
echo "Error: unexpected forge response for $forge_path" >&2
;;
error)
echo "Error: unexpected CI status response" >&2
echo "$raw" | sed '/token/Id; /Bearer/d; /Authorization/Id' >&2
exit 3
fi
pid=$1
st=$2
url=$3
jobs_s=${4:-}
empty_since=$now
if [[ "$first_seen" -eq 0 ]]; then
echo "Watching pipeline $pid status=$st $url" >&2
first_seen=1
fi
case "$st" in
;;
success)
echo "OK pipeline=$url sha=$sha id=$pid jobs=$jobs_s"
exit 0
;;
failed|failure|canceled|cancelled|skipped)
if [[ "$forge_kind" == gitlab ]]; then
jobs_s=$(gitlab_jobs "$pid" | summarize_jobs || true)
fi
echo "FAIL reason=job_$st sha=$sha pipeline=$url jobs=$jobs_s" >&2
echo "FAIL reason=job_${st} sha=$sha pipeline=$url jobs=$jobs_s" >&2
exit 1
;;
created|pending|running|waiting_for_resource|preparing|waiting|queued|in_progress|requested)
running|pending|queued|waiting|in_progress|created|preparing)
if [[ "$first_seen" -eq 0 ]]; then
echo "Watching pipeline $pid status=$st $url" >&2
first_seen=1
fi
empty_since=$now
sleep "$poll_sec"
;;
*)
empty_since=$now
sleep "$poll_sec"
;;
esac
+99
View File
@@ -0,0 +1,99 @@
#!/usr/bin/env bash
set -euo pipefail
# Query Gitea Actions for an exact SHA via curl. Output STATUS= keys.
# Auth: GITEA_TOKEN in the environment or .env.builder. GITEA_API_URL optional.
sha=
while (($#)); do
case "$1" in
--sha) sha=$2; shift 2 ;;
*) echo "Usage: ci-status.sh --sha SHA" >&2; exit 2 ;;
esac
done
[[ -n "$sha" ]] || { echo "Usage: ci-status.sh --sha SHA" >&2; exit 2; }
if ! command -v curl >/dev/null 2>&1; then
echo "Error: curl is required" >&2
exit 3
fi
root=${BUILDER_PROJECT_ROOT:-$(git rev-parse --show-toplevel)}
origin=$(git -C "$root" remote get-url origin 2>/dev/null || true)
[[ -n "$origin" ]] || { echo "Error: no origin remote" >&2; exit 3; }
read -r host path base <<<"$(WAIT_CI_ORIGIN=$origin python3 -c '
import os, re
url = os.environ["WAIT_CI_ORIGIN"].strip()
url = re.sub(r"^git@([^:]+):", r"ssh://git@\1/", url)
url = re.sub(r"^ssh://git@([^/]+):", r"ssh://git@\1/", url)
if re.match(r"^https?://", url):
m = re.match(r"^(https?)://([^/]+)(?:/(.*))?$", url)
scheme, host, path = (m.group(1), m.group(2), m.group(3) or "") if m else ("", "", "")
else:
m = re.match(r"^ssh://git@([^/]+)/(.+)$", url)
scheme, host, path = ("https", m.group(1), m.group(2)) if m else ("", "", "")
path = re.sub(r"\.git$", "", path)
print(host, path, (scheme or "https") + "://" + host)
')"
api=${GITEA_API_URL:-$base/api/v1}
api=${api%/}
[[ "$api" == */api/v1 ]] || api+=/api/v1
repo=$(WAIT_CI_PATH=$path python3 -c 'import os,urllib.parse; print(urllib.parse.quote(os.environ["WAIT_CI_PATH"], safe="/"))')
gitea_curl() {
if [[ -n "${GITEA_TOKEN:-}" ]]; then
curl -fsS --retry 2 --connect-timeout 10 --max-time 30 \
--config <(printf 'header = "Authorization: token %s"\n' "$GITEA_TOKEN") "$1"
else
curl -fsS --retry 2 --connect-timeout 10 --max-time 30 "$1"
fi
}
set +e
runs=$(gitea_curl "${api}/repos/${repo}/actions/runs?head_sha=${sha}&limit=20&page=1")
rc=$?
set -e
if [[ $rc -ne 0 ]]; then
echo "Error: curl cannot read Gitea Actions for $path" >&2
exit 3
fi
python3 -c '
import json, sys
sha = sys.argv[1]
try:
data = json.loads(sys.stdin.read())
except json.JSONDecodeError:
print("STATUS=error")
raise SystemExit(0)
if isinstance(data, dict) and data.get("message") and not (data.get("workflow_runs") or data.get("runs")):
print("STATUS=error")
raise SystemExit(0)
runs = data.get("workflow_runs", data.get("runs", [])) if isinstance(data, dict) else data
if not isinstance(runs, list) or not runs:
print("STATUS=empty")
raise SystemExit(0)
matched = []
for run in runs:
head = run.get("head_sha") or run.get("headSha") or (run.get("head_commit") or {}).get("id")
if head and head.lower() == sha.lower() and (run.get("id") or run.get("database_id")):
matched.append(run)
if not matched:
print("STATUS=empty")
raise SystemExit(0)
run = matched[0]
run_id = run.get("id") or run.get("database_id")
url = run.get("html_url") or run.get("web_url") or run.get("url") or "-"
status = str(run.get("status") or "").lower()
conclusion = str(run.get("conclusion") or run.get("result") or "").lower()
if conclusion in ("failure", "failed", "cancelled", "canceled", "skipped", "timed_out"):
state = "failed"
elif status in ("completed", "success") and conclusion in ("", "success", "succeeded"):
state = "success"
elif status == "completed" and conclusion:
state = "failed"
else:
state = "running"
print("STATUS=%s ID=%s PIPELINE=%s" % (state, run_id, url))
' "$sha" <<<"$runs"
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env bash
set -euo pipefail
# Query GitHub Actions for an exact SHA via gh. Output STATUS= keys.
sha=
while (($#)); do
case "$1" in
--sha) sha=$2; shift 2 ;;
*) echo "Usage: ci-status.sh --sha SHA" >&2; exit 2 ;;
esac
done
[[ -n "$sha" ]] || { echo "Usage: ci-status.sh --sha SHA" >&2; exit 2; }
if ! command -v gh >/dev/null 2>&1; then
echo "Error: gh is required" >&2
exit 3
fi
root=${BUILDER_PROJECT_ROOT:-$(git rev-parse --show-toplevel)}
origin=$(git -C "$root" remote get-url origin 2>/dev/null || true)
[[ -n "$origin" ]] || { echo "Error: no origin remote" >&2; exit 3; }
repo=$(WAIT_CI_ORIGIN=$origin python3 -c '
import os, re
url = os.environ["WAIT_CI_ORIGIN"].strip()
url = re.sub(r"^git@([^:]+):", r"ssh://git@\1/", url)
url = re.sub(r"^ssh://git@([^/]+):", r"ssh://git@\1/", url)
if re.match(r"^https?://", url):
m = re.match(r"^https?://[^/]+(?:/(.*))?$", url)
path = m.group(1) if m else ""
else:
m = re.match(r"^ssh://git@[^/]+/(.+)$", url)
path = m.group(1) if m else ""
print(re.sub(r"\.git$", "", path or ""))
')
set +e
raw=$(gh run list --repo "$repo" --commit "$sha" --limit 20 --json databaseId,status,conclusion,url,headSha)
rc=$?
set -e
if [[ $rc -ne 0 ]]; then
echo "Error: gh cannot list runs for $repo" >&2
exit 3
fi
python3 -c '
import json, sys
try:
data = json.loads(sys.stdin.read())
except json.JSONDecodeError:
print("STATUS=error")
raise SystemExit(0)
if not data:
print("STATUS=empty")
raise SystemExit(0)
r = data[0]
status = str(r.get("status") or "").lower()
conc = str(r.get("conclusion") or "").lower()
if status == "completed" and conc == "success":
state = "success"
elif status == "completed":
state = "failed"
else:
state = "running"
print("STATUS=%s ID=%s PIPELINE=%s" % (state, r.get("databaseId") or "-", r.get("url") or "-"))
' <<<"$raw"
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env bash
set -euo pipefail
# Query GitLab CI for an exact SHA via glab. Output STATUS= keys.
sha=
while (($#)); do
case "$1" in
--sha) sha=$2; shift 2 ;;
*) echo "Usage: ci-status.sh --sha SHA" >&2; exit 2 ;;
esac
done
[[ -n "$sha" ]] || { echo "Usage: ci-status.sh --sha SHA" >&2; exit 2; }
if ! command -v glab >/dev/null 2>&1; then
echo "Error: glab is required" >&2
exit 3
fi
root=${BUILDER_PROJECT_ROOT:-$(git rev-parse --show-toplevel)}
origin=$(git -C "$root" remote get-url origin 2>/dev/null || true)
[[ -n "$origin" ]] || { echo "Error: no origin remote" >&2; exit 3; }
path=$(WAIT_CI_ORIGIN=$origin python3 -c '
import os, re
url = os.environ["WAIT_CI_ORIGIN"].strip()
url = re.sub(r"^git@([^:]+):", r"ssh://git@\1/", url)
url = re.sub(r"^ssh://git@([^/]+):", r"ssh://git@\1/", url)
if re.match(r"^https?://", url):
m = re.match(r"^https?://[^/]+(?:/(.*))?$", url)
path = m.group(1) if m else ""
else:
m = re.match(r"^ssh://git@[^/]+/(.+)$", url)
path = m.group(1) if m else ""
print(re.sub(r"\.git$", "", path or ""))
')
encoded=$(WAIT_CI_PATH=$path python3 -c 'import os,urllib.parse; print(urllib.parse.quote(os.environ["WAIT_CI_PATH"], safe=""))')
set +e
raw=$(glab api "projects/${encoded}/pipelines?sha=${sha}&per_page=20")
rc=$?
set -e
if [[ $rc -ne 0 ]]; then
echo "Error: glab cannot list pipelines for $path" >&2
exit 3
fi
python3 -c '
import json, sys
try:
data = json.loads(sys.stdin.read())
except json.JSONDecodeError:
print("STATUS=error")
raise SystemExit(0)
if isinstance(data, dict) and data.get("message"):
print("STATUS=error")
raise SystemExit(0)
if not isinstance(data, list) or not data:
print("STATUS=empty")
raise SystemExit(0)
p = data[0]
st = str(p.get("status") or "").lower()
if st in ("success", "passed"):
state = "success"
elif st in ("failed", "canceled", "cancelled", "skipped"):
state = "failed"
elif st in ("",):
state = "empty"
else:
state = "running"
print("STATUS=%s ID=%s PIPELINE=%s" % (state, p.get("id") or "-", p.get("web_url") or "-"))
' <<<"$raw"
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env bash
set -euo pipefail
# Query Gitea Actions for an exact SHA via tea. Output STATUS= keys.
sha=
while (($#)); do
case "$1" in
--sha) sha=$2; shift 2 ;;
*) echo "Usage: ci-status.sh --sha SHA" >&2; exit 2 ;;
esac
done
[[ -n "$sha" ]] || { echo "Usage: ci-status.sh --sha SHA" >&2; exit 2; }
if ! command -v tea >/dev/null 2>&1; then
echo "Error: tea is required" >&2
exit 3
fi
root=${BUILDER_PROJECT_ROOT:-$(git rev-parse --show-toplevel)}
origin=$(git -C "$root" remote get-url origin 2>/dev/null || true)
[[ -n "$origin" ]] || { echo "Error: no origin remote" >&2; exit 3; }
path=$(WAIT_CI_ORIGIN=$origin python3 -c '
import os, re
url = os.environ["WAIT_CI_ORIGIN"].strip()
url = re.sub(r"^git@([^:]+):", r"ssh://git@\1/", url)
url = re.sub(r"^ssh://git@([^/]+):", r"ssh://git@\1/", url)
if re.match(r"^https?://", url):
m = re.match(r"^https?://[^/]+(?:/(.*))?$", url)
path = m.group(1) if m else ""
else:
m = re.match(r"^ssh://git@[^/]+/(.+)$", url)
path = m.group(1) if m else ""
print(re.sub(r"\.git$", "", path or ""))
')
[[ -n "$path" ]] || { echo "Error: cannot parse origin" >&2; exit 3; }
set +e
runs=$(tea api "repos/${path}/actions/runs?head_sha=${sha}&limit=20")
rc=$?
set -e
if [[ $rc -ne 0 ]]; then
echo "Error: tea cannot read Actions for $path" >&2
exit 3
fi
python3 -c '
import json, sys
sha = sys.argv[1]
try:
data = json.loads(sys.stdin.read())
except json.JSONDecodeError:
print("STATUS=error")
raise SystemExit(0)
if isinstance(data, dict) and data.get("message") and not (data.get("workflow_runs") or data.get("runs")):
print("STATUS=error")
raise SystemExit(0)
runs = data.get("workflow_runs", data.get("runs", [])) if isinstance(data, dict) else data
if not isinstance(runs, list) or not runs:
print("STATUS=empty")
raise SystemExit(0)
matched = [run for run in runs if str(run.get("head_sha") or run.get("headSha") or "").lower() == sha.lower()]
if not matched:
print("STATUS=empty")
raise SystemExit(0)
run = matched[0]
run_id = run.get("id") or run.get("database_id") or "-"
url = run.get("html_url") or run.get("web_url") or run.get("url") or "-"
status = str(run.get("status") or "").lower()
conclusion = str(run.get("conclusion") or run.get("result") or "").lower()
if conclusion in ("failure", "failed", "cancelled", "canceled", "skipped", "timed_out"):
state = "failed"
elif status in ("completed", "success") and conclusion in ("", "success", "succeeded"):
state = "success"
elif status == "completed" and conclusion:
state = "failed"
else:
state = "running"
print("STATUS=%s ID=%s PIPELINE=%s" % (state, run_id, url))
' "$sha" <<<"$runs"
+1 -2
View File
@@ -11,8 +11,7 @@
# DOCKER_REPOSITORY=
# IMAGE_TAG=
# PLATFORMS= # optional override; project ARCHES lives in makefile.builder
# BUILDER_CI_FORGE= # optional: github, gitlab, or gitea; useful for custom hosts
# GITEA_API_URL= # optional: defaults to https://<origin-host>/api/v1
# GITEA_API_URL= # optional for the curl CI status script
# GITEA_TOKEN= # optional token for private Gitea repositories
# COPY_DEST= # fallback: local dir, or docker dest container:/absolute/dir
# COPY_SCRIPT= # external/temporary executable override; project-local paths belong in makefile.builder
+3 -2
View File
@@ -8,9 +8,10 @@
ARCHES ?= amd64
ARCH ?= $(firstword $(ARCHES))
# local: make docker/deb on this machine. ci: obtain.sh waits for the
# forge pipeline of an exact SHA, then inspects the artifact. Not a track.
# local: make docker/deb on this machine. ci: obtain.sh polls
# CI_STATUS_SCRIPT for an exact SHA, then inspects the artifact. Not a track.
BUILD_VENUE ?= local
CI_STATUS_SCRIPT ?= .pouch/builder/ci-status.sh
ifeq ($(strip $(ARCHES)),)
$(error ARCHES must not be empty)
endif
+3
View File
@@ -34,6 +34,9 @@ description: >-
新的 ship。
- tester / builder / deployer 的 SSOT 仍在各自目录。ship 只追加指针。本轮跑过的
测试、obtain、部署若没写入 `current.yaml` 的 `evidence`,该阶段不算完成。
- 新建 ship 的任务带 `testPlan`,开发 worker 把新增/修改测试写入
`2-develop/test-manifest.yaml`;verify 按 manifest 的 selector 执行,regress 再叠加
所有 tester catalog 根和项目原生完整测试。没有测试时必须有明确批准的 `testGap`。
- 项目构建/部署契约仍在 `.pouch/ack/delivery.yaml` 和 builder/deployer 配置里,不要
复制进每个 ship。
- 旧的 `.pouch/specifier/features/<id>/` 在继续该需求时迁到
+10
View File
@@ -16,6 +16,7 @@
current.yaml # 总账:kind、状态、stopAt、指针
spec.md # 短需求说明(short 和 long 都有)
tasks.yaml # 本 ship 的任务 DAG
2-develop/test-manifest.yaml # 开发者新增/修改测试的机器清单
status.md # 给人看的一页摘要
1-spec/ # 规格、评审、根因和 handoff
2-develop/ # 开发任务和变更
@@ -131,6 +132,12 @@ evidence:
`tasks.yaml` 只描述本 ship 的工作项,不是 ACK 缺陷板。
新建 ship 的任务默认带 `testPlan`。开发 worker 在 manifest 中记录测试的稳定 id、
runner、target、selector、覆盖的验收项和 `added|changed` 状态。`verify` 必须逐条执行
manifest;`regress` 在完整 tester catalog 和项目原生测试之外再次确认这些条目通过。
仅文档或明确无行为变化的任务可使用 `policy: not_applicable`,但必须填写被主会话接受的
`testGap.reason` 与 `acceptedBy`。
```yaml
shipId: add-export-button
tasks:
@@ -141,6 +148,9 @@ tasks:
- 登录用户在列表页看到「导出」
verification:
- 浏览器打开列表页,按钮可见且可点
testPlan:
policy: required
manifest: 2-develop/test-manifest.yaml
status: open
stopAt: review_ready
```
+11 -2
View File
@@ -21,9 +21,18 @@ scripts/append_evidence.py 追加后跑 scripts/validate_ship.py。
验收:见 spec.md 与 tasks.yaml 里该任务的 expected / verification。
授权:<本步允许的 git/PR/部署动作;未写的不要做>
测试交接:`implement` / `implement_task` 必须在 ship 的 `2-develop/test-manifest.yaml`
登记本次新增或修改的测试。每条记录至少包含稳定 `id`、`kind`、`runner`、`target`、
`selector`、`covers` 和 `change: added|changed`。若确实没有行为变化或无法补测,必须
填写 `testGap.reason`,并由主会话明确写入 `acceptedBy`;不能留空后声称已验证。
测试边界:`verify` / `verify_task` 只负责开发阶段验证,执行任务相关的单元测试、
新增功能集成测试和静态检查,不执行合并后的完整 tester 回归;`regress` 只在合并后
负责完整回归,必须覆盖发现的每个 tester catalog 根和项目声明的原生测试套件。
新增功能集成测试和静态检查,不执行合并后的完整 tester 回归。验证前先使用 ship skill
的 `scripts/validate_test_manifest.py <ship-dir>/2-develop/test-manifest.yaml`
带上当前 ship id、任务的 testPlan.policy 执行 `--mode verify`;逐条执行 manifest 中的
测试 selector,并在证据中记录测试 id。
`regress` 只在合并后负责完整回归,除 manifest 中的新增/修改测试外,必须覆盖发现的每个
tester catalog 根和项目声明的原生测试套件。
若步骤是 `verify_task` 或 `regress` 且 skill 是 `tester`:先执行
`python3 <tester-skill>/scripts/catalog.py roots --root <project-root>`。`verify_task` 按
+39
View File
@@ -19,6 +19,7 @@ from ship_lib import (
check_ship_id,
load_yaml,
)
from validate_test_manifest import validate_manifest
def _issue(issues: list[str], msg: str) -> None:
@@ -144,6 +145,7 @@ def validate_ship_dir(ship_dir: Path) -> list[str]:
_issue(issues, f"缺少 {tasks_rel}")
else:
issues.extend(_validate_tasks(tasks_path, ship_id, kind))
issues.extend(_validate_manifest_pointers(tasks_path, ship_dir, ship_id))
delivery = data.get("delivery")
if delivery is not None:
@@ -247,6 +249,43 @@ def _validate_tasks(path: Path, ship_id: Any, kind: Any) -> list[str]:
return issues
def _validate_manifest_pointers(path: Path, ship_dir: Path, ship_id: str) -> list[str]:
issues: list[str] = []
try:
data = load_yaml(path)
except Exception:
return issues
tasks = data.get("tasks") if isinstance(data, dict) else None
if not isinstance(tasks, list):
return issues
for index, item in enumerate(tasks):
if not isinstance(item, dict):
continue
plan = item.get("testPlan")
if not isinstance(plan, dict):
continue
manifest = plan.get("manifest")
if not isinstance(manifest, str) or not manifest.strip():
continue
target = (ship_dir / manifest).resolve()
try:
target.relative_to(ship_dir.resolve())
except ValueError:
_issue(issues, f"tasks[{index}].testPlan.manifest 必须位于 ship 目录内: {manifest}")
continue
if not target.is_file():
_issue(issues, f"tasks[{index}].testPlan.manifest 不存在: {manifest}")
continue
manifest_issues, _ = validate_manifest(
target,
expected_ship_id=ship_id,
policy=str(plan.get("policy") or "required"),
mode="schema",
)
issues.extend(f"tasks[{index}].{issue}" for issue in manifest_issues)
return issues
def _validate_evidence(evidence: dict[str, Any], project_root: Path) -> list[str]:
issues: list[str] = []
for key in ("testRuns", "artifacts", "deploys", "prs", "commits"):
+14
View File
@@ -0,0 +1,14 @@
# test-preflight
Reusable preflight for tester matrices. It supports read-only `verify` and
test-environment-only `prepare`, and writes per-check YAML/CSV evidence for
artifacts, tester nodes, devices, network, services, and fixture/cleanup contracts.
This project assumes mobile album/photo/IME capabilities; journey cases exercise them
and preflight does not gate them or read authorization environment variables.
Run the self-tests with:
```bash
python3 -m unittest discover -s skills/test-preflight/tests
python3 /home/ace/.codex/skills/.system/skill-creator/scripts/quick_validate.py skills/test-preflight
```
+58
View File
@@ -0,0 +1,58 @@
---
name: test-preflight
description: >-
Verify or prepare a mobile/integration test environment from a declared matrix:
artifacts, tester bindings, HOST/PUBLIC_HOST, service probes, and fixture/cleanup
contracts. Use before tester runs; it is not a deployment,
build, or business-data skill.
---
# test-preflight
Use this skill before `integration` or `journey` tests. It answers whether the
declared environment can start a run and emits one YAML and one CSV evidence
file. It never logs in, creates business data, builds/downloads an artifact, or
deploys production.
## Modes
- `verify` is read-only: validate the matrix, artifact identity, binding, actual
tester-node platform/device state, DNS/TCP/HTTP probes, backend probes, and
fixture/cleanup declarations. It must not install, launch, or deploy.
- `prepare` is side-effecting only within the test environment: call the
caller-supplied deployer command contract, run the same checks, then use the
tester-node command contract to install and launch the already supplied APK or
IPA. It never calls a builder and refuses `environment: prod`.
The command is:
```bash
python3 <skill-dir>/scripts/preflight.py verify \
--config <matrix.yaml> --yaml <preflight.yaml> --csv <preflight.csv>
python3 <skill-dir>/scripts/preflight.py prepare \
--config <matrix.yaml> --yaml <preflight.yaml> --csv <preflight.csv>
```
Read [references/schema.md](references/schema.md) for the input contract and
[references/boundaries.md](references/boundaries.md) before wiring a real
agent-runtime/deployer adapter. The included command-contract fixture is the
safe way to test remote-node behavior without pretending that this machine is a
Mac or that a deployment happened.
Overall status is `ready`, `blocked`, or `environment-fail`:
- `blocked` means a required declaration, artifact, binding, runner, fixture,
cleanup contract, or authorization is absent/unsafe.
- `environment-fail` means a declared environment exists but DNS/TCP/HTTP,
device/toolchain, installation, launch, or deployment execution failed.
- `ready` means every recorded check passed. A ready preflight permits the
caller to continue with `integration`; this skill does not run test cases.
This project assumes the client has the required album picker, photo permission, and Chinese
IME capabilities. Preflight does not probe or gate these capabilities, and does not read
`AUTHORIZE_ALBUM` or `IME_PINYIN`. Their real behavior is exercised by journey cases; a failure
there is a product test failure, not an environment block.
Do not treat a preflight report as a product result. A preflight failure means
no client session may be created; the tester owns `not-run` propagation and must write
one result row per selected testcase with `blocked_by: [test-preflight]`.
+3
View File
@@ -0,0 +1,3 @@
interface:
display_name: "Test Preflight"
short_description: "Verify or prepare tester environments"
@@ -0,0 +1,4 @@
# Reference moved
Read [schema.md](schema.md) for the matrix and evidence contract, and
[boundaries.md](boundaries.md) for agent-runtime/deployer/builder integration.
@@ -0,0 +1,44 @@
# Integration boundaries
`test-preflight` is an orchestration and evidence layer. It consumes existing
skills instead of implementing their policy or storage formats.
| Boundary | Caller-supplied contract | What preflight does | What it must not do |
| --- | --- | --- | --- |
| agent-runtime | resolved `tester.<name>` binding: binding id, node, environment, project/path, and a node runner | validate that the binding is explicit and route node probes through the runner | parse Orca state, invent a host, or create/reuse a terminal |
| deployer | `prepare` only: an argv command returning a JSON deployment receipt for `environment: test` | call it once and record its receipt/status | copy Compose/Argo logic, SSH directly, or accept production |
| builder/obtain | caller-provided local artifact path (and optional remote path), hash, source commit, package/bundle identity | hash and inspect the exact file; pass it to install contract | build, guess an old artifact, or silently download/upload |
## Deployer command contract
`prepare` requires:
```yaml
deployment:
environment: test
command: [python3, /path/to/existing-deployer-adapter.py]
```
The command receives JSON on stdin:
```json
{"operation":"prepare","environment":"test","project_root":"/abs/project","host":"...","public_host":"..."}
```
It must return `{ "status": "ready", "observed": "..." }` (an optional
`checks` mapping is recorded). Non-zero/invalid output is an
`environment-fail`; absent command or a non-test environment is `blocked`.
The adapter owns deployer-specific configuration, receipts, cleanup, and
authorization.
## Device contract
The node runner owns remote execution. For a local node the implementation uses
read-only `adb`/`xcrun simctl` checks in `verify`; `prepare` alone invokes
`adb install`/launch or `simctl install`/launch. For a remote node the same
operations are JSON requests to the runner. Missing xcrun on Linux, Android
`unauthorized`/`offline`, a missing device, or a failed install/start is an
`environment-fail`, not a product failure.
No client session is created by this skill. The runner's `install` and `start`
operations are the only device mutations, and are never called by `verify`.
@@ -0,0 +1,95 @@
# test-preflight matrix schema
The input is YAML. `mode` is supplied by the CLI, so a file may be reused for
`verify` and `prepare`.
```yaml
schema_version: 1
project_root: /absolute/project/root
host: https://test.example.invalid
public_host: https://public-test.example.invalid # optional
service_probes:
- id: api-health
url: "{HOST}/health"
expected_status: [200]
fixtures:
required: true
references: [isolated-test-account]
cleanup:
required: true
steps: [delete-run-data]
# Optional project metadata. It is not a preflight gate.
capabilities:
assumed: [album_picker, photo_permission, ime_pinyin]
matrix:
- platform: android
tester:
binding: tester.android
node: local
runner: {kind: local}
artifact:
path: /absolute/dist/writes.apk
sha256: 012345...
package: page.writes
source_commit: 40-character-commit-sha
identity_command: [apkanalyzer, manifest, application-id, "{artifact}"]
```
Required matrix fields are `platform` (`android` or `ios`), `tester.binding`,
`tester.node`, and `artifact.path`. Artifact identity is required as
`package` for Android or `bundle_id` for iOS. `sha256` and `source_commit` are
optional but recommended; when present they are checked exactly.
`service_probes` must contain at least one URL. `{HOST}` and `{PUBLIC_HOST}`
are expanded from the top-level values. Each probe is checked from the actual
tester node as DNS, TCP, and HTTP. A status outside 2xx–4xx fails transport
readiness; `expected_status` can require an exact allow-list. HTTP 401/403 can
therefore be a valid reachability result when explicitly expected.
`fixtures` and `cleanup` are declarations, not commands. A required fixture
needs non-empty `references`; a required cleanup needs non-empty `steps` or
`run`. For a read-only run, use `required: false` with a non-empty `reason`.
The skill never executes these values.
## Client capabilities
The project assumes the mobile test nodes provide the album picker, photo permission,
test image, and Chinese IME. These are not preflight declarations or gates. Journey
cases exercise the real UI; failures are recorded as product failures. `AUTHORIZE_ALBUM`
and `IME_PINYIN` are not read.
## Tester-node contract
`tester.runner.kind: local` executes platform and network probes on the current
node. It is only valid for `node: local`/`localhost`. A non-local node must use
`kind: contract` and a non-empty argv `command`. The command receives one JSON
request on stdin and returns one JSON object on stdout; no shell string is
accepted.
Requests have `operation` (`platform_probe`, `network_probe`, `install`, or
`start`), `platform`, `node`, and operation-specific fields. The response is:
```json
{
"status": "ready",
"observed": "short evidence",
"checks": {
"macos": {"status": "ready", "observed": "Darwin 24", "expected": "Darwin"}
}
}
```
`checks` is optional; each key becomes a separate evidence row. A non-zero,
invalid, or unavailable runner is `environment-fail`. The adapter is expected
to execute on the resolved tester node, not on the coordinator. See
[boundaries.md](boundaries.md).
## Evidence
YAML contains `mode`, overall `status`, timestamps, a summary, and `checks`.
CSV uses these stable columns:
`check_id,mode,platform,node,target,observed,expected,status,remediation,timestamp`
Every check must use one of `ready`, `blocked`, or `environment-fail`; the
report does not invent `fail` or `not-run` testcase results.
+553
View File
@@ -0,0 +1,553 @@
#!/usr/bin/env python3
"""Run the test-preflight verify/prepare contract."""
from __future__ import annotations
import argparse
import csv
import hashlib
import json
import os
import platform as host_platform
import plistlib
import re
import shutil
import socket
import subprocess
import sys
import urllib.error
import urllib.parse
import urllib.request
import zipfile
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Mapping
try:
import yaml
except ImportError as exc: # pragma: no cover
raise SystemExit("test-preflight 需要 PyYAML;请安装运行依赖后重试") from exc
PLATFORMS = {"android", "ios"}
STATUSES = {"ready", "blocked", "environment-fail"}
EVIDENCE_FIELDS = [
"check_id", "mode", "platform", "node", "target", "observed", "expected",
"status", "remediation", "timestamp",
]
SHA256 = re.compile(r"^[0-9a-fA-F]{64}$")
COMMIT = re.compile(r"^[0-9a-fA-F]{40}$")
def now() -> str:
return datetime.now(timezone.utc).isoformat()
def text(value: Any) -> str:
if isinstance(value, (dict, list)):
return json.dumps(value, ensure_ascii=False, sort_keys=True)
return "" if value is None else str(value)
def status_from(value: Any, default: str = "environment-fail") -> str:
return value if value in STATUSES else default
def sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for block in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(block)
return digest.hexdigest()
def command_text(command: list[str], *, payload: Mapping[str, Any] | None = None) -> tuple[int, str]:
try:
process = subprocess.run(
command,
input=(json.dumps(payload, ensure_ascii=False) if payload is not None else None),
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
check=False,
)
except OSError as exc:
return 127, str(exc)
return process.returncode, process.stdout.strip()
class Evidence:
def __init__(self, mode: str) -> None:
self.mode = mode
self.started_at = now()
self.rows: list[dict[str, str]] = []
def add(
self,
check_id: str,
*,
platform: str = "all",
node: str = "coordinator",
target: str = "",
observed: Any = "",
expected: Any = "",
status: str,
remediation: str = "",
) -> None:
if status not in STATUSES:
raise ValueError(f"invalid evidence status: {status}")
self.rows.append({
"check_id": check_id,
"mode": self.mode,
"platform": platform,
"node": node,
"target": target,
"observed": text(observed),
"expected": text(expected),
"status": status,
"remediation": remediation,
"timestamp": now(),
})
def overall(self) -> str:
statuses = {row["status"] for row in self.rows}
if "environment-fail" in statuses:
return "environment-fail"
if "blocked" in statuses:
return "blocked"
return "ready"
def payload(self) -> dict[str, Any]:
return {
"schema_version": 1,
"mode": self.mode,
"status": self.overall(),
"started_at": self.started_at,
"finished_at": now(),
"summary": {state: sum(row["status"] == state for row in self.rows) for state in sorted(STATUSES)},
"checks": self.rows,
}
def load_config(path: Path) -> dict[str, Any]:
data = yaml.safe_load(path.read_text(encoding="utf-8"))
if not isinstance(data, dict):
raise ValueError("配置根必须是 mapping")
return data
def expand(value: str, config: Mapping[str, Any]) -> str:
return value.replace("{HOST}", text(config.get("host"))).replace("{PUBLIC_HOST}", text(config.get("public_host")))
def normalize_url(value: str) -> str:
return value if urllib.parse.urlsplit(value).scheme else "http://" + value.lstrip("/")
def argv(value: Any) -> list[str] | None:
if not isinstance(value, list) or not value or not all(isinstance(item, str) and item for item in value):
return None
return value
def validate_shape(config: Mapping[str, Any], evidence: Evidence) -> bool:
ok = True
if config.get("schema_version") != 1:
evidence.add("config-schema", target="config", observed=config.get("schema_version"), expected="1", status="blocked", remediation="将 schema_version 设为 1")
ok = False
if not isinstance(config.get("project_root"), str) or not Path(text(config.get("project_root"))).is_absolute():
evidence.add("config-project-root", target="project_root", observed=config.get("project_root"), expected="absolute project_root", status="blocked", remediation="提供项目根绝对路径")
ok = False
if not isinstance(config.get("host"), str) or not config["host"].strip():
evidence.add("config-host", target="HOST", expected="non-empty host URL", status="blocked", remediation="提供 HOST")
ok = False
probes = config.get("service_probes")
if not isinstance(probes, list) or not probes:
evidence.add("config-service-probes", target="service_probes", expected="at least one probe", status="blocked", remediation="声明服务 URL 探针")
ok = False
else:
for index, probe in enumerate(probes):
if not isinstance(probe, dict) or not isinstance(probe.get("url"), str) or not probe["url"].strip():
evidence.add(f"config-service-probe-{index}", target=f"service_probes[{index}]", expected="url", status="blocked", remediation="为每个服务探针提供 URL")
ok = False
matrix = config.get("matrix")
if not isinstance(matrix, list) or not matrix:
evidence.add("config-matrix", target="matrix", expected="non-empty list", status="blocked", remediation="声明至少一个平台矩阵行")
return False
seen: set[str] = set()
for index, row in enumerate(matrix):
prefix = f"matrix-{index}"
if not isinstance(row, dict):
evidence.add(prefix, target=f"matrix[{index}]", expected="mapping", status="blocked", remediation="修正矩阵行")
ok = False
continue
platform_name = text(row.get("platform")).lower()
if platform_name not in PLATFORMS or platform_name in seen:
evidence.add(f"{prefix}-platform", platform=platform_name or "unknown", target=f"matrix[{index}].platform", expected="unique android or ios", status="blocked", remediation="为每个平台保留唯一矩阵行")
ok = False
seen.add(platform_name)
tester = row.get("tester")
if not isinstance(tester, dict) or not isinstance(tester.get("binding"), str) or not tester["binding"].startswith("tester."):
evidence.add(f"{prefix}-binding", platform=platform_name, target=f"matrix[{index}].tester.binding", expected="tester.<name>", status="blocked", remediation="由 agent-runtime 解析并传入 tester binding")
ok = False
if not isinstance(tester, dict) or not isinstance(tester.get("node"), str) or not tester["node"].strip():
evidence.add(f"{prefix}-node", platform=platform_name, target=f"matrix[{index}].tester.node", expected="actual tester node", status="blocked", remediation="提供实际 tester 节点名")
ok = False
artifact = row.get("artifact")
identity_key = "package" if platform_name == "android" else "bundle_id"
if not isinstance(artifact, dict) or not isinstance(artifact.get("path"), str) or not artifact["path"].strip():
evidence.add(f"{prefix}-artifact", platform=platform_name, target=f"matrix[{index}].artifact.path", expected="caller-provided artifact path", status="blocked", remediation="提供本轮 APK/IPA,不要让 preflight 猜产物")
ok = False
if not isinstance(artifact, dict) or not isinstance(artifact.get(identity_key), str) or not artifact[identity_key].strip():
evidence.add(f"{prefix}-identity", platform=platform_name, target=f"matrix[{index}].artifact.{identity_key}", expected="declared artifact identity", status="blocked", remediation=f"声明 artifact.{identity_key}")
ok = False
if isinstance(artifact, dict) and artifact.get("sha256") is not None and not SHA256.fullmatch(text(artifact.get("sha256"))):
evidence.add(f"{prefix}-sha256", platform=platform_name, target=f"matrix[{index}].artifact.sha256", expected="64 hex characters", status="blocked", remediation="提供正确 SHA-256 或删除字段")
ok = False
if isinstance(artifact, dict) and artifact.get("source_commit") is not None and not COMMIT.fullmatch(text(artifact.get("source_commit"))):
evidence.add(f"{prefix}-commit", platform=platform_name, target=f"matrix[{index}].artifact.source_commit", expected="40-character commit SHA", status="blocked", remediation="提供产物来源 commit SHA")
ok = False
return ok
def artifact_identity(path: Path, platform_name: str, artifact: Mapping[str, Any]) -> tuple[str | None, str]:
if platform_name == "ios":
try:
with zipfile.ZipFile(path) as archive:
names = [name for name in archive.namelist() if re.fullmatch(r"Payload/[^/]+\.app/Info\.plist", name)]
if len(names) != 1:
return None, "IPA 中没有唯一 Payload/*.app/Info.plist"
plist = plistlib.loads(archive.read(names[0]))
value = plist.get("CFBundleIdentifier")
return (value, "Info.plist CFBundleIdentifier") if isinstance(value, str) else (None, "Info.plist 缺少 CFBundleIdentifier")
except (OSError, zipfile.BadZipFile, plistlib.InvalidFileException) as exc:
return None, f"IPA 读取失败: {exc}"
command = argv(artifact.get("identity_command"))
if command:
command = [item.replace("{artifact}", str(path)) for item in command]
else:
tool = shutil.which("apkanalyzer") or shutil.which("aapt2") or shutil.which("aapt")
if not tool:
return None, "缺少 apkanalyzer/aapt2/aapt;无法读取 APK package identity"
command = [tool, "manifest", "application-id", str(path)] if tool.endswith("apkanalyzer") else [tool, "dump", "badging", str(path)]
code, output = command_text(command)
if code != 0:
return None, output[-500:] or f"identity command exited {code}"
match = re.search(r"(?:application-id|name)\s*[:=]\s*['\"]?([A-Za-z0-9_.]+)", output)
if not match:
match = re.search(r"package:\s+name='([^']+)'", output)
if not match:
match = re.fullmatch(r"\s*([A-Za-z0-9_.]+)\s*", output)
return (match.group(1), output[-500:]) if match else (None, "identity command output did not contain package")
def record_artifact(evidence: Evidence, row: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
artifact = row.get("artifact") or {}
path = Path(text(artifact.get("path"))).expanduser()
prefix = f"{platform_name}-{index}-artifact"
if not path.is_file():
evidence.add(prefix, platform=platform_name, node=node, target=str(path), expected="existing caller-provided artifact", status="blocked", remediation="补齐 APK/IPA 路径;不要默认构建")
return False
actual_hash = sha256(path)
expected_hash = artifact.get("sha256")
if expected_hash and actual_hash.lower() != text(expected_hash).lower():
evidence.add(prefix + "-hash", platform=platform_name, node=node, target=str(path), observed=actual_hash, expected=expected_hash, status="blocked", remediation="确认产物路径或 SHA-256 与本轮构建身份一致")
return False
evidence.add(prefix + "-hash", platform=platform_name, node=node, target=str(path), observed=actual_hash, expected=expected_hash or "file SHA-256 recorded", status="ready")
identity, observed = artifact_identity(path, platform_name, artifact)
expected_identity = text(artifact.get("package" if platform_name == "android" else "bundle_id"))
if identity != expected_identity:
evidence.add(prefix + "-identity", platform=platform_name, node=node, target=str(path), observed=identity or observed, expected=expected_identity, status="environment-fail", remediation="使用与 catalog 身份一致的 APK/IPA,并重新校验产物")
return False
evidence.add(prefix + "-identity", platform=platform_name, node=node, target=str(path), observed=identity, expected=expected_identity, status="ready")
if artifact.get("source_commit"):
evidence.add(prefix + "-commit", platform=platform_name, node=node, target=str(path), observed=artifact.get("source_commit"), expected=artifact.get("source_commit"), status="ready")
return True
def local_platform_probe(platform_name: str) -> tuple[str, dict[str, dict[str, Any]]]:
if platform_name == "android":
adb = shutil.which("adb")
if not adb:
return "environment-fail", {"adb": {"status": "environment-fail", "observed": "missing", "expected": "Android SDK platform-tools", "remediation": "在 tester.android 节点安装并配置 adb"}}
code, output = command_text([adb, "devices", "-l"])
states = [line.split() for line in output.splitlines()[1:] if line.strip()]
device = next((parts[0] for parts in states if len(parts) > 1 and parts[1] == "device"), "")
bad = next((parts for parts in states if len(parts) > 1 and parts[1] in {"unauthorized", "offline"}), None)
if bad:
return "environment-fail", {"adb": {"status": "ready", "observed": output, "expected": "adb responds"}, "device": {"status": "environment-fail", "observed": " ".join(bad), "expected": "authorized device", "remediation": "解锁 Android 设备并接受 RSA 授权;offline 时重启 adb/设备"}}
if code != 0 or not device:
return "environment-fail", {"adb": {"status": "environment-fail", "observed": output, "expected": "one authorized device", "remediation": "连接并授权 Android device/emulator"}}
boot_code, boot = command_text([adb, "-s", device, "shell", "getprop", "sys.boot_completed"])
booted = boot_code == 0 and boot.strip() == "1"
return ("ready" if booted else "environment-fail", {
"adb": {"status": "ready", "observed": adb, "expected": "adb available"},
"device": {"status": "ready", "observed": device, "expected": "authorized device"},
"boot": {"status": "ready" if booted else "environment-fail", "observed": boot, "expected": "sys.boot_completed=1", "remediation": "等待 emulator boot complete 后重跑"},
})
if host_platform.system() != "Darwin":
return "environment-fail", {"macos": {"status": "environment-fail", "observed": host_platform.system(), "expected": "Darwin/macOS tester.ios node", "remediation": "将 tester.ios 指向可用 macOS,不要在 Linux 伪装 iOS"}}
xcrun = shutil.which("xcrun")
if not xcrun:
return "environment-fail", {"xcrun": {"status": "environment-fail", "observed": "missing", "expected": "xcrun and simctl", "remediation": "在 Mac 安装 Xcode Command Line Tools/Xcode"}}
code, output = command_text([xcrun, "simctl", "list", "devices", "available"])
booted = any("(Booted)" in line for line in output.splitlines())
return ("ready" if code == 0 and booted else "environment-fail", {
"macos": {"status": "ready", "observed": "Darwin", "expected": "Darwin/macOS"},
"xcrun": {"status": "ready" if code == 0 else "environment-fail", "observed": output[-500:], "expected": "simctl available", "remediation": "安装 Xcode 并接受许可"},
"boot": {"status": "ready" if booted else "environment-fail", "observed": "Booted" if booted else output[-300:], "expected": "Booted simulator", "remediation": "启动目标 Simulator 后重跑"},
})
def runner_call(runner: Mapping[str, Any], request: Mapping[str, Any]) -> tuple[str, str, dict[str, Any]]:
command = argv(runner.get("command"))
if not command:
return "blocked", "runner command missing", {}
code, output = command_text(command, payload=request)
if code != 0:
return "environment-fail", output[-800:], {}
try:
result = json.loads(output)
except json.JSONDecodeError:
return "environment-fail", output[-800:], {}
if not isinstance(result, dict):
return "environment-fail", "runner returned non-object", {}
return status_from(result.get("status")), text(result.get("observed", "")), result
def add_runner_checks(evidence: Evidence, result: Mapping[str, Any], *, prefix: str, platform_name: str, node: str, default_expected: str) -> str:
checks = result.get("checks")
if isinstance(checks, dict) and checks:
statuses: list[str] = []
for name, detail in checks.items():
detail = detail if isinstance(detail, dict) else {"observed": detail}
status = status_from(detail.get("status"), status_from(result.get("status")))
statuses.append(status)
evidence.add(f"{prefix}-{name}", platform=platform_name, node=node, target=text(detail.get("target", name)), observed=detail.get("observed", result.get("observed", "")), expected=detail.get("expected", default_expected), status=status, remediation=text(detail.get("remediation", "")))
return "environment-fail" if "environment-fail" in statuses else ("blocked" if "blocked" in statuses else "ready")
status = status_from(result.get("status"))
evidence.add(prefix, platform=platform_name, node=node, target=prefix, observed=result.get("observed", ""), expected=default_expected, status=status, remediation=text(result.get("remediation", "")))
return status
def require_runner_checks(evidence: Evidence, result: Mapping[str, Any], *, prefix: str, platform_name: str, node: str, required: set[str]) -> bool:
checks = result.get("checks")
missing = sorted(required - set(checks)) if isinstance(checks, dict) else sorted(required)
if not missing:
return True
evidence.add(prefix + "-contract", platform=platform_name, node=node, target="runner response", observed={"missing": missing}, expected=sorted(required), status="blocked", remediation="让 tester 节点 adapter 返回每个必需的 adb/boot、macOS/xcrun/simctl 或 DNS/TCP/HTTP 证据")
return False
def network_local(url: str) -> list[dict[str, Any]]:
parsed = urllib.parse.urlsplit(url)
host = parsed.hostname or ""
port = parsed.port or (443 if parsed.scheme == "https" else 80)
try:
addresses = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
except OSError as exc:
return [{"name": "dns", "status": "environment-fail", "observed": str(exc), "expected": "resolvable hostname", "remediation": "检查 tester 节点 DNS/VPN/网络"}]
rows: list[dict[str, Any]] = [{"name": "dns", "status": "ready", "observed": sorted({item[4][0] for item in addresses}), "expected": "resolvable hostname"}]
try:
with socket.create_connection((host, port), timeout=5):
rows.append({"name": "tcp", "status": "ready", "observed": f"{host}:{port}", "expected": "TCP connection"})
except OSError as exc:
rows.append({"name": "tcp", "status": "environment-fail", "observed": str(exc), "expected": f"TCP {host}:{port}", "remediation": "检查端口、防火墙和测试节点到 HOST 的路由"})
return rows
try:
with urllib.request.urlopen(urllib.request.Request(url, method="GET"), timeout=5) as response:
code = response.status
rows.append({"name": "http", "status": "ready" if 200 <= code < 500 else "environment-fail", "observed": code, "expected": "HTTP 2xx-4xx", "remediation": "检查后端健康状态" if code >= 500 else ""})
except urllib.error.HTTPError as exc:
rows.append({"name": "http", "status": "ready" if 200 <= exc.code < 500 else "environment-fail", "observed": exc.code, "expected": "HTTP 2xx-4xx", "remediation": "检查后端服务,不把 5xx 当成可用" if exc.code >= 500 else ""})
except (OSError, urllib.error.URLError) as exc:
rows.append({"name": "http", "status": "environment-fail", "observed": str(exc), "expected": "HTTP response", "remediation": "检查服务监听、代理、TLS 和节点网络"})
return rows
def check_network(evidence: Evidence, config: Mapping[str, Any], row: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
tester = row.get("tester") or {}
runner = tester.get("runner") if isinstance(tester, dict) else None
is_local = node in {"local", "localhost", "127.0.0.1"}
statuses: list[str] = []
for probe_index, probe in enumerate(config.get("service_probes") or []):
raw_url = text(probe.get("url"))
if "{PUBLIC_HOST}" in raw_url and not text(config.get("public_host")).strip():
evidence.add(f"{platform_name}-{index}-network-{probe_index}-public-host", platform=platform_name, node=node, target=raw_url, expected="PUBLIC_HOST configured", status="blocked", remediation="提供 PUBLIC_HOST 或改用 HOST 探针")
statuses.append("blocked")
continue
url = normalize_url(expand(raw_url, config))
if not urllib.parse.urlsplit(url).hostname:
evidence.add(f"{platform_name}-{index}-network-{probe_index}-url", platform=platform_name, node=node, target=url, expected="valid probe URL", status="blocked", remediation="修正 service_probes URL 或 HOST/PUBLIC_HOST")
statuses.append("blocked")
continue
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
details = network_local(url)
expected_statuses = probe.get("expected_status")
for detail in details:
if detail["name"] == "http" and isinstance(expected_statuses, list) and detail["status"] == "ready" and detail["observed"] not in expected_statuses:
detail = {**detail, "status": "environment-fail", "expected": expected_statuses, "remediation": "检查服务探针的 expected_status 或后端响应"}
status = status_from(detail["status"])
statuses.append(status)
evidence.add(f"{platform_name}-{index}-network-{probe_index}-{detail['name']}", platform=platform_name, node=node, target=url, observed=detail["observed"], expected=detail["expected"], status=status, remediation=detail.get("remediation", ""))
elif isinstance(runner, dict) and runner.get("kind") == "contract":
binding = text(tester.get("binding")) if isinstance(tester, dict) else ""
status, observed, result = runner_call(runner, {"operation": "network_probe", "platform": platform_name, "node": node, "binding": binding, "url": url, "probe": probe})
result = result or {"status": status, "observed": observed}
complete = require_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-network-{probe_index}", platform_name=platform_name, node=node, required={"dns", "tcp", "http"})
network_status = add_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-network-{probe_index}", platform_name=platform_name, node=node, default_expected="DNS/TCP/HTTP reachable from actual tester node")
statuses.append(network_status if complete else "blocked")
else:
evidence.add(f"{platform_name}-{index}-network-{probe_index}", platform=platform_name, node=node, target=url, expected="node runner contract", status="blocked", remediation="远端 tester 必须提供 agent-runtime adapter runner")
statuses.append("blocked")
return "environment-fail" not in statuses and "blocked" not in statuses
def check_fixture_cleanup(evidence: Evidence, config: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
ok = True
fixture = config.get("fixtures")
valid_fixture = isinstance(fixture, dict) and (not fixture.get("required", True) or bool(fixture.get("references")))
if not valid_fixture:
evidence.add(f"{platform_name}-{index}-fixture", platform=platform_name, node=node, target="fixtures", expected="required fixture references", status="blocked", remediation="声明隔离测试账号/fixture;不要在运行中临时创建")
ok = False
else:
evidence.add(f"{platform_name}-{index}-fixture", platform=platform_name, node=node, target="fixtures", observed=fixture.get("references"), expected="declared fixture references", status="ready")
cleanup = config.get("cleanup")
valid_noop = isinstance(cleanup, dict) and cleanup.get("required") is False and bool(text(cleanup.get("reason")).strip())
valid_steps = isinstance(cleanup, dict) and (bool(cleanup.get("steps")) or bool(text(cleanup.get("run")).strip()))
if not (valid_noop or valid_steps):
evidence.add(f"{platform_name}-{index}-cleanup", platform=platform_name, node=node, target="cleanup", expected="steps/run or explicit read-only reason", status="blocked", remediation="声明 teardown/rollback;清理缺失时禁止开始测试")
ok = False
else:
evidence.add(f"{platform_name}-{index}-cleanup", platform=platform_name, node=node, target="cleanup", observed=cleanup, expected="cleanup contract declared", status="ready")
return ok
def run(config: Mapping[str, Any], mode: str) -> Evidence:
evidence = Evidence(mode)
shape_ok = validate_shape(config, evidence)
deployment_ok = True
if mode == "prepare":
deployment = config.get("deployment")
environment = deployment.get("environment") if isinstance(deployment, dict) else None
command = argv(deployment.get("command")) if isinstance(deployment, dict) else None
if environment != "test":
evidence.add("deployment-environment", target="deployment.environment", observed=environment, expected="test", status="blocked", remediation="prepare 只允许 deployer 的 test 环境")
if not command:
evidence.add("deployment-command", target="deployment.command", expected="existing deployer adapter command", status="blocked", remediation="提供既有 deployer 的 JSON command contract;不要在 skill 内实现部署")
deployment_ok = False
elif not command:
evidence.add("deployment-command", target="deployment.command", expected="existing deployer adapter command", status="blocked", remediation="提供既有 deployer 的 JSON command contract;不要在 skill 内实现部署")
deployment_ok = False
else:
code, output = command_text(command, payload={"operation": "prepare", "environment": "test", "project_root": config.get("project_root"), "host": config.get("host"), "public_host": config.get("public_host")})
if code != 0:
evidence.add("deployment", target="test", observed=output[-800:], expected="deployer receipt status=ready", status="environment-fail", remediation="按 deployer 返回证据修复测试后端,再重跑 prepare")
deployment_ok = False
else:
try:
receipt = json.loads(output)
except json.JSONDecodeError:
receipt = {}
status = status_from(receipt.get("status"), "environment-fail") if isinstance(receipt, dict) else "environment-fail"
evidence.add("deployment", target="test", observed=receipt.get("observed", output[-800:]) if isinstance(receipt, dict) else output[-800:], expected="deployer receipt status=ready", status=status, remediation="检查既有 deployer 的测试环境 receipt" if status != "ready" else "")
deployment_ok = status == "ready"
if not shape_ok:
return evidence
for index, row in enumerate(config["matrix"]):
platform_name = text(row.get("platform")).lower()
tester = row.get("tester") or {}
node = text(tester.get("node"))
binding = text(tester.get("binding"))
evidence.add(f"{platform_name}-{index}-binding", platform=platform_name, node=node, target=binding, observed=binding, expected="resolved tester.<name> binding", status="ready")
artifact_ok = record_artifact(evidence, row, platform_name, node, index)
fixture_ok = check_fixture_cleanup(evidence, config, platform_name, node, index)
runner = tester.get("runner") if isinstance(tester, dict) else None
is_local = node in {"local", "localhost", "127.0.0.1"}
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
platform_status, details = local_platform_probe(platform_name)
for name, detail in details.items():
evidence.add(f"{platform_name}-{index}-platform-{name}", platform=platform_name, node=node, target=name, observed=detail.get("observed", ""), expected=detail.get("expected", "platform ready"), status=status_from(detail.get("status"), platform_status), remediation=detail.get("remediation", ""))
platform_ok = platform_status == "ready"
elif isinstance(runner, dict) and runner.get("kind") == "contract":
status, observed, result = runner_call(runner, {"operation": "platform_probe", "platform": platform_name, "node": node, "binding": binding})
result = result or {"status": status, "observed": observed}
required = {"adb", "device", "boot"} if platform_name == "android" else {"macos", "xcrun", "simctl", "boot"}
complete = require_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-platform", platform_name=platform_name, node=node, required=required)
platform_status = add_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-platform", platform_name=platform_name, node=node, default_expected="platform/device ready")
platform_ok = complete and platform_status == "ready"
else:
evidence.add(f"{platform_name}-{index}-platform-runner", platform=platform_name, node=node, target=node, expected="node runner contract", status="blocked", remediation="远端 tester 必须由 agent-runtime 提供可执行 runner")
platform_ok = False
network_ok = check_network(evidence, config, row, platform_name, node, index)
# This isolated project assumes the client capabilities used by its
# journeys. Their real behavior is a product assertion, not a preflight
# environment gate.
row_ok = artifact_ok and fixture_ok and platform_ok and network_ok
if mode == "prepare":
artifact = row.get("artifact") or {}
identity = artifact.get("package" if platform_name == "android" else "bundle_id")
target_path = text(artifact.get("remote_path") or artifact.get("path"))
if not row_ok or not deployment_ok:
evidence.add(f"{platform_name}-{index}-install", platform=platform_name, node=node, target=target_path, observed="not-run", expected="verify and deployment ready", status="blocked", remediation="先修复前置检查;prepare 不伪造安装成功")
evidence.add(f"{platform_name}-{index}-start", platform=platform_name, node=node, target=text(identity), observed="not-run", expected="install ready", status="blocked", remediation="安装前置失败,未启动 app")
else:
install_request = {"operation": "install", "platform": platform_name, "node": node, "artifact": target_path, "identity": identity}
start_request = {"operation": "start", "platform": platform_name, "node": node, "identity": identity}
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
if platform_name == "android":
adb = shutil.which("adb") or "adb"
serial = os.environ.get("ANDROID_SERIAL", "")
install_code, install_output = command_text([adb, "-s", serial, "install", "-r", target_path] if serial else [adb, "install", "-r", target_path])
install_result = {"status": "ready" if install_code == 0 else "environment-fail", "observed": install_output}
launch = [adb, "-s", serial, "shell", "monkey", "-p", text(identity), "1"] if serial else [adb, "shell", "monkey", "-p", text(identity), "1"]
start_code, start_output = command_text(launch) if install_code == 0 else (1, "install failed")
start_result = {"status": "ready" if start_code == 0 else "environment-fail", "observed": start_output}
else:
install_result = {"status": "environment-fail", "observed": "local iOS install requires a macOS runner", "remediation": "use tester.ios contract runner on Mac"}
start_result = {"status": "blocked", "observed": "not-run", "remediation": "install failed"}
add_runner_checks(evidence, install_result, prefix=f"{platform_name}-{index}-install", platform_name=platform_name, node=node, default_expected="artifact installed")
add_runner_checks(evidence, start_result, prefix=f"{platform_name}-{index}-start", platform_name=platform_name, node=node, default_expected="app launched")
elif isinstance(runner, dict) and runner.get("kind") == "contract":
install_status, install_observed, install_result = runner_call(runner, install_request)
add_runner_checks(evidence, install_result or {"status": install_status, "observed": install_observed}, prefix=f"{platform_name}-{index}-install", platform_name=platform_name, node=node, default_expected="artifact installed")
if install_status == "ready":
start_status, start_observed, start_result = runner_call(runner, start_request)
add_runner_checks(evidence, start_result or {"status": start_status, "observed": start_observed}, prefix=f"{platform_name}-{index}-start", platform_name=platform_name, node=node, default_expected="app launched")
else:
evidence.add(f"{platform_name}-{index}-start", platform=platform_name, node=node, target=text(identity), observed="not-run", expected="install ready", status="blocked", remediation="安装失败,先修复 tester 节点")
return evidence
def write_evidence(evidence: Evidence, yaml_path: Path, csv_path: Path) -> dict[str, Any]:
payload = evidence.payload()
yaml_path.parent.mkdir(parents=True, exist_ok=True)
csv_path.parent.mkdir(parents=True, exist_ok=True)
yaml_path.write_text(yaml.safe_dump(payload, allow_unicode=True, sort_keys=False), encoding="utf-8")
with csv_path.open("w", newline="", encoding="utf-8") as handle:
writer = csv.DictWriter(handle, fieldnames=EVIDENCE_FIELDS)
writer.writeheader()
writer.writerows(evidence.rows)
return {"status": payload["status"], "yaml": str(yaml_path), "csv": str(csv_path), "checks": len(evidence.rows), "summary": payload["summary"]}
def main() -> int:
parser = argparse.ArgumentParser(description="Verify or prepare a test environment and emit YAML/CSV evidence")
parser.add_argument("mode", choices=("verify", "prepare"))
parser.add_argument("--config", required=True, type=Path)
parser.add_argument("--yaml", dest="yaml_path", type=Path, default=Path("preflight.yaml"))
parser.add_argument("--csv", dest="csv_path", type=Path, default=Path("preflight.csv"))
args = parser.parse_args()
try:
config = load_config(args.config)
evidence = run(config, args.mode)
result = write_evidence(evidence, args.yaml_path, args.csv_path)
except (OSError, ValueError, yaml.YAMLError) as exc:
print(json.dumps({"status": "blocked", "error": str(exc)}, ensure_ascii=False), file=sys.stderr)
return 2
print(json.dumps(result, ensure_ascii=False, indent=2))
return 0 if result["status"] == "ready" else 1
if __name__ == "__main__":
raise SystemExit(main())
+75
View File
@@ -0,0 +1,75 @@
#!/usr/bin/env python3
"""Deterministic runner fixture for test-preflight contract tests."""
import json
import os
import sys
request = json.load(sys.stdin)
scenario = os.environ.get("PREFLIGHT_RUNNER_SCENARIO", "ready")
operation = request.get("operation")
if operation in {"deployment", "prepare"}:
print(json.dumps({"status": "ready", "observed": "mock test deploy receipt"}))
elif operation == "platform_probe":
platform = request.get("platform")
if scenario == "android-unauthorized" and platform == "android":
print(json.dumps({"status": "environment-fail", "checks": {
"adb": {"status": "ready", "observed": "adb 35", "expected": "adb responds"},
"device": {"status": "environment-fail", "observed": "emulator-5554 unauthorized", "expected": "authorized device", "remediation": "accept RSA authorization"},
"boot": {"status": "environment-fail", "observed": "unknown", "expected": "boot complete"},
}}))
elif scenario == "mac-unavailable" and platform == "ios":
print(json.dumps({"status": "environment-fail", "checks": {
"macos": {"status": "environment-fail", "observed": "Mac unavailable", "expected": "Darwin tester node", "remediation": "restore tester.ios Mac"},
}}))
else:
print(json.dumps({"status": "ready", "checks": {
"adb": {"status": "ready", "observed": "adb 35", "expected": "adb responds"},
"device": {"status": "ready", "observed": "authorized", "expected": "authorized device"},
"boot": {"status": "ready", "observed": "boot complete", "expected": "boot complete"},
"macos": {"status": "ready", "observed": "Darwin", "expected": "Darwin"},
"xcrun": {"status": "ready", "observed": "xcrun", "expected": "xcrun"},
"simctl": {"status": "ready", "observed": "available", "expected": "simctl"},
}}))
elif operation == "network_probe":
if scenario == "host-unreachable":
print(json.dumps({"status": "environment-fail", "checks": {
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
"tcp": {"status": "environment-fail", "observed": "connection refused", "expected": "TCP reachable", "remediation": "check HOST route"},
}}))
elif scenario == "http-failure":
print(json.dumps({"status": "environment-fail", "checks": {
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
"tcp": {"status": "ready", "observed": "443 open", "expected": "TCP reachable"},
"http": {"status": "environment-fail", "observed": 503, "expected": "HTTP 2xx-4xx", "remediation": "check service health"},
}}))
else:
print(json.dumps({"status": "ready", "checks": {
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
"tcp": {"status": "ready", "observed": "443 open", "expected": "TCP reachable"},
"http": {"status": "ready", "observed": 200, "expected": "HTTP 2xx-4xx"},
}}))
elif operation == "capability_probe":
capability = request.get("capability")
checks = {
"album_picker": {
"picker": {"status": "ready", "observed": "real picker opened and returned image-001"},
},
"photo_permission": {
"permission": {"status": "ready", "observed": "permission state checked on device"},
"test_image": {"status": "ready", "observed": "image-001 available in test album"},
},
"ime_pinyin": {
"composition": {"status": "ready", "observed": "nihao composed in Chinese IME"},
"candidates": {"status": "ready", "observed": "你好 candidate selected"},
},
}
if scenario == "capability-missing" and capability == "album_picker":
print(json.dumps({"status": "ready", "checks": {}}))
else:
print(json.dumps({"status": "ready", "checks": checks.get(capability, {})}))
elif operation in {"install", "start"}:
print(json.dumps({"status": "ready", "observed": operation + "ed mock artifact"}))
else:
print(json.dumps({"status": "environment-fail", "observed": "unknown operation"}))
@@ -0,0 +1,225 @@
#!/usr/bin/env python3
from __future__ import annotations
import json
import os
import plistlib
import subprocess
import sys
import tempfile
import unittest
import zipfile
from pathlib import Path
from unittest import mock
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "scripts"))
import preflight # noqa: E402
FIXTURE_RUNNER = Path(__file__).resolve().parent / "fixtures" / "contract_runner.py"
class PreflightTests(unittest.TestCase):
def setUp(self) -> None:
self.temp = tempfile.TemporaryDirectory()
self.root = Path(self.temp.name)
self.apk = self.root / "writes.apk"
self.apk.write_bytes(b"mock apk")
self.ipa = self.root / "writes.ipa"
with zipfile.ZipFile(self.ipa, "w") as archive:
archive.writestr("Payload/Writes.app/Info.plist", plistlib.dumps({"CFBundleIdentifier": "page.writes"}))
self.identity = self.root / "identity.py"
self.identity.write_text("print('page.writes')\n", encoding="utf-8")
def tearDown(self) -> None:
self.temp.cleanup()
def runner(self, scenario: str = "ready") -> dict[str, object]:
return {"kind": "contract", "command": [sys.executable, str(FIXTURE_RUNNER)]}
def config(self, *, platform: str = "android", artifact: Path | None = None, node: str = "tester.android", scenario: str = "ready") -> dict[str, object]:
artifact = artifact or self.apk
artifact_row: dict[str, object] = {
"path": str(artifact),
"package": "page.writes" if platform == "android" else None,
"bundle_id": "page.writes" if platform == "ios" else None,
}
if platform == "android":
artifact_row["identity_command"] = [sys.executable, str(self.identity)]
artifact_row.pop("bundle_id")
else:
artifact_row.pop("package")
return {
"schema_version": 1,
"project_root": str(self.root),
"host": "https://test.example.invalid",
"public_host": "https://public.example.invalid",
"service_probes": [{"id": "health", "url": "{HOST}/health", "expected_status": [200]}],
"fixtures": {"required": True, "references": ["isolated-account"]},
"cleanup": {"required": True, "steps": ["cleanup-run-data"]},
"matrix": [{
"platform": platform,
"tester": {"binding": "tester.android" if platform == "android" else "tester.ios", "node": node, "runner": self.runner(scenario)},
"artifact": artifact_row,
}],
}
def write_config(self, data: dict[str, object]) -> Path:
path = self.root / "matrix.yaml"
import yaml
path.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8")
return path
def invoke(self, mode: str, data: dict[str, object]) -> tuple[int, dict[str, object], dict[str, object]]:
config = self.write_config(data)
yaml_path, csv_path = self.root / f"{mode}.yaml", self.root / f"{mode}.csv"
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "ready"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), mode,
"--config", str(config), "--yaml", str(yaml_path), "--csv", str(csv_path),
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
import yaml
return process.returncode, yaml.safe_load(yaml_path.read_text(encoding="utf-8")), {"stdout": process.stdout, "stderr": process.stderr}
def test_missing_artifact_is_blocked_and_report_has_contract_columns(self):
data = self.config(artifact=self.root / "missing.apk")
code, report, _ = self.invoke("verify", data)
self.assertEqual(code, 1)
self.assertEqual(report["status"], "blocked")
row = next(row for row in report["checks"] if row["check_id"] == "android-0-artifact")
self.assertEqual(row["status"], "blocked")
self.assertEqual(set(row), set(preflight.EVIDENCE_FIELDS))
def test_android_unauthorized_is_environment_fail(self):
data = self.config()
config = self.write_config(data)
out = self.root / "unauthorized.yaml"
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "android-unauthorized"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config), "--yaml", str(out), "--csv", str(self.root / "u.csv")
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
report = json.loads(json.dumps(__import__("yaml").safe_load(out.read_text(encoding="utf-8"))))
self.assertEqual(process.returncode, 1)
self.assertEqual(report["status"], "environment-fail")
self.assertEqual(next(row for row in report["checks"] if row["check_id"].endswith("-device"))["status"], "environment-fail")
def test_linux_without_xcrun_is_environment_fail(self):
data = self.config(platform="ios", artifact=self.ipa, node="local")
data["matrix"][0]["tester"]["runner"] = {"kind": "local"}
with mock.patch.object(preflight.host_platform, "system", return_value="Linux"), mock.patch.object(preflight, "network_local", return_value=[{"name": "dns", "status": "ready", "observed": "ok", "expected": "DNS"}, {"name": "tcp", "status": "ready", "observed": "ok", "expected": "TCP"}, {"name": "http", "status": "ready", "observed": 200, "expected": "HTTP"}]):
result = preflight.run(data, "verify")
self.assertEqual(result.overall(), "environment-fail")
row = next(row for row in result.rows if row["check_id"].endswith("-platform-macos"))
self.assertEqual(row["status"], "environment-fail")
def test_mac_unavailable_contract_is_environment_fail(self):
data = self.config(platform="ios", artifact=self.ipa)
config_path = self.write_config(data)
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "mac-unavailable"
output = self.root / "mac.yaml"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "mac.csv")
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
import yaml
report = yaml.safe_load(output.read_text(encoding="utf-8"))
self.assertEqual(process.returncode, 1)
self.assertEqual(report["status"], "environment-fail")
self.assertTrue(any(row["check_id"].endswith("-macos") and row["status"] == "environment-fail" for row in report["checks"]))
def test_missing_fixture_or_cleanup_is_blocked(self):
data = self.config()
data["fixtures"] = {"required": True, "references": []}
data["cleanup"] = {"required": True, "steps": []}
result = preflight.run(data, "verify")
self.assertEqual(result.overall(), "blocked")
self.assertTrue(any(row["check_id"].endswith("-fixture") and row["status"] == "blocked" for row in result.rows))
self.assertTrue(any(row["check_id"].endswith("-cleanup") and row["status"] == "blocked" for row in result.rows))
def test_mobile_capabilities_are_assumed_and_not_a_preflight_gate(self):
data = self.config()
config_path = self.write_config(data)
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "capability-missing"
output = self.root / "capabilities.yaml"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify",
"--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "capabilities.csv"),
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
report = __import__("yaml").safe_load(output.read_text(encoding="utf-8"))
self.assertEqual(process.returncode, 0)
self.assertEqual(report["status"], "ready")
self.assertFalse(any("capability-" in row["check_id"] for row in report["checks"]))
def test_host_unreachable_from_actual_runner_is_environment_fail(self):
data = self.config()
config_path = self.write_config(data)
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "host-unreachable"
output = self.root / "host.yaml"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "host.csv")
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
import yaml
report = yaml.safe_load(output.read_text(encoding="utf-8"))
self.assertEqual(process.returncode, 1)
self.assertEqual(report["status"], "environment-fail")
self.assertTrue(any("network" in row["check_id"] and row["status"] == "environment-fail" for row in report["checks"]))
def test_http_500_is_environment_fail_not_product_fail(self):
data = self.config()
config_path = self.write_config(data)
env = os.environ.copy()
env["PREFLIGHT_RUNNER_SCENARIO"] = "http-failure"
output = self.root / "http.yaml"
process = subprocess.run([
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "http.csv")
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
import yaml
report = yaml.safe_load(output.read_text(encoding="utf-8"))
self.assertEqual(process.returncode, 1)
http_row = next(row for row in report["checks"] if row["check_id"].endswith("-http"))
self.assertEqual(http_row["observed"], "503")
self.assertEqual(http_row["status"], "environment-fail")
def test_prepare_without_deployer_is_blocked_and_never_installs(self):
data = self.config()
code, report, _ = self.invoke("prepare", data)
self.assertEqual(code, 1)
self.assertEqual(report["status"], "blocked")
self.assertFalse(any(row["check_id"].endswith("-install") and row["status"] == "ready" for row in report["checks"]))
self.assertTrue(any(row["check_id"] == "deployment-command" for row in report["checks"]))
def test_prepare_mock_contract_calls_install_and_start(self):
data = self.config()
data["deployment"] = {"environment": "test", "command": [sys.executable, str(FIXTURE_RUNNER)]}
code, report, _ = self.invoke("prepare", data)
self.assertEqual(code, 0)
self.assertEqual(report["status"], "ready")
ids = {row["check_id"] for row in report["checks"]}
self.assertIn("android-0-install", ids)
self.assertIn("android-0-start", ids)
def test_verify_never_calls_install_or_start(self):
data = self.config()
calls: list[str] = []
original = preflight.runner_call
def record(runner, request):
calls.append(str(request["operation"]))
return original(runner, request)
with mock.patch.object(preflight, "runner_call", side_effect=record):
result = preflight.run(data, "verify")
self.assertEqual(result.overall(), "ready")
self.assertNotIn("install", calls)
self.assertNotIn("start", calls)
self.assertNotIn("capability_probe", calls)
if __name__ == "__main__":
unittest.main()
+84 -5
View File
@@ -8,6 +8,7 @@
- 「加一条相机开盖的测试」「完善 TC-CAM-003 的验证」
- 「跑 smoke」「测 CAM」「跑 TC-INF-001 和 TC-CAM-001」
- 「现场复测」
- 「用这个 APK/IPA 地址跑 Android+iOS」
不管 ACK。这是项目自己的测试目录。
@@ -16,18 +17,36 @@
- 目录在目标测试项目根的 `.pouch/tester/`。父仓库和 `app` 等 Git submodule 可以各自有
catalog;先运行 `catalog.py roots --root <repo-root>` 发现它们,不要把两个项目的 cases 合并。
没有任何 catalog 才说「初始化 tester」。
- 从父仓库直接执行 tester 命令时,只有一个子 catalog 会自动选中;多个子 catalog 会自动
fan-out,逐个执行且不合并 cases。无需在根 `.pouch` 里增加未定义的 `include`。
- 跑真实环境前,复制 `.pouch/tester/local.yaml.example` 为 `local.yaml`;模板默认
`HOST: "127.0.0.1"`。如果目标不是本机,显式改成目标地址。也可以用环境变量
`HX_TEST_HOST` 覆盖。
`HOST` 覆盖。
- 测试默认在当前项目执行。也可以直接提供 Orca project id、项目名称或 terminal id;
tester 会自动解析 Agent、环境、项目和目录,写入 `local.yaml.agent`,并输出更新后的配置列表。
每轮测试会创建新的 terminal,结束后自动关闭;不会复用或关闭已有 terminal。
- 测试环境部署由 deployer 在本机 Compose 中触发;正式环境部署由本机 `~/code/app00`
项目的 fresh Agent terminal 触发,tester 不直接 SSH 或执行正式 Compose。
- 会动硬件、会改数据的用例带风险级别(`readonly` / `mutating` / `destructive`)。默认只跑最低档 `readonly`。要跑更高风险,需要你明确说,例如「风险到 mutating」;碰哪个子系统另由 `gate`(如 hardware / weld)控制。
- 会动硬件、会改数据的用例带风险级别(`readonly` / `mutating` / `destructive`)。默认
`standard` 只跑最低档 `readonly`。隔离、可重置的 Writes 类环境在 catalog 设置
`execution_profile: isolated-full` 后,所有 active 自动化用例都进入执行,不因
`AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或
`AUTHORIZE_PASSWORD` skipped;risk 原值仍保留。设备、产物、服务和 cleanup/rollback
由 test-preflight 保证;客户端相册、权限和中文输入能力默认具备,由 journey 用例验证。
## 使用示例
```bash
<skill-dir>/tester run \
--artifact android=<path-or-url> \
--artifact ios=<path-or-url> \
--platform android,ios
```
调用方必须给出每个平台的产物位置,可以是本地文件或下载地址。tester 按这个输入取用或下载
APK/IPA,为本轮创建隔离 worktree,在 Android 和 iOS tester 节点安装并验证身份,然后交给
真实移动 session 执行 catalog。默认全量闸门,只有显式 `--partial` 才允许单平台继续。
```text
列出测试用例
```
@@ -56,9 +75,14 @@
1. 用脚本读 `.pouch/tester/`,按分类统计或筛出你点名的用例,不会把整库塞进对话。
2. 改用例前先给你看将要写入的内容,你确认再保存。
3. 执行时:若用户要求准备环境,先按目标环境走 deployer 路由;再解析并更新 Agent 配置,输出生效列表,再准备 Orca 项目并创建临时 terminal;脚本类、
浏览器和移动模拟器都在该 Agent 环境中执行。
4. 超出你允许的风险级别会跳过,不会偷偷执行。
3. 执行时先建立“产物 × 平台 × tester 绑定 × 执行环境”矩阵,并把平台、产物和 binding
传给 `catalog.py select` 做硬性校验,再对矩阵全部行做
preflight。任一产物、平台或环境不可用,都先返回完整阻塞清单,不创建测试 session、
不执行可用平台;只有全部可用才按目标环境走 deployer 路由、解析 Agent、输出生效
列表、准备 Orca 项目并创建临时 terminal。脚本类、浏览器和移动模拟器都在对应
Agent 环境中执行。`builder.<platform>` 不能替代 `tester.<platform>`。
4. `standard` 下超出你允许的风险级别会跳过;`isolated-full` 由 catalog profile 明确
全量 active 执行,不把风险悄悄改成 readonly。
5. 无论测试成功、失败或异常,都会关闭本轮创建的 terminal;跑完给出按分类计数的
结果,以及 `pass / fail / environment-fail / skipped / not-run`。
@@ -67,3 +91,58 @@
- **列出**:能看到分类表和总条数;`.pouch/tester/cases.csv` 已由 skill 里的 `generate_cases_csv.py` 从 `cases/` 重新生成。每次运行的结果保存在独立的 `.pouch/tester/runs/<run-id>.*` 文件中。
- **维护**:目标 id 能 `查看` 到,且检查通过。
- **运行**:有本次 `runs/*.yaml` 快照和同名 `.csv` 成绩单(`executed` 列区分已跑 / 未跑);没填 HOST 时会停,而不是编造失败。
### 客户端用户旅程
测试用例支持两种顶层类型:`test_type: integration` 和 `test_type: journey`。集成套件
至少包括 `api-contract`、`authorization`、`build-contract`;客户端套件仍是
`client-smoke`、`client-regression`、`safety`。没有显式范围时默认不选择 integration。
执行顺序固定为 `test-preflight -> integration -> journey`。preflight 状态为 blocked 或
environment-fail 时不创建客户端 session;integration 失败时依赖 journey 为 `not-run`。
服务不可达是 environment-fail,契约断言错误是 integration fail,正确账号登录失败仍是
journey fail。`tester run`/bundle 必须传入 caller 提供的 matrix:
```bash
<skill-dir>/tester run --preflight-config /abs/preflight.yaml \
--artifact android=/abs/app.apk --artifact ios=/abs/app.ipa
```
没有真实 deployer、Mac 或设备时,只能使用 test-preflight 的 command-contract mock fixture
验证协议,不能把 mock 结果写成真实运行成功。
客户端 E2E 用例使用 `client_suite: client-smoke | client-regression | safety`,并明确
`journey`、`fixture`、`session`、`cleanup`、`platform`。标准生命周期是:
```text
preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告
```
正确测试账号登录失败记 `fail`;设备、产物或服务不可达记 `environment-fail`。fixture 或
cleanup 缺失会在开始前阻断;前置失败后的剩余用例是 `not-run`,明确排除才是 `skipped`。
`safety` 必须显式选择和授权,不进入默认 smoke/full。它与普通 tester 用例共用原有
CSV/YAML 运行记录和汇总校验。
移动 preflight 还必须在实际 tester 节点验证相册 picker、照片权限与测试图片,以及中文
拼音的真实 composition 和候选词选择;缺失时所有 testcase 都保留 `not-run` +
`blocked_by`,不会静默 `skipped`。
失败回流使用 run YAML 顶层 `defects`/`test_issues` 与 `feedback`:PR 触发先更新 PR/关联
issue、通知 dispatcher 重新分配开发;需求/bug 触发直接回开发流程。可用
`scripts/record_issues.py` 生成同名 CSV 的结构化 issue 列。
### 汇总已有运行结果
对已经生成的同一运行 CSV/YAML,使用显式路径汇总:
```bash
python3 <skill-dir>/scripts/summarize_run.py \
--csv .pouch/tester/runs/<run-id>.csv \
--yaml .pouch/tester/runs/<run-id>.yaml
```
脚本只从 CSV 的 `result` 列计算 `pass`、`fail`、`environment-fail`、`skipped`、
`not-run` 五类状态,并要求五类计数和等于 CSV 数据行数。YAML 只用于校验
`results` 数量和 testcase id 集合;数量、集合、重复/缺失 id 或 CSV 缺失/未知状态
时返回非零并输出诊断。校验成功才表示这份汇总可作为运行报告;非零表示报告不一致,
不能据此判定产品失败。
+70 -4
View File
@@ -30,12 +30,17 @@ python3 <skill-dir>/scripts/catalog.py roots --root <repo-root>
- `<repo-root>/.pouch/tester/catalog.yaml` 存在时,它是一个独立测试根;
- 同时检查 Git submodule 和一层普通子项目目录;
- 只有一个嵌套 catalog 时,可直接用该路径继续 `list/select/run`;
- 有多个 catalog 时按用户指定的组件分别执行,例如 `--root app`,不要猜测或合并;
- 有多个 catalog 时由 tester 自动 fan-out,逐个项目分别执行,不等待用户选择,也不合并 cases;
- 每个测试根单独生成 `.pouch/tester/runs/*.yaml`/`.csv`,交付记录可分别引用这些快照。
如果父仓库没有 catalog 但子项目有 catalog,不能把“父目录无 catalog”报告为环境失败;先用
`roots` 找到子项目并继续。只有发现不到任何 catalog,才按缺少 tester 目录处理。
CLI 也有同样的保护:从父仓库执行 `list/select/validate/run-script` 时,若只发现一个
子 tester 根会自动进入该根;发现多个候选会自动 fan-out,每个根独立输出、独立校验、
独立生成运行记录。不需要、也不能通过在根 `.pouch` 里写一个未定义的 `include` 来拼接
子项目配置。
Agent 执行目标优先来自项目根 `.pouch/agent.yaml` 的 `tester.<name>` binding,由共享
`agent-runtime` 解析 environment、host、project、path 和 profile。`.pouch/tester/local.yaml`
仍是兼容覆盖层:解析 binding 后可 materialize 为旧 `agent` mapping,但不应再维护两套
@@ -69,10 +74,10 @@ tester agent 配置。
- `local.yaml.example` 的 HOST 默认是 `127.0.0.1`;如果目标不是本机,必须在
`local.yaml` 或环境变量中显式覆盖。`local.yaml` 不存在且没有环境变量时仍不猜
HOST,停下并记 environment-fail。
- 默认风险上限是 catalog 里的 `default_max_risk`。超出上限的条目标 `skipped`,禁止执行。
- 默认执行 profile 是 `standard`:风险上限来自 catalog 的 `default_max_risk`,超出上限的条目标 `skipped`,禁止执行。若 catalog 声明 `execution_profile: isolated-full`,只允许用于隔离且可重置的测试环境:所有 active 自动化用例都进入执行矩阵,不因 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或 `AUTHORIZE_PASSWORD` skipped;原有 `risk` 值必须保留并进入报告。设备、产物、服务和清理契约由 test-preflight 检查;相册、照片权限/测试图片和中文拼音能力在本项目中默认具备,由 journey 用例真实验证。
- `risk` 只能是 `readonly` / `mutating` / `destructive`,默认只跑 `readonly`。`tag=smoke` 只能配 `risk=readonly`。规则见 [levels.md](references/levels.md)。
- `driver=browser` 必须开真实页面;`driver=mobile` 必须挂模拟器 session。二者都不得改成只打 API。
- `automation=assisted|manual` 需要人在场:无人值守时记 `not-run`,不要自己代跑人工步骤。见 [concepts.md](references/concepts.md)。
- `automation=assisted|manual` 需要人在场:无人值守时记 `not-run`,不要自己代跑人工步骤。isolated-full 取消的是风险/授权过滤,不会伪造人工操作。见 [concepts.md](references/concepts.md)。
- 浏览器、移动模拟器按 catalog `sessions` 初始化一次,同一次运行内复用。
- Agent、环境、项目目录和 Orca terminal 生命周期见 [execution.md](references/execution.md):可以从 project id/名称或 terminal id 自动解析,更新后输出配置列表;每轮都新建 terminal,结束后关闭本轮创建的 terminal,不复用或关闭已有 terminal。
- 环境准备见 [deployment.md](references/deployment.md):测试环境只调用 deployer 的本机 Compose 路径;正式环境只委派给本机 `~/code/app00` 的 fresh Agent terminal,tester 不直接 SSH 或执行生产 Compose。
@@ -132,4 +137,65 @@ python3 <skill-dir>/scripts/catalog.py show --root <project-root> --id <CASE_ID>
## 运行
读 [run.md](references/run.md)、[execution.md](references/execution.md) 和 [deployment.md](references/deployment.md)。若本轮要求准备环境,先按目标环境完成部署,再 `select` 得到命中列表;测试环境由本机 deployer 触发,正式环境由 app00 Agent terminal 委派。随后按 agent 解析目标项目、目录和 Agent,更新配置并输出生效列表,创建测试用临时 terminal 并按 driver 分流。未指定范围时等同「tag=smoke 且风险上限=默认且 automation=automated」。`select` 的 `run_together` 组按 [run.md](references/run.md) 整组执行。移动端还要先识别目标平台并完成 Android 或 iOS 环境闸门;闸门未通过时输出对应的用户引导,不继续猜测或代跑。无论部署或测试成功、失败、超时还是中断,都要清理各自本轮创建的 terminal。
读 [run.md](references/run.md)、[execution.md](references/execution.md)、[deployment.md](references/deployment.md) 和 [test-bundle.md](references/test-bundle.md)。运行必须按以下顺序执行:
1. 先建立本轮测试矩阵,列出实际要测的产物、平台、commit、包/Bundle 身份、产物路径或下载地址,以及对应的 `tester.<platform>` 执行绑定。用户要求 Android+iOS 时,两行都必须出现;不能因为只拿到 Android 产物就缩小范围。
2. 对所有 tester 根的矩阵行做完整 preflight:先用 `catalog.py select` 校验平台、产物定位和 tester binding,再检查目标 Agent/环境/项目路径、Android/iOS 设备闸门和 HOST 可达。所有根的检查完成前不得创建测试 session 或开始任何测试用例。
3. 有任一根或任一行不可用时,先完成其它根的 preflight,再一次性输出完整不可用清单、观测结果、修复动作和可复制的验证命令,然后停止测试执行;不能先执行已通过 preflight 的根,也不能把缺失平台记成测试通过。
4. 只有矩阵所有行都可用,才创建本轮测试 terminal/session,按 `select` 结果执行并记录成绩。
测试环境由本机 deployer 触发,正式环境由 app00 Agent terminal 委派。随后按 agent 解析目标项目、目录和 Agent,更新配置并输出生效列表。移动端每个平台的产物位置必须由调用方给出,值为本地文件或 URL;缺了就停,不要猜 builder、CI release 或本机旧包。矩阵行记产物位置、身份和 `tester.<platform>` binding,并用 `<skill-dir>/tester run --artifact android=<path-or-url> --artifact ios=<path-or-url> --platform android,ios` 下载或取用、安装并准备本轮隔离 worktree。Android 在 `tester.android`、iOS 在 `tester.ios` 节点执行。未指定范围时等同「tag=smoke 且风险上限=默认且 automation=automated」。`select` 的 `run_together` 组按 [run.md](references/run.md) 整组执行。无论部署或测试成功、失败、超时还是中断,都要清理各自本轮创建的 terminal。
### 客户端用户旅程标准
### 双层测试与执行顺序
用例顶层类型是 `test_type: integration | journey`。旧目录没有该字段时保持兼容:已有
`client_suite`/`journey` 视为 journey,已有 `integration_suite` 视为 integration。集成
套件至少包括 `api-contract`、`authorization`、`build-contract`;旅程套件仍固定为
`client-smoke`、`client-regression`、`safety`。集成用例不需要客户端 session、platform
或 journey 字段,不能被客户端套件选中。
一次运行严格遵守:
```text
test-preflight (blocked/environment-fail -> 不创建 session)
-> integration
-> journey (integration fail -> not-run,不创建 session)
```
`test-preflight` 的报告是环境证据,不计入 testcase 总数。服务不可达记
`environment-fail`;服务已达但契约断言错误记 integration `fail`;正确测试账号登录失败
仍记 journey `fail`。tester 的 `run`/bundle 入口必须接收 caller 提供的 preflight matrix,
没有 matrix 不得绕过闸门或伪造成功。
声明 `client_suite`(迁移期也接受 `suite`)的用例采用固定的三套客户端集合:
`client-smoke`、`client-regression`、`safety`。前两者分别覆盖最短只读主路径和隔离测试
环境中的普通用户回归;`safety` 覆盖危险/权限边界场景,必须显式选择和授权,不能被
默认 smoke/full 扩入。选择某套客户端旅程可用:
```bash
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-smoke
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-regression --max-risk mutating
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite safety --max-risk destructive
```
客户端用例必须声明 `journey`、`fixture`、`session`、`cleanup` 和 `platform`。旅程包含
`goal`、`entry_state`、`actions`、`assertions`、`final_state`、`failure_policy`;fixture
至少声明测试账号;session 指向 catalog 中可复用的 browser/mobile session;cleanup 要有
步骤或命令(只读也要显式说明无需清理);platform 只能是 Android/iOS。具体字段见
`references/schema.md`。平台适配只能改操作层,不能把真实客户端操作换成 API 调用。
客户端运行器统一遵守 `preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告`。正确测试
账号登录失败是产品 `fail`;设备、产物、服务或 session 不可达才是 `environment-fail`。
fixture 或 cleanup 缺失会在 `catalog.py select` 的 preflight 阶段阻断,不产生虚假逐条结果。
设备、产物、服务或 session 能力缺失时 preflight 必须明确 blocked/environment-fail,并为所有选中的 testcase 生成
`not-run` 结果行;本项目默认具备相册、照片权限、测试图片或中文拼音 composition/candidate
能力,不在 preflight 中校验或静默变成 `skipped`,由 journey 真实验证并记录产品失败。独立用例继续执行,依赖失败的用例写 `not-run` 和 `blocked_by`;
修复后按 targeted rerun,再执行 full integration → journey。成绩仍写现有
`runs/<run-id>.yaml` 并用 `generate_run_csv.py` 生成同名 CSV;YAML 可带 `defects`/
`test_issues` 与 `feedback`,记录 testcase、product/test issue 类型、expected/observed、
artifact hash、environment、证据路径、复现和下一步,不添加新的 result 值。
PR 触发的失败回流必须先更新 PR/关联 issue,再通知 dispatcher 重新分配开发;需求/bug
触发的失败直接返回开发流程。tester 只生成结构化回流计划,不在本地偷偷修改远端。
+25 -1
View File
@@ -35,11 +35,19 @@ catalog(整个测试集)
| `session` | 可复用会话(浏览器 / 模拟器) | pytest fixture;Robot suite variable | 不是一条用例 |
| `gate` | 项目细分闸门名 → 环境变量(`motion` / `hardware` / `weld`) | 授权开关 | 不是 `risk`,不要拿它代替风险等级 |
| `authorize` | 风险档 → 环境变量。该档执行前必须为 `true` | pytest `skipif` 条件 | 不是认证(auth),是**授权** |
| `execution_profile` | 执行环境策略:`standard` 或隔离可重置的 `isolated-full` | pytest marker/fixture policy | 不是风险等级;isolated-full 不会抹掉 `risk` |
| `assume` | 额外跳过条件(gate / authorize 之外的环境表达式) | JUnit `assumeTrue`;pytest `skipif` | 不是授权开关的重复写法 |
| `status_probe` | 分类级前置条件:请求一个 URL,命中即视为前置不满足 | pytest session fixture;Robot Suite Setup | 不是产品失败,不是整轮硬闸 |
| `how` | 怎么测:`steps` / `run` / `url` / `timeout` | test steps | 不是判定标准 |
| `expected` | 怎么算过:`{kind, value}` 列表 | pytest `assert`;Robot `Should …` | 不是步骤 |
| `run_together` | 必须成组执行的用例(有序、可选 continue_on_fail) | 场景组(scenario) | 不是 suite,不是 tag |
| `client_suite` | 客户端用户旅程套件:`client-smoke` / `client-regression` / `safety` | mobile/browser E2E suite | 不是 `category`,也不是通用 tag;迁移期可读 `suite` 别名 |
| `test_type` | 顶层测试类型:`journey` 或 `integration` | E2E vs service/contract test | 不是 driver;旧目录可由既有字段推断 |
| `integration_suite` | 集成套件:`api-contract` / `authorization` / `build-contract` | API/contract test suite | 不是客户端 `client_suite`;迁移期可读 `suite` 别名 |
| `journey` | 用户目标、入口、操作、UI 断言、最终状态和失败继续策略 | Cucumber scenario / Robot task | 不是 API contract |
| `fixture` | 旅程开始前声明的测试账号与隔离数据 | pytest fixture | 不是运行中临时猜出的数据 |
| `cleanup` | 本轮数据的 teardown 契约;缺失会在 preflight 阻断 | pytest teardown / Robot teardown | 不是测试通过后的可选备注 |
| `platform` | 旅程适用的 `android`/`ios` 平台列表 | device matrix | 不是 tester Agent binding;binding 仍在命令参数/运行环境 |
### 结果词
@@ -48,7 +56,7 @@ catalog(整个测试集)
| `pass` | 通过 | passed |
| `fail` | 产品缺陷 | failed |
| `environment-fail` | 前置/环境导致无法执行,**永远不是产品失败** | pytest error;JUnit aborted |
| `skipped` | **主动**不跑:`retired` / `pending` / 超风险上限 / 未授权 / `assume` 不成立 | pytest skipped;JUnit disabled |
| `skipped` | **主动**不跑:standard 下的 `retired` / `pending` / 超风险上限 / 未授权 / `assume` 不成立 | pytest skipped;JUnit disabled |
| `not-run` | **被动**没轮到:失败即停后剩下的;分类被 `status_probe` 挡住整组没执行;`automation` 需人工而无人值守 | pytest not run;deselected |
## 2. 命名迁移对照(v1 → v2)
@@ -75,6 +83,8 @@ catalog(整个测试集)
- 三级**单调**:`readonly ⊂ mutating ⊂ destructive`,所以只需要一个上限 `default_max_risk` 和一个 `--max-risk`,不用组合矩阵。
- 与 `gate` 正交:`risk` 说「影响多大」,`gate` 说「碰哪个子系统」。点动 vs 焊接这种细分授权用 `gate`,不要再造 risk 名。
- `tag=smoke` 只能配 `risk=readonly`:冒烟测试不许留下变化。
- 隔离可重置环境可设置 `execution_profile: isolated-full` 跑所有 active 用例;安全前置由
test-preflight 的设备、服务与 cleanup/rollback 契约保证,原始 risk 仍必须报告;客户端能力由 journey 用例验证。
### 3.2 `automation`:为什么是 `automated / assisted / manual`
@@ -86,6 +96,7 @@ catalog(整个测试集)
- `assisted`:Agent 驱动,但至少一步需要人在场;该步在 `how.steps` 里以 `[人工]` 开头(可被 `validate` 机器判定)。
- `manual`:只能人工执行;禁止 `driver`、禁止 `how.run`,必须有 `how.steps`。
- 默认只跑 `automated`。无人值守时 `assisted` / `manual` 记 `not-run`;有人在场时 `assisted` 暂停等人确认,`manual` 由人工回报结果。
- isolated-full 只取消风险/授权过滤,不伪造 assisted/manual 的人工步骤。
### 3.3 结果为什么拆出 `not-run`
@@ -99,6 +110,19 @@ catalog(整个测试集)
- 因此它是**分类级前置条件**:`catalog.status_probe` 是全局默认,`categories[].status_probe` 可覆盖(缺省继承 / `false` 关闭 / mapping 整体替换)。对应 pytest 的 session fixture、Robot 的 Suite Setup。
- 命中或不可达只影响用它的分类;只有所有选中分类都被挡住,整轮结论才是 `environment-fail`。`select` 会按分类返回解析后的生效值,Agent 不要自己读 catalog 猜。
### 3.5 客户端套件和生命周期
客户端套件固定为:
- `client-smoke`:启动、正确账号登录和少量只读主路径,通常配 `tag=smoke`、`risk=readonly`。
- `client-regression`:完整的普通用户旅程,可包含隔离环境中的 `mutating` 操作;不等于所有 full 用例。
- `safety`:删除、权限边界、改密、停用或故障注入等危险旅程;必须显式选择和授权,不进入默认运行。
每条旅程按 `preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告` 执行。登录失败在
客户端和正确测试账号都已就绪时记 `fail`;设备、产物、服务或 session 不可达才记
`environment-fail`。fixture/cleanup 缺失是 contract block,不伪装成逐条结果。前置失败
后的剩余条目是 `not-run`,主动排除仍是 `skipped`。
## 4. 不采用的词
| 词 | 不用它的原因 |
+26 -5
View File
@@ -10,12 +10,31 @@
```yaml
bindings:
tester.test:
tester.android:
adapter: tester
location: local-app
profile: test-agent
tester.ios:
adapter: tester
location: remote-ios
profile: test-agent
```
`tester.android` / `tester.ios` 是推荐的平台绑定名。`builder.android`、`builder.ios`
只负责构建,不能被 tester 自动继承;这条边界用于防止 iOS 构建成功却仍在 Linux
上执行测试。`tester.test` 只能用于明确的单平台或非移动测试;同一轮包含 Android
和 iOS 时必须分别解析两个 tester 绑定。
移动 bundle 入口必须把平台绑定显式传入:
```bash
--binding android=tester.android --binding ios=tester.ios
```
入口会校验 preflight matrix 的每个平台 binding、node 和 runner。远程 iOS 必须是
`contract` runner;多平台运行不会读取旧的单一 `local.yaml.agent` 来决定执行节点,
也不会在协调节点直接执行 Android/iOS 平台命令。
`locations` 和 `profiles` 的定义见共享 `agent-runtime`。tester adapter 只把它 materialize
到本轮兼容配置;每轮 terminal 仍然新建并在结束时关闭,不能因为 binding 是 persistent
就复用测试 terminal。
@@ -54,7 +73,8 @@ terminal。
- Orca terminal id,例如 `term_...`;
- 不提供定位信息,使用当前项目。
有定位信息时优先使用解析脚本,避免手工拼接 Orca JSON:
有定位信息时优先使用解析脚本,避免手工拼接 Orca JSON。多平台运行要对每个
`tester.<platform>` 绑定分别解析,不能只解析一次 `tester.test` 后复用到所有平台:
```bash
python3 <skill-dir>/scripts/resolve_agent.py \
@@ -101,9 +121,10 @@ python3 <skill-dir>/scripts/resolve_agent.py \
之后重新查询并取得 `worktree` selector。项目准备失败要记为
`environment-fail`,不要继续创建 terminal。
这里的“没有项目就创建”指创建/登记 Orca 的项目记录,不是删除或重建用户目录,
也不是创建新的 Git worktree。目标目录必须已经存在且包含待测项目;如果目录本身
不存在,停止并说明需要先在目标机器准备项目。
这里的“没有项目就创建”指创建/登记 Orca 的项目记录,不是删除或重建用户目录。
目标目录必须已经存在且包含待测项目;如果目录本身不存在,停止并说明需要先在
目标机器准备项目。移动端准备阶段会另外为本轮创建隔离 Git worktree,那一步
不替代这里的 Orca 项目登记。
## 配置更新结果
+5 -1
View File
@@ -36,7 +36,11 @@
| `destructive` | 难恢复或伤人/伤钱/伤环境 | 删库、真实扣费 | 清数据、内购 | 急停、断电、真焊 |
默认只跑 `readonly`。项目细分闸(点动 vs 焊接)用可选 `gate`,不要再发明 risk 名。
隔离且可重置的项目可在 catalog 设置 `execution_profile: isolated-full`,此时所有 active
用例按原始 risk 执行;`risk` 仍用于报告和 cleanup 设计,不能改写为 readonly。
`tag=smoke` 且 `risk!=readonly` 为非法。
不想跑某一档时,`authorize.<risk>` 指向的环境变量保持 false 即可——该档用例记 `skipped`,不是 `not-run`。
standard 下不想跑某一档时,`authorize.<risk>` 指向的环境变量保持 false 即可——该档用例记
`skipped`,不是 `not-run`。isolated-full 不使用这些授权开关,改由 preflight 能力探针和
cleanup/rollback 契约阻断不安全环境。
+141 -26
View File
@@ -7,19 +7,76 @@ Agent、执行目标和临时 terminal 的完整协议见 [execution.md](executi
已有 terminal。
测试结束、失败、超时或中断时,都要关闭本轮创建的 terminal。
## 0. 准备执行目标
## 0. 建立测试矩阵并完成全量 preflight
`test-preflight` 是 integration 和 journey 的共同硬闸。tester 必须先调用其 `verify`
或 `prepare`,保存独立的 preflight YAML/CSV;状态为 `blocked` 或 `environment-fail`
时不创建 browser/mobile session,所有尚未执行的 testcase 记 `not-run`。preflight 证据
不算 testcase,也不进入 testcase CSV/YAML 的总数。
测试不是“找到一个能跑的设备就开始”。每轮先建立一份测试矩阵;移动端每个目标产物
必须单独占一行:
```text
平台 | 产物路径/下载地址 | commit | 包名或 Bundle ID | tester 绑定 | 执行环境
Android | … | … | page.writes | tester.android | local-app
iOS | … | … | page.writes | tester.ios | remote-ios
```
矩阵里的产物位置必须由调用方给出,值为本地文件或 URL。不能根据“曾经构建成功过”、
`builder.<platform>` 绑定、CI release 名或本机旧文件猜测本轮产物。缺少路径或下载
地址时,先报告“待确认”并停止。
平台绑定是测试执行目标,不是构建目标:`builder.ios` 只能证明构建配置,不能替代
`tester.ios`;`builder.android` 同理。多平台运行不得使用无法表达平台的通用
`tester.test` 绑定。每一行都必须解析出独立的 tester Agent、Orca 环境、项目和路径。
完成矩阵后,按以下顺序检查**所有行**,但不执行测试用例:
1. 产物:调用方给出的文件或下载地址存在;Android APK 的 package、iOS IPA 的
Bundle ID 与 catalog 契约(若有)一致。
2. 执行目标:按 [execution.md](execution.md) 精确解析每个平台的 `tester` 绑定;
不允许把本机 worktree 当成远程 Mac 的 worktree,也不允许用旧 commit 的远程目录
代替本轮产物对应的项目目录。
3. 平台环境:Android 执行 adb、设备、APK、HOST 检查;iOS 在 macOS 执行
`uname`、`xcrun`/`simctl`、模拟器或真机、App 安装、HOST 检查。
4. 依赖环境:浏览器、权限、网络、测试账号或 catalog 明确要求的其它 session 前置。
所有检查完成后才能进入下方的 `select` 和 session 初始化。任一行不可用时,必须返回
完整清单并停止:
```text
## tester preflight:阻塞
目标产物: Android, iOS
可用: <平台 + 产物 + 环境>
不可用:
- <平台>: <具体检查项和观测结果>
修复: <用户可执行动作>
验证: <可复制命令>
本轮未创建测试 session,未执行测试用例。
完成修复后重跑原 tester 命令。
```
不能因为 Android 可用就先测 Android,也不能用 API、另一平台或旧产物替代不可用行。
先按 `execution.md` 处理目标机器和项目目录,输出配置更新结果,再进入测试流程。目标测试项目必须包含
可读取的 `.pouch/tester/catalog.yaml`。父仓库可能只是包含多个测试子项目;运行前用
`catalog.py roots --root <repo-root>` 发现并选择目标根,不能把父仓库无 catalog 当成失败,
也不能偷偷回退到另一个子项目或复制测试文件。每个目标根单独生成运行快照。
可读取的 `.pouch/tester/catalog.yaml`。父仓库可能包含多个测试子项目;运行前用
`catalog.py roots --root <repo-root>` 发现全部目标根,不能把父仓库无 catalog 当成失败,
也不能偷偷回退到某一个子项目或复制测试文件。所有目标根都要被 fan-out 调度,每个根独立
做产物/环境 preflight,并单独生成运行快照。
`catalog.py select` 输出的 `agent` 是本轮使用的配置快照。记录中要保留 Agent、目标
环境、主机、项目、路径、worktree id、terminal handle,以及 terminal 是否成功关闭。
## 1. 选范围
未指定范围 → `--tag smoke`,风险上限 = `default_max_risk`,`automation` 默认只跑 `automated`。
未指定范围 → `--tag smoke`,执行 profile = catalog 的 `execution_profile`(默认 `standard`),
`standard` 的风险上限 = `default_max_risk`,`automation` 默认只跑 `automated`。
`execution_profile=isolated-full` 只用于隔离可重置环境:不按风险上限或 `AUTHORIZE_*`/gate
授权变量过滤 active 用例,保留每条用例原始 `risk`;安全门禁由 test-preflight 能力探针和
cleanup/rollback 契约承担。
用户说分类、id、tag、风险上限时,原样传给 select:
@@ -27,24 +84,65 @@ Agent、执行目标和临时 terminal 的完整协议见 [execution.md](executi
python3 <skill-dir>/scripts/catalog.py select --root <root> \
[--category CAT] [--id ID]... [--tag smoke|full|slow] \
[--max-risk <risk>] [--driver script|browser|mobile] \
[--automation automated|assisted|manual|any]
[--execution-profile standard|isolated-full] \
[--automation automated|assisted|manual|any] \
[--test-type integration|journey] [--suite <client-or-integration-suite>] \
[--platform android|ios]... \
[--artifact android=<path-or-url>] [--artifact ios=<path-or-url>] \
[--binding android=tester.android] [--binding ios=tester.ios]
```
`--automation` 默认 `automated`(需人工的条目会被跳过,并在 JSON `skipped` 里说明);用户明确要求现场配合时用 `assisted`,`any` 表示不按人工参与度筛。
客户端旅程还可按标准套件选择:
`select` 的 JSON 字段:`max_risk` / `automation` / `count` / `vars`(已展开的变量,用来替换 `how` 里的 `${…}`)/ `skipped` / `run_together` / `status_probe`(按分类,`url` 已展开)/ `cases`。
```bash
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-smoke
```
`select` 已排除 `retired` 和风险高于上限的条目(后者若用户点了具体 id,会在 JSON 的 `skipped` 里说明,不执行)。
`client-smoke`、`client-regression`、`safety` 是 `client_suite` 的固定值,不替换既有
`tag=smoke|full|slow`。`safety` 不参与未指定范围的默认选择。
集成测试显式选择:
```bash
python3 <skill-dir>/scripts/catalog.py select --root <root> --test-type integration
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite api-contract
```
未指定 `--test-type`/`--suite` 时,默认选择不会包含 `test_type=integration`。
选中任一 `driver=mobile` 用例时,`--platform`、每个平台的 `--artifact` 和对应
`tester` binding 是硬性参数;`catalog.py select` 会检查产物文件/下载地址和
`.pouch/agent.yaml` 中的 binding。缺任何一项直接返回非零并打印 `preflight: blocked`,
不能拿到 cases 后绕过闸门开始测试。默认 binding 是 `tester.<platform>`,所以 iOS
会强制检查 `tester.ios`,不会回退到 `builder.ios` 或通用 `tester.test`。
`--automation` 默认 `automated`(需人工的条目会被记为 `not-run`,不会伪造人工步骤);用户明确要求现场配合时用 `assisted`,`any` 表示不按人工参与度筛。
单根 `select` 的 JSON 字段:`max_risk` / `automation` / `count` / `vars`(已展开的变量,用来替换 `how` 里的 `${…}`)/ `artifacts` / `preflight` / `skipped` / `run_together` / `status_probe`(按分类,`url` 已展开)/ `cases`。从父仓库 fan-out 时返回 `mode: fan-out` 和每个根的独立结果,不能把多个根的 cases 拼成一个列表。
`standard` 的 `select` 已排除 `retired` 和风险高于上限的条目(后者若用户点了具体 id,会在 JSON 的 `skipped` 里说明);`isolated-full` 保留所有 active 条目,不因风险或授权变量 skipped。
若 catalog 有 `run_together`,命中组内任一条就把整组按 `ids` 顺序扩进列表(JSON 带 `run_together`)。这不是擅自扩大范围。
无命中:停下说明筛选条件,不要扩大范围。
## 2. 运行前闸
## 2. 双层执行
- `HOST` 未设(环境变量 / `HX_TEST_HOST` / `local.yaml` 都没有):全部停,environment-fail。
- 命中条的 `risk` 在 catalog `authorize` 里有对应变量:变量不是 `true` 则该条 `skipped`。有 `gate` 时还要 `catalog.gates` 对应变量为 `true`;`assume` 不成立同样 `skipped`。即使用户点了 id 也不执行。
- 授权/假设不成立是**主动不跑**(`skipped`),不是 `not-run`:这是我们决定不跑,不是没轮到。
preflight 通过后,先按 catalog 顺序执行 `integration`,再按客户端套件顺序执行
`journey`。集成服务不可达是 `environment-fail`;服务已达但 API/权限/数据契约断言错误
是 `fail`。任何 integration `fail` 都阻断依赖的 journey,并把它们记为 `not-run`;此时
不得创建客户端 session。正确测试账号登录失败发生在客户端已启动后,仍是产品 `fail`。
运行器可复用 `scripts/two_tier.py` 的 `run_integration_first`,它只接收外部 driver
回调,不实现 API、浏览器或移动执行器;mock fixture 只能验证调用顺序和失败传播,不能
冒充真实部署、Mac 或设备成功。
## 3. 运行前闸
- `HOST` 未设(环境变量 / `local.yaml` 都没有):全部停,environment-fail。
- `standard`:命中条的 `risk` 在 catalog `authorize` 里有对应变量时,变量不是 `true` 则该条 `skipped`;有 `gate` 时还要对应变量为 `true`,`assume` 不成立同样 `skipped`。即使用户点了 id 也不执行。
- `isolated-full`:不读取 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM`、`AUTHORIZE_PASSWORD` 等人工开关;相册、照片权限/测试图片和中文拼音能力默认具备,由 journey 用例真实验证,失败记为产品失败。
- standard 中授权/假设不成立是**主动不跑**(`skipped`);isolated-full 不因客户端能力授权变量停止,设备、产物、服务或清理前置失败时才为全部 testcase 写 `not-run` + `blocked_by: [test-preflight]`。
- `status_probe` 是**分类级前置条件**(业界对应 pytest 的 session fixture / Robot 的 Suite Setup)。用 `select` 返回的 `status_probe` 解析生效值,不要自己读 catalog 猜。返回的 `probe.url` 已用 `vars` 展开,直接请求即可;`select` 同批返回 `vars`:
| `categories[].status_probe` | 生效探测 |
@@ -54,10 +152,10 @@ python3 <skill-dir>/scripts/catalog.py select --root <root> \
| mapping | 整体覆盖顶层,不合并(`source: category`)|
- 同一生效探测一次运行只请求一次,结果复用。
- 探测不可达:用它的分类标 environment-fail,其余分类照跑。
- 命中 `stop_if`:把用它的分类标 environment-fail 并记 `observed` 值,其余分类照跑;所有选中分类都被挡住时,整轮结论才是 environment-fail。
- 探测不可达或命中 `stop_if`:在 preflight 阶段记录对应分类/平台的不可用项并停止整轮;
先返回用户处理,不创建后续测试 session。只有所有本轮目标环境通过,才进入用例执行。
## 2.1 移动端平台环境闸门与用户引导
## 3.1 移动端平台环境闸门与用户引导
`driver=mobile` 不是一个可以互换的平台。运行前必须根据 catalog session、用例变量或用户指定,明确本轮是 Android 还是 iOS;无法判断时先停下询问平台,不得猜测。两类平台分别执行以下闸门:
@@ -67,7 +165,7 @@ python3 <skill-dir>/scripts/catalog.py select --root <root> \
1. `adb` 可执行,且版本与当前 SDK 配置一致。
2. `adb devices -l` 至少有一个状态为 `device` 的目标;`unauthorized`、`offline` 或空列表都不能继续。多个目标时必须显式设置 `ANDROID_SERIAL`。
3. 被测 APK 已安装,包名与 catalog 的变量一致;可用 `adb -s "$ANDROID_SERIAL" shell pm path "$PACKAGE"` 验证。
3. 调用方已给出本轮 APK 的本地路径或 URL。由 tester skill 取用或下载该文件,读取 APK 身份,自动执行 `adb install -r`,再用 `adb -s "$ANDROID_SERIAL" shell pm path <package>` 验证。
4. 测试所需的 HOST、端口、局域网/DNS 和权限可从设备访问。
### iOS
@@ -76,10 +174,11 @@ python3 <skill-dir>/scripts/catalog.py select --root <root> \
1. 运行节点是 macOS,并且 `xcrun`、Xcode Command Line Tools 和 `simctl` 可用。Linux/Windows 节点不能伪装成 iOS 环境;需要 macOS 或可访问的远程 Mac。
2. 模拟器用 `xcrun simctl list devices available` 确认目标存在且已启动,再用 `xcrun simctl bootstatus <UDID> -b` 确认完成启动;真机用 `xcrun xctrace list devices` 确认已连接、已信任且允许开发。
3. 被测 IPA/App 已安装,Bundle ID 与 catalog 的变量一致;模拟器可用 `xcrun simctl get_app_container <UDID> <BUNDLE_ID>` 验证。
3. 调用方已给出本轮 IPA 的本地路径或 URL。由 tester skill 在 tester.ios 的 Mac 取用或下载该文件,解包读取 `Payload/*.app/Info.plist`,校验 Bundle ID,自动执行 `xcrun simctl install`,再用 `xcrun simctl get_app_container <UDID> <BUNDLE_ID>` 验证。
4. 测试所需的 HOST、端口、局域网/DNS 和权限可从该模拟器/真机访问。
任一项不满足,都要在执行记录中把受影响的移动用例标为 `environment-fail`,不能改成 API 测试、不能把它们标成产品 `fail`,也不要继续初始化后续移动 session。记录实际探测命令及简短输出,不要把整段设备日志塞进对话。
Android+iOS 默认是全量 preflight 闸门:任一项不满足,另一平台也不得安装或执行,并在记录中标为 `not-run`。只有用户显式加 `--partial` 才允许已通过的平台继续。不能改成 API 测试、不能把它们标成产品 `fail`,也不要继续初始化被闸门挡住的平台 session。记录实际探测命令及简短输出,不要把整段设备日志塞进对话。
任一项不满足,都要在 preflight 汇总中列出平台、具体检查项、实际探测命令和简短输出,然后停止整轮;只有 preflight 全部通过后,才把对应移动用例的结果写入运行快照。
### 环境缺失时的用户引导
@@ -105,7 +204,7 @@ python3 <skill-dir>/scripts/catalog.py select --root <root> \
可按实际缺失替换为以下平台化提示:
- Android 无设备:提示用户连接并解锁设备、开启 USB 调试、接受 RSA 授权;多个设备时设置 `ANDROID_SERIAL`;再运行 `adb devices -l`,直到目标状态为 `device`。
- Android 未安装 APK:提示用户构建/安装与当前测试环境匹配的 APK,并用 `adb -s "$ANDROID_SERIAL" shell pm path "$PACKAGE"` 验证包名。
- Android 缺少产物位置:停下向调用方要本轮 APK 的本地路径或 URL,不要手工 scp 或沿用旧包。
- iOS 非 macOS 或无 Xcode:明确说明 iOS 测试需要 macOS + Xcode/Command Line Tools,提示用户切换到本机 Mac 或配置可访问的远程 Mac;不要建议在 Linux 上安装一个同名命令来绕过检查。
- iOS 模拟器未启动:提示用户在 Xcode 或 `simctl` 中创建/启动可用设备,使用 `xcrun simctl bootstatus <UDID> -b` 验证。
- iOS 真机未就绪:提示用户连接数据线、解锁并信任此电脑、开启开发者模式并确认设备出现在 `xcrun xctrace list devices`。
@@ -117,7 +216,7 @@ python3 <skill-dir>/scripts/catalog.py select --root <root> \
环境引导只负责让用户补齐前置条件;除 catalog 已明确授权的 session 初始化外,不自动安装 SDK、配对真机、修改系统安全设置或上传产物。
## 3. Session
## 4. Session
看命中条用到的 `session` 键:
@@ -126,9 +225,9 @@ python3 <skill-dir>/scripts/catalog.py select --root <root> \
- `reuse: run`:整轮结束再关。`persist`:不关。
- 中途 session 挂了:后续同 driver 标 environment-fail,不要标产品失败。
## 4. 执行
## 5. 执行
按 select 返回的顺序,一条一条来。select 按 `catalog.yaml` 的 `categories` 顺序,同一分类内按该 yaml 列表顺序(不是文件名字母序)。默认失败即停(产品 `fail` 或探测到的不安全状态)。用户说继续 / keep-going 才往后跑。
按 select 返回的顺序,一条一条来。select 按 `catalog.yaml` 的 `categories` 顺序,同一分类内按该 yaml 列表顺序(不是文件名字母序)。同一 phase 的独立用例继续执行;只有显式 `depends_on` 的失败依赖才写 `not-run` + `blocked_by`。用户说 targeted rerun 时只重跑受影响用例,修复后再 full integration → journey。
例外:当前条和下一条同属 `run_together` 且该组 `continue_on_fail: true` 时,前一条产品 `fail` 仍跑后一条。`environment-fail` 仍整轮停。
@@ -158,9 +257,9 @@ python3 <skill-dir>/scripts/catalog.py run-script --root <root> --id <id>
挂到已有 session,执行 `how.steps`,对照 `expected`。必须留下可见文本、URL 或交互证据。
## 5. 记录与报告
## 6. 记录与报告
每条:`id, result, evidence, notes`。`result` 只能是 `pass` / `fail` / `environment-fail` / `skipped` / `not-run`。`skipped` 是主动不跑(retired / pending / 超风险上限 / 未授权 / `assume` 不成立),`not-run` 是没轮到(失败即停后剩下的 / 被前置整组挡住的 / 需人工而无人值守的)。
每条:`id, result, evidence, notes`,依赖阻断时另有 `blocked_by`。`result` 只能是 `pass` / `fail` / `environment-fail` / `skipped` / `not-run`。`skipped` 是 standard 中主动不跑(retired / pending / 超风险上限 / 未授权 / `assume` 不成立);`not-run` 是没轮到或 preflight/依赖阻断。失败用例可在 YAML 顶层 `defects`(兼容 `test_issues`)记录 testcase、`issue_type=product|test`、expected/observed、artifact hash、environment、evidence path、reproduction、next step。
写入 `.pouch/tester/runs/<UTC时间>-<范围>-<随机后缀>.yaml`,可用
`generate_run_csv.new_run_id(<范围>)` 生成同名 `runId`。记录包含唯一 `runId`、筛选条件、HOST 是否已设(可以写 HOST 值)、session 是否复用、每个分类的 `status_probe` 生效值与观测结果(`source` / `disabled` 原因 / `hit` / `observed`)、逐条结果。记录文件只创建一次,不覆盖已有文件。立刻用同一快照生成 CSV,不要手改:
@@ -169,7 +268,23 @@ python3 <skill-dir>/scripts/catalog.py run-script --root <root> --id <id>
python3 <skill-dir>/scripts/generate_run_csv.py --root <root> --run <yaml>
```
这会写出与快照同名的 `.pouch/tester/runs/<run-id>.csv`(`executed=yes|no`),不会修改 `cases.csv`。对话里两个路径都报。
这会写出与快照同名的 `.pouch/tester/runs/<run-id>.csv`(`executed=yes|no`),不会修改
`cases.csv`。对话里两个路径都报。
需要复核已有成绩单时,显式同时传入同一次运行的 CSV 和 YAML:
```bash
python3 <skill-dir>/scripts/summarize_run.py \
--csv .pouch/tester/runs/<run-id>.csv \
--yaml .pouch/tester/runs/<run-id>.yaml
```
汇总状态只读取 CSV `result` 列,必须输出五类 `pass / fail / environment-fail /
skipped / not-run`,且计数和等于 CSV 数据行数。汇总前会检查 CSV 数据行数、CSV 唯一
testcase id 数、YAML `results` 数量和 CSV/YAML testcase id 集合;重复或缺失 id、缺失或
未知 CSV 状态、任一数量或集合不一致都会非零退出并给出诊断。YAML 的 `counts` 或其它
历史汇总字段不会参与状态计数。退出码 0 表示汇总校验成功,非零表示报告不一致或输入
无效,不等同于产品测试失败。
对话报告:
+92 -1
View File
@@ -14,6 +14,10 @@ runs/<run-id>.yaml 每次运行快照(执行源);run-id 必须唯一
runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对应 yaml 生成
```
两类用例共用同一套 cases/runs 和五态报告,不另造成绩单。新目录显式声明顶层
`test_type: journey | integration`;旧目录没有该字段时按已有 `client_suite`/`journey`
或 `integration_suite` 推断。`category` 仍是文件/执行顺序容器,不被套件字段替代。
## catalog.yaml
| 字段 | 含义 |
@@ -21,17 +25,25 @@ runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对
| `categories` | `{id, title, status_probe?}` 列表,id 与 `cases/<id小写>.yaml` 对应;列表顺序即 `select` / 执行顺序。`status_probe` 见下 |
| `risks` | 从低到高,只能是 `readonly` → `mutating` → `destructive` 的一段,且以 `readonly` 开头。含义见 [levels.md](levels.md) |
| `default_max_risk` | 未指定时的运行上限,必须是 `risks` 里的值 |
| `execution_profile` | `standard`(默认)或 `isolated-full`。后者只用于隔离可重置环境,保留 risk 语义但不按 risk/authorize/gate 过滤 active 用例;设备、服务和 cleanup/rollback 契约仍由 preflight 检查,客户端能力由 journey 真实验证 |
| `vars` | 可替换变量,如 `API_BASE: http://${HOST}:8095` |
| `artifacts` | 可选的移动产物身份契约。按 `android` / `ios` 声明 `required` 和 `package` / `bundle_id`;每轮文件路径或下载地址由调用方提供,不提交临时产物路径。 |
| `authorize` | 风险名 → 环境变量名。该档执行前变量必须为 `true`(`readonly` 不需要) |
| `gates` | 可选。项目细分闸名 → 环境变量。用在用例的 `gate` 上,不要拿它代替 `risk` |
| `sessions` | 可复用会话。`kind`: `browser` / `mobile`;`reuse`: `run`(本轮结束关掉)或 `persist`(probe 活着就附着) |
| `id_pattern` | 用例 id 正则 |
| `run_together` | 可选。成组执行:`[{ids: [TC-…, TC-…], continue_on_fail: true}]`。点到组内任一条,`select` 会按 `ids` 顺序把整组扩进命中列表。`continue_on_fail: true` 时组内前一条产品 fail 仍跑后一条(返回/收尾)。一个 id 只能属于一组。 |
| `status_probe` | 可选。顶层默认值,一个**分类级前置条件**:`{url, stop_if: {字段: [命中值]}}`。运行前请求 `url`,响应命中任一 `stop_if` 条件即视为前置不满足。分类项可用同名键覆盖:缺省继承顶层、`false` 关闭、mapping 整体替换。命中或不可达只影响用它的分类,不是产品失败。业界对应 pytest 的 session fixture / Robot 的 Suite Setup。`select` 会按分类返回解析后的生效值,`url` 已用 `vars` 展开 |
| `client_suites` | 可选的标准套件声明;出现时必须完整包含 `client-smoke`、`client-regression`、`safety`。缺省运行仍按 `tag` 兼容旧目录 |
| `integration_suites` | 可选的集成套件声明;标准值为 `api-contract`、`authorization`、`build-contract` |
`isolated-full` 不得把风险字段改成 `readonly`,也不得把 `AUTHORIZE_ALBUM`、
`AUTHORIZE_PASSWORD`、`IME_PINYIN` 等变量当成门禁。移动用例必须通过 test-preflight
的设备、产物、服务和网络检查;相册、权限和输入法能力默认具备,交由 journey 用例验证。
`local.yaml` 与环境变量覆盖 `HOST` 和授权开关。模板中的 `HOST` 默认是
`127.0.0.1`;远程目标必须显式覆盖。查找顺序:环境变量 > `local.yaml` >
`catalog.yaml` 的 vars。也认 `HX_TEST_HOST` 作为 `HOST`。
`catalog.yaml` 的 vars。
## agent
@@ -69,6 +81,7 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
| 字段 | 必填 | 含义 |
|------|------|------|
| `id` | 是 | 匹配 `id_pattern` |
| `test_type` | 新用例是 | `journey` 或 `integration`;缺省兼容旧目录并按字段推断 |
| `category` | 是 | 必须等于文件声明的 category |
| `title` | 是 | 短名 |
| `what` | 是 | 一句可观察的「测什么」 |
@@ -83,10 +96,67 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
| `expected` | 是 | 怎么过,至少一条 `{kind, value}` |
| `maps_to` | 否 | 对人文档的功能项 id |
| `assume` | 否 | 额外跳过条件(环境变量表达式)。不要重复 `risk` / `gate` 已覆盖的授权变量,否则 `validate` 报错 |
| `depends_on` | 否 | 依赖的 testcase id 列表;依赖 fail/environment-fail/not-run 时本条写 `not-run` 和 `blocked_by`,独立用例继续 |
| `setup` | 否 | 人话前置 |
| `teardown` | 否 | 跑完回退 |
| `notes` | 否 | 备注 |
### 集成测试用例契约
`test_type: integration` 的用例不创建 browser/mobile session,也不能声明 `platform` 或
客户端 `journey`。必须有 `integration_suite`(迁移期可用 `suite`)且值为
`api-contract`、`authorization` 或 `build-contract`;`fixture` 必须是非空 mapping;
`request`、`input`、`call`、`integration` 之一或旧式 `how` 用于描述调用;`expected` 是
协议/数据断言。`mutating`/`destructive` 集成用例必须提供 `cleanup.steps` 或
`cleanup.run`(只读探针可用 `required: false` + `reason`)。
选择 `--suite api-contract` 或 `--test-type integration` 才会进入集成阶段;无显式类型
时的默认选择不会包含 integration。
### 客户端旅程用例契约
声明 `client_suite` 后,以下字段都必须存在;`catalog.py validate` 和 `select` 使用同一
套检查。契约缺失是 `preflight: blocked`,不生成逐条产品失败。
| 字段 | 必填 | 约束 |
|------|------|------|
| `client_suite` | 是 | `client-smoke`、`client-regression` 或 `safety`;安全套件必须显式选择,不属于默认 smoke/full |
| `journey.goal` | 是 | 用户目标,不写接口目标 |
| `journey.entry_state` | 是 | 启动/登录前的可观察入口状态 |
| `journey.actions` | 是 | 非空用户操作列表;真实 App/browser 驱动从这里开始 |
| `journey.assertions` | 是 | 非空可观察 UI 断言列表;最终业务状态可另由 `expected` 的 api-field/api-status 佐证 |
| `journey.final_state` | 是 | 旅程完成后的业务状态 |
| `journey.failure_policy` | 是 | `stop` 或 `continue`;默认不隐式吞掉前置失败 |
| `fixture` | 是 | mapping,至少有 `account`;账号和数据由 app catalog 提供,skill 不写具体账号 |
| `session` | 是 | 复用的 browser/mobile session 名,必须在 catalog.sessions 声明 |
| `cleanup` | 是 | mapping,必须有非空 `steps`/`run`;只读旅程也要用 `required: false` + `reason` 显式说明无需清理 |
| `platform` | 是 | `android`/`ios` 的字符串或列表,不能重复;平台差异只能进入执行适配层 |
一个最小形状如下(值是占位符,不是 app 规则):
```yaml
client_suite: client-smoke
platform: [android, ios]
journey:
goal: "用户完成一个可观察的客户端目标"
entry_state: logged_out
actions: ["启动客户端", "输入测试账号并登录", "完成目标操作"]
assertions: ["看到目标页面", "看到成功提示"]
final_state: "目标状态已保存"
failure_policy: stop
fixture:
account: test-role
data: []
session: mobile
cleanup:
steps: ["删除本轮 run-id 创建的数据"]
```
账号登录属于 session 初始化:正确测试账号登录失败是产品 `fail`;设备、产物或服务
不可达才是 `environment-fail`。fixture/cleanup 缺失必须在启动、登录和任何旅程操作前
阻断。旅程失败后,尚未轮到的用例记 `not-run`;风险、生命周期或明确范围排除记
`skipped`。清理失败单独写入运行快照的 cleanup 证据,不改造为一个新的 result 值。
## how
按 `driver` 分派;`automation=manual` 时没有 `driver`,只有 `how.steps`(人工步骤)。
@@ -106,3 +176,24 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
## 判定
`pass` / `fail` / `environment-fail` / `skipped` / `not-run`
## 运行失败回流
run YAML 的 `results` 必须保留每条 testcase 行。失败可在顶层 `defects`(兼容别名
`test_issues`)追加:
```yaml
defects:
- testcase: TC-CLIENT-001
issue_type: product # product | test
expected: "候选词可选择"
observed: "composition 成功但候选列表为空"
artifact_hash: sha256:...
environment: "Android emulator API 35 / tester.android"
evidence_path: .pouch/tester/evidence/TC-CLIENT-001.png
reproduction: "打开相册后输入 nihao"
next_step: "开发修复候选词渲染后 targeted rerun"
```
PR 触发的 run 可带 `feedback` 计划:先更新 PR/关联 issue,再通知 dispatcher 重新分配开发;
需求/bug 触发则 route 直接为 development。tester 只记录计划,不隐式调用远端写操作。
+25
View File
@@ -0,0 +1,25 @@
# 移动产物准备
调用方必须给出每个目标平台的产物位置:本地文件或可下载 URL。tester 不从
CI release、builder 绑定或本机旧文件猜测产物。
```bash
<skill-dir>/tester run \
--artifact android=<path-or-url> \
--artifact ios=<path-or-url> \
--platform android,ios \
--binding android=tester.android \
--binding ios=tester.ios
```
URL 就下载,本地文件就直接用。每个平台都通过自己的 `tester.<platform>` binding
校验、探测和安装;`agent-runtime` 负责把 Android 请求送到 Android 节点、把 iOS
请求送到 Mac 节点。旧的单一 `local.yaml.agent` 只能作为兼容配置,不能作为双平台
运行目标。随后由现有 tester catalog 驱动真实移动 session。默认先完成 Android+iOS 双平台
preflight;任一平台无法准备时整轮停止。只有显式加 `--partial` 才允许运行可用平台。
每轮另外创建隔离 Git worktree,不切用户当前分支。`--commit` 缺省为当前 HEAD。
运行记录写入目标项目 `.pouch/tester/runs/`,包含产物位置、sha256、身份、实际
worktree commit、节点、安装证据和 terminal 生命周期;`automatic-repair-failed`
与产品测试失败分开记录。
+416 -15
View File
@@ -4,15 +4,30 @@
from __future__ import annotations
import argparse
import contextlib
import io
import json
import os
import re
import subprocess
import sys
import urllib.parse
import urllib.request
import zipfile
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from journey import (
CLIENT_SUITES,
INTEGRATION_SUITES,
TEST_TYPES,
contract_errors,
preflight as journey_preflight,
suite as case_suite,
test_type,
)
try:
import yaml
except ImportError: # pragma: no cover
@@ -27,6 +42,7 @@ LIFECYCLE_SKIP_REASON = {
}
TAGS = ("smoke", "full", "slow")
CANONICAL_RISKS = ("readonly", "mutating", "destructive")
EXECUTION_PROFILES = ("standard", "isolated-full")
AUTOMATIONS = ("automated", "assisted", "manual")
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
EXPECTED_KINDS = (
@@ -59,6 +75,60 @@ def catalog_path(root: Path) -> Path:
return catalog_dir(root) / "catalog.yaml"
def discover_or_self(root: Path) -> list[Path]:
"""Return all independent tester projects below a parent repository."""
root = root.resolve()
return discover_catalog_roots(root)
def cmd_fanout(args: argparse.Namespace, roots: list[Path]) -> int:
"""Run a catalog command independently for every discovered tester root."""
results: list[dict[str, Any]] = []
for child_root in roots:
output = io.StringIO()
with contextlib.redirect_stdout(output):
if args.cmd == "list":
exit_code = cmd_list(child_root)
elif args.cmd == "select":
exit_code = cmd_select(args, child_root)
elif args.cmd == "show":
exit_code = cmd_show(child_root, args.case_id)
elif args.cmd == "validate":
exit_code = cmd_validate(child_root)
elif args.cmd == "export-csv":
exit_code = cmd_export_csv(child_root)
elif args.cmd == "run-script":
exit_code = cmd_run_script(child_root, args.case_id)
else:
exit_code = 2
print(f"不支持从父仓库 fan-out 命令: {args.cmd}")
raw = output.getvalue().strip()
try:
result: Any = json.loads(raw) if raw else None
except json.JSONDecodeError:
result = raw
results.append({
"root": str(child_root),
"exit": exit_code,
"result": result,
})
if args.cmd == "select":
print(json.dumps({"mode": "fan-out", "roots": results}, ensure_ascii=False, indent=2))
else:
for item in results:
print(f"\n## tester root: {item['root']} (exit={item['exit']})")
result = item["result"]
if isinstance(result, (dict, list)):
print(json.dumps(result, ensure_ascii=False, indent=2))
elif result:
print(result)
return 0 if all(item["exit"] == 0 for item in results) else 2
FANOUT_COMMANDS = {"list", "select", "show", "validate", "export-csv", "run-script"}
def discover_catalog_roots(root: Path) -> list[Path]:
"""Find tester projects owned by a repository and its Git submodules.
@@ -249,17 +319,12 @@ def resolve_vars(root: Path, meta: dict[str, Any]) -> dict[str, str]:
for key, value in os.environ.items():
if key in merged or key in auth_keys or key.startswith("AUTHORIZE_") or key in {
"HOST",
"HX_TEST_HOST",
"WEB_BASE",
"API_BASE",
"WS_URL",
"STATUS_BASE",
}:
merged[key] = value
if not merged.get("HOST") and merged.get("HX_TEST_HOST"):
merged["HOST"] = merged["HX_TEST_HOST"]
if os.environ.get("HX_TEST_HOST") and not os.environ.get("HOST"):
merged["HOST"] = os.environ["HX_TEST_HOST"]
for _ in range(10):
changed = False
for key, value in list(merged.items()):
@@ -287,6 +352,21 @@ def risk_index(meta: dict[str, Any], name: str) -> int:
return risks.index(name)
def resolve_execution_profile(meta: dict[str, Any], requested: str | None = None) -> str:
"""Resolve the run profile without changing the meaning of ``risk``.
``isolated-full`` is an execution policy for a resettable test environment:
it disables risk/authorization filtering, but leaves the catalog risk value
on every case so reports still describe the side effect that was exercised.
"""
profile = requested or meta.get("execution_profile") or "standard"
if profile not in EXECUTION_PROFILES:
raise SystemExit(
f"未知 execution_profile: {profile!r}(可选: {', '.join(EXECUTION_PROFILES)})"
)
return str(profile)
def host_set(vars_map: dict[str, str]) -> bool:
host = (vars_map.get("HOST") or "").strip()
return bool(host) and "${HOST}" not in host
@@ -314,7 +394,7 @@ def cmd_list(root: Path) -> int:
pending = [c for c in cases if c.get("lifecycle") == "pending"]
print(f"## 测试目录 active {len(active)} / pending {len(pending)} / retired {len(retired)}")
print()
headers = ["分类", "标题", "条数", "smoke"] + risks + list(DRIVERS)
headers = ["分类", "标题", "条数", "journey", "integration", "smoke"] + risks + list(DRIVERS)
print("| " + " | ".join(headers) + " |")
print("| " + " | ".join("---" for _ in headers) + " |")
@@ -332,7 +412,14 @@ def cmd_list(root: Path) -> int:
title = (cats_meta.get(cat) or {}).get("title") or (
group[0].get("_file_title") if group else ""
)
row = [cat, str(title), str(len(group)), str(sum(1 for c in group if c.get("tag") == "smoke"))]
row = [
cat,
str(title),
str(len(group)),
str(sum(1 for c in group if test_type(c) == "journey")),
str(sum(1 for c in group if test_type(c) == "integration")),
str(sum(1 for c in group if c.get("tag") == "smoke")),
]
for risk in risks:
row.append(str(sum(1 for c in group if c.get("risk") == risk)))
for driver in DRIVERS:
@@ -499,6 +586,7 @@ def expand_run_together(
skipped: list[dict[str, str]],
max_i: int,
automation_filter: str = "automated",
bypass_risk: bool = False,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
groups = run_together_groups(meta)
if not groups:
@@ -536,7 +624,7 @@ def expand_run_together(
except SystemExit:
skipped.append({"id": cid, "reason": f"未知风险 {risk}"})
continue
if r_i > max_i:
if not bypass_risk and r_i > max_i:
skipped.append({"id": cid, "reason": f"risk={risk} 超过上限"})
continue
members.append(public_case(case))
@@ -560,12 +648,150 @@ def expand_run_together(
return selected, hit_groups
def parse_platform_pairs(values: list[str], label: str) -> tuple[dict[str, str], list[str]]:
pairs: dict[str, str] = {}
errors: list[str] = []
for raw in values:
platform, sep, value = raw.partition("=")
platform = platform.strip().lower()
value = value.strip()
if not sep or platform not in ("android", "ios") or not value:
errors.append(f"{label} 格式必须是 android=<值> 或 ios=<值>: {raw}")
continue
if platform in pairs:
errors.append(f"{label} 重复平台: {platform}")
continue
pairs[platform] = value
return pairs, errors
def safe_locator(locator: str) -> str:
parsed = urllib.parse.urlsplit(locator)
if parsed.scheme in ("http", "https"):
path = parsed.path or "/"
return f"{parsed.scheme}://{parsed.netloc}{path}"
return locator
def artifact_ready(locator: str) -> tuple[bool, str]:
parsed = urllib.parse.urlsplit(locator)
if parsed.scheme in ("http", "https"):
try:
request = urllib.request.Request(locator, method="HEAD")
with urllib.request.urlopen(request, timeout=10) as response:
if 200 <= response.status < 400:
return True, f"HTTP {response.status}"
return False, f"HTTP {response.status}"
except Exception as exc:
return False, f"下载地址不可达: {type(exc).__name__}: {exc}"
path = Path(locator).expanduser()
if not path.is_file():
return False, "文件不存在"
if path.stat().st_size <= 0:
return False, "文件为空"
if path.suffix.lower() in (".apk", ".ipa") and not zipfile.is_zipfile(path):
return False, "不是有效的 APK/IPA zip 产物"
return True, f"文件 {path.stat().st_size} bytes"
def runtime_binding(root: Path, binding_id: str) -> tuple[bool, str]:
config_path = root / ".pouch" / "agent.yaml"
if not config_path.is_file():
return False, f"缺少 {config_path}"
try:
config = load_yaml(config_path) or {}
except Exception as exc:
return False, f"读取 agent.yaml 失败: {exc}"
binding = (config.get("bindings") or {}).get(binding_id)
if not isinstance(binding, dict):
return False, "binding 不存在"
if binding.get("adapter") != "tester":
return False, f"adapter={binding.get('adapter')!r},不是 tester"
location = binding.get("location")
if not location:
return False, "未配置 location"
return True, f"location={location}"
def mobile_preflight(
root: Path,
meta: dict[str, Any],
selected: list[dict[str, Any]],
platforms_raw: list[str],
artifacts_raw: list[str],
bindings_raw: list[str],
) -> dict[str, Any]:
mobile_selected = [case for case in selected if case.get("driver") == "mobile"]
if not mobile_selected:
return {"required": False, "status": "not-needed", "checks": [], "errors": []}
platforms = [item.strip().lower() for item in platforms_raw if item.strip()]
errors: list[str] = []
if not platforms:
errors.append("选中了 mobile 用例,必须显式提供 --platform android/ios")
if len(platforms) != len(set(platforms)):
errors.append("--platform 不能重复")
invalid_platforms = [item for item in platforms if item not in ("android", "ios")]
if invalid_platforms:
errors.append("非法平台: " + ", ".join(invalid_platforms))
artifacts, pair_errors = parse_platform_pairs(artifacts_raw, "--artifact")
errors.extend(pair_errors)
bindings, pair_errors = parse_platform_pairs(bindings_raw, "--binding")
errors.extend(pair_errors)
contract = meta.get("artifacts")
if not isinstance(contract, dict) or not contract:
errors.append("catalog 缺少 artifacts 产物契约")
checks: list[dict[str, Any]] = []
for platform in platforms:
if platform not in ("android", "ios"):
continue
spec = (contract or {}).get(platform) if isinstance(contract, dict) else None
if not isinstance(spec, dict):
errors.append(f"{platform}: catalog.artifacts 未声明")
if platform not in artifacts:
errors.append(f"{platform}: 缺少 --artifact {platform}=<路径或下载地址>")
else:
ok, detail = artifact_ready(artifacts[platform])
checks.append({
"platform": platform,
"artifact": safe_locator(artifacts[platform]),
"artifact_ok": ok,
"artifact_detail": detail,
})
if not ok:
errors.append(f"{platform}: 产物不可用: {detail}")
binding_id = bindings.get(platform, f"tester.{platform}")
ok, detail = runtime_binding(root, binding_id)
checks.append({
"platform": platform,
"binding": binding_id,
"binding_ok": ok,
"binding_detail": detail,
})
if not ok:
errors.append(f"{platform}: binding {binding_id} 不可用: {detail}")
return {
"required": True,
"status": "blocked" if errors else "ready",
"platforms": platforms,
"checks": checks,
"errors": errors,
}
def cmd_select(args: argparse.Namespace, root: Path) -> int:
meta = load_catalog_meta(root)
cases = load_all_cases(root)
execution_profile = resolve_execution_profile(meta, args.execution_profile)
bypass_risk = execution_profile == "isolated-full"
max_risk = args.max_risk or meta.get("default_max_risk") or "readonly"
max_i = risk_index(meta, max_risk)
automation_filter = args.automation or "automated"
automation_filter = args.automation or ("any" if bypass_risk else "automated")
wanted = set(parse_ids(args.ids or []))
selected: list[dict[str, Any]] = []
skipped: list[dict[str, str]] = []
@@ -591,6 +817,20 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
continue
if args.tag and case.get("tag") != args.tag:
continue
if args.test_type and test_type(case) != args.test_type:
continue
if args.suite and case_suite(case) != args.suite:
continue
# A mixed catalog keeps legacy cases runnable. Standard keeps the
# historical client-only implicit default; isolated-full means
# exactly what it says and includes every active test type.
if (
not bypass_risk
and not args.test_type
and not args.suite
and test_type(case) == "integration"
):
continue
if args.driver and case.get("driver") != args.driver:
continue
automation = str(case.get("automation") or "automated")
@@ -606,7 +846,7 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
except SystemExit:
skipped.append({"id": cid, "reason": f"未知风险 {risk}"})
continue
if r_i > max_i:
if not bypass_risk and r_i > max_i:
skipped.append({"id": cid, "reason": f"risk={risk} 超过上限 {max_risk}"})
continue
selected.append(public_case(case))
@@ -618,7 +858,34 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
)
return 2
selected, together = expand_run_together(meta, cases, selected, skipped, max_i, automation_filter)
selected, together = expand_run_together(
meta,
cases,
selected,
skipped,
max_i,
automation_filter,
bypass_risk=bypass_risk,
)
# The historical key is kept in the JSON, but the gate now validates both
# test types before any caller can start a phase.
journey_gate = journey_preflight(selected, sessions=meta.get("sessions") or {})
if journey_gate["status"] == "blocked":
print(json.dumps({"preflight": journey_gate, "journey_preflight": journey_gate}, ensure_ascii=False, indent=2))
return 2
preflight = mobile_preflight(
root,
meta,
selected,
args.platform or [],
args.artifacts or [],
args.bindings or [],
)
if preflight["status"] == "blocked":
print(json.dumps({"preflight": preflight}, ensure_ascii=False, indent=2))
return 2
# 按分类解析生效的 status_probe,让 Agent 不再自己按 catalog 猜作用范围。
vars_map = resolve_vars(root, meta)
@@ -633,9 +900,18 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
json.dumps(
{
"max_risk": max_risk,
"execution_profile": execution_profile,
"risk_filter": "disabled" if bypass_risk else "enforced",
"automation": automation_filter,
"count": len(selected),
"vars": vars_map,
"artifacts": meta.get("artifacts") or {},
"preflight": preflight,
"journey_preflight": journey_gate,
"test_types": {
"journey": sum(1 for case in selected if test_type(case) == "journey"),
"integration": sum(1 for case in selected if test_type(case) == "integration"),
},
"skipped": skipped,
"run_together": together,
"status_probe": status_probes,
@@ -706,6 +982,12 @@ def validate(root: Path) -> list[str]:
default_max = meta.get("default_max_risk")
if default_max not in risks:
errors.append(f"default_max_risk={default_max!r} 不在 risks 中")
profile = meta.get("execution_profile", "standard")
if profile not in EXECUTION_PROFILES:
errors.append(
f"execution_profile={profile!r} 非法,只能是 {', '.join(EXECUTION_PROFILES)}"
)
isolated_full = profile == "isolated-full"
gates = meta.get("gates") or {}
if gates and not isinstance(gates, dict):
errors.append("catalog.gates 必须是 mapping")
@@ -718,6 +1000,50 @@ def validate(root: Path) -> list[str]:
if not isinstance(sessions, dict):
errors.append("catalog.sessions 必须是 mapping")
sessions = {}
client_suites = meta.get("client_suites")
if client_suites is not None:
if not isinstance(client_suites, dict):
errors.append("catalog.client_suites 必须是 mapping")
else:
unknown = sorted(set(client_suites) - set(CLIENT_SUITES))
if unknown:
errors.append("catalog.client_suites 含非法套件: " + ", ".join(unknown))
missing_suites = sorted(set(CLIENT_SUITES) - set(client_suites))
if missing_suites:
errors.append("catalog.client_suites 缺少标准套件: " + ", ".join(missing_suites))
integration_suites = meta.get("integration_suites")
if integration_suites is not None:
if not isinstance(integration_suites, dict):
errors.append("catalog.integration_suites 必须是 mapping")
else:
unknown = sorted(set(integration_suites) - set(INTEGRATION_SUITES))
if unknown:
errors.append("catalog.integration_suites 含非法套件: " + ", ".join(unknown))
missing_suites = sorted(set(INTEGRATION_SUITES) - set(integration_suites))
if missing_suites:
errors.append("catalog.integration_suites 缺少标准套件: " + ", ".join(missing_suites))
artifacts = meta.get("artifacts")
if artifacts is not None:
if not isinstance(artifacts, dict):
errors.append("catalog.artifacts 必须是 mapping")
else:
for platform, spec in artifacts.items():
where = f"catalog.artifacts.{platform}"
if platform not in ("android", "ios"):
errors.append(f"{where}: 平台只能是 android/ios")
if not isinstance(spec, dict):
errors.append(f"{where}: 必须是 mapping")
continue
if "required" in spec and not isinstance(spec["required"], bool):
errors.append(f"{where}.required 必须是 boolean")
if platform == "android" and not spec.get("package"):
errors.append(f"{where}: Android 需要 package")
if platform == "ios" and not spec.get("bundle_id"):
errors.append(f"{where}: iOS 需要 bundle_id")
if platform == "android" and spec.get("bundle_id"):
errors.append(f"{where}: Android 不应填 bundle_id")
if platform == "ios" and spec.get("package"):
errors.append(f"{where}: iOS 不应填 package")
pattern = meta.get("id_pattern") or r"^.+$"
try:
id_re = re.compile(pattern)
@@ -772,6 +1098,7 @@ def validate(root: Path) -> list[str]:
errors.append(f"{where}: category={cat} 与文件 category={file_cat} 不一致")
if cat and cat_ids and cat not in cat_ids:
errors.append(f"{where}: category={cat} 不在 catalog.categories")
errors.extend(contract_errors(case, sessions=sessions))
if case.get("lifecycle", "active") not in LIFECYCLES:
errors.append(f"{where}: 非法 lifecycle")
if case.get("tag") not in TAGS:
@@ -787,8 +1114,17 @@ def validate(root: Path) -> list[str]:
if case.get("risk") in (None, "readonly"):
errors.append(f"{where}: risk=readonly 不要填 gate")
risk = case.get("risk")
if risk and risk != "readonly" and risk not in authorize and not gate:
if risk and risk != "readonly" and risk not in authorize and not gate and not isolated_full:
errors.append(f"{where}: {risk} 需要 authorize.{risk} 或 gate")
if isolated_full and case.get("lifecycle", "active") == "active" and risk != "readonly":
cleanup = case.get("cleanup")
cleanup_ok = isinstance(cleanup, dict) and (
bool(cleanup.get("steps"))
or bool(str(cleanup.get("run") or "").strip())
or (cleanup.get("required") is False and bool(str(cleanup.get("reason") or "").strip()))
)
if not cleanup_ok:
errors.append(f"{where}: isolated-full 的写入型用例必须声明 cleanup/rollback steps/run")
automation = str(case.get("automation") or "automated")
if automation not in AUTOMATIONS:
errors.append(f"{where}: 非法 automation={case.get('automation')}")
@@ -920,7 +1256,19 @@ def cmd_next_id(root: Path, category: str) -> int:
return 0
def authorized(vars_map: dict[str, str], meta: dict[str, Any], case: dict[str, Any]) -> bool:
def authorized(
vars_map: dict[str, str],
meta: dict[str, Any],
case: dict[str, Any],
*,
execution_profile: str | None = None,
) -> bool:
profile = resolve_execution_profile(meta, execution_profile)
if profile == "isolated-full":
# AUTHORIZE_* and gate variables are intentionally not run-time
# switches for an isolated resettable environment. Capability and
# cleanup preflight are the safety boundary for this profile.
return True
risk = str(case.get("risk") or "readonly")
risks = list(meta.get("risks") or ["readonly"])
lowest = risks[0] if risks else "readonly"
@@ -940,6 +1288,7 @@ def authorized(vars_map: dict[str, str], meta: dict[str, Any], case: dict[str, A
def cmd_run_script(root: Path, case_id: str) -> int:
meta = load_catalog_meta(root)
execution_profile = resolve_execution_profile(meta)
vars_map = resolve_vars(root, meta)
case = next((c for c in load_all_cases(root) if c.get("id") == case_id), None)
if not case:
@@ -977,10 +1326,10 @@ def cmd_run_script(root: Path, case_id: str) -> int:
)
return 2
risk = str(case.get("risk") or "readonly")
if not authorized(vars_map, meta, case):
if not authorized(vars_map, meta, case, execution_profile=execution_profile):
print(
json.dumps(
{"id": public_id, "result": "skipped", "evidence": f"风险 {risk} 未授权"},
{"id": public_id, "result": "skipped", "evidence": f"风险 {risk} 未授权", "execution_profile": execution_profile},
ensure_ascii=False,
)
)
@@ -1029,6 +1378,7 @@ def cmd_run_script(root: Path, case_id: str) -> int:
{
"id": public_id,
"result": "executed",
"execution_profile": execution_profile,
"exit": proc.returncode,
"evidence": evidence,
},
@@ -1049,7 +1399,22 @@ def build_parser() -> argparse.ArgumentParser:
p_sel.add_argument("--category")
p_sel.add_argument("--id", action="append", dest="ids", default=[])
p_sel.add_argument("--tag", choices=TAGS)
p_sel.add_argument(
"--suite", "--client-suite", dest="suite",
choices=[*CLIENT_SUITES, *INTEGRATION_SUITES],
help="客户端或集成测试套件;不传时默认不选 integration",
)
p_sel.add_argument(
"--test-type", choices=TEST_TYPES, default=None,
help="按顶层 test_type 筛选;integration 必须显式选择",
)
p_sel.add_argument("--max-risk")
p_sel.add_argument(
"--execution-profile",
choices=EXECUTION_PROFILES,
default=None,
help="执行策略;isolated-full 在隔离可重置环境中不按 risk/authorize 筛选",
)
p_sel.add_argument("--driver", choices=DRIVERS)
p_sel.add_argument(
"--automation",
@@ -1057,6 +1422,29 @@ def build_parser() -> argparse.ArgumentParser:
default=None,
help="默认 automated;any 表示不按人工参与度筛选",
)
p_sel.add_argument(
"--platform",
action="append",
choices=["android", "ios"],
default=[],
help="mobile 运行平台,可重复;选中 mobile 用例时必填",
)
p_sel.add_argument(
"--artifact",
action="append",
dest="artifacts",
default=[],
metavar="PLATFORM=PATH_OR_URL",
help="本轮移动产物,可重复,例如 android=dist/app.apk",
)
p_sel.add_argument(
"--binding",
action="append",
dest="bindings",
default=[],
metavar="PLATFORM=TESTER_BINDING",
help="平台 tester binding,可重复;默认 tester.<platform>",
)
p_show = sub.add_parser("show", help="打印一条用例", parents=[parent])
p_show.add_argument("--id", required=True, dest="case_id")
sub.add_parser("validate", help="校验目录", parents=[parent])
@@ -1073,6 +1461,19 @@ def main(argv: list[str] | None = None) -> int:
parser = build_parser()
args = parser.parse_args(argv)
root = Path(args.root).resolve()
if args.cmd != "roots":
roots = discover_or_self(root)
if len(roots) > 1:
if args.cmd in FANOUT_COMMANDS:
return cmd_fanout(args, roots)
print(
f"发现多个 tester 根目录,命令 {args.cmd} 不是可 fan-out 的执行命令;"
"请对具体项目根目录执行该维护命令。",
file=sys.stderr,
)
return 2
if len(roots) == 1:
root = roots[0]
if args.cmd == "list":
return cmd_list(root)
if args.cmd == "select":
+150
View File
@@ -0,0 +1,150 @@
#!/usr/bin/env python3
"""Build structured tester issues and the development feedback route.
This module only writes the run contract. Updating a remote PR/issue or
notifying a dispatcher remains an explicit caller-side integration step; no
network mutation is hidden in the tester.
"""
from __future__ import annotations
from copy import deepcopy
from datetime import datetime, timezone
from typing import Any, Iterable, Mapping
ISSUE_TYPES = ("product", "test")
TRIGGERS = ("pr", "requirement", "bug")
ISSUE_RESULTS = frozenset({"fail", "environment-fail"})
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def issue_type_for(result: str) -> str:
if result == "fail":
return "product"
if result == "environment-fail":
return "test"
raise ValueError(f"只有 fail/environment-fail 可以生成 issue: {result}")
def _value(row: Mapping[str, Any], key: str, defaults: Mapping[str, Any]) -> Any:
value = row.get(key)
return value if value not in (None, "") else defaults.get(key, "")
def build_issue(
row: Mapping[str, Any],
*,
defaults: Mapping[str, Any] | None = None,
issue_type: str | None = None,
) -> dict[str, Any]:
"""Create the stable issue shape required by the run report."""
defaults = defaults or {}
result = str(row.get("result") or "")
if result not in ISSUE_RESULTS:
raise ValueError(f"不能为 {result!r} 结果创建 defect: {row.get('id')}")
selected_type = issue_type or row.get("issue_type") or issue_type_for(result)
if selected_type not in ISSUE_TYPES:
raise ValueError(f"issue_type 必须是 product/test: {selected_type}")
testcase = str(row.get("testcase") or row.get("id") or "")
if not testcase:
raise ValueError("issue 缺少 testcase")
return {
"id": str(row.get("issue_id") or f"TEST-{testcase}-{result}"),
"testcase": testcase,
"issue_type": selected_type,
"result": result,
"phase": _value(row, "phase", defaults),
"expected": _value(row, "expected", defaults),
"observed": _value(row, "observed", defaults),
"artifact_hash": _value(row, "artifact_hash", defaults),
"environment": _value(row, "environment", defaults),
"evidence_path": _value(row, "evidence_path", defaults),
"reproduction": _value(row, "reproduction", defaults),
"next_step": _value(row, "next_step", defaults),
"created_at": str(row.get("created_at") or _now()),
}
def collect_issues(
results: Iterable[Mapping[str, Any]],
*,
defaults: Mapping[str, Any] | None = None,
) -> list[dict[str, Any]]:
issues: list[dict[str, Any]] = []
for row in results:
if not isinstance(row, Mapping) or str(row.get("result") or "") not in ISSUE_RESULTS:
continue
explicit = row.get("issue")
source = dict(row)
if isinstance(explicit, Mapping):
source.update(explicit)
issues.append(build_issue(source, defaults=defaults))
return issues
def feedback_plan(
trigger: str,
*,
target: str = "",
related_issue: str = "",
dispatcher: str = "dispatcher",
) -> dict[str, Any]:
"""Describe the required handoff without performing remote mutations."""
if trigger not in TRIGGERS:
raise ValueError(f"trigger 必须是 {', '.join(TRIGGERS)}")
if trigger == "pr":
actions = [
{"action": "update_pr", "target": target, "status": "required"},
{"action": "update_related_issue", "target": related_issue, "status": "required"},
{"action": "notify_dispatcher", "target": dispatcher, "status": "required"},
{"action": "reassign_development", "target": dispatcher, "status": "pending"},
]
route = "pr-update-issue-dispatcher-reassign"
else:
actions = [
{"action": "return_to_development", "target": target or trigger, "status": "required"},
]
route = "development"
return {
"trigger": trigger,
"route": route,
"target": target,
"related_issue": related_issue,
"actions": actions,
"remote_mutation": "caller-required",
"created_at": _now(),
}
def attach_feedback(
run: Mapping[str, Any],
*,
trigger: str | None = None,
target: str = "",
related_issue: str = "",
dispatcher: str = "dispatcher",
defaults: Mapping[str, Any] | None = None,
) -> dict[str, Any]:
"""Return a new run mapping with results, defects and routing preserved."""
output = deepcopy(dict(run))
results = output.get("results")
if not isinstance(results, list):
raise ValueError("run 缺少 results 列表")
issues = collect_issues(results, defaults=defaults)
output["defects"] = issues
# Compatibility name for consumers that call the same structure
# ``test-issues``. It is deliberately a copy, not a second source of truth.
output["test_issues"] = deepcopy(issues)
if trigger:
output["feedback"] = feedback_plan(
trigger,
target=target,
related_issue=related_issue,
dispatcher=dispatcher,
)
return output
@@ -13,6 +13,8 @@ import sys
from pathlib import Path
from typing import Any
from journey import suite as case_suite, test_type
try:
import yaml
except ImportError: # pragma: no cover
@@ -22,6 +24,10 @@ except ImportError: # pragma: no cover
FIELDS = (
"category",
"id",
"test_type",
"suite",
"client_suite",
"platform",
"maps_to",
"title",
"what",
@@ -156,10 +162,17 @@ def export_cases_csv(root: Path, out: Path | None = None) -> Path:
writer.writeheader()
for case in cases:
cid = str(case.get("id") or "")
client_suite_value = case.get("client_suite") or (
case.get("suite") if test_type(case) == "journey" else ""
)
writer.writerow(
{
"category": case.get("category") or "",
"id": cid,
"test_type": test_type(case),
"suite": case_suite(case),
"client_suite": client_suite_value or "",
"platform": one_line(case.get("platform")),
"maps_to": case.get("maps_to") or "",
"title": case.get("title") or "",
"what": one_line(case.get("what")),
+53
View File
@@ -18,6 +18,7 @@ except ImportError: # pragma: no cover
raise SystemExit(2)
from generate_cases_csv import load_cases, one_line, resolve_root, tester_dir
from journey import suite as case_suite, test_type
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
RUN_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
@@ -25,6 +26,10 @@ EXECUTED_YES = frozenset({"pass", "fail", "environment-fail"})
EXECUTED_NO = frozenset({"skipped", "not-run"})
FIELDS = (
"id",
"test_type",
"suite",
"client_suite",
"platform",
"result",
"executed",
"evidence",
@@ -35,6 +40,16 @@ FIELDS = (
"risk",
"driver",
"maps_to",
"blocked_by",
"issue_id",
"issue_type",
"expected",
"observed",
"artifact_hash",
"environment",
"evidence_path",
"reproduction",
"next_step",
)
@@ -107,6 +122,23 @@ def case_index(root: Path) -> dict[str, dict[str, Any]]:
return out
def issue_index(run: dict[str, Any]) -> dict[str, dict[str, Any]]:
"""Index canonical defects and compatibility test_issues by testcase."""
indexed: dict[str, dict[str, Any]] = {}
raw = run.get("defects")
if not isinstance(raw, list):
raw = run.get("test_issues")
if not isinstance(raw, list):
return indexed
for issue in raw:
if not isinstance(issue, dict):
continue
testcase = str(issue.get("testcase") or issue.get("id") or "")
if testcase:
indexed.setdefault(testcase, issue)
return indexed
def export_run_csv(
root: Path,
run_path: Path | None = None,
@@ -118,6 +150,7 @@ def export_run_csv(
path = path.resolve()
run = load_run(path)
cases = case_index(root)
issues = issue_index(run)
dest = out or path.with_suffix(".csv")
dest.parent.mkdir(parents=True, exist_ok=True)
with dest.open("w", encoding="utf-8-sig", newline="") as handle:
@@ -129,9 +162,19 @@ def export_run_csv(
cid = str(item["id"])
result = str(item.get("result") or "")
case = cases.get(cid) or {}
issue = dict(issues.get(cid, {}))
if isinstance(item.get("issue"), dict):
issue.update(item["issue"])
client_suite_value = case.get("client_suite") or (
case.get("suite") if test_type(case) == "journey" else ""
)
writer.writerow(
{
"id": cid,
"test_type": test_type(case),
"suite": case_suite(case),
"client_suite": client_suite_value or "",
"platform": one_line(case.get("platform")),
"result": result,
"executed": executed_cell(result),
"evidence": one_line(item.get("evidence")),
@@ -142,6 +185,16 @@ def export_run_csv(
"risk": case.get("risk") or "",
"driver": case.get("driver") or "",
"maps_to": case.get("maps_to") or "",
"blocked_by": one_line(item.get("blocked_by")),
"issue_id": issue.get("id") or item.get("issue_id") or "",
"issue_type": issue.get("issue_type") or item.get("issue_type") or "",
"expected": one_line(issue.get("expected") or item.get("expected")),
"observed": one_line(issue.get("observed") or item.get("observed")),
"artifact_hash": issue.get("artifact_hash") or item.get("artifact_hash") or "",
"environment": one_line(issue.get("environment") or item.get("environment")),
"evidence_path": one_line(issue.get("evidence_path") or item.get("evidence_path")),
"reproduction": one_line(issue.get("reproduction") or item.get("reproduction")),
"next_step": one_line(issue.get("next_step") or item.get("next_step")),
}
)
return dest
+290
View File
@@ -0,0 +1,290 @@
#!/usr/bin/env python3
"""Shared client-journey contract and result lifecycle helpers.
The mobile/browser driver still owns the real UI interaction. This module
owns the parts that must be identical for every client catalog: contract
validation, preflight blocking, and result propagation. It deliberately
returns the existing tester result vocabulary instead of introducing a second
report format.
"""
from __future__ import annotations
from typing import Any, Iterable, Mapping
TEST_TYPES = ("journey", "integration")
CLIENT_SUITES = ("client-smoke", "client-regression", "safety")
INTEGRATION_SUITES = ("api-contract", "authorization", "build-contract")
PLATFORMS = ("android", "ios")
FAILURE_POLICIES = ("stop", "continue")
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
def client_suite(case: Mapping[str, Any]) -> str:
"""Return the canonical client suite, accepting ``suite`` as a migration alias."""
value = case.get("client_suite")
alias = case.get("suite")
if value and alias and str(value) != str(alias):
return ""
return str(value or alias or "")
def integration_suite(case: Mapping[str, Any]) -> str:
"""Return the integration suite, accepting ``suite`` during migration."""
value = case.get("integration_suite")
alias = case.get("suite")
if value and alias and str(value) != str(alias):
return ""
return str(value or alias or "")
def test_type(case: Mapping[str, Any]) -> str:
"""Return an explicit type or infer one for an old catalog entry."""
declared = str(case.get("test_type") or "").strip().lower()
if declared:
return declared
if client_suite(case) or case.get("journey") is not None:
return "journey"
if integration_suite(case):
return "integration"
return ""
def suite(case: Mapping[str, Any]) -> str:
"""Return the report/selection suite for either test type."""
kind = test_type(case)
return client_suite(case) if kind == "journey" else integration_suite(case)
def normalize_platforms(value: Any) -> list[str]:
if isinstance(value, str):
values = [value]
elif isinstance(value, (list, tuple)):
values = list(value)
else:
return []
return [str(item).strip().lower() for item in values if str(item).strip()]
def _non_empty_list(value: Any) -> bool:
return isinstance(value, list) and bool(value) and all(str(item).strip() for item in value)
def _has_account(fixture: Mapping[str, Any]) -> bool:
account = fixture.get("account")
if isinstance(account, Mapping):
return bool(str(account.get("role") or account.get("username") or "").strip())
return bool(str(account or "").strip())
def contract_errors(case: Mapping[str, Any], sessions: Mapping[str, Any] | None = None) -> list[str]:
"""Validate a client journey without requiring app-specific field names.
A case becomes a client journey when it declares ``client_suite`` (or the
temporary ``suite`` alias). Legacy non-client cases keep the existing
schema. Once opted in, all lifecycle inputs are explicit so a missing
fixture/cleanup can stop the run before any case result is fabricated.
"""
kind = test_type(case)
declared_kind = str(case.get("test_type") or "").strip().lower()
if declared_kind and declared_kind not in TEST_TYPES:
return [f"{case.get('id') or '?'}: test_type 必须是 journey 或 integration"]
if not kind:
return []
if kind == "integration":
return integration_contract_errors(case)
suite_name = client_suite(case)
journey = case.get("journey")
errors: list[str] = []
cid = str(case.get("id") or "?")
where = f"{cid}: client journey"
if case.get("client_suite") and case.get("suite") and str(case["client_suite"]) != str(case["suite"]):
errors.append(f"{where}: client_suite 与 suite 不一致")
if suite_name not in CLIENT_SUITES:
errors.append(f"{where}: client_suite 必须是 {', '.join(CLIENT_SUITES)}")
if not isinstance(journey, Mapping):
errors.append(f"{where}: journey 必须是 mapping")
journey = {}
for key in ("goal", "entry_state", "final_state"):
if not str(journey.get(key) or "").strip():
errors.append(f"{where}: journey.{key} 必填")
if not _non_empty_list(journey.get("actions")):
errors.append(f"{where}: journey.actions 必须是非空列表")
if not _non_empty_list(journey.get("assertions")):
errors.append(f"{where}: journey.assertions 必须是非空列表")
policy = str(journey.get("failure_policy") or "")
if policy not in FAILURE_POLICIES:
errors.append(f"{where}: journey.failure_policy 必须是 stop 或 continue")
fixture = case.get("fixture")
if not isinstance(fixture, Mapping):
errors.append(f"{where}: fixture 缺失,必须在 preflight 前声明")
elif not _has_account(fixture):
errors.append(f"{where}: fixture.account 缺失,必须声明测试账号或角色")
cleanup = case.get("cleanup")
if not isinstance(cleanup, Mapping):
errors.append(f"{where}: cleanup 缺失,必须在 preflight 前声明")
else:
steps = cleanup.get("steps")
command = cleanup.get("run")
explicitly_noop = cleanup.get("required") is False and str(cleanup.get("reason") or "").strip()
if not explicitly_noop and not _non_empty_list(steps) and not str(command or "").strip():
errors.append(f"{where}: cleanup 必须有非空 steps/run;只读用例也要显式声明 reason")
platforms = normalize_platforms(case.get("platform"))
if not platforms:
errors.append(f"{where}: platform 必须声明 android/ios")
elif any(platform not in PLATFORMS for platform in platforms):
errors.append(f"{where}: platform 只能是 {', '.join(PLATFORMS)}")
elif len(platforms) != len(set(platforms)):
errors.append(f"{where}: platform 不能重复")
driver = case.get("driver")
if driver not in ("mobile", "browser"):
errors.append(f"{where}: 客户端旅程 driver 必须是 mobile 或 browser")
if driver and driver in ("mobile", "browser"):
session = case.get("session")
if not str(session or "").strip():
errors.append(f"{where}: session 必须声明可复用客户端会话")
elif sessions is not None and session not in sessions:
errors.append(f"{where}: session={session} 未在 catalog.sessions 声明")
if suite_name == "safety" and str(case.get("tag") or "") == "smoke":
errors.append(f"{where}: safety 不得标记为 smoke")
return errors
def integration_contract_errors(case: Mapping[str, Any]) -> list[str]:
"""Validate the non-client half of the tester catalog.
Integration cases deliberately do not require a browser/mobile session. A
``how`` block is the legacy call declaration; newer catalogs may use
``request``/``input``/``call`` or an ``integration`` mapping as well.
``expected`` remains the common protocol/data assertion list.
"""
cid = str(case.get("id") or "?")
where = f"{cid}: integration"
errors: list[str] = []
value = case.get("integration_suite")
alias = case.get("suite")
if value and alias and str(value) != str(alias):
errors.append(f"{where}: integration_suite 与 suite 不一致")
suite_name = integration_suite(case)
if suite_name not in INTEGRATION_SUITES:
errors.append(
f"{where}: integration_suite 必须是 {', '.join(INTEGRATION_SUITES)}"
)
fixture = case.get("fixture")
if not isinstance(fixture, Mapping) or not fixture:
errors.append(f"{where}: fixture 缺失,必须在 preflight 前声明")
declared_call = any(
key in case and case.get(key) not in (None, "", [], {})
for key in ("request", "input", "call", "integration")
)
how = case.get("how")
if not declared_call and not isinstance(how, Mapping):
errors.append(f"{where}: 必须声明 request/input/call/integration 或 how")
# Integration writes need an explicit rollback/cleanup contract. Read-only
# probes may state why cleanup is unnecessary, matching test-preflight.
risk = str(case.get("risk") or "")
cleanup = case.get("cleanup")
cleanup_steps = isinstance(cleanup, Mapping) and (
bool(cleanup.get("steps")) or bool(str(cleanup.get("run") or "").strip())
)
cleanup_noop = isinstance(cleanup, Mapping) and cleanup.get("required") is False and bool(
str(cleanup.get("reason") or "").strip()
)
if risk != "readonly" and not (cleanup_steps or cleanup_noop):
errors.append(f"{where}: 写入型 integration 必须声明 cleanup/rollback steps/run")
elif cleanup is not None and not (cleanup_steps or cleanup_noop):
errors.append(f"{where}: cleanup 必须有 steps/run 或 required=false + reason")
forbidden = []
if case.get("client_suite") or case.get("journey") is not None:
forbidden.append("client_suite/journey")
if case.get("session") or case.get("platform"):
forbidden.append("session/platform")
if forbidden:
errors.append(f"{where}: integration 不应声明 " + ", ".join(forbidden))
return errors
def preflight(cases: Iterable[Mapping[str, Any]], sessions: Mapping[str, Any] | None = None) -> dict[str, Any]:
"""Return a contract preflight result; no per-case result is created on block."""
cases = list(cases)
errors: list[str] = []
for case in cases:
errors.extend(contract_errors(case, sessions=sessions))
return {
"required": bool(errors or any(client_suite(case) for case in cases)),
"status": "blocked" if errors else "ready",
"failure_class": "contract" if errors else None,
"errors": errors,
}
def classify_outcome(
*,
ok: bool,
environment_ready: bool = True,
phase: str = "journey",
) -> str:
"""Classify one observed phase using the tester's five result states.
A valid test account is part of the client journey: a failed login is a
product ``fail``. Only an unavailable environment is ``environment-fail``.
"""
if not environment_ready:
return "environment-fail"
if not ok:
return "fail"
return "pass"
def propagate_results(
case_ids: Iterable[str],
observed: Mapping[str, str],
*,
skipped: Mapping[str, str] | None = None,
continue_on_fail: bool = False,
policies: Mapping[str, str] | None = None,
blocked_by: Mapping[str, Iterable[str]] | None = None,
) -> list[dict[str, str]]:
"""Build ordered result rows and mark cases after a blocking failure not-run.
``skipped`` is intentional selection (risk/lifecycle/assumption); it is
never converted to ``not-run``. A product or environment failure stops
subsequent cases unless the caller explicitly opts into continuation.
"""
skipped = skipped or {}
policies = policies or {}
blocked_by = blocked_by or {}
rows: list[dict[str, str]] = []
blocking_case = ""
for raw_id in case_ids:
case_id = str(raw_id)
if case_id in skipped:
rows.append({"id": case_id, "result": "skipped", "notes": skipped[case_id]})
continue
if blocking_case:
rows.append({
"id": case_id,
"result": "not-run",
"notes": "前置用例失败,未轮到执行",
"blocked_by": ",".join(blocked_by.get(case_id) or (blocking_case,)),
})
continue
result = observed.get(case_id, "not-run")
if result not in RESULTS:
raise ValueError(f"未知结果: {case_id}={result}")
rows.append({"id": case_id, "result": result})
policy = policies.get(case_id)
if result in ("fail", "environment-fail") and not (continue_on_fail or policy == "continue"):
blocking_case = case_id
return rows
+777
View File
@@ -0,0 +1,777 @@
#!/usr/bin/env python3
"""Provision caller-provided mobile artifacts for tester.
This is shared tester infrastructure. It never replaces real Android/iOS
execution with HTTP calls: it only obtains, verifies and installs the
artifact the caller supplied, then the catalog runner drives the mobile
session.
"""
from __future__ import annotations
import argparse
import hashlib
import importlib.util
import json
import os
import plistlib
import re
import shutil
import subprocess
import sys
import tempfile
import time
import urllib.error
import urllib.parse
import urllib.request
import zipfile
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from platform import system as platform_system
from typing import Any
COMMIT = re.compile(r"^[0-9a-fA-F]{40}$")
PLATFORMS = {"android", "ios"}
PREFLIGHT_ROOT = Path(__file__).resolve().parents[2] / "test-preflight" / "scripts" / "preflight.py"
class EnvironmentFailure(RuntimeError):
"""An external environment or recoverable provisioning failure."""
class ArtifactFailure(RuntimeError):
"""A missing or unusable caller-provided artifact locator."""
@dataclass
class CommandResult:
code: int
output: str
@dataclass
class RuntimeContext:
root: Path
bindings: dict[str, str] = field(default_factory=dict)
retries: dict[str, int] = field(default_factory=lambda: {"download": 3, "start": 3, "install": 3})
evidence: list[dict[str, Any]] = field(default_factory=list)
def record(self, command: str, result: CommandResult) -> None:
self.evidence.append({
"command": command,
"status": "ok" if result.code == 0 else "error",
"output": result.output[-800:],
})
def redact_url(value: str) -> str:
parsed = urllib.parse.urlsplit(value)
if not parsed.query:
return value
return urllib.parse.urlunsplit((parsed.scheme, parsed.netloc, parsed.path, "<redacted>", ""))
def run_command(args: list[str], *, context: RuntimeContext | None = None, cwd: Path | None = None) -> CommandResult:
try:
proc = subprocess.run(args, cwd=cwd, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, check=False)
result = CommandResult(proc.returncode, proc.stdout.strip())
except OSError as exc:
result = CommandResult(127, str(exc))
if context:
context.record(" ".join(args), result)
return result
def fetch_bytes(url: str, *, attempts: int = 3, token: str = "") -> bytes:
if url.startswith("file://"):
return Path(urllib.parse.urlsplit(url).path).read_bytes()
if not urllib.parse.urlsplit(url).scheme:
return Path(url).read_bytes()
last: Exception | None = None
for attempt in range(1, attempts + 1):
try:
headers = {"User-Agent": "pouch-tester/1"}
if token:
headers["Authorization"] = f"token {token}"
request = urllib.request.Request(url, headers=headers)
with urllib.request.urlopen(request, timeout=30) as response:
return response.read()
except (OSError, urllib.error.URLError) as exc:
last = exc
if attempt < attempts:
time.sleep(min(attempt, 2))
raise EnvironmentFailure(f"下载失败({attempts} 次): {redact_url(url)}: {last}")
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for block in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(block)
return digest.hexdigest()
def parse_artifacts(values: list[str]) -> dict[str, str]:
pairs: dict[str, str] = {}
for raw in values:
platform, sep, value = raw.partition("=")
platform = platform.strip().lower()
value = value.strip()
if not sep or platform not in PLATFORMS or not value:
raise ArtifactFailure(f"--artifact 格式必须是 android=<路径或 URL> 或 ios=<路径或 URL>: {raw}")
if platform in pairs:
raise ArtifactFailure(f"--artifact 重复平台: {platform}")
pairs[platform] = value
return pairs
def parse_bindings(values: list[str], requested: list[str]) -> dict[str, str]:
"""Resolve one tester binding per requested platform.
A single legacy ``agent`` mapping cannot safely represent a multi-platform
run. Keep the platform distinction in the command contract all the way to
agent-runtime instead of treating binding as report metadata.
"""
pairs: dict[str, str] = {}
for raw in values:
platform, sep, value = raw.partition("=")
platform = platform.strip().lower()
value = value.strip()
if not sep or platform not in PLATFORMS or not value or not re.fullmatch(r"[a-z][a-z0-9-]*(?:\.[a-z][a-z0-9-]*)+", value):
raise ArtifactFailure(f"--binding 格式必须是 android=tester.<name> 或 ios=tester.<name>: {raw}")
if platform in pairs:
raise ArtifactFailure(f"--binding 重复平台: {platform}")
pairs[platform] = value
resolved = {platform: pairs.get(platform, f"tester.{platform}") for platform in requested}
if len(set(resolved.values())) != len(resolved) and len(resolved) > 1:
raise ArtifactFailure("多平台运行不能复用同一个 tester binding;请分别配置 tester.android/tester.ios")
return resolved
def load_agent_runtime():
candidates = [
Path.home() / ".codex" / "skills" / "agent-runtime" / "scripts" / "agent.py",
Path.home() / ".pouch" / "skills" / "agent-runtime" / "scripts" / "agent.py",
Path(__file__).resolve().parents[2] / "agent-runtime" / "scripts" / "agent.py",
]
runtime_path = next((item for item in candidates if item.is_file()), None)
if runtime_path is None:
raise EnvironmentFailure("找不到共享 agent-runtime: " + ", ".join(str(item) for item in candidates))
spec = importlib.util.spec_from_file_location("pouch_agent_runtime_for_mobile_bundle", runtime_path)
if spec is None or spec.loader is None:
raise EnvironmentFailure(f"无法加载共享 agent-runtime: {runtime_path}")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def binding_location(root: Path, binding_id: str) -> dict[str, Any]:
runtime = load_agent_runtime()
try:
value = runtime.binding(runtime.load_config(root), binding_id)
except (ValueError, OSError) as exc:
raise EnvironmentFailure(f"无法解析 {binding_id}: {exc}") from exc
if value.get("adapter") != "tester":
raise EnvironmentFailure(f"{binding_id} 不是 tester binding")
return value
def agent_runner(root: Path, request: dict[str, Any]) -> dict[str, Any]:
"""Dispatch a platform operation through the resolved tester binding."""
runtime = load_agent_runtime()
return runtime.execute_runner_request(root, request)
def platform_node(root: Path, binding_id: str) -> str:
location = binding_location(root, binding_id)["location"]
return "local" if location.get("environment") == "local" else str(location.get("environment"))
def validate_preflight_matrix(root: Path, config_path: Path, bindings: dict[str, str], requested: list[str]) -> None:
"""Fail closed when a matrix routes a platform to the wrong node/runner."""
try:
import yaml
raw = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
except (OSError, ValueError, ImportError) as exc:
raise EnvironmentFailure(f"无法读取 preflight matrix: {config_path}: {exc}") from exc
rows = raw.get("matrix") if isinstance(raw, dict) else None
if not isinstance(rows, list):
raise EnvironmentFailure("preflight matrix 缺少 matrix;不能推断平台执行节点")
by_platform = {str(row.get("platform", "")).lower(): row for row in rows if isinstance(row, dict)}
for platform in requested:
expected_binding = bindings[platform]
row = by_platform.get(platform)
if not row:
raise EnvironmentFailure(f"preflight matrix 缺少 {platform} 行;不能创建该平台 session")
tester = row.get("tester") if isinstance(row, dict) else None
actual_binding = tester.get("binding") if isinstance(tester, dict) else None
if actual_binding != expected_binding:
raise EnvironmentFailure(f"{platform} preflight binding={actual_binding!r},期望 {expected_binding!r}")
expected_node = platform_node(root, expected_binding)
actual_node = tester.get("node") if isinstance(tester, dict) else None
if expected_node == "local" and actual_node not in {"local", "localhost", "127.0.0.1"}:
raise EnvironmentFailure(f"{platform} binding 是 local,但 preflight node={actual_node!r}")
runner = tester.get("runner") if isinstance(tester, dict) else None
if expected_node != "local" and (not isinstance(runner, dict) or runner.get("kind") != "contract"):
raise EnvironmentFailure(f"{platform} 绑定到 {expected_node},必须使用 contract runner;禁止在协调节点执行")
def preflight_remote_artifacts(config_path: Path, requested: list[str]) -> dict[str, str]:
"""Return caller-declared artifact paths readable by remote tester nodes."""
try:
import yaml
raw = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
except (OSError, ValueError, ImportError) as exc:
raise EnvironmentFailure(f"无法读取 preflight matrix: {config_path}: {exc}") from exc
rows = raw.get("matrix") if isinstance(raw, dict) else None
if not isinstance(rows, list):
raise EnvironmentFailure("preflight matrix 缺少 matrix;不能解析远程产物路径")
by_platform = {str(row.get("platform", "")).lower(): row for row in rows if isinstance(row, dict)}
paths: dict[str, str] = {}
for platform in requested:
row = by_platform.get(platform) or {}
artifact = row.get("artifact") if isinstance(row, dict) else None
remote_path = artifact.get("remote_path") if isinstance(artifact, dict) else None
if isinstance(remote_path, str) and remote_path.strip():
paths[platform] = remote_path.strip()
return paths
def local_path(locator: str) -> Path | None:
parsed = urllib.parse.urlsplit(locator)
if parsed.scheme in ("http", "https"):
return None
if parsed.scheme == "file":
return Path(parsed.path)
if parsed.scheme:
raise ArtifactFailure(f"不支持的产物地址 scheme: {parsed.scheme}")
return Path(locator).expanduser()
def obtain_artifact(platform: str, locator: str, destination: Path, context: RuntimeContext) -> dict[str, Any]:
path = local_path(locator)
if path is not None:
if not path.is_file():
raise ArtifactFailure(f"{platform} 产物不是现有文件: {path}")
resolved = path.resolve()
return {
"path": str(resolved),
"locator": str(resolved),
"sha256": sha256_file(resolved),
"attempts": 1,
"source": "file",
"verified": True,
}
last: Exception | None = None
for attempt in range(1, context.retries["download"] + 1):
try:
destination.parent.mkdir(parents=True, exist_ok=True)
destination.write_bytes(fetch_bytes(locator, attempts=1, token=os.environ.get("GITEA_TOKEN", "")))
return {
"path": str(destination),
"locator": redact_url(locator),
"sha256": sha256_file(destination),
"attempts": attempt,
"source": "url",
"verified": True,
}
except (OSError, EnvironmentFailure) as exc:
last = exc
if attempt < context.retries["download"]:
time.sleep(min(attempt, 2))
raise EnvironmentFailure(f"{platform} 下载失败({context.retries['download']} 次): {last}")
def catalog_identities(root: Path) -> dict[str, str]:
path = root / ".pouch" / "tester" / "catalog.yaml"
if not path.is_file():
return {}
try:
import yaml
except ImportError:
return {}
try:
data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
except (OSError, yaml.YAMLError):
return {}
artifacts = data.get("artifacts")
if not isinstance(artifacts, dict):
return {}
identities: dict[str, str] = {}
android = artifacts.get("android")
ios = artifacts.get("ios")
if isinstance(android, dict) and isinstance(android.get("package"), str) and android["package"]:
identities["android"] = android["package"]
if isinstance(ios, dict) and isinstance(ios.get("bundle_id"), str) and ios["bundle_id"]:
identities["ios"] = ios["bundle_id"]
return identities
def android_identity(path: Path, context: RuntimeContext) -> str:
commands: list[list[str]] = []
if shutil.which("apkanalyzer"):
commands.append(["apkanalyzer", "manifest", "application-id", str(path)])
if shutil.which("aapt2"):
commands.append(["aapt2", "dump", "badging", str(path)])
if shutil.which("aapt"):
commands.append(["aapt", "dump", "badging", str(path)])
if not commands:
raise EnvironmentFailure("缺少 apkanalyzer/aapt2/aapt;请在 Android tester 节点配置 Android SDK build-tools")
for command in commands:
result = run_command(command, context=context)
if result.code == 0:
match = re.search(r"(?:name|application-id)[=:]['\"]?([A-Za-z0-9_.]+)", result.output)
if not match:
# apkanalyzer manifest application-id prints the bare package
# name, while aapt/aapt2 print a package: name=... record.
match = re.fullmatch(r"\s*([A-Za-z0-9_.]+)\s*", result.output)
if match:
return match.group(1)
raise EnvironmentFailure("APK 身份读取失败;请检查 Android SDK build-tools")
def ios_identity(path: Path) -> str:
try:
with zipfile.ZipFile(path) as archive:
candidates = [name for name in archive.namelist() if re.fullmatch(r"Payload/[^/]+\.app/Info\.plist", name)]
if len(candidates) != 1:
raise EnvironmentFailure("IPA 中没有唯一 Payload/*.app/Info.plist")
plist = plistlib.loads(archive.read(candidates[0]))
except (OSError, zipfile.BadZipFile, plistlib.InvalidFileException) as exc:
raise EnvironmentFailure(f"IPA 解包/读取 Info.plist 失败: {exc}") from exc
bundle_id = plist.get("CFBundleIdentifier")
if not isinstance(bundle_id, str) or not bundle_id:
raise EnvironmentFailure("IPA Info.plist 缺少 CFBundleIdentifier")
return bundle_id
def retry_command(args: list[str], *, context: RuntimeContext, action: str) -> CommandResult:
last = CommandResult(1, "")
for attempt in range(1, context.retries[action] + 1):
last = run_command(args, context=context)
if last.code == 0:
return last
if attempt < context.retries[action]:
time.sleep(min(attempt, 2))
raise EnvironmentFailure(f"{action} 失败({context.retries[action]} 次): {' '.join(args)}\n{last.output[-500:]}")
def android_preflight(context: RuntimeContext) -> dict[str, str]:
adb = os.environ.get("ADB_BIN") or shutil.which("adb")
if not adb:
raise EnvironmentFailure("Android 节点缺少 adb;请安装/配置 Android SDK platform-tools")
retry_command([adb, "start-server"], context=context, action="start")
devices = run_command([adb, "devices", "-l"], context=context)
available = [line.split()[0] for line in devices.output.splitlines()[1:] if re.search(r"\sdevice\s", line)]
serial = os.environ.get("ANDROID_SERIAL", "").strip()
if serial and serial not in available:
raise EnvironmentFailure(f"ANDROID_SERIAL={serial} 不在 device 状态;请解锁/授权设备后检查 adb devices -l")
if not serial and len(available) != 1:
if not available:
raise EnvironmentFailure("没有 Android device;请连接、解锁、开启 USB 调试并接受 RSA 授权")
raise EnvironmentFailure(f"检测到多个 Android device ({', '.join(available)});请设置 ANDROID_SERIAL")
return {"adb": adb, "serial": serial or available[0], "session_lifecycle": "reused-or-started"}
def ios_preflight(context: RuntimeContext) -> dict[str, str]:
if platform_system() != "Darwin":
raise EnvironmentFailure("iOS 必须由 tester.ios 的 macOS 节点执行;Linux 不能伪装 Simulator")
xcrun = shutil.which("xcrun")
if not xcrun:
raise EnvironmentFailure("iOS 节点缺少 xcrun;请安装 Xcode Command Line Tools")
devices = run_command([xcrun, "simctl", "list", "devices", "available"], context=context)
if devices.code != 0:
raise EnvironmentFailure("simctl 不可用;请安装 Xcode 并确认已接受许可")
wanted = os.environ.get("IOS_SIMULATOR_UDID", "").strip()
matches: list[tuple[str, str, str]] = []
for line in devices.output.splitlines():
match = re.search(r"^\s*(.+?)\s+\(([0-9A-Fa-f-]{36})\)\s+\((Booted|Shutdown)\)", line)
if match:
matches.append((match.group(2), match.group(1), match.group(3)))
selected = next((item for item in matches if item[0] == wanted), None) if wanted else None
selected = selected or next((item for item in matches if item[2] == "Booted"), None) or (matches[0] if matches else None)
if not selected:
raise EnvironmentFailure("没有可用 iOS Simulator;请在 Mac 上安装一个可用 runtime/device")
udid, name, state = selected
if state != "Booted":
retry_command([xcrun, "simctl", "boot", udid], context=context, action="start")
retry_command([xcrun, "simctl", "bootstatus", udid, "-b"], context=context, action="start")
return {"xcrun": xcrun, "udid": udid, "name": name, "session_lifecycle": "reused" if state == "Booted" else "started"}
def check_network(context: RuntimeContext, node: dict[str, str] | None = None) -> None:
host = os.environ.get("HOST", "").strip()
api = os.environ.get("API_BASE", "").strip() or (f"http://{host}" if host else "")
if not api:
raise EnvironmentFailure("缺少 HOST/API_BASE;请在 tester.local.yaml 配置测试服务地址")
for attempt in range(1, context.retries["start"] + 1):
try:
with urllib.request.urlopen(urllib.request.Request(api.rstrip("/") + "/v1/me"), timeout=5) as response:
if response.status < 500:
break
except urllib.error.HTTPError as exc:
if exc.code < 500:
break
except (OSError, urllib.error.URLError):
pass
if attempt == context.retries["start"]:
raise EnvironmentFailure(f"测试服务不可达: {api};请检查 HOST、VPN、DNS 和局域网后重跑")
time.sleep(min(attempt, 2))
if node and "adb" in node:
if shutil.which("ping"):
result = run_command([node["adb"], "-s", node["serial"], "shell", "ping", "-c", "1", "-W", "3", host], context=context)
if result.code != 0:
raise EnvironmentFailure(f"Android device 无法访问 HOST={host};请检查设备 Wi-Fi/DNS/VPN")
elif node:
result = run_command([node["xcrun"], "simctl", "spawn", node["udid"], "/usr/bin/curl", "-sS", "-o", "/dev/null", "-w", "%{http_code}", api.rstrip("/") + "/v1/me"], context=context)
if result.code != 0 or result.output.strip() in {"", "000"} or result.output.strip().startswith("5"):
raise EnvironmentFailure(f"iOS Simulator 无法访问 {api};请检查 Mac 的 VPN/DNS/局域网权限")
def install_android(node: dict[str, str], path: Path, package: str, context: RuntimeContext) -> dict[str, Any]:
retry_command([node["adb"], "-s", node["serial"], "install", "-r", str(path)], context=context, action="install")
result = run_command([node["adb"], "-s", node["serial"], "shell", "pm", "path", package], context=context)
if result.code != 0 or "package:" not in result.output:
raise EnvironmentFailure(f"安装后未找到 Android package {package};请检查签名/设备空间")
return {"serial": node["serial"], "package": package, "installed": True, "verified": True}
def install_ios(node: dict[str, str], path: Path, bundle_id: str, context: RuntimeContext) -> dict[str, Any]:
retry_command([node["xcrun"], "simctl", "install", node["udid"], str(path)], context=context, action="install")
result = run_command([node["xcrun"], "simctl", "get_app_container", node["udid"], bundle_id], context=context)
if result.code != 0 or not result.output:
raise EnvironmentFailure(f"安装后未找到 iOS bundle {bundle_id};请检查签名/Simulator 架构")
return {"udid": node["udid"], "simulator": node["name"], "bundle_id": bundle_id, "installed": True, "verified": True}
def sanitize(value: str) -> str:
return re.sub(r"[^A-Za-z0-9._-]+", "-", value)
def resolve_commit(explicit: str, context: RuntimeContext) -> str:
if explicit:
if not COMMIT.fullmatch(explicit):
raise ArtifactFailure("--commit 必须是 40 位 commit SHA")
return explicit
git = shutil.which("git") or "git"
result = run_command([git, "rev-parse", "HEAD"], context=context, cwd=context.root)
if result.code != 0 or not COMMIT.fullmatch(result.output):
raise EnvironmentFailure(f"无法读取当前 commit: {context.root}")
return result.output
def ensure_worktree(commit: str, context: RuntimeContext, parent: Path | None = None) -> tuple[Path, str]:
git = shutil.which("git") or "git"
check = run_command([git, "rev-parse", "--show-toplevel"], context=context, cwd=context.root)
if check.code != 0:
raise EnvironmentFailure(f"目标目录不是 Git worktree: {context.root}")
repo = Path(check.output).resolve()
verify = run_command([git, "rev-parse", f"{commit}^{{commit}}"], context=context, cwd=repo)
if verify.code != 0:
fetched = run_command([git, "fetch", "--no-tags", "origin", commit], context=context, cwd=repo)
if fetched.code != 0:
raise EnvironmentFailure(f"目标 commit 不在本地且无法从 origin 获取: {commit}")
base = parent or Path(os.environ.get("TESTER_WORKTREE_ROOT", Path(tempfile.gettempdir()) / "pouch-tester" / "worktrees"))
base.mkdir(parents=True, exist_ok=True)
token = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + "-" + os.urandom(3).hex()
path = (base / f"{sanitize(commit[:12])}-{token}").resolve()
if path.exists():
raise EnvironmentFailure(f"隔离 worktree 路径已存在,拒绝覆盖: {path}")
added = run_command([git, "worktree", "add", "--detach", str(path), commit], context=context, cwd=repo)
if added.code != 0:
raise EnvironmentFailure(f"创建精确 commit worktree 失败: {path}\n{added.output[-500:]}")
actual = run_command([git, "-C", str(path), "rev-parse", "HEAD"], context=context)
if actual.code != 0 or actual.output.lower() != commit.lower():
raise EnvironmentFailure(f"worktree commit 不匹配: expected={commit} actual={actual.output}")
return path, actual.output
def platforms(raw: str) -> list[str]:
values = [item.strip().lower() for item in raw.split(",") if item.strip()]
if not values:
return ["android", "ios"]
if any(item not in PLATFORMS for item in values) or len(values) != len(set(values)):
raise ArtifactFailure("--platform 只能是 android,ios 且不能重复")
return values
def write_run(root: Path, record: dict[str, Any]) -> Path:
runs = root / ".pouch" / "tester" / "runs"
runs.mkdir(parents=True, exist_ok=True)
run_id = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + "-bundle-" + os.urandom(3).hex()
path = runs / f"{run_id}.yaml"
path.write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
return path
def all_active_result_rows(root: Path, *, blocked_by: str) -> list[dict[str, Any]]:
"""Keep a testcase row for every active catalog entry when preflight blocks."""
try:
from generate_cases_csv import load_cases
cases = load_cases(root)
except (ImportError, OSError, ValueError, SystemExit):
cases = []
rows: list[dict[str, Any]] = []
for case in cases:
if case.get("lifecycle", "active") != "active":
continue
rows.append({
"id": str(case.get("id") or ""),
"result": "not-run",
"notes": "test-preflight 未通过,未创建移动 session",
"blocked_by": [blocked_by],
})
return rows
def run_test_preflight(
config_path: Path,
*,
mode: str = "verify",
yaml_path: Path | None = None,
csv_path: Path | None = None,
) -> dict[str, Any]:
"""Run the shared preflight contract before any client session work."""
if mode not in {"verify", "prepare"}:
raise ArtifactFailure("preflight mode 只能是 verify 或 prepare")
if not PREFLIGHT_ROOT.is_file():
raise EnvironmentFailure(f"找不到 test-preflight skill: {PREFLIGHT_ROOT}")
spec = importlib.util.spec_from_file_location("pouch_test_preflight", PREFLIGHT_ROOT)
if spec is None or spec.loader is None:
raise EnvironmentFailure("无法加载 test-preflight skill")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
try:
config = module.load_config(config_path)
evidence = module.run(config, mode)
yaml_path = yaml_path or config_path.with_name("preflight.yaml")
csv_path = csv_path or config_path.with_name("preflight.csv")
return module.write_evidence(evidence, yaml_path, csv_path)
except (OSError, ValueError, module.yaml.YAMLError) as exc:
return {"status": "blocked", "error": str(exc), "yaml": str(yaml_path or ""), "csv": str(csv_path or "")}
def execute(args: argparse.Namespace) -> int:
requested = platforms(args.platform)
root = Path(args.root).resolve()
bindings = parse_bindings(args.bindings, requested)
context = RuntimeContext(root, bindings=bindings)
record: dict[str, Any] = {
"schema_version": 1,
"started_at": datetime.now(timezone.utc).isoformat(),
"platforms_requested": requested,
"partial": bool(args.partial),
"bindings": bindings,
"node": os.environ.get("TESTER_NODE", "local"),
"terminal": {"handle": os.environ.get("ORCA_TERMINAL_HANDLE"), "created": os.environ.get("TESTER_TERMINAL_CREATED", "false").lower() == "true", "closed": os.environ.get("TESTER_TERMINAL_CLOSED", "unknown")},
"product_test": {"status": "not-run", "failure_class": "automatic-repair-failed", "note": "准备完成后由 tester catalog 驱动真实移动 session;本工具不以 API 替代移动测试"},
"platforms": {},
}
remote_artifacts: dict[str, str] = {}
try:
locators = parse_artifacts(args.artifacts)
missing = [platform for platform in requested if platform not in locators]
extra = [platform for platform in locators if platform not in requested]
if missing:
raise ArtifactFailure("缺少调用方产物位置: " + ", ".join(f"--artifact {item}=<路径或 URL>" for item in missing))
if extra:
raise ArtifactFailure("产物平台超出 --platform: " + ", ".join(extra))
record["artifacts_requested"] = {platform: redact_url(locator) if local_path(locator) is None else locator for platform, locator in locators.items()}
commit = resolve_commit(args.commit, context)
worktree, actual_commit = ensure_worktree(commit, context, Path(args.worktree_root).resolve() if args.worktree_root else None)
record["worktree"] = {"path": str(worktree), "expected_commit": commit, "actual_commit": actual_commit, "isolated": True}
artifact_dir = Path(tempfile.mkdtemp(prefix="pouch-tester-artifacts-"))
record["artifact_dir"] = str(artifact_dir)
expected = catalog_identities(context.root)
prepared: dict[str, tuple[dict[str, Any], dict[str, str] | None, str]] = {}
failures: dict[str, str] = {}
for platform in requested:
result: dict[str, Any] = {"node": "tester.android" if platform == "android" else "tester.ios", "status": "preparing"}
record["platforms"][platform] = result
try:
path = artifact_dir / ("app.apk" if platform == "android" else "app.ipa")
result["artifact"] = obtain_artifact(platform, locators[platform], path, context)
identity = android_identity(Path(result["artifact"]["path"]), context) if platform == "android" else ios_identity(Path(result["artifact"]["path"]))
wanted = expected.get(platform)
if wanted and identity != wanted:
raise EnvironmentFailure(f"{platform} 身份不匹配: expected={wanted} actual={identity}")
result.update({"identity": identity, "status": "artifact-ready"})
prepared[platform] = (result, None, identity)
except (EnvironmentFailure, ArtifactFailure) as exc:
failures[platform] = str(exc)
result.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc)})
# This is the hard gate shared by integration and journey callers.
# There is no safe implicit matrix: without a caller-supplied config we
# cannot prove HOST, fixtures, tester bindings, or node reachability.
preflight_config = Path(args.preflight_config).expanduser() if args.preflight_config else None
if not failures:
if preflight_config is None:
preflight_result = {
"status": "blocked",
"error": "缺少 --preflight-config;不能跳过 test-preflight",
}
else:
validate_preflight_matrix(context.root, preflight_config, bindings, requested)
remote_artifacts = preflight_remote_artifacts(preflight_config, requested)
preflight_result = run_test_preflight(
preflight_config,
mode=args.preflight_mode,
yaml_path=Path(args.preflight_yaml).expanduser() if args.preflight_yaml else None,
csv_path=Path(args.preflight_csv).expanduser() if args.preflight_csv else None,
)
record["preflight"] = preflight_result
if preflight_result.get("status") != "ready":
failures["preflight"] = str(preflight_result.get("error") or preflight_result.get("status"))
if not failures and args.preflight_mode == "prepare":
# test-preflight prepare owns platform probing, installation and
# launch; do not install/start the same artifact a second time.
for result, _, _ in prepared.values():
result.update({"status": "preflight-prepared"})
elif not failures:
for platform in requested:
result, _, identity = prepared[platform]
try:
binding_id = bindings[platform]
node_name = platform_node(context.root, binding_id)
probe = agent_runner(context.root, {
"operation": "platform_probe",
"platform": platform,
"node": node_name,
"binding": binding_id,
})
if probe.get("status") != "ready":
raise EnvironmentFailure(str(probe.get("observed") or probe))
host = os.environ.get("HOST", "").strip()
network = agent_runner(context.root, {
"operation": "network_probe",
"platform": platform,
"node": node_name,
"binding": binding_id,
"url": f"http://{host}/v1/me",
}) if host else {"status": "blocked", "observed": "HOST 未设置"}
if network.get("status") != "ready":
raise EnvironmentFailure(str(network.get("observed") or network))
result.update({"node_preflight": probe, "network_preflight": network, "status": "preflight-passed"})
prepared[platform] = (result, None, identity)
except EnvironmentFailure as exc:
failures[platform] = str(exc)
result.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc)})
if failures and not args.partial:
for platform in requested:
if platform not in failures:
record["platforms"][platform].update({"status": "not-run", "reason": "全量 preflight 闸门被另一平台阻断"})
overall = record.get("preflight", {}).get("status") if "preflight" in record else "environment-fail"
record.update({"status": overall if overall in {"blocked", "environment-fail"} else "environment-fail", "failure_class": "test-preflight" if "preflight" in record else "automatic-repair-failed", "next_action": "修复列出的平台环境后重跑;默认不会静默降级"})
record["results"] = all_active_result_rows(context.root, blocked_by="test-preflight")
else:
for platform, (result, node, identity) in prepared.items():
try:
if args.preflight_mode == "prepare":
# test-preflight already ran platform_probe, network_probe,
# install and start through each matrix binding.
installed = {"delegated": "test-preflight.prepare", "binding": bindings[platform]}
else:
binding_id = bindings[platform]
node_name = platform_node(context.root, binding_id)
probe = agent_runner(context.root, {
"operation": "platform_probe",
"platform": platform,
"node": node_name,
"binding": binding_id,
})
if probe.get("status") != "ready":
raise EnvironmentFailure(str(probe.get("observed") or probe))
host = os.environ.get("HOST", "").strip()
if host:
network = agent_runner(context.root, {
"operation": "network_probe",
"platform": platform,
"node": node_name,
"binding": binding_id,
"url": f"http://{host}/v1/me",
})
if network.get("status") != "ready":
raise EnvironmentFailure(str(network.get("observed") or network))
artifact_path = result["artifact"]["path"]
if node_name != "local":
artifact_path = remote_artifacts.get(platform, "")
if not artifact_path:
raise EnvironmentFailure(f"远程 {platform} 安装必须在 matrix.artifact.remote_path 声明节点可读产物")
installed = agent_runner(context.root, {
"operation": "install",
"platform": platform,
"node": node_name,
"binding": binding_id,
"artifact": artifact_path,
"identity": identity,
})
if installed.get("status") != "ready":
raise EnvironmentFailure(str(installed.get("observed") or installed))
result.update({"status": "ready", "install": installed, "failure_class": None})
except EnvironmentFailure as exc:
failures[platform] = str(exc)
result.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc)})
record.update({"status": "environment-fail" if failures else "ready", "failure_class": "automatic-repair-failed" if failures else None})
if not failures:
record["product_test"].update({"status": "pending", "failure_class": None})
record["evidence"] = context.evidence
except (EnvironmentFailure, ArtifactFailure) as exc:
record.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc), "evidence": context.evidence})
if record.get("status") != "ready" and "results" not in record:
record["results"] = all_active_result_rows(context.root, blocked_by="test-preflight")
path = write_run(context.root, record)
try:
from generate_run_csv import export_run_csv
record["csv"] = str(export_run_csv(context.root, run_path=path))
except (OSError, SystemExit, ValueError):
# The YAML run remains the source of truth when a partial/provisioning
# root has no catalog cases yet.
record["csv"] = ""
path.write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
output = {"status": record["status"], "run": str(path), "platforms": record["platforms"], "worktree": record.get("worktree")}
if record.get("error"):
output["error"] = record["error"]
if record.get("next_action"):
output["next_action"] = record["next_action"]
print(json.dumps(output, ensure_ascii=False, indent=2), file=sys.stderr if record["status"] != "ready" else sys.stdout)
return 0 if record["status"] == "ready" else 1
def parser() -> argparse.ArgumentParser:
root = argparse.ArgumentParser(prog="tester", description="Obtain and provision caller-provided mobile artifacts")
sub = root.add_subparsers(dest="command", required=True)
run = sub.add_parser("run")
run.add_argument("--artifact", action="append", dest="artifacts", default=[], metavar="PLATFORM=PATH_OR_URL", help="调用方产物,可重复,例如 android=dist/app.apk 或 ios=https://...")
run.add_argument("--platform", default="android,ios")
run.add_argument("--commit", default="", help="本轮隔离 worktree 的 40 位 commit;缺省使用当前 HEAD")
run.add_argument("--partial", action="store_true")
run.add_argument("--root", default=".")
run.add_argument("--worktree-root", default="")
run.add_argument(
"--binding",
action="append",
dest="bindings",
default=[],
metavar="PLATFORM=TESTER_BINDING",
help="每个平台的 tester binding,可重复;默认 tester.<platform>",
)
run.add_argument("--preflight-config", default="", help="test-preflight matrix YAML(必填,不得跳过)")
run.add_argument("--preflight-mode", choices=("verify", "prepare"), default="verify")
run.add_argument("--preflight-yaml", default="", help="preflight YAML 证据输出路径")
run.add_argument("--preflight-csv", default="", help="preflight CSV 证据输出路径")
run.set_defaults(func=execute)
return root
if __name__ == "__main__":
parsed = parser().parse_args()
raise SystemExit(parsed.func(parsed))
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env python3
"""Append structured tester issue evidence to an existing run snapshot."""
from __future__ import annotations
import argparse
from pathlib import Path
from typing import Any
import yaml
from failure_feedback import attach_feedback
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="将 tester 失败结构化回流到 run YAML/CSV")
parser.add_argument("--run", required=True, type=Path)
parser.add_argument("--trigger", choices=("pr", "requirement", "bug"))
parser.add_argument("--target", default="")
parser.add_argument("--related-issue", default="")
parser.add_argument("--dispatcher", default="dispatcher")
parser.add_argument("--testcase", default="", help="为一条失败结果补充 testcase issue 字段")
parser.add_argument("--issue-type", choices=("product", "test"))
parser.add_argument("--expected", default="")
parser.add_argument("--observed", default="")
parser.add_argument("--artifact-hash", default="")
parser.add_argument("--environment", default="")
parser.add_argument("--evidence-path", default="")
parser.add_argument("--reproduction", default="")
parser.add_argument("--next-step", default="")
args = parser.parse_args(argv)
data: dict[str, Any] = yaml.safe_load(args.run.read_text(encoding="utf-8")) or {}
if args.testcase:
rows = data.get("results")
if not isinstance(rows, list):
raise SystemExit("run 缺少 results 列表")
row = next((item for item in rows if isinstance(item, dict) and str(item.get("id")) == args.testcase), None)
if row is None:
raise SystemExit(f"找不到 testcase: {args.testcase}")
row["issue"] = {
key: value
for key, value in {
"issue_type": args.issue_type,
"expected": args.expected,
"observed": args.observed,
"artifact_hash": args.artifact_hash,
"environment": args.environment,
"evidence_path": args.evidence_path,
"reproduction": args.reproduction,
"next_step": args.next_step,
}.items()
if value
}
output = attach_feedback(
data,
trigger=args.trigger,
target=args.target,
related_issue=args.related_issue,
dispatcher=args.dispatcher,
)
args.run.write_text(yaml.safe_dump(output, allow_unicode=True, sort_keys=False), encoding="utf-8")
csv_path = ""
try:
from generate_run_csv import export_run_csv
csv_path = str(export_run_csv(args.run.parents[3], run_path=args.run))
except (ImportError, OSError, SystemExit, ValueError):
pass
print(f"updated {args.run} defects={len(output.get('defects') or [])} csv={csv_path}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+8 -3
View File
@@ -180,9 +180,14 @@ def clean_config(config: dict[str, Any]) -> dict[str, Any]:
def config_from_binding(root: Path, selector: str) -> dict[str, Any]:
"""Materialize a common agent-runtime tester binding for legacy local.yaml."""
runtime_path = Path(__file__).resolve().parents[3] / "skills" / "agent-runtime" / "scripts" / "agent.py"
if not runtime_path.is_file():
raise RuntimeError(f"找不到共享 agent-runtime: {runtime_path}")
candidates = [
Path.home() / ".codex" / "skills" / "agent-runtime" / "scripts" / "agent.py",
Path.home() / ".pouch" / "skills" / "agent-runtime" / "scripts" / "agent.py",
Path(__file__).resolve().parents[3] / "skills" / "agent-runtime" / "scripts" / "agent.py",
]
runtime_path = next((item for item in candidates if item.is_file()), None)
if runtime_path is None:
raise RuntimeError("找不到共享 agent-runtime: " + ", ".join(str(item) for item in candidates))
spec = importlib.util.spec_from_file_location("pouch_agent_runtime", runtime_path)
if spec is None or spec.loader is None:
raise RuntimeError(f"无法加载共享 agent-runtime: {runtime_path}")
+196
View File
@@ -0,0 +1,196 @@
#!/usr/bin/env python3
"""Validate a tester CSV/YAML pair and summarize statuses from CSV only."""
from __future__ import annotations
import argparse
import csv
import sys
from collections import Counter
from pathlib import Path
from typing import Any
try:
import yaml
except ImportError: # pragma: no cover
print("需要 PyYAML: python3 -m pip install pyyaml", file=sys.stderr)
raise SystemExit(2)
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
RESULT_SET = frozenset(RESULTS)
class SummaryError(ValueError):
"""Raised when the CSV/YAML pair cannot be safely summarized."""
def _format_ids(ids: set[str]) -> str:
return ", ".join(sorted(ids)) or "(none)"
def _duplicate_ids(ids: list[str]) -> list[str]:
counts = Counter(ids)
return sorted(case_id for case_id, count in counts.items() if count > 1)
def load_csv(path: Path) -> list[dict[str, str]]:
"""Load CSV rows and reject malformed IDs/results with diagnostics."""
try:
handle = path.open(encoding="utf-8-sig", newline="")
except OSError as exc:
raise SummaryError(f"无法读取 CSV {path}: {exc}") from exc
with handle:
reader = csv.DictReader(handle)
if not reader.fieldnames:
raise SummaryError(f"CSV {path} 缺少表头")
missing_fields = [field for field in ("id", "result") if field not in reader.fieldnames]
if missing_fields:
raise SummaryError(f"CSV {path} 缺少必需列: {', '.join(missing_fields)}")
rows = list(reader)
ids: list[str] = []
errors: list[str] = []
for line_number, row in enumerate(rows, start=2):
case_id = (row.get("id") or "").strip()
result = (row.get("result") or "").strip()
ids.append(case_id)
if not case_id:
errors.append(f"CSV 第 {line_number} 行 testcase id 缺失")
if not result:
errors.append(f"CSV 第 {line_number} 行 result 缺失 (testcase id={case_id or '(missing)'})")
elif result not in RESULT_SET:
errors.append(
f"CSV 第 {line_number} 行 result 未知: {result!r} "
f"(testcase id={case_id or '(missing)'}; 允许值={', '.join(RESULTS)})"
)
duplicates = _duplicate_ids([case_id for case_id in ids if case_id])
if duplicates:
errors.append(f"CSV testcase id 重复: {_format_ids(set(duplicates))}")
if errors:
raise SummaryError("CSV 校验失败:\n- " + "\n- ".join(errors))
return rows
def load_yaml_results(path: Path) -> list[dict[str, Any]]:
"""Load only YAML results structure needed for count/ID validation."""
try:
data = yaml.safe_load(path.read_text(encoding="utf-8"))
except OSError as exc:
raise SummaryError(f"无法读取 YAML {path}: {exc}") from exc
except yaml.YAMLError as exc:
raise SummaryError(f"YAML 解析失败 {path}: {exc}") from exc
if not isinstance(data, dict) or not isinstance(data.get("results"), list):
raise SummaryError(f"YAML {path} 缺少 results 列表")
results = data["results"]
ids: list[str] = []
errors: list[str] = []
for index, item in enumerate(results, start=1):
if not isinstance(item, dict):
errors.append(f"YAML results[{index}] 不是 mapping")
continue
case_id = str(item.get("id") or "").strip()
ids.append(case_id)
if not case_id:
errors.append(f"YAML results[{index}] testcase id 缺失")
duplicates = _duplicate_ids([case_id for case_id in ids if case_id])
if duplicates:
errors.append(f"YAML testcase id 重复: {_format_ids(set(duplicates))}")
if errors:
raise SummaryError("YAML 校验失败:\n- " + "\n- ".join(errors))
issues = data.get("defects", data.get("test_issues"))
if issues is not None:
if not isinstance(issues, list):
raise SummaryError("YAML defects/test_issues 必须是列表")
result_ids = set(ids)
issue_errors: list[str] = []
for index, issue in enumerate(issues, start=1):
if not isinstance(issue, dict):
issue_errors.append(f"YAML defects[{index}] 不是 mapping")
continue
testcase = str(issue.get("testcase") or "")
issue_type = str(issue.get("issue_type") or "")
if not testcase or testcase not in result_ids:
issue_errors.append(f"YAML defects[{index}] testcase 无效: {testcase or '(missing)'}")
if issue_type not in {"product", "test"}:
issue_errors.append(f"YAML defects[{index}] issue_type 无效: {issue_type or '(missing)'}")
for field in ("expected", "observed", "artifact_hash", "environment", "evidence_path", "reproduction", "next_step"):
if field not in issue:
issue_errors.append(f"YAML defects[{index}] 缺少 {field}")
if issue_errors:
raise SummaryError("YAML defects 校验失败:\n- " + "\n- ".join(issue_errors))
return results
def summarize(csv_path: Path, yaml_path: Path) -> tuple[Counter[str], dict[str, int]]:
"""Validate the pair and return counts calculated from CSV result values."""
rows = load_csv(csv_path)
yaml_results = load_yaml_results(yaml_path)
csv_ids = [(row.get("id") or "").strip() for row in rows]
yaml_ids = [str(item.get("id") or "").strip() for item in yaml_results]
csv_id_set = set(csv_ids)
yaml_id_set = set(yaml_ids)
diagnostics: list[str] = []
if len(rows) != len(csv_id_set):
diagnostics.append(
f"CSV 数据行总数与唯一 testcase id 数不一致: rows={len(rows)} unique_ids={len(csv_id_set)}"
)
if len(rows) != len(yaml_results):
diagnostics.append(
f"CSV 数据行总数与 YAML results 数量不一致: csv_rows={len(rows)} yaml_results={len(yaml_results)}"
)
if csv_id_set != yaml_id_set:
diagnostics.append(
"CSV/YAML testcase id 集合不一致: "
f"仅 CSV 有 [{_format_ids(csv_id_set - yaml_id_set)}]; "
f"仅 YAML 有 [{_format_ids(yaml_id_set - csv_id_set)}]"
)
if diagnostics:
raise SummaryError("CSV/YAML 汇总校验失败:\n- " + "\n- ".join(diagnostics))
counts: Counter[str] = Counter((row.get("result") or "").strip() for row in rows)
total = sum(counts[name] for name in RESULTS)
if total != len(rows):
raise SummaryError(
"CSV 状态计数和不等于数据行总数: "
f"count_sum={total} csv_rows={len(rows)}"
)
metadata = {
"csv_rows": len(rows),
"csv_unique_ids": len(csv_id_set),
"yaml_results": len(yaml_results),
"count_sum": total,
}
return counts, metadata
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description="校验 tester CSV/YAML 并从 CSV result 汇总状态")
parser.add_argument("--csv", required=True, type=Path, help="运行成绩单 CSV")
parser.add_argument("--yaml", required=True, type=Path, help="对应运行快照 YAML")
args = parser.parse_args(argv)
try:
counts, metadata = summarize(args.csv, args.yaml)
except SummaryError as exc:
print(f"汇总失败: {exc}", file=sys.stderr)
return 1
bits = " ".join(f"{name}={counts[name]}" for name in RESULTS)
print(f"summary source=CSV.result {bits} total={metadata['count_sum']}")
print(
"validated "
f"csv_rows={metadata['csv_rows']} "
f"csv_unique_ids={metadata['csv_unique_ids']} "
f"yaml_results={metadata['yaml_results']}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+185
View File
@@ -0,0 +1,185 @@
#!/usr/bin/env python3
"""Shared execution policy for tester's integration-first runs.
This module intentionally has no browser, mobile, deployment, or API driver.
Those adapters belong to the caller. It only controls the phase gate and the
stable five-state testcase result propagation so a caller cannot create a
journey session before preflight and integration have passed.
"""
from __future__ import annotations
from collections.abc import Callable, Iterable, Mapping
from typing import Any
from failure_feedback import collect_issues
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
PHASES = ("preflight", "integration", "journey")
def classify_integration(*, ok: bool, service_reachable: bool = True) -> str:
"""Separate a contract assertion failure from an unavailable service."""
if not service_reachable:
return "environment-fail"
return "pass" if ok else "fail"
def _case_id(case: Mapping[str, Any] | str) -> str:
if isinstance(case, Mapping):
return str(case.get("id") or "")
return str(case)
def _not_run(
cases: Iterable[Mapping[str, Any] | str],
reason: str,
*,
blocked_by: Iterable[str] = (),
) -> list[dict[str, Any]]:
rows: list[dict[str, str]] = []
blockers = [str(item) for item in blocked_by if str(item)]
for case in cases:
row: dict[str, Any] = {"id": _case_id(case), "result": "not-run", "notes": reason}
if blockers:
row["blocked_by"] = blockers
if isinstance(case, Mapping):
row["test_type"] = str(case.get("test_type") or "")
row["suite"] = str(
case.get("client_suite")
or case.get("integration_suite")
or case.get("suite")
or ""
)
rows.append(row)
return rows
def _dependencies(
case: Mapping[str, Any] | str,
*,
default: Iterable[str] = (),
) -> list[str]:
if not isinstance(case, Mapping):
return [str(item) for item in default]
declared = case.get("depends_on", case.get("dependencies"))
if declared is None:
return [str(item) for item in default]
if isinstance(declared, str):
return [declared]
if isinstance(declared, Iterable):
return [str(item) for item in declared if str(item)]
raise ValueError(f"depends_on 必须是字符串或列表: {_case_id(case)}")
def _failed_dependencies(dependencies: Iterable[str], results: Mapping[str, Mapping[str, Any]]) -> list[str]:
return [
dependency
for dependency in dependencies
if str((results.get(dependency) or {}).get("result"))
in {"fail", "environment-fail", "not-run"}
]
def _issue_structures(results: list[dict[str, Any]]) -> dict[str, list[dict[str, Any]]]:
issues = collect_issues(results)
return {"defects": issues, "test_issues": [dict(issue) for issue in issues]}
def run_integration_first(
*,
preflight_status: str,
integration_cases: Iterable[Mapping[str, Any] | str],
journey_cases: Iterable[Mapping[str, Any] | str],
execute_integration: Callable[[Mapping[str, Any] | str], str | Mapping[str, Any]],
create_journey_session: Callable[[], Any],
execute_journey: Callable[[Any, Mapping[str, Any] | str], str | Mapping[str, Any]],
) -> dict[str, Any]:
"""Execute integration cases before creating any journey session.
``preflight_status`` must be the status emitted by test-preflight. A
blocked/environment-fail gate records only ``not-run`` testcase rows and
never invokes either adapter. An integration failure stops the phase and
propagates ``not-run`` to all journey cases.
"""
integration = list(integration_cases)
journey = list(journey_cases)
if preflight_status != "ready":
reason = f"test-preflight={preflight_status},未创建客户端 session"
result_rows = _not_run(integration, reason, blocked_by=("test-preflight",)) + _not_run(journey, reason, blocked_by=("test-preflight",))
return {
"status": preflight_status,
"phase": "preflight",
"session_created": False,
"results": result_rows,
**_issue_structures(result_rows),
}
results: list[dict[str, Any]] = []
results_by_id: dict[str, dict[str, Any]] = {}
for case in integration:
case_id = _case_id(case)
dependencies = _dependencies(case)
blocked_by = _failed_dependencies(dependencies, results_by_id)
if blocked_by:
rows = _not_run(
[case],
"依赖的 integration 失败,未执行",
blocked_by=blocked_by,
)
results.extend(rows)
results_by_id[case_id] = rows[0]
continue
outcome = execute_integration(case)
row = dict(outcome) if isinstance(outcome, Mapping) else {"result": outcome}
row["id"] = _case_id(case)
if isinstance(case, Mapping):
row.setdefault("test_type", str(case.get("test_type") or "integration"))
row.setdefault("suite", str(case.get("integration_suite") or case.get("suite") or ""))
result = str(row.get("result") or "not-run")
if result not in RESULTS:
raise ValueError(f"未知 integration 结果: {row['id']}={result}")
row["result"] = result
results.append(row)
results_by_id[case_id] = row
integration_ids = [_case_id(case) for case in integration]
has_runnable_journey = any(
not _failed_dependencies(_dependencies(case, default=integration_ids), results_by_id)
for case in journey
)
session = create_journey_session() if has_runnable_journey else None
for case in journey:
dependencies = _dependencies(case, default=integration_ids)
blocked_by = _failed_dependencies(dependencies, results_by_id)
if blocked_by:
row = _not_run(
[case],
"依赖的用例失败,journey 未执行",
blocked_by=blocked_by,
)[0]
results.append(row)
results_by_id[_case_id(case)] = row
continue
outcome = execute_journey(session, case)
row = dict(outcome) if isinstance(outcome, Mapping) else {"result": outcome}
row["id"] = _case_id(case)
if isinstance(case, Mapping):
row.setdefault("test_type", str(case.get("test_type") or "journey"))
row.setdefault("suite", str(case.get("client_suite") or case.get("suite") or ""))
result = str(row.get("result") or "not-run")
if result not in RESULTS:
raise ValueError(f"未知 journey 结果: {row['id']}={result}")
row["result"] = result
results.append(row)
results_by_id[_case_id(case)] = row
status = "fail" if any(row["result"] == "fail" for row in results) else (
"environment-fail" if any(row["result"] == "environment-fail" for row in results) else "pass"
)
return {
"status": status,
"phase": "journey" if journey else "integration",
"session_created": has_runnable_journey,
"results": results,
**_issue_structures(results),
}
+18
View File
@@ -22,6 +22,24 @@ risks:
- mutating
- destructive
default_max_risk: readonly
# standard enforces the default_max_risk/authorize policy. Writes-style isolated
# resettable environments may set this to isolated-full; risk values remain on
# cases and are still reported.
execution_profile: standard
client_suites:
client-smoke:
default: true
client-regression:
default: false
safety:
default: false
integration_suites:
api-contract:
default: false
authorization:
default: false
build-contract:
default: false
vars: {}
authorize:
mutating: AUTHORIZE_MUTATING
+2
View File
@@ -0,0 +1,2 @@
#!/bin/sh
exec python3 "$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)/scripts/mobile_bundle.py" "$@"
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env python3
import json
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
import yaml
ROOT = Path(__file__).resolve().parents[1]
CATALOG = ROOT / "scripts" / "catalog.py"
def write_fixture(root: Path, *, complete: bool = True) -> None:
tester = root / ".pouch" / "tester"
(tester / "cases").mkdir(parents=True)
(tester / "runs").mkdir()
(tester / "catalog.yaml").write_text(
yaml.safe_dump(
{
"version": 2,
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
"categories": [{"id": "CLIENT"}],
"risks": ["readonly", "mutating", "destructive"],
"default_max_risk": "readonly",
"client_suites": {
"client-smoke": {"default": True},
"client-regression": {"default": False},
"safety": {"default": False},
},
"sessions": {"browser": {"kind": "browser", "reuse": "run"}},
},
sort_keys=False,
),
encoding="utf-8",
)
case = {
"id": "TC-CLIENT-001",
"category": "CLIENT",
"title": "journey",
"what": "client journey",
"lifecycle": "active",
"tag": "smoke",
"risk": "readonly",
"driver": "browser",
"session": "browser",
"client_suite": "client-smoke",
"platform": ["android", "ios"],
"journey": {
"goal": "goal",
"entry_state": "logged_out",
"actions": ["start", "login"],
"assertions": ["visible"],
"final_state": "done",
"failure_policy": "stop",
},
"fixture": {"account": "test-role"},
"cleanup": {"steps": ["cleanup"]},
"how": {"steps": ["start", "login"]},
"expected": [{"kind": "visible-text", "value": "done"}],
}
if not complete:
case.pop("fixture")
case.pop("cleanup")
(tester / "cases" / "client.yaml").write_text(
yaml.safe_dump({"category": "CLIENT", "cases": [case]}, sort_keys=False),
encoding="utf-8",
)
class CatalogJourneyTests(unittest.TestCase):
def run_catalog(self, root: Path, *args: str):
return subprocess.run(
[sys.executable, str(CATALOG), *args, "--root", str(root)],
text=True,
capture_output=True,
)
def test_valid_client_catalog_and_suite_filter(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
write_fixture(root)
valid = self.run_catalog(root, "validate")
self.assertEqual(valid.returncode, 0, valid.stderr)
selected = self.run_catalog(root, "select", "--suite", "client-smoke")
self.assertEqual(selected.returncode, 0, selected.stderr)
self.assertEqual(json.loads(selected.stdout)["count"], 1)
def test_missing_fixture_or_cleanup_blocks_before_results(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
write_fixture(root, complete=False)
selected = self.run_catalog(root, "select", "--suite", "client-smoke")
self.assertNotEqual(selected.returncode, 0)
payload = json.loads(selected.stdout)
self.assertEqual(payload["preflight"]["failure_class"], "contract")
self.assertTrue(any("fixture 缺失" in error for error in payload["preflight"]["errors"]))
self.assertEqual(payload["preflight"]["status"], "blocked")
if __name__ == "__main__":
unittest.main()
+69
View File
@@ -0,0 +1,69 @@
#!/usr/bin/env python3
import sys
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
from journey import classify_outcome, contract_errors, propagate_results
def valid_case(**overrides):
case = {
"id": "TC-CLIENT-001",
"client_suite": "client-smoke",
"driver": "browser",
"session": "browser",
"platform": ["android", "ios"],
"journey": {
"goal": "完成客户端目标",
"entry_state": "logged_out",
"actions": ["启动", "登录", "完成目标"],
"assertions": ["看到目标页面"],
"final_state": "目标已完成",
"failure_policy": "stop",
},
"fixture": {"account": "test-role", "data": []},
"cleanup": {"steps": ["清理本轮数据"]},
}
case.update(overrides)
return case
class JourneyContractTests(unittest.TestCase):
def test_contract_accepts_shared_client_shape(self):
self.assertEqual(contract_errors(valid_case(), {"browser": {}}), [])
def test_missing_fixture_and_cleanup_are_contract_errors(self):
errors = contract_errors(
valid_case(fixture=None, cleanup=None),
{"browser": {}},
)
self.assertTrue(any("fixture 缺失" in error for error in errors))
self.assertTrue(any("cleanup 缺失" in error for error in errors))
def test_login_failure_is_product_failure_but_unavailable_environment_is_not(self):
self.assertEqual(classify_outcome(ok=False, phase="login"), "fail")
self.assertEqual(
classify_outcome(ok=False, phase="login", environment_ready=False),
"environment-fail",
)
def test_failed_case_blocks_remaining_cases_and_keeps_explicit_skip(self):
rows = propagate_results(
["TC-1", "TC-2", "TC-3"],
{"TC-1": "fail", "TC-2": "pass", "TC-3": "pass"},
skipped={"TC-3": "未授权"},
)
self.assertEqual([row["result"] for row in rows], ["fail", "not-run", "skipped"])
def test_continue_policy_allows_next_case(self):
rows = propagate_results(
["TC-1", "TC-2"],
{"TC-1": "fail", "TC-2": "pass"},
policies={"TC-1": "continue"},
)
self.assertEqual([row["result"] for row in rows], ["fail", "pass"])
if __name__ == "__main__":
unittest.main()
+220
View File
@@ -0,0 +1,220 @@
#!/usr/bin/env python3
import json
import plistlib
import subprocess
import tempfile
import unittest
import zipfile
from pathlib import Path
from unittest import mock
import sys
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
import mobile_bundle as bundle
def init_repo(root: Path) -> str:
run = lambda args: subprocess.run(args, check=True, stdout=subprocess.DEVNULL)
run(["git", "init", "-q", str(root)])
run(["git", "-C", str(root), "config", "user.email", "tester@example.invalid"])
run(["git", "-C", str(root), "config", "user.name", "Tester"])
(root / "marker").write_text("x")
run(["git", "-C", str(root), "add", "marker"])
run(["git", "-C", str(root), "commit", "-qm", "bundle"])
return subprocess.check_output(["git", "-C", str(root), "rev-parse", "HEAD"], text=True).strip()
class MobileBundleTests(unittest.TestCase):
def test_artifact_locator_is_required(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp) / "repo"
root.mkdir()
init_repo(root)
args = bundle.parser().parse_args(["run", "--platform", "android,ios", "--root", str(root)])
self.assertEqual(bundle.execute(args), 1)
runs = list((root / ".pouch" / "tester" / "runs").glob("*.yaml"))
self.assertEqual(len(runs), 1)
record = json.loads(runs[0].read_text(encoding="utf-8"))
self.assertIn("缺少调用方产物位置", record["error"])
def test_parse_artifacts_accepts_file_and_url(self):
parsed = bundle.parse_artifacts([
"android=/tmp/app.apk",
"ios=https://ci.example/app.ipa",
])
self.assertEqual(parsed["android"], "/tmp/app.apk")
self.assertEqual(parsed["ios"], "https://ci.example/app.ipa")
with self.assertRaises(bundle.ArtifactFailure):
bundle.parse_artifacts(["android"])
with self.assertRaises(bundle.ArtifactFailure):
bundle.parse_artifacts(["android=/tmp/a.apk", "android=/tmp/b.apk"])
def test_parse_bindings_keeps_platforms_separate(self):
self.assertEqual(
bundle.parse_bindings(["android=tester.android", "ios=tester.ios"], ["android", "ios"]),
{"android": "tester.android", "ios": "tester.ios"},
)
self.assertEqual(
bundle.parse_bindings([], ["android"]),
{"android": "tester.android"},
)
with self.assertRaises(bundle.ArtifactFailure):
bundle.parse_bindings(["android=tester.test", "ios=tester.test"], ["android", "ios"])
def test_preflight_matrix_rejects_remote_platform_using_local_runner(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
agent_dir = root / ".pouch"
agent_dir.mkdir()
(agent_dir / "agent.yaml").write_text(
"""
schemaVersion: 1
locations:
local-app:
environment: local
host: local
project: test
path: /tmp/test
remote-ios:
environment: macmini
host: runtime:12345678-1234-1234-1234-123456789012
project: test
path: /tmp/test-ios
profiles:
test-agent:
role: test
cli: codex
tier: standard
model: gpt-5
reasoningEffort: medium
permissionMode: workspace-write
bindings:
tester.android: {adapter: tester, location: local-app, profile: test-agent}
tester.ios: {adapter: tester, location: remote-ios, profile: test-agent}
""".strip()
+ "\n",
encoding="utf-8",
)
config = root / "matrix.yaml"
config.write_text(
"""
matrix:
- platform: android
tester: {binding: tester.android, node: local, runner: {kind: local}}
- platform: ios
tester: {binding: tester.ios, node: local, runner: {kind: local}}
""".strip()
+ "\n",
encoding="utf-8",
)
with self.assertRaisesRegex(bundle.EnvironmentFailure, "ios.*contract runner"):
bundle.validate_preflight_matrix(
root,
config,
{"android": "tester.android", "ios": "tester.ios"},
["android", "ios"],
)
def test_remote_artifact_path_comes_from_preflight_matrix(self):
with tempfile.TemporaryDirectory() as temp:
config = Path(temp) / "matrix.yaml"
config.write_text(
"matrix:\n - platform: ios\n artifact:\n remote_path: /Users/tester/app.ipa\n",
encoding="utf-8",
)
self.assertEqual(
bundle.preflight_remote_artifacts(config, ["ios"]),
{"ios": "/Users/tester/app.ipa"},
)
def test_new_worktree_each_run(self):
with tempfile.TemporaryDirectory() as temp:
repo, worktrees = Path(temp) / "repo", Path(temp) / "worktrees"
repo.mkdir()
commit = init_repo(repo)
first, first_commit = bundle.ensure_worktree(commit, bundle.RuntimeContext(repo), worktrees)
second, second_commit = bundle.ensure_worktree(commit, bundle.RuntimeContext(repo), worktrees)
self.assertEqual(first_commit, commit)
self.assertEqual(second_commit, commit)
self.assertNotEqual(first, second)
self.assertTrue(first.is_dir())
self.assertTrue(second.is_dir())
def test_ipa_identity_and_install(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
ipa = root / "app.ipa"
with zipfile.ZipFile(ipa, "w") as archive:
archive.writestr("Payload/Writes.app/Info.plist", plistlib.dumps({"CFBundleIdentifier": "page.writes"}))
self.assertEqual(bundle.ios_identity(ipa), "page.writes")
context = bundle.RuntimeContext(root)
calls = []
def fake(args, **kwargs):
calls.append(args)
return bundle.CommandResult(0, "/container" if "get_app_container" in args else "")
with mock.patch.object(bundle, "run_command", side_effect=fake):
result = bundle.install_ios({"xcrun": "xcrun", "udid": "UDID", "name": "Test"}, ipa, "page.writes", context)
self.assertTrue(result["verified"])
self.assertTrue(any("install" in call for call in calls))
def test_android_identity_accepts_bare_apkanalyzer_output(self):
with tempfile.TemporaryDirectory() as temp:
apk = Path(temp) / "app.apk"
apk.write_bytes(b"apk")
context = bundle.RuntimeContext(Path(temp))
with mock.patch.object(bundle.shutil, "which", side_effect=lambda name: "apkanalyzer" if name == "apkanalyzer" else None), mock.patch.object(
bundle, "run_command", return_value=bundle.CommandResult(0, "page.writes\n")
):
self.assertEqual(bundle.android_identity(apk, context), "page.writes")
def test_android_missing_device_is_environment_failure(self):
context = bundle.RuntimeContext(Path("."))
with mock.patch.object(bundle.shutil, "which", return_value="/sdk/adb"), mock.patch.object(
bundle, "run_command", side_effect=[bundle.CommandResult(0, ""), bundle.CommandResult(0, "List of devices attached")]
):
with self.assertRaises(bundle.EnvironmentFailure):
bundle.android_preflight(context)
def test_network_failure_is_actionable(self):
context = bundle.RuntimeContext(Path("."))
with mock.patch.object(bundle.urllib.request, "urlopen", side_effect=OSError("offline")), mock.patch.object(bundle.time, "sleep"):
with mock.patch.dict(bundle.os.environ, {"HOST": "test.invalid"}, clear=False):
with self.assertRaisesRegex(bundle.EnvironmentFailure, "VPN"):
bundle.check_network(context)
def test_retry_recovers(self):
context = bundle.RuntimeContext(Path("."))
responses = [bundle.CommandResult(1, "busy"), bundle.CommandResult(0, "ok")]
with mock.patch.object(bundle, "run_command", side_effect=responses), mock.patch.object(bundle.time, "sleep"):
self.assertEqual(bundle.retry_command(["fake"], context=context, action="install").code, 0)
def test_full_gate_blocks_install_when_ios_fails(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp) / "repo"
root.mkdir()
commit = init_repo(root)
apk, ipa = Path(temp) / "a.apk", Path(temp) / "a.ipa"
apk.write_bytes(b"apk")
ipa.write_bytes(b"ipa")
args = bundle.parser().parse_args([
"run",
"--artifact", f"android={apk}",
"--artifact", f"ios={ipa}",
"--root", str(root),
"--worktree-root", str(Path(temp) / "wt"),
])
with mock.patch.object(bundle, "android_identity", return_value="page.writes"), mock.patch.object(bundle, "ios_identity", return_value="page.writes"), mock.patch.object(bundle, "android_preflight", return_value={"adb": "adb", "serial": "device"}), mock.patch.object(bundle, "ios_preflight", side_effect=bundle.EnvironmentFailure("Mac unavailable")), mock.patch.object(bundle, "check_network"), mock.patch.object(bundle, "install_android") as install:
self.assertEqual(bundle.execute(args), 1)
install.assert_not_called()
self.assertEqual(args.commit, "")
self.assertTrue(COMMIT_HEAD(root, commit))
def COMMIT_HEAD(root: Path, commit: str) -> bool:
actual = subprocess.check_output(["git", "-C", str(root), "rev-parse", "HEAD"], text=True).strip()
return actual == commit
if __name__ == "__main__":
unittest.main()
+129
View File
@@ -0,0 +1,129 @@
#!/usr/bin/env python3
import csv
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
import summarize_run
class RunSummaryTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.root = Path(self.temp.name)
self.csv_path = self.root / "run.csv"
self.yaml_path = self.root / "run.yaml"
self.rows = [
{"id": "TC-001", "result": "pass"},
{"id": "TC-002", "result": "fail"},
{"id": "TC-003", "result": "environment-fail"},
{"id": "TC-004", "result": "skipped"},
{"id": "TC-005", "result": "not-run"},
]
self.write_csv(self.rows)
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows])
def tearDown(self):
self.temp.cleanup()
def write_csv(self, rows):
with self.csv_path.open("w", encoding="utf-8", newline="") as handle:
writer = csv.DictWriter(handle, fieldnames=["id", "result"])
writer.writeheader()
writer.writerows(rows)
def write_yaml(self, results):
import yaml
self.yaml_path.write_text(
yaml.safe_dump(
{"counts": {"pass": 999}, "results": results},
sort_keys=False,
),
encoding="utf-8",
)
def run_cli(self):
return subprocess.run(
[
sys.executable,
str(Path(summarize_run.__file__).resolve()),
"--csv",
str(self.csv_path),
"--yaml",
str(self.yaml_path),
],
text=True,
capture_output=True,
)
def test_valid_summary_uses_csv_results(self):
result = self.run_cli()
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("source=CSV.result", result.stdout)
self.assertIn("pass=1 fail=1 environment-fail=1 skipped=1 not-run=1 total=5", result.stdout)
def test_yaml_counts_do_not_drive_status_summary(self):
self.write_csv([{"id": "TC-001", "result": "fail"}])
self.write_yaml([{"id": "TC-001", "result": "fail"}])
result = self.run_cli()
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("pass=0 fail=1", result.stdout)
self.assertNotIn("pass=999", result.stdout)
def test_yaml_result_count_mismatch_is_nonzero(self):
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows[:-1]])
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn("YAML results 数量不一致", result.stderr)
def test_id_set_mismatch_is_nonzero(self):
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows[:-1]] + [{"id": "TC-999"}])
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn("testcase id 集合不一致", result.stderr)
def test_duplicate_csv_id_is_nonzero(self):
rows = self.rows.copy()
rows[-1] = {"id": "TC-001", "result": "not-run"}
self.write_csv(rows)
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn("CSV testcase id 重复", result.stderr)
def test_missing_csv_id_is_nonzero(self):
rows = self.rows.copy()
rows[0] = {"id": "", "result": "pass"}
self.write_csv(rows)
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn("CSV 第 2 行 testcase id 缺失", result.stderr)
def test_duplicate_or_missing_yaml_id_is_nonzero(self):
cases = [
([{"id": "TC-001"}, {"id": "TC-001"}] + [{"id": row["id"]} for row in self.rows[2:]], "YAML testcase id 重复"),
([{"id": ""}] + [{"id": row["id"]} for row in self.rows[1:]], "YAML results[1] testcase id 缺失"),
]
for results, expected in cases:
with self.subTest(expected=expected):
self.write_yaml(results)
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn(expected, result.stderr)
def test_unknown_or_missing_csv_result_is_nonzero(self):
for bad_result, expected in [("unexpected", "result 未知"), ("", "result 缺失")]:
with self.subTest(bad_result=bad_result):
rows = self.rows.copy()
rows[0] = {"id": "TC-001", "result": bad_result}
self.write_csv(rows)
result = self.run_cli()
self.assertNotEqual(result.returncode, 0)
self.assertIn(expected, result.stderr)
if __name__ == "__main__":
unittest.main()
+299
View File
@@ -0,0 +1,299 @@
import csv
import json
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
import yaml
SCRIPTS = Path(__file__).resolve().parents[1] / "scripts"
CATALOG = SCRIPTS / "catalog.py"
CASES_CSV = SCRIPTS / "generate_cases_csv.py"
RUN_CSV = SCRIPTS / "generate_run_csv.py"
sys.path.insert(0, str(SCRIPTS))
from two_tier import classify_integration, run_integration_first
def write_mixed_catalog(root: Path) -> None:
tester = root / ".pouch" / "tester"
(tester / "cases").mkdir(parents=True)
(tester / "runs").mkdir()
(tester / "catalog.yaml").write_text(
yaml.safe_dump(
{
"version": 2,
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
"categories": [{"id": "MIXED"}],
"risks": ["readonly", "mutating", "destructive"],
"default_max_risk": "readonly",
"client_suites": {
"client-smoke": {"default": True},
"client-regression": {"default": False},
"safety": {"default": False},
},
"sessions": {"browser": {"kind": "browser", "reuse": "run"}},
},
sort_keys=False,
),
encoding="utf-8",
)
journey = {
"id": "TC-JOURNEY-001",
"test_type": "journey",
"category": "MIXED",
"title": "journey",
"what": "client path",
"lifecycle": "active",
"tag": "smoke",
"risk": "readonly",
"driver": "browser",
"automation": "automated",
"session": "browser",
"client_suite": "client-smoke",
"platform": "android",
"journey": {
"goal": "goal",
"entry_state": "logged_out",
"actions": ["login"],
"assertions": ["visible"],
"final_state": "done",
"failure_policy": "stop",
},
"fixture": {"account": "test-role"},
"cleanup": {"required": False, "reason": "read-only"},
"how": {"steps": ["open"]},
"expected": [{"kind": "visible-text", "value": "done"}],
}
integration = {
"id": "TC-INTEGRATION-001",
"test_type": "integration",
"category": "MIXED",
"title": "contract",
"what": "API contract",
"lifecycle": "active",
"tag": "slow",
"risk": "readonly",
"driver": "script",
"automation": "automated",
"integration_suite": "api-contract",
"fixture": {"references": ["isolated-account"]},
"request": {"method": "GET", "path": "/health"},
"how": {"run": "true"},
"cleanup": {"required": False, "reason": "read-only"},
"expected": [{"kind": "api-status", "value": 200}],
}
(tester / "cases" / "mixed.yaml").write_text(
yaml.safe_dump({"category": "MIXED", "cases": [journey, integration]}, sort_keys=False),
encoding="utf-8",
)
class TwoTierCatalogTests(unittest.TestCase):
def run_catalog(self, root: Path, *args: str) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, str(CATALOG), *args, "--root", str(root)],
text=True,
capture_output=True,
)
def test_default_selection_excludes_integration_and_explicit_selection_includes_it(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
write_mixed_catalog(root)
valid = self.run_catalog(root, "validate")
self.assertEqual(valid.returncode, 0, valid.stderr)
default = self.run_catalog(root, "select")
self.assertEqual(default.returncode, 0, default.stderr)
default_payload = json.loads(default.stdout)
self.assertEqual(default_payload["count"], 1)
self.assertEqual(default_payload["test_types"], {"journey": 1, "integration": 0})
integration = self.run_catalog(root, "select", "--test-type", "integration")
self.assertEqual(integration.returncode, 0, integration.stderr)
integration_payload = json.loads(integration.stdout)
self.assertEqual(integration_payload["count"], 1)
self.assertEqual(integration_payload["cases"][0]["test_type"], "integration")
csv_result = subprocess.run(
[sys.executable, str(CASES_CSV), "--root", str(root)],
text=True,
capture_output=True,
)
self.assertEqual(csv_result.returncode, 0, csv_result.stderr)
with (root / ".pouch/tester/cases.csv").open(encoding="utf-8-sig", newline="") as handle:
rows = list(csv.DictReader(handle))
self.assertEqual({row["test_type"] for row in rows}, {"journey", "integration"})
self.assertEqual({row["suite"] for row in rows}, {"client-smoke", "api-contract"})
run_path = root / ".pouch/tester/runs/two-tier.yaml"
run_path.write_text(
yaml.safe_dump(
{"runId": "two-tier", "results": [
{"id": "TC-INTEGRATION-001", "result": "pass"},
{"id": "TC-JOURNEY-001", "result": "not-run"},
]},
sort_keys=False,
),
encoding="utf-8",
)
run_csv = subprocess.run(
[sys.executable, str(RUN_CSV), "--root", str(root), "--run", str(run_path)],
text=True,
capture_output=True,
)
self.assertEqual(run_csv.returncode, 0, run_csv.stderr)
with run_path.with_suffix(".csv").open(encoding="utf-8-sig", newline="") as handle:
report_rows = list(csv.DictReader(handle))
self.assertEqual(report_rows[0]["test_type"], "integration")
self.assertEqual(report_rows[0]["suite"], "api-contract")
def test_isolated_full_keeps_risk_and_ignores_authorize_filter(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
write_mixed_catalog(root)
catalog_path = root / ".pouch/tester/catalog.yaml"
catalog = yaml.safe_load(catalog_path.read_text(encoding="utf-8"))
catalog["authorize"] = {"mutating": "AUTHORIZE_MUTATING"}
catalog["execution_profile"] = "isolated-full"
catalog_path.write_text(yaml.safe_dump(catalog, sort_keys=False), encoding="utf-8")
case_path = root / ".pouch/tester/cases/mixed.yaml"
cases = yaml.safe_load(case_path.read_text(encoding="utf-8"))
journey = cases["cases"][0]
journey.update({"risk": "mutating", "tag": "full", "client_suite": "client-regression"})
case_path.write_text(yaml.safe_dump(cases, sort_keys=False), encoding="utf-8")
selected = self.run_catalog(root, "select", "--suite", "client-regression")
self.assertEqual(selected.returncode, 0, selected.stderr)
payload = json.loads(selected.stdout)
self.assertEqual(payload["execution_profile"], "isolated-full")
self.assertEqual(payload["risk_filter"], "disabled")
self.assertEqual(payload["count"], 1)
self.assertEqual(payload["cases"][0]["risk"], "mutating")
self.assertEqual(payload["skipped"], [])
all_active = self.run_catalog(root, "select")
self.assertEqual(all_active.returncode, 0, all_active.stderr)
all_payload = json.loads(all_active.stdout)
self.assertEqual(all_payload["count"], 2)
self.assertEqual(all_payload["test_types"], {"journey": 1, "integration": 1})
def test_isolated_full_run_script_does_not_use_authorize_switch(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
tester = root / ".pouch" / "tester"
(tester / "cases").mkdir(parents=True)
(tester / "local.yaml").write_text("HOST: http://127.0.0.1\nAUTHORIZE_MUTATING: false\n", encoding="utf-8")
(tester / "catalog.yaml").write_text(
yaml.safe_dump(
{
"version": 2,
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
"categories": [{"id": "SCRIPT"}],
"risks": ["readonly", "mutating", "destructive"],
"default_max_risk": "readonly",
"execution_profile": "isolated-full",
"authorize": {"mutating": "AUTHORIZE_MUTATING"},
},
sort_keys=False,
),
encoding="utf-8",
)
(tester / "cases" / "script.yaml").write_text(
yaml.safe_dump(
{
"category": "SCRIPT",
"cases": [{
"id": "TC-SCRIPT-001",
"title": "mutating script",
"what": "run isolated script",
"lifecycle": "active",
"tag": "full",
"risk": "mutating",
"driver": "script",
"how": {"run": "exit 0"},
"cleanup": {"steps": ["reset"]},
"expected": [{"kind": "script-exit", "value": 0}],
}],
},
sort_keys=False,
),
encoding="utf-8",
)
result = subprocess.run(
[sys.executable, str(CATALOG), "run-script", "--root", str(root), "--id", "TC-SCRIPT-001"],
text=True,
capture_output=True,
)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(json.loads(result.stdout)["result"], "executed")
class IntegrationFirstTests(unittest.TestCase):
def test_preflight_block_does_not_create_session(self):
calls: list[str] = []
result = run_integration_first(
preflight_status="blocked",
integration_cases=["INT-1"],
journey_cases=["J-1"],
execute_integration=lambda case: calls.append("integration") or "pass",
create_journey_session=lambda: calls.append("session"),
execute_journey=lambda session, case: "pass",
)
self.assertEqual(result["results"][0]["result"], "not-run")
self.assertFalse(result["session_created"])
self.assertEqual(calls, [])
def test_integration_failure_propagates_journey_not_run(self):
calls: list[str] = []
result = run_integration_first(
preflight_status="ready",
integration_cases=["INT-1", "INT-2"],
journey_cases=["J-1", "J-2"],
execute_integration=lambda case: calls.append(case) or ("fail" if case == "INT-1" else "pass"),
create_journey_session=lambda: calls.append("session"),
execute_journey=lambda session, case: "pass",
)
self.assertEqual(result["status"], "fail")
self.assertEqual(calls, ["INT-1", "INT-2"])
self.assertFalse(result["session_created"])
self.assertEqual([row["result"] for row in result["results"]], ["fail", "pass", "not-run", "not-run"])
self.assertEqual(result["results"][2]["blocked_by"], ["INT-1"])
def test_independent_integration_continues_and_dependency_is_blocked(self):
result = run_integration_first(
preflight_status="ready",
integration_cases=[
{"id": "INT-1"},
{"id": "INT-2"},
{"id": "INT-3", "depends_on": ["INT-1"]},
],
journey_cases=[],
execute_integration=lambda case: "fail" if case["id"] == "INT-1" else "pass",
create_journey_session=lambda: object(),
execute_journey=lambda session, case: "pass",
)
self.assertEqual([row["result"] for row in result["results"]], ["fail", "pass", "not-run"])
self.assertEqual(result["results"][2]["blocked_by"], ["INT-1"])
def test_preflight_block_creates_every_result_row(self):
result = run_integration_first(
preflight_status="blocked",
integration_cases=[{"id": "INT-1"}],
journey_cases=[{"id": "J-1"}],
execute_integration=lambda case: "pass",
create_journey_session=lambda: object(),
execute_journey=lambda session, case: "pass",
)
self.assertEqual([row["result"] for row in result["results"]], ["not-run", "not-run"])
self.assertEqual(result["results"][0]["blocked_by"], ["test-preflight"])
def test_service_unreachable_is_environment_failure(self):
self.assertEqual(classify_integration(ok=False, service_reachable=False), "environment-fail")
self.assertEqual(classify_integration(ok=False, service_reachable=True), "fail")
if __name__ == "__main__":
unittest.main()
+43 -6
View File
@@ -112,6 +112,9 @@ class BuilderCheckTests(unittest.TestCase):
self.assertIn("copy_artifact.sh", skill)
self.assertIn("android", skill.lower())
self.assertIn("COPY_DEST", contract)
self.assertIn("init_ci_status.sh", skill)
self.assertIn("CI_STATUS_SCRIPT", contract)
self.assertIn("ci-status.md", skill)
def test_no_makefile_without_ready_is_usage_error(self) -> None:
with tempfile.TemporaryDirectory() as temp:
@@ -213,28 +216,62 @@ class BuilderCheckTests(unittest.TestCase):
self.assertIn("[FAIL] 10. 轨道工具链", text)
self.assertIn("docker: MISSING", text)
def test_ready_uses_curl_for_gitea_actions(self) -> None:
def test_ready_ci_requires_status_script(self) -> None:
with tempfile.TemporaryDirectory() as temp:
project = Path(temp)
init_repo(project)
write_contract_makefile(project)
(project / "Dockerfile").write_text("FROM scratch\n", encoding="utf-8")
git(project, "remote", "add", "origin", "https://gitea.example.test/acme/widget.git")
code, text = run_check(
project,
"--ready",
env={"BUILD_VENUE": "ci"},
which={
"docker": "/usr/bin/docker",
"curl": "/usr/bin/curl",
"curl": None,
"gh": None,
"glab": None,
"tea": None,
},
)
self.assertEqual(code, 1, text)
self.assertIn("[FAIL] 15. CI_STATUS_SCRIPT 可执行", text)
self.assertIn("does not point to a file", text)
self.assertIn("CI status: project CI_STATUS_SCRIPT", text)
self.assertNotIn("curl: found", text)
self.assertNotIn("Gitea API", text)
def test_ready_ci_passes_with_executable_status_script(self) -> None:
with tempfile.TemporaryDirectory() as temp:
project = Path(temp)
init_repo(project)
write_contract_makefile(project)
(project / "Dockerfile").write_text("FROM scratch\n", encoding="utf-8")
script = project / ".pouch" / "builder" / "ci-status.sh"
script.parent.mkdir(parents=True)
script.write_text("#!/bin/sh\necho STATUS=empty\n", encoding="utf-8")
script.chmod(0o755)
code, text = run_check(
project,
"--ready",
env={"BUILD_VENUE": "ci"},
which={"docker": "/usr/bin/docker", "curl": None, "gh": None, "glab": None},
)
self.assertEqual(code, 0, text)
self.assertIn("curl: found", text)
self.assertIn("Gitea API: /api/v1", text)
self.assertNotIn("glab: ", text)
self.assertIn("[PASS] 15. CI_STATUS_SCRIPT 可执行", text)
self.assertIn("CI status: project CI_STATUS_SCRIPT", text)
def test_ready_local_skips_status_script(self) -> None:
with tempfile.TemporaryDirectory() as temp:
project = Path(temp)
init_repo(project)
write_contract_makefile(project)
code, text = run_check(
project, "--ready", which={"dpkg-deb": "/usr/bin/dpkg-deb"}
)
self.assertEqual(code, 0, text)
self.assertIn("[SKIP] 15. CI_STATUS_SCRIPT 可执行", text)
self.assertIn("BUILD_VENUE is not ci", text)
def test_user_makefile_does_not_satisfy_contract(self) -> None:
with tempfile.TemporaryDirectory() as temp:
+124 -152
View File
@@ -1,6 +1,5 @@
from __future__ import annotations
import json
import os
import subprocess
import tempfile
@@ -9,7 +8,10 @@ from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
SCRIPT = REPO_ROOT / "skills" / "builder" / "scripts" / "wait_ci.sh"
SCRIPTS = REPO_ROOT / "skills" / "builder" / "scripts"
WAIT_CI = SCRIPTS / "wait_ci.sh"
INIT_CI = SCRIPTS / "init_ci_status.sh"
TEMPLATES = REPO_ROOT / "skills" / "builder" / "templates" / "ci-status"
def git(cwd: Path, *args: str) -> str:
@@ -33,7 +35,7 @@ def git(cwd: Path, *args: str) -> str:
class BuilderCiTests(unittest.TestCase):
def make_repo(self, root: Path, origin: str) -> str:
def make_repo(self, root: Path, origin: str = "https://gitea.example.test/acme/widget.git") -> str:
git(root, "init", "-b", "main")
git(root, "config", "user.email", "test@example.com")
git(root, "config", "user.name", "Test")
@@ -43,6 +45,27 @@ class BuilderCiTests(unittest.TestCase):
git(root, "remote", "add", "origin", origin)
return git(root, "rev-parse", "HEAD")
def write_makefile(self, root: Path, venue: str = "ci") -> None:
(root / "makefile.builder").write_text(
"BUILD_VENUE ?= "
+ venue
+ "\n"
"CI_STATUS_SCRIPT ?= .pouch/builder/ci-status.sh\n"
"print-%:\n"
"\t@echo $($*)\n"
"venue:\n"
"\t@echo $(BUILD_VENUE)\n",
encoding="utf-8",
)
def write_status_script(self, root: Path, body: str, executable: bool = True) -> Path:
path = root / ".pouch" / "builder" / "ci-status.sh"
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("#!/bin/sh\n" + body + "\n", encoding="utf-8")
if executable:
path.chmod(0o755)
return path
def fake_command(self, bin_dir: Path, name: str, body: str) -> None:
command = bin_dir / name
command.write_text(f"#!/bin/sh\n{body}\n", encoding="utf-8")
@@ -52,20 +75,22 @@ class BuilderCiTests(unittest.TestCase):
self,
root: Path,
sha: str,
bin_dir: Path,
bin_dir: Path | None = None,
extra_env: dict[str, str] | None = None,
) -> subprocess.CompletedProcess[str]:
path = os.environ["PATH"]
if bin_dir is not None:
path = f"{bin_dir}{os.pathsep}{path}"
env = {
**os.environ,
"PATH": f"{bin_dir}{os.pathsep}{os.environ['PATH']}",
"PATH": path,
"BUILDER_CI_POLL_SEC": "0",
"BUILDER_CI_APPEAR_SEC": "0",
"BUILDER_CI_FORGE": "gitea",
}
if extra_env:
env.update(extra_env)
return subprocess.run(
["bash", str(SCRIPT), "--sha", sha, "--root", str(root)],
["bash", str(WAIT_CI), "--sha", sha, "--root", str(root)],
cwd=root,
env=env,
text=True,
@@ -73,176 +98,123 @@ class BuilderCiTests(unittest.TestCase):
check=False,
)
def test_gitea_success_reads_exact_run_and_jobs(self) -> None:
def test_init_copies_each_kind_template(self) -> None:
for kind in ("tea", "glab", "gh", "curl"):
with self.subTest(kind=kind), tempfile.TemporaryDirectory() as temp:
root = Path(temp)
result = subprocess.run(
["bash", str(INIT_CI), "--root", str(root), "--kind", kind],
text=True,
capture_output=True,
check=False,
)
dest = root / ".pouch" / "builder" / "ci-status.sh"
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(result.stdout.strip(), str(dest))
self.assertTrue(dest.is_file())
self.assertTrue(os.access(dest, os.X_OK))
self.assertEqual(
dest.read_text(encoding="utf-8"),
(TEMPLATES / f"{kind}.sh").read_text(encoding="utf-8"),
)
def test_init_rejects_unknown_kind(self) -> None:
with tempfile.TemporaryDirectory() as temp:
result = subprocess.run(
["bash", str(INIT_CI), "--root", temp, "--kind", "jenkins"],
text=True,
capture_output=True,
check=False,
)
self.assertEqual(result.returncode, 2)
self.assertIn("--kind must be tea, glab, gh, or curl", result.stderr)
def test_success_reads_project_script_not_forge_cli(self) -> None:
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
sha = self.make_repo(root, "https://gitea.example.test/acme/widget.git")
sha = self.make_repo(root)
self.write_makefile(root)
self.write_status_script(
root,
'echo "STATUS=success ID=12 PIPELINE=https://gitea.example.test/run/12 JOBS=build=success"',
)
bin_dir = root / "bin"
bin_dir.mkdir()
self.fake_command(
bin_dir,
"curl",
"""
case "$*" in
*"/jobs?"*) cat "$BUILDER_CI_FIXTURE/jobs.json" ;;
*"/actions/runs?"*) cat "$BUILDER_CI_FIXTURE/runs.json" ;;
*) exit 9 ;;
esac
""",
)
fixture = root / "fixture"
fixture.mkdir()
(fixture / "runs.json").write_text(
json.dumps(
{
"workflow_runs": [
{
"id": 9,
"head_sha": "0" * 40,
"status": "completed",
"conclusion": "success",
"html_url": "https://gitea.example.test/wrong",
},
{
"id": 12,
"head_sha": sha,
"status": "completed",
"conclusion": "success",
"html_url": "https://gitea.example.test/run/12",
},
]
}
),
encoding="utf-8",
)
(fixture / "jobs.json").write_text(
json.dumps(
{"jobs": [{"id": 1, "name": "build", "head_sha": sha, "status": "completed", "conclusion": "success"}]}
),
encoding="utf-8",
)
result = self.run_wait(
root,
sha,
bin_dir,
{"BUILDER_CI_FIXTURE": str(fixture)},
)
for name in ("curl", "tea", "glab", "gh"):
self.fake_command(bin_dir, name, "exit 9")
result = self.run_wait(root, sha, bin_dir)
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
self.assertIn("OK pipeline=https://gitea.example.test/run/12", result.stdout)
self.assertIn("id=12", result.stdout)
self.assertIn("jobs=build=success", result.stdout)
self.assertNotIn("wrong", result.stdout + result.stderr)
def test_gitea_failed_job_is_failure(self) -> None:
def test_failed_status_is_failure(self) -> None:
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
sha = self.make_repo(root, "git@gitea.example.test:acme/widget.git")
bin_dir = root / "bin"
bin_dir.mkdir()
fixture = root / "fixture"
fixture.mkdir()
self.fake_command(
bin_dir,
"curl",
"""
case "$*" in
*"/jobs?"*) cat "$BUILDER_CI_FIXTURE/jobs.json" ;;
*"/actions/runs?"*) cat "$BUILDER_CI_FIXTURE/runs.json" ;;
*) exit 9 ;;
esac
""",
)
(fixture / "runs.json").write_text(
json.dumps(
{
"workflow_runs": [
{
"id": 21,
"head_sha": sha,
"status": "completed",
"conclusion": "failure",
"html_url": "https://gitea.example.test/run/21",
}
]
}
),
encoding="utf-8",
)
(fixture / "jobs.json").write_text(
json.dumps(
{"jobs": [{"id": 2, "name": "test", "head_sha": sha, "status": "completed", "conclusion": "failure"}]}
),
encoding="utf-8",
)
result = self.run_wait(
sha = self.make_repo(root)
self.write_makefile(root)
self.write_status_script(
root,
sha,
bin_dir,
{"BUILDER_CI_FIXTURE": str(fixture)},
'echo "STATUS=failed ID=21 PIPELINE=https://gitea.example.test/run/21 JOBS=test=failure"',
)
result = self.run_wait(root, sha)
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
self.assertIn("reason=job_failed", result.stderr)
self.assertIn("test=failure", result.stderr)
def test_gitea_does_not_treat_other_sha_as_pipeline(self) -> None:
def test_empty_status_is_no_pipeline(self) -> None:
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
sha = self.make_repo(root, "https://gitea.example.test/acme/widget.git")
bin_dir = root / "bin"
bin_dir.mkdir()
fixture = root / "fixture"
fixture.mkdir()
self.fake_command(
bin_dir,
"curl",
'cat "$BUILDER_CI_FIXTURE/runs.json"',
)
(fixture / "runs.json").write_text(
json.dumps(
{
"workflow_runs": [
{
"id": 99,
"head_sha": "f" * 40,
"status": "completed",
"conclusion": "success",
}
]
}
),
encoding="utf-8",
)
result = self.run_wait(
root,
sha,
bin_dir,
{"BUILDER_CI_FIXTURE": str(fixture)},
)
sha = self.make_repo(root)
self.write_makefile(root)
self.write_status_script(root, 'echo "STATUS=empty"')
result = self.run_wait(root, sha)
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
self.assertIn("reason=no_pipeline", result.stderr)
def test_github_and_gitlab_branches_still_use_their_cli(self) -> None:
for forge, command, output in (
("github", "gh", "printf '%s\\n' '[]'"),
("gitlab", "glab", "printf '%s\\n' '[]'"),
):
with self.subTest(forge=forge), tempfile.TemporaryDirectory() as temp:
def test_missing_script_is_usage_error(self) -> None:
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
sha = self.make_repo(root, f"https://{forge}.example.test/acme/widget.git")
bin_dir = root / "bin"
bin_dir.mkdir()
self.fake_command(bin_dir, command, output)
sha = self.make_repo(root)
self.write_makefile(root)
result = self.run_wait(root, sha)
self.assertEqual(result.returncode, 2, result.stdout + result.stderr)
self.assertIn("CI status script missing", result.stderr)
def test_non_executable_script_is_usage_error(self) -> None:
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
sha = self.make_repo(root)
self.write_makefile(root)
self.write_status_script(root, 'echo "STATUS=success"', executable=False)
result = self.run_wait(root, sha)
self.assertEqual(result.returncode, 2, result.stdout + result.stderr)
self.assertIn("not executable", result.stderr)
def test_relative_script_outside_builder_dir_is_rejected(self) -> None:
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
sha = self.make_repo(root)
self.write_makefile(root)
rogue = root / "ci-status.sh"
rogue.write_text('#!/bin/sh\necho STATUS=success\n', encoding="utf-8")
rogue.chmod(0o755)
result = self.run_wait(
root,
sha,
bin_dir,
{
"BUILDER_CI_FORGE": forge,
"BUILDER_CI_APPEAR_SEC": "0",
},
root, sha, extra_env={"CI_STATUS_SCRIPT": "ci-status.sh"}
)
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
self.assertIn("reason=no_pipeline", result.stderr)
self.assertEqual(result.returncode, 2, result.stdout + result.stderr)
self.assertIn("must be under .pouch/builder/", result.stderr)
def test_script_exit_3_is_cannot_query(self) -> None:
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
sha = self.make_repo(root)
self.write_makefile(root)
self.write_status_script(root, 'echo "Error: not logged in" >&2; exit 3')
result = self.run_wait(root, sha)
self.assertEqual(result.returncode, 3, result.stdout + result.stderr)
self.assertIn("CI status script exited 3", result.stderr)
self.assertIn("not logged in", result.stderr)
if __name__ == "__main__":
+5 -2
View File
@@ -67,8 +67,11 @@ class DeployerCheckTests(unittest.TestCase):
self.assertIn("不要读 compose.md", compose_section)
self.assertIn("remote.py <svc> recreate", compose_section)
self.assertIn("一句问", compose_section)
self.assertIn("没有 `test` 就用非 `prod` 的那个", compose_section)
self.assertIn("不要** `sync.py` / `remote.py`", compose_section)
self.assertIn(
"没有 `test` 时只能选择明确为本机的非 `prod` 环境",
compose_section,
)
self.assertIn("不要 `sync.py`、`remote.py`、SSH", compose_section)
self.assertIn("docker compose up -d --force-recreate", compose_section)
self.assertNotIn(
"读 [compose.md](references/compose.md)。用 `list.py` 摸底",
+7 -2
View File
@@ -25,6 +25,7 @@ sync_spec.loader.exec_module(deployer_sync)
def _info(service_dir: str, **overrides) -> dict:
info = {
"service_dir": service_dir,
"name": "demo-service",
"node": "edge",
"remote_dir": "/opt/app/svc",
"sync_exclude": ["data", "_data"],
@@ -93,11 +94,15 @@ class DeployerSyncTests(unittest.TestCase):
self.assertEqual(rc, 0)
self.assertTrue(commands)
mkdir_cmd = commands[0]
mkdir_cmd = next(
command
for command in commands
if command and command[0] == "ssh" and command[-1].startswith("mkdir -p ")
)
self.assertEqual(mkdir_cmd[0], "ssh")
self.assertIn("-p", mkdir_cmd)
self.assertIn("2222", mkdir_cmd)
self.assertTrue(any("/opt/app/svc/cfg" in part for part in mkdir_cmd))
self.assertIn("/opt/app/svc/cfg", " ".join(mkdir_cmd))
scp_cmds = [c for c in commands if c and c[0] == "scp"]
self.assertEqual(len(scp_cmds), 2)
+100
View File
@@ -0,0 +1,100 @@
from __future__ import annotations
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
from pouch.yaml_io import safe_dump, safe_load
ROOT = Path(__file__).resolve().parents[1]
CREATE = ROOT / "skills/ship/scripts/create_ship.py"
VALIDATE = ROOT / "skills/ship/scripts/validate_ship.py"
MANIFEST = ROOT / "skills/ship/scripts/validate_test_manifest.py"
def run(script: Path, *args: str) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, str(script), *args],
cwd=ROOT,
text=True,
capture_output=True,
check=False,
)
class TestManifestTests(unittest.TestCase):
def _create(self, root: Path) -> Path:
result = run(
CREATE,
"--project-root", str(root),
"--ship-id", "manifest-demo",
"--kind", "short",
"--title", "测试 manifest",
"--stop-at", "develop_verified",
)
self.assertEqual(result.returncode, 0, result.stderr)
return root / ".pouch/ship/manifest-demo"
def test_new_ship_creates_manifest_and_schema_validates(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
ship = self._create(Path(tmp))
manifest = ship / "2-develop/test-manifest.yaml"
self.assertTrue(manifest.is_file())
checked = run(
MANIFEST, str(manifest), "--ship-id", "manifest-demo", "--mode", "schema"
)
self.assertEqual(checked.returncode, 0, checked.stderr)
record = run(VALIDATE, str(ship))
self.assertEqual(record.returncode, 0, record.stderr)
def test_required_empty_manifest_blocks_verify(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
ship = self._create(Path(tmp))
checked = run(
MANIFEST,
str(ship / "2-develop/test-manifest.yaml"),
"--ship-id", "manifest-demo",
"--policy", "required",
"--mode", "verify",
)
self.assertNotEqual(checked.returncode, 0)
self.assertIn("登记至少一个测试", checked.stderr)
def test_manifest_lists_exact_test_selector(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
ship = self._create(Path(tmp))
path = ship / "2-develop/test-manifest.yaml"
data = safe_load(path.read_text(encoding="utf-8"))
data["tests"] = [{
"id": "go:internal/store/drafts_test.go#TestDraft",
"kind": "unit",
"runner": "go-test",
"target": "./internal/store",
"selector": "^TestDraft$",
"covers": ["T-01"],
"change": "added",
}]
path.write_text(safe_dump(data), encoding="utf-8")
checked = run(
MANIFEST,
str(path),
"--ship-id", "manifest-demo",
"--policy", "required",
"--mode", "verify",
)
self.assertEqual(checked.returncode, 0, checked.stderr)
self.assertIn("go:internal/store/drafts_test.go#TestDraft", checked.stdout)
def test_ship_validation_rejects_missing_manifest(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
ship = self._create(Path(tmp))
(ship / "2-develop/test-manifest.yaml").unlink()
checked = run(VALIDATE, str(ship))
self.assertNotEqual(checked.returncode, 0)
self.assertIn("manifest 不存在", checked.stderr)
if __name__ == "__main__":
unittest.main()
+59
View File
@@ -12,6 +12,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1]
CASES = REPO_ROOT / "skills" / "tester" / "scripts" / "generate_cases_csv.py"
RUNS = REPO_ROOT / "skills" / "tester" / "scripts" / "generate_run_csv.py"
sys.path.insert(0, str(RUNS.parent))
from failure_feedback import attach_feedback
def load_module(name: str, path: Path):
@@ -29,6 +30,7 @@ class TesterRunRecordTests(unittest.TestCase):
root = Path(temp)
cases_dir = root / ".pouch" / "tester" / "cases"
cases_dir.mkdir(parents=True)
(root / ".pouch" / "tester" / "runs").mkdir()
(root / ".pouch" / "tester" / "catalog.yaml").write_text(
"categories:\n - id: smoke\n title: Smoke\n",
encoding="utf-8",
@@ -51,6 +53,63 @@ class TesterRunRecordTests(unittest.TestCase):
self.assertNotEqual(first, second)
self.assertRegex(first, r"^\d{8}T\d{6}Z-smoke-[0-9a-f]{6}$")
def test_run_csv_preserves_all_results_and_structured_issue_fields(self) -> None:
runs = load_module("tester_run_csv_issues", RUNS)
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
cases_dir = root / ".pouch" / "tester" / "cases"
cases_dir.mkdir(parents=True)
(root / ".pouch" / "tester" / "runs").mkdir()
(root / ".pouch" / "tester" / "catalog.yaml").write_text(
"categories:\n - id: smoke\n title: Smoke\n", encoding="utf-8"
)
(cases_dir / "smoke.yaml").write_text(
"category: smoke\ncases:\n"
" - id: TC-1\n title: One\n what: one\n lifecycle: active\n"
" tag: smoke\n risk: readonly\n how: {steps: [open]}\n"
" expected: [{kind: visible-text, value: ready}]\n"
" - id: TC-2\n title: Two\n what: two\n lifecycle: active\n"
" tag: smoke\n risk: readonly\n how: {steps: [open]}\n"
" expected: [{kind: visible-text, value: ready}]\n",
encoding="utf-8",
)
run = attach_feedback(
{
"runId": "issues",
"results": [
{
"id": "TC-1",
"result": "fail",
"issue": {
"expected": "ready",
"observed": "blank",
"artifact_hash": "sha256:abc",
"environment": "Android tester",
"evidence_path": "evidence/1.png",
"reproduction": "open",
"next_step": "targeted rerun",
},
},
{"id": "TC-2", "result": "not-run", "blocked_by": ["TC-1"]},
],
},
trigger="pr",
target="PR-1",
related_issue="ISSUE-1",
)
run_path = root / ".pouch" / "tester" / "runs" / "issues.yaml"
import yaml
run_path.write_text(yaml.safe_dump(run, sort_keys=False), encoding="utf-8")
out = runs.export_run_csv(root, run_path=run_path)
with out.open(encoding="utf-8-sig", newline="") as handle:
rows = list(csv.DictReader(handle))
self.assertEqual([row["id"] for row in rows], ["TC-1", "TC-2"])
self.assertEqual(rows[0]["issue_type"], "product")
self.assertEqual(rows[0]["expected"], "ready")
self.assertEqual(rows[0]["artifact_hash"], "sha256:abc")
self.assertEqual(rows[1]["blocked_by"], "TC-1")
self.assertEqual(run["feedback"]["route"], "pr-update-issue-dispatcher-reassign")
if __name__ == "__main__":
unittest.main()