Compare commits
13 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| b94aa597ca | |||
| 841a2e0ae6 | |||
| 4d31d074ad | |||
| 29f725fb95 | |||
| 3a757e3376 | |||
| 5b9a693fda | |||
| 9b2d0a28d8 | |||
| 3a18a27dc6 | |||
| c2eb207bd2 | |||
| ffa7cbd63e | |||
| baed23d02a | |||
| 9ba9b3da0f | |||
| 0c9a530390 |
@@ -47,7 +47,7 @@ pouch add builder -g # 全局
|
|||||||
并负责上层编排。
|
并负责上层编排。
|
||||||
|
|
||||||
1. `makefile.builder` 目标:`help/build/clean/version` + 条件 `deb/docker/push*`,
|
1. `makefile.builder` 目标:`help/build/clean/version` + 条件 `deb/docker/push*`,
|
||||||
以及 `obtain` / `venue`。变量 `ARCHES/ARCH/VERSION/DIST_DIR/PROJECT_NAME/BUILD_VENUE`。
|
以及 `obtain` / `venue`。变量 `ARCHES/ARCH/VERSION/DIST_DIR/PROJECT_NAME/BUILD_VENUE/CI_STATUS_SCRIPT`。
|
||||||
`VERSION` 通过 `include <builder>/scripts/version.mk` 从 Git 推导。
|
`VERSION` 通过 `include <builder>/scripts/version.mk` 从 Git 推导。
|
||||||
多架构清单是 `ARCHES`,见下方「多架构」。
|
多架构清单是 `ARCHES`,见下方「多架构」。
|
||||||
2. 运行 `python3 -I -S <builder>/scripts/check.py . --ready` 直到构建项 PASS。
|
2. 运行 `python3 -I -S <builder>/scripts/check.py . --ready` 直到构建项 PASS。
|
||||||
@@ -66,7 +66,7 @@ COPY_AS=latest.apk
|
|||||||
# COPY_SCRIPT= # 仅外部/临时脚本覆盖;项目内脚本写 makefile.builder
|
# COPY_SCRIPT= # 仅外部/临时脚本覆盖;项目内脚本写 makefile.builder
|
||||||
```
|
```
|
||||||
|
|
||||||
`BUILD_VENUE=ci` 写在 `makefile.builder`:产物由 CI 构建。`make -f makefile.builder obtain`(或 `push-docker`)等待该 commit 的流水线并 inspect registry,不在本机 `docker push`。
|
`BUILD_VENUE=ci` 写在 `makefile.builder`:产物由 CI 构建。初始化时确认查询命令(tea / glab / gh / curl),写成 `.pouch/builder/ci-status.sh`。`make -f makefile.builder obtain`(或 `push-docker`)执行该脚本等待流水线并 inspect registry,不在本机 `docker push`。
|
||||||
|
|
||||||
4. 日常发布就是两条命令:
|
4. 日常发布就是两条命令:
|
||||||
`make -f makefile.builder deb && make -f makefile.builder push-deb`、
|
`make -f makefile.builder deb && make -f makefile.builder push-deb`、
|
||||||
@@ -195,7 +195,7 @@ detached HEAD 用 `detached`,CI 可注入 `BUILD_BRANCH` / `CI_COMMIT_BRANCH`
|
|||||||
|
|
||||||
## Agent 会做什么
|
## Agent 会做什么
|
||||||
|
|
||||||
1. 初始化:探测 DEB / Docker / Android / iOS / copy 轨道,写出 `makefile.builder` 和 `.env.builder` 骨架,按仓库证据填 `ARCHES`(不默认双架构),不改你已有的 `Makefile` 或 `.env`。
|
1. 初始化:探测 DEB / Docker / Android / iOS / copy 轨道,写出 `makefile.builder` 和 `.env.builder` 骨架,确认 `BUILD_VENUE`;选 ci 时生成 `.pouch/builder/ci-status.sh`。按仓库证据填 `ARCHES`(不默认双架构),不改你已有的 `Makefile` 或 `.env`。
|
||||||
2. 检查:只跑 `scripts/check.py --ready`,不改文件。
|
2. 检查:只跑 `scripts/check.py --ready`,不改文件。
|
||||||
3. 构建:走 `make -f makefile.builder`;只获构建授权时不走会自动上传的目标。
|
3. 构建:走 `make -f makefile.builder`;只获构建授权时不走会自动上传的目标。
|
||||||
4. 发布:先校验产物,再上传。缺发布键、未授权、或可能覆盖同版本时停下问你。
|
4. 发布:先校验产物,再上传。缺发布键、未授权、或可能覆盖同版本时停下问你。
|
||||||
@@ -223,7 +223,9 @@ Docker 只有你点名镜像时才走。token 不会写进输出、提交或回
|
|||||||
| `scripts/verify_deb.sh` | 核对包元数据、内容与 SHA-256 |
|
| `scripts/verify_deb.sh` | 核对包元数据、内容与 SHA-256 |
|
||||||
| `scripts/copy_artifact.sh` | 按 `COPY_SCRIPT` 分发产物,未配置时拷到 `COPY_DEST` |
|
| `scripts/copy_artifact.sh` | 按 `COPY_SCRIPT` 分发产物,未配置时拷到 `COPY_DEST` |
|
||||||
| `scripts/obtain.sh` | 对精确 SHA 拿到产物;`BUILD_VENUE=ci` 时等流水线再 inspect |
|
| `scripts/obtain.sh` | 对精确 SHA 拿到产物;`BUILD_VENUE=ci` 时等流水线再 inspect |
|
||||||
| `scripts/wait_ci.sh` | 只等该 SHA 的 GitLab/GitHub/Gitea 流水线,不触发;Gitea 读取 workflow jobs |
|
| `scripts/wait_ci.sh` | 轮询项目 `CI_STATUS_SCRIPT`,只等该 SHA,不触发 |
|
||||||
|
| `scripts/init_ci_status.sh` | 初始化时把 tea/glab/gh/curl 模板拷到 `.pouch/builder/ci-status.sh` |
|
||||||
|
| `templates/ci-status/` | tea / glab / gh / curl 查询脚本模板;契约见 [ci-status.md](references/ci-status.md) |
|
||||||
| `templates/makefile.android.builder` | Android APK + copy 轨道骨架 |
|
| `templates/makefile.android.builder` | Android APK + copy 轨道骨架 |
|
||||||
| `templates/makefile.ios.builder` | iOS IPA + copy 轨道骨架 |
|
| `templates/makefile.ios.builder` | iOS IPA + copy 轨道骨架 |
|
||||||
|
|
||||||
|
|||||||
@@ -47,7 +47,8 @@ worktree,也可以指向已配对的远程 Mac/其它构建机;不要把远
|
|||||||
3. 没有 `makefile.builder`:按轨道拷模板。deb/docker 用 `templates/makefile.builder`;仅 android 用 `templates/makefile.android.builder`;仅 ios 用 `templates/makefile.ios.builder`。按轨道删掉未使用的 deb/docker/push* 段,把 `build` 的 TODO 换成仓库里已有的真实编译命令(可从用户 Makefile 抄配方,但不要 `include` 或递归调用它),且配方必须使用 `$(ARCH)`。Android/iOS 的 `ARCHES` 是构建机 ISA(fat APK/IPA 仍只留本机一项)。按 [contract.md](references/contract.md) 填 `ARCHES`:`debian/control` / CI matrix / 用户点名的子集;否则只写本机(`x86_64`→`amd64`,`aarch64`→`arm64`)。不要默认双架构;看不出交叉编法时只留一项并说明。`include` builder 的 `scripts/version.mk`。多产物把 `push` 改成 `push: push-deb push-docker push-copy` 的子集。不要改用户 Makefile。
|
3. 没有 `makefile.builder`:按轨道拷模板。deb/docker 用 `templates/makefile.builder`;仅 android 用 `templates/makefile.android.builder`;仅 ios 用 `templates/makefile.ios.builder`。按轨道删掉未使用的 deb/docker/push* 段,把 `build` 的 TODO 换成仓库里已有的真实编译命令(可从用户 Makefile 抄配方,但不要 `include` 或递归调用它),且配方必须使用 `$(ARCH)`。Android/iOS 的 `ARCHES` 是构建机 ISA(fat APK/IPA 仍只留本机一项)。按 [contract.md](references/contract.md) 填 `ARCHES`:`debian/control` / CI matrix / 用户点名的子集;否则只写本机(`x86_64`→`amd64`,`aarch64`→`arm64`)。不要默认双架构;看不出交叉编法时只留一项并说明。`include` builder 的 `scripts/version.mk`。多产物把 `push` 改成 `push: push-deb push-docker push-copy` 的子集。不要改用户 Makefile。
|
||||||
4. 已有 `makefile.builder`:跑检查;按 FAIL 给出修补说明。不覆盖 `build`。用户明确要求按契约改时再加 `ARCHES`,并确认配方里有 `$(ARCH)`。不要调用 create-makefile(版本规则冲突)。
|
4. 已有 `makefile.builder`:跑检查;按 FAIL 给出修补说明。不覆盖 `build`。用户明确要求按契约改时再加 `ARCHES`,并确认配方里有 `$(ARCH)`。不要调用 create-makefile(版本规则冲突)。
|
||||||
5. 没有 `.env.builder`:把 `<skill-dir>/templates/env.builder` 拷到项目根(注释键,不含值)。缺发布键时在报告里给出可粘贴示例,并把 `.env.builder` 加入 `.gitignore`,不要提交。
|
5. 没有 `.env.builder`:把 `<skill-dir>/templates/env.builder` 拷到项目根(注释键,不含值)。缺发布键时在报告里给出可粘贴示例,并把 `.env.builder` 加入 `.gitignore`,不要提交。
|
||||||
6. 运行 `python3 -I -S <skill-dir>/scripts/check.py <project-dir> --ready`。结构校验通过且当前轨道能构建才称「完成」;只缺发布键是「部分完成」。契约 FAIL 或轨道工具缺失是「阻塞」。除非用户明确要求,不提交、不推送、不上传。
|
6. 确认 `BUILD_VENUE`:`local` 或 `ci`,不要猜。选 `ci` 时读 [ci-status.md](references/ci-status.md),确认查询命令(`tea` / `glab` / `gh` / `curl`),执行 `<skill-dir>/scripts/init_ci_status.sh --root <project-dir> --kind <kind>`,在 `makefile.builder` 写入 `BUILD_VENUE=ci` 和 `CI_STATUS_SCRIPT=.pouch/builder/ci-status.sh`。用户给了自定义查询命令时,把命令写进同一份脚本并保持 `STATUS=` 输出契约。
|
||||||
|
7. 运行 `python3 -I -S <skill-dir>/scripts/check.py <project-dir> --ready`。结构校验通过且当前轨道能构建才称「完成」;只缺发布键是「部分完成」。契约 FAIL 或轨道工具缺失是「阻塞」。除非用户明确要求,不提交、不推送、不上传。
|
||||||
|
|
||||||
```text
|
```text
|
||||||
## builder 初始化:完成 | 部分完成 | 阻塞
|
## builder 初始化:完成 | 部分完成 | 阻塞
|
||||||
@@ -90,6 +91,6 @@ COPY_AS=latest.apk
|
|||||||
按 [obtain.md](references/obtain.md) 执行 `scripts/obtain.sh --sha <commit>`。
|
按 [obtain.md](references/obtain.md) 执行 `scripts/obtain.sh --sha <commit>`。
|
||||||
|
|
||||||
- `BUILD_VENUE=local`:当前 HEAD 必须是该 SHA,跑 makefile 的 docker/deb。
|
- `BUILD_VENUE=local`:当前 HEAD 必须是该 SHA,跑 makefile 的 docker/deb。
|
||||||
- `BUILD_VENUE=ci`:等该 SHA 的 GitLab/GitHub/Gitea 流水线,再 inspect `version.sh --docker` 的镜像 tag。不触发流水线,不本机 push。
|
- `BUILD_VENUE=ci`:`wait_ci.sh` 轮询项目的 `CI_STATUS_SCRIPT`,再 inspect `version.sh --docker` 的镜像 tag。不触发流水线,不本机 push。缺脚本则停下,回到初始化。
|
||||||
- 失败(无流水线、job 失败、digest 不对)必须原样返回给调用方,不得标 `artifact_built`。
|
- 失败(无流水线、job 失败、digest 不对)必须原样返回给调用方,不得标 `artifact_built`。
|
||||||
- 合进默认分支若项目不跑 CI,这是正确失败;预览用 `dev/*`,正式用 tag。
|
- 合进默认分支若项目不跑 CI,这是正确失败;预览用 `dev/*`,正式用 tag。
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# CI 状态查询脚本
|
||||||
|
|
||||||
|
`BUILD_VENUE=ci` 时,流水线状态由**项目脚本**查询。builder 只轮询这份脚本,
|
||||||
|
不在 skill 里猜 gh / glab / tea / curl。
|
||||||
|
|
||||||
|
## 初始化
|
||||||
|
|
||||||
|
选 `ci` 后确认查询方式,再生成脚本:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
<skill-dir>/scripts/init_ci_status.sh --root <project> --kind tea|glab|gh|curl
|
||||||
|
```
|
||||||
|
|
||||||
|
写出 `.pouch/builder/ci-status.sh`(可执行、进 git)。`makefile.builder` 写:
|
||||||
|
|
||||||
|
```makefile
|
||||||
|
BUILD_VENUE ?= ci
|
||||||
|
CI_STATUS_SCRIPT ?= .pouch/builder/ci-status.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
相对路径只能在 `.pouch/builder/` 下。凭据仍放 `.env.builder`(如 `GITEA_TOKEN`),
|
||||||
|
不要写进脚本参数或日志。
|
||||||
|
|
||||||
|
用户给了自定义命令时,把该命令写进同一份脚本,输出必须符合下方契约。
|
||||||
|
|
||||||
|
## 调用
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.pouch/builder/ci-status.sh --sha <40-character-sha>
|
||||||
|
```
|
||||||
|
|
||||||
|
只查询、不触发流水线。`BUILDER_PROJECT_ROOT` 是项目根。
|
||||||
|
|
||||||
|
## 输出
|
||||||
|
|
||||||
|
退出 0 且 stdout 含 `STATUS=` 键值(可多行或同一行):
|
||||||
|
|
||||||
|
| STATUS | 含义 |
|
||||||
|
| --- | --- |
|
||||||
|
| `empty` | 该 SHA 还没有流水线 |
|
||||||
|
| `running` | 已出现、尚未结束 |
|
||||||
|
| `success` | 该 SHA 的流水线成功 |
|
||||||
|
| `failed` | 失败、取消或超时 |
|
||||||
|
| `error` | 查询本身失败(鉴权、CLI 缺失) |
|
||||||
|
|
||||||
|
可选:`ID=`、`PIPELINE=`(URL)、`JOBS=`。
|
||||||
|
|
||||||
|
| 退出码 | 含义 |
|
||||||
|
| --- | --- |
|
||||||
|
| 0 | 已查出状态(含 empty/running/failed) |
|
||||||
|
| 2 | 用法错误 |
|
||||||
|
| 3 | 无法查询(缺 CLI、未登录、被拒绝) |
|
||||||
|
|
||||||
|
`wait_ci.sh` 按这份输出轮询:`empty` 超过出现窗口记 `no_pipeline`;
|
||||||
|
`success` 才继续 obtain;`failed` 原样失败。
|
||||||
@@ -52,7 +52,7 @@ make -f makefile.builder <target>
|
|||||||
| `push-docker` | 同时有多种发布轨道的项目 | `BUILD_VENUE=local` 时仅调 `publish_docker.sh`;`ci` 时仅调 `obtain.sh` |
|
| `push-docker` | 同时有多种发布轨道的项目 | `BUILD_VENUE=local` 时仅调 `publish_docker.sh`;`ci` 时仅调 `obtain.sh` |
|
||||||
| `push-copy` | 同时有多种发布轨道的项目 | 仅调 builder 的 `copy_artifact.sh`;由它按配置选择内置拷贝或自定义脚本 |
|
| `push-copy` | 同时有多种发布轨道的项目 | 仅调 builder 的 `copy_artifact.sh`;由它按配置选择内置拷贝或自定义脚本 |
|
||||||
| `push` | 单一产物类型时必备;多产物项目为聚合 | 依序调用对应 push-* 或直接调脚本;是发布的唯一 make 入口 |
|
| `push` | 单一产物类型时必备;多产物项目为聚合 | 依序调用对应 push-* 或直接调脚本;是发布的唯一 make 入口 |
|
||||||
| `obtain` | 有产物的项目 | 调用 `obtain.sh`,对精确 commit SHA 拿到已存在的产物。`BUILD_VENUE=ci` 时等待该 SHA 的 forge 流水线再 inspect,不在本机 `docker push` |
|
| `obtain` | 有产物的项目 | 调用 `obtain.sh`,对精确 commit SHA 拿到已存在的产物。`BUILD_VENUE=ci` 时轮询 `CI_STATUS_SCRIPT` 再 inspect,不在本机 `docker push` |
|
||||||
| `venue` | 可选(新模板必备) | 打印一行 `BUILD_VENUE`(`local` 或 `ci`) |
|
| `venue` | 可选(新模板必备) | 打印一行 `BUILD_VENUE`(`local` 或 `ci`) |
|
||||||
|
|
||||||
规则:
|
规则:
|
||||||
@@ -82,7 +82,8 @@ make -f makefile.builder <target>
|
|||||||
|
|
||||||
| 变量 | 默认 | 说明 |
|
| 变量 | 默认 | 说明 |
|
||||||
|------|------|------|
|
|------|------|------|
|
||||||
| `BUILD_VENUE` | `local` | 产物在哪构建:`local` 本机 make;`ci` 由 GitLab/GitHub/Gitea 流水线构建。这不是新轨道。写在 `makefile.builder` 并提交。`ci` 时 `push-docker` 委托 `obtain.sh`(等流水线 + inspect),不跑 `publish_docker.sh` |
|
| `BUILD_VENUE` | `local` | 产物在哪构建:`local` 本机 make;`ci` 由项目 CI 流水线构建。这不是新轨道。写在 `makefile.builder` 并提交。`ci` 时 `push-docker` 委托 `obtain.sh`(等流水线 + inspect),不跑 `publish_docker.sh` |
|
||||||
|
| `CI_STATUS_SCRIPT` | `.pouch/builder/ci-status.sh` | `BUILD_VENUE=ci` 时必填。项目内查询该 SHA 流水线状态的可执行脚本,相对路径只能在 `.pouch/builder/`。初始化时按 tea / glab / gh / curl 生成,契约见 [ci-status.md](ci-status.md) |
|
||||||
| `ARCHES` | `amd64` | 本项目要发布的架构清单,空格分隔,子集只能是 `amd64` `arm64`。写在 `makefile.builder` 里并提交;不要放进 `.env.builder`。缺省或无 `arches` 目标时视为单架构 `amd64`。不要默认写成双架构 |
|
| `ARCHES` | `amd64` | 本项目要发布的架构清单,空格分隔,子集只能是 `amd64` `arm64`。写在 `makefile.builder` 里并提交;不要放进 `.env.builder`。缺省或无 `arches` 目标时视为单架构 `amd64`。不要默认写成双架构 |
|
||||||
| `ARCH` | `$(firstword $(ARCHES))` | 这一次构建的架构。必须落在 `ARCHES` 里;非法值必须 `$(error)` 并提示合法值。命令行/环境传入时只构建/上传这一项 |
|
| `ARCH` | `$(firstword $(ARCHES))` | 这一次构建的架构。必须落在 `ARCHES` 里;非法值必须 `$(error)` 并提示合法值。命令行/环境传入时只构建/上传这一项 |
|
||||||
| `VERSION` | `` (空) | 为空时按「版本号」节从 Git 祖先稳定 tag 推导;禁止 `sort -V` 取全局最新 |
|
| `VERSION` | `` (空) | 为空时按「版本号」节从 Git 祖先稳定 tag 推导;禁止 `sort -V` 取全局最新 |
|
||||||
@@ -189,7 +190,7 @@ include $(HOME)/.pouch/skills/builder/scripts/version.mk
|
|||||||
|
|
||||||
`PLATFORMS` 优先级(高 → 低):用户这轮 `--platform` 或已 export 的 `PLATFORMS` → 命令行 `ARCH=` 映射的 `linux/$(ARCH)`(由 `push-docker` 传入)→ `.env.builder` 的 `PLATFORMS` → `makefile.builder` 的 `ARCHES` → `linux/amd64`。
|
`PLATFORMS` 优先级(高 → 低):用户这轮 `--platform` 或已 export 的 `PLATFORMS` → 命令行 `ARCH=` 映射的 `linux/$(ARCH)`(由 `push-docker` 传入)→ `.env.builder` 的 `PLATFORMS` → `makefile.builder` 的 `ARCHES` → `linux/amd64`。
|
||||||
|
|
||||||
`BUILD_VENUE=ci` 时 Docker 发布路径是 `obtain.sh`:只认 **该 commit SHA** 的流水线。没有流水线、job 失败、超时、或 Harbor inspect 不到 `version.sh --docker` 算出的 tag,一律非零退出。不触发流水线。GitHub 用 `gh`、GitLab 用 `glab`、Gitea 用 `curl` 读取 Actions API;Gitea 会再次校验 `head_sha` 并读取 workflow jobs。合进默认分支若项目声明不构建,这是正确失败;调用方应打 tag 或推送到项目会构建的 ref(例如 `dev/*`)。CI job 若已经 `docker push`,obtain 成功即视为产物已在 registry,不要再本机 push 一次。
|
`BUILD_VENUE=ci` 时 Docker 发布路径是 `obtain.sh`:`wait_ci.sh` 只轮询 `CI_STATUS_SCRIPT`,只认 **该 commit SHA**。没有流水线、job 失败、超时、或 Harbor inspect 不到 `version.sh --docker` 算出的 tag,一律非零退出。不触发流水线。查询命令在初始化时写成项目脚本,运行时不再猜测 gh / glab / tea / curl。合进默认分支若项目声明不构建,这是正确失败;调用方应打 tag 或推送到项目会构建的 ref(例如 `dev/*`)。CI job 若已经 `docker push`,obtain 成功即视为产物已在 registry,不要再本机 push 一次。
|
||||||
|
|
||||||
配置来源优先级:shell 已显式设置的值 > 项目根 `.env.builder` > 失败并询问用户。
|
配置来源优先级:shell 已显式设置的值 > 项目根 `.env.builder` > 失败并询问用户。
|
||||||
`.env.builder` 由 builder 脚本加载(不回显任何值);当前 shell 已设置的值优先。
|
`.env.builder` 由 builder 脚本加载(不回显任何值);当前 shell 已设置的值优先。
|
||||||
|
|||||||
@@ -36,13 +36,13 @@ TAG=1.4.2
|
|||||||
| 0 | 产物已核验 | `artifact_built` 完成 |
|
| 0 | 产物已核验 | `artifact_built` 完成 |
|
||||||
| 1 | 该 SHA 无流水线、job 失败、超时、inspect 失败 | 标 `failed`,不得 publish/deploy |
|
| 1 | 该 SHA 无流水线、job 失败、超时、inspect 失败 | 标 `failed`,不得 publish/deploy |
|
||||||
| 2 | 用法或配置错误 | 标 `blocked` |
|
| 2 | 用法或配置错误 | 标 `blocked` |
|
||||||
| 3 | glab/gh/curl 缺失或读不到该项目 | 停;可用**已打开且已登录**的同 host Orca tab 查流水线。禁止新开未登录页 |
|
| 3 | 项目 CI 状态脚本无法查询 | 停;回到初始化补 `CI_STATUS_SCRIPT`,或用**已打开且已登录**的同 host Orca tab 查流水线。禁止新开未登录页 |
|
||||||
|
|
||||||
## `ci` 规则
|
## `ci` 规则
|
||||||
|
|
||||||
- 只认该 SHA 的 pipeline,不认默认分支「最新一次绿」。
|
- 只认该 SHA 的 pipeline,不认默认分支「最新一次绿」。
|
||||||
- 不触发流水线。
|
- 不触发流水线。
|
||||||
- Gitea 使用 Actions REST API 的 `head_sha` 精确过滤,并读取每个 workflow run 的 jobs;私有仓库可在 `.env.builder` 配置 `GITEA_TOKEN`。自定义 Gitea host 可设置 `BUILDER_CI_FORGE=gitea`,API 前缀默认是 `https://<origin-host>/api/v1`,也可用 `GITEA_API_URL` 覆盖。
|
- 查询命令是项目 `.pouch/builder/ci-status.sh`(`CI_STATUS_SCRIPT`),初始化时按 tea / glab / gh / curl 写好。凭据仍放 `.env.builder`。
|
||||||
- job 变绿不够,docker 轨道必须 inspect 到 digest。
|
- job 变绿不够,docker 轨道必须 inspect 到 digest。
|
||||||
- 合进 `main` 若项目不构建,`no_pipeline` 是正确失败。预览走项目会构建的 ref(如 `dev/*`),正式走 tag。
|
- 合进 `main` 若项目不构建,`no_pipeline` 是正确失败。预览走项目会构建的 ref(如 `dev/*`),正式走 tag。
|
||||||
|
|
||||||
|
|||||||
@@ -551,8 +551,8 @@ def configured_value(
|
|||||||
return os.environ.get(key, "") or env_values.get(key, "") or (make_values or {}).get(key, "")
|
return os.environ.get(key, "") or env_values.get(key, "") or (make_values or {}).get(key, "")
|
||||||
|
|
||||||
|
|
||||||
def resolve_copy_script(project: Path, script: str) -> tuple[Path | None, str | None]:
|
def resolve_builder_local_script(project: Path, script: str) -> tuple[Path | None, str | None]:
|
||||||
"""Resolve COPY_SCRIPT without allowing arbitrary project-relative paths."""
|
"""Resolve a project script; relative paths must stay under .pouch/builder/."""
|
||||||
configured = Path(script).expanduser()
|
configured = Path(script).expanduser()
|
||||||
if configured.is_absolute():
|
if configured.is_absolute():
|
||||||
return configured, None
|
return configured, None
|
||||||
@@ -561,10 +561,17 @@ def resolve_copy_script(project: Path, script: str) -> tuple[Path | None, str |
|
|||||||
allowed_root = (project / ".pouch" / "builder").resolve()
|
allowed_root = (project / ".pouch" / "builder").resolve()
|
||||||
resolved.relative_to(allowed_root)
|
resolved.relative_to(allowed_root)
|
||||||
except ValueError:
|
except ValueError:
|
||||||
return None, "relative COPY_SCRIPT must be under .pouch/builder/"
|
return None, "relative script must be under .pouch/builder/"
|
||||||
return resolved, None
|
return resolved, None
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_copy_script(project: Path, script: str) -> tuple[Path | None, str | None]:
|
||||||
|
path, error = resolve_builder_local_script(project, script)
|
||||||
|
if error:
|
||||||
|
return None, "relative COPY_SCRIPT must be under .pouch/builder/"
|
||||||
|
return path, None
|
||||||
|
|
||||||
|
|
||||||
def check_copy_script(
|
def check_copy_script(
|
||||||
report: Report,
|
report: Report,
|
||||||
project: Path,
|
project: Path,
|
||||||
@@ -591,6 +598,9 @@ def check_copy_script(
|
|||||||
|
|
||||||
|
|
||||||
def read_venue(project: Path) -> str:
|
def read_venue(project: Path) -> str:
|
||||||
|
env = (os.environ.get("BUILD_VENUE") or "").strip()
|
||||||
|
if env in ("local", "ci"):
|
||||||
|
return env
|
||||||
result = run_make(project, "venue")
|
result = run_make(project, "venue")
|
||||||
if not has_no_rule(result):
|
if not has_no_rule(result):
|
||||||
echoed = parse_dry_echo(result)
|
echoed = parse_dry_echo(result)
|
||||||
@@ -604,34 +614,35 @@ def read_venue(project: Path) -> str:
|
|||||||
return raw or "local"
|
return raw or "local"
|
||||||
|
|
||||||
|
|
||||||
def origin_forge(project: Path) -> str:
|
def check_ci_status_script(
|
||||||
result = subprocess.run(
|
report: Report,
|
||||||
["git", "-C", str(project), "remote", "get-url", "origin"],
|
project: Path,
|
||||||
capture_output=True,
|
env_values: dict[str, str],
|
||||||
text=True,
|
make_values: dict[str, str],
|
||||||
check=False,
|
) -> None:
|
||||||
|
if read_venue(project) != "ci":
|
||||||
|
report.add(SKIP, 15, "CI_STATUS_SCRIPT 可执行", "BUILD_VENUE is not ci")
|
||||||
|
return
|
||||||
|
script = configured_value("CI_STATUS_SCRIPT", env_values, make_values)
|
||||||
|
problems = []
|
||||||
|
if not script:
|
||||||
|
problems.append("BUILD_VENUE=ci requires CI_STATUS_SCRIPT")
|
||||||
|
path = None
|
||||||
|
resolve_error = None
|
||||||
|
else:
|
||||||
|
path, resolve_error = resolve_builder_local_script(project, script)
|
||||||
|
if resolve_error:
|
||||||
|
problems.append(resolve_error)
|
||||||
|
elif path is not None and not path.is_file():
|
||||||
|
problems.append("CI_STATUS_SCRIPT does not point to a file")
|
||||||
|
elif path is not None and not os.access(path, os.X_OK):
|
||||||
|
problems.append("CI_STATUS_SCRIPT is not executable")
|
||||||
|
report.add(
|
||||||
|
FAIL if problems else PASS,
|
||||||
|
15,
|
||||||
|
"CI_STATUS_SCRIPT 可执行",
|
||||||
|
"\n".join(problems) if problems else f"ci status script: {script}",
|
||||||
)
|
)
|
||||||
url = (result.stdout or "").strip()
|
|
||||||
configured = env_file_values(project).get("BUILDER_CI_FORGE", "").strip().lower()
|
|
||||||
forced = (os.environ.get("BUILDER_CI_FORGE") or configured).strip().lower()
|
|
||||||
if forced in ("github", "gitlab", "gitea"):
|
|
||||||
return forced
|
|
||||||
if "github.com" in url:
|
|
||||||
return "github"
|
|
||||||
host_match = re.search(r"(?:https?://|ssh://git@|git@)([^/:]+)", url)
|
|
||||||
host = host_match.group(1).lower() if host_match else ""
|
|
||||||
builder_keys = env_file_keys(project)
|
|
||||||
if (
|
|
||||||
"gitea" in host
|
|
||||||
or "GITEA_TOKEN" in builder_keys
|
|
||||||
or "GITEA_API_URL" in builder_keys
|
|
||||||
or os.environ.get("GITEA_TOKEN")
|
|
||||||
or os.environ.get("GITEA_API_URL")
|
|
||||||
):
|
|
||||||
return "gitea"
|
|
||||||
if url:
|
|
||||||
return "gitlab"
|
|
||||||
return ""
|
|
||||||
|
|
||||||
|
|
||||||
def check_ready_toolchain(
|
def check_ready_toolchain(
|
||||||
@@ -655,28 +666,7 @@ def check_ready_toolchain(
|
|||||||
lines.append(
|
lines.append(
|
||||||
f"docker: {'found' if docker_ok else 'optional for inspect; obtain fails without it'}"
|
f"docker: {'found' if docker_ok else 'optional for inspect; obtain fails without it'}"
|
||||||
)
|
)
|
||||||
forge = origin_forge(project) if project is not None else ""
|
lines.append("CI status: project CI_STATUS_SCRIPT (tea/glab/gh/curl)")
|
||||||
if forge == "github":
|
|
||||||
gh_ok = shutil.which("gh") is not None
|
|
||||||
lines.append(f"gh: {'found' if gh_ok else 'MISSING (blocks ci venue)'}")
|
|
||||||
if not gh_ok:
|
|
||||||
problems.append("install gh and authenticate for this GitHub repo")
|
|
||||||
elif forge == "gitlab":
|
|
||||||
glab_ok = shutil.which("glab") is not None
|
|
||||||
lines.append(f"glab: {'found' if glab_ok else 'MISSING (blocks ci venue)'}")
|
|
||||||
if not glab_ok:
|
|
||||||
problems.append("install glab and authenticate for this GitLab project")
|
|
||||||
elif forge == "gitea":
|
|
||||||
curl_ok = shutil.which("curl") is not None
|
|
||||||
lines.append(f"curl: {'found' if curl_ok else 'MISSING (blocks ci venue)'}")
|
|
||||||
lines.append("Gitea API: /api/v1; GITEA_TOKEN optional for public repositories")
|
|
||||||
if not curl_ok:
|
|
||||||
problems.append("install curl to read Gitea Actions for this repository")
|
|
||||||
else:
|
|
||||||
glab_ok = shutil.which("glab") is not None
|
|
||||||
lines.append(f"glab: {'found' if glab_ok else 'MISSING (blocks ci venue)'}")
|
|
||||||
if not glab_ok:
|
|
||||||
problems.append("install glab and authenticate for this GitLab project")
|
|
||||||
elif docker_project:
|
elif docker_project:
|
||||||
docker_ok = shutil.which("docker") is not None
|
docker_ok = shutil.which("docker") is not None
|
||||||
lines.append(f"docker: {'found' if docker_ok else 'MISSING (blocks docker track)'}")
|
lines.append(f"docker: {'found' if docker_ok else 'MISSING (blocks docker track)'}")
|
||||||
@@ -867,6 +857,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
check_ready_env_keys(
|
check_ready_env_keys(
|
||||||
report, project, deb_project=deb_project, docker_project=docker_project
|
report, project, deb_project=deb_project, docker_project=docker_project
|
||||||
)
|
)
|
||||||
|
check_ci_status_script(report, project, env_file_values(project), makefile_values(project) if (project / BUILDER_MAKEFILE).is_file() else {})
|
||||||
print()
|
print()
|
||||||
print(f"RESULT: FAILED ({report.failures} check(s) failed)")
|
print(f"RESULT: FAILED ({report.failures} check(s) failed)")
|
||||||
return 1
|
return 1
|
||||||
@@ -963,6 +954,12 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
docker_project=docker_project,
|
docker_project=docker_project,
|
||||||
copy_project=copy_project,
|
copy_project=copy_project,
|
||||||
)
|
)
|
||||||
|
check_ci_status_script(
|
||||||
|
report,
|
||||||
|
project,
|
||||||
|
env_file_values(project),
|
||||||
|
makefile_values(project),
|
||||||
|
)
|
||||||
|
|
||||||
total_fail = report.failures
|
total_fail = report.failures
|
||||||
print()
|
print()
|
||||||
|
|||||||
Executable
+42
@@ -0,0 +1,42 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Copy a CI status template into the project.
|
||||||
|
# Usage: init_ci_status.sh --root DIR --kind tea|glab|gh|curl
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
echo "Usage: init_ci_status.sh --root DIR --kind tea|glab|gh|curl" >&2
|
||||||
|
}
|
||||||
|
|
||||||
|
root=
|
||||||
|
kind=
|
||||||
|
while (($#)); do
|
||||||
|
case "$1" in
|
||||||
|
--root) [[ $# -ge 2 ]] || { usage; exit 2; }; root=$2; shift 2 ;;
|
||||||
|
--kind) [[ $# -ge 2 ]] || { usage; exit 2; }; kind=$2; shift 2 ;;
|
||||||
|
-h|--help) usage; exit 0 ;;
|
||||||
|
*) usage; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ -z "$root" || -z "$kind" ]]; then
|
||||||
|
usage
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
case "$kind" in
|
||||||
|
tea|glab|gh|curl) ;;
|
||||||
|
*) echo "Error: --kind must be tea, glab, gh, or curl" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||||
|
src=$script_dir/../templates/ci-status/${kind}.sh
|
||||||
|
if [[ ! -f "$src" ]]; then
|
||||||
|
echo "Error: missing template: $src" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
dest=$root/.pouch/builder/ci-status.sh
|
||||||
|
mkdir -p "$(dirname "$dest")"
|
||||||
|
cp "$src" "$dest"
|
||||||
|
chmod +x "$dest"
|
||||||
|
echo "$dest"
|
||||||
@@ -10,14 +10,13 @@ set -euo pipefail
|
|||||||
#
|
#
|
||||||
# BUILD_VENUE (makefile.builder, default local):
|
# BUILD_VENUE (makefile.builder, default local):
|
||||||
# local HEAD must be SHA; run `make -f makefile.builder docker` or `deb`
|
# local HEAD must be SHA; run `make -f makefile.builder docker` or `deb`
|
||||||
# ci wait for the forge pipeline of this SHA, then inspect the image
|
# ci wait for this SHA via the project CI_STATUS_SCRIPT, then inspect the image
|
||||||
# (GitHub Actions, GitLab CI, or Gitea Actions)
|
|
||||||
#
|
#
|
||||||
# Exit:
|
# Exit:
|
||||||
# 0 artifact verified; prints ARTIFACT= / DIGEST= / SHA= / VENUE=
|
# 0 artifact verified; prints ARTIFACT= / DIGEST= / SHA= / VENUE=
|
||||||
# 1 no pipeline, job failed, inspect failed
|
# 1 no pipeline, job failed, inspect failed
|
||||||
# 2 usage / config
|
# 2 usage / config
|
||||||
# 3 forge CLI cannot read the project (ci venue)
|
# 3 CI status script cannot query (ci venue)
|
||||||
|
|
||||||
usage() {
|
usage() {
|
||||||
sed -n '2,20p' "$0"
|
sed -n '2,20p' "$0"
|
||||||
|
|||||||
@@ -1,25 +1,17 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
# Wait until the forge pipeline for an exact commit SHA finishes.
|
# Poll the project CI status script until the exact SHA finishes.
|
||||||
# Does not trigger a pipeline. Does not print tokens.
|
# Does not trigger a pipeline. Does not print tokens.
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# wait_ci.sh --sha SHA [--root DIR] [--timeout SEC]
|
# wait_ci.sh --sha SHA [--root DIR] [--timeout SEC]
|
||||||
#
|
#
|
||||||
# Supported forges:
|
# The project script is CI_STATUS_SCRIPT (makefile.builder, default
|
||||||
# GitHub Actions (gh), GitLab CI (glab), and Gitea Actions (curl + REST API).
|
# .pouch/builder/ci-status.sh). See references/ci-status.md.
|
||||||
# Set BUILDER_CI_FORGE=gitea for a custom Gitea host that does not contain
|
|
||||||
# "gitea" in its hostname. GITEA_API_URL may override the /api/v1 endpoint.
|
|
||||||
#
|
|
||||||
# Exit:
|
|
||||||
# 0 pipeline succeeded; prints "OK pipeline=<url> sha=<sha>"
|
|
||||||
# 1 no pipeline for this SHA, or a job failed/canceled
|
|
||||||
# 2 usage / bad arguments
|
|
||||||
# 3 forge CLI missing, or it cannot read this project
|
|
||||||
|
|
||||||
usage() {
|
usage() {
|
||||||
sed -n '2,18p' "$0"
|
sed -n '2,14p' "$0"
|
||||||
}
|
}
|
||||||
|
|
||||||
sha=
|
sha=
|
||||||
@@ -51,7 +43,6 @@ if [[ -z "$sha" ]]; then
|
|||||||
fi
|
fi
|
||||||
sha=$(git -C "$root" rev-parse "$sha")
|
sha=$(git -C "$root" rev-parse "$sha")
|
||||||
|
|
||||||
# Load only builder configuration. Shell values win and values are never echoed.
|
|
||||||
if [[ -f "$root/.env.builder" ]]; then
|
if [[ -f "$root/.env.builder" ]]; then
|
||||||
while IFS='=' read -r key value; do
|
while IFS='=' read -r key value; do
|
||||||
key=${key%%[[:space:]]*}
|
key=${key%%[[:space:]]*}
|
||||||
@@ -67,391 +58,126 @@ if [[ -f "$root/.env.builder" ]]; then
|
|||||||
done < <(grep -v '^[[:space:]]*$' "$root/.env.builder")
|
done < <(grep -v '^[[:space:]]*$' "$root/.env.builder")
|
||||||
fi
|
fi
|
||||||
|
|
||||||
python_json() {
|
resolve_status_script() {
|
||||||
python3 -c "$1" "${@:2}"
|
local configured=${CI_STATUS_SCRIPT:-}
|
||||||
}
|
if [[ -z "$configured" && -f "$root/makefile.builder" ]]; then
|
||||||
|
configured=$(make -s --no-print-directory -C "$root" -f makefile.builder print-CI_STATUS_SCRIPT 2>/dev/null | tail -n 1 || true)
|
||||||
parse_origin() {
|
fi
|
||||||
python_json '
|
configured=${configured:-.pouch/builder/ci-status.sh}
|
||||||
import os, re
|
local path
|
||||||
url = os.environ["WAIT_CI_ORIGIN"].strip()
|
if [[ "$configured" == /* ]]; then
|
||||||
url = re.sub(r"^git@([^:]+):", r"ssh://git@\1/", url)
|
path=$configured
|
||||||
url = re.sub(r"^ssh://git@([^/]+):", r"ssh://git@\1/", url)
|
|
||||||
if re.match(r"^https?://", url):
|
|
||||||
m = re.match(r"^(https?)://([^/]+)(?:/(.*))?$", url)
|
|
||||||
scheme, host, path = (m.group(1), m.group(2), m.group(3) or "") if m else ("", "", "")
|
|
||||||
else:
|
|
||||||
m = re.match(r"^ssh://git@([^/]+)/(.+)$", url)
|
|
||||||
scheme, host, path = ("https", m.group(1), m.group(2)) if m else ("", "", "")
|
|
||||||
path = re.sub(r"\.git$", "", path)
|
|
||||||
override = os.environ.get("BUILDER_CI_FORGE", "").strip().lower()
|
|
||||||
if override in ("github", "gitlab", "gitea"):
|
|
||||||
kind = override
|
|
||||||
elif host == "github.com" or host.endswith(".github.com"):
|
|
||||||
kind = "github"
|
|
||||||
elif ("gitea" in host.lower() or os.environ.get("GITEA_TOKEN") or
|
|
||||||
os.environ.get("GITEA_API_URL")):
|
|
||||||
kind = "gitea"
|
|
||||||
else:
|
|
||||||
kind = "gitlab"
|
|
||||||
print(kind, host, path, (scheme or "https") + "://" + host)
|
|
||||||
'
|
|
||||||
}
|
|
||||||
|
|
||||||
origin=$(git -C "$root" remote get-url origin 2>/dev/null || true)
|
|
||||||
if [[ -z "$origin" ]]; then
|
|
||||||
echo "Error: no origin remote" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
|
|
||||||
export WAIT_CI_ORIGIN=$origin
|
|
||||||
read -r forge_kind forge_host forge_path forge_base <<<"$(parse_origin)"
|
|
||||||
if [[ -z "$forge_kind" || -z "$forge_path" ]]; then
|
|
||||||
echo "Error: cannot parse origin: $origin" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
export WAIT_CI_PATH=$forge_path
|
|
||||||
export WAIT_CI_SHA=$sha
|
|
||||||
|
|
||||||
gitlab_pipelines() {
|
|
||||||
local encoded
|
|
||||||
encoded=$(python_json 'import os,urllib.parse; print(urllib.parse.quote(os.environ["WAIT_CI_PATH"], safe=""))')
|
|
||||||
glab api "projects/${encoded}/pipelines?sha=${sha}&per_page=20"
|
|
||||||
}
|
|
||||||
|
|
||||||
gitlab_jobs() {
|
|
||||||
local encoded pid=$1
|
|
||||||
encoded=$(python_json 'import os,urllib.parse; print(urllib.parse.quote(os.environ["WAIT_CI_PATH"], safe=""))')
|
|
||||||
glab api "projects/${encoded}/pipelines/${pid}/jobs"
|
|
||||||
}
|
|
||||||
|
|
||||||
github_runs() {
|
|
||||||
gh run list --repo "$forge_path" --commit "$sha" --limit 20 \
|
|
||||||
--json databaseId,status,conclusion,url,headSha
|
|
||||||
}
|
|
||||||
|
|
||||||
gitea_api_base=${GITEA_API_URL:-$forge_base/api/v1}
|
|
||||||
gitea_api_base=${gitea_api_base%/}
|
|
||||||
if [[ "$gitea_api_base" != */api/v1 ]]; then
|
|
||||||
gitea_api_base+="/api/v1"
|
|
||||||
fi
|
|
||||||
gitea_repo_path=$(python_json 'import os,urllib.parse; print(urllib.parse.quote(os.environ["WAIT_CI_PATH"], safe="/"))')
|
|
||||||
|
|
||||||
gitea_curl() {
|
|
||||||
local url=$1
|
|
||||||
if [[ -n "${GITEA_TOKEN:-}" ]]; then
|
|
||||||
# Process substitution keeps the token out of curl's argument list and logs.
|
|
||||||
curl -fsS --retry 2 --connect-timeout 10 --max-time 30 \
|
|
||||||
--config <(printf 'header = "Authorization: token %s"\n' "$GITEA_TOKEN") "$url"
|
|
||||||
else
|
else
|
||||||
curl -fsS --retry 2 --connect-timeout 10 --max-time 30 "$url"
|
path=$root/$configured
|
||||||
|
case "$configured" in
|
||||||
|
.pouch/builder/*) ;;
|
||||||
|
*)
|
||||||
|
echo "Error: relative CI_STATUS_SCRIPT must be under .pouch/builder/: $configured" >&2
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
fi
|
fi
|
||||||
|
if [[ ! -f "$path" ]]; then
|
||||||
|
echo "Error: CI status script missing: $path" >&2
|
||||||
|
echo "Run builder init with BUILD_VENUE=ci to write .pouch/builder/ci-status.sh" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
if [[ ! -x "$path" ]]; then
|
||||||
|
echo "Error: CI status script is not executable: $path" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
printf '%s' "$path"
|
||||||
}
|
}
|
||||||
|
|
||||||
gitea_runs() {
|
parse_status() {
|
||||||
gitea_curl "${gitea_api_base}/repos/${gitea_repo_path}/actions/runs?head_sha=${sha}&limit=20&page=1"
|
python3 -c '
|
||||||
}
|
import sys
|
||||||
|
text = sys.stdin.read()
|
||||||
gitea_jobs() {
|
values = {}
|
||||||
gitea_curl "${gitea_api_base}/repos/${gitea_repo_path}/actions/runs/$1/jobs?limit=100&page=1"
|
for raw in text.replace("\n", " ").split():
|
||||||
}
|
if "=" in raw:
|
||||||
|
key, value = raw.split("=", 1)
|
||||||
summarize_gitlab() {
|
values[key.upper()] = value
|
||||||
python_json '
|
print(values.get("STATUS", "error").lower())
|
||||||
import json, sys
|
print(values.get("ID", "-"))
|
||||||
try:
|
print(values.get("PIPELINE", "-"))
|
||||||
data = json.loads(sys.stdin.read())
|
print(values.get("JOBS", ""))
|
||||||
except json.JSONDecodeError:
|
|
||||||
print("error")
|
|
||||||
raise SystemExit(0)
|
|
||||||
if isinstance(data, dict) and data.get("message"):
|
|
||||||
print("auth", data.get("message", "")[:80].replace(" ", "_"))
|
|
||||||
raise SystemExit(0)
|
|
||||||
if not isinstance(data, list):
|
|
||||||
print("error")
|
|
||||||
raise SystemExit(0)
|
|
||||||
if not data:
|
|
||||||
print("empty")
|
|
||||||
raise SystemExit(0)
|
|
||||||
p = data[0]
|
|
||||||
print(p.get("id"), p.get("status"), p.get("web_url", ""))
|
|
||||||
'
|
'
|
||||||
}
|
}
|
||||||
|
|
||||||
summarize_jobs() {
|
status_script=$(resolve_status_script)
|
||||||
python_json '
|
export BUILDER_PROJECT_ROOT=$root
|
||||||
import json, sys
|
|
||||||
data = json.loads(sys.stdin.read())
|
|
||||||
parts = []
|
|
||||||
for j in data if isinstance(data, list) else []:
|
|
||||||
parts.append("%s=%s" % (j.get("name"), j.get("status")))
|
|
||||||
print(",".join(parts))
|
|
||||||
'
|
|
||||||
}
|
|
||||||
|
|
||||||
summarize_github() {
|
query() {
|
||||||
python_json '
|
local out err rc
|
||||||
import json, sys
|
err=$(mktemp)
|
||||||
data = json.loads(sys.stdin.read())
|
set +e
|
||||||
if not data:
|
out=$(BUILDER_PROJECT_ROOT="$root" "$status_script" --sha "$sha" 2>"$err")
|
||||||
print("empty")
|
rc=$?
|
||||||
raise SystemExit(0)
|
set -e
|
||||||
r = data[0]
|
if [[ $rc -ne 0 ]]; then
|
||||||
status = r.get("status")
|
echo "Error: CI status script exited $rc" >&2
|
||||||
conc = r.get("conclusion")
|
sed '/token/Id; /Bearer/d; /Authorization/Id' "$err" >&2
|
||||||
if status == "completed" and conc == "success":
|
rm -f "$err"
|
||||||
st = "success"
|
if [[ $rc -eq 2 ]]; then
|
||||||
elif status == "completed":
|
exit 2
|
||||||
st = conc or "failed"
|
|
||||||
else:
|
|
||||||
st = status or "running"
|
|
||||||
print(r.get("databaseId"), st, r.get("url", ""))
|
|
||||||
'
|
|
||||||
}
|
|
||||||
|
|
||||||
summarize_gitea_runs() {
|
|
||||||
python_json '
|
|
||||||
import json, sys
|
|
||||||
try:
|
|
||||||
data = json.loads(sys.stdin.read())
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
print("error")
|
|
||||||
raise SystemExit(0)
|
|
||||||
if isinstance(data, dict) and data.get("message") and not (data.get("workflow_runs") or data.get("runs")):
|
|
||||||
print("auth", data.get("message", "")[:80].replace(" ", "_"))
|
|
||||||
raise SystemExit(0)
|
|
||||||
if isinstance(data, dict):
|
|
||||||
data = data.get("workflow_runs", data.get("runs", []))
|
|
||||||
if not isinstance(data, list):
|
|
||||||
print("error")
|
|
||||||
raise SystemExit(0)
|
|
||||||
want = sys.argv[1] if len(sys.argv) > 1 else ""
|
|
||||||
for run in data:
|
|
||||||
head = run.get("head_sha") or run.get("headSha") or (run.get("head_commit") or {}).get("id")
|
|
||||||
if not head or head.lower() != want.lower():
|
|
||||||
continue
|
|
||||||
run_id = run.get("id") or run.get("database_id")
|
|
||||||
if run_id is None:
|
|
||||||
continue
|
|
||||||
status = str(run.get("status") or "").lower()
|
|
||||||
conclusion = str(run.get("conclusion") or run.get("result") or "").lower()
|
|
||||||
url = run.get("html_url") or run.get("web_url") or run.get("url") or "-"
|
|
||||||
print("%s\t%s\t%s\t%s" % (run_id, status, conclusion, url))
|
|
||||||
' "$sha"
|
|
||||||
}
|
|
||||||
|
|
||||||
summarize_gitea_jobs() {
|
|
||||||
python_json '
|
|
||||||
import json, sys
|
|
||||||
try:
|
|
||||||
data = json.loads(sys.stdin.read())
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
print("error")
|
|
||||||
raise SystemExit(0)
|
|
||||||
if isinstance(data, dict) and data.get("message") and not data.get("jobs"):
|
|
||||||
print("auth", data.get("message", "")[:80].replace(" ", "_"))
|
|
||||||
raise SystemExit(0)
|
|
||||||
if isinstance(data, dict):
|
|
||||||
data = data.get("jobs", [])
|
|
||||||
if not isinstance(data, list):
|
|
||||||
print("error")
|
|
||||||
raise SystemExit(0)
|
|
||||||
want = sys.argv[1] if len(sys.argv) > 1 else ""
|
|
||||||
parts = []
|
|
||||||
state = "success" if data else "empty"
|
|
||||||
for job in data:
|
|
||||||
head = job.get("head_sha") or job.get("headSha")
|
|
||||||
if head and want and head.lower() != want.lower():
|
|
||||||
continue
|
|
||||||
name = str(job.get("name") or job.get("id") or "job").replace(" ", "_")
|
|
||||||
status = str(job.get("status") or "").lower()
|
|
||||||
result = str(job.get("conclusion") or job.get("result") or "").lower()
|
|
||||||
value = result or status or "unknown"
|
|
||||||
parts.append("%s=%s" % (name, value))
|
|
||||||
if value in ("failure", "failed", "cancelled", "canceled", "skipped", "timed_out", "action_required"):
|
|
||||||
state = "failed"
|
|
||||||
elif value not in ("success", "succeeded", "completed"):
|
|
||||||
state = "running"
|
|
||||||
if not parts:
|
|
||||||
state = "empty"
|
|
||||||
print("%s\t%s" % (state, ",".join(parts)))
|
|
||||||
' "$sha"
|
|
||||||
}
|
|
||||||
|
|
||||||
gitea_status_line() {
|
|
||||||
local raw runs
|
|
||||||
if ! raw=$(gitea_runs); then
|
|
||||||
echo "error"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
runs=$(printf '%s' "$raw" | summarize_gitea_runs)
|
|
||||||
if [[ "$runs" == auth* || "$runs" == error ]]; then
|
|
||||||
printf '%s\n' "$runs"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
if [[ -z "$runs" ]]; then
|
|
||||||
echo "empty"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
local overall=success ids= urls= jobs_all= first=1
|
|
||||||
while IFS=$'\t' read -r id run_status conclusion url; do
|
|
||||||
[[ -z "$id" ]] && continue
|
|
||||||
local jobs job_line job_state job_summary
|
|
||||||
if ! jobs=$(gitea_jobs "$id"); then
|
|
||||||
echo "error"
|
|
||||||
return 0
|
|
||||||
fi
|
fi
|
||||||
job_line=$(printf '%s' "$jobs" | summarize_gitea_jobs "$sha")
|
|
||||||
job_state=${job_line%%$'\t'*}
|
|
||||||
job_summary=${job_line#*$'\t'}
|
|
||||||
if [[ "$job_state" == auth* || "$job_state" == error ]]; then
|
|
||||||
printf '%s\n' "$job_state"
|
|
||||||
return 0
|
|
||||||
fi
|
|
||||||
if [[ "$first" -eq 0 ]]; then
|
|
||||||
ids+=","; urls+=","; jobs_all+=","
|
|
||||||
fi
|
|
||||||
ids+="$id"; urls+="$url"; jobs_all+="$job_summary"; first=0
|
|
||||||
if [[ "$run_status" =~ ^(failure|failed|cancelled|canceled|skipped|timed_out|action_required)$ || "$conclusion" =~ ^(failure|failed|cancelled|canceled|skipped|timed_out|action_required)$ || "$job_state" == failed ]]; then
|
|
||||||
overall=failed
|
|
||||||
elif [[ "$overall" == success && ( "$job_state" == running || "$job_state" == empty || ( "$run_status" != completed && "$run_status" != success ) ) ]]; then
|
|
||||||
overall=running
|
|
||||||
fi
|
|
||||||
done <<<"$runs"
|
|
||||||
printf '%s %s %s %s\n' "$ids" "$overall" "$urls" "$jobs_all"
|
|
||||||
}
|
|
||||||
|
|
||||||
if [[ "$forge_kind" == github ]]; then
|
|
||||||
if ! command -v gh >/dev/null 2>&1; then
|
|
||||||
echo "Error: gh is required to watch GitHub Actions for $forge_host" >&2
|
|
||||||
exit 3
|
exit 3
|
||||||
fi
|
fi
|
||||||
elif [[ "$forge_kind" == gitlab ]]; then
|
rm -f "$err"
|
||||||
if ! command -v glab >/dev/null 2>&1; then
|
printf '%s' "$out"
|
||||||
echo "Error: glab is required to watch GitLab CI for $forge_host" >&2
|
}
|
||||||
exit 3
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
if ! command -v curl >/dev/null 2>&1; then
|
|
||||||
echo "Error: curl is required to watch Gitea Actions for $forge_host" >&2
|
|
||||||
exit 3
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
started=$(date +%s)
|
started=$(date +%s)
|
||||||
first_seen=0
|
first_seen=0
|
||||||
empty_since=$started
|
empty_since=$started
|
||||||
|
|
||||||
probe_auth() {
|
|
||||||
local out rc kind
|
|
||||||
if [[ "$forge_kind" == github ]]; then
|
|
||||||
set +e
|
|
||||||
out=$(github_runs 2>&1)
|
|
||||||
rc=$?
|
|
||||||
set -e
|
|
||||||
if [[ $rc -ne 0 ]]; then
|
|
||||||
echo "Error: gh cannot list runs for $forge_path" >&2
|
|
||||||
echo "$out" | sed '/token/Id; /Bearer/d' >&2
|
|
||||||
exit 3
|
|
||||||
fi
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
if [[ "$forge_kind" == gitlab ]]; then
|
|
||||||
set +e
|
|
||||||
out=$(gitlab_pipelines 2>&1)
|
|
||||||
rc=$?
|
|
||||||
set -e
|
|
||||||
if [[ $rc -ne 0 ]]; then
|
|
||||||
echo "Error: glab cannot list pipelines for $forge_path on $forge_host" >&2
|
|
||||||
echo "$out" | sed '/token/Id; /Bearer/d' >&2
|
|
||||||
exit 3
|
|
||||||
fi
|
|
||||||
kind=$(printf '%s' "$out" | summarize_gitlab | awk '{print $1}')
|
|
||||||
if [[ "$kind" == auth ]]; then
|
|
||||||
echo "Error: glab is authenticated but cannot read $forge_path" >&2
|
|
||||||
exit 3
|
|
||||||
fi
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
set +e
|
|
||||||
out=$(gitea_runs 2>&1)
|
|
||||||
rc=$?
|
|
||||||
set -e
|
|
||||||
if [[ $rc -ne 0 ]]; then
|
|
||||||
echo "Error: curl cannot read Gitea Actions for $forge_path on $forge_host" >&2
|
|
||||||
echo "$out" | sed '/token/Id; /Bearer/d; /Authorization/Id' >&2
|
|
||||||
exit 3
|
|
||||||
fi
|
|
||||||
kind=$(printf '%s' "$out" | summarize_gitea_runs | awk '{print $1}')
|
|
||||||
if [[ "$kind" == auth ]]; then
|
|
||||||
echo "Error: Gitea API rejected access to $forge_path" >&2
|
|
||||||
exit 3
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
probe_auth
|
|
||||||
|
|
||||||
status_line() {
|
|
||||||
if [[ "$forge_kind" == github ]]; then
|
|
||||||
github_runs | summarize_github
|
|
||||||
elif [[ "$forge_kind" == gitlab ]]; then
|
|
||||||
gitlab_pipelines | summarize_gitlab
|
|
||||||
else
|
|
||||||
gitea_status_line
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
while true; do
|
while true; do
|
||||||
now=$(date +%s)
|
now=$(date +%s)
|
||||||
if (( now - started > timeout_sec )); then
|
if (( now - started > timeout_sec )); then
|
||||||
echo "FAIL reason=timeout sha=$sha" >&2
|
echo "FAIL reason=timeout sha=$sha" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
line=$(status_line)
|
raw=$(query)
|
||||||
set -- $line
|
mapfile -t fields < <(printf '%s' "$raw" | parse_status)
|
||||||
kind=${1:-empty}
|
st=${fields[0]:-error}
|
||||||
|
pid=${fields[1]:--}
|
||||||
if [[ "$kind" == empty ]]; then
|
url=${fields[2]:--}
|
||||||
if (( now - empty_since >= appear_sec )); then
|
jobs_s=${fields[3]:-}
|
||||||
echo "FAIL reason=no_pipeline sha=$sha origin=$forge_host/$forge_path" >&2
|
|
||||||
echo "This SHA has no CI pipeline. Tag or push a ref the project actually builds." >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
sleep "$poll_sec"
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
if [[ "$kind" == error || "$kind" == auth ]]; then
|
|
||||||
echo "Error: unexpected forge response for $forge_path" >&2
|
|
||||||
exit 3
|
|
||||||
fi
|
|
||||||
|
|
||||||
pid=$1
|
|
||||||
st=$2
|
|
||||||
url=$3
|
|
||||||
jobs_s=${4:-}
|
|
||||||
empty_since=$now
|
|
||||||
if [[ "$first_seen" -eq 0 ]]; then
|
|
||||||
echo "Watching pipeline $pid status=$st $url" >&2
|
|
||||||
first_seen=1
|
|
||||||
fi
|
|
||||||
|
|
||||||
case "$st" in
|
case "$st" in
|
||||||
|
empty)
|
||||||
|
if (( now - empty_since >= appear_sec )); then
|
||||||
|
echo "FAIL reason=no_pipeline sha=$sha" >&2
|
||||||
|
echo "This SHA has no CI pipeline. Tag or push a ref the project actually builds." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep "$poll_sec"
|
||||||
|
continue
|
||||||
|
;;
|
||||||
|
error)
|
||||||
|
echo "Error: unexpected CI status response" >&2
|
||||||
|
echo "$raw" | sed '/token/Id; /Bearer/d; /Authorization/Id' >&2
|
||||||
|
exit 3
|
||||||
|
;;
|
||||||
success)
|
success)
|
||||||
echo "OK pipeline=$url sha=$sha id=$pid jobs=$jobs_s"
|
echo "OK pipeline=$url sha=$sha id=$pid jobs=$jobs_s"
|
||||||
exit 0
|
exit 0
|
||||||
;;
|
;;
|
||||||
failed|failure|canceled|cancelled|skipped)
|
failed|failure|canceled|cancelled|skipped)
|
||||||
if [[ "$forge_kind" == gitlab ]]; then
|
echo "FAIL reason=job_${st} sha=$sha pipeline=$url jobs=$jobs_s" >&2
|
||||||
jobs_s=$(gitlab_jobs "$pid" | summarize_jobs || true)
|
|
||||||
fi
|
|
||||||
echo "FAIL reason=job_$st sha=$sha pipeline=$url jobs=$jobs_s" >&2
|
|
||||||
exit 1
|
exit 1
|
||||||
;;
|
;;
|
||||||
created|pending|running|waiting_for_resource|preparing|waiting|queued|in_progress|requested)
|
running|pending|queued|waiting|in_progress|created|preparing)
|
||||||
|
if [[ "$first_seen" -eq 0 ]]; then
|
||||||
|
echo "Watching pipeline $pid status=$st $url" >&2
|
||||||
|
first_seen=1
|
||||||
|
fi
|
||||||
|
empty_since=$now
|
||||||
sleep "$poll_sec"
|
sleep "$poll_sec"
|
||||||
;;
|
;;
|
||||||
*)
|
*)
|
||||||
|
empty_since=$now
|
||||||
sleep "$poll_sec"
|
sleep "$poll_sec"
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
|
|||||||
Executable
+99
@@ -0,0 +1,99 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Query Gitea Actions for an exact SHA via curl. Output STATUS= keys.
|
||||||
|
# Auth: GITEA_TOKEN in the environment or .env.builder. GITEA_API_URL optional.
|
||||||
|
|
||||||
|
sha=
|
||||||
|
while (($#)); do
|
||||||
|
case "$1" in
|
||||||
|
--sha) sha=$2; shift 2 ;;
|
||||||
|
*) echo "Usage: ci-status.sh --sha SHA" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
[[ -n "$sha" ]] || { echo "Usage: ci-status.sh --sha SHA" >&2; exit 2; }
|
||||||
|
|
||||||
|
if ! command -v curl >/dev/null 2>&1; then
|
||||||
|
echo "Error: curl is required" >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
root=${BUILDER_PROJECT_ROOT:-$(git rev-parse --show-toplevel)}
|
||||||
|
origin=$(git -C "$root" remote get-url origin 2>/dev/null || true)
|
||||||
|
[[ -n "$origin" ]] || { echo "Error: no origin remote" >&2; exit 3; }
|
||||||
|
|
||||||
|
read -r host path base <<<"$(WAIT_CI_ORIGIN=$origin python3 -c '
|
||||||
|
import os, re
|
||||||
|
url = os.environ["WAIT_CI_ORIGIN"].strip()
|
||||||
|
url = re.sub(r"^git@([^:]+):", r"ssh://git@\1/", url)
|
||||||
|
url = re.sub(r"^ssh://git@([^/]+):", r"ssh://git@\1/", url)
|
||||||
|
if re.match(r"^https?://", url):
|
||||||
|
m = re.match(r"^(https?)://([^/]+)(?:/(.*))?$", url)
|
||||||
|
scheme, host, path = (m.group(1), m.group(2), m.group(3) or "") if m else ("", "", "")
|
||||||
|
else:
|
||||||
|
m = re.match(r"^ssh://git@([^/]+)/(.+)$", url)
|
||||||
|
scheme, host, path = ("https", m.group(1), m.group(2)) if m else ("", "", "")
|
||||||
|
path = re.sub(r"\.git$", "", path)
|
||||||
|
print(host, path, (scheme or "https") + "://" + host)
|
||||||
|
')"
|
||||||
|
api=${GITEA_API_URL:-$base/api/v1}
|
||||||
|
api=${api%/}
|
||||||
|
[[ "$api" == */api/v1 ]] || api+=/api/v1
|
||||||
|
repo=$(WAIT_CI_PATH=$path python3 -c 'import os,urllib.parse; print(urllib.parse.quote(os.environ["WAIT_CI_PATH"], safe="/"))')
|
||||||
|
|
||||||
|
gitea_curl() {
|
||||||
|
if [[ -n "${GITEA_TOKEN:-}" ]]; then
|
||||||
|
curl -fsS --retry 2 --connect-timeout 10 --max-time 30 \
|
||||||
|
--config <(printf 'header = "Authorization: token %s"\n' "$GITEA_TOKEN") "$1"
|
||||||
|
else
|
||||||
|
curl -fsS --retry 2 --connect-timeout 10 --max-time 30 "$1"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
set +e
|
||||||
|
runs=$(gitea_curl "${api}/repos/${repo}/actions/runs?head_sha=${sha}&limit=20&page=1")
|
||||||
|
rc=$?
|
||||||
|
set -e
|
||||||
|
if [[ $rc -ne 0 ]]; then
|
||||||
|
echo "Error: curl cannot read Gitea Actions for $path" >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -c '
|
||||||
|
import json, sys
|
||||||
|
sha = sys.argv[1]
|
||||||
|
try:
|
||||||
|
data = json.loads(sys.stdin.read())
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
print("STATUS=error")
|
||||||
|
raise SystemExit(0)
|
||||||
|
if isinstance(data, dict) and data.get("message") and not (data.get("workflow_runs") or data.get("runs")):
|
||||||
|
print("STATUS=error")
|
||||||
|
raise SystemExit(0)
|
||||||
|
runs = data.get("workflow_runs", data.get("runs", [])) if isinstance(data, dict) else data
|
||||||
|
if not isinstance(runs, list) or not runs:
|
||||||
|
print("STATUS=empty")
|
||||||
|
raise SystemExit(0)
|
||||||
|
matched = []
|
||||||
|
for run in runs:
|
||||||
|
head = run.get("head_sha") or run.get("headSha") or (run.get("head_commit") or {}).get("id")
|
||||||
|
if head and head.lower() == sha.lower() and (run.get("id") or run.get("database_id")):
|
||||||
|
matched.append(run)
|
||||||
|
if not matched:
|
||||||
|
print("STATUS=empty")
|
||||||
|
raise SystemExit(0)
|
||||||
|
run = matched[0]
|
||||||
|
run_id = run.get("id") or run.get("database_id")
|
||||||
|
url = run.get("html_url") or run.get("web_url") or run.get("url") or "-"
|
||||||
|
status = str(run.get("status") or "").lower()
|
||||||
|
conclusion = str(run.get("conclusion") or run.get("result") or "").lower()
|
||||||
|
if conclusion in ("failure", "failed", "cancelled", "canceled", "skipped", "timed_out"):
|
||||||
|
state = "failed"
|
||||||
|
elif status in ("completed", "success") and conclusion in ("", "success", "succeeded"):
|
||||||
|
state = "success"
|
||||||
|
elif status == "completed" and conclusion:
|
||||||
|
state = "failed"
|
||||||
|
else:
|
||||||
|
state = "running"
|
||||||
|
print("STATUS=%s ID=%s PIPELINE=%s" % (state, run_id, url))
|
||||||
|
' "$sha" <<<"$runs"
|
||||||
Executable
+66
@@ -0,0 +1,66 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Query GitHub Actions for an exact SHA via gh. Output STATUS= keys.
|
||||||
|
|
||||||
|
sha=
|
||||||
|
while (($#)); do
|
||||||
|
case "$1" in
|
||||||
|
--sha) sha=$2; shift 2 ;;
|
||||||
|
*) echo "Usage: ci-status.sh --sha SHA" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
[[ -n "$sha" ]] || { echo "Usage: ci-status.sh --sha SHA" >&2; exit 2; }
|
||||||
|
|
||||||
|
if ! command -v gh >/dev/null 2>&1; then
|
||||||
|
echo "Error: gh is required" >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
root=${BUILDER_PROJECT_ROOT:-$(git rev-parse --show-toplevel)}
|
||||||
|
origin=$(git -C "$root" remote get-url origin 2>/dev/null || true)
|
||||||
|
[[ -n "$origin" ]] || { echo "Error: no origin remote" >&2; exit 3; }
|
||||||
|
repo=$(WAIT_CI_ORIGIN=$origin python3 -c '
|
||||||
|
import os, re
|
||||||
|
url = os.environ["WAIT_CI_ORIGIN"].strip()
|
||||||
|
url = re.sub(r"^git@([^:]+):", r"ssh://git@\1/", url)
|
||||||
|
url = re.sub(r"^ssh://git@([^/]+):", r"ssh://git@\1/", url)
|
||||||
|
if re.match(r"^https?://", url):
|
||||||
|
m = re.match(r"^https?://[^/]+(?:/(.*))?$", url)
|
||||||
|
path = m.group(1) if m else ""
|
||||||
|
else:
|
||||||
|
m = re.match(r"^ssh://git@[^/]+/(.+)$", url)
|
||||||
|
path = m.group(1) if m else ""
|
||||||
|
print(re.sub(r"\.git$", "", path or ""))
|
||||||
|
')
|
||||||
|
|
||||||
|
set +e
|
||||||
|
raw=$(gh run list --repo "$repo" --commit "$sha" --limit 20 --json databaseId,status,conclusion,url,headSha)
|
||||||
|
rc=$?
|
||||||
|
set -e
|
||||||
|
if [[ $rc -ne 0 ]]; then
|
||||||
|
echo "Error: gh cannot list runs for $repo" >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -c '
|
||||||
|
import json, sys
|
||||||
|
try:
|
||||||
|
data = json.loads(sys.stdin.read())
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
print("STATUS=error")
|
||||||
|
raise SystemExit(0)
|
||||||
|
if not data:
|
||||||
|
print("STATUS=empty")
|
||||||
|
raise SystemExit(0)
|
||||||
|
r = data[0]
|
||||||
|
status = str(r.get("status") or "").lower()
|
||||||
|
conc = str(r.get("conclusion") or "").lower()
|
||||||
|
if status == "completed" and conc == "success":
|
||||||
|
state = "success"
|
||||||
|
elif status == "completed":
|
||||||
|
state = "failed"
|
||||||
|
else:
|
||||||
|
state = "running"
|
||||||
|
print("STATUS=%s ID=%s PIPELINE=%s" % (state, r.get("databaseId") or "-", r.get("url") or "-"))
|
||||||
|
' <<<"$raw"
|
||||||
Executable
+71
@@ -0,0 +1,71 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Query GitLab CI for an exact SHA via glab. Output STATUS= keys.
|
||||||
|
|
||||||
|
sha=
|
||||||
|
while (($#)); do
|
||||||
|
case "$1" in
|
||||||
|
--sha) sha=$2; shift 2 ;;
|
||||||
|
*) echo "Usage: ci-status.sh --sha SHA" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
[[ -n "$sha" ]] || { echo "Usage: ci-status.sh --sha SHA" >&2; exit 2; }
|
||||||
|
|
||||||
|
if ! command -v glab >/dev/null 2>&1; then
|
||||||
|
echo "Error: glab is required" >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
root=${BUILDER_PROJECT_ROOT:-$(git rev-parse --show-toplevel)}
|
||||||
|
origin=$(git -C "$root" remote get-url origin 2>/dev/null || true)
|
||||||
|
[[ -n "$origin" ]] || { echo "Error: no origin remote" >&2; exit 3; }
|
||||||
|
path=$(WAIT_CI_ORIGIN=$origin python3 -c '
|
||||||
|
import os, re
|
||||||
|
url = os.environ["WAIT_CI_ORIGIN"].strip()
|
||||||
|
url = re.sub(r"^git@([^:]+):", r"ssh://git@\1/", url)
|
||||||
|
url = re.sub(r"^ssh://git@([^/]+):", r"ssh://git@\1/", url)
|
||||||
|
if re.match(r"^https?://", url):
|
||||||
|
m = re.match(r"^https?://[^/]+(?:/(.*))?$", url)
|
||||||
|
path = m.group(1) if m else ""
|
||||||
|
else:
|
||||||
|
m = re.match(r"^ssh://git@[^/]+/(.+)$", url)
|
||||||
|
path = m.group(1) if m else ""
|
||||||
|
print(re.sub(r"\.git$", "", path or ""))
|
||||||
|
')
|
||||||
|
encoded=$(WAIT_CI_PATH=$path python3 -c 'import os,urllib.parse; print(urllib.parse.quote(os.environ["WAIT_CI_PATH"], safe=""))')
|
||||||
|
|
||||||
|
set +e
|
||||||
|
raw=$(glab api "projects/${encoded}/pipelines?sha=${sha}&per_page=20")
|
||||||
|
rc=$?
|
||||||
|
set -e
|
||||||
|
if [[ $rc -ne 0 ]]; then
|
||||||
|
echo "Error: glab cannot list pipelines for $path" >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -c '
|
||||||
|
import json, sys
|
||||||
|
try:
|
||||||
|
data = json.loads(sys.stdin.read())
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
print("STATUS=error")
|
||||||
|
raise SystemExit(0)
|
||||||
|
if isinstance(data, dict) and data.get("message"):
|
||||||
|
print("STATUS=error")
|
||||||
|
raise SystemExit(0)
|
||||||
|
if not isinstance(data, list) or not data:
|
||||||
|
print("STATUS=empty")
|
||||||
|
raise SystemExit(0)
|
||||||
|
p = data[0]
|
||||||
|
st = str(p.get("status") or "").lower()
|
||||||
|
if st in ("success", "passed"):
|
||||||
|
state = "success"
|
||||||
|
elif st in ("failed", "canceled", "cancelled", "skipped"):
|
||||||
|
state = "failed"
|
||||||
|
elif st in ("",):
|
||||||
|
state = "empty"
|
||||||
|
else:
|
||||||
|
state = "running"
|
||||||
|
print("STATUS=%s ID=%s PIPELINE=%s" % (state, p.get("id") or "-", p.get("web_url") or "-"))
|
||||||
|
' <<<"$raw"
|
||||||
Executable
+80
@@ -0,0 +1,80 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Query Gitea Actions for an exact SHA via tea. Output STATUS= keys.
|
||||||
|
|
||||||
|
sha=
|
||||||
|
while (($#)); do
|
||||||
|
case "$1" in
|
||||||
|
--sha) sha=$2; shift 2 ;;
|
||||||
|
*) echo "Usage: ci-status.sh --sha SHA" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
[[ -n "$sha" ]] || { echo "Usage: ci-status.sh --sha SHA" >&2; exit 2; }
|
||||||
|
|
||||||
|
if ! command -v tea >/dev/null 2>&1; then
|
||||||
|
echo "Error: tea is required" >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
root=${BUILDER_PROJECT_ROOT:-$(git rev-parse --show-toplevel)}
|
||||||
|
origin=$(git -C "$root" remote get-url origin 2>/dev/null || true)
|
||||||
|
[[ -n "$origin" ]] || { echo "Error: no origin remote" >&2; exit 3; }
|
||||||
|
path=$(WAIT_CI_ORIGIN=$origin python3 -c '
|
||||||
|
import os, re
|
||||||
|
url = os.environ["WAIT_CI_ORIGIN"].strip()
|
||||||
|
url = re.sub(r"^git@([^:]+):", r"ssh://git@\1/", url)
|
||||||
|
url = re.sub(r"^ssh://git@([^/]+):", r"ssh://git@\1/", url)
|
||||||
|
if re.match(r"^https?://", url):
|
||||||
|
m = re.match(r"^https?://[^/]+(?:/(.*))?$", url)
|
||||||
|
path = m.group(1) if m else ""
|
||||||
|
else:
|
||||||
|
m = re.match(r"^ssh://git@[^/]+/(.+)$", url)
|
||||||
|
path = m.group(1) if m else ""
|
||||||
|
print(re.sub(r"\.git$", "", path or ""))
|
||||||
|
')
|
||||||
|
[[ -n "$path" ]] || { echo "Error: cannot parse origin" >&2; exit 3; }
|
||||||
|
|
||||||
|
set +e
|
||||||
|
runs=$(tea api "repos/${path}/actions/runs?head_sha=${sha}&limit=20")
|
||||||
|
rc=$?
|
||||||
|
set -e
|
||||||
|
if [[ $rc -ne 0 ]]; then
|
||||||
|
echo "Error: tea cannot read Actions for $path" >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 -c '
|
||||||
|
import json, sys
|
||||||
|
sha = sys.argv[1]
|
||||||
|
try:
|
||||||
|
data = json.loads(sys.stdin.read())
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
print("STATUS=error")
|
||||||
|
raise SystemExit(0)
|
||||||
|
if isinstance(data, dict) and data.get("message") and not (data.get("workflow_runs") or data.get("runs")):
|
||||||
|
print("STATUS=error")
|
||||||
|
raise SystemExit(0)
|
||||||
|
runs = data.get("workflow_runs", data.get("runs", [])) if isinstance(data, dict) else data
|
||||||
|
if not isinstance(runs, list) or not runs:
|
||||||
|
print("STATUS=empty")
|
||||||
|
raise SystemExit(0)
|
||||||
|
matched = [run for run in runs if str(run.get("head_sha") or run.get("headSha") or "").lower() == sha.lower()]
|
||||||
|
if not matched:
|
||||||
|
print("STATUS=empty")
|
||||||
|
raise SystemExit(0)
|
||||||
|
run = matched[0]
|
||||||
|
run_id = run.get("id") or run.get("database_id") or "-"
|
||||||
|
url = run.get("html_url") or run.get("web_url") or run.get("url") or "-"
|
||||||
|
status = str(run.get("status") or "").lower()
|
||||||
|
conclusion = str(run.get("conclusion") or run.get("result") or "").lower()
|
||||||
|
if conclusion in ("failure", "failed", "cancelled", "canceled", "skipped", "timed_out"):
|
||||||
|
state = "failed"
|
||||||
|
elif status in ("completed", "success") and conclusion in ("", "success", "succeeded"):
|
||||||
|
state = "success"
|
||||||
|
elif status == "completed" and conclusion:
|
||||||
|
state = "failed"
|
||||||
|
else:
|
||||||
|
state = "running"
|
||||||
|
print("STATUS=%s ID=%s PIPELINE=%s" % (state, run_id, url))
|
||||||
|
' "$sha" <<<"$runs"
|
||||||
@@ -11,8 +11,7 @@
|
|||||||
# DOCKER_REPOSITORY=
|
# DOCKER_REPOSITORY=
|
||||||
# IMAGE_TAG=
|
# IMAGE_TAG=
|
||||||
# PLATFORMS= # optional override; project ARCHES lives in makefile.builder
|
# PLATFORMS= # optional override; project ARCHES lives in makefile.builder
|
||||||
# BUILDER_CI_FORGE= # optional: github, gitlab, or gitea; useful for custom hosts
|
# GITEA_API_URL= # optional for the curl CI status script
|
||||||
# GITEA_API_URL= # optional: defaults to https://<origin-host>/api/v1
|
|
||||||
# GITEA_TOKEN= # optional token for private Gitea repositories
|
# GITEA_TOKEN= # optional token for private Gitea repositories
|
||||||
# COPY_DEST= # fallback: local dir, or docker dest container:/absolute/dir
|
# COPY_DEST= # fallback: local dir, or docker dest container:/absolute/dir
|
||||||
# COPY_SCRIPT= # external/temporary executable override; project-local paths belong in makefile.builder
|
# COPY_SCRIPT= # external/temporary executable override; project-local paths belong in makefile.builder
|
||||||
|
|||||||
@@ -8,9 +8,10 @@
|
|||||||
|
|
||||||
ARCHES ?= amd64
|
ARCHES ?= amd64
|
||||||
ARCH ?= $(firstword $(ARCHES))
|
ARCH ?= $(firstword $(ARCHES))
|
||||||
# local: make docker/deb on this machine. ci: obtain.sh waits for the
|
# local: make docker/deb on this machine. ci: obtain.sh polls
|
||||||
# forge pipeline of an exact SHA, then inspects the artifact. Not a track.
|
# CI_STATUS_SCRIPT for an exact SHA, then inspects the artifact. Not a track.
|
||||||
BUILD_VENUE ?= local
|
BUILD_VENUE ?= local
|
||||||
|
CI_STATUS_SCRIPT ?= .pouch/builder/ci-status.sh
|
||||||
ifeq ($(strip $(ARCHES)),)
|
ifeq ($(strip $(ARCHES)),)
|
||||||
$(error ARCHES must not be empty)
|
$(error ARCHES must not be empty)
|
||||||
endif
|
endif
|
||||||
|
|||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# test-preflight
|
||||||
|
|
||||||
|
Reusable preflight for tester matrices. It supports read-only `verify` and
|
||||||
|
test-environment-only `prepare`, and writes per-check YAML/CSV evidence for
|
||||||
|
artifacts, tester nodes, devices, network, services, and fixture/cleanup contracts.
|
||||||
|
This project assumes mobile album/photo/IME capabilities; journey cases exercise them
|
||||||
|
and preflight does not gate them or read authorization environment variables.
|
||||||
|
|
||||||
|
Run the self-tests with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m unittest discover -s skills/test-preflight/tests
|
||||||
|
python3 /home/ace/.codex/skills/.system/skill-creator/scripts/quick_validate.py skills/test-preflight
|
||||||
|
```
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
---
|
||||||
|
name: test-preflight
|
||||||
|
description: >-
|
||||||
|
Verify or prepare a mobile/integration test environment from a declared matrix:
|
||||||
|
artifacts, tester bindings, HOST/PUBLIC_HOST, service probes, and fixture/cleanup
|
||||||
|
contracts. Use before tester runs; it is not a deployment,
|
||||||
|
build, or business-data skill.
|
||||||
|
---
|
||||||
|
|
||||||
|
# test-preflight
|
||||||
|
|
||||||
|
Use this skill before `integration` or `journey` tests. It answers whether the
|
||||||
|
declared environment can start a run and emits one YAML and one CSV evidence
|
||||||
|
file. It never logs in, creates business data, builds/downloads an artifact, or
|
||||||
|
deploys production.
|
||||||
|
|
||||||
|
## Modes
|
||||||
|
|
||||||
|
- `verify` is read-only: validate the matrix, artifact identity, binding, actual
|
||||||
|
tester-node platform/device state, DNS/TCP/HTTP probes, backend probes, and
|
||||||
|
fixture/cleanup declarations. It must not install, launch, or deploy.
|
||||||
|
- `prepare` is side-effecting only within the test environment: call the
|
||||||
|
caller-supplied deployer command contract, run the same checks, then use the
|
||||||
|
tester-node command contract to install and launch the already supplied APK or
|
||||||
|
IPA. It never calls a builder and refuses `environment: prod`.
|
||||||
|
|
||||||
|
The command is:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 <skill-dir>/scripts/preflight.py verify \
|
||||||
|
--config <matrix.yaml> --yaml <preflight.yaml> --csv <preflight.csv>
|
||||||
|
python3 <skill-dir>/scripts/preflight.py prepare \
|
||||||
|
--config <matrix.yaml> --yaml <preflight.yaml> --csv <preflight.csv>
|
||||||
|
```
|
||||||
|
|
||||||
|
Read [references/schema.md](references/schema.md) for the input contract and
|
||||||
|
[references/boundaries.md](references/boundaries.md) before wiring a real
|
||||||
|
agent-runtime/deployer adapter. The included command-contract fixture is the
|
||||||
|
safe way to test remote-node behavior without pretending that this machine is a
|
||||||
|
Mac or that a deployment happened.
|
||||||
|
|
||||||
|
Overall status is `ready`, `blocked`, or `environment-fail`:
|
||||||
|
|
||||||
|
- `blocked` means a required declaration, artifact, binding, runner, fixture,
|
||||||
|
cleanup contract, or authorization is absent/unsafe.
|
||||||
|
- `environment-fail` means a declared environment exists but DNS/TCP/HTTP,
|
||||||
|
device/toolchain, installation, launch, or deployment execution failed.
|
||||||
|
- `ready` means every recorded check passed. A ready preflight permits the
|
||||||
|
caller to continue with `integration`; this skill does not run test cases.
|
||||||
|
|
||||||
|
This project assumes the client has the required album picker, photo permission, and Chinese
|
||||||
|
IME capabilities. Preflight does not probe or gate these capabilities, and does not read
|
||||||
|
`AUTHORIZE_ALBUM` or `IME_PINYIN`. Their real behavior is exercised by journey cases; a failure
|
||||||
|
there is a product test failure, not an environment block.
|
||||||
|
|
||||||
|
Do not treat a preflight report as a product result. A preflight failure means
|
||||||
|
no client session may be created; the tester owns `not-run` propagation and must write
|
||||||
|
one result row per selected testcase with `blocked_by: [test-preflight]`.
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
interface:
|
||||||
|
display_name: "Test Preflight"
|
||||||
|
short_description: "Verify or prepare tester environments"
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
# Reference moved
|
||||||
|
|
||||||
|
Read [schema.md](schema.md) for the matrix and evidence contract, and
|
||||||
|
[boundaries.md](boundaries.md) for agent-runtime/deployer/builder integration.
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Integration boundaries
|
||||||
|
|
||||||
|
`test-preflight` is an orchestration and evidence layer. It consumes existing
|
||||||
|
skills instead of implementing their policy or storage formats.
|
||||||
|
|
||||||
|
| Boundary | Caller-supplied contract | What preflight does | What it must not do |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| agent-runtime | resolved `tester.<name>` binding: binding id, node, environment, project/path, and a node runner | validate that the binding is explicit and route node probes through the runner | parse Orca state, invent a host, or create/reuse a terminal |
|
||||||
|
| deployer | `prepare` only: an argv command returning a JSON deployment receipt for `environment: test` | call it once and record its receipt/status | copy Compose/Argo logic, SSH directly, or accept production |
|
||||||
|
| builder/obtain | caller-provided local artifact path (and optional remote path), hash, source commit, package/bundle identity | hash and inspect the exact file; pass it to install contract | build, guess an old artifact, or silently download/upload |
|
||||||
|
|
||||||
|
## Deployer command contract
|
||||||
|
|
||||||
|
`prepare` requires:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
deployment:
|
||||||
|
environment: test
|
||||||
|
command: [python3, /path/to/existing-deployer-adapter.py]
|
||||||
|
```
|
||||||
|
|
||||||
|
The command receives JSON on stdin:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"operation":"prepare","environment":"test","project_root":"/abs/project","host":"...","public_host":"..."}
|
||||||
|
```
|
||||||
|
|
||||||
|
It must return `{ "status": "ready", "observed": "..." }` (an optional
|
||||||
|
`checks` mapping is recorded). Non-zero/invalid output is an
|
||||||
|
`environment-fail`; absent command or a non-test environment is `blocked`.
|
||||||
|
The adapter owns deployer-specific configuration, receipts, cleanup, and
|
||||||
|
authorization.
|
||||||
|
|
||||||
|
## Device contract
|
||||||
|
|
||||||
|
The node runner owns remote execution. For a local node the implementation uses
|
||||||
|
read-only `adb`/`xcrun simctl` checks in `verify`; `prepare` alone invokes
|
||||||
|
`adb install`/launch or `simctl install`/launch. For a remote node the same
|
||||||
|
operations are JSON requests to the runner. Missing xcrun on Linux, Android
|
||||||
|
`unauthorized`/`offline`, a missing device, or a failed install/start is an
|
||||||
|
`environment-fail`, not a product failure.
|
||||||
|
|
||||||
|
No client session is created by this skill. The runner's `install` and `start`
|
||||||
|
operations are the only device mutations, and are never called by `verify`.
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# test-preflight matrix schema
|
||||||
|
|
||||||
|
The input is YAML. `mode` is supplied by the CLI, so a file may be reused for
|
||||||
|
`verify` and `prepare`.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
schema_version: 1
|
||||||
|
project_root: /absolute/project/root
|
||||||
|
host: https://test.example.invalid
|
||||||
|
public_host: https://public-test.example.invalid # optional
|
||||||
|
service_probes:
|
||||||
|
- id: api-health
|
||||||
|
url: "{HOST}/health"
|
||||||
|
expected_status: [200]
|
||||||
|
fixtures:
|
||||||
|
required: true
|
||||||
|
references: [isolated-test-account]
|
||||||
|
cleanup:
|
||||||
|
required: true
|
||||||
|
steps: [delete-run-data]
|
||||||
|
# Optional project metadata. It is not a preflight gate.
|
||||||
|
capabilities:
|
||||||
|
assumed: [album_picker, photo_permission, ime_pinyin]
|
||||||
|
matrix:
|
||||||
|
- platform: android
|
||||||
|
tester:
|
||||||
|
binding: tester.android
|
||||||
|
node: local
|
||||||
|
runner: {kind: local}
|
||||||
|
artifact:
|
||||||
|
path: /absolute/dist/writes.apk
|
||||||
|
sha256: 012345...
|
||||||
|
package: page.writes
|
||||||
|
source_commit: 40-character-commit-sha
|
||||||
|
identity_command: [apkanalyzer, manifest, application-id, "{artifact}"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Required matrix fields are `platform` (`android` or `ios`), `tester.binding`,
|
||||||
|
`tester.node`, and `artifact.path`. Artifact identity is required as
|
||||||
|
`package` for Android or `bundle_id` for iOS. `sha256` and `source_commit` are
|
||||||
|
optional but recommended; when present they are checked exactly.
|
||||||
|
|
||||||
|
`service_probes` must contain at least one URL. `{HOST}` and `{PUBLIC_HOST}`
|
||||||
|
are expanded from the top-level values. Each probe is checked from the actual
|
||||||
|
tester node as DNS, TCP, and HTTP. A status outside 2xx–4xx fails transport
|
||||||
|
readiness; `expected_status` can require an exact allow-list. HTTP 401/403 can
|
||||||
|
therefore be a valid reachability result when explicitly expected.
|
||||||
|
|
||||||
|
`fixtures` and `cleanup` are declarations, not commands. A required fixture
|
||||||
|
needs non-empty `references`; a required cleanup needs non-empty `steps` or
|
||||||
|
`run`. For a read-only run, use `required: false` with a non-empty `reason`.
|
||||||
|
The skill never executes these values.
|
||||||
|
|
||||||
|
## Client capabilities
|
||||||
|
|
||||||
|
The project assumes the mobile test nodes provide the album picker, photo permission,
|
||||||
|
test image, and Chinese IME. These are not preflight declarations or gates. Journey
|
||||||
|
cases exercise the real UI; failures are recorded as product failures. `AUTHORIZE_ALBUM`
|
||||||
|
and `IME_PINYIN` are not read.
|
||||||
|
|
||||||
|
## Tester-node contract
|
||||||
|
|
||||||
|
`tester.runner.kind: local` executes platform and network probes on the current
|
||||||
|
node. It is only valid for `node: local`/`localhost`. A non-local node must use
|
||||||
|
`kind: contract` and a non-empty argv `command`. The command receives one JSON
|
||||||
|
request on stdin and returns one JSON object on stdout; no shell string is
|
||||||
|
accepted.
|
||||||
|
|
||||||
|
Requests have `operation` (`platform_probe`, `network_probe`, `install`, or
|
||||||
|
`start`), `platform`, `node`, and operation-specific fields. The response is:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "ready",
|
||||||
|
"observed": "short evidence",
|
||||||
|
"checks": {
|
||||||
|
"macos": {"status": "ready", "observed": "Darwin 24", "expected": "Darwin"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`checks` is optional; each key becomes a separate evidence row. A non-zero,
|
||||||
|
invalid, or unavailable runner is `environment-fail`. The adapter is expected
|
||||||
|
to execute on the resolved tester node, not on the coordinator. See
|
||||||
|
[boundaries.md](boundaries.md).
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
YAML contains `mode`, overall `status`, timestamps, a summary, and `checks`.
|
||||||
|
CSV uses these stable columns:
|
||||||
|
|
||||||
|
`check_id,mode,platform,node,target,observed,expected,status,remediation,timestamp`
|
||||||
|
|
||||||
|
Every check must use one of `ready`, `blocked`, or `environment-fail`; the
|
||||||
|
report does not invent `fail` or `not-run` testcase results.
|
||||||
@@ -0,0 +1,553 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Run the test-preflight verify/prepare contract."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import platform as host_platform
|
||||||
|
import plistlib
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import socket
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
import zipfile
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Mapping
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml
|
||||||
|
except ImportError as exc: # pragma: no cover
|
||||||
|
raise SystemExit("test-preflight 需要 PyYAML;请安装运行依赖后重试") from exc
|
||||||
|
|
||||||
|
|
||||||
|
PLATFORMS = {"android", "ios"}
|
||||||
|
STATUSES = {"ready", "blocked", "environment-fail"}
|
||||||
|
EVIDENCE_FIELDS = [
|
||||||
|
"check_id", "mode", "platform", "node", "target", "observed", "expected",
|
||||||
|
"status", "remediation", "timestamp",
|
||||||
|
]
|
||||||
|
SHA256 = re.compile(r"^[0-9a-fA-F]{64}$")
|
||||||
|
COMMIT = re.compile(r"^[0-9a-fA-F]{40}$")
|
||||||
|
|
||||||
|
|
||||||
|
def now() -> str:
|
||||||
|
return datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def text(value: Any) -> str:
|
||||||
|
if isinstance(value, (dict, list)):
|
||||||
|
return json.dumps(value, ensure_ascii=False, sort_keys=True)
|
||||||
|
return "" if value is None else str(value)
|
||||||
|
|
||||||
|
|
||||||
|
def status_from(value: Any, default: str = "environment-fail") -> str:
|
||||||
|
return value if value in STATUSES else default
|
||||||
|
|
||||||
|
|
||||||
|
def sha256(path: Path) -> str:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
with path.open("rb") as handle:
|
||||||
|
for block in iter(lambda: handle.read(1024 * 1024), b""):
|
||||||
|
digest.update(block)
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def command_text(command: list[str], *, payload: Mapping[str, Any] | None = None) -> tuple[int, str]:
|
||||||
|
try:
|
||||||
|
process = subprocess.run(
|
||||||
|
command,
|
||||||
|
input=(json.dumps(payload, ensure_ascii=False) if payload is not None else None),
|
||||||
|
text=True,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.STDOUT,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
except OSError as exc:
|
||||||
|
return 127, str(exc)
|
||||||
|
return process.returncode, process.stdout.strip()
|
||||||
|
|
||||||
|
|
||||||
|
class Evidence:
|
||||||
|
def __init__(self, mode: str) -> None:
|
||||||
|
self.mode = mode
|
||||||
|
self.started_at = now()
|
||||||
|
self.rows: list[dict[str, str]] = []
|
||||||
|
|
||||||
|
def add(
|
||||||
|
self,
|
||||||
|
check_id: str,
|
||||||
|
*,
|
||||||
|
platform: str = "all",
|
||||||
|
node: str = "coordinator",
|
||||||
|
target: str = "",
|
||||||
|
observed: Any = "",
|
||||||
|
expected: Any = "",
|
||||||
|
status: str,
|
||||||
|
remediation: str = "",
|
||||||
|
) -> None:
|
||||||
|
if status not in STATUSES:
|
||||||
|
raise ValueError(f"invalid evidence status: {status}")
|
||||||
|
self.rows.append({
|
||||||
|
"check_id": check_id,
|
||||||
|
"mode": self.mode,
|
||||||
|
"platform": platform,
|
||||||
|
"node": node,
|
||||||
|
"target": target,
|
||||||
|
"observed": text(observed),
|
||||||
|
"expected": text(expected),
|
||||||
|
"status": status,
|
||||||
|
"remediation": remediation,
|
||||||
|
"timestamp": now(),
|
||||||
|
})
|
||||||
|
|
||||||
|
def overall(self) -> str:
|
||||||
|
statuses = {row["status"] for row in self.rows}
|
||||||
|
if "environment-fail" in statuses:
|
||||||
|
return "environment-fail"
|
||||||
|
if "blocked" in statuses:
|
||||||
|
return "blocked"
|
||||||
|
return "ready"
|
||||||
|
|
||||||
|
def payload(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"schema_version": 1,
|
||||||
|
"mode": self.mode,
|
||||||
|
"status": self.overall(),
|
||||||
|
"started_at": self.started_at,
|
||||||
|
"finished_at": now(),
|
||||||
|
"summary": {state: sum(row["status"] == state for row in self.rows) for state in sorted(STATUSES)},
|
||||||
|
"checks": self.rows,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_config(path: Path) -> dict[str, Any]:
|
||||||
|
data = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
raise ValueError("配置根必须是 mapping")
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def expand(value: str, config: Mapping[str, Any]) -> str:
|
||||||
|
return value.replace("{HOST}", text(config.get("host"))).replace("{PUBLIC_HOST}", text(config.get("public_host")))
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_url(value: str) -> str:
|
||||||
|
return value if urllib.parse.urlsplit(value).scheme else "http://" + value.lstrip("/")
|
||||||
|
|
||||||
|
|
||||||
|
def argv(value: Any) -> list[str] | None:
|
||||||
|
if not isinstance(value, list) or not value or not all(isinstance(item, str) and item for item in value):
|
||||||
|
return None
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def validate_shape(config: Mapping[str, Any], evidence: Evidence) -> bool:
|
||||||
|
ok = True
|
||||||
|
if config.get("schema_version") != 1:
|
||||||
|
evidence.add("config-schema", target="config", observed=config.get("schema_version"), expected="1", status="blocked", remediation="将 schema_version 设为 1")
|
||||||
|
ok = False
|
||||||
|
if not isinstance(config.get("project_root"), str) or not Path(text(config.get("project_root"))).is_absolute():
|
||||||
|
evidence.add("config-project-root", target="project_root", observed=config.get("project_root"), expected="absolute project_root", status="blocked", remediation="提供项目根绝对路径")
|
||||||
|
ok = False
|
||||||
|
if not isinstance(config.get("host"), str) or not config["host"].strip():
|
||||||
|
evidence.add("config-host", target="HOST", expected="non-empty host URL", status="blocked", remediation="提供 HOST")
|
||||||
|
ok = False
|
||||||
|
probes = config.get("service_probes")
|
||||||
|
if not isinstance(probes, list) or not probes:
|
||||||
|
evidence.add("config-service-probes", target="service_probes", expected="at least one probe", status="blocked", remediation="声明服务 URL 探针")
|
||||||
|
ok = False
|
||||||
|
else:
|
||||||
|
for index, probe in enumerate(probes):
|
||||||
|
if not isinstance(probe, dict) or not isinstance(probe.get("url"), str) or not probe["url"].strip():
|
||||||
|
evidence.add(f"config-service-probe-{index}", target=f"service_probes[{index}]", expected="url", status="blocked", remediation="为每个服务探针提供 URL")
|
||||||
|
ok = False
|
||||||
|
matrix = config.get("matrix")
|
||||||
|
if not isinstance(matrix, list) or not matrix:
|
||||||
|
evidence.add("config-matrix", target="matrix", expected="non-empty list", status="blocked", remediation="声明至少一个平台矩阵行")
|
||||||
|
return False
|
||||||
|
seen: set[str] = set()
|
||||||
|
for index, row in enumerate(matrix):
|
||||||
|
prefix = f"matrix-{index}"
|
||||||
|
if not isinstance(row, dict):
|
||||||
|
evidence.add(prefix, target=f"matrix[{index}]", expected="mapping", status="blocked", remediation="修正矩阵行")
|
||||||
|
ok = False
|
||||||
|
continue
|
||||||
|
platform_name = text(row.get("platform")).lower()
|
||||||
|
if platform_name not in PLATFORMS or platform_name in seen:
|
||||||
|
evidence.add(f"{prefix}-platform", platform=platform_name or "unknown", target=f"matrix[{index}].platform", expected="unique android or ios", status="blocked", remediation="为每个平台保留唯一矩阵行")
|
||||||
|
ok = False
|
||||||
|
seen.add(platform_name)
|
||||||
|
tester = row.get("tester")
|
||||||
|
if not isinstance(tester, dict) or not isinstance(tester.get("binding"), str) or not tester["binding"].startswith("tester."):
|
||||||
|
evidence.add(f"{prefix}-binding", platform=platform_name, target=f"matrix[{index}].tester.binding", expected="tester.<name>", status="blocked", remediation="由 agent-runtime 解析并传入 tester binding")
|
||||||
|
ok = False
|
||||||
|
if not isinstance(tester, dict) or not isinstance(tester.get("node"), str) or not tester["node"].strip():
|
||||||
|
evidence.add(f"{prefix}-node", platform=platform_name, target=f"matrix[{index}].tester.node", expected="actual tester node", status="blocked", remediation="提供实际 tester 节点名")
|
||||||
|
ok = False
|
||||||
|
artifact = row.get("artifact")
|
||||||
|
identity_key = "package" if platform_name == "android" else "bundle_id"
|
||||||
|
if not isinstance(artifact, dict) or not isinstance(artifact.get("path"), str) or not artifact["path"].strip():
|
||||||
|
evidence.add(f"{prefix}-artifact", platform=platform_name, target=f"matrix[{index}].artifact.path", expected="caller-provided artifact path", status="blocked", remediation="提供本轮 APK/IPA,不要让 preflight 猜产物")
|
||||||
|
ok = False
|
||||||
|
if not isinstance(artifact, dict) or not isinstance(artifact.get(identity_key), str) or not artifact[identity_key].strip():
|
||||||
|
evidence.add(f"{prefix}-identity", platform=platform_name, target=f"matrix[{index}].artifact.{identity_key}", expected="declared artifact identity", status="blocked", remediation=f"声明 artifact.{identity_key}")
|
||||||
|
ok = False
|
||||||
|
if isinstance(artifact, dict) and artifact.get("sha256") is not None and not SHA256.fullmatch(text(artifact.get("sha256"))):
|
||||||
|
evidence.add(f"{prefix}-sha256", platform=platform_name, target=f"matrix[{index}].artifact.sha256", expected="64 hex characters", status="blocked", remediation="提供正确 SHA-256 或删除字段")
|
||||||
|
ok = False
|
||||||
|
if isinstance(artifact, dict) and artifact.get("source_commit") is not None and not COMMIT.fullmatch(text(artifact.get("source_commit"))):
|
||||||
|
evidence.add(f"{prefix}-commit", platform=platform_name, target=f"matrix[{index}].artifact.source_commit", expected="40-character commit SHA", status="blocked", remediation="提供产物来源 commit SHA")
|
||||||
|
ok = False
|
||||||
|
return ok
|
||||||
|
|
||||||
|
|
||||||
|
def artifact_identity(path: Path, platform_name: str, artifact: Mapping[str, Any]) -> tuple[str | None, str]:
|
||||||
|
if platform_name == "ios":
|
||||||
|
try:
|
||||||
|
with zipfile.ZipFile(path) as archive:
|
||||||
|
names = [name for name in archive.namelist() if re.fullmatch(r"Payload/[^/]+\.app/Info\.plist", name)]
|
||||||
|
if len(names) != 1:
|
||||||
|
return None, "IPA 中没有唯一 Payload/*.app/Info.plist"
|
||||||
|
plist = plistlib.loads(archive.read(names[0]))
|
||||||
|
value = plist.get("CFBundleIdentifier")
|
||||||
|
return (value, "Info.plist CFBundleIdentifier") if isinstance(value, str) else (None, "Info.plist 缺少 CFBundleIdentifier")
|
||||||
|
except (OSError, zipfile.BadZipFile, plistlib.InvalidFileException) as exc:
|
||||||
|
return None, f"IPA 读取失败: {exc}"
|
||||||
|
command = argv(artifact.get("identity_command"))
|
||||||
|
if command:
|
||||||
|
command = [item.replace("{artifact}", str(path)) for item in command]
|
||||||
|
else:
|
||||||
|
tool = shutil.which("apkanalyzer") or shutil.which("aapt2") or shutil.which("aapt")
|
||||||
|
if not tool:
|
||||||
|
return None, "缺少 apkanalyzer/aapt2/aapt;无法读取 APK package identity"
|
||||||
|
command = [tool, "manifest", "application-id", str(path)] if tool.endswith("apkanalyzer") else [tool, "dump", "badging", str(path)]
|
||||||
|
code, output = command_text(command)
|
||||||
|
if code != 0:
|
||||||
|
return None, output[-500:] or f"identity command exited {code}"
|
||||||
|
match = re.search(r"(?:application-id|name)\s*[:=]\s*['\"]?([A-Za-z0-9_.]+)", output)
|
||||||
|
if not match:
|
||||||
|
match = re.search(r"package:\s+name='([^']+)'", output)
|
||||||
|
if not match:
|
||||||
|
match = re.fullmatch(r"\s*([A-Za-z0-9_.]+)\s*", output)
|
||||||
|
return (match.group(1), output[-500:]) if match else (None, "identity command output did not contain package")
|
||||||
|
|
||||||
|
|
||||||
|
def record_artifact(evidence: Evidence, row: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
|
||||||
|
artifact = row.get("artifact") or {}
|
||||||
|
path = Path(text(artifact.get("path"))).expanduser()
|
||||||
|
prefix = f"{platform_name}-{index}-artifact"
|
||||||
|
if not path.is_file():
|
||||||
|
evidence.add(prefix, platform=platform_name, node=node, target=str(path), expected="existing caller-provided artifact", status="blocked", remediation="补齐 APK/IPA 路径;不要默认构建")
|
||||||
|
return False
|
||||||
|
actual_hash = sha256(path)
|
||||||
|
expected_hash = artifact.get("sha256")
|
||||||
|
if expected_hash and actual_hash.lower() != text(expected_hash).lower():
|
||||||
|
evidence.add(prefix + "-hash", platform=platform_name, node=node, target=str(path), observed=actual_hash, expected=expected_hash, status="blocked", remediation="确认产物路径或 SHA-256 与本轮构建身份一致")
|
||||||
|
return False
|
||||||
|
evidence.add(prefix + "-hash", platform=platform_name, node=node, target=str(path), observed=actual_hash, expected=expected_hash or "file SHA-256 recorded", status="ready")
|
||||||
|
identity, observed = artifact_identity(path, platform_name, artifact)
|
||||||
|
expected_identity = text(artifact.get("package" if platform_name == "android" else "bundle_id"))
|
||||||
|
if identity != expected_identity:
|
||||||
|
evidence.add(prefix + "-identity", platform=platform_name, node=node, target=str(path), observed=identity or observed, expected=expected_identity, status="environment-fail", remediation="使用与 catalog 身份一致的 APK/IPA,并重新校验产物")
|
||||||
|
return False
|
||||||
|
evidence.add(prefix + "-identity", platform=platform_name, node=node, target=str(path), observed=identity, expected=expected_identity, status="ready")
|
||||||
|
if artifact.get("source_commit"):
|
||||||
|
evidence.add(prefix + "-commit", platform=platform_name, node=node, target=str(path), observed=artifact.get("source_commit"), expected=artifact.get("source_commit"), status="ready")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def local_platform_probe(platform_name: str) -> tuple[str, dict[str, dict[str, Any]]]:
|
||||||
|
if platform_name == "android":
|
||||||
|
adb = shutil.which("adb")
|
||||||
|
if not adb:
|
||||||
|
return "environment-fail", {"adb": {"status": "environment-fail", "observed": "missing", "expected": "Android SDK platform-tools", "remediation": "在 tester.android 节点安装并配置 adb"}}
|
||||||
|
code, output = command_text([adb, "devices", "-l"])
|
||||||
|
states = [line.split() for line in output.splitlines()[1:] if line.strip()]
|
||||||
|
device = next((parts[0] for parts in states if len(parts) > 1 and parts[1] == "device"), "")
|
||||||
|
bad = next((parts for parts in states if len(parts) > 1 and parts[1] in {"unauthorized", "offline"}), None)
|
||||||
|
if bad:
|
||||||
|
return "environment-fail", {"adb": {"status": "ready", "observed": output, "expected": "adb responds"}, "device": {"status": "environment-fail", "observed": " ".join(bad), "expected": "authorized device", "remediation": "解锁 Android 设备并接受 RSA 授权;offline 时重启 adb/设备"}}
|
||||||
|
if code != 0 or not device:
|
||||||
|
return "environment-fail", {"adb": {"status": "environment-fail", "observed": output, "expected": "one authorized device", "remediation": "连接并授权 Android device/emulator"}}
|
||||||
|
boot_code, boot = command_text([adb, "-s", device, "shell", "getprop", "sys.boot_completed"])
|
||||||
|
booted = boot_code == 0 and boot.strip() == "1"
|
||||||
|
return ("ready" if booted else "environment-fail", {
|
||||||
|
"adb": {"status": "ready", "observed": adb, "expected": "adb available"},
|
||||||
|
"device": {"status": "ready", "observed": device, "expected": "authorized device"},
|
||||||
|
"boot": {"status": "ready" if booted else "environment-fail", "observed": boot, "expected": "sys.boot_completed=1", "remediation": "等待 emulator boot complete 后重跑"},
|
||||||
|
})
|
||||||
|
if host_platform.system() != "Darwin":
|
||||||
|
return "environment-fail", {"macos": {"status": "environment-fail", "observed": host_platform.system(), "expected": "Darwin/macOS tester.ios node", "remediation": "将 tester.ios 指向可用 macOS,不要在 Linux 伪装 iOS"}}
|
||||||
|
xcrun = shutil.which("xcrun")
|
||||||
|
if not xcrun:
|
||||||
|
return "environment-fail", {"xcrun": {"status": "environment-fail", "observed": "missing", "expected": "xcrun and simctl", "remediation": "在 Mac 安装 Xcode Command Line Tools/Xcode"}}
|
||||||
|
code, output = command_text([xcrun, "simctl", "list", "devices", "available"])
|
||||||
|
booted = any("(Booted)" in line for line in output.splitlines())
|
||||||
|
return ("ready" if code == 0 and booted else "environment-fail", {
|
||||||
|
"macos": {"status": "ready", "observed": "Darwin", "expected": "Darwin/macOS"},
|
||||||
|
"xcrun": {"status": "ready" if code == 0 else "environment-fail", "observed": output[-500:], "expected": "simctl available", "remediation": "安装 Xcode 并接受许可"},
|
||||||
|
"boot": {"status": "ready" if booted else "environment-fail", "observed": "Booted" if booted else output[-300:], "expected": "Booted simulator", "remediation": "启动目标 Simulator 后重跑"},
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
def runner_call(runner: Mapping[str, Any], request: Mapping[str, Any]) -> tuple[str, str, dict[str, Any]]:
|
||||||
|
command = argv(runner.get("command"))
|
||||||
|
if not command:
|
||||||
|
return "blocked", "runner command missing", {}
|
||||||
|
code, output = command_text(command, payload=request)
|
||||||
|
if code != 0:
|
||||||
|
return "environment-fail", output[-800:], {}
|
||||||
|
try:
|
||||||
|
result = json.loads(output)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
return "environment-fail", output[-800:], {}
|
||||||
|
if not isinstance(result, dict):
|
||||||
|
return "environment-fail", "runner returned non-object", {}
|
||||||
|
return status_from(result.get("status")), text(result.get("observed", "")), result
|
||||||
|
|
||||||
|
|
||||||
|
def add_runner_checks(evidence: Evidence, result: Mapping[str, Any], *, prefix: str, platform_name: str, node: str, default_expected: str) -> str:
|
||||||
|
checks = result.get("checks")
|
||||||
|
if isinstance(checks, dict) and checks:
|
||||||
|
statuses: list[str] = []
|
||||||
|
for name, detail in checks.items():
|
||||||
|
detail = detail if isinstance(detail, dict) else {"observed": detail}
|
||||||
|
status = status_from(detail.get("status"), status_from(result.get("status")))
|
||||||
|
statuses.append(status)
|
||||||
|
evidence.add(f"{prefix}-{name}", platform=platform_name, node=node, target=text(detail.get("target", name)), observed=detail.get("observed", result.get("observed", "")), expected=detail.get("expected", default_expected), status=status, remediation=text(detail.get("remediation", "")))
|
||||||
|
return "environment-fail" if "environment-fail" in statuses else ("blocked" if "blocked" in statuses else "ready")
|
||||||
|
status = status_from(result.get("status"))
|
||||||
|
evidence.add(prefix, platform=platform_name, node=node, target=prefix, observed=result.get("observed", ""), expected=default_expected, status=status, remediation=text(result.get("remediation", "")))
|
||||||
|
return status
|
||||||
|
|
||||||
|
|
||||||
|
def require_runner_checks(evidence: Evidence, result: Mapping[str, Any], *, prefix: str, platform_name: str, node: str, required: set[str]) -> bool:
|
||||||
|
checks = result.get("checks")
|
||||||
|
missing = sorted(required - set(checks)) if isinstance(checks, dict) else sorted(required)
|
||||||
|
if not missing:
|
||||||
|
return True
|
||||||
|
evidence.add(prefix + "-contract", platform=platform_name, node=node, target="runner response", observed={"missing": missing}, expected=sorted(required), status="blocked", remediation="让 tester 节点 adapter 返回每个必需的 adb/boot、macOS/xcrun/simctl 或 DNS/TCP/HTTP 证据")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def network_local(url: str) -> list[dict[str, Any]]:
|
||||||
|
parsed = urllib.parse.urlsplit(url)
|
||||||
|
host = parsed.hostname or ""
|
||||||
|
port = parsed.port or (443 if parsed.scheme == "https" else 80)
|
||||||
|
try:
|
||||||
|
addresses = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
|
||||||
|
except OSError as exc:
|
||||||
|
return [{"name": "dns", "status": "environment-fail", "observed": str(exc), "expected": "resolvable hostname", "remediation": "检查 tester 节点 DNS/VPN/网络"}]
|
||||||
|
rows: list[dict[str, Any]] = [{"name": "dns", "status": "ready", "observed": sorted({item[4][0] for item in addresses}), "expected": "resolvable hostname"}]
|
||||||
|
try:
|
||||||
|
with socket.create_connection((host, port), timeout=5):
|
||||||
|
rows.append({"name": "tcp", "status": "ready", "observed": f"{host}:{port}", "expected": "TCP connection"})
|
||||||
|
except OSError as exc:
|
||||||
|
rows.append({"name": "tcp", "status": "environment-fail", "observed": str(exc), "expected": f"TCP {host}:{port}", "remediation": "检查端口、防火墙和测试节点到 HOST 的路由"})
|
||||||
|
return rows
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(urllib.request.Request(url, method="GET"), timeout=5) as response:
|
||||||
|
code = response.status
|
||||||
|
rows.append({"name": "http", "status": "ready" if 200 <= code < 500 else "environment-fail", "observed": code, "expected": "HTTP 2xx-4xx", "remediation": "检查后端健康状态" if code >= 500 else ""})
|
||||||
|
except urllib.error.HTTPError as exc:
|
||||||
|
rows.append({"name": "http", "status": "ready" if 200 <= exc.code < 500 else "environment-fail", "observed": exc.code, "expected": "HTTP 2xx-4xx", "remediation": "检查后端服务,不把 5xx 当成可用" if exc.code >= 500 else ""})
|
||||||
|
except (OSError, urllib.error.URLError) as exc:
|
||||||
|
rows.append({"name": "http", "status": "environment-fail", "observed": str(exc), "expected": "HTTP response", "remediation": "检查服务监听、代理、TLS 和节点网络"})
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def check_network(evidence: Evidence, config: Mapping[str, Any], row: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
|
||||||
|
tester = row.get("tester") or {}
|
||||||
|
runner = tester.get("runner") if isinstance(tester, dict) else None
|
||||||
|
is_local = node in {"local", "localhost", "127.0.0.1"}
|
||||||
|
statuses: list[str] = []
|
||||||
|
for probe_index, probe in enumerate(config.get("service_probes") or []):
|
||||||
|
raw_url = text(probe.get("url"))
|
||||||
|
if "{PUBLIC_HOST}" in raw_url and not text(config.get("public_host")).strip():
|
||||||
|
evidence.add(f"{platform_name}-{index}-network-{probe_index}-public-host", platform=platform_name, node=node, target=raw_url, expected="PUBLIC_HOST configured", status="blocked", remediation="提供 PUBLIC_HOST 或改用 HOST 探针")
|
||||||
|
statuses.append("blocked")
|
||||||
|
continue
|
||||||
|
url = normalize_url(expand(raw_url, config))
|
||||||
|
if not urllib.parse.urlsplit(url).hostname:
|
||||||
|
evidence.add(f"{platform_name}-{index}-network-{probe_index}-url", platform=platform_name, node=node, target=url, expected="valid probe URL", status="blocked", remediation="修正 service_probes URL 或 HOST/PUBLIC_HOST")
|
||||||
|
statuses.append("blocked")
|
||||||
|
continue
|
||||||
|
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
|
||||||
|
details = network_local(url)
|
||||||
|
expected_statuses = probe.get("expected_status")
|
||||||
|
for detail in details:
|
||||||
|
if detail["name"] == "http" and isinstance(expected_statuses, list) and detail["status"] == "ready" and detail["observed"] not in expected_statuses:
|
||||||
|
detail = {**detail, "status": "environment-fail", "expected": expected_statuses, "remediation": "检查服务探针的 expected_status 或后端响应"}
|
||||||
|
status = status_from(detail["status"])
|
||||||
|
statuses.append(status)
|
||||||
|
evidence.add(f"{platform_name}-{index}-network-{probe_index}-{detail['name']}", platform=platform_name, node=node, target=url, observed=detail["observed"], expected=detail["expected"], status=status, remediation=detail.get("remediation", ""))
|
||||||
|
elif isinstance(runner, dict) and runner.get("kind") == "contract":
|
||||||
|
binding = text(tester.get("binding")) if isinstance(tester, dict) else ""
|
||||||
|
status, observed, result = runner_call(runner, {"operation": "network_probe", "platform": platform_name, "node": node, "binding": binding, "url": url, "probe": probe})
|
||||||
|
result = result or {"status": status, "observed": observed}
|
||||||
|
complete = require_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-network-{probe_index}", platform_name=platform_name, node=node, required={"dns", "tcp", "http"})
|
||||||
|
network_status = add_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-network-{probe_index}", platform_name=platform_name, node=node, default_expected="DNS/TCP/HTTP reachable from actual tester node")
|
||||||
|
statuses.append(network_status if complete else "blocked")
|
||||||
|
else:
|
||||||
|
evidence.add(f"{platform_name}-{index}-network-{probe_index}", platform=platform_name, node=node, target=url, expected="node runner contract", status="blocked", remediation="远端 tester 必须提供 agent-runtime adapter runner")
|
||||||
|
statuses.append("blocked")
|
||||||
|
return "environment-fail" not in statuses and "blocked" not in statuses
|
||||||
|
|
||||||
|
|
||||||
|
def check_fixture_cleanup(evidence: Evidence, config: Mapping[str, Any], platform_name: str, node: str, index: int) -> bool:
|
||||||
|
ok = True
|
||||||
|
fixture = config.get("fixtures")
|
||||||
|
valid_fixture = isinstance(fixture, dict) and (not fixture.get("required", True) or bool(fixture.get("references")))
|
||||||
|
if not valid_fixture:
|
||||||
|
evidence.add(f"{platform_name}-{index}-fixture", platform=platform_name, node=node, target="fixtures", expected="required fixture references", status="blocked", remediation="声明隔离测试账号/fixture;不要在运行中临时创建")
|
||||||
|
ok = False
|
||||||
|
else:
|
||||||
|
evidence.add(f"{platform_name}-{index}-fixture", platform=platform_name, node=node, target="fixtures", observed=fixture.get("references"), expected="declared fixture references", status="ready")
|
||||||
|
cleanup = config.get("cleanup")
|
||||||
|
valid_noop = isinstance(cleanup, dict) and cleanup.get("required") is False and bool(text(cleanup.get("reason")).strip())
|
||||||
|
valid_steps = isinstance(cleanup, dict) and (bool(cleanup.get("steps")) or bool(text(cleanup.get("run")).strip()))
|
||||||
|
if not (valid_noop or valid_steps):
|
||||||
|
evidence.add(f"{platform_name}-{index}-cleanup", platform=platform_name, node=node, target="cleanup", expected="steps/run or explicit read-only reason", status="blocked", remediation="声明 teardown/rollback;清理缺失时禁止开始测试")
|
||||||
|
ok = False
|
||||||
|
else:
|
||||||
|
evidence.add(f"{platform_name}-{index}-cleanup", platform=platform_name, node=node, target="cleanup", observed=cleanup, expected="cleanup contract declared", status="ready")
|
||||||
|
return ok
|
||||||
|
|
||||||
|
|
||||||
|
def run(config: Mapping[str, Any], mode: str) -> Evidence:
|
||||||
|
evidence = Evidence(mode)
|
||||||
|
shape_ok = validate_shape(config, evidence)
|
||||||
|
deployment_ok = True
|
||||||
|
if mode == "prepare":
|
||||||
|
deployment = config.get("deployment")
|
||||||
|
environment = deployment.get("environment") if isinstance(deployment, dict) else None
|
||||||
|
command = argv(deployment.get("command")) if isinstance(deployment, dict) else None
|
||||||
|
if environment != "test":
|
||||||
|
evidence.add("deployment-environment", target="deployment.environment", observed=environment, expected="test", status="blocked", remediation="prepare 只允许 deployer 的 test 环境")
|
||||||
|
if not command:
|
||||||
|
evidence.add("deployment-command", target="deployment.command", expected="existing deployer adapter command", status="blocked", remediation="提供既有 deployer 的 JSON command contract;不要在 skill 内实现部署")
|
||||||
|
deployment_ok = False
|
||||||
|
elif not command:
|
||||||
|
evidence.add("deployment-command", target="deployment.command", expected="existing deployer adapter command", status="blocked", remediation="提供既有 deployer 的 JSON command contract;不要在 skill 内实现部署")
|
||||||
|
deployment_ok = False
|
||||||
|
else:
|
||||||
|
code, output = command_text(command, payload={"operation": "prepare", "environment": "test", "project_root": config.get("project_root"), "host": config.get("host"), "public_host": config.get("public_host")})
|
||||||
|
if code != 0:
|
||||||
|
evidence.add("deployment", target="test", observed=output[-800:], expected="deployer receipt status=ready", status="environment-fail", remediation="按 deployer 返回证据修复测试后端,再重跑 prepare")
|
||||||
|
deployment_ok = False
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
receipt = json.loads(output)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
receipt = {}
|
||||||
|
status = status_from(receipt.get("status"), "environment-fail") if isinstance(receipt, dict) else "environment-fail"
|
||||||
|
evidence.add("deployment", target="test", observed=receipt.get("observed", output[-800:]) if isinstance(receipt, dict) else output[-800:], expected="deployer receipt status=ready", status=status, remediation="检查既有 deployer 的测试环境 receipt" if status != "ready" else "")
|
||||||
|
deployment_ok = status == "ready"
|
||||||
|
if not shape_ok:
|
||||||
|
return evidence
|
||||||
|
for index, row in enumerate(config["matrix"]):
|
||||||
|
platform_name = text(row.get("platform")).lower()
|
||||||
|
tester = row.get("tester") or {}
|
||||||
|
node = text(tester.get("node"))
|
||||||
|
binding = text(tester.get("binding"))
|
||||||
|
evidence.add(f"{platform_name}-{index}-binding", platform=platform_name, node=node, target=binding, observed=binding, expected="resolved tester.<name> binding", status="ready")
|
||||||
|
artifact_ok = record_artifact(evidence, row, platform_name, node, index)
|
||||||
|
fixture_ok = check_fixture_cleanup(evidence, config, platform_name, node, index)
|
||||||
|
runner = tester.get("runner") if isinstance(tester, dict) else None
|
||||||
|
is_local = node in {"local", "localhost", "127.0.0.1"}
|
||||||
|
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
|
||||||
|
platform_status, details = local_platform_probe(platform_name)
|
||||||
|
for name, detail in details.items():
|
||||||
|
evidence.add(f"{platform_name}-{index}-platform-{name}", platform=platform_name, node=node, target=name, observed=detail.get("observed", ""), expected=detail.get("expected", "platform ready"), status=status_from(detail.get("status"), platform_status), remediation=detail.get("remediation", ""))
|
||||||
|
platform_ok = platform_status == "ready"
|
||||||
|
elif isinstance(runner, dict) and runner.get("kind") == "contract":
|
||||||
|
status, observed, result = runner_call(runner, {"operation": "platform_probe", "platform": platform_name, "node": node, "binding": binding})
|
||||||
|
result = result or {"status": status, "observed": observed}
|
||||||
|
required = {"adb", "device", "boot"} if platform_name == "android" else {"macos", "xcrun", "simctl", "boot"}
|
||||||
|
complete = require_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-platform", platform_name=platform_name, node=node, required=required)
|
||||||
|
platform_status = add_runner_checks(evidence, result, prefix=f"{platform_name}-{index}-platform", platform_name=platform_name, node=node, default_expected="platform/device ready")
|
||||||
|
platform_ok = complete and platform_status == "ready"
|
||||||
|
else:
|
||||||
|
evidence.add(f"{platform_name}-{index}-platform-runner", platform=platform_name, node=node, target=node, expected="node runner contract", status="blocked", remediation="远端 tester 必须由 agent-runtime 提供可执行 runner")
|
||||||
|
platform_ok = False
|
||||||
|
network_ok = check_network(evidence, config, row, platform_name, node, index)
|
||||||
|
# This isolated project assumes the client capabilities used by its
|
||||||
|
# journeys. Their real behavior is a product assertion, not a preflight
|
||||||
|
# environment gate.
|
||||||
|
row_ok = artifact_ok and fixture_ok and platform_ok and network_ok
|
||||||
|
if mode == "prepare":
|
||||||
|
artifact = row.get("artifact") or {}
|
||||||
|
identity = artifact.get("package" if platform_name == "android" else "bundle_id")
|
||||||
|
target_path = text(artifact.get("remote_path") or artifact.get("path"))
|
||||||
|
if not row_ok or not deployment_ok:
|
||||||
|
evidence.add(f"{platform_name}-{index}-install", platform=platform_name, node=node, target=target_path, observed="not-run", expected="verify and deployment ready", status="blocked", remediation="先修复前置检查;prepare 不伪造安装成功")
|
||||||
|
evidence.add(f"{platform_name}-{index}-start", platform=platform_name, node=node, target=text(identity), observed="not-run", expected="install ready", status="blocked", remediation="安装前置失败,未启动 app")
|
||||||
|
else:
|
||||||
|
install_request = {"operation": "install", "platform": platform_name, "node": node, "artifact": target_path, "identity": identity}
|
||||||
|
start_request = {"operation": "start", "platform": platform_name, "node": node, "identity": identity}
|
||||||
|
if is_local and (not isinstance(runner, dict) or runner.get("kind", "local") == "local"):
|
||||||
|
if platform_name == "android":
|
||||||
|
adb = shutil.which("adb") or "adb"
|
||||||
|
serial = os.environ.get("ANDROID_SERIAL", "")
|
||||||
|
install_code, install_output = command_text([adb, "-s", serial, "install", "-r", target_path] if serial else [adb, "install", "-r", target_path])
|
||||||
|
install_result = {"status": "ready" if install_code == 0 else "environment-fail", "observed": install_output}
|
||||||
|
launch = [adb, "-s", serial, "shell", "monkey", "-p", text(identity), "1"] if serial else [adb, "shell", "monkey", "-p", text(identity), "1"]
|
||||||
|
start_code, start_output = command_text(launch) if install_code == 0 else (1, "install failed")
|
||||||
|
start_result = {"status": "ready" if start_code == 0 else "environment-fail", "observed": start_output}
|
||||||
|
else:
|
||||||
|
install_result = {"status": "environment-fail", "observed": "local iOS install requires a macOS runner", "remediation": "use tester.ios contract runner on Mac"}
|
||||||
|
start_result = {"status": "blocked", "observed": "not-run", "remediation": "install failed"}
|
||||||
|
add_runner_checks(evidence, install_result, prefix=f"{platform_name}-{index}-install", platform_name=platform_name, node=node, default_expected="artifact installed")
|
||||||
|
add_runner_checks(evidence, start_result, prefix=f"{platform_name}-{index}-start", platform_name=platform_name, node=node, default_expected="app launched")
|
||||||
|
elif isinstance(runner, dict) and runner.get("kind") == "contract":
|
||||||
|
install_status, install_observed, install_result = runner_call(runner, install_request)
|
||||||
|
add_runner_checks(evidence, install_result or {"status": install_status, "observed": install_observed}, prefix=f"{platform_name}-{index}-install", platform_name=platform_name, node=node, default_expected="artifact installed")
|
||||||
|
if install_status == "ready":
|
||||||
|
start_status, start_observed, start_result = runner_call(runner, start_request)
|
||||||
|
add_runner_checks(evidence, start_result or {"status": start_status, "observed": start_observed}, prefix=f"{platform_name}-{index}-start", platform_name=platform_name, node=node, default_expected="app launched")
|
||||||
|
else:
|
||||||
|
evidence.add(f"{platform_name}-{index}-start", platform=platform_name, node=node, target=text(identity), observed="not-run", expected="install ready", status="blocked", remediation="安装失败,先修复 tester 节点")
|
||||||
|
return evidence
|
||||||
|
|
||||||
|
|
||||||
|
def write_evidence(evidence: Evidence, yaml_path: Path, csv_path: Path) -> dict[str, Any]:
|
||||||
|
payload = evidence.payload()
|
||||||
|
yaml_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
csv_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
yaml_path.write_text(yaml.safe_dump(payload, allow_unicode=True, sort_keys=False), encoding="utf-8")
|
||||||
|
with csv_path.open("w", newline="", encoding="utf-8") as handle:
|
||||||
|
writer = csv.DictWriter(handle, fieldnames=EVIDENCE_FIELDS)
|
||||||
|
writer.writeheader()
|
||||||
|
writer.writerows(evidence.rows)
|
||||||
|
return {"status": payload["status"], "yaml": str(yaml_path), "csv": str(csv_path), "checks": len(evidence.rows), "summary": payload["summary"]}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description="Verify or prepare a test environment and emit YAML/CSV evidence")
|
||||||
|
parser.add_argument("mode", choices=("verify", "prepare"))
|
||||||
|
parser.add_argument("--config", required=True, type=Path)
|
||||||
|
parser.add_argument("--yaml", dest="yaml_path", type=Path, default=Path("preflight.yaml"))
|
||||||
|
parser.add_argument("--csv", dest="csv_path", type=Path, default=Path("preflight.csv"))
|
||||||
|
args = parser.parse_args()
|
||||||
|
try:
|
||||||
|
config = load_config(args.config)
|
||||||
|
evidence = run(config, args.mode)
|
||||||
|
result = write_evidence(evidence, args.yaml_path, args.csv_path)
|
||||||
|
except (OSError, ValueError, yaml.YAMLError) as exc:
|
||||||
|
print(json.dumps({"status": "blocked", "error": str(exc)}, ensure_ascii=False), file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||||
|
return 0 if result["status"] == "ready" else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Deterministic runner fixture for test-preflight contract tests."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
request = json.load(sys.stdin)
|
||||||
|
scenario = os.environ.get("PREFLIGHT_RUNNER_SCENARIO", "ready")
|
||||||
|
operation = request.get("operation")
|
||||||
|
|
||||||
|
if operation in {"deployment", "prepare"}:
|
||||||
|
print(json.dumps({"status": "ready", "observed": "mock test deploy receipt"}))
|
||||||
|
elif operation == "platform_probe":
|
||||||
|
platform = request.get("platform")
|
||||||
|
if scenario == "android-unauthorized" and platform == "android":
|
||||||
|
print(json.dumps({"status": "environment-fail", "checks": {
|
||||||
|
"adb": {"status": "ready", "observed": "adb 35", "expected": "adb responds"},
|
||||||
|
"device": {"status": "environment-fail", "observed": "emulator-5554 unauthorized", "expected": "authorized device", "remediation": "accept RSA authorization"},
|
||||||
|
"boot": {"status": "environment-fail", "observed": "unknown", "expected": "boot complete"},
|
||||||
|
}}))
|
||||||
|
elif scenario == "mac-unavailable" and platform == "ios":
|
||||||
|
print(json.dumps({"status": "environment-fail", "checks": {
|
||||||
|
"macos": {"status": "environment-fail", "observed": "Mac unavailable", "expected": "Darwin tester node", "remediation": "restore tester.ios Mac"},
|
||||||
|
}}))
|
||||||
|
else:
|
||||||
|
print(json.dumps({"status": "ready", "checks": {
|
||||||
|
"adb": {"status": "ready", "observed": "adb 35", "expected": "adb responds"},
|
||||||
|
"device": {"status": "ready", "observed": "authorized", "expected": "authorized device"},
|
||||||
|
"boot": {"status": "ready", "observed": "boot complete", "expected": "boot complete"},
|
||||||
|
"macos": {"status": "ready", "observed": "Darwin", "expected": "Darwin"},
|
||||||
|
"xcrun": {"status": "ready", "observed": "xcrun", "expected": "xcrun"},
|
||||||
|
"simctl": {"status": "ready", "observed": "available", "expected": "simctl"},
|
||||||
|
}}))
|
||||||
|
elif operation == "network_probe":
|
||||||
|
if scenario == "host-unreachable":
|
||||||
|
print(json.dumps({"status": "environment-fail", "checks": {
|
||||||
|
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
|
||||||
|
"tcp": {"status": "environment-fail", "observed": "connection refused", "expected": "TCP reachable", "remediation": "check HOST route"},
|
||||||
|
}}))
|
||||||
|
elif scenario == "http-failure":
|
||||||
|
print(json.dumps({"status": "environment-fail", "checks": {
|
||||||
|
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
|
||||||
|
"tcp": {"status": "ready", "observed": "443 open", "expected": "TCP reachable"},
|
||||||
|
"http": {"status": "environment-fail", "observed": 503, "expected": "HTTP 2xx-4xx", "remediation": "check service health"},
|
||||||
|
}}))
|
||||||
|
else:
|
||||||
|
print(json.dumps({"status": "ready", "checks": {
|
||||||
|
"dns": {"status": "ready", "observed": "10.0.0.2", "expected": "DNS resolves"},
|
||||||
|
"tcp": {"status": "ready", "observed": "443 open", "expected": "TCP reachable"},
|
||||||
|
"http": {"status": "ready", "observed": 200, "expected": "HTTP 2xx-4xx"},
|
||||||
|
}}))
|
||||||
|
elif operation == "capability_probe":
|
||||||
|
capability = request.get("capability")
|
||||||
|
checks = {
|
||||||
|
"album_picker": {
|
||||||
|
"picker": {"status": "ready", "observed": "real picker opened and returned image-001"},
|
||||||
|
},
|
||||||
|
"photo_permission": {
|
||||||
|
"permission": {"status": "ready", "observed": "permission state checked on device"},
|
||||||
|
"test_image": {"status": "ready", "observed": "image-001 available in test album"},
|
||||||
|
},
|
||||||
|
"ime_pinyin": {
|
||||||
|
"composition": {"status": "ready", "observed": "nihao composed in Chinese IME"},
|
||||||
|
"candidates": {"status": "ready", "observed": "你好 candidate selected"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if scenario == "capability-missing" and capability == "album_picker":
|
||||||
|
print(json.dumps({"status": "ready", "checks": {}}))
|
||||||
|
else:
|
||||||
|
print(json.dumps({"status": "ready", "checks": checks.get(capability, {})}))
|
||||||
|
elif operation in {"install", "start"}:
|
||||||
|
print(json.dumps({"status": "ready", "observed": operation + "ed mock artifact"}))
|
||||||
|
else:
|
||||||
|
print(json.dumps({"status": "environment-fail", "observed": "unknown operation"}))
|
||||||
@@ -0,0 +1,225 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import plistlib
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
import zipfile
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(ROOT / "scripts"))
|
||||||
|
import preflight # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
FIXTURE_RUNNER = Path(__file__).resolve().parent / "fixtures" / "contract_runner.py"
|
||||||
|
|
||||||
|
|
||||||
|
class PreflightTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.root = Path(self.temp.name)
|
||||||
|
self.apk = self.root / "writes.apk"
|
||||||
|
self.apk.write_bytes(b"mock apk")
|
||||||
|
self.ipa = self.root / "writes.ipa"
|
||||||
|
with zipfile.ZipFile(self.ipa, "w") as archive:
|
||||||
|
archive.writestr("Payload/Writes.app/Info.plist", plistlib.dumps({"CFBundleIdentifier": "page.writes"}))
|
||||||
|
self.identity = self.root / "identity.py"
|
||||||
|
self.identity.write_text("print('page.writes')\n", encoding="utf-8")
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
self.temp.cleanup()
|
||||||
|
|
||||||
|
def runner(self, scenario: str = "ready") -> dict[str, object]:
|
||||||
|
return {"kind": "contract", "command": [sys.executable, str(FIXTURE_RUNNER)]}
|
||||||
|
|
||||||
|
def config(self, *, platform: str = "android", artifact: Path | None = None, node: str = "tester.android", scenario: str = "ready") -> dict[str, object]:
|
||||||
|
artifact = artifact or self.apk
|
||||||
|
artifact_row: dict[str, object] = {
|
||||||
|
"path": str(artifact),
|
||||||
|
"package": "page.writes" if platform == "android" else None,
|
||||||
|
"bundle_id": "page.writes" if platform == "ios" else None,
|
||||||
|
}
|
||||||
|
if platform == "android":
|
||||||
|
artifact_row["identity_command"] = [sys.executable, str(self.identity)]
|
||||||
|
artifact_row.pop("bundle_id")
|
||||||
|
else:
|
||||||
|
artifact_row.pop("package")
|
||||||
|
return {
|
||||||
|
"schema_version": 1,
|
||||||
|
"project_root": str(self.root),
|
||||||
|
"host": "https://test.example.invalid",
|
||||||
|
"public_host": "https://public.example.invalid",
|
||||||
|
"service_probes": [{"id": "health", "url": "{HOST}/health", "expected_status": [200]}],
|
||||||
|
"fixtures": {"required": True, "references": ["isolated-account"]},
|
||||||
|
"cleanup": {"required": True, "steps": ["cleanup-run-data"]},
|
||||||
|
"matrix": [{
|
||||||
|
"platform": platform,
|
||||||
|
"tester": {"binding": "tester.android" if platform == "android" else "tester.ios", "node": node, "runner": self.runner(scenario)},
|
||||||
|
"artifact": artifact_row,
|
||||||
|
}],
|
||||||
|
}
|
||||||
|
|
||||||
|
def write_config(self, data: dict[str, object]) -> Path:
|
||||||
|
path = self.root / "matrix.yaml"
|
||||||
|
import yaml
|
||||||
|
path.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8")
|
||||||
|
return path
|
||||||
|
|
||||||
|
def invoke(self, mode: str, data: dict[str, object]) -> tuple[int, dict[str, object], dict[str, object]]:
|
||||||
|
config = self.write_config(data)
|
||||||
|
yaml_path, csv_path = self.root / f"{mode}.yaml", self.root / f"{mode}.csv"
|
||||||
|
env = os.environ.copy()
|
||||||
|
env["PREFLIGHT_RUNNER_SCENARIO"] = "ready"
|
||||||
|
process = subprocess.run([
|
||||||
|
sys.executable, str(ROOT / "scripts" / "preflight.py"), mode,
|
||||||
|
"--config", str(config), "--yaml", str(yaml_path), "--csv", str(csv_path),
|
||||||
|
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||||
|
import yaml
|
||||||
|
return process.returncode, yaml.safe_load(yaml_path.read_text(encoding="utf-8")), {"stdout": process.stdout, "stderr": process.stderr}
|
||||||
|
|
||||||
|
def test_missing_artifact_is_blocked_and_report_has_contract_columns(self):
|
||||||
|
data = self.config(artifact=self.root / "missing.apk")
|
||||||
|
code, report, _ = self.invoke("verify", data)
|
||||||
|
self.assertEqual(code, 1)
|
||||||
|
self.assertEqual(report["status"], "blocked")
|
||||||
|
row = next(row for row in report["checks"] if row["check_id"] == "android-0-artifact")
|
||||||
|
self.assertEqual(row["status"], "blocked")
|
||||||
|
self.assertEqual(set(row), set(preflight.EVIDENCE_FIELDS))
|
||||||
|
|
||||||
|
def test_android_unauthorized_is_environment_fail(self):
|
||||||
|
data = self.config()
|
||||||
|
config = self.write_config(data)
|
||||||
|
out = self.root / "unauthorized.yaml"
|
||||||
|
env = os.environ.copy()
|
||||||
|
env["PREFLIGHT_RUNNER_SCENARIO"] = "android-unauthorized"
|
||||||
|
process = subprocess.run([
|
||||||
|
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config), "--yaml", str(out), "--csv", str(self.root / "u.csv")
|
||||||
|
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||||
|
report = json.loads(json.dumps(__import__("yaml").safe_load(out.read_text(encoding="utf-8"))))
|
||||||
|
self.assertEqual(process.returncode, 1)
|
||||||
|
self.assertEqual(report["status"], "environment-fail")
|
||||||
|
self.assertEqual(next(row for row in report["checks"] if row["check_id"].endswith("-device"))["status"], "environment-fail")
|
||||||
|
|
||||||
|
def test_linux_without_xcrun_is_environment_fail(self):
|
||||||
|
data = self.config(platform="ios", artifact=self.ipa, node="local")
|
||||||
|
data["matrix"][0]["tester"]["runner"] = {"kind": "local"}
|
||||||
|
with mock.patch.object(preflight.host_platform, "system", return_value="Linux"), mock.patch.object(preflight, "network_local", return_value=[{"name": "dns", "status": "ready", "observed": "ok", "expected": "DNS"}, {"name": "tcp", "status": "ready", "observed": "ok", "expected": "TCP"}, {"name": "http", "status": "ready", "observed": 200, "expected": "HTTP"}]):
|
||||||
|
result = preflight.run(data, "verify")
|
||||||
|
self.assertEqual(result.overall(), "environment-fail")
|
||||||
|
row = next(row for row in result.rows if row["check_id"].endswith("-platform-macos"))
|
||||||
|
self.assertEqual(row["status"], "environment-fail")
|
||||||
|
|
||||||
|
def test_mac_unavailable_contract_is_environment_fail(self):
|
||||||
|
data = self.config(platform="ios", artifact=self.ipa)
|
||||||
|
config_path = self.write_config(data)
|
||||||
|
env = os.environ.copy()
|
||||||
|
env["PREFLIGHT_RUNNER_SCENARIO"] = "mac-unavailable"
|
||||||
|
output = self.root / "mac.yaml"
|
||||||
|
process = subprocess.run([
|
||||||
|
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "mac.csv")
|
||||||
|
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||||
|
import yaml
|
||||||
|
report = yaml.safe_load(output.read_text(encoding="utf-8"))
|
||||||
|
self.assertEqual(process.returncode, 1)
|
||||||
|
self.assertEqual(report["status"], "environment-fail")
|
||||||
|
self.assertTrue(any(row["check_id"].endswith("-macos") and row["status"] == "environment-fail" for row in report["checks"]))
|
||||||
|
|
||||||
|
def test_missing_fixture_or_cleanup_is_blocked(self):
|
||||||
|
data = self.config()
|
||||||
|
data["fixtures"] = {"required": True, "references": []}
|
||||||
|
data["cleanup"] = {"required": True, "steps": []}
|
||||||
|
result = preflight.run(data, "verify")
|
||||||
|
self.assertEqual(result.overall(), "blocked")
|
||||||
|
self.assertTrue(any(row["check_id"].endswith("-fixture") and row["status"] == "blocked" for row in result.rows))
|
||||||
|
self.assertTrue(any(row["check_id"].endswith("-cleanup") and row["status"] == "blocked" for row in result.rows))
|
||||||
|
|
||||||
|
def test_mobile_capabilities_are_assumed_and_not_a_preflight_gate(self):
|
||||||
|
data = self.config()
|
||||||
|
config_path = self.write_config(data)
|
||||||
|
env = os.environ.copy()
|
||||||
|
env["PREFLIGHT_RUNNER_SCENARIO"] = "capability-missing"
|
||||||
|
output = self.root / "capabilities.yaml"
|
||||||
|
process = subprocess.run([
|
||||||
|
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify",
|
||||||
|
"--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "capabilities.csv"),
|
||||||
|
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||||
|
report = __import__("yaml").safe_load(output.read_text(encoding="utf-8"))
|
||||||
|
self.assertEqual(process.returncode, 0)
|
||||||
|
self.assertEqual(report["status"], "ready")
|
||||||
|
self.assertFalse(any("capability-" in row["check_id"] for row in report["checks"]))
|
||||||
|
|
||||||
|
def test_host_unreachable_from_actual_runner_is_environment_fail(self):
|
||||||
|
data = self.config()
|
||||||
|
config_path = self.write_config(data)
|
||||||
|
env = os.environ.copy()
|
||||||
|
env["PREFLIGHT_RUNNER_SCENARIO"] = "host-unreachable"
|
||||||
|
output = self.root / "host.yaml"
|
||||||
|
process = subprocess.run([
|
||||||
|
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "host.csv")
|
||||||
|
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||||
|
import yaml
|
||||||
|
report = yaml.safe_load(output.read_text(encoding="utf-8"))
|
||||||
|
self.assertEqual(process.returncode, 1)
|
||||||
|
self.assertEqual(report["status"], "environment-fail")
|
||||||
|
self.assertTrue(any("network" in row["check_id"] and row["status"] == "environment-fail" for row in report["checks"]))
|
||||||
|
|
||||||
|
def test_http_500_is_environment_fail_not_product_fail(self):
|
||||||
|
data = self.config()
|
||||||
|
config_path = self.write_config(data)
|
||||||
|
env = os.environ.copy()
|
||||||
|
env["PREFLIGHT_RUNNER_SCENARIO"] = "http-failure"
|
||||||
|
output = self.root / "http.yaml"
|
||||||
|
process = subprocess.run([
|
||||||
|
sys.executable, str(ROOT / "scripts" / "preflight.py"), "verify", "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "http.csv")
|
||||||
|
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
|
||||||
|
import yaml
|
||||||
|
report = yaml.safe_load(output.read_text(encoding="utf-8"))
|
||||||
|
self.assertEqual(process.returncode, 1)
|
||||||
|
http_row = next(row for row in report["checks"] if row["check_id"].endswith("-http"))
|
||||||
|
self.assertEqual(http_row["observed"], "503")
|
||||||
|
self.assertEqual(http_row["status"], "environment-fail")
|
||||||
|
|
||||||
|
def test_prepare_without_deployer_is_blocked_and_never_installs(self):
|
||||||
|
data = self.config()
|
||||||
|
code, report, _ = self.invoke("prepare", data)
|
||||||
|
self.assertEqual(code, 1)
|
||||||
|
self.assertEqual(report["status"], "blocked")
|
||||||
|
self.assertFalse(any(row["check_id"].endswith("-install") and row["status"] == "ready" for row in report["checks"]))
|
||||||
|
self.assertTrue(any(row["check_id"] == "deployment-command" for row in report["checks"]))
|
||||||
|
|
||||||
|
def test_prepare_mock_contract_calls_install_and_start(self):
|
||||||
|
data = self.config()
|
||||||
|
data["deployment"] = {"environment": "test", "command": [sys.executable, str(FIXTURE_RUNNER)]}
|
||||||
|
code, report, _ = self.invoke("prepare", data)
|
||||||
|
self.assertEqual(code, 0)
|
||||||
|
self.assertEqual(report["status"], "ready")
|
||||||
|
ids = {row["check_id"] for row in report["checks"]}
|
||||||
|
self.assertIn("android-0-install", ids)
|
||||||
|
self.assertIn("android-0-start", ids)
|
||||||
|
|
||||||
|
def test_verify_never_calls_install_or_start(self):
|
||||||
|
data = self.config()
|
||||||
|
calls: list[str] = []
|
||||||
|
original = preflight.runner_call
|
||||||
|
|
||||||
|
def record(runner, request):
|
||||||
|
calls.append(str(request["operation"]))
|
||||||
|
return original(runner, request)
|
||||||
|
|
||||||
|
with mock.patch.object(preflight, "runner_call", side_effect=record):
|
||||||
|
result = preflight.run(data, "verify")
|
||||||
|
self.assertEqual(result.overall(), "ready")
|
||||||
|
self.assertNotIn("install", calls)
|
||||||
|
self.assertNotIn("start", calls)
|
||||||
|
self.assertNotIn("capability_probe", calls)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
+64
-3
@@ -21,13 +21,18 @@
|
|||||||
fan-out,逐个执行且不合并 cases。无需在根 `.pouch` 里增加未定义的 `include`。
|
fan-out,逐个执行且不合并 cases。无需在根 `.pouch` 里增加未定义的 `include`。
|
||||||
- 跑真实环境前,复制 `.pouch/tester/local.yaml.example` 为 `local.yaml`;模板默认
|
- 跑真实环境前,复制 `.pouch/tester/local.yaml.example` 为 `local.yaml`;模板默认
|
||||||
`HOST: "127.0.0.1"`。如果目标不是本机,显式改成目标地址。也可以用环境变量
|
`HOST: "127.0.0.1"`。如果目标不是本机,显式改成目标地址。也可以用环境变量
|
||||||
`HX_TEST_HOST` 覆盖。
|
`HOST` 覆盖。
|
||||||
- 测试默认在当前项目执行。也可以直接提供 Orca project id、项目名称或 terminal id;
|
- 测试默认在当前项目执行。也可以直接提供 Orca project id、项目名称或 terminal id;
|
||||||
tester 会自动解析 Agent、环境、项目和目录,写入 `local.yaml.agent`,并输出更新后的配置列表。
|
tester 会自动解析 Agent、环境、项目和目录,写入 `local.yaml.agent`,并输出更新后的配置列表。
|
||||||
每轮测试会创建新的 terminal,结束后自动关闭;不会复用或关闭已有 terminal。
|
每轮测试会创建新的 terminal,结束后自动关闭;不会复用或关闭已有 terminal。
|
||||||
- 测试环境部署由 deployer 在本机 Compose 中触发;正式环境部署由本机 `~/code/app00`
|
- 测试环境部署由 deployer 在本机 Compose 中触发;正式环境部署由本机 `~/code/app00`
|
||||||
项目的 fresh Agent terminal 触发,tester 不直接 SSH 或执行正式 Compose。
|
项目的 fresh Agent terminal 触发,tester 不直接 SSH 或执行正式 Compose。
|
||||||
- 会动硬件、会改数据的用例带风险级别(`readonly` / `mutating` / `destructive`)。默认只跑最低档 `readonly`。要跑更高风险,需要你明确说,例如「风险到 mutating」;碰哪个子系统另由 `gate`(如 hardware / weld)控制。
|
- 会动硬件、会改数据的用例带风险级别(`readonly` / `mutating` / `destructive`)。默认
|
||||||
|
`standard` 只跑最低档 `readonly`。隔离、可重置的 Writes 类环境在 catalog 设置
|
||||||
|
`execution_profile: isolated-full` 后,所有 active 自动化用例都进入执行,不因
|
||||||
|
`AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或
|
||||||
|
`AUTHORIZE_PASSWORD` skipped;risk 原值仍保留。设备、产物、服务和 cleanup/rollback
|
||||||
|
由 test-preflight 保证;客户端相册、权限和中文输入能力默认具备,由 journey 用例验证。
|
||||||
|
|
||||||
## 使用示例
|
## 使用示例
|
||||||
|
|
||||||
@@ -76,7 +81,8 @@ APK/IPA,为本轮创建隔离 worktree,在 Android 和 iOS tester 节点安
|
|||||||
不执行可用平台;只有全部可用才按目标环境走 deployer 路由、解析 Agent、输出生效
|
不执行可用平台;只有全部可用才按目标环境走 deployer 路由、解析 Agent、输出生效
|
||||||
列表、准备 Orca 项目并创建临时 terminal。脚本类、浏览器和移动模拟器都在对应
|
列表、准备 Orca 项目并创建临时 terminal。脚本类、浏览器和移动模拟器都在对应
|
||||||
Agent 环境中执行。`builder.<platform>` 不能替代 `tester.<platform>`。
|
Agent 环境中执行。`builder.<platform>` 不能替代 `tester.<platform>`。
|
||||||
4. 超出你允许的风险级别会跳过,不会偷偷执行。
|
4. `standard` 下超出你允许的风险级别会跳过;`isolated-full` 由 catalog profile 明确
|
||||||
|
全量 active 执行,不把风险悄悄改成 readonly。
|
||||||
5. 无论测试成功、失败或异常,都会关闭本轮创建的 terminal;跑完给出按分类计数的
|
5. 无论测试成功、失败或异常,都会关闭本轮创建的 terminal;跑完给出按分类计数的
|
||||||
结果,以及 `pass / fail / environment-fail / skipped / not-run`。
|
结果,以及 `pass / fail / environment-fail / skipped / not-run`。
|
||||||
|
|
||||||
@@ -85,3 +91,58 @@ APK/IPA,为本轮创建隔离 worktree,在 Android 和 iOS tester 节点安
|
|||||||
- **列出**:能看到分类表和总条数;`.pouch/tester/cases.csv` 已由 skill 里的 `generate_cases_csv.py` 从 `cases/` 重新生成。每次运行的结果保存在独立的 `.pouch/tester/runs/<run-id>.*` 文件中。
|
- **列出**:能看到分类表和总条数;`.pouch/tester/cases.csv` 已由 skill 里的 `generate_cases_csv.py` 从 `cases/` 重新生成。每次运行的结果保存在独立的 `.pouch/tester/runs/<run-id>.*` 文件中。
|
||||||
- **维护**:目标 id 能 `查看` 到,且检查通过。
|
- **维护**:目标 id 能 `查看` 到,且检查通过。
|
||||||
- **运行**:有本次 `runs/*.yaml` 快照和同名 `.csv` 成绩单(`executed` 列区分已跑 / 未跑);没填 HOST 时会停,而不是编造失败。
|
- **运行**:有本次 `runs/*.yaml` 快照和同名 `.csv` 成绩单(`executed` 列区分已跑 / 未跑);没填 HOST 时会停,而不是编造失败。
|
||||||
|
|
||||||
|
### 客户端用户旅程
|
||||||
|
|
||||||
|
测试用例支持两种顶层类型:`test_type: integration` 和 `test_type: journey`。集成套件
|
||||||
|
至少包括 `api-contract`、`authorization`、`build-contract`;客户端套件仍是
|
||||||
|
`client-smoke`、`client-regression`、`safety`。没有显式范围时默认不选择 integration。
|
||||||
|
|
||||||
|
执行顺序固定为 `test-preflight -> integration -> journey`。preflight 状态为 blocked 或
|
||||||
|
environment-fail 时不创建客户端 session;integration 失败时依赖 journey 为 `not-run`。
|
||||||
|
服务不可达是 environment-fail,契约断言错误是 integration fail,正确账号登录失败仍是
|
||||||
|
journey fail。`tester run`/bundle 必须传入 caller 提供的 matrix:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
<skill-dir>/tester run --preflight-config /abs/preflight.yaml \
|
||||||
|
--artifact android=/abs/app.apk --artifact ios=/abs/app.ipa
|
||||||
|
```
|
||||||
|
|
||||||
|
没有真实 deployer、Mac 或设备时,只能使用 test-preflight 的 command-contract mock fixture
|
||||||
|
验证协议,不能把 mock 结果写成真实运行成功。
|
||||||
|
|
||||||
|
客户端 E2E 用例使用 `client_suite: client-smoke | client-regression | safety`,并明确
|
||||||
|
`journey`、`fixture`、`session`、`cleanup`、`platform`。标准生命周期是:
|
||||||
|
|
||||||
|
```text
|
||||||
|
preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告
|
||||||
|
```
|
||||||
|
|
||||||
|
正确测试账号登录失败记 `fail`;设备、产物或服务不可达记 `environment-fail`。fixture 或
|
||||||
|
cleanup 缺失会在开始前阻断;前置失败后的剩余用例是 `not-run`,明确排除才是 `skipped`。
|
||||||
|
`safety` 必须显式选择和授权,不进入默认 smoke/full。它与普通 tester 用例共用原有
|
||||||
|
CSV/YAML 运行记录和汇总校验。
|
||||||
|
|
||||||
|
移动 preflight 还必须在实际 tester 节点验证相册 picker、照片权限与测试图片,以及中文
|
||||||
|
拼音的真实 composition 和候选词选择;缺失时所有 testcase 都保留 `not-run` +
|
||||||
|
`blocked_by`,不会静默 `skipped`。
|
||||||
|
|
||||||
|
失败回流使用 run YAML 顶层 `defects`/`test_issues` 与 `feedback`:PR 触发先更新 PR/关联
|
||||||
|
issue、通知 dispatcher 重新分配开发;需求/bug 触发直接回开发流程。可用
|
||||||
|
`scripts/record_issues.py` 生成同名 CSV 的结构化 issue 列。
|
||||||
|
|
||||||
|
### 汇总已有运行结果
|
||||||
|
|
||||||
|
对已经生成的同一运行 CSV/YAML,使用显式路径汇总:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 <skill-dir>/scripts/summarize_run.py \
|
||||||
|
--csv .pouch/tester/runs/<run-id>.csv \
|
||||||
|
--yaml .pouch/tester/runs/<run-id>.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
脚本只从 CSV 的 `result` 列计算 `pass`、`fail`、`environment-fail`、`skipped`、
|
||||||
|
`not-run` 五类状态,并要求五类计数和等于 CSV 数据行数。YAML 只用于校验
|
||||||
|
`results` 数量和 testcase id 集合;数量、集合、重复/缺失 id 或 CSV 缺失/未知状态
|
||||||
|
时返回非零并输出诊断。校验成功才表示这份汇总可作为运行报告;非零表示报告不一致,
|
||||||
|
不能据此判定产品失败。
|
||||||
|
|||||||
+56
-2
@@ -74,10 +74,10 @@ tester agent 配置。
|
|||||||
- `local.yaml.example` 的 HOST 默认是 `127.0.0.1`;如果目标不是本机,必须在
|
- `local.yaml.example` 的 HOST 默认是 `127.0.0.1`;如果目标不是本机,必须在
|
||||||
`local.yaml` 或环境变量中显式覆盖。`local.yaml` 不存在且没有环境变量时仍不猜
|
`local.yaml` 或环境变量中显式覆盖。`local.yaml` 不存在且没有环境变量时仍不猜
|
||||||
HOST,停下并记 environment-fail。
|
HOST,停下并记 environment-fail。
|
||||||
- 默认风险上限是 catalog 里的 `default_max_risk`。超出上限的条目标 `skipped`,禁止执行。
|
- 默认执行 profile 是 `standard`:风险上限来自 catalog 的 `default_max_risk`,超出上限的条目标 `skipped`,禁止执行。若 catalog 声明 `execution_profile: isolated-full`,只允许用于隔离且可重置的测试环境:所有 active 自动化用例都进入执行矩阵,不因 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或 `AUTHORIZE_PASSWORD` skipped;原有 `risk` 值必须保留并进入报告。设备、产物、服务和清理契约由 test-preflight 检查;相册、照片权限/测试图片和中文拼音能力在本项目中默认具备,由 journey 用例真实验证。
|
||||||
- `risk` 只能是 `readonly` / `mutating` / `destructive`,默认只跑 `readonly`。`tag=smoke` 只能配 `risk=readonly`。规则见 [levels.md](references/levels.md)。
|
- `risk` 只能是 `readonly` / `mutating` / `destructive`,默认只跑 `readonly`。`tag=smoke` 只能配 `risk=readonly`。规则见 [levels.md](references/levels.md)。
|
||||||
- `driver=browser` 必须开真实页面;`driver=mobile` 必须挂模拟器 session。二者都不得改成只打 API。
|
- `driver=browser` 必须开真实页面;`driver=mobile` 必须挂模拟器 session。二者都不得改成只打 API。
|
||||||
- `automation=assisted|manual` 需要人在场:无人值守时记 `not-run`,不要自己代跑人工步骤。见 [concepts.md](references/concepts.md)。
|
- `automation=assisted|manual` 需要人在场:无人值守时记 `not-run`,不要自己代跑人工步骤。isolated-full 取消的是风险/授权过滤,不会伪造人工操作。见 [concepts.md](references/concepts.md)。
|
||||||
- 浏览器、移动模拟器按 catalog `sessions` 初始化一次,同一次运行内复用。
|
- 浏览器、移动模拟器按 catalog `sessions` 初始化一次,同一次运行内复用。
|
||||||
- Agent、环境、项目目录和 Orca terminal 生命周期见 [execution.md](references/execution.md):可以从 project id/名称或 terminal id 自动解析,更新后输出配置列表;每轮都新建 terminal,结束后关闭本轮创建的 terminal,不复用或关闭已有 terminal。
|
- Agent、环境、项目目录和 Orca terminal 生命周期见 [execution.md](references/execution.md):可以从 project id/名称或 terminal id 自动解析,更新后输出配置列表;每轮都新建 terminal,结束后关闭本轮创建的 terminal,不复用或关闭已有 terminal。
|
||||||
- 环境准备见 [deployment.md](references/deployment.md):测试环境只调用 deployer 的本机 Compose 路径;正式环境只委派给本机 `~/code/app00` 的 fresh Agent terminal,tester 不直接 SSH 或执行生产 Compose。
|
- 环境准备见 [deployment.md](references/deployment.md):测试环境只调用 deployer 的本机 Compose 路径;正式环境只委派给本机 `~/code/app00` 的 fresh Agent terminal,tester 不直接 SSH 或执行生产 Compose。
|
||||||
@@ -145,3 +145,57 @@ python3 <skill-dir>/scripts/catalog.py show --root <project-root> --id <CASE_ID>
|
|||||||
4. 只有矩阵所有行都可用,才创建本轮测试 terminal/session,按 `select` 结果执行并记录成绩。
|
4. 只有矩阵所有行都可用,才创建本轮测试 terminal/session,按 `select` 结果执行并记录成绩。
|
||||||
|
|
||||||
测试环境由本机 deployer 触发,正式环境由 app00 Agent terminal 委派。随后按 agent 解析目标项目、目录和 Agent,更新配置并输出生效列表。移动端每个平台的产物位置必须由调用方给出,值为本地文件或 URL;缺了就停,不要猜 builder、CI release 或本机旧包。矩阵行记产物位置、身份和 `tester.<platform>` binding,并用 `<skill-dir>/tester run --artifact android=<path-or-url> --artifact ios=<path-or-url> --platform android,ios` 下载或取用、安装并准备本轮隔离 worktree。Android 在 `tester.android`、iOS 在 `tester.ios` 节点执行。未指定范围时等同「tag=smoke 且风险上限=默认且 automation=automated」。`select` 的 `run_together` 组按 [run.md](references/run.md) 整组执行。无论部署或测试成功、失败、超时还是中断,都要清理各自本轮创建的 terminal。
|
测试环境由本机 deployer 触发,正式环境由 app00 Agent terminal 委派。随后按 agent 解析目标项目、目录和 Agent,更新配置并输出生效列表。移动端每个平台的产物位置必须由调用方给出,值为本地文件或 URL;缺了就停,不要猜 builder、CI release 或本机旧包。矩阵行记产物位置、身份和 `tester.<platform>` binding,并用 `<skill-dir>/tester run --artifact android=<path-or-url> --artifact ios=<path-or-url> --platform android,ios` 下载或取用、安装并准备本轮隔离 worktree。Android 在 `tester.android`、iOS 在 `tester.ios` 节点执行。未指定范围时等同「tag=smoke 且风险上限=默认且 automation=automated」。`select` 的 `run_together` 组按 [run.md](references/run.md) 整组执行。无论部署或测试成功、失败、超时还是中断,都要清理各自本轮创建的 terminal。
|
||||||
|
|
||||||
|
### 客户端用户旅程标准
|
||||||
|
|
||||||
|
### 双层测试与执行顺序
|
||||||
|
|
||||||
|
用例顶层类型是 `test_type: integration | journey`。旧目录没有该字段时保持兼容:已有
|
||||||
|
`client_suite`/`journey` 视为 journey,已有 `integration_suite` 视为 integration。集成
|
||||||
|
套件至少包括 `api-contract`、`authorization`、`build-contract`;旅程套件仍固定为
|
||||||
|
`client-smoke`、`client-regression`、`safety`。集成用例不需要客户端 session、platform
|
||||||
|
或 journey 字段,不能被客户端套件选中。
|
||||||
|
|
||||||
|
一次运行严格遵守:
|
||||||
|
|
||||||
|
```text
|
||||||
|
test-preflight (blocked/environment-fail -> 不创建 session)
|
||||||
|
-> integration
|
||||||
|
-> journey (integration fail -> not-run,不创建 session)
|
||||||
|
```
|
||||||
|
|
||||||
|
`test-preflight` 的报告是环境证据,不计入 testcase 总数。服务不可达记
|
||||||
|
`environment-fail`;服务已达但契约断言错误记 integration `fail`;正确测试账号登录失败
|
||||||
|
仍记 journey `fail`。tester 的 `run`/bundle 入口必须接收 caller 提供的 preflight matrix,
|
||||||
|
没有 matrix 不得绕过闸门或伪造成功。
|
||||||
|
|
||||||
|
声明 `client_suite`(迁移期也接受 `suite`)的用例采用固定的三套客户端集合:
|
||||||
|
`client-smoke`、`client-regression`、`safety`。前两者分别覆盖最短只读主路径和隔离测试
|
||||||
|
环境中的普通用户回归;`safety` 覆盖危险/权限边界场景,必须显式选择和授权,不能被
|
||||||
|
默认 smoke/full 扩入。选择某套客户端旅程可用:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-smoke
|
||||||
|
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-regression --max-risk mutating
|
||||||
|
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite safety --max-risk destructive
|
||||||
|
```
|
||||||
|
|
||||||
|
客户端用例必须声明 `journey`、`fixture`、`session`、`cleanup` 和 `platform`。旅程包含
|
||||||
|
`goal`、`entry_state`、`actions`、`assertions`、`final_state`、`failure_policy`;fixture
|
||||||
|
至少声明测试账号;session 指向 catalog 中可复用的 browser/mobile session;cleanup 要有
|
||||||
|
步骤或命令(只读也要显式说明无需清理);platform 只能是 Android/iOS。具体字段见
|
||||||
|
`references/schema.md`。平台适配只能改操作层,不能把真实客户端操作换成 API 调用。
|
||||||
|
|
||||||
|
客户端运行器统一遵守 `preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告`。正确测试
|
||||||
|
账号登录失败是产品 `fail`;设备、产物、服务或 session 不可达才是 `environment-fail`。
|
||||||
|
fixture 或 cleanup 缺失会在 `catalog.py select` 的 preflight 阶段阻断,不产生虚假逐条结果。
|
||||||
|
设备、产物、服务或 session 能力缺失时 preflight 必须明确 blocked/environment-fail,并为所有选中的 testcase 生成
|
||||||
|
`not-run` 结果行;本项目默认具备相册、照片权限、测试图片或中文拼音 composition/candidate
|
||||||
|
能力,不在 preflight 中校验或静默变成 `skipped`,由 journey 真实验证并记录产品失败。独立用例继续执行,依赖失败的用例写 `not-run` 和 `blocked_by`;
|
||||||
|
修复后按 targeted rerun,再执行 full integration → journey。成绩仍写现有
|
||||||
|
`runs/<run-id>.yaml` 并用 `generate_run_csv.py` 生成同名 CSV;YAML 可带 `defects`/
|
||||||
|
`test_issues` 与 `feedback`,记录 testcase、product/test issue 类型、expected/observed、
|
||||||
|
artifact hash、environment、证据路径、复现和下一步,不添加新的 result 值。
|
||||||
|
|
||||||
|
PR 触发的失败回流必须先更新 PR/关联 issue,再通知 dispatcher 重新分配开发;需求/bug
|
||||||
|
触发的失败直接返回开发流程。tester 只生成结构化回流计划,不在本地偷偷修改远端。
|
||||||
|
|||||||
@@ -35,11 +35,19 @@ catalog(整个测试集)
|
|||||||
| `session` | 可复用会话(浏览器 / 模拟器) | pytest fixture;Robot suite variable | 不是一条用例 |
|
| `session` | 可复用会话(浏览器 / 模拟器) | pytest fixture;Robot suite variable | 不是一条用例 |
|
||||||
| `gate` | 项目细分闸门名 → 环境变量(`motion` / `hardware` / `weld`) | 授权开关 | 不是 `risk`,不要拿它代替风险等级 |
|
| `gate` | 项目细分闸门名 → 环境变量(`motion` / `hardware` / `weld`) | 授权开关 | 不是 `risk`,不要拿它代替风险等级 |
|
||||||
| `authorize` | 风险档 → 环境变量。该档执行前必须为 `true` | pytest `skipif` 条件 | 不是认证(auth),是**授权** |
|
| `authorize` | 风险档 → 环境变量。该档执行前必须为 `true` | pytest `skipif` 条件 | 不是认证(auth),是**授权** |
|
||||||
|
| `execution_profile` | 执行环境策略:`standard` 或隔离可重置的 `isolated-full` | pytest marker/fixture policy | 不是风险等级;isolated-full 不会抹掉 `risk` |
|
||||||
| `assume` | 额外跳过条件(gate / authorize 之外的环境表达式) | JUnit `assumeTrue`;pytest `skipif` | 不是授权开关的重复写法 |
|
| `assume` | 额外跳过条件(gate / authorize 之外的环境表达式) | JUnit `assumeTrue`;pytest `skipif` | 不是授权开关的重复写法 |
|
||||||
| `status_probe` | 分类级前置条件:请求一个 URL,命中即视为前置不满足 | pytest session fixture;Robot Suite Setup | 不是产品失败,不是整轮硬闸 |
|
| `status_probe` | 分类级前置条件:请求一个 URL,命中即视为前置不满足 | pytest session fixture;Robot Suite Setup | 不是产品失败,不是整轮硬闸 |
|
||||||
| `how` | 怎么测:`steps` / `run` / `url` / `timeout` | test steps | 不是判定标准 |
|
| `how` | 怎么测:`steps` / `run` / `url` / `timeout` | test steps | 不是判定标准 |
|
||||||
| `expected` | 怎么算过:`{kind, value}` 列表 | pytest `assert`;Robot `Should …` | 不是步骤 |
|
| `expected` | 怎么算过:`{kind, value}` 列表 | pytest `assert`;Robot `Should …` | 不是步骤 |
|
||||||
| `run_together` | 必须成组执行的用例(有序、可选 continue_on_fail) | 场景组(scenario) | 不是 suite,不是 tag |
|
| `run_together` | 必须成组执行的用例(有序、可选 continue_on_fail) | 场景组(scenario) | 不是 suite,不是 tag |
|
||||||
|
| `client_suite` | 客户端用户旅程套件:`client-smoke` / `client-regression` / `safety` | mobile/browser E2E suite | 不是 `category`,也不是通用 tag;迁移期可读 `suite` 别名 |
|
||||||
|
| `test_type` | 顶层测试类型:`journey` 或 `integration` | E2E vs service/contract test | 不是 driver;旧目录可由既有字段推断 |
|
||||||
|
| `integration_suite` | 集成套件:`api-contract` / `authorization` / `build-contract` | API/contract test suite | 不是客户端 `client_suite`;迁移期可读 `suite` 别名 |
|
||||||
|
| `journey` | 用户目标、入口、操作、UI 断言、最终状态和失败继续策略 | Cucumber scenario / Robot task | 不是 API contract |
|
||||||
|
| `fixture` | 旅程开始前声明的测试账号与隔离数据 | pytest fixture | 不是运行中临时猜出的数据 |
|
||||||
|
| `cleanup` | 本轮数据的 teardown 契约;缺失会在 preflight 阻断 | pytest teardown / Robot teardown | 不是测试通过后的可选备注 |
|
||||||
|
| `platform` | 旅程适用的 `android`/`ios` 平台列表 | device matrix | 不是 tester Agent binding;binding 仍在命令参数/运行环境 |
|
||||||
|
|
||||||
### 结果词
|
### 结果词
|
||||||
|
|
||||||
@@ -48,7 +56,7 @@ catalog(整个测试集)
|
|||||||
| `pass` | 通过 | passed |
|
| `pass` | 通过 | passed |
|
||||||
| `fail` | 产品缺陷 | failed |
|
| `fail` | 产品缺陷 | failed |
|
||||||
| `environment-fail` | 前置/环境导致无法执行,**永远不是产品失败** | pytest error;JUnit aborted |
|
| `environment-fail` | 前置/环境导致无法执行,**永远不是产品失败** | pytest error;JUnit aborted |
|
||||||
| `skipped` | **主动**不跑:`retired` / `pending` / 超风险上限 / 未授权 / `assume` 不成立 | pytest skipped;JUnit disabled |
|
| `skipped` | **主动**不跑:standard 下的 `retired` / `pending` / 超风险上限 / 未授权 / `assume` 不成立 | pytest skipped;JUnit disabled |
|
||||||
| `not-run` | **被动**没轮到:失败即停后剩下的;分类被 `status_probe` 挡住整组没执行;`automation` 需人工而无人值守 | pytest not run;deselected |
|
| `not-run` | **被动**没轮到:失败即停后剩下的;分类被 `status_probe` 挡住整组没执行;`automation` 需人工而无人值守 | pytest not run;deselected |
|
||||||
|
|
||||||
## 2. 命名迁移对照(v1 → v2)
|
## 2. 命名迁移对照(v1 → v2)
|
||||||
@@ -75,6 +83,8 @@ catalog(整个测试集)
|
|||||||
- 三级**单调**:`readonly ⊂ mutating ⊂ destructive`,所以只需要一个上限 `default_max_risk` 和一个 `--max-risk`,不用组合矩阵。
|
- 三级**单调**:`readonly ⊂ mutating ⊂ destructive`,所以只需要一个上限 `default_max_risk` 和一个 `--max-risk`,不用组合矩阵。
|
||||||
- 与 `gate` 正交:`risk` 说「影响多大」,`gate` 说「碰哪个子系统」。点动 vs 焊接这种细分授权用 `gate`,不要再造 risk 名。
|
- 与 `gate` 正交:`risk` 说「影响多大」,`gate` 说「碰哪个子系统」。点动 vs 焊接这种细分授权用 `gate`,不要再造 risk 名。
|
||||||
- `tag=smoke` 只能配 `risk=readonly`:冒烟测试不许留下变化。
|
- `tag=smoke` 只能配 `risk=readonly`:冒烟测试不许留下变化。
|
||||||
|
- 隔离可重置环境可设置 `execution_profile: isolated-full` 跑所有 active 用例;安全前置由
|
||||||
|
test-preflight 的设备、服务与 cleanup/rollback 契约保证,原始 risk 仍必须报告;客户端能力由 journey 用例验证。
|
||||||
|
|
||||||
### 3.2 `automation`:为什么是 `automated / assisted / manual`
|
### 3.2 `automation`:为什么是 `automated / assisted / manual`
|
||||||
|
|
||||||
@@ -86,6 +96,7 @@ catalog(整个测试集)
|
|||||||
- `assisted`:Agent 驱动,但至少一步需要人在场;该步在 `how.steps` 里以 `[人工]` 开头(可被 `validate` 机器判定)。
|
- `assisted`:Agent 驱动,但至少一步需要人在场;该步在 `how.steps` 里以 `[人工]` 开头(可被 `validate` 机器判定)。
|
||||||
- `manual`:只能人工执行;禁止 `driver`、禁止 `how.run`,必须有 `how.steps`。
|
- `manual`:只能人工执行;禁止 `driver`、禁止 `how.run`,必须有 `how.steps`。
|
||||||
- 默认只跑 `automated`。无人值守时 `assisted` / `manual` 记 `not-run`;有人在场时 `assisted` 暂停等人确认,`manual` 由人工回报结果。
|
- 默认只跑 `automated`。无人值守时 `assisted` / `manual` 记 `not-run`;有人在场时 `assisted` 暂停等人确认,`manual` 由人工回报结果。
|
||||||
|
- isolated-full 只取消风险/授权过滤,不伪造 assisted/manual 的人工步骤。
|
||||||
|
|
||||||
### 3.3 结果为什么拆出 `not-run`
|
### 3.3 结果为什么拆出 `not-run`
|
||||||
|
|
||||||
@@ -99,6 +110,19 @@ catalog(整个测试集)
|
|||||||
- 因此它是**分类级前置条件**:`catalog.status_probe` 是全局默认,`categories[].status_probe` 可覆盖(缺省继承 / `false` 关闭 / mapping 整体替换)。对应 pytest 的 session fixture、Robot 的 Suite Setup。
|
- 因此它是**分类级前置条件**:`catalog.status_probe` 是全局默认,`categories[].status_probe` 可覆盖(缺省继承 / `false` 关闭 / mapping 整体替换)。对应 pytest 的 session fixture、Robot 的 Suite Setup。
|
||||||
- 命中或不可达只影响用它的分类;只有所有选中分类都被挡住,整轮结论才是 `environment-fail`。`select` 会按分类返回解析后的生效值,Agent 不要自己读 catalog 猜。
|
- 命中或不可达只影响用它的分类;只有所有选中分类都被挡住,整轮结论才是 `environment-fail`。`select` 会按分类返回解析后的生效值,Agent 不要自己读 catalog 猜。
|
||||||
|
|
||||||
|
### 3.5 客户端套件和生命周期
|
||||||
|
|
||||||
|
客户端套件固定为:
|
||||||
|
|
||||||
|
- `client-smoke`:启动、正确账号登录和少量只读主路径,通常配 `tag=smoke`、`risk=readonly`。
|
||||||
|
- `client-regression`:完整的普通用户旅程,可包含隔离环境中的 `mutating` 操作;不等于所有 full 用例。
|
||||||
|
- `safety`:删除、权限边界、改密、停用或故障注入等危险旅程;必须显式选择和授权,不进入默认运行。
|
||||||
|
|
||||||
|
每条旅程按 `preflight → 启动 → 登录 → 旅程 → 断言 → 清理 → 报告` 执行。登录失败在
|
||||||
|
客户端和正确测试账号都已就绪时记 `fail`;设备、产物、服务或 session 不可达才记
|
||||||
|
`environment-fail`。fixture/cleanup 缺失是 contract block,不伪装成逐条结果。前置失败
|
||||||
|
后的剩余条目是 `not-run`,主动排除仍是 `skipped`。
|
||||||
|
|
||||||
## 4. 不采用的词
|
## 4. 不采用的词
|
||||||
|
|
||||||
| 词 | 不用它的原因 |
|
| 词 | 不用它的原因 |
|
||||||
|
|||||||
@@ -25,6 +25,16 @@ bindings:
|
|||||||
上执行测试。`tester.test` 只能用于明确的单平台或非移动测试;同一轮包含 Android
|
上执行测试。`tester.test` 只能用于明确的单平台或非移动测试;同一轮包含 Android
|
||||||
和 iOS 时必须分别解析两个 tester 绑定。
|
和 iOS 时必须分别解析两个 tester 绑定。
|
||||||
|
|
||||||
|
移动 bundle 入口必须把平台绑定显式传入:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
--binding android=tester.android --binding ios=tester.ios
|
||||||
|
```
|
||||||
|
|
||||||
|
入口会校验 preflight matrix 的每个平台 binding、node 和 runner。远程 iOS 必须是
|
||||||
|
`contract` runner;多平台运行不会读取旧的单一 `local.yaml.agent` 来决定执行节点,
|
||||||
|
也不会在协调节点直接执行 Android/iOS 平台命令。
|
||||||
|
|
||||||
`locations` 和 `profiles` 的定义见共享 `agent-runtime`。tester adapter 只把它 materialize
|
`locations` 和 `profiles` 的定义见共享 `agent-runtime`。tester adapter 只把它 materialize
|
||||||
到本轮兼容配置;每轮 terminal 仍然新建并在结束时关闭,不能因为 binding 是 persistent
|
到本轮兼容配置;每轮 terminal 仍然新建并在结束时关闭,不能因为 binding 是 persistent
|
||||||
就复用测试 terminal。
|
就复用测试 terminal。
|
||||||
|
|||||||
@@ -36,7 +36,11 @@
|
|||||||
| `destructive` | 难恢复或伤人/伤钱/伤环境 | 删库、真实扣费 | 清数据、内购 | 急停、断电、真焊 |
|
| `destructive` | 难恢复或伤人/伤钱/伤环境 | 删库、真实扣费 | 清数据、内购 | 急停、断电、真焊 |
|
||||||
|
|
||||||
默认只跑 `readonly`。项目细分闸(点动 vs 焊接)用可选 `gate`,不要再发明 risk 名。
|
默认只跑 `readonly`。项目细分闸(点动 vs 焊接)用可选 `gate`,不要再发明 risk 名。
|
||||||
|
隔离且可重置的项目可在 catalog 设置 `execution_profile: isolated-full`,此时所有 active
|
||||||
|
用例按原始 risk 执行;`risk` 仍用于报告和 cleanup 设计,不能改写为 readonly。
|
||||||
|
|
||||||
`tag=smoke` 且 `risk!=readonly` 为非法。
|
`tag=smoke` 且 `risk!=readonly` 为非法。
|
||||||
|
|
||||||
不想跑某一档时,`authorize.<risk>` 指向的环境变量保持 false 即可——该档用例记 `skipped`,不是 `not-run`。
|
standard 下不想跑某一档时,`authorize.<risk>` 指向的环境变量保持 false 即可——该档用例记
|
||||||
|
`skipped`,不是 `not-run`。isolated-full 不使用这些授权开关,改由 preflight 能力探针和
|
||||||
|
cleanup/rollback 契约阻断不安全环境。
|
||||||
|
|||||||
@@ -9,6 +9,11 @@ Agent、执行目标和临时 terminal 的完整协议见 [execution.md](executi
|
|||||||
|
|
||||||
## 0. 建立测试矩阵并完成全量 preflight
|
## 0. 建立测试矩阵并完成全量 preflight
|
||||||
|
|
||||||
|
`test-preflight` 是 integration 和 journey 的共同硬闸。tester 必须先调用其 `verify`
|
||||||
|
或 `prepare`,保存独立的 preflight YAML/CSV;状态为 `blocked` 或 `environment-fail`
|
||||||
|
时不创建 browser/mobile session,所有尚未执行的 testcase 记 `not-run`。preflight 证据
|
||||||
|
不算 testcase,也不进入 testcase CSV/YAML 的总数。
|
||||||
|
|
||||||
测试不是“找到一个能跑的设备就开始”。每轮先建立一份测试矩阵;移动端每个目标产物
|
测试不是“找到一个能跑的设备就开始”。每轮先建立一份测试矩阵;移动端每个目标产物
|
||||||
必须单独占一行:
|
必须单独占一行:
|
||||||
|
|
||||||
@@ -67,7 +72,11 @@ iOS | … | … | page.writes | tester.ios | remote-ios
|
|||||||
|
|
||||||
## 1. 选范围
|
## 1. 选范围
|
||||||
|
|
||||||
未指定范围 → `--tag smoke`,风险上限 = `default_max_risk`,`automation` 默认只跑 `automated`。
|
未指定范围 → `--tag smoke`,执行 profile = catalog 的 `execution_profile`(默认 `standard`),
|
||||||
|
`standard` 的风险上限 = `default_max_risk`,`automation` 默认只跑 `automated`。
|
||||||
|
`execution_profile=isolated-full` 只用于隔离可重置环境:不按风险上限或 `AUTHORIZE_*`/gate
|
||||||
|
授权变量过滤 active 用例,保留每条用例原始 `risk`;安全门禁由 test-preflight 能力探针和
|
||||||
|
cleanup/rollback 契约承担。
|
||||||
|
|
||||||
用户说分类、id、tag、风险上限时,原样传给 select:
|
用户说分类、id、tag、风险上限时,原样传给 select:
|
||||||
|
|
||||||
@@ -75,33 +84,65 @@ iOS | … | … | page.writes | tester.ios | remote-ios
|
|||||||
python3 <skill-dir>/scripts/catalog.py select --root <root> \
|
python3 <skill-dir>/scripts/catalog.py select --root <root> \
|
||||||
[--category CAT] [--id ID]... [--tag smoke|full|slow] \
|
[--category CAT] [--id ID]... [--tag smoke|full|slow] \
|
||||||
[--max-risk <risk>] [--driver script|browser|mobile] \
|
[--max-risk <risk>] [--driver script|browser|mobile] \
|
||||||
|
[--execution-profile standard|isolated-full] \
|
||||||
[--automation automated|assisted|manual|any] \
|
[--automation automated|assisted|manual|any] \
|
||||||
|
[--test-type integration|journey] [--suite <client-or-integration-suite>] \
|
||||||
[--platform android|ios]... \
|
[--platform android|ios]... \
|
||||||
[--artifact android=<path-or-url>] [--artifact ios=<path-or-url>] \
|
[--artifact android=<path-or-url>] [--artifact ios=<path-or-url>] \
|
||||||
[--binding android=tester.android] [--binding ios=tester.ios]
|
[--binding android=tester.android] [--binding ios=tester.ios]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
客户端旅程还可按标准套件选择:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite client-smoke
|
||||||
|
```
|
||||||
|
|
||||||
|
`client-smoke`、`client-regression`、`safety` 是 `client_suite` 的固定值,不替换既有
|
||||||
|
`tag=smoke|full|slow`。`safety` 不参与未指定范围的默认选择。
|
||||||
|
|
||||||
|
集成测试显式选择:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 <skill-dir>/scripts/catalog.py select --root <root> --test-type integration
|
||||||
|
python3 <skill-dir>/scripts/catalog.py select --root <root> --suite api-contract
|
||||||
|
```
|
||||||
|
|
||||||
|
未指定 `--test-type`/`--suite` 时,默认选择不会包含 `test_type=integration`。
|
||||||
|
|
||||||
选中任一 `driver=mobile` 用例时,`--platform`、每个平台的 `--artifact` 和对应
|
选中任一 `driver=mobile` 用例时,`--platform`、每个平台的 `--artifact` 和对应
|
||||||
`tester` binding 是硬性参数;`catalog.py select` 会检查产物文件/下载地址和
|
`tester` binding 是硬性参数;`catalog.py select` 会检查产物文件/下载地址和
|
||||||
`.pouch/agent.yaml` 中的 binding。缺任何一项直接返回非零并打印 `preflight: blocked`,
|
`.pouch/agent.yaml` 中的 binding。缺任何一项直接返回非零并打印 `preflight: blocked`,
|
||||||
不能拿到 cases 后绕过闸门开始测试。默认 binding 是 `tester.<platform>`,所以 iOS
|
不能拿到 cases 后绕过闸门开始测试。默认 binding 是 `tester.<platform>`,所以 iOS
|
||||||
会强制检查 `tester.ios`,不会回退到 `builder.ios` 或通用 `tester.test`。
|
会强制检查 `tester.ios`,不会回退到 `builder.ios` 或通用 `tester.test`。
|
||||||
|
|
||||||
`--automation` 默认 `automated`(需人工的条目会被跳过,并在 JSON `skipped` 里说明);用户明确要求现场配合时用 `assisted`,`any` 表示不按人工参与度筛。
|
`--automation` 默认 `automated`(需人工的条目会被记为 `not-run`,不会伪造人工步骤);用户明确要求现场配合时用 `assisted`,`any` 表示不按人工参与度筛。
|
||||||
|
|
||||||
单根 `select` 的 JSON 字段:`max_risk` / `automation` / `count` / `vars`(已展开的变量,用来替换 `how` 里的 `${…}`)/ `artifacts` / `preflight` / `skipped` / `run_together` / `status_probe`(按分类,`url` 已展开)/ `cases`。从父仓库 fan-out 时返回 `mode: fan-out` 和每个根的独立结果,不能把多个根的 cases 拼成一个列表。
|
单根 `select` 的 JSON 字段:`max_risk` / `automation` / `count` / `vars`(已展开的变量,用来替换 `how` 里的 `${…}`)/ `artifacts` / `preflight` / `skipped` / `run_together` / `status_probe`(按分类,`url` 已展开)/ `cases`。从父仓库 fan-out 时返回 `mode: fan-out` 和每个根的独立结果,不能把多个根的 cases 拼成一个列表。
|
||||||
|
|
||||||
`select` 已排除 `retired` 和风险高于上限的条目(后者若用户点了具体 id,会在 JSON 的 `skipped` 里说明,不执行)。
|
`standard` 的 `select` 已排除 `retired` 和风险高于上限的条目(后者若用户点了具体 id,会在 JSON 的 `skipped` 里说明);`isolated-full` 保留所有 active 条目,不因风险或授权变量 skipped。
|
||||||
|
|
||||||
若 catalog 有 `run_together`,命中组内任一条就把整组按 `ids` 顺序扩进列表(JSON 带 `run_together`)。这不是擅自扩大范围。
|
若 catalog 有 `run_together`,命中组内任一条就把整组按 `ids` 顺序扩进列表(JSON 带 `run_together`)。这不是擅自扩大范围。
|
||||||
|
|
||||||
无命中:停下说明筛选条件,不要扩大范围。
|
无命中:停下说明筛选条件,不要扩大范围。
|
||||||
|
|
||||||
## 2. 运行前闸
|
## 2. 双层执行
|
||||||
|
|
||||||
- `HOST` 未设(环境变量 / `HX_TEST_HOST` / `local.yaml` 都没有):全部停,environment-fail。
|
preflight 通过后,先按 catalog 顺序执行 `integration`,再按客户端套件顺序执行
|
||||||
- 命中条的 `risk` 在 catalog `authorize` 里有对应变量:变量不是 `true` 则该条 `skipped`。有 `gate` 时还要 `catalog.gates` 对应变量为 `true`;`assume` 不成立同样 `skipped`。即使用户点了 id 也不执行。
|
`journey`。集成服务不可达是 `environment-fail`;服务已达但 API/权限/数据契约断言错误
|
||||||
- 授权/假设不成立是**主动不跑**(`skipped`),不是 `not-run`:这是我们决定不跑,不是没轮到。
|
是 `fail`。任何 integration `fail` 都阻断依赖的 journey,并把它们记为 `not-run`;此时
|
||||||
|
不得创建客户端 session。正确测试账号登录失败发生在客户端已启动后,仍是产品 `fail`。
|
||||||
|
|
||||||
|
运行器可复用 `scripts/two_tier.py` 的 `run_integration_first`,它只接收外部 driver
|
||||||
|
回调,不实现 API、浏览器或移动执行器;mock fixture 只能验证调用顺序和失败传播,不能
|
||||||
|
冒充真实部署、Mac 或设备成功。
|
||||||
|
|
||||||
|
## 3. 运行前闸
|
||||||
|
|
||||||
|
- `HOST` 未设(环境变量 / `local.yaml` 都没有):全部停,environment-fail。
|
||||||
|
- `standard`:命中条的 `risk` 在 catalog `authorize` 里有对应变量时,变量不是 `true` 则该条 `skipped`;有 `gate` 时还要对应变量为 `true`,`assume` 不成立同样 `skipped`。即使用户点了 id 也不执行。
|
||||||
|
- `isolated-full`:不读取 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM`、`AUTHORIZE_PASSWORD` 等人工开关;相册、照片权限/测试图片和中文拼音能力默认具备,由 journey 用例真实验证,失败记为产品失败。
|
||||||
|
- standard 中授权/假设不成立是**主动不跑**(`skipped`);isolated-full 不因客户端能力授权变量停止,设备、产物、服务或清理前置失败时才为全部 testcase 写 `not-run` + `blocked_by: [test-preflight]`。
|
||||||
- `status_probe` 是**分类级前置条件**(业界对应 pytest 的 session fixture / Robot 的 Suite Setup)。用 `select` 返回的 `status_probe` 解析生效值,不要自己读 catalog 猜。返回的 `probe.url` 已用 `vars` 展开,直接请求即可;`select` 同批返回 `vars`:
|
- `status_probe` 是**分类级前置条件**(业界对应 pytest 的 session fixture / Robot 的 Suite Setup)。用 `select` 返回的 `status_probe` 解析生效值,不要自己读 catalog 猜。返回的 `probe.url` 已用 `vars` 展开,直接请求即可;`select` 同批返回 `vars`:
|
||||||
|
|
||||||
| `categories[].status_probe` | 生效探测 |
|
| `categories[].status_probe` | 生效探测 |
|
||||||
@@ -114,7 +155,7 @@ python3 <skill-dir>/scripts/catalog.py select --root <root> \
|
|||||||
- 探测不可达或命中 `stop_if`:在 preflight 阶段记录对应分类/平台的不可用项并停止整轮;
|
- 探测不可达或命中 `stop_if`:在 preflight 阶段记录对应分类/平台的不可用项并停止整轮;
|
||||||
先返回用户处理,不创建后续测试 session。只有所有本轮目标环境通过,才进入用例执行。
|
先返回用户处理,不创建后续测试 session。只有所有本轮目标环境通过,才进入用例执行。
|
||||||
|
|
||||||
## 2.1 移动端平台环境闸门与用户引导
|
## 3.1 移动端平台环境闸门与用户引导
|
||||||
|
|
||||||
`driver=mobile` 不是一个可以互换的平台。运行前必须根据 catalog session、用例变量或用户指定,明确本轮是 Android 还是 iOS;无法判断时先停下询问平台,不得猜测。两类平台分别执行以下闸门:
|
`driver=mobile` 不是一个可以互换的平台。运行前必须根据 catalog session、用例变量或用户指定,明确本轮是 Android 还是 iOS;无法判断时先停下询问平台,不得猜测。两类平台分别执行以下闸门:
|
||||||
|
|
||||||
@@ -175,7 +216,7 @@ Android+iOS 默认是全量 preflight 闸门:任一项不满足,另一平台
|
|||||||
|
|
||||||
环境引导只负责让用户补齐前置条件;除 catalog 已明确授权的 session 初始化外,不自动安装 SDK、配对真机、修改系统安全设置或上传产物。
|
环境引导只负责让用户补齐前置条件;除 catalog 已明确授权的 session 初始化外,不自动安装 SDK、配对真机、修改系统安全设置或上传产物。
|
||||||
|
|
||||||
## 3. Session
|
## 4. Session
|
||||||
|
|
||||||
看命中条用到的 `session` 键:
|
看命中条用到的 `session` 键:
|
||||||
|
|
||||||
@@ -184,9 +225,9 @@ Android+iOS 默认是全量 preflight 闸门:任一项不满足,另一平台
|
|||||||
- `reuse: run`:整轮结束再关。`persist`:不关。
|
- `reuse: run`:整轮结束再关。`persist`:不关。
|
||||||
- 中途 session 挂了:后续同 driver 标 environment-fail,不要标产品失败。
|
- 中途 session 挂了:后续同 driver 标 environment-fail,不要标产品失败。
|
||||||
|
|
||||||
## 4. 执行
|
## 5. 执行
|
||||||
|
|
||||||
按 select 返回的顺序,一条一条来。select 按 `catalog.yaml` 的 `categories` 顺序,同一分类内按该 yaml 列表顺序(不是文件名字母序)。默认失败即停(产品 `fail` 或探测到的不安全状态)。用户说继续 / keep-going 才往后跑。
|
按 select 返回的顺序,一条一条来。select 按 `catalog.yaml` 的 `categories` 顺序,同一分类内按该 yaml 列表顺序(不是文件名字母序)。同一 phase 的独立用例继续执行;只有显式 `depends_on` 的失败依赖才写 `not-run` + `blocked_by`。用户说 targeted rerun 时只重跑受影响用例,修复后再 full integration → journey。
|
||||||
|
|
||||||
例外:当前条和下一条同属 `run_together` 且该组 `continue_on_fail: true` 时,前一条产品 `fail` 仍跑后一条。`environment-fail` 仍整轮停。
|
例外:当前条和下一条同属 `run_together` 且该组 `continue_on_fail: true` 时,前一条产品 `fail` 仍跑后一条。`environment-fail` 仍整轮停。
|
||||||
|
|
||||||
@@ -216,9 +257,9 @@ python3 <skill-dir>/scripts/catalog.py run-script --root <root> --id <id>
|
|||||||
|
|
||||||
挂到已有 session,执行 `how.steps`,对照 `expected`。必须留下可见文本、URL 或交互证据。
|
挂到已有 session,执行 `how.steps`,对照 `expected`。必须留下可见文本、URL 或交互证据。
|
||||||
|
|
||||||
## 5. 记录与报告
|
## 6. 记录与报告
|
||||||
|
|
||||||
每条:`id, result, evidence, notes`。`result` 只能是 `pass` / `fail` / `environment-fail` / `skipped` / `not-run`。`skipped` 是主动不跑(retired / pending / 超风险上限 / 未授权 / `assume` 不成立),`not-run` 是没轮到(失败即停后剩下的 / 被前置整组挡住的 / 需人工而无人值守的)。
|
每条:`id, result, evidence, notes`,依赖阻断时另有 `blocked_by`。`result` 只能是 `pass` / `fail` / `environment-fail` / `skipped` / `not-run`。`skipped` 是 standard 中主动不跑(retired / pending / 超风险上限 / 未授权 / `assume` 不成立);`not-run` 是没轮到或 preflight/依赖阻断。失败用例可在 YAML 顶层 `defects`(兼容 `test_issues`)记录 testcase、`issue_type=product|test`、expected/observed、artifact hash、environment、evidence path、reproduction、next step。
|
||||||
|
|
||||||
写入 `.pouch/tester/runs/<UTC时间>-<范围>-<随机后缀>.yaml`,可用
|
写入 `.pouch/tester/runs/<UTC时间>-<范围>-<随机后缀>.yaml`,可用
|
||||||
`generate_run_csv.new_run_id(<范围>)` 生成同名 `runId`。记录包含唯一 `runId`、筛选条件、HOST 是否已设(可以写 HOST 值)、session 是否复用、每个分类的 `status_probe` 生效值与观测结果(`source` / `disabled` 原因 / `hit` / `observed`)、逐条结果。记录文件只创建一次,不覆盖已有文件。立刻用同一快照生成 CSV,不要手改:
|
`generate_run_csv.new_run_id(<范围>)` 生成同名 `runId`。记录包含唯一 `runId`、筛选条件、HOST 是否已设(可以写 HOST 值)、session 是否复用、每个分类的 `status_probe` 生效值与观测结果(`source` / `disabled` 原因 / `hit` / `observed`)、逐条结果。记录文件只创建一次,不覆盖已有文件。立刻用同一快照生成 CSV,不要手改:
|
||||||
@@ -227,7 +268,23 @@ python3 <skill-dir>/scripts/catalog.py run-script --root <root> --id <id>
|
|||||||
python3 <skill-dir>/scripts/generate_run_csv.py --root <root> --run <yaml>
|
python3 <skill-dir>/scripts/generate_run_csv.py --root <root> --run <yaml>
|
||||||
```
|
```
|
||||||
|
|
||||||
这会写出与快照同名的 `.pouch/tester/runs/<run-id>.csv`(`executed=yes|no`),不会修改 `cases.csv`。对话里两个路径都报。
|
这会写出与快照同名的 `.pouch/tester/runs/<run-id>.csv`(`executed=yes|no`),不会修改
|
||||||
|
`cases.csv`。对话里两个路径都报。
|
||||||
|
|
||||||
|
需要复核已有成绩单时,显式同时传入同一次运行的 CSV 和 YAML:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 <skill-dir>/scripts/summarize_run.py \
|
||||||
|
--csv .pouch/tester/runs/<run-id>.csv \
|
||||||
|
--yaml .pouch/tester/runs/<run-id>.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
汇总状态只读取 CSV `result` 列,必须输出五类 `pass / fail / environment-fail /
|
||||||
|
skipped / not-run`,且计数和等于 CSV 数据行数。汇总前会检查 CSV 数据行数、CSV 唯一
|
||||||
|
testcase id 数、YAML `results` 数量和 CSV/YAML testcase id 集合;重复或缺失 id、缺失或
|
||||||
|
未知 CSV 状态、任一数量或集合不一致都会非零退出并给出诊断。YAML 的 `counts` 或其它
|
||||||
|
历史汇总字段不会参与状态计数。退出码 0 表示汇总校验成功,非零表示报告不一致或输入
|
||||||
|
无效,不等同于产品测试失败。
|
||||||
|
|
||||||
对话报告:
|
对话报告:
|
||||||
|
|
||||||
|
|||||||
@@ -14,6 +14,10 @@ runs/<run-id>.yaml 每次运行快照(执行源);run-id 必须唯一
|
|||||||
runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对应 yaml 生成
|
runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对应 yaml 生成
|
||||||
```
|
```
|
||||||
|
|
||||||
|
两类用例共用同一套 cases/runs 和五态报告,不另造成绩单。新目录显式声明顶层
|
||||||
|
`test_type: journey | integration`;旧目录没有该字段时按已有 `client_suite`/`journey`
|
||||||
|
或 `integration_suite` 推断。`category` 仍是文件/执行顺序容器,不被套件字段替代。
|
||||||
|
|
||||||
## catalog.yaml
|
## catalog.yaml
|
||||||
|
|
||||||
| 字段 | 含义 |
|
| 字段 | 含义 |
|
||||||
@@ -21,6 +25,7 @@ runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对
|
|||||||
| `categories` | `{id, title, status_probe?}` 列表,id 与 `cases/<id小写>.yaml` 对应;列表顺序即 `select` / 执行顺序。`status_probe` 见下 |
|
| `categories` | `{id, title, status_probe?}` 列表,id 与 `cases/<id小写>.yaml` 对应;列表顺序即 `select` / 执行顺序。`status_probe` 见下 |
|
||||||
| `risks` | 从低到高,只能是 `readonly` → `mutating` → `destructive` 的一段,且以 `readonly` 开头。含义见 [levels.md](levels.md) |
|
| `risks` | 从低到高,只能是 `readonly` → `mutating` → `destructive` 的一段,且以 `readonly` 开头。含义见 [levels.md](levels.md) |
|
||||||
| `default_max_risk` | 未指定时的运行上限,必须是 `risks` 里的值 |
|
| `default_max_risk` | 未指定时的运行上限,必须是 `risks` 里的值 |
|
||||||
|
| `execution_profile` | `standard`(默认)或 `isolated-full`。后者只用于隔离可重置环境,保留 risk 语义但不按 risk/authorize/gate 过滤 active 用例;设备、服务和 cleanup/rollback 契约仍由 preflight 检查,客户端能力由 journey 真实验证 |
|
||||||
| `vars` | 可替换变量,如 `API_BASE: http://${HOST}:8095` |
|
| `vars` | 可替换变量,如 `API_BASE: http://${HOST}:8095` |
|
||||||
| `artifacts` | 可选的移动产物身份契约。按 `android` / `ios` 声明 `required` 和 `package` / `bundle_id`;每轮文件路径或下载地址由调用方提供,不提交临时产物路径。 |
|
| `artifacts` | 可选的移动产物身份契约。按 `android` / `ios` 声明 `required` 和 `package` / `bundle_id`;每轮文件路径或下载地址由调用方提供,不提交临时产物路径。 |
|
||||||
| `authorize` | 风险名 → 环境变量名。该档执行前变量必须为 `true`(`readonly` 不需要) |
|
| `authorize` | 风险名 → 环境变量名。该档执行前变量必须为 `true`(`readonly` 不需要) |
|
||||||
@@ -29,10 +34,16 @@ runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对
|
|||||||
| `id_pattern` | 用例 id 正则 |
|
| `id_pattern` | 用例 id 正则 |
|
||||||
| `run_together` | 可选。成组执行:`[{ids: [TC-…, TC-…], continue_on_fail: true}]`。点到组内任一条,`select` 会按 `ids` 顺序把整组扩进命中列表。`continue_on_fail: true` 时组内前一条产品 fail 仍跑后一条(返回/收尾)。一个 id 只能属于一组。 |
|
| `run_together` | 可选。成组执行:`[{ids: [TC-…, TC-…], continue_on_fail: true}]`。点到组内任一条,`select` 会按 `ids` 顺序把整组扩进命中列表。`continue_on_fail: true` 时组内前一条产品 fail 仍跑后一条(返回/收尾)。一个 id 只能属于一组。 |
|
||||||
| `status_probe` | 可选。顶层默认值,一个**分类级前置条件**:`{url, stop_if: {字段: [命中值]}}`。运行前请求 `url`,响应命中任一 `stop_if` 条件即视为前置不满足。分类项可用同名键覆盖:缺省继承顶层、`false` 关闭、mapping 整体替换。命中或不可达只影响用它的分类,不是产品失败。业界对应 pytest 的 session fixture / Robot 的 Suite Setup。`select` 会按分类返回解析后的生效值,`url` 已用 `vars` 展开 |
|
| `status_probe` | 可选。顶层默认值,一个**分类级前置条件**:`{url, stop_if: {字段: [命中值]}}`。运行前请求 `url`,响应命中任一 `stop_if` 条件即视为前置不满足。分类项可用同名键覆盖:缺省继承顶层、`false` 关闭、mapping 整体替换。命中或不可达只影响用它的分类,不是产品失败。业界对应 pytest 的 session fixture / Robot 的 Suite Setup。`select` 会按分类返回解析后的生效值,`url` 已用 `vars` 展开 |
|
||||||
|
| `client_suites` | 可选的标准套件声明;出现时必须完整包含 `client-smoke`、`client-regression`、`safety`。缺省运行仍按 `tag` 兼容旧目录 |
|
||||||
|
| `integration_suites` | 可选的集成套件声明;标准值为 `api-contract`、`authorization`、`build-contract` |
|
||||||
|
|
||||||
|
`isolated-full` 不得把风险字段改成 `readonly`,也不得把 `AUTHORIZE_ALBUM`、
|
||||||
|
`AUTHORIZE_PASSWORD`、`IME_PINYIN` 等变量当成门禁。移动用例必须通过 test-preflight
|
||||||
|
的设备、产物、服务和网络检查;相册、权限和输入法能力默认具备,交由 journey 用例验证。
|
||||||
|
|
||||||
`local.yaml` 与环境变量覆盖 `HOST` 和授权开关。模板中的 `HOST` 默认是
|
`local.yaml` 与环境变量覆盖 `HOST` 和授权开关。模板中的 `HOST` 默认是
|
||||||
`127.0.0.1`;远程目标必须显式覆盖。查找顺序:环境变量 > `local.yaml` >
|
`127.0.0.1`;远程目标必须显式覆盖。查找顺序:环境变量 > `local.yaml` >
|
||||||
`catalog.yaml` 的 vars。也认 `HX_TEST_HOST` 作为 `HOST`。
|
`catalog.yaml` 的 vars。
|
||||||
|
|
||||||
## agent
|
## agent
|
||||||
|
|
||||||
@@ -70,6 +81,7 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
|
|||||||
| 字段 | 必填 | 含义 |
|
| 字段 | 必填 | 含义 |
|
||||||
|------|------|------|
|
|------|------|------|
|
||||||
| `id` | 是 | 匹配 `id_pattern` |
|
| `id` | 是 | 匹配 `id_pattern` |
|
||||||
|
| `test_type` | 新用例是 | `journey` 或 `integration`;缺省兼容旧目录并按字段推断 |
|
||||||
| `category` | 是 | 必须等于文件声明的 category |
|
| `category` | 是 | 必须等于文件声明的 category |
|
||||||
| `title` | 是 | 短名 |
|
| `title` | 是 | 短名 |
|
||||||
| `what` | 是 | 一句可观察的「测什么」 |
|
| `what` | 是 | 一句可观察的「测什么」 |
|
||||||
@@ -84,10 +96,67 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
|
|||||||
| `expected` | 是 | 怎么过,至少一条 `{kind, value}` |
|
| `expected` | 是 | 怎么过,至少一条 `{kind, value}` |
|
||||||
| `maps_to` | 否 | 对人文档的功能项 id |
|
| `maps_to` | 否 | 对人文档的功能项 id |
|
||||||
| `assume` | 否 | 额外跳过条件(环境变量表达式)。不要重复 `risk` / `gate` 已覆盖的授权变量,否则 `validate` 报错 |
|
| `assume` | 否 | 额外跳过条件(环境变量表达式)。不要重复 `risk` / `gate` 已覆盖的授权变量,否则 `validate` 报错 |
|
||||||
|
| `depends_on` | 否 | 依赖的 testcase id 列表;依赖 fail/environment-fail/not-run 时本条写 `not-run` 和 `blocked_by`,独立用例继续 |
|
||||||
| `setup` | 否 | 人话前置 |
|
| `setup` | 否 | 人话前置 |
|
||||||
| `teardown` | 否 | 跑完回退 |
|
| `teardown` | 否 | 跑完回退 |
|
||||||
| `notes` | 否 | 备注 |
|
| `notes` | 否 | 备注 |
|
||||||
|
|
||||||
|
### 集成测试用例契约
|
||||||
|
|
||||||
|
`test_type: integration` 的用例不创建 browser/mobile session,也不能声明 `platform` 或
|
||||||
|
客户端 `journey`。必须有 `integration_suite`(迁移期可用 `suite`)且值为
|
||||||
|
`api-contract`、`authorization` 或 `build-contract`;`fixture` 必须是非空 mapping;
|
||||||
|
`request`、`input`、`call`、`integration` 之一或旧式 `how` 用于描述调用;`expected` 是
|
||||||
|
协议/数据断言。`mutating`/`destructive` 集成用例必须提供 `cleanup.steps` 或
|
||||||
|
`cleanup.run`(只读探针可用 `required: false` + `reason`)。
|
||||||
|
|
||||||
|
选择 `--suite api-contract` 或 `--test-type integration` 才会进入集成阶段;无显式类型
|
||||||
|
时的默认选择不会包含 integration。
|
||||||
|
|
||||||
|
### 客户端旅程用例契约
|
||||||
|
|
||||||
|
声明 `client_suite` 后,以下字段都必须存在;`catalog.py validate` 和 `select` 使用同一
|
||||||
|
套检查。契约缺失是 `preflight: blocked`,不生成逐条产品失败。
|
||||||
|
|
||||||
|
| 字段 | 必填 | 约束 |
|
||||||
|
|------|------|------|
|
||||||
|
| `client_suite` | 是 | `client-smoke`、`client-regression` 或 `safety`;安全套件必须显式选择,不属于默认 smoke/full |
|
||||||
|
| `journey.goal` | 是 | 用户目标,不写接口目标 |
|
||||||
|
| `journey.entry_state` | 是 | 启动/登录前的可观察入口状态 |
|
||||||
|
| `journey.actions` | 是 | 非空用户操作列表;真实 App/browser 驱动从这里开始 |
|
||||||
|
| `journey.assertions` | 是 | 非空可观察 UI 断言列表;最终业务状态可另由 `expected` 的 api-field/api-status 佐证 |
|
||||||
|
| `journey.final_state` | 是 | 旅程完成后的业务状态 |
|
||||||
|
| `journey.failure_policy` | 是 | `stop` 或 `continue`;默认不隐式吞掉前置失败 |
|
||||||
|
| `fixture` | 是 | mapping,至少有 `account`;账号和数据由 app catalog 提供,skill 不写具体账号 |
|
||||||
|
| `session` | 是 | 复用的 browser/mobile session 名,必须在 catalog.sessions 声明 |
|
||||||
|
| `cleanup` | 是 | mapping,必须有非空 `steps`/`run`;只读旅程也要用 `required: false` + `reason` 显式说明无需清理 |
|
||||||
|
| `platform` | 是 | `android`/`ios` 的字符串或列表,不能重复;平台差异只能进入执行适配层 |
|
||||||
|
|
||||||
|
一个最小形状如下(值是占位符,不是 app 规则):
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
client_suite: client-smoke
|
||||||
|
platform: [android, ios]
|
||||||
|
journey:
|
||||||
|
goal: "用户完成一个可观察的客户端目标"
|
||||||
|
entry_state: logged_out
|
||||||
|
actions: ["启动客户端", "输入测试账号并登录", "完成目标操作"]
|
||||||
|
assertions: ["看到目标页面", "看到成功提示"]
|
||||||
|
final_state: "目标状态已保存"
|
||||||
|
failure_policy: stop
|
||||||
|
fixture:
|
||||||
|
account: test-role
|
||||||
|
data: []
|
||||||
|
session: mobile
|
||||||
|
cleanup:
|
||||||
|
steps: ["删除本轮 run-id 创建的数据"]
|
||||||
|
```
|
||||||
|
|
||||||
|
账号登录属于 session 初始化:正确测试账号登录失败是产品 `fail`;设备、产物或服务
|
||||||
|
不可达才是 `environment-fail`。fixture/cleanup 缺失必须在启动、登录和任何旅程操作前
|
||||||
|
阻断。旅程失败后,尚未轮到的用例记 `not-run`;风险、生命周期或明确范围排除记
|
||||||
|
`skipped`。清理失败单独写入运行快照的 cleanup 证据,不改造为一个新的 result 值。
|
||||||
|
|
||||||
## how
|
## how
|
||||||
|
|
||||||
按 `driver` 分派;`automation=manual` 时没有 `driver`,只有 `how.steps`(人工步骤)。
|
按 `driver` 分派;`automation=manual` 时没有 `driver`,只有 `how.steps`(人工步骤)。
|
||||||
@@ -107,3 +176,24 @@ terminal;运行结束用 `orca terminal close --terminal <handle>` 关闭本
|
|||||||
## 判定
|
## 判定
|
||||||
|
|
||||||
`pass` / `fail` / `environment-fail` / `skipped` / `not-run`
|
`pass` / `fail` / `environment-fail` / `skipped` / `not-run`
|
||||||
|
|
||||||
|
## 运行失败回流
|
||||||
|
|
||||||
|
run YAML 的 `results` 必须保留每条 testcase 行。失败可在顶层 `defects`(兼容别名
|
||||||
|
`test_issues`)追加:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
defects:
|
||||||
|
- testcase: TC-CLIENT-001
|
||||||
|
issue_type: product # product | test
|
||||||
|
expected: "候选词可选择"
|
||||||
|
observed: "composition 成功但候选列表为空"
|
||||||
|
artifact_hash: sha256:...
|
||||||
|
environment: "Android emulator API 35 / tester.android"
|
||||||
|
evidence_path: .pouch/tester/evidence/TC-CLIENT-001.png
|
||||||
|
reproduction: "打开相册后输入 nihao"
|
||||||
|
next_step: "开发修复候选词渲染后 targeted rerun"
|
||||||
|
```
|
||||||
|
|
||||||
|
PR 触发的 run 可带 `feedback` 计划:先更新 PR/关联 issue,再通知 dispatcher 重新分配开发;
|
||||||
|
需求/bug 触发则 route 直接为 development。tester 只记录计划,不隐式调用远端写操作。
|
||||||
|
|||||||
@@ -7,11 +7,15 @@ CI release、builder 绑定或本机旧文件猜测产物。
|
|||||||
<skill-dir>/tester run \
|
<skill-dir>/tester run \
|
||||||
--artifact android=<path-or-url> \
|
--artifact android=<path-or-url> \
|
||||||
--artifact ios=<path-or-url> \
|
--artifact ios=<path-or-url> \
|
||||||
--platform android,ios
|
--platform android,ios \
|
||||||
|
--binding android=tester.android \
|
||||||
|
--binding ios=tester.ios
|
||||||
```
|
```
|
||||||
|
|
||||||
URL 就下载,本地文件就直接用。每个平台都在自己的 `tester.<platform>` 节点校验身份并安装,
|
URL 就下载,本地文件就直接用。每个平台都通过自己的 `tester.<platform>` binding
|
||||||
随后由现有 tester catalog 驱动真实移动 session。默认先完成 Android+iOS 双平台
|
校验、探测和安装;`agent-runtime` 负责把 Android 请求送到 Android 节点、把 iOS
|
||||||
|
请求送到 Mac 节点。旧的单一 `local.yaml.agent` 只能作为兼容配置,不能作为双平台
|
||||||
|
运行目标。随后由现有 tester catalog 驱动真实移动 session。默认先完成 Android+iOS 双平台
|
||||||
preflight;任一平台无法准备时整轮停止。只有显式加 `--partial` 才允许运行可用平台。
|
preflight;任一平台无法准备时整轮停止。只有显式加 `--partial` 才允许运行可用平台。
|
||||||
|
|
||||||
每轮另外创建隔离 Git worktree,不切用户当前分支。`--commit` 缺省为当前 HEAD。
|
每轮另外创建隔离 Git worktree,不切用户当前分支。`--commit` 缺省为当前 HEAD。
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ from datetime import datetime, timezone
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
from journey import (
|
||||||
|
CLIENT_SUITES,
|
||||||
|
INTEGRATION_SUITES,
|
||||||
|
TEST_TYPES,
|
||||||
|
contract_errors,
|
||||||
|
preflight as journey_preflight,
|
||||||
|
suite as case_suite,
|
||||||
|
test_type,
|
||||||
|
)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import yaml
|
import yaml
|
||||||
except ImportError: # pragma: no cover
|
except ImportError: # pragma: no cover
|
||||||
@@ -32,6 +42,7 @@ LIFECYCLE_SKIP_REASON = {
|
|||||||
}
|
}
|
||||||
TAGS = ("smoke", "full", "slow")
|
TAGS = ("smoke", "full", "slow")
|
||||||
CANONICAL_RISKS = ("readonly", "mutating", "destructive")
|
CANONICAL_RISKS = ("readonly", "mutating", "destructive")
|
||||||
|
EXECUTION_PROFILES = ("standard", "isolated-full")
|
||||||
AUTOMATIONS = ("automated", "assisted", "manual")
|
AUTOMATIONS = ("automated", "assisted", "manual")
|
||||||
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
||||||
EXPECTED_KINDS = (
|
EXPECTED_KINDS = (
|
||||||
@@ -308,17 +319,12 @@ def resolve_vars(root: Path, meta: dict[str, Any]) -> dict[str, str]:
|
|||||||
for key, value in os.environ.items():
|
for key, value in os.environ.items():
|
||||||
if key in merged or key in auth_keys or key.startswith("AUTHORIZE_") or key in {
|
if key in merged or key in auth_keys or key.startswith("AUTHORIZE_") or key in {
|
||||||
"HOST",
|
"HOST",
|
||||||
"HX_TEST_HOST",
|
|
||||||
"WEB_BASE",
|
"WEB_BASE",
|
||||||
"API_BASE",
|
"API_BASE",
|
||||||
"WS_URL",
|
"WS_URL",
|
||||||
"STATUS_BASE",
|
"STATUS_BASE",
|
||||||
}:
|
}:
|
||||||
merged[key] = value
|
merged[key] = value
|
||||||
if not merged.get("HOST") and merged.get("HX_TEST_HOST"):
|
|
||||||
merged["HOST"] = merged["HX_TEST_HOST"]
|
|
||||||
if os.environ.get("HX_TEST_HOST") and not os.environ.get("HOST"):
|
|
||||||
merged["HOST"] = os.environ["HX_TEST_HOST"]
|
|
||||||
for _ in range(10):
|
for _ in range(10):
|
||||||
changed = False
|
changed = False
|
||||||
for key, value in list(merged.items()):
|
for key, value in list(merged.items()):
|
||||||
@@ -346,6 +352,21 @@ def risk_index(meta: dict[str, Any], name: str) -> int:
|
|||||||
return risks.index(name)
|
return risks.index(name)
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_execution_profile(meta: dict[str, Any], requested: str | None = None) -> str:
|
||||||
|
"""Resolve the run profile without changing the meaning of ``risk``.
|
||||||
|
|
||||||
|
``isolated-full`` is an execution policy for a resettable test environment:
|
||||||
|
it disables risk/authorization filtering, but leaves the catalog risk value
|
||||||
|
on every case so reports still describe the side effect that was exercised.
|
||||||
|
"""
|
||||||
|
profile = requested or meta.get("execution_profile") or "standard"
|
||||||
|
if profile not in EXECUTION_PROFILES:
|
||||||
|
raise SystemExit(
|
||||||
|
f"未知 execution_profile: {profile!r}(可选: {', '.join(EXECUTION_PROFILES)})"
|
||||||
|
)
|
||||||
|
return str(profile)
|
||||||
|
|
||||||
|
|
||||||
def host_set(vars_map: dict[str, str]) -> bool:
|
def host_set(vars_map: dict[str, str]) -> bool:
|
||||||
host = (vars_map.get("HOST") or "").strip()
|
host = (vars_map.get("HOST") or "").strip()
|
||||||
return bool(host) and "${HOST}" not in host
|
return bool(host) and "${HOST}" not in host
|
||||||
@@ -373,7 +394,7 @@ def cmd_list(root: Path) -> int:
|
|||||||
pending = [c for c in cases if c.get("lifecycle") == "pending"]
|
pending = [c for c in cases if c.get("lifecycle") == "pending"]
|
||||||
print(f"## 测试目录 active {len(active)} / pending {len(pending)} / retired {len(retired)}")
|
print(f"## 测试目录 active {len(active)} / pending {len(pending)} / retired {len(retired)}")
|
||||||
print()
|
print()
|
||||||
headers = ["分类", "标题", "条数", "smoke"] + risks + list(DRIVERS)
|
headers = ["分类", "标题", "条数", "journey", "integration", "smoke"] + risks + list(DRIVERS)
|
||||||
print("| " + " | ".join(headers) + " |")
|
print("| " + " | ".join(headers) + " |")
|
||||||
print("| " + " | ".join("---" for _ in headers) + " |")
|
print("| " + " | ".join("---" for _ in headers) + " |")
|
||||||
|
|
||||||
@@ -391,7 +412,14 @@ def cmd_list(root: Path) -> int:
|
|||||||
title = (cats_meta.get(cat) or {}).get("title") or (
|
title = (cats_meta.get(cat) or {}).get("title") or (
|
||||||
group[0].get("_file_title") if group else ""
|
group[0].get("_file_title") if group else ""
|
||||||
)
|
)
|
||||||
row = [cat, str(title), str(len(group)), str(sum(1 for c in group if c.get("tag") == "smoke"))]
|
row = [
|
||||||
|
cat,
|
||||||
|
str(title),
|
||||||
|
str(len(group)),
|
||||||
|
str(sum(1 for c in group if test_type(c) == "journey")),
|
||||||
|
str(sum(1 for c in group if test_type(c) == "integration")),
|
||||||
|
str(sum(1 for c in group if c.get("tag") == "smoke")),
|
||||||
|
]
|
||||||
for risk in risks:
|
for risk in risks:
|
||||||
row.append(str(sum(1 for c in group if c.get("risk") == risk)))
|
row.append(str(sum(1 for c in group if c.get("risk") == risk)))
|
||||||
for driver in DRIVERS:
|
for driver in DRIVERS:
|
||||||
@@ -558,6 +586,7 @@ def expand_run_together(
|
|||||||
skipped: list[dict[str, str]],
|
skipped: list[dict[str, str]],
|
||||||
max_i: int,
|
max_i: int,
|
||||||
automation_filter: str = "automated",
|
automation_filter: str = "automated",
|
||||||
|
bypass_risk: bool = False,
|
||||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||||
groups = run_together_groups(meta)
|
groups = run_together_groups(meta)
|
||||||
if not groups:
|
if not groups:
|
||||||
@@ -595,7 +624,7 @@ def expand_run_together(
|
|||||||
except SystemExit:
|
except SystemExit:
|
||||||
skipped.append({"id": cid, "reason": f"未知风险 {risk}"})
|
skipped.append({"id": cid, "reason": f"未知风险 {risk}"})
|
||||||
continue
|
continue
|
||||||
if r_i > max_i:
|
if not bypass_risk and r_i > max_i:
|
||||||
skipped.append({"id": cid, "reason": f"risk={risk} 超过上限"})
|
skipped.append({"id": cid, "reason": f"risk={risk} 超过上限"})
|
||||||
continue
|
continue
|
||||||
members.append(public_case(case))
|
members.append(public_case(case))
|
||||||
@@ -758,9 +787,11 @@ def mobile_preflight(
|
|||||||
def cmd_select(args: argparse.Namespace, root: Path) -> int:
|
def cmd_select(args: argparse.Namespace, root: Path) -> int:
|
||||||
meta = load_catalog_meta(root)
|
meta = load_catalog_meta(root)
|
||||||
cases = load_all_cases(root)
|
cases = load_all_cases(root)
|
||||||
|
execution_profile = resolve_execution_profile(meta, args.execution_profile)
|
||||||
|
bypass_risk = execution_profile == "isolated-full"
|
||||||
max_risk = args.max_risk or meta.get("default_max_risk") or "readonly"
|
max_risk = args.max_risk or meta.get("default_max_risk") or "readonly"
|
||||||
max_i = risk_index(meta, max_risk)
|
max_i = risk_index(meta, max_risk)
|
||||||
automation_filter = args.automation or "automated"
|
automation_filter = args.automation or ("any" if bypass_risk else "automated")
|
||||||
wanted = set(parse_ids(args.ids or []))
|
wanted = set(parse_ids(args.ids or []))
|
||||||
selected: list[dict[str, Any]] = []
|
selected: list[dict[str, Any]] = []
|
||||||
skipped: list[dict[str, str]] = []
|
skipped: list[dict[str, str]] = []
|
||||||
@@ -786,6 +817,20 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
|
|||||||
continue
|
continue
|
||||||
if args.tag and case.get("tag") != args.tag:
|
if args.tag and case.get("tag") != args.tag:
|
||||||
continue
|
continue
|
||||||
|
if args.test_type and test_type(case) != args.test_type:
|
||||||
|
continue
|
||||||
|
if args.suite and case_suite(case) != args.suite:
|
||||||
|
continue
|
||||||
|
# A mixed catalog keeps legacy cases runnable. Standard keeps the
|
||||||
|
# historical client-only implicit default; isolated-full means
|
||||||
|
# exactly what it says and includes every active test type.
|
||||||
|
if (
|
||||||
|
not bypass_risk
|
||||||
|
and not args.test_type
|
||||||
|
and not args.suite
|
||||||
|
and test_type(case) == "integration"
|
||||||
|
):
|
||||||
|
continue
|
||||||
if args.driver and case.get("driver") != args.driver:
|
if args.driver and case.get("driver") != args.driver:
|
||||||
continue
|
continue
|
||||||
automation = str(case.get("automation") or "automated")
|
automation = str(case.get("automation") or "automated")
|
||||||
@@ -801,7 +846,7 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
|
|||||||
except SystemExit:
|
except SystemExit:
|
||||||
skipped.append({"id": cid, "reason": f"未知风险 {risk}"})
|
skipped.append({"id": cid, "reason": f"未知风险 {risk}"})
|
||||||
continue
|
continue
|
||||||
if r_i > max_i:
|
if not bypass_risk and r_i > max_i:
|
||||||
skipped.append({"id": cid, "reason": f"risk={risk} 超过上限 {max_risk}"})
|
skipped.append({"id": cid, "reason": f"risk={risk} 超过上限 {max_risk}"})
|
||||||
continue
|
continue
|
||||||
selected.append(public_case(case))
|
selected.append(public_case(case))
|
||||||
@@ -813,7 +858,22 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
|
|||||||
)
|
)
|
||||||
return 2
|
return 2
|
||||||
|
|
||||||
selected, together = expand_run_together(meta, cases, selected, skipped, max_i, automation_filter)
|
selected, together = expand_run_together(
|
||||||
|
meta,
|
||||||
|
cases,
|
||||||
|
selected,
|
||||||
|
skipped,
|
||||||
|
max_i,
|
||||||
|
automation_filter,
|
||||||
|
bypass_risk=bypass_risk,
|
||||||
|
)
|
||||||
|
|
||||||
|
# The historical key is kept in the JSON, but the gate now validates both
|
||||||
|
# test types before any caller can start a phase.
|
||||||
|
journey_gate = journey_preflight(selected, sessions=meta.get("sessions") or {})
|
||||||
|
if journey_gate["status"] == "blocked":
|
||||||
|
print(json.dumps({"preflight": journey_gate, "journey_preflight": journey_gate}, ensure_ascii=False, indent=2))
|
||||||
|
return 2
|
||||||
|
|
||||||
preflight = mobile_preflight(
|
preflight = mobile_preflight(
|
||||||
root,
|
root,
|
||||||
@@ -840,11 +900,18 @@ def cmd_select(args: argparse.Namespace, root: Path) -> int:
|
|||||||
json.dumps(
|
json.dumps(
|
||||||
{
|
{
|
||||||
"max_risk": max_risk,
|
"max_risk": max_risk,
|
||||||
|
"execution_profile": execution_profile,
|
||||||
|
"risk_filter": "disabled" if bypass_risk else "enforced",
|
||||||
"automation": automation_filter,
|
"automation": automation_filter,
|
||||||
"count": len(selected),
|
"count": len(selected),
|
||||||
"vars": vars_map,
|
"vars": vars_map,
|
||||||
"artifacts": meta.get("artifacts") or {},
|
"artifacts": meta.get("artifacts") or {},
|
||||||
"preflight": preflight,
|
"preflight": preflight,
|
||||||
|
"journey_preflight": journey_gate,
|
||||||
|
"test_types": {
|
||||||
|
"journey": sum(1 for case in selected if test_type(case) == "journey"),
|
||||||
|
"integration": sum(1 for case in selected if test_type(case) == "integration"),
|
||||||
|
},
|
||||||
"skipped": skipped,
|
"skipped": skipped,
|
||||||
"run_together": together,
|
"run_together": together,
|
||||||
"status_probe": status_probes,
|
"status_probe": status_probes,
|
||||||
@@ -915,6 +982,12 @@ def validate(root: Path) -> list[str]:
|
|||||||
default_max = meta.get("default_max_risk")
|
default_max = meta.get("default_max_risk")
|
||||||
if default_max not in risks:
|
if default_max not in risks:
|
||||||
errors.append(f"default_max_risk={default_max!r} 不在 risks 中")
|
errors.append(f"default_max_risk={default_max!r} 不在 risks 中")
|
||||||
|
profile = meta.get("execution_profile", "standard")
|
||||||
|
if profile not in EXECUTION_PROFILES:
|
||||||
|
errors.append(
|
||||||
|
f"execution_profile={profile!r} 非法,只能是 {', '.join(EXECUTION_PROFILES)}"
|
||||||
|
)
|
||||||
|
isolated_full = profile == "isolated-full"
|
||||||
gates = meta.get("gates") or {}
|
gates = meta.get("gates") or {}
|
||||||
if gates and not isinstance(gates, dict):
|
if gates and not isinstance(gates, dict):
|
||||||
errors.append("catalog.gates 必须是 mapping")
|
errors.append("catalog.gates 必须是 mapping")
|
||||||
@@ -927,6 +1000,28 @@ def validate(root: Path) -> list[str]:
|
|||||||
if not isinstance(sessions, dict):
|
if not isinstance(sessions, dict):
|
||||||
errors.append("catalog.sessions 必须是 mapping")
|
errors.append("catalog.sessions 必须是 mapping")
|
||||||
sessions = {}
|
sessions = {}
|
||||||
|
client_suites = meta.get("client_suites")
|
||||||
|
if client_suites is not None:
|
||||||
|
if not isinstance(client_suites, dict):
|
||||||
|
errors.append("catalog.client_suites 必须是 mapping")
|
||||||
|
else:
|
||||||
|
unknown = sorted(set(client_suites) - set(CLIENT_SUITES))
|
||||||
|
if unknown:
|
||||||
|
errors.append("catalog.client_suites 含非法套件: " + ", ".join(unknown))
|
||||||
|
missing_suites = sorted(set(CLIENT_SUITES) - set(client_suites))
|
||||||
|
if missing_suites:
|
||||||
|
errors.append("catalog.client_suites 缺少标准套件: " + ", ".join(missing_suites))
|
||||||
|
integration_suites = meta.get("integration_suites")
|
||||||
|
if integration_suites is not None:
|
||||||
|
if not isinstance(integration_suites, dict):
|
||||||
|
errors.append("catalog.integration_suites 必须是 mapping")
|
||||||
|
else:
|
||||||
|
unknown = sorted(set(integration_suites) - set(INTEGRATION_SUITES))
|
||||||
|
if unknown:
|
||||||
|
errors.append("catalog.integration_suites 含非法套件: " + ", ".join(unknown))
|
||||||
|
missing_suites = sorted(set(INTEGRATION_SUITES) - set(integration_suites))
|
||||||
|
if missing_suites:
|
||||||
|
errors.append("catalog.integration_suites 缺少标准套件: " + ", ".join(missing_suites))
|
||||||
artifacts = meta.get("artifacts")
|
artifacts = meta.get("artifacts")
|
||||||
if artifacts is not None:
|
if artifacts is not None:
|
||||||
if not isinstance(artifacts, dict):
|
if not isinstance(artifacts, dict):
|
||||||
@@ -1003,6 +1098,7 @@ def validate(root: Path) -> list[str]:
|
|||||||
errors.append(f"{where}: category={cat} 与文件 category={file_cat} 不一致")
|
errors.append(f"{where}: category={cat} 与文件 category={file_cat} 不一致")
|
||||||
if cat and cat_ids and cat not in cat_ids:
|
if cat and cat_ids and cat not in cat_ids:
|
||||||
errors.append(f"{where}: category={cat} 不在 catalog.categories")
|
errors.append(f"{where}: category={cat} 不在 catalog.categories")
|
||||||
|
errors.extend(contract_errors(case, sessions=sessions))
|
||||||
if case.get("lifecycle", "active") not in LIFECYCLES:
|
if case.get("lifecycle", "active") not in LIFECYCLES:
|
||||||
errors.append(f"{where}: 非法 lifecycle")
|
errors.append(f"{where}: 非法 lifecycle")
|
||||||
if case.get("tag") not in TAGS:
|
if case.get("tag") not in TAGS:
|
||||||
@@ -1018,8 +1114,17 @@ def validate(root: Path) -> list[str]:
|
|||||||
if case.get("risk") in (None, "readonly"):
|
if case.get("risk") in (None, "readonly"):
|
||||||
errors.append(f"{where}: risk=readonly 不要填 gate")
|
errors.append(f"{where}: risk=readonly 不要填 gate")
|
||||||
risk = case.get("risk")
|
risk = case.get("risk")
|
||||||
if risk and risk != "readonly" and risk not in authorize and not gate:
|
if risk and risk != "readonly" and risk not in authorize and not gate and not isolated_full:
|
||||||
errors.append(f"{where}: {risk} 需要 authorize.{risk} 或 gate")
|
errors.append(f"{where}: {risk} 需要 authorize.{risk} 或 gate")
|
||||||
|
if isolated_full and case.get("lifecycle", "active") == "active" and risk != "readonly":
|
||||||
|
cleanup = case.get("cleanup")
|
||||||
|
cleanup_ok = isinstance(cleanup, dict) and (
|
||||||
|
bool(cleanup.get("steps"))
|
||||||
|
or bool(str(cleanup.get("run") or "").strip())
|
||||||
|
or (cleanup.get("required") is False and bool(str(cleanup.get("reason") or "").strip()))
|
||||||
|
)
|
||||||
|
if not cleanup_ok:
|
||||||
|
errors.append(f"{where}: isolated-full 的写入型用例必须声明 cleanup/rollback steps/run")
|
||||||
automation = str(case.get("automation") or "automated")
|
automation = str(case.get("automation") or "automated")
|
||||||
if automation not in AUTOMATIONS:
|
if automation not in AUTOMATIONS:
|
||||||
errors.append(f"{where}: 非法 automation={case.get('automation')}")
|
errors.append(f"{where}: 非法 automation={case.get('automation')}")
|
||||||
@@ -1151,7 +1256,19 @@ def cmd_next_id(root: Path, category: str) -> int:
|
|||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def authorized(vars_map: dict[str, str], meta: dict[str, Any], case: dict[str, Any]) -> bool:
|
def authorized(
|
||||||
|
vars_map: dict[str, str],
|
||||||
|
meta: dict[str, Any],
|
||||||
|
case: dict[str, Any],
|
||||||
|
*,
|
||||||
|
execution_profile: str | None = None,
|
||||||
|
) -> bool:
|
||||||
|
profile = resolve_execution_profile(meta, execution_profile)
|
||||||
|
if profile == "isolated-full":
|
||||||
|
# AUTHORIZE_* and gate variables are intentionally not run-time
|
||||||
|
# switches for an isolated resettable environment. Capability and
|
||||||
|
# cleanup preflight are the safety boundary for this profile.
|
||||||
|
return True
|
||||||
risk = str(case.get("risk") or "readonly")
|
risk = str(case.get("risk") or "readonly")
|
||||||
risks = list(meta.get("risks") or ["readonly"])
|
risks = list(meta.get("risks") or ["readonly"])
|
||||||
lowest = risks[0] if risks else "readonly"
|
lowest = risks[0] if risks else "readonly"
|
||||||
@@ -1171,6 +1288,7 @@ def authorized(vars_map: dict[str, str], meta: dict[str, Any], case: dict[str, A
|
|||||||
|
|
||||||
def cmd_run_script(root: Path, case_id: str) -> int:
|
def cmd_run_script(root: Path, case_id: str) -> int:
|
||||||
meta = load_catalog_meta(root)
|
meta = load_catalog_meta(root)
|
||||||
|
execution_profile = resolve_execution_profile(meta)
|
||||||
vars_map = resolve_vars(root, meta)
|
vars_map = resolve_vars(root, meta)
|
||||||
case = next((c for c in load_all_cases(root) if c.get("id") == case_id), None)
|
case = next((c for c in load_all_cases(root) if c.get("id") == case_id), None)
|
||||||
if not case:
|
if not case:
|
||||||
@@ -1208,10 +1326,10 @@ def cmd_run_script(root: Path, case_id: str) -> int:
|
|||||||
)
|
)
|
||||||
return 2
|
return 2
|
||||||
risk = str(case.get("risk") or "readonly")
|
risk = str(case.get("risk") or "readonly")
|
||||||
if not authorized(vars_map, meta, case):
|
if not authorized(vars_map, meta, case, execution_profile=execution_profile):
|
||||||
print(
|
print(
|
||||||
json.dumps(
|
json.dumps(
|
||||||
{"id": public_id, "result": "skipped", "evidence": f"风险 {risk} 未授权"},
|
{"id": public_id, "result": "skipped", "evidence": f"风险 {risk} 未授权", "execution_profile": execution_profile},
|
||||||
ensure_ascii=False,
|
ensure_ascii=False,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -1260,6 +1378,7 @@ def cmd_run_script(root: Path, case_id: str) -> int:
|
|||||||
{
|
{
|
||||||
"id": public_id,
|
"id": public_id,
|
||||||
"result": "executed",
|
"result": "executed",
|
||||||
|
"execution_profile": execution_profile,
|
||||||
"exit": proc.returncode,
|
"exit": proc.returncode,
|
||||||
"evidence": evidence,
|
"evidence": evidence,
|
||||||
},
|
},
|
||||||
@@ -1280,7 +1399,22 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
p_sel.add_argument("--category")
|
p_sel.add_argument("--category")
|
||||||
p_sel.add_argument("--id", action="append", dest="ids", default=[])
|
p_sel.add_argument("--id", action="append", dest="ids", default=[])
|
||||||
p_sel.add_argument("--tag", choices=TAGS)
|
p_sel.add_argument("--tag", choices=TAGS)
|
||||||
|
p_sel.add_argument(
|
||||||
|
"--suite", "--client-suite", dest="suite",
|
||||||
|
choices=[*CLIENT_SUITES, *INTEGRATION_SUITES],
|
||||||
|
help="客户端或集成测试套件;不传时默认不选 integration",
|
||||||
|
)
|
||||||
|
p_sel.add_argument(
|
||||||
|
"--test-type", choices=TEST_TYPES, default=None,
|
||||||
|
help="按顶层 test_type 筛选;integration 必须显式选择",
|
||||||
|
)
|
||||||
p_sel.add_argument("--max-risk")
|
p_sel.add_argument("--max-risk")
|
||||||
|
p_sel.add_argument(
|
||||||
|
"--execution-profile",
|
||||||
|
choices=EXECUTION_PROFILES,
|
||||||
|
default=None,
|
||||||
|
help="执行策略;isolated-full 在隔离可重置环境中不按 risk/authorize 筛选",
|
||||||
|
)
|
||||||
p_sel.add_argument("--driver", choices=DRIVERS)
|
p_sel.add_argument("--driver", choices=DRIVERS)
|
||||||
p_sel.add_argument(
|
p_sel.add_argument(
|
||||||
"--automation",
|
"--automation",
|
||||||
|
|||||||
@@ -0,0 +1,150 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Build structured tester issues and the development feedback route.
|
||||||
|
|
||||||
|
This module only writes the run contract. Updating a remote PR/issue or
|
||||||
|
notifying a dispatcher remains an explicit caller-side integration step; no
|
||||||
|
network mutation is hidden in the tester.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from copy import deepcopy
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Any, Iterable, Mapping
|
||||||
|
|
||||||
|
|
||||||
|
ISSUE_TYPES = ("product", "test")
|
||||||
|
TRIGGERS = ("pr", "requirement", "bug")
|
||||||
|
ISSUE_RESULTS = frozenset({"fail", "environment-fail"})
|
||||||
|
|
||||||
|
|
||||||
|
def _now() -> str:
|
||||||
|
return datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def issue_type_for(result: str) -> str:
|
||||||
|
if result == "fail":
|
||||||
|
return "product"
|
||||||
|
if result == "environment-fail":
|
||||||
|
return "test"
|
||||||
|
raise ValueError(f"只有 fail/environment-fail 可以生成 issue: {result}")
|
||||||
|
|
||||||
|
|
||||||
|
def _value(row: Mapping[str, Any], key: str, defaults: Mapping[str, Any]) -> Any:
|
||||||
|
value = row.get(key)
|
||||||
|
return value if value not in (None, "") else defaults.get(key, "")
|
||||||
|
|
||||||
|
|
||||||
|
def build_issue(
|
||||||
|
row: Mapping[str, Any],
|
||||||
|
*,
|
||||||
|
defaults: Mapping[str, Any] | None = None,
|
||||||
|
issue_type: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Create the stable issue shape required by the run report."""
|
||||||
|
defaults = defaults or {}
|
||||||
|
result = str(row.get("result") or "")
|
||||||
|
if result not in ISSUE_RESULTS:
|
||||||
|
raise ValueError(f"不能为 {result!r} 结果创建 defect: {row.get('id')}")
|
||||||
|
selected_type = issue_type or row.get("issue_type") or issue_type_for(result)
|
||||||
|
if selected_type not in ISSUE_TYPES:
|
||||||
|
raise ValueError(f"issue_type 必须是 product/test: {selected_type}")
|
||||||
|
testcase = str(row.get("testcase") or row.get("id") or "")
|
||||||
|
if not testcase:
|
||||||
|
raise ValueError("issue 缺少 testcase")
|
||||||
|
return {
|
||||||
|
"id": str(row.get("issue_id") or f"TEST-{testcase}-{result}"),
|
||||||
|
"testcase": testcase,
|
||||||
|
"issue_type": selected_type,
|
||||||
|
"result": result,
|
||||||
|
"phase": _value(row, "phase", defaults),
|
||||||
|
"expected": _value(row, "expected", defaults),
|
||||||
|
"observed": _value(row, "observed", defaults),
|
||||||
|
"artifact_hash": _value(row, "artifact_hash", defaults),
|
||||||
|
"environment": _value(row, "environment", defaults),
|
||||||
|
"evidence_path": _value(row, "evidence_path", defaults),
|
||||||
|
"reproduction": _value(row, "reproduction", defaults),
|
||||||
|
"next_step": _value(row, "next_step", defaults),
|
||||||
|
"created_at": str(row.get("created_at") or _now()),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def collect_issues(
|
||||||
|
results: Iterable[Mapping[str, Any]],
|
||||||
|
*,
|
||||||
|
defaults: Mapping[str, Any] | None = None,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
issues: list[dict[str, Any]] = []
|
||||||
|
for row in results:
|
||||||
|
if not isinstance(row, Mapping) or str(row.get("result") or "") not in ISSUE_RESULTS:
|
||||||
|
continue
|
||||||
|
explicit = row.get("issue")
|
||||||
|
source = dict(row)
|
||||||
|
if isinstance(explicit, Mapping):
|
||||||
|
source.update(explicit)
|
||||||
|
issues.append(build_issue(source, defaults=defaults))
|
||||||
|
return issues
|
||||||
|
|
||||||
|
|
||||||
|
def feedback_plan(
|
||||||
|
trigger: str,
|
||||||
|
*,
|
||||||
|
target: str = "",
|
||||||
|
related_issue: str = "",
|
||||||
|
dispatcher: str = "dispatcher",
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Describe the required handoff without performing remote mutations."""
|
||||||
|
if trigger not in TRIGGERS:
|
||||||
|
raise ValueError(f"trigger 必须是 {', '.join(TRIGGERS)}")
|
||||||
|
if trigger == "pr":
|
||||||
|
actions = [
|
||||||
|
{"action": "update_pr", "target": target, "status": "required"},
|
||||||
|
{"action": "update_related_issue", "target": related_issue, "status": "required"},
|
||||||
|
{"action": "notify_dispatcher", "target": dispatcher, "status": "required"},
|
||||||
|
{"action": "reassign_development", "target": dispatcher, "status": "pending"},
|
||||||
|
]
|
||||||
|
route = "pr-update-issue-dispatcher-reassign"
|
||||||
|
else:
|
||||||
|
actions = [
|
||||||
|
{"action": "return_to_development", "target": target or trigger, "status": "required"},
|
||||||
|
]
|
||||||
|
route = "development"
|
||||||
|
return {
|
||||||
|
"trigger": trigger,
|
||||||
|
"route": route,
|
||||||
|
"target": target,
|
||||||
|
"related_issue": related_issue,
|
||||||
|
"actions": actions,
|
||||||
|
"remote_mutation": "caller-required",
|
||||||
|
"created_at": _now(),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def attach_feedback(
|
||||||
|
run: Mapping[str, Any],
|
||||||
|
*,
|
||||||
|
trigger: str | None = None,
|
||||||
|
target: str = "",
|
||||||
|
related_issue: str = "",
|
||||||
|
dispatcher: str = "dispatcher",
|
||||||
|
defaults: Mapping[str, Any] | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Return a new run mapping with results, defects and routing preserved."""
|
||||||
|
output = deepcopy(dict(run))
|
||||||
|
results = output.get("results")
|
||||||
|
if not isinstance(results, list):
|
||||||
|
raise ValueError("run 缺少 results 列表")
|
||||||
|
issues = collect_issues(results, defaults=defaults)
|
||||||
|
output["defects"] = issues
|
||||||
|
# Compatibility name for consumers that call the same structure
|
||||||
|
# ``test-issues``. It is deliberately a copy, not a second source of truth.
|
||||||
|
output["test_issues"] = deepcopy(issues)
|
||||||
|
if trigger:
|
||||||
|
output["feedback"] = feedback_plan(
|
||||||
|
trigger,
|
||||||
|
target=target,
|
||||||
|
related_issue=related_issue,
|
||||||
|
dispatcher=dispatcher,
|
||||||
|
)
|
||||||
|
return output
|
||||||
|
|
||||||
@@ -13,6 +13,8 @@ import sys
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
from journey import suite as case_suite, test_type
|
||||||
|
|
||||||
try:
|
try:
|
||||||
import yaml
|
import yaml
|
||||||
except ImportError: # pragma: no cover
|
except ImportError: # pragma: no cover
|
||||||
@@ -22,6 +24,10 @@ except ImportError: # pragma: no cover
|
|||||||
FIELDS = (
|
FIELDS = (
|
||||||
"category",
|
"category",
|
||||||
"id",
|
"id",
|
||||||
|
"test_type",
|
||||||
|
"suite",
|
||||||
|
"client_suite",
|
||||||
|
"platform",
|
||||||
"maps_to",
|
"maps_to",
|
||||||
"title",
|
"title",
|
||||||
"what",
|
"what",
|
||||||
@@ -156,10 +162,17 @@ def export_cases_csv(root: Path, out: Path | None = None) -> Path:
|
|||||||
writer.writeheader()
|
writer.writeheader()
|
||||||
for case in cases:
|
for case in cases:
|
||||||
cid = str(case.get("id") or "")
|
cid = str(case.get("id") or "")
|
||||||
|
client_suite_value = case.get("client_suite") or (
|
||||||
|
case.get("suite") if test_type(case) == "journey" else ""
|
||||||
|
)
|
||||||
writer.writerow(
|
writer.writerow(
|
||||||
{
|
{
|
||||||
"category": case.get("category") or "",
|
"category": case.get("category") or "",
|
||||||
"id": cid,
|
"id": cid,
|
||||||
|
"test_type": test_type(case),
|
||||||
|
"suite": case_suite(case),
|
||||||
|
"client_suite": client_suite_value or "",
|
||||||
|
"platform": one_line(case.get("platform")),
|
||||||
"maps_to": case.get("maps_to") or "",
|
"maps_to": case.get("maps_to") or "",
|
||||||
"title": case.get("title") or "",
|
"title": case.get("title") or "",
|
||||||
"what": one_line(case.get("what")),
|
"what": one_line(case.get("what")),
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ except ImportError: # pragma: no cover
|
|||||||
raise SystemExit(2)
|
raise SystemExit(2)
|
||||||
|
|
||||||
from generate_cases_csv import load_cases, one_line, resolve_root, tester_dir
|
from generate_cases_csv import load_cases, one_line, resolve_root, tester_dir
|
||||||
|
from journey import suite as case_suite, test_type
|
||||||
|
|
||||||
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
||||||
RUN_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
|
RUN_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
|
||||||
@@ -25,6 +26,10 @@ EXECUTED_YES = frozenset({"pass", "fail", "environment-fail"})
|
|||||||
EXECUTED_NO = frozenset({"skipped", "not-run"})
|
EXECUTED_NO = frozenset({"skipped", "not-run"})
|
||||||
FIELDS = (
|
FIELDS = (
|
||||||
"id",
|
"id",
|
||||||
|
"test_type",
|
||||||
|
"suite",
|
||||||
|
"client_suite",
|
||||||
|
"platform",
|
||||||
"result",
|
"result",
|
||||||
"executed",
|
"executed",
|
||||||
"evidence",
|
"evidence",
|
||||||
@@ -35,6 +40,16 @@ FIELDS = (
|
|||||||
"risk",
|
"risk",
|
||||||
"driver",
|
"driver",
|
||||||
"maps_to",
|
"maps_to",
|
||||||
|
"blocked_by",
|
||||||
|
"issue_id",
|
||||||
|
"issue_type",
|
||||||
|
"expected",
|
||||||
|
"observed",
|
||||||
|
"artifact_hash",
|
||||||
|
"environment",
|
||||||
|
"evidence_path",
|
||||||
|
"reproduction",
|
||||||
|
"next_step",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -107,6 +122,23 @@ def case_index(root: Path) -> dict[str, dict[str, Any]]:
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def issue_index(run: dict[str, Any]) -> dict[str, dict[str, Any]]:
|
||||||
|
"""Index canonical defects and compatibility test_issues by testcase."""
|
||||||
|
indexed: dict[str, dict[str, Any]] = {}
|
||||||
|
raw = run.get("defects")
|
||||||
|
if not isinstance(raw, list):
|
||||||
|
raw = run.get("test_issues")
|
||||||
|
if not isinstance(raw, list):
|
||||||
|
return indexed
|
||||||
|
for issue in raw:
|
||||||
|
if not isinstance(issue, dict):
|
||||||
|
continue
|
||||||
|
testcase = str(issue.get("testcase") or issue.get("id") or "")
|
||||||
|
if testcase:
|
||||||
|
indexed.setdefault(testcase, issue)
|
||||||
|
return indexed
|
||||||
|
|
||||||
|
|
||||||
def export_run_csv(
|
def export_run_csv(
|
||||||
root: Path,
|
root: Path,
|
||||||
run_path: Path | None = None,
|
run_path: Path | None = None,
|
||||||
@@ -118,6 +150,7 @@ def export_run_csv(
|
|||||||
path = path.resolve()
|
path = path.resolve()
|
||||||
run = load_run(path)
|
run = load_run(path)
|
||||||
cases = case_index(root)
|
cases = case_index(root)
|
||||||
|
issues = issue_index(run)
|
||||||
dest = out or path.with_suffix(".csv")
|
dest = out or path.with_suffix(".csv")
|
||||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||||
with dest.open("w", encoding="utf-8-sig", newline="") as handle:
|
with dest.open("w", encoding="utf-8-sig", newline="") as handle:
|
||||||
@@ -129,9 +162,19 @@ def export_run_csv(
|
|||||||
cid = str(item["id"])
|
cid = str(item["id"])
|
||||||
result = str(item.get("result") or "")
|
result = str(item.get("result") or "")
|
||||||
case = cases.get(cid) or {}
|
case = cases.get(cid) or {}
|
||||||
|
issue = dict(issues.get(cid, {}))
|
||||||
|
if isinstance(item.get("issue"), dict):
|
||||||
|
issue.update(item["issue"])
|
||||||
|
client_suite_value = case.get("client_suite") or (
|
||||||
|
case.get("suite") if test_type(case) == "journey" else ""
|
||||||
|
)
|
||||||
writer.writerow(
|
writer.writerow(
|
||||||
{
|
{
|
||||||
"id": cid,
|
"id": cid,
|
||||||
|
"test_type": test_type(case),
|
||||||
|
"suite": case_suite(case),
|
||||||
|
"client_suite": client_suite_value or "",
|
||||||
|
"platform": one_line(case.get("platform")),
|
||||||
"result": result,
|
"result": result,
|
||||||
"executed": executed_cell(result),
|
"executed": executed_cell(result),
|
||||||
"evidence": one_line(item.get("evidence")),
|
"evidence": one_line(item.get("evidence")),
|
||||||
@@ -142,6 +185,16 @@ def export_run_csv(
|
|||||||
"risk": case.get("risk") or "",
|
"risk": case.get("risk") or "",
|
||||||
"driver": case.get("driver") or "",
|
"driver": case.get("driver") or "",
|
||||||
"maps_to": case.get("maps_to") or "",
|
"maps_to": case.get("maps_to") or "",
|
||||||
|
"blocked_by": one_line(item.get("blocked_by")),
|
||||||
|
"issue_id": issue.get("id") or item.get("issue_id") or "",
|
||||||
|
"issue_type": issue.get("issue_type") or item.get("issue_type") or "",
|
||||||
|
"expected": one_line(issue.get("expected") or item.get("expected")),
|
||||||
|
"observed": one_line(issue.get("observed") or item.get("observed")),
|
||||||
|
"artifact_hash": issue.get("artifact_hash") or item.get("artifact_hash") or "",
|
||||||
|
"environment": one_line(issue.get("environment") or item.get("environment")),
|
||||||
|
"evidence_path": one_line(issue.get("evidence_path") or item.get("evidence_path")),
|
||||||
|
"reproduction": one_line(issue.get("reproduction") or item.get("reproduction")),
|
||||||
|
"next_step": one_line(issue.get("next_step") or item.get("next_step")),
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
return dest
|
return dest
|
||||||
|
|||||||
@@ -0,0 +1,290 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Shared client-journey contract and result lifecycle helpers.
|
||||||
|
|
||||||
|
The mobile/browser driver still owns the real UI interaction. This module
|
||||||
|
owns the parts that must be identical for every client catalog: contract
|
||||||
|
validation, preflight blocking, and result propagation. It deliberately
|
||||||
|
returns the existing tester result vocabulary instead of introducing a second
|
||||||
|
report format.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any, Iterable, Mapping
|
||||||
|
|
||||||
|
|
||||||
|
TEST_TYPES = ("journey", "integration")
|
||||||
|
CLIENT_SUITES = ("client-smoke", "client-regression", "safety")
|
||||||
|
INTEGRATION_SUITES = ("api-contract", "authorization", "build-contract")
|
||||||
|
PLATFORMS = ("android", "ios")
|
||||||
|
FAILURE_POLICIES = ("stop", "continue")
|
||||||
|
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
||||||
|
|
||||||
|
|
||||||
|
def client_suite(case: Mapping[str, Any]) -> str:
|
||||||
|
"""Return the canonical client suite, accepting ``suite`` as a migration alias."""
|
||||||
|
value = case.get("client_suite")
|
||||||
|
alias = case.get("suite")
|
||||||
|
if value and alias and str(value) != str(alias):
|
||||||
|
return ""
|
||||||
|
return str(value or alias or "")
|
||||||
|
|
||||||
|
|
||||||
|
def integration_suite(case: Mapping[str, Any]) -> str:
|
||||||
|
"""Return the integration suite, accepting ``suite`` during migration."""
|
||||||
|
value = case.get("integration_suite")
|
||||||
|
alias = case.get("suite")
|
||||||
|
if value and alias and str(value) != str(alias):
|
||||||
|
return ""
|
||||||
|
return str(value or alias or "")
|
||||||
|
|
||||||
|
|
||||||
|
def test_type(case: Mapping[str, Any]) -> str:
|
||||||
|
"""Return an explicit type or infer one for an old catalog entry."""
|
||||||
|
declared = str(case.get("test_type") or "").strip().lower()
|
||||||
|
if declared:
|
||||||
|
return declared
|
||||||
|
if client_suite(case) or case.get("journey") is not None:
|
||||||
|
return "journey"
|
||||||
|
if integration_suite(case):
|
||||||
|
return "integration"
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def suite(case: Mapping[str, Any]) -> str:
|
||||||
|
"""Return the report/selection suite for either test type."""
|
||||||
|
kind = test_type(case)
|
||||||
|
return client_suite(case) if kind == "journey" else integration_suite(case)
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_platforms(value: Any) -> list[str]:
|
||||||
|
if isinstance(value, str):
|
||||||
|
values = [value]
|
||||||
|
elif isinstance(value, (list, tuple)):
|
||||||
|
values = list(value)
|
||||||
|
else:
|
||||||
|
return []
|
||||||
|
return [str(item).strip().lower() for item in values if str(item).strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def _non_empty_list(value: Any) -> bool:
|
||||||
|
return isinstance(value, list) and bool(value) and all(str(item).strip() for item in value)
|
||||||
|
|
||||||
|
|
||||||
|
def _has_account(fixture: Mapping[str, Any]) -> bool:
|
||||||
|
account = fixture.get("account")
|
||||||
|
if isinstance(account, Mapping):
|
||||||
|
return bool(str(account.get("role") or account.get("username") or "").strip())
|
||||||
|
return bool(str(account or "").strip())
|
||||||
|
|
||||||
|
|
||||||
|
def contract_errors(case: Mapping[str, Any], sessions: Mapping[str, Any] | None = None) -> list[str]:
|
||||||
|
"""Validate a client journey without requiring app-specific field names.
|
||||||
|
|
||||||
|
A case becomes a client journey when it declares ``client_suite`` (or the
|
||||||
|
temporary ``suite`` alias). Legacy non-client cases keep the existing
|
||||||
|
schema. Once opted in, all lifecycle inputs are explicit so a missing
|
||||||
|
fixture/cleanup can stop the run before any case result is fabricated.
|
||||||
|
"""
|
||||||
|
kind = test_type(case)
|
||||||
|
declared_kind = str(case.get("test_type") or "").strip().lower()
|
||||||
|
if declared_kind and declared_kind not in TEST_TYPES:
|
||||||
|
return [f"{case.get('id') or '?'}: test_type 必须是 journey 或 integration"]
|
||||||
|
if not kind:
|
||||||
|
return []
|
||||||
|
if kind == "integration":
|
||||||
|
return integration_contract_errors(case)
|
||||||
|
|
||||||
|
suite_name = client_suite(case)
|
||||||
|
journey = case.get("journey")
|
||||||
|
errors: list[str] = []
|
||||||
|
cid = str(case.get("id") or "?")
|
||||||
|
where = f"{cid}: client journey"
|
||||||
|
if case.get("client_suite") and case.get("suite") and str(case["client_suite"]) != str(case["suite"]):
|
||||||
|
errors.append(f"{where}: client_suite 与 suite 不一致")
|
||||||
|
if suite_name not in CLIENT_SUITES:
|
||||||
|
errors.append(f"{where}: client_suite 必须是 {', '.join(CLIENT_SUITES)}")
|
||||||
|
|
||||||
|
if not isinstance(journey, Mapping):
|
||||||
|
errors.append(f"{where}: journey 必须是 mapping")
|
||||||
|
journey = {}
|
||||||
|
for key in ("goal", "entry_state", "final_state"):
|
||||||
|
if not str(journey.get(key) or "").strip():
|
||||||
|
errors.append(f"{where}: journey.{key} 必填")
|
||||||
|
if not _non_empty_list(journey.get("actions")):
|
||||||
|
errors.append(f"{where}: journey.actions 必须是非空列表")
|
||||||
|
if not _non_empty_list(journey.get("assertions")):
|
||||||
|
errors.append(f"{where}: journey.assertions 必须是非空列表")
|
||||||
|
policy = str(journey.get("failure_policy") or "")
|
||||||
|
if policy not in FAILURE_POLICIES:
|
||||||
|
errors.append(f"{where}: journey.failure_policy 必须是 stop 或 continue")
|
||||||
|
|
||||||
|
fixture = case.get("fixture")
|
||||||
|
if not isinstance(fixture, Mapping):
|
||||||
|
errors.append(f"{where}: fixture 缺失,必须在 preflight 前声明")
|
||||||
|
elif not _has_account(fixture):
|
||||||
|
errors.append(f"{where}: fixture.account 缺失,必须声明测试账号或角色")
|
||||||
|
|
||||||
|
cleanup = case.get("cleanup")
|
||||||
|
if not isinstance(cleanup, Mapping):
|
||||||
|
errors.append(f"{where}: cleanup 缺失,必须在 preflight 前声明")
|
||||||
|
else:
|
||||||
|
steps = cleanup.get("steps")
|
||||||
|
command = cleanup.get("run")
|
||||||
|
explicitly_noop = cleanup.get("required") is False and str(cleanup.get("reason") or "").strip()
|
||||||
|
if not explicitly_noop and not _non_empty_list(steps) and not str(command or "").strip():
|
||||||
|
errors.append(f"{where}: cleanup 必须有非空 steps/run;只读用例也要显式声明 reason")
|
||||||
|
|
||||||
|
platforms = normalize_platforms(case.get("platform"))
|
||||||
|
if not platforms:
|
||||||
|
errors.append(f"{where}: platform 必须声明 android/ios")
|
||||||
|
elif any(platform not in PLATFORMS for platform in platforms):
|
||||||
|
errors.append(f"{where}: platform 只能是 {', '.join(PLATFORMS)}")
|
||||||
|
elif len(platforms) != len(set(platforms)):
|
||||||
|
errors.append(f"{where}: platform 不能重复")
|
||||||
|
|
||||||
|
driver = case.get("driver")
|
||||||
|
if driver not in ("mobile", "browser"):
|
||||||
|
errors.append(f"{where}: 客户端旅程 driver 必须是 mobile 或 browser")
|
||||||
|
if driver and driver in ("mobile", "browser"):
|
||||||
|
session = case.get("session")
|
||||||
|
if not str(session or "").strip():
|
||||||
|
errors.append(f"{where}: session 必须声明可复用客户端会话")
|
||||||
|
elif sessions is not None and session not in sessions:
|
||||||
|
errors.append(f"{where}: session={session} 未在 catalog.sessions 声明")
|
||||||
|
if suite_name == "safety" and str(case.get("tag") or "") == "smoke":
|
||||||
|
errors.append(f"{where}: safety 不得标记为 smoke")
|
||||||
|
return errors
|
||||||
|
|
||||||
|
|
||||||
|
def integration_contract_errors(case: Mapping[str, Any]) -> list[str]:
|
||||||
|
"""Validate the non-client half of the tester catalog.
|
||||||
|
|
||||||
|
Integration cases deliberately do not require a browser/mobile session. A
|
||||||
|
``how`` block is the legacy call declaration; newer catalogs may use
|
||||||
|
``request``/``input``/``call`` or an ``integration`` mapping as well.
|
||||||
|
``expected`` remains the common protocol/data assertion list.
|
||||||
|
"""
|
||||||
|
cid = str(case.get("id") or "?")
|
||||||
|
where = f"{cid}: integration"
|
||||||
|
errors: list[str] = []
|
||||||
|
value = case.get("integration_suite")
|
||||||
|
alias = case.get("suite")
|
||||||
|
if value and alias and str(value) != str(alias):
|
||||||
|
errors.append(f"{where}: integration_suite 与 suite 不一致")
|
||||||
|
suite_name = integration_suite(case)
|
||||||
|
if suite_name not in INTEGRATION_SUITES:
|
||||||
|
errors.append(
|
||||||
|
f"{where}: integration_suite 必须是 {', '.join(INTEGRATION_SUITES)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
fixture = case.get("fixture")
|
||||||
|
if not isinstance(fixture, Mapping) or not fixture:
|
||||||
|
errors.append(f"{where}: fixture 缺失,必须在 preflight 前声明")
|
||||||
|
|
||||||
|
declared_call = any(
|
||||||
|
key in case and case.get(key) not in (None, "", [], {})
|
||||||
|
for key in ("request", "input", "call", "integration")
|
||||||
|
)
|
||||||
|
how = case.get("how")
|
||||||
|
if not declared_call and not isinstance(how, Mapping):
|
||||||
|
errors.append(f"{where}: 必须声明 request/input/call/integration 或 how")
|
||||||
|
|
||||||
|
# Integration writes need an explicit rollback/cleanup contract. Read-only
|
||||||
|
# probes may state why cleanup is unnecessary, matching test-preflight.
|
||||||
|
risk = str(case.get("risk") or "")
|
||||||
|
cleanup = case.get("cleanup")
|
||||||
|
cleanup_steps = isinstance(cleanup, Mapping) and (
|
||||||
|
bool(cleanup.get("steps")) or bool(str(cleanup.get("run") or "").strip())
|
||||||
|
)
|
||||||
|
cleanup_noop = isinstance(cleanup, Mapping) and cleanup.get("required") is False and bool(
|
||||||
|
str(cleanup.get("reason") or "").strip()
|
||||||
|
)
|
||||||
|
if risk != "readonly" and not (cleanup_steps or cleanup_noop):
|
||||||
|
errors.append(f"{where}: 写入型 integration 必须声明 cleanup/rollback steps/run")
|
||||||
|
elif cleanup is not None and not (cleanup_steps or cleanup_noop):
|
||||||
|
errors.append(f"{where}: cleanup 必须有 steps/run 或 required=false + reason")
|
||||||
|
|
||||||
|
forbidden = []
|
||||||
|
if case.get("client_suite") or case.get("journey") is not None:
|
||||||
|
forbidden.append("client_suite/journey")
|
||||||
|
if case.get("session") or case.get("platform"):
|
||||||
|
forbidden.append("session/platform")
|
||||||
|
if forbidden:
|
||||||
|
errors.append(f"{where}: integration 不应声明 " + ", ".join(forbidden))
|
||||||
|
return errors
|
||||||
|
|
||||||
|
|
||||||
|
def preflight(cases: Iterable[Mapping[str, Any]], sessions: Mapping[str, Any] | None = None) -> dict[str, Any]:
|
||||||
|
"""Return a contract preflight result; no per-case result is created on block."""
|
||||||
|
cases = list(cases)
|
||||||
|
errors: list[str] = []
|
||||||
|
for case in cases:
|
||||||
|
errors.extend(contract_errors(case, sessions=sessions))
|
||||||
|
return {
|
||||||
|
"required": bool(errors or any(client_suite(case) for case in cases)),
|
||||||
|
"status": "blocked" if errors else "ready",
|
||||||
|
"failure_class": "contract" if errors else None,
|
||||||
|
"errors": errors,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def classify_outcome(
|
||||||
|
*,
|
||||||
|
ok: bool,
|
||||||
|
environment_ready: bool = True,
|
||||||
|
phase: str = "journey",
|
||||||
|
) -> str:
|
||||||
|
"""Classify one observed phase using the tester's five result states.
|
||||||
|
|
||||||
|
A valid test account is part of the client journey: a failed login is a
|
||||||
|
product ``fail``. Only an unavailable environment is ``environment-fail``.
|
||||||
|
"""
|
||||||
|
if not environment_ready:
|
||||||
|
return "environment-fail"
|
||||||
|
if not ok:
|
||||||
|
return "fail"
|
||||||
|
return "pass"
|
||||||
|
|
||||||
|
|
||||||
|
def propagate_results(
|
||||||
|
case_ids: Iterable[str],
|
||||||
|
observed: Mapping[str, str],
|
||||||
|
*,
|
||||||
|
skipped: Mapping[str, str] | None = None,
|
||||||
|
continue_on_fail: bool = False,
|
||||||
|
policies: Mapping[str, str] | None = None,
|
||||||
|
blocked_by: Mapping[str, Iterable[str]] | None = None,
|
||||||
|
) -> list[dict[str, str]]:
|
||||||
|
"""Build ordered result rows and mark cases after a blocking failure not-run.
|
||||||
|
|
||||||
|
``skipped`` is intentional selection (risk/lifecycle/assumption); it is
|
||||||
|
never converted to ``not-run``. A product or environment failure stops
|
||||||
|
subsequent cases unless the caller explicitly opts into continuation.
|
||||||
|
"""
|
||||||
|
skipped = skipped or {}
|
||||||
|
policies = policies or {}
|
||||||
|
blocked_by = blocked_by or {}
|
||||||
|
rows: list[dict[str, str]] = []
|
||||||
|
blocking_case = ""
|
||||||
|
for raw_id in case_ids:
|
||||||
|
case_id = str(raw_id)
|
||||||
|
if case_id in skipped:
|
||||||
|
rows.append({"id": case_id, "result": "skipped", "notes": skipped[case_id]})
|
||||||
|
continue
|
||||||
|
if blocking_case:
|
||||||
|
rows.append({
|
||||||
|
"id": case_id,
|
||||||
|
"result": "not-run",
|
||||||
|
"notes": "前置用例失败,未轮到执行",
|
||||||
|
"blocked_by": ",".join(blocked_by.get(case_id) or (blocking_case,)),
|
||||||
|
})
|
||||||
|
continue
|
||||||
|
result = observed.get(case_id, "not-run")
|
||||||
|
if result not in RESULTS:
|
||||||
|
raise ValueError(f"未知结果: {case_id}={result}")
|
||||||
|
rows.append({"id": case_id, "result": result})
|
||||||
|
policy = policies.get(case_id)
|
||||||
|
if result in ("fail", "environment-fail") and not (continue_on_fail or policy == "continue"):
|
||||||
|
blocking_case = case_id
|
||||||
|
return rows
|
||||||
@@ -11,6 +11,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import hashlib
|
import hashlib
|
||||||
|
import importlib.util
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import plistlib
|
import plistlib
|
||||||
@@ -33,6 +34,7 @@ from typing import Any
|
|||||||
|
|
||||||
COMMIT = re.compile(r"^[0-9a-fA-F]{40}$")
|
COMMIT = re.compile(r"^[0-9a-fA-F]{40}$")
|
||||||
PLATFORMS = {"android", "ios"}
|
PLATFORMS = {"android", "ios"}
|
||||||
|
PREFLIGHT_ROOT = Path(__file__).resolve().parents[2] / "test-preflight" / "scripts" / "preflight.py"
|
||||||
|
|
||||||
|
|
||||||
class EnvironmentFailure(RuntimeError):
|
class EnvironmentFailure(RuntimeError):
|
||||||
@@ -52,6 +54,7 @@ class CommandResult:
|
|||||||
@dataclass
|
@dataclass
|
||||||
class RuntimeContext:
|
class RuntimeContext:
|
||||||
root: Path
|
root: Path
|
||||||
|
bindings: dict[str, str] = field(default_factory=dict)
|
||||||
retries: dict[str, int] = field(default_factory=lambda: {"download": 3, "start": 3, "install": 3})
|
retries: dict[str, int] = field(default_factory=lambda: {"download": 3, "start": 3, "install": 3})
|
||||||
evidence: list[dict[str, Any]] = field(default_factory=list)
|
evidence: list[dict[str, Any]] = field(default_factory=list)
|
||||||
|
|
||||||
@@ -124,6 +127,120 @@ def parse_artifacts(values: list[str]) -> dict[str, str]:
|
|||||||
return pairs
|
return pairs
|
||||||
|
|
||||||
|
|
||||||
|
def parse_bindings(values: list[str], requested: list[str]) -> dict[str, str]:
|
||||||
|
"""Resolve one tester binding per requested platform.
|
||||||
|
|
||||||
|
A single legacy ``agent`` mapping cannot safely represent a multi-platform
|
||||||
|
run. Keep the platform distinction in the command contract all the way to
|
||||||
|
agent-runtime instead of treating binding as report metadata.
|
||||||
|
"""
|
||||||
|
pairs: dict[str, str] = {}
|
||||||
|
for raw in values:
|
||||||
|
platform, sep, value = raw.partition("=")
|
||||||
|
platform = platform.strip().lower()
|
||||||
|
value = value.strip()
|
||||||
|
if not sep or platform not in PLATFORMS or not value or not re.fullmatch(r"[a-z][a-z0-9-]*(?:\.[a-z][a-z0-9-]*)+", value):
|
||||||
|
raise ArtifactFailure(f"--binding 格式必须是 android=tester.<name> 或 ios=tester.<name>: {raw}")
|
||||||
|
if platform in pairs:
|
||||||
|
raise ArtifactFailure(f"--binding 重复平台: {platform}")
|
||||||
|
pairs[platform] = value
|
||||||
|
resolved = {platform: pairs.get(platform, f"tester.{platform}") for platform in requested}
|
||||||
|
if len(set(resolved.values())) != len(resolved) and len(resolved) > 1:
|
||||||
|
raise ArtifactFailure("多平台运行不能复用同一个 tester binding;请分别配置 tester.android/tester.ios")
|
||||||
|
return resolved
|
||||||
|
|
||||||
|
|
||||||
|
def load_agent_runtime():
|
||||||
|
candidates = [
|
||||||
|
Path.home() / ".codex" / "skills" / "agent-runtime" / "scripts" / "agent.py",
|
||||||
|
Path.home() / ".pouch" / "skills" / "agent-runtime" / "scripts" / "agent.py",
|
||||||
|
Path(__file__).resolve().parents[2] / "agent-runtime" / "scripts" / "agent.py",
|
||||||
|
]
|
||||||
|
runtime_path = next((item for item in candidates if item.is_file()), None)
|
||||||
|
if runtime_path is None:
|
||||||
|
raise EnvironmentFailure("找不到共享 agent-runtime: " + ", ".join(str(item) for item in candidates))
|
||||||
|
spec = importlib.util.spec_from_file_location("pouch_agent_runtime_for_mobile_bundle", runtime_path)
|
||||||
|
if spec is None or spec.loader is None:
|
||||||
|
raise EnvironmentFailure(f"无法加载共享 agent-runtime: {runtime_path}")
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
def binding_location(root: Path, binding_id: str) -> dict[str, Any]:
|
||||||
|
runtime = load_agent_runtime()
|
||||||
|
try:
|
||||||
|
value = runtime.binding(runtime.load_config(root), binding_id)
|
||||||
|
except (ValueError, OSError) as exc:
|
||||||
|
raise EnvironmentFailure(f"无法解析 {binding_id}: {exc}") from exc
|
||||||
|
if value.get("adapter") != "tester":
|
||||||
|
raise EnvironmentFailure(f"{binding_id} 不是 tester binding")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def agent_runner(root: Path, request: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""Dispatch a platform operation through the resolved tester binding."""
|
||||||
|
runtime = load_agent_runtime()
|
||||||
|
return runtime.execute_runner_request(root, request)
|
||||||
|
|
||||||
|
|
||||||
|
def platform_node(root: Path, binding_id: str) -> str:
|
||||||
|
location = binding_location(root, binding_id)["location"]
|
||||||
|
return "local" if location.get("environment") == "local" else str(location.get("environment"))
|
||||||
|
|
||||||
|
|
||||||
|
def validate_preflight_matrix(root: Path, config_path: Path, bindings: dict[str, str], requested: list[str]) -> None:
|
||||||
|
"""Fail closed when a matrix routes a platform to the wrong node/runner."""
|
||||||
|
try:
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
raw = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
|
||||||
|
except (OSError, ValueError, ImportError) as exc:
|
||||||
|
raise EnvironmentFailure(f"无法读取 preflight matrix: {config_path}: {exc}") from exc
|
||||||
|
rows = raw.get("matrix") if isinstance(raw, dict) else None
|
||||||
|
if not isinstance(rows, list):
|
||||||
|
raise EnvironmentFailure("preflight matrix 缺少 matrix;不能推断平台执行节点")
|
||||||
|
by_platform = {str(row.get("platform", "")).lower(): row for row in rows if isinstance(row, dict)}
|
||||||
|
for platform in requested:
|
||||||
|
expected_binding = bindings[platform]
|
||||||
|
row = by_platform.get(platform)
|
||||||
|
if not row:
|
||||||
|
raise EnvironmentFailure(f"preflight matrix 缺少 {platform} 行;不能创建该平台 session")
|
||||||
|
tester = row.get("tester") if isinstance(row, dict) else None
|
||||||
|
actual_binding = tester.get("binding") if isinstance(tester, dict) else None
|
||||||
|
if actual_binding != expected_binding:
|
||||||
|
raise EnvironmentFailure(f"{platform} preflight binding={actual_binding!r},期望 {expected_binding!r}")
|
||||||
|
expected_node = platform_node(root, expected_binding)
|
||||||
|
actual_node = tester.get("node") if isinstance(tester, dict) else None
|
||||||
|
if expected_node == "local" and actual_node not in {"local", "localhost", "127.0.0.1"}:
|
||||||
|
raise EnvironmentFailure(f"{platform} binding 是 local,但 preflight node={actual_node!r}")
|
||||||
|
runner = tester.get("runner") if isinstance(tester, dict) else None
|
||||||
|
if expected_node != "local" and (not isinstance(runner, dict) or runner.get("kind") != "contract"):
|
||||||
|
raise EnvironmentFailure(f"{platform} 绑定到 {expected_node},必须使用 contract runner;禁止在协调节点执行")
|
||||||
|
|
||||||
|
|
||||||
|
def preflight_remote_artifacts(config_path: Path, requested: list[str]) -> dict[str, str]:
|
||||||
|
"""Return caller-declared artifact paths readable by remote tester nodes."""
|
||||||
|
try:
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
raw = yaml.safe_load(config_path.read_text(encoding="utf-8")) or {}
|
||||||
|
except (OSError, ValueError, ImportError) as exc:
|
||||||
|
raise EnvironmentFailure(f"无法读取 preflight matrix: {config_path}: {exc}") from exc
|
||||||
|
rows = raw.get("matrix") if isinstance(raw, dict) else None
|
||||||
|
if not isinstance(rows, list):
|
||||||
|
raise EnvironmentFailure("preflight matrix 缺少 matrix;不能解析远程产物路径")
|
||||||
|
by_platform = {str(row.get("platform", "")).lower(): row for row in rows if isinstance(row, dict)}
|
||||||
|
paths: dict[str, str] = {}
|
||||||
|
for platform in requested:
|
||||||
|
row = by_platform.get(platform) or {}
|
||||||
|
artifact = row.get("artifact") if isinstance(row, dict) else None
|
||||||
|
remote_path = artifact.get("remote_path") if isinstance(artifact, dict) else None
|
||||||
|
if isinstance(remote_path, str) and remote_path.strip():
|
||||||
|
paths[platform] = remote_path.strip()
|
||||||
|
return paths
|
||||||
|
|
||||||
|
|
||||||
def local_path(locator: str) -> Path | None:
|
def local_path(locator: str) -> Path | None:
|
||||||
parsed = urllib.parse.urlsplit(locator)
|
parsed = urllib.parse.urlsplit(locator)
|
||||||
if parsed.scheme in ("http", "https"):
|
if parsed.scheme in ("http", "https"):
|
||||||
@@ -209,6 +326,10 @@ def android_identity(path: Path, context: RuntimeContext) -> str:
|
|||||||
result = run_command(command, context=context)
|
result = run_command(command, context=context)
|
||||||
if result.code == 0:
|
if result.code == 0:
|
||||||
match = re.search(r"(?:name|application-id)[=:]['\"]?([A-Za-z0-9_.]+)", result.output)
|
match = re.search(r"(?:name|application-id)[=:]['\"]?([A-Za-z0-9_.]+)", result.output)
|
||||||
|
if not match:
|
||||||
|
# apkanalyzer manifest application-id prints the bare package
|
||||||
|
# name, while aapt/aapt2 print a package: name=... record.
|
||||||
|
match = re.fullmatch(r"\s*([A-Za-z0-9_.]+)\s*", result.output)
|
||||||
if match:
|
if match:
|
||||||
return match.group(1)
|
return match.group(1)
|
||||||
raise EnvironmentFailure("APK 身份读取失败;请检查 Android SDK build-tools")
|
raise EnvironmentFailure("APK 身份读取失败;请检查 Android SDK build-tools")
|
||||||
@@ -388,20 +509,71 @@ def write_run(root: Path, record: dict[str, Any]) -> Path:
|
|||||||
return path
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def all_active_result_rows(root: Path, *, blocked_by: str) -> list[dict[str, Any]]:
|
||||||
|
"""Keep a testcase row for every active catalog entry when preflight blocks."""
|
||||||
|
try:
|
||||||
|
from generate_cases_csv import load_cases
|
||||||
|
|
||||||
|
cases = load_cases(root)
|
||||||
|
except (ImportError, OSError, ValueError, SystemExit):
|
||||||
|
cases = []
|
||||||
|
rows: list[dict[str, Any]] = []
|
||||||
|
for case in cases:
|
||||||
|
if case.get("lifecycle", "active") != "active":
|
||||||
|
continue
|
||||||
|
rows.append({
|
||||||
|
"id": str(case.get("id") or ""),
|
||||||
|
"result": "not-run",
|
||||||
|
"notes": "test-preflight 未通过,未创建移动 session",
|
||||||
|
"blocked_by": [blocked_by],
|
||||||
|
})
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def run_test_preflight(
|
||||||
|
config_path: Path,
|
||||||
|
*,
|
||||||
|
mode: str = "verify",
|
||||||
|
yaml_path: Path | None = None,
|
||||||
|
csv_path: Path | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Run the shared preflight contract before any client session work."""
|
||||||
|
if mode not in {"verify", "prepare"}:
|
||||||
|
raise ArtifactFailure("preflight mode 只能是 verify 或 prepare")
|
||||||
|
if not PREFLIGHT_ROOT.is_file():
|
||||||
|
raise EnvironmentFailure(f"找不到 test-preflight skill: {PREFLIGHT_ROOT}")
|
||||||
|
spec = importlib.util.spec_from_file_location("pouch_test_preflight", PREFLIGHT_ROOT)
|
||||||
|
if spec is None or spec.loader is None:
|
||||||
|
raise EnvironmentFailure("无法加载 test-preflight skill")
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
try:
|
||||||
|
config = module.load_config(config_path)
|
||||||
|
evidence = module.run(config, mode)
|
||||||
|
yaml_path = yaml_path or config_path.with_name("preflight.yaml")
|
||||||
|
csv_path = csv_path or config_path.with_name("preflight.csv")
|
||||||
|
return module.write_evidence(evidence, yaml_path, csv_path)
|
||||||
|
except (OSError, ValueError, module.yaml.YAMLError) as exc:
|
||||||
|
return {"status": "blocked", "error": str(exc), "yaml": str(yaml_path or ""), "csv": str(csv_path or "")}
|
||||||
|
|
||||||
|
|
||||||
def execute(args: argparse.Namespace) -> int:
|
def execute(args: argparse.Namespace) -> int:
|
||||||
requested = platforms(args.platform)
|
requested = platforms(args.platform)
|
||||||
context = RuntimeContext(Path(args.root).resolve())
|
root = Path(args.root).resolve()
|
||||||
|
bindings = parse_bindings(args.bindings, requested)
|
||||||
|
context = RuntimeContext(root, bindings=bindings)
|
||||||
record: dict[str, Any] = {
|
record: dict[str, Any] = {
|
||||||
"schema_version": 1,
|
"schema_version": 1,
|
||||||
"started_at": datetime.now(timezone.utc).isoformat(),
|
"started_at": datetime.now(timezone.utc).isoformat(),
|
||||||
"platforms_requested": requested,
|
"platforms_requested": requested,
|
||||||
"partial": bool(args.partial),
|
"partial": bool(args.partial),
|
||||||
"binding": args.binding,
|
"bindings": bindings,
|
||||||
"node": os.environ.get("TESTER_NODE", "local"),
|
"node": os.environ.get("TESTER_NODE", "local"),
|
||||||
"terminal": {"handle": os.environ.get("ORCA_TERMINAL_HANDLE"), "created": os.environ.get("TESTER_TERMINAL_CREATED", "false").lower() == "true", "closed": os.environ.get("TESTER_TERMINAL_CLOSED", "unknown")},
|
"terminal": {"handle": os.environ.get("ORCA_TERMINAL_HANDLE"), "created": os.environ.get("TESTER_TERMINAL_CREATED", "false").lower() == "true", "closed": os.environ.get("TESTER_TERMINAL_CLOSED", "unknown")},
|
||||||
"product_test": {"status": "not-run", "failure_class": "automatic-repair-failed", "note": "准备完成后由 tester catalog 驱动真实移动 session;本工具不以 API 替代移动测试"},
|
"product_test": {"status": "not-run", "failure_class": "automatic-repair-failed", "note": "准备完成后由 tester catalog 驱动真实移动 session;本工具不以 API 替代移动测试"},
|
||||||
"platforms": {},
|
"platforms": {},
|
||||||
}
|
}
|
||||||
|
remote_artifacts: dict[str, str] = {}
|
||||||
try:
|
try:
|
||||||
locators = parse_artifacts(args.artifacts)
|
locators = parse_artifacts(args.artifacts)
|
||||||
missing = [platform for platform in requested if platform not in locators]
|
missing = [platform for platform in requested if platform not in locators]
|
||||||
@@ -417,7 +589,7 @@ def execute(args: argparse.Namespace) -> int:
|
|||||||
artifact_dir = Path(tempfile.mkdtemp(prefix="pouch-tester-artifacts-"))
|
artifact_dir = Path(tempfile.mkdtemp(prefix="pouch-tester-artifacts-"))
|
||||||
record["artifact_dir"] = str(artifact_dir)
|
record["artifact_dir"] = str(artifact_dir)
|
||||||
expected = catalog_identities(context.root)
|
expected = catalog_identities(context.root)
|
||||||
prepared: dict[str, tuple[dict[str, Any], dict[str, str], str]] = {}
|
prepared: dict[str, tuple[dict[str, Any], dict[str, str] | None, str]] = {}
|
||||||
failures: dict[str, str] = {}
|
failures: dict[str, str] = {}
|
||||||
for platform in requested:
|
for platform in requested:
|
||||||
result: dict[str, Any] = {"node": "tester.android" if platform == "android" else "tester.ios", "status": "preparing"}
|
result: dict[str, Any] = {"node": "tester.android" if platform == "android" else "tester.ios", "status": "preparing"}
|
||||||
@@ -429,22 +601,120 @@ def execute(args: argparse.Namespace) -> int:
|
|||||||
wanted = expected.get(platform)
|
wanted = expected.get(platform)
|
||||||
if wanted and identity != wanted:
|
if wanted and identity != wanted:
|
||||||
raise EnvironmentFailure(f"{platform} 身份不匹配: expected={wanted} actual={identity}")
|
raise EnvironmentFailure(f"{platform} 身份不匹配: expected={wanted} actual={identity}")
|
||||||
node = android_preflight(context) if platform == "android" else ios_preflight(context)
|
result.update({"identity": identity, "status": "artifact-ready"})
|
||||||
check_network(context, node)
|
prepared[platform] = (result, None, identity)
|
||||||
result.update({"identity": identity, "node_preflight": node, "status": "preflight-passed"})
|
|
||||||
prepared[platform] = (result, node, identity)
|
|
||||||
except (EnvironmentFailure, ArtifactFailure) as exc:
|
except (EnvironmentFailure, ArtifactFailure) as exc:
|
||||||
failures[platform] = str(exc)
|
failures[platform] = str(exc)
|
||||||
result.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc)})
|
result.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc)})
|
||||||
|
|
||||||
|
# This is the hard gate shared by integration and journey callers.
|
||||||
|
# There is no safe implicit matrix: without a caller-supplied config we
|
||||||
|
# cannot prove HOST, fixtures, tester bindings, or node reachability.
|
||||||
|
preflight_config = Path(args.preflight_config).expanduser() if args.preflight_config else None
|
||||||
|
if not failures:
|
||||||
|
if preflight_config is None:
|
||||||
|
preflight_result = {
|
||||||
|
"status": "blocked",
|
||||||
|
"error": "缺少 --preflight-config;不能跳过 test-preflight",
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
validate_preflight_matrix(context.root, preflight_config, bindings, requested)
|
||||||
|
remote_artifacts = preflight_remote_artifacts(preflight_config, requested)
|
||||||
|
preflight_result = run_test_preflight(
|
||||||
|
preflight_config,
|
||||||
|
mode=args.preflight_mode,
|
||||||
|
yaml_path=Path(args.preflight_yaml).expanduser() if args.preflight_yaml else None,
|
||||||
|
csv_path=Path(args.preflight_csv).expanduser() if args.preflight_csv else None,
|
||||||
|
)
|
||||||
|
record["preflight"] = preflight_result
|
||||||
|
if preflight_result.get("status") != "ready":
|
||||||
|
failures["preflight"] = str(preflight_result.get("error") or preflight_result.get("status"))
|
||||||
|
|
||||||
|
if not failures and args.preflight_mode == "prepare":
|
||||||
|
# test-preflight prepare owns platform probing, installation and
|
||||||
|
# launch; do not install/start the same artifact a second time.
|
||||||
|
for result, _, _ in prepared.values():
|
||||||
|
result.update({"status": "preflight-prepared"})
|
||||||
|
elif not failures:
|
||||||
|
for platform in requested:
|
||||||
|
result, _, identity = prepared[platform]
|
||||||
|
try:
|
||||||
|
binding_id = bindings[platform]
|
||||||
|
node_name = platform_node(context.root, binding_id)
|
||||||
|
probe = agent_runner(context.root, {
|
||||||
|
"operation": "platform_probe",
|
||||||
|
"platform": platform,
|
||||||
|
"node": node_name,
|
||||||
|
"binding": binding_id,
|
||||||
|
})
|
||||||
|
if probe.get("status") != "ready":
|
||||||
|
raise EnvironmentFailure(str(probe.get("observed") or probe))
|
||||||
|
host = os.environ.get("HOST", "").strip()
|
||||||
|
network = agent_runner(context.root, {
|
||||||
|
"operation": "network_probe",
|
||||||
|
"platform": platform,
|
||||||
|
"node": node_name,
|
||||||
|
"binding": binding_id,
|
||||||
|
"url": f"http://{host}/v1/me",
|
||||||
|
}) if host else {"status": "blocked", "observed": "HOST 未设置"}
|
||||||
|
if network.get("status") != "ready":
|
||||||
|
raise EnvironmentFailure(str(network.get("observed") or network))
|
||||||
|
result.update({"node_preflight": probe, "network_preflight": network, "status": "preflight-passed"})
|
||||||
|
prepared[platform] = (result, None, identity)
|
||||||
|
except EnvironmentFailure as exc:
|
||||||
|
failures[platform] = str(exc)
|
||||||
|
result.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc)})
|
||||||
if failures and not args.partial:
|
if failures and not args.partial:
|
||||||
for platform in requested:
|
for platform in requested:
|
||||||
if platform not in failures:
|
if platform not in failures:
|
||||||
record["platforms"][platform].update({"status": "not-run", "reason": "全量 preflight 闸门被另一平台阻断"})
|
record["platforms"][platform].update({"status": "not-run", "reason": "全量 preflight 闸门被另一平台阻断"})
|
||||||
record.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "next_action": "修复列出的平台环境后重跑;默认不会静默降级"})
|
overall = record.get("preflight", {}).get("status") if "preflight" in record else "environment-fail"
|
||||||
|
record.update({"status": overall if overall in {"blocked", "environment-fail"} else "environment-fail", "failure_class": "test-preflight" if "preflight" in record else "automatic-repair-failed", "next_action": "修复列出的平台环境后重跑;默认不会静默降级"})
|
||||||
|
record["results"] = all_active_result_rows(context.root, blocked_by="test-preflight")
|
||||||
else:
|
else:
|
||||||
for platform, (result, node, identity) in prepared.items():
|
for platform, (result, node, identity) in prepared.items():
|
||||||
try:
|
try:
|
||||||
installed = install_android(node, Path(result["artifact"]["path"]), identity, context) if platform == "android" else install_ios(node, Path(result["artifact"]["path"]), identity, context)
|
if args.preflight_mode == "prepare":
|
||||||
|
# test-preflight already ran platform_probe, network_probe,
|
||||||
|
# install and start through each matrix binding.
|
||||||
|
installed = {"delegated": "test-preflight.prepare", "binding": bindings[platform]}
|
||||||
|
else:
|
||||||
|
binding_id = bindings[platform]
|
||||||
|
node_name = platform_node(context.root, binding_id)
|
||||||
|
probe = agent_runner(context.root, {
|
||||||
|
"operation": "platform_probe",
|
||||||
|
"platform": platform,
|
||||||
|
"node": node_name,
|
||||||
|
"binding": binding_id,
|
||||||
|
})
|
||||||
|
if probe.get("status") != "ready":
|
||||||
|
raise EnvironmentFailure(str(probe.get("observed") or probe))
|
||||||
|
host = os.environ.get("HOST", "").strip()
|
||||||
|
if host:
|
||||||
|
network = agent_runner(context.root, {
|
||||||
|
"operation": "network_probe",
|
||||||
|
"platform": platform,
|
||||||
|
"node": node_name,
|
||||||
|
"binding": binding_id,
|
||||||
|
"url": f"http://{host}/v1/me",
|
||||||
|
})
|
||||||
|
if network.get("status") != "ready":
|
||||||
|
raise EnvironmentFailure(str(network.get("observed") or network))
|
||||||
|
artifact_path = result["artifact"]["path"]
|
||||||
|
if node_name != "local":
|
||||||
|
artifact_path = remote_artifacts.get(platform, "")
|
||||||
|
if not artifact_path:
|
||||||
|
raise EnvironmentFailure(f"远程 {platform} 安装必须在 matrix.artifact.remote_path 声明节点可读产物")
|
||||||
|
installed = agent_runner(context.root, {
|
||||||
|
"operation": "install",
|
||||||
|
"platform": platform,
|
||||||
|
"node": node_name,
|
||||||
|
"binding": binding_id,
|
||||||
|
"artifact": artifact_path,
|
||||||
|
"identity": identity,
|
||||||
|
})
|
||||||
|
if installed.get("status") != "ready":
|
||||||
|
raise EnvironmentFailure(str(installed.get("observed") or installed))
|
||||||
result.update({"status": "ready", "install": installed, "failure_class": None})
|
result.update({"status": "ready", "install": installed, "failure_class": None})
|
||||||
except EnvironmentFailure as exc:
|
except EnvironmentFailure as exc:
|
||||||
failures[platform] = str(exc)
|
failures[platform] = str(exc)
|
||||||
@@ -455,7 +725,18 @@ def execute(args: argparse.Namespace) -> int:
|
|||||||
record["evidence"] = context.evidence
|
record["evidence"] = context.evidence
|
||||||
except (EnvironmentFailure, ArtifactFailure) as exc:
|
except (EnvironmentFailure, ArtifactFailure) as exc:
|
||||||
record.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc), "evidence": context.evidence})
|
record.update({"status": "environment-fail", "failure_class": "automatic-repair-failed", "error": str(exc), "evidence": context.evidence})
|
||||||
|
if record.get("status") != "ready" and "results" not in record:
|
||||||
|
record["results"] = all_active_result_rows(context.root, blocked_by="test-preflight")
|
||||||
path = write_run(context.root, record)
|
path = write_run(context.root, record)
|
||||||
|
try:
|
||||||
|
from generate_run_csv import export_run_csv
|
||||||
|
|
||||||
|
record["csv"] = str(export_run_csv(context.root, run_path=path))
|
||||||
|
except (OSError, SystemExit, ValueError):
|
||||||
|
# The YAML run remains the source of truth when a partial/provisioning
|
||||||
|
# root has no catalog cases yet.
|
||||||
|
record["csv"] = ""
|
||||||
|
path.write_text(json.dumps(record, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||||
output = {"status": record["status"], "run": str(path), "platforms": record["platforms"], "worktree": record.get("worktree")}
|
output = {"status": record["status"], "run": str(path), "platforms": record["platforms"], "worktree": record.get("worktree")}
|
||||||
if record.get("error"):
|
if record.get("error"):
|
||||||
output["error"] = record["error"]
|
output["error"] = record["error"]
|
||||||
@@ -475,7 +756,18 @@ def parser() -> argparse.ArgumentParser:
|
|||||||
run.add_argument("--partial", action="store_true")
|
run.add_argument("--partial", action="store_true")
|
||||||
run.add_argument("--root", default=".")
|
run.add_argument("--root", default=".")
|
||||||
run.add_argument("--worktree-root", default="")
|
run.add_argument("--worktree-root", default="")
|
||||||
run.add_argument("--binding", default="")
|
run.add_argument(
|
||||||
|
"--binding",
|
||||||
|
action="append",
|
||||||
|
dest="bindings",
|
||||||
|
default=[],
|
||||||
|
metavar="PLATFORM=TESTER_BINDING",
|
||||||
|
help="每个平台的 tester binding,可重复;默认 tester.<platform>",
|
||||||
|
)
|
||||||
|
run.add_argument("--preflight-config", default="", help="test-preflight matrix YAML(必填,不得跳过)")
|
||||||
|
run.add_argument("--preflight-mode", choices=("verify", "prepare"), default="verify")
|
||||||
|
run.add_argument("--preflight-yaml", default="", help="preflight YAML 证据输出路径")
|
||||||
|
run.add_argument("--preflight-csv", default="", help="preflight CSV 证据输出路径")
|
||||||
run.set_defaults(func=execute)
|
run.set_defaults(func=execute)
|
||||||
return root
|
return root
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,76 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Append structured tester issue evidence to an existing run snapshot."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
from failure_feedback import attach_feedback
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
parser = argparse.ArgumentParser(description="将 tester 失败结构化回流到 run YAML/CSV")
|
||||||
|
parser.add_argument("--run", required=True, type=Path)
|
||||||
|
parser.add_argument("--trigger", choices=("pr", "requirement", "bug"))
|
||||||
|
parser.add_argument("--target", default="")
|
||||||
|
parser.add_argument("--related-issue", default="")
|
||||||
|
parser.add_argument("--dispatcher", default="dispatcher")
|
||||||
|
parser.add_argument("--testcase", default="", help="为一条失败结果补充 testcase issue 字段")
|
||||||
|
parser.add_argument("--issue-type", choices=("product", "test"))
|
||||||
|
parser.add_argument("--expected", default="")
|
||||||
|
parser.add_argument("--observed", default="")
|
||||||
|
parser.add_argument("--artifact-hash", default="")
|
||||||
|
parser.add_argument("--environment", default="")
|
||||||
|
parser.add_argument("--evidence-path", default="")
|
||||||
|
parser.add_argument("--reproduction", default="")
|
||||||
|
parser.add_argument("--next-step", default="")
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
|
data: dict[str, Any] = yaml.safe_load(args.run.read_text(encoding="utf-8")) or {}
|
||||||
|
if args.testcase:
|
||||||
|
rows = data.get("results")
|
||||||
|
if not isinstance(rows, list):
|
||||||
|
raise SystemExit("run 缺少 results 列表")
|
||||||
|
row = next((item for item in rows if isinstance(item, dict) and str(item.get("id")) == args.testcase), None)
|
||||||
|
if row is None:
|
||||||
|
raise SystemExit(f"找不到 testcase: {args.testcase}")
|
||||||
|
row["issue"] = {
|
||||||
|
key: value
|
||||||
|
for key, value in {
|
||||||
|
"issue_type": args.issue_type,
|
||||||
|
"expected": args.expected,
|
||||||
|
"observed": args.observed,
|
||||||
|
"artifact_hash": args.artifact_hash,
|
||||||
|
"environment": args.environment,
|
||||||
|
"evidence_path": args.evidence_path,
|
||||||
|
"reproduction": args.reproduction,
|
||||||
|
"next_step": args.next_step,
|
||||||
|
}.items()
|
||||||
|
if value
|
||||||
|
}
|
||||||
|
output = attach_feedback(
|
||||||
|
data,
|
||||||
|
trigger=args.trigger,
|
||||||
|
target=args.target,
|
||||||
|
related_issue=args.related_issue,
|
||||||
|
dispatcher=args.dispatcher,
|
||||||
|
)
|
||||||
|
args.run.write_text(yaml.safe_dump(output, allow_unicode=True, sort_keys=False), encoding="utf-8")
|
||||||
|
csv_path = ""
|
||||||
|
try:
|
||||||
|
from generate_run_csv import export_run_csv
|
||||||
|
|
||||||
|
csv_path = str(export_run_csv(args.run.parents[3], run_path=args.run))
|
||||||
|
except (ImportError, OSError, SystemExit, ValueError):
|
||||||
|
pass
|
||||||
|
print(f"updated {args.run} defects={len(output.get('defects') or [])} csv={csv_path}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
||||||
@@ -180,9 +180,14 @@ def clean_config(config: dict[str, Any]) -> dict[str, Any]:
|
|||||||
def config_from_binding(root: Path, selector: str) -> dict[str, Any]:
|
def config_from_binding(root: Path, selector: str) -> dict[str, Any]:
|
||||||
"""Materialize a common agent-runtime tester binding for legacy local.yaml."""
|
"""Materialize a common agent-runtime tester binding for legacy local.yaml."""
|
||||||
|
|
||||||
runtime_path = Path(__file__).resolve().parents[3] / "skills" / "agent-runtime" / "scripts" / "agent.py"
|
candidates = [
|
||||||
if not runtime_path.is_file():
|
Path.home() / ".codex" / "skills" / "agent-runtime" / "scripts" / "agent.py",
|
||||||
raise RuntimeError(f"找不到共享 agent-runtime: {runtime_path}")
|
Path.home() / ".pouch" / "skills" / "agent-runtime" / "scripts" / "agent.py",
|
||||||
|
Path(__file__).resolve().parents[3] / "skills" / "agent-runtime" / "scripts" / "agent.py",
|
||||||
|
]
|
||||||
|
runtime_path = next((item for item in candidates if item.is_file()), None)
|
||||||
|
if runtime_path is None:
|
||||||
|
raise RuntimeError("找不到共享 agent-runtime: " + ", ".join(str(item) for item in candidates))
|
||||||
spec = importlib.util.spec_from_file_location("pouch_agent_runtime", runtime_path)
|
spec = importlib.util.spec_from_file_location("pouch_agent_runtime", runtime_path)
|
||||||
if spec is None or spec.loader is None:
|
if spec is None or spec.loader is None:
|
||||||
raise RuntimeError(f"无法加载共享 agent-runtime: {runtime_path}")
|
raise RuntimeError(f"无法加载共享 agent-runtime: {runtime_path}")
|
||||||
|
|||||||
@@ -0,0 +1,196 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Validate a tester CSV/YAML pair and summarize statuses from CSV only."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import sys
|
||||||
|
from collections import Counter
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
try:
|
||||||
|
import yaml
|
||||||
|
except ImportError: # pragma: no cover
|
||||||
|
print("需要 PyYAML: python3 -m pip install pyyaml", file=sys.stderr)
|
||||||
|
raise SystemExit(2)
|
||||||
|
|
||||||
|
|
||||||
|
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
||||||
|
RESULT_SET = frozenset(RESULTS)
|
||||||
|
|
||||||
|
|
||||||
|
class SummaryError(ValueError):
|
||||||
|
"""Raised when the CSV/YAML pair cannot be safely summarized."""
|
||||||
|
|
||||||
|
|
||||||
|
def _format_ids(ids: set[str]) -> str:
|
||||||
|
return ", ".join(sorted(ids)) or "(none)"
|
||||||
|
|
||||||
|
|
||||||
|
def _duplicate_ids(ids: list[str]) -> list[str]:
|
||||||
|
counts = Counter(ids)
|
||||||
|
return sorted(case_id for case_id, count in counts.items() if count > 1)
|
||||||
|
|
||||||
|
|
||||||
|
def load_csv(path: Path) -> list[dict[str, str]]:
|
||||||
|
"""Load CSV rows and reject malformed IDs/results with diagnostics."""
|
||||||
|
try:
|
||||||
|
handle = path.open(encoding="utf-8-sig", newline="")
|
||||||
|
except OSError as exc:
|
||||||
|
raise SummaryError(f"无法读取 CSV {path}: {exc}") from exc
|
||||||
|
|
||||||
|
with handle:
|
||||||
|
reader = csv.DictReader(handle)
|
||||||
|
if not reader.fieldnames:
|
||||||
|
raise SummaryError(f"CSV {path} 缺少表头")
|
||||||
|
missing_fields = [field for field in ("id", "result") if field not in reader.fieldnames]
|
||||||
|
if missing_fields:
|
||||||
|
raise SummaryError(f"CSV {path} 缺少必需列: {', '.join(missing_fields)}")
|
||||||
|
rows = list(reader)
|
||||||
|
|
||||||
|
ids: list[str] = []
|
||||||
|
errors: list[str] = []
|
||||||
|
for line_number, row in enumerate(rows, start=2):
|
||||||
|
case_id = (row.get("id") or "").strip()
|
||||||
|
result = (row.get("result") or "").strip()
|
||||||
|
ids.append(case_id)
|
||||||
|
if not case_id:
|
||||||
|
errors.append(f"CSV 第 {line_number} 行 testcase id 缺失")
|
||||||
|
if not result:
|
||||||
|
errors.append(f"CSV 第 {line_number} 行 result 缺失 (testcase id={case_id or '(missing)'})")
|
||||||
|
elif result not in RESULT_SET:
|
||||||
|
errors.append(
|
||||||
|
f"CSV 第 {line_number} 行 result 未知: {result!r} "
|
||||||
|
f"(testcase id={case_id or '(missing)'}; 允许值={', '.join(RESULTS)})"
|
||||||
|
)
|
||||||
|
|
||||||
|
duplicates = _duplicate_ids([case_id for case_id in ids if case_id])
|
||||||
|
if duplicates:
|
||||||
|
errors.append(f"CSV testcase id 重复: {_format_ids(set(duplicates))}")
|
||||||
|
if errors:
|
||||||
|
raise SummaryError("CSV 校验失败:\n- " + "\n- ".join(errors))
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def load_yaml_results(path: Path) -> list[dict[str, Any]]:
|
||||||
|
"""Load only YAML results structure needed for count/ID validation."""
|
||||||
|
try:
|
||||||
|
data = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||||
|
except OSError as exc:
|
||||||
|
raise SummaryError(f"无法读取 YAML {path}: {exc}") from exc
|
||||||
|
except yaml.YAMLError as exc:
|
||||||
|
raise SummaryError(f"YAML 解析失败 {path}: {exc}") from exc
|
||||||
|
|
||||||
|
if not isinstance(data, dict) or not isinstance(data.get("results"), list):
|
||||||
|
raise SummaryError(f"YAML {path} 缺少 results 列表")
|
||||||
|
|
||||||
|
results = data["results"]
|
||||||
|
ids: list[str] = []
|
||||||
|
errors: list[str] = []
|
||||||
|
for index, item in enumerate(results, start=1):
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
errors.append(f"YAML results[{index}] 不是 mapping")
|
||||||
|
continue
|
||||||
|
case_id = str(item.get("id") or "").strip()
|
||||||
|
ids.append(case_id)
|
||||||
|
if not case_id:
|
||||||
|
errors.append(f"YAML results[{index}] testcase id 缺失")
|
||||||
|
|
||||||
|
duplicates = _duplicate_ids([case_id for case_id in ids if case_id])
|
||||||
|
if duplicates:
|
||||||
|
errors.append(f"YAML testcase id 重复: {_format_ids(set(duplicates))}")
|
||||||
|
if errors:
|
||||||
|
raise SummaryError("YAML 校验失败:\n- " + "\n- ".join(errors))
|
||||||
|
issues = data.get("defects", data.get("test_issues"))
|
||||||
|
if issues is not None:
|
||||||
|
if not isinstance(issues, list):
|
||||||
|
raise SummaryError("YAML defects/test_issues 必须是列表")
|
||||||
|
result_ids = set(ids)
|
||||||
|
issue_errors: list[str] = []
|
||||||
|
for index, issue in enumerate(issues, start=1):
|
||||||
|
if not isinstance(issue, dict):
|
||||||
|
issue_errors.append(f"YAML defects[{index}] 不是 mapping")
|
||||||
|
continue
|
||||||
|
testcase = str(issue.get("testcase") or "")
|
||||||
|
issue_type = str(issue.get("issue_type") or "")
|
||||||
|
if not testcase or testcase not in result_ids:
|
||||||
|
issue_errors.append(f"YAML defects[{index}] testcase 无效: {testcase or '(missing)'}")
|
||||||
|
if issue_type not in {"product", "test"}:
|
||||||
|
issue_errors.append(f"YAML defects[{index}] issue_type 无效: {issue_type or '(missing)'}")
|
||||||
|
for field in ("expected", "observed", "artifact_hash", "environment", "evidence_path", "reproduction", "next_step"):
|
||||||
|
if field not in issue:
|
||||||
|
issue_errors.append(f"YAML defects[{index}] 缺少 {field}")
|
||||||
|
if issue_errors:
|
||||||
|
raise SummaryError("YAML defects 校验失败:\n- " + "\n- ".join(issue_errors))
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def summarize(csv_path: Path, yaml_path: Path) -> tuple[Counter[str], dict[str, int]]:
|
||||||
|
"""Validate the pair and return counts calculated from CSV result values."""
|
||||||
|
rows = load_csv(csv_path)
|
||||||
|
yaml_results = load_yaml_results(yaml_path)
|
||||||
|
csv_ids = [(row.get("id") or "").strip() for row in rows]
|
||||||
|
yaml_ids = [str(item.get("id") or "").strip() for item in yaml_results]
|
||||||
|
csv_id_set = set(csv_ids)
|
||||||
|
yaml_id_set = set(yaml_ids)
|
||||||
|
diagnostics: list[str] = []
|
||||||
|
|
||||||
|
if len(rows) != len(csv_id_set):
|
||||||
|
diagnostics.append(
|
||||||
|
f"CSV 数据行总数与唯一 testcase id 数不一致: rows={len(rows)} unique_ids={len(csv_id_set)}"
|
||||||
|
)
|
||||||
|
if len(rows) != len(yaml_results):
|
||||||
|
diagnostics.append(
|
||||||
|
f"CSV 数据行总数与 YAML results 数量不一致: csv_rows={len(rows)} yaml_results={len(yaml_results)}"
|
||||||
|
)
|
||||||
|
if csv_id_set != yaml_id_set:
|
||||||
|
diagnostics.append(
|
||||||
|
"CSV/YAML testcase id 集合不一致: "
|
||||||
|
f"仅 CSV 有 [{_format_ids(csv_id_set - yaml_id_set)}]; "
|
||||||
|
f"仅 YAML 有 [{_format_ids(yaml_id_set - csv_id_set)}]"
|
||||||
|
)
|
||||||
|
if diagnostics:
|
||||||
|
raise SummaryError("CSV/YAML 汇总校验失败:\n- " + "\n- ".join(diagnostics))
|
||||||
|
|
||||||
|
counts: Counter[str] = Counter((row.get("result") or "").strip() for row in rows)
|
||||||
|
total = sum(counts[name] for name in RESULTS)
|
||||||
|
if total != len(rows):
|
||||||
|
raise SummaryError(
|
||||||
|
"CSV 状态计数和不等于数据行总数: "
|
||||||
|
f"count_sum={total} csv_rows={len(rows)}"
|
||||||
|
)
|
||||||
|
metadata = {
|
||||||
|
"csv_rows": len(rows),
|
||||||
|
"csv_unique_ids": len(csv_id_set),
|
||||||
|
"yaml_results": len(yaml_results),
|
||||||
|
"count_sum": total,
|
||||||
|
}
|
||||||
|
return counts, metadata
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
parser = argparse.ArgumentParser(description="校验 tester CSV/YAML 并从 CSV result 汇总状态")
|
||||||
|
parser.add_argument("--csv", required=True, type=Path, help="运行成绩单 CSV")
|
||||||
|
parser.add_argument("--yaml", required=True, type=Path, help="对应运行快照 YAML")
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
try:
|
||||||
|
counts, metadata = summarize(args.csv, args.yaml)
|
||||||
|
except SummaryError as exc:
|
||||||
|
print(f"汇总失败: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
bits = " ".join(f"{name}={counts[name]}" for name in RESULTS)
|
||||||
|
print(f"summary source=CSV.result {bits} total={metadata['count_sum']}")
|
||||||
|
print(
|
||||||
|
"validated "
|
||||||
|
f"csv_rows={metadata['csv_rows']} "
|
||||||
|
f"csv_unique_ids={metadata['csv_unique_ids']} "
|
||||||
|
f"yaml_results={metadata['yaml_results']}"
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,185 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Shared execution policy for tester's integration-first runs.
|
||||||
|
|
||||||
|
This module intentionally has no browser, mobile, deployment, or API driver.
|
||||||
|
Those adapters belong to the caller. It only controls the phase gate and the
|
||||||
|
stable five-state testcase result propagation so a caller cannot create a
|
||||||
|
journey session before preflight and integration have passed.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable, Iterable, Mapping
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from failure_feedback import collect_issues
|
||||||
|
|
||||||
|
RESULTS = ("pass", "fail", "environment-fail", "skipped", "not-run")
|
||||||
|
PHASES = ("preflight", "integration", "journey")
|
||||||
|
|
||||||
|
|
||||||
|
def classify_integration(*, ok: bool, service_reachable: bool = True) -> str:
|
||||||
|
"""Separate a contract assertion failure from an unavailable service."""
|
||||||
|
if not service_reachable:
|
||||||
|
return "environment-fail"
|
||||||
|
return "pass" if ok else "fail"
|
||||||
|
|
||||||
|
|
||||||
|
def _case_id(case: Mapping[str, Any] | str) -> str:
|
||||||
|
if isinstance(case, Mapping):
|
||||||
|
return str(case.get("id") or "")
|
||||||
|
return str(case)
|
||||||
|
|
||||||
|
|
||||||
|
def _not_run(
|
||||||
|
cases: Iterable[Mapping[str, Any] | str],
|
||||||
|
reason: str,
|
||||||
|
*,
|
||||||
|
blocked_by: Iterable[str] = (),
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
rows: list[dict[str, str]] = []
|
||||||
|
blockers = [str(item) for item in blocked_by if str(item)]
|
||||||
|
for case in cases:
|
||||||
|
row: dict[str, Any] = {"id": _case_id(case), "result": "not-run", "notes": reason}
|
||||||
|
if blockers:
|
||||||
|
row["blocked_by"] = blockers
|
||||||
|
if isinstance(case, Mapping):
|
||||||
|
row["test_type"] = str(case.get("test_type") or "")
|
||||||
|
row["suite"] = str(
|
||||||
|
case.get("client_suite")
|
||||||
|
or case.get("integration_suite")
|
||||||
|
or case.get("suite")
|
||||||
|
or ""
|
||||||
|
)
|
||||||
|
rows.append(row)
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def _dependencies(
|
||||||
|
case: Mapping[str, Any] | str,
|
||||||
|
*,
|
||||||
|
default: Iterable[str] = (),
|
||||||
|
) -> list[str]:
|
||||||
|
if not isinstance(case, Mapping):
|
||||||
|
return [str(item) for item in default]
|
||||||
|
declared = case.get("depends_on", case.get("dependencies"))
|
||||||
|
if declared is None:
|
||||||
|
return [str(item) for item in default]
|
||||||
|
if isinstance(declared, str):
|
||||||
|
return [declared]
|
||||||
|
if isinstance(declared, Iterable):
|
||||||
|
return [str(item) for item in declared if str(item)]
|
||||||
|
raise ValueError(f"depends_on 必须是字符串或列表: {_case_id(case)}")
|
||||||
|
|
||||||
|
|
||||||
|
def _failed_dependencies(dependencies: Iterable[str], results: Mapping[str, Mapping[str, Any]]) -> list[str]:
|
||||||
|
return [
|
||||||
|
dependency
|
||||||
|
for dependency in dependencies
|
||||||
|
if str((results.get(dependency) or {}).get("result"))
|
||||||
|
in {"fail", "environment-fail", "not-run"}
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _issue_structures(results: list[dict[str, Any]]) -> dict[str, list[dict[str, Any]]]:
|
||||||
|
issues = collect_issues(results)
|
||||||
|
return {"defects": issues, "test_issues": [dict(issue) for issue in issues]}
|
||||||
|
|
||||||
|
|
||||||
|
def run_integration_first(
|
||||||
|
*,
|
||||||
|
preflight_status: str,
|
||||||
|
integration_cases: Iterable[Mapping[str, Any] | str],
|
||||||
|
journey_cases: Iterable[Mapping[str, Any] | str],
|
||||||
|
execute_integration: Callable[[Mapping[str, Any] | str], str | Mapping[str, Any]],
|
||||||
|
create_journey_session: Callable[[], Any],
|
||||||
|
execute_journey: Callable[[Any, Mapping[str, Any] | str], str | Mapping[str, Any]],
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Execute integration cases before creating any journey session.
|
||||||
|
|
||||||
|
``preflight_status`` must be the status emitted by test-preflight. A
|
||||||
|
blocked/environment-fail gate records only ``not-run`` testcase rows and
|
||||||
|
never invokes either adapter. An integration failure stops the phase and
|
||||||
|
propagates ``not-run`` to all journey cases.
|
||||||
|
"""
|
||||||
|
integration = list(integration_cases)
|
||||||
|
journey = list(journey_cases)
|
||||||
|
if preflight_status != "ready":
|
||||||
|
reason = f"test-preflight={preflight_status},未创建客户端 session"
|
||||||
|
result_rows = _not_run(integration, reason, blocked_by=("test-preflight",)) + _not_run(journey, reason, blocked_by=("test-preflight",))
|
||||||
|
return {
|
||||||
|
"status": preflight_status,
|
||||||
|
"phase": "preflight",
|
||||||
|
"session_created": False,
|
||||||
|
"results": result_rows,
|
||||||
|
**_issue_structures(result_rows),
|
||||||
|
}
|
||||||
|
|
||||||
|
results: list[dict[str, Any]] = []
|
||||||
|
results_by_id: dict[str, dict[str, Any]] = {}
|
||||||
|
for case in integration:
|
||||||
|
case_id = _case_id(case)
|
||||||
|
dependencies = _dependencies(case)
|
||||||
|
blocked_by = _failed_dependencies(dependencies, results_by_id)
|
||||||
|
if blocked_by:
|
||||||
|
rows = _not_run(
|
||||||
|
[case],
|
||||||
|
"依赖的 integration 失败,未执行",
|
||||||
|
blocked_by=blocked_by,
|
||||||
|
)
|
||||||
|
results.extend(rows)
|
||||||
|
results_by_id[case_id] = rows[0]
|
||||||
|
continue
|
||||||
|
outcome = execute_integration(case)
|
||||||
|
row = dict(outcome) if isinstance(outcome, Mapping) else {"result": outcome}
|
||||||
|
row["id"] = _case_id(case)
|
||||||
|
if isinstance(case, Mapping):
|
||||||
|
row.setdefault("test_type", str(case.get("test_type") or "integration"))
|
||||||
|
row.setdefault("suite", str(case.get("integration_suite") or case.get("suite") or ""))
|
||||||
|
result = str(row.get("result") or "not-run")
|
||||||
|
if result not in RESULTS:
|
||||||
|
raise ValueError(f"未知 integration 结果: {row['id']}={result}")
|
||||||
|
row["result"] = result
|
||||||
|
results.append(row)
|
||||||
|
results_by_id[case_id] = row
|
||||||
|
|
||||||
|
integration_ids = [_case_id(case) for case in integration]
|
||||||
|
has_runnable_journey = any(
|
||||||
|
not _failed_dependencies(_dependencies(case, default=integration_ids), results_by_id)
|
||||||
|
for case in journey
|
||||||
|
)
|
||||||
|
session = create_journey_session() if has_runnable_journey else None
|
||||||
|
for case in journey:
|
||||||
|
dependencies = _dependencies(case, default=integration_ids)
|
||||||
|
blocked_by = _failed_dependencies(dependencies, results_by_id)
|
||||||
|
if blocked_by:
|
||||||
|
row = _not_run(
|
||||||
|
[case],
|
||||||
|
"依赖的用例失败,journey 未执行",
|
||||||
|
blocked_by=blocked_by,
|
||||||
|
)[0]
|
||||||
|
results.append(row)
|
||||||
|
results_by_id[_case_id(case)] = row
|
||||||
|
continue
|
||||||
|
outcome = execute_journey(session, case)
|
||||||
|
row = dict(outcome) if isinstance(outcome, Mapping) else {"result": outcome}
|
||||||
|
row["id"] = _case_id(case)
|
||||||
|
if isinstance(case, Mapping):
|
||||||
|
row.setdefault("test_type", str(case.get("test_type") or "journey"))
|
||||||
|
row.setdefault("suite", str(case.get("client_suite") or case.get("suite") or ""))
|
||||||
|
result = str(row.get("result") or "not-run")
|
||||||
|
if result not in RESULTS:
|
||||||
|
raise ValueError(f"未知 journey 结果: {row['id']}={result}")
|
||||||
|
row["result"] = result
|
||||||
|
results.append(row)
|
||||||
|
results_by_id[_case_id(case)] = row
|
||||||
|
status = "fail" if any(row["result"] == "fail" for row in results) else (
|
||||||
|
"environment-fail" if any(row["result"] == "environment-fail" for row in results) else "pass"
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"status": status,
|
||||||
|
"phase": "journey" if journey else "integration",
|
||||||
|
"session_created": has_runnable_journey,
|
||||||
|
"results": results,
|
||||||
|
**_issue_structures(results),
|
||||||
|
}
|
||||||
@@ -22,6 +22,24 @@ risks:
|
|||||||
- mutating
|
- mutating
|
||||||
- destructive
|
- destructive
|
||||||
default_max_risk: readonly
|
default_max_risk: readonly
|
||||||
|
# standard enforces the default_max_risk/authorize policy. Writes-style isolated
|
||||||
|
# resettable environments may set this to isolated-full; risk values remain on
|
||||||
|
# cases and are still reported.
|
||||||
|
execution_profile: standard
|
||||||
|
client_suites:
|
||||||
|
client-smoke:
|
||||||
|
default: true
|
||||||
|
client-regression:
|
||||||
|
default: false
|
||||||
|
safety:
|
||||||
|
default: false
|
||||||
|
integration_suites:
|
||||||
|
api-contract:
|
||||||
|
default: false
|
||||||
|
authorization:
|
||||||
|
default: false
|
||||||
|
build-contract:
|
||||||
|
default: false
|
||||||
vars: {}
|
vars: {}
|
||||||
authorize:
|
authorize:
|
||||||
mutating: AUTHORIZE_MUTATING
|
mutating: AUTHORIZE_MUTATING
|
||||||
|
|||||||
@@ -0,0 +1,104 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
CATALOG = ROOT / "scripts" / "catalog.py"
|
||||||
|
|
||||||
|
|
||||||
|
def write_fixture(root: Path, *, complete: bool = True) -> None:
|
||||||
|
tester = root / ".pouch" / "tester"
|
||||||
|
(tester / "cases").mkdir(parents=True)
|
||||||
|
(tester / "runs").mkdir()
|
||||||
|
(tester / "catalog.yaml").write_text(
|
||||||
|
yaml.safe_dump(
|
||||||
|
{
|
||||||
|
"version": 2,
|
||||||
|
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
|
||||||
|
"categories": [{"id": "CLIENT"}],
|
||||||
|
"risks": ["readonly", "mutating", "destructive"],
|
||||||
|
"default_max_risk": "readonly",
|
||||||
|
"client_suites": {
|
||||||
|
"client-smoke": {"default": True},
|
||||||
|
"client-regression": {"default": False},
|
||||||
|
"safety": {"default": False},
|
||||||
|
},
|
||||||
|
"sessions": {"browser": {"kind": "browser", "reuse": "run"}},
|
||||||
|
},
|
||||||
|
sort_keys=False,
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
case = {
|
||||||
|
"id": "TC-CLIENT-001",
|
||||||
|
"category": "CLIENT",
|
||||||
|
"title": "journey",
|
||||||
|
"what": "client journey",
|
||||||
|
"lifecycle": "active",
|
||||||
|
"tag": "smoke",
|
||||||
|
"risk": "readonly",
|
||||||
|
"driver": "browser",
|
||||||
|
"session": "browser",
|
||||||
|
"client_suite": "client-smoke",
|
||||||
|
"platform": ["android", "ios"],
|
||||||
|
"journey": {
|
||||||
|
"goal": "goal",
|
||||||
|
"entry_state": "logged_out",
|
||||||
|
"actions": ["start", "login"],
|
||||||
|
"assertions": ["visible"],
|
||||||
|
"final_state": "done",
|
||||||
|
"failure_policy": "stop",
|
||||||
|
},
|
||||||
|
"fixture": {"account": "test-role"},
|
||||||
|
"cleanup": {"steps": ["cleanup"]},
|
||||||
|
"how": {"steps": ["start", "login"]},
|
||||||
|
"expected": [{"kind": "visible-text", "value": "done"}],
|
||||||
|
}
|
||||||
|
if not complete:
|
||||||
|
case.pop("fixture")
|
||||||
|
case.pop("cleanup")
|
||||||
|
(tester / "cases" / "client.yaml").write_text(
|
||||||
|
yaml.safe_dump({"category": "CLIENT", "cases": [case]}, sort_keys=False),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class CatalogJourneyTests(unittest.TestCase):
|
||||||
|
def run_catalog(self, root: Path, *args: str):
|
||||||
|
return subprocess.run(
|
||||||
|
[sys.executable, str(CATALOG), *args, "--root", str(root)],
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_valid_client_catalog_and_suite_filter(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
write_fixture(root)
|
||||||
|
valid = self.run_catalog(root, "validate")
|
||||||
|
self.assertEqual(valid.returncode, 0, valid.stderr)
|
||||||
|
selected = self.run_catalog(root, "select", "--suite", "client-smoke")
|
||||||
|
self.assertEqual(selected.returncode, 0, selected.stderr)
|
||||||
|
self.assertEqual(json.loads(selected.stdout)["count"], 1)
|
||||||
|
|
||||||
|
def test_missing_fixture_or_cleanup_blocks_before_results(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
write_fixture(root, complete=False)
|
||||||
|
selected = self.run_catalog(root, "select", "--suite", "client-smoke")
|
||||||
|
self.assertNotEqual(selected.returncode, 0)
|
||||||
|
payload = json.loads(selected.stdout)
|
||||||
|
self.assertEqual(payload["preflight"]["failure_class"], "contract")
|
||||||
|
self.assertTrue(any("fixture 缺失" in error for error in payload["preflight"]["errors"]))
|
||||||
|
self.assertEqual(payload["preflight"]["status"], "blocked")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
import sys
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
|
||||||
|
from journey import classify_outcome, contract_errors, propagate_results
|
||||||
|
|
||||||
|
|
||||||
|
def valid_case(**overrides):
|
||||||
|
case = {
|
||||||
|
"id": "TC-CLIENT-001",
|
||||||
|
"client_suite": "client-smoke",
|
||||||
|
"driver": "browser",
|
||||||
|
"session": "browser",
|
||||||
|
"platform": ["android", "ios"],
|
||||||
|
"journey": {
|
||||||
|
"goal": "完成客户端目标",
|
||||||
|
"entry_state": "logged_out",
|
||||||
|
"actions": ["启动", "登录", "完成目标"],
|
||||||
|
"assertions": ["看到目标页面"],
|
||||||
|
"final_state": "目标已完成",
|
||||||
|
"failure_policy": "stop",
|
||||||
|
},
|
||||||
|
"fixture": {"account": "test-role", "data": []},
|
||||||
|
"cleanup": {"steps": ["清理本轮数据"]},
|
||||||
|
}
|
||||||
|
case.update(overrides)
|
||||||
|
return case
|
||||||
|
|
||||||
|
|
||||||
|
class JourneyContractTests(unittest.TestCase):
|
||||||
|
def test_contract_accepts_shared_client_shape(self):
|
||||||
|
self.assertEqual(contract_errors(valid_case(), {"browser": {}}), [])
|
||||||
|
|
||||||
|
def test_missing_fixture_and_cleanup_are_contract_errors(self):
|
||||||
|
errors = contract_errors(
|
||||||
|
valid_case(fixture=None, cleanup=None),
|
||||||
|
{"browser": {}},
|
||||||
|
)
|
||||||
|
self.assertTrue(any("fixture 缺失" in error for error in errors))
|
||||||
|
self.assertTrue(any("cleanup 缺失" in error for error in errors))
|
||||||
|
|
||||||
|
def test_login_failure_is_product_failure_but_unavailable_environment_is_not(self):
|
||||||
|
self.assertEqual(classify_outcome(ok=False, phase="login"), "fail")
|
||||||
|
self.assertEqual(
|
||||||
|
classify_outcome(ok=False, phase="login", environment_ready=False),
|
||||||
|
"environment-fail",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_failed_case_blocks_remaining_cases_and_keeps_explicit_skip(self):
|
||||||
|
rows = propagate_results(
|
||||||
|
["TC-1", "TC-2", "TC-3"],
|
||||||
|
{"TC-1": "fail", "TC-2": "pass", "TC-3": "pass"},
|
||||||
|
skipped={"TC-3": "未授权"},
|
||||||
|
)
|
||||||
|
self.assertEqual([row["result"] for row in rows], ["fail", "not-run", "skipped"])
|
||||||
|
|
||||||
|
def test_continue_policy_allows_next_case(self):
|
||||||
|
rows = propagate_results(
|
||||||
|
["TC-1", "TC-2"],
|
||||||
|
{"TC-1": "fail", "TC-2": "pass"},
|
||||||
|
policies={"TC-1": "continue"},
|
||||||
|
)
|
||||||
|
self.assertEqual([row["result"] for row in rows], ["fail", "pass"])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -50,6 +50,84 @@ class MobileBundleTests(unittest.TestCase):
|
|||||||
with self.assertRaises(bundle.ArtifactFailure):
|
with self.assertRaises(bundle.ArtifactFailure):
|
||||||
bundle.parse_artifacts(["android=/tmp/a.apk", "android=/tmp/b.apk"])
|
bundle.parse_artifacts(["android=/tmp/a.apk", "android=/tmp/b.apk"])
|
||||||
|
|
||||||
|
def test_parse_bindings_keeps_platforms_separate(self):
|
||||||
|
self.assertEqual(
|
||||||
|
bundle.parse_bindings(["android=tester.android", "ios=tester.ios"], ["android", "ios"]),
|
||||||
|
{"android": "tester.android", "ios": "tester.ios"},
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
bundle.parse_bindings([], ["android"]),
|
||||||
|
{"android": "tester.android"},
|
||||||
|
)
|
||||||
|
with self.assertRaises(bundle.ArtifactFailure):
|
||||||
|
bundle.parse_bindings(["android=tester.test", "ios=tester.test"], ["android", "ios"])
|
||||||
|
|
||||||
|
def test_preflight_matrix_rejects_remote_platform_using_local_runner(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
agent_dir = root / ".pouch"
|
||||||
|
agent_dir.mkdir()
|
||||||
|
(agent_dir / "agent.yaml").write_text(
|
||||||
|
"""
|
||||||
|
schemaVersion: 1
|
||||||
|
locations:
|
||||||
|
local-app:
|
||||||
|
environment: local
|
||||||
|
host: local
|
||||||
|
project: test
|
||||||
|
path: /tmp/test
|
||||||
|
remote-ios:
|
||||||
|
environment: macmini
|
||||||
|
host: runtime:12345678-1234-1234-1234-123456789012
|
||||||
|
project: test
|
||||||
|
path: /tmp/test-ios
|
||||||
|
profiles:
|
||||||
|
test-agent:
|
||||||
|
role: test
|
||||||
|
cli: codex
|
||||||
|
tier: standard
|
||||||
|
model: gpt-5
|
||||||
|
reasoningEffort: medium
|
||||||
|
permissionMode: workspace-write
|
||||||
|
bindings:
|
||||||
|
tester.android: {adapter: tester, location: local-app, profile: test-agent}
|
||||||
|
tester.ios: {adapter: tester, location: remote-ios, profile: test-agent}
|
||||||
|
""".strip()
|
||||||
|
+ "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
config = root / "matrix.yaml"
|
||||||
|
config.write_text(
|
||||||
|
"""
|
||||||
|
matrix:
|
||||||
|
- platform: android
|
||||||
|
tester: {binding: tester.android, node: local, runner: {kind: local}}
|
||||||
|
- platform: ios
|
||||||
|
tester: {binding: tester.ios, node: local, runner: {kind: local}}
|
||||||
|
""".strip()
|
||||||
|
+ "\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
with self.assertRaisesRegex(bundle.EnvironmentFailure, "ios.*contract runner"):
|
||||||
|
bundle.validate_preflight_matrix(
|
||||||
|
root,
|
||||||
|
config,
|
||||||
|
{"android": "tester.android", "ios": "tester.ios"},
|
||||||
|
["android", "ios"],
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_remote_artifact_path_comes_from_preflight_matrix(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
config = Path(temp) / "matrix.yaml"
|
||||||
|
config.write_text(
|
||||||
|
"matrix:\n - platform: ios\n artifact:\n remote_path: /Users/tester/app.ipa\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
bundle.preflight_remote_artifacts(config, ["ios"]),
|
||||||
|
{"ios": "/Users/tester/app.ipa"},
|
||||||
|
)
|
||||||
|
|
||||||
def test_new_worktree_each_run(self):
|
def test_new_worktree_each_run(self):
|
||||||
with tempfile.TemporaryDirectory() as temp:
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
repo, worktrees = Path(temp) / "repo", Path(temp) / "worktrees"
|
repo, worktrees = Path(temp) / "repo", Path(temp) / "worktrees"
|
||||||
@@ -80,6 +158,16 @@ class MobileBundleTests(unittest.TestCase):
|
|||||||
self.assertTrue(result["verified"])
|
self.assertTrue(result["verified"])
|
||||||
self.assertTrue(any("install" in call for call in calls))
|
self.assertTrue(any("install" in call for call in calls))
|
||||||
|
|
||||||
|
def test_android_identity_accepts_bare_apkanalyzer_output(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
apk = Path(temp) / "app.apk"
|
||||||
|
apk.write_bytes(b"apk")
|
||||||
|
context = bundle.RuntimeContext(Path(temp))
|
||||||
|
with mock.patch.object(bundle.shutil, "which", side_effect=lambda name: "apkanalyzer" if name == "apkanalyzer" else None), mock.patch.object(
|
||||||
|
bundle, "run_command", return_value=bundle.CommandResult(0, "page.writes\n")
|
||||||
|
):
|
||||||
|
self.assertEqual(bundle.android_identity(apk, context), "page.writes")
|
||||||
|
|
||||||
def test_android_missing_device_is_environment_failure(self):
|
def test_android_missing_device_is_environment_failure(self):
|
||||||
context = bundle.RuntimeContext(Path("."))
|
context = bundle.RuntimeContext(Path("."))
|
||||||
with mock.patch.object(bundle.shutil, "which", return_value="/sdk/adb"), mock.patch.object(
|
with mock.patch.object(bundle.shutil, "which", return_value="/sdk/adb"), mock.patch.object(
|
||||||
|
|||||||
@@ -0,0 +1,129 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
import csv
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
|
||||||
|
import summarize_run
|
||||||
|
|
||||||
|
|
||||||
|
class RunSummaryTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.root = Path(self.temp.name)
|
||||||
|
self.csv_path = self.root / "run.csv"
|
||||||
|
self.yaml_path = self.root / "run.yaml"
|
||||||
|
self.rows = [
|
||||||
|
{"id": "TC-001", "result": "pass"},
|
||||||
|
{"id": "TC-002", "result": "fail"},
|
||||||
|
{"id": "TC-003", "result": "environment-fail"},
|
||||||
|
{"id": "TC-004", "result": "skipped"},
|
||||||
|
{"id": "TC-005", "result": "not-run"},
|
||||||
|
]
|
||||||
|
self.write_csv(self.rows)
|
||||||
|
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows])
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.temp.cleanup()
|
||||||
|
|
||||||
|
def write_csv(self, rows):
|
||||||
|
with self.csv_path.open("w", encoding="utf-8", newline="") as handle:
|
||||||
|
writer = csv.DictWriter(handle, fieldnames=["id", "result"])
|
||||||
|
writer.writeheader()
|
||||||
|
writer.writerows(rows)
|
||||||
|
|
||||||
|
def write_yaml(self, results):
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
self.yaml_path.write_text(
|
||||||
|
yaml.safe_dump(
|
||||||
|
{"counts": {"pass": 999}, "results": results},
|
||||||
|
sort_keys=False,
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
def run_cli(self):
|
||||||
|
return subprocess.run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
str(Path(summarize_run.__file__).resolve()),
|
||||||
|
"--csv",
|
||||||
|
str(self.csv_path),
|
||||||
|
"--yaml",
|
||||||
|
str(self.yaml_path),
|
||||||
|
],
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_valid_summary_uses_csv_results(self):
|
||||||
|
result = self.run_cli()
|
||||||
|
self.assertEqual(result.returncode, 0, result.stderr)
|
||||||
|
self.assertIn("source=CSV.result", result.stdout)
|
||||||
|
self.assertIn("pass=1 fail=1 environment-fail=1 skipped=1 not-run=1 total=5", result.stdout)
|
||||||
|
|
||||||
|
def test_yaml_counts_do_not_drive_status_summary(self):
|
||||||
|
self.write_csv([{"id": "TC-001", "result": "fail"}])
|
||||||
|
self.write_yaml([{"id": "TC-001", "result": "fail"}])
|
||||||
|
result = self.run_cli()
|
||||||
|
self.assertEqual(result.returncode, 0, result.stderr)
|
||||||
|
self.assertIn("pass=0 fail=1", result.stdout)
|
||||||
|
self.assertNotIn("pass=999", result.stdout)
|
||||||
|
|
||||||
|
def test_yaml_result_count_mismatch_is_nonzero(self):
|
||||||
|
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows[:-1]])
|
||||||
|
result = self.run_cli()
|
||||||
|
self.assertNotEqual(result.returncode, 0)
|
||||||
|
self.assertIn("YAML results 数量不一致", result.stderr)
|
||||||
|
|
||||||
|
def test_id_set_mismatch_is_nonzero(self):
|
||||||
|
self.write_yaml([{"id": row["id"], "result": row["result"]} for row in self.rows[:-1]] + [{"id": "TC-999"}])
|
||||||
|
result = self.run_cli()
|
||||||
|
self.assertNotEqual(result.returncode, 0)
|
||||||
|
self.assertIn("testcase id 集合不一致", result.stderr)
|
||||||
|
|
||||||
|
def test_duplicate_csv_id_is_nonzero(self):
|
||||||
|
rows = self.rows.copy()
|
||||||
|
rows[-1] = {"id": "TC-001", "result": "not-run"}
|
||||||
|
self.write_csv(rows)
|
||||||
|
result = self.run_cli()
|
||||||
|
self.assertNotEqual(result.returncode, 0)
|
||||||
|
self.assertIn("CSV testcase id 重复", result.stderr)
|
||||||
|
|
||||||
|
def test_missing_csv_id_is_nonzero(self):
|
||||||
|
rows = self.rows.copy()
|
||||||
|
rows[0] = {"id": "", "result": "pass"}
|
||||||
|
self.write_csv(rows)
|
||||||
|
result = self.run_cli()
|
||||||
|
self.assertNotEqual(result.returncode, 0)
|
||||||
|
self.assertIn("CSV 第 2 行 testcase id 缺失", result.stderr)
|
||||||
|
|
||||||
|
def test_duplicate_or_missing_yaml_id_is_nonzero(self):
|
||||||
|
cases = [
|
||||||
|
([{"id": "TC-001"}, {"id": "TC-001"}] + [{"id": row["id"]} for row in self.rows[2:]], "YAML testcase id 重复"),
|
||||||
|
([{"id": ""}] + [{"id": row["id"]} for row in self.rows[1:]], "YAML results[1] testcase id 缺失"),
|
||||||
|
]
|
||||||
|
for results, expected in cases:
|
||||||
|
with self.subTest(expected=expected):
|
||||||
|
self.write_yaml(results)
|
||||||
|
result = self.run_cli()
|
||||||
|
self.assertNotEqual(result.returncode, 0)
|
||||||
|
self.assertIn(expected, result.stderr)
|
||||||
|
|
||||||
|
def test_unknown_or_missing_csv_result_is_nonzero(self):
|
||||||
|
for bad_result, expected in [("unexpected", "result 未知"), ("", "result 缺失")]:
|
||||||
|
with self.subTest(bad_result=bad_result):
|
||||||
|
rows = self.rows.copy()
|
||||||
|
rows[0] = {"id": "TC-001", "result": bad_result}
|
||||||
|
self.write_csv(rows)
|
||||||
|
result = self.run_cli()
|
||||||
|
self.assertNotEqual(result.returncode, 0)
|
||||||
|
self.assertIn(expected, result.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,299 @@
|
|||||||
|
import csv
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
SCRIPTS = Path(__file__).resolve().parents[1] / "scripts"
|
||||||
|
CATALOG = SCRIPTS / "catalog.py"
|
||||||
|
CASES_CSV = SCRIPTS / "generate_cases_csv.py"
|
||||||
|
RUN_CSV = SCRIPTS / "generate_run_csv.py"
|
||||||
|
sys.path.insert(0, str(SCRIPTS))
|
||||||
|
from two_tier import classify_integration, run_integration_first
|
||||||
|
|
||||||
|
|
||||||
|
def write_mixed_catalog(root: Path) -> None:
|
||||||
|
tester = root / ".pouch" / "tester"
|
||||||
|
(tester / "cases").mkdir(parents=True)
|
||||||
|
(tester / "runs").mkdir()
|
||||||
|
(tester / "catalog.yaml").write_text(
|
||||||
|
yaml.safe_dump(
|
||||||
|
{
|
||||||
|
"version": 2,
|
||||||
|
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
|
||||||
|
"categories": [{"id": "MIXED"}],
|
||||||
|
"risks": ["readonly", "mutating", "destructive"],
|
||||||
|
"default_max_risk": "readonly",
|
||||||
|
"client_suites": {
|
||||||
|
"client-smoke": {"default": True},
|
||||||
|
"client-regression": {"default": False},
|
||||||
|
"safety": {"default": False},
|
||||||
|
},
|
||||||
|
"sessions": {"browser": {"kind": "browser", "reuse": "run"}},
|
||||||
|
},
|
||||||
|
sort_keys=False,
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
journey = {
|
||||||
|
"id": "TC-JOURNEY-001",
|
||||||
|
"test_type": "journey",
|
||||||
|
"category": "MIXED",
|
||||||
|
"title": "journey",
|
||||||
|
"what": "client path",
|
||||||
|
"lifecycle": "active",
|
||||||
|
"tag": "smoke",
|
||||||
|
"risk": "readonly",
|
||||||
|
"driver": "browser",
|
||||||
|
"automation": "automated",
|
||||||
|
"session": "browser",
|
||||||
|
"client_suite": "client-smoke",
|
||||||
|
"platform": "android",
|
||||||
|
"journey": {
|
||||||
|
"goal": "goal",
|
||||||
|
"entry_state": "logged_out",
|
||||||
|
"actions": ["login"],
|
||||||
|
"assertions": ["visible"],
|
||||||
|
"final_state": "done",
|
||||||
|
"failure_policy": "stop",
|
||||||
|
},
|
||||||
|
"fixture": {"account": "test-role"},
|
||||||
|
"cleanup": {"required": False, "reason": "read-only"},
|
||||||
|
"how": {"steps": ["open"]},
|
||||||
|
"expected": [{"kind": "visible-text", "value": "done"}],
|
||||||
|
}
|
||||||
|
integration = {
|
||||||
|
"id": "TC-INTEGRATION-001",
|
||||||
|
"test_type": "integration",
|
||||||
|
"category": "MIXED",
|
||||||
|
"title": "contract",
|
||||||
|
"what": "API contract",
|
||||||
|
"lifecycle": "active",
|
||||||
|
"tag": "slow",
|
||||||
|
"risk": "readonly",
|
||||||
|
"driver": "script",
|
||||||
|
"automation": "automated",
|
||||||
|
"integration_suite": "api-contract",
|
||||||
|
"fixture": {"references": ["isolated-account"]},
|
||||||
|
"request": {"method": "GET", "path": "/health"},
|
||||||
|
"how": {"run": "true"},
|
||||||
|
"cleanup": {"required": False, "reason": "read-only"},
|
||||||
|
"expected": [{"kind": "api-status", "value": 200}],
|
||||||
|
}
|
||||||
|
(tester / "cases" / "mixed.yaml").write_text(
|
||||||
|
yaml.safe_dump({"category": "MIXED", "cases": [journey, integration]}, sort_keys=False),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TwoTierCatalogTests(unittest.TestCase):
|
||||||
|
def run_catalog(self, root: Path, *args: str) -> subprocess.CompletedProcess[str]:
|
||||||
|
return subprocess.run(
|
||||||
|
[sys.executable, str(CATALOG), *args, "--root", str(root)],
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_default_selection_excludes_integration_and_explicit_selection_includes_it(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
write_mixed_catalog(root)
|
||||||
|
valid = self.run_catalog(root, "validate")
|
||||||
|
self.assertEqual(valid.returncode, 0, valid.stderr)
|
||||||
|
|
||||||
|
default = self.run_catalog(root, "select")
|
||||||
|
self.assertEqual(default.returncode, 0, default.stderr)
|
||||||
|
default_payload = json.loads(default.stdout)
|
||||||
|
self.assertEqual(default_payload["count"], 1)
|
||||||
|
self.assertEqual(default_payload["test_types"], {"journey": 1, "integration": 0})
|
||||||
|
|
||||||
|
integration = self.run_catalog(root, "select", "--test-type", "integration")
|
||||||
|
self.assertEqual(integration.returncode, 0, integration.stderr)
|
||||||
|
integration_payload = json.loads(integration.stdout)
|
||||||
|
self.assertEqual(integration_payload["count"], 1)
|
||||||
|
self.assertEqual(integration_payload["cases"][0]["test_type"], "integration")
|
||||||
|
|
||||||
|
csv_result = subprocess.run(
|
||||||
|
[sys.executable, str(CASES_CSV), "--root", str(root)],
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
)
|
||||||
|
self.assertEqual(csv_result.returncode, 0, csv_result.stderr)
|
||||||
|
with (root / ".pouch/tester/cases.csv").open(encoding="utf-8-sig", newline="") as handle:
|
||||||
|
rows = list(csv.DictReader(handle))
|
||||||
|
self.assertEqual({row["test_type"] for row in rows}, {"journey", "integration"})
|
||||||
|
self.assertEqual({row["suite"] for row in rows}, {"client-smoke", "api-contract"})
|
||||||
|
|
||||||
|
run_path = root / ".pouch/tester/runs/two-tier.yaml"
|
||||||
|
run_path.write_text(
|
||||||
|
yaml.safe_dump(
|
||||||
|
{"runId": "two-tier", "results": [
|
||||||
|
{"id": "TC-INTEGRATION-001", "result": "pass"},
|
||||||
|
{"id": "TC-JOURNEY-001", "result": "not-run"},
|
||||||
|
]},
|
||||||
|
sort_keys=False,
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
run_csv = subprocess.run(
|
||||||
|
[sys.executable, str(RUN_CSV), "--root", str(root), "--run", str(run_path)],
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
)
|
||||||
|
self.assertEqual(run_csv.returncode, 0, run_csv.stderr)
|
||||||
|
with run_path.with_suffix(".csv").open(encoding="utf-8-sig", newline="") as handle:
|
||||||
|
report_rows = list(csv.DictReader(handle))
|
||||||
|
self.assertEqual(report_rows[0]["test_type"], "integration")
|
||||||
|
self.assertEqual(report_rows[0]["suite"], "api-contract")
|
||||||
|
|
||||||
|
def test_isolated_full_keeps_risk_and_ignores_authorize_filter(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
write_mixed_catalog(root)
|
||||||
|
catalog_path = root / ".pouch/tester/catalog.yaml"
|
||||||
|
catalog = yaml.safe_load(catalog_path.read_text(encoding="utf-8"))
|
||||||
|
catalog["authorize"] = {"mutating": "AUTHORIZE_MUTATING"}
|
||||||
|
catalog["execution_profile"] = "isolated-full"
|
||||||
|
catalog_path.write_text(yaml.safe_dump(catalog, sort_keys=False), encoding="utf-8")
|
||||||
|
case_path = root / ".pouch/tester/cases/mixed.yaml"
|
||||||
|
cases = yaml.safe_load(case_path.read_text(encoding="utf-8"))
|
||||||
|
journey = cases["cases"][0]
|
||||||
|
journey.update({"risk": "mutating", "tag": "full", "client_suite": "client-regression"})
|
||||||
|
case_path.write_text(yaml.safe_dump(cases, sort_keys=False), encoding="utf-8")
|
||||||
|
|
||||||
|
selected = self.run_catalog(root, "select", "--suite", "client-regression")
|
||||||
|
self.assertEqual(selected.returncode, 0, selected.stderr)
|
||||||
|
payload = json.loads(selected.stdout)
|
||||||
|
self.assertEqual(payload["execution_profile"], "isolated-full")
|
||||||
|
self.assertEqual(payload["risk_filter"], "disabled")
|
||||||
|
self.assertEqual(payload["count"], 1)
|
||||||
|
self.assertEqual(payload["cases"][0]["risk"], "mutating")
|
||||||
|
self.assertEqual(payload["skipped"], [])
|
||||||
|
|
||||||
|
all_active = self.run_catalog(root, "select")
|
||||||
|
self.assertEqual(all_active.returncode, 0, all_active.stderr)
|
||||||
|
all_payload = json.loads(all_active.stdout)
|
||||||
|
self.assertEqual(all_payload["count"], 2)
|
||||||
|
self.assertEqual(all_payload["test_types"], {"journey": 1, "integration": 1})
|
||||||
|
|
||||||
|
def test_isolated_full_run_script_does_not_use_authorize_switch(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
tester = root / ".pouch" / "tester"
|
||||||
|
(tester / "cases").mkdir(parents=True)
|
||||||
|
(tester / "local.yaml").write_text("HOST: http://127.0.0.1\nAUTHORIZE_MUTATING: false\n", encoding="utf-8")
|
||||||
|
(tester / "catalog.yaml").write_text(
|
||||||
|
yaml.safe_dump(
|
||||||
|
{
|
||||||
|
"version": 2,
|
||||||
|
"id_pattern": "^TC-[A-Z]+-[0-9]+$",
|
||||||
|
"categories": [{"id": "SCRIPT"}],
|
||||||
|
"risks": ["readonly", "mutating", "destructive"],
|
||||||
|
"default_max_risk": "readonly",
|
||||||
|
"execution_profile": "isolated-full",
|
||||||
|
"authorize": {"mutating": "AUTHORIZE_MUTATING"},
|
||||||
|
},
|
||||||
|
sort_keys=False,
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(tester / "cases" / "script.yaml").write_text(
|
||||||
|
yaml.safe_dump(
|
||||||
|
{
|
||||||
|
"category": "SCRIPT",
|
||||||
|
"cases": [{
|
||||||
|
"id": "TC-SCRIPT-001",
|
||||||
|
"title": "mutating script",
|
||||||
|
"what": "run isolated script",
|
||||||
|
"lifecycle": "active",
|
||||||
|
"tag": "full",
|
||||||
|
"risk": "mutating",
|
||||||
|
"driver": "script",
|
||||||
|
"how": {"run": "exit 0"},
|
||||||
|
"cleanup": {"steps": ["reset"]},
|
||||||
|
"expected": [{"kind": "script-exit", "value": 0}],
|
||||||
|
}],
|
||||||
|
},
|
||||||
|
sort_keys=False,
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
result = subprocess.run(
|
||||||
|
[sys.executable, str(CATALOG), "run-script", "--root", str(root), "--id", "TC-SCRIPT-001"],
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
)
|
||||||
|
self.assertEqual(result.returncode, 0, result.stderr)
|
||||||
|
self.assertEqual(json.loads(result.stdout)["result"], "executed")
|
||||||
|
|
||||||
|
|
||||||
|
class IntegrationFirstTests(unittest.TestCase):
|
||||||
|
def test_preflight_block_does_not_create_session(self):
|
||||||
|
calls: list[str] = []
|
||||||
|
result = run_integration_first(
|
||||||
|
preflight_status="blocked",
|
||||||
|
integration_cases=["INT-1"],
|
||||||
|
journey_cases=["J-1"],
|
||||||
|
execute_integration=lambda case: calls.append("integration") or "pass",
|
||||||
|
create_journey_session=lambda: calls.append("session"),
|
||||||
|
execute_journey=lambda session, case: "pass",
|
||||||
|
)
|
||||||
|
self.assertEqual(result["results"][0]["result"], "not-run")
|
||||||
|
self.assertFalse(result["session_created"])
|
||||||
|
self.assertEqual(calls, [])
|
||||||
|
|
||||||
|
def test_integration_failure_propagates_journey_not_run(self):
|
||||||
|
calls: list[str] = []
|
||||||
|
result = run_integration_first(
|
||||||
|
preflight_status="ready",
|
||||||
|
integration_cases=["INT-1", "INT-2"],
|
||||||
|
journey_cases=["J-1", "J-2"],
|
||||||
|
execute_integration=lambda case: calls.append(case) or ("fail" if case == "INT-1" else "pass"),
|
||||||
|
create_journey_session=lambda: calls.append("session"),
|
||||||
|
execute_journey=lambda session, case: "pass",
|
||||||
|
)
|
||||||
|
self.assertEqual(result["status"], "fail")
|
||||||
|
self.assertEqual(calls, ["INT-1", "INT-2"])
|
||||||
|
self.assertFalse(result["session_created"])
|
||||||
|
self.assertEqual([row["result"] for row in result["results"]], ["fail", "pass", "not-run", "not-run"])
|
||||||
|
self.assertEqual(result["results"][2]["blocked_by"], ["INT-1"])
|
||||||
|
|
||||||
|
def test_independent_integration_continues_and_dependency_is_blocked(self):
|
||||||
|
result = run_integration_first(
|
||||||
|
preflight_status="ready",
|
||||||
|
integration_cases=[
|
||||||
|
{"id": "INT-1"},
|
||||||
|
{"id": "INT-2"},
|
||||||
|
{"id": "INT-3", "depends_on": ["INT-1"]},
|
||||||
|
],
|
||||||
|
journey_cases=[],
|
||||||
|
execute_integration=lambda case: "fail" if case["id"] == "INT-1" else "pass",
|
||||||
|
create_journey_session=lambda: object(),
|
||||||
|
execute_journey=lambda session, case: "pass",
|
||||||
|
)
|
||||||
|
self.assertEqual([row["result"] for row in result["results"]], ["fail", "pass", "not-run"])
|
||||||
|
self.assertEqual(result["results"][2]["blocked_by"], ["INT-1"])
|
||||||
|
|
||||||
|
def test_preflight_block_creates_every_result_row(self):
|
||||||
|
result = run_integration_first(
|
||||||
|
preflight_status="blocked",
|
||||||
|
integration_cases=[{"id": "INT-1"}],
|
||||||
|
journey_cases=[{"id": "J-1"}],
|
||||||
|
execute_integration=lambda case: "pass",
|
||||||
|
create_journey_session=lambda: object(),
|
||||||
|
execute_journey=lambda session, case: "pass",
|
||||||
|
)
|
||||||
|
self.assertEqual([row["result"] for row in result["results"]], ["not-run", "not-run"])
|
||||||
|
self.assertEqual(result["results"][0]["blocked_by"], ["test-preflight"])
|
||||||
|
|
||||||
|
def test_service_unreachable_is_environment_failure(self):
|
||||||
|
self.assertEqual(classify_integration(ok=False, service_reachable=False), "environment-fail")
|
||||||
|
self.assertEqual(classify_integration(ok=False, service_reachable=True), "fail")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -112,6 +112,9 @@ class BuilderCheckTests(unittest.TestCase):
|
|||||||
self.assertIn("copy_artifact.sh", skill)
|
self.assertIn("copy_artifact.sh", skill)
|
||||||
self.assertIn("android", skill.lower())
|
self.assertIn("android", skill.lower())
|
||||||
self.assertIn("COPY_DEST", contract)
|
self.assertIn("COPY_DEST", contract)
|
||||||
|
self.assertIn("init_ci_status.sh", skill)
|
||||||
|
self.assertIn("CI_STATUS_SCRIPT", contract)
|
||||||
|
self.assertIn("ci-status.md", skill)
|
||||||
|
|
||||||
def test_no_makefile_without_ready_is_usage_error(self) -> None:
|
def test_no_makefile_without_ready_is_usage_error(self) -> None:
|
||||||
with tempfile.TemporaryDirectory() as temp:
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
@@ -213,28 +216,62 @@ class BuilderCheckTests(unittest.TestCase):
|
|||||||
self.assertIn("[FAIL] 10. 轨道工具链", text)
|
self.assertIn("[FAIL] 10. 轨道工具链", text)
|
||||||
self.assertIn("docker: MISSING", text)
|
self.assertIn("docker: MISSING", text)
|
||||||
|
|
||||||
def test_ready_uses_curl_for_gitea_actions(self) -> None:
|
def test_ready_ci_requires_status_script(self) -> None:
|
||||||
with tempfile.TemporaryDirectory() as temp:
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
project = Path(temp)
|
project = Path(temp)
|
||||||
init_repo(project)
|
init_repo(project)
|
||||||
write_contract_makefile(project)
|
write_contract_makefile(project)
|
||||||
(project / "Dockerfile").write_text("FROM scratch\n", encoding="utf-8")
|
(project / "Dockerfile").write_text("FROM scratch\n", encoding="utf-8")
|
||||||
git(project, "remote", "add", "origin", "https://gitea.example.test/acme/widget.git")
|
|
||||||
code, text = run_check(
|
code, text = run_check(
|
||||||
project,
|
project,
|
||||||
"--ready",
|
"--ready",
|
||||||
env={"BUILD_VENUE": "ci"},
|
env={"BUILD_VENUE": "ci"},
|
||||||
which={
|
which={
|
||||||
"docker": "/usr/bin/docker",
|
"docker": "/usr/bin/docker",
|
||||||
"curl": "/usr/bin/curl",
|
"curl": None,
|
||||||
"gh": None,
|
"gh": None,
|
||||||
"glab": None,
|
"glab": None,
|
||||||
|
"tea": None,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
self.assertEqual(code, 1, text)
|
||||||
|
self.assertIn("[FAIL] 15. CI_STATUS_SCRIPT 可执行", text)
|
||||||
|
self.assertIn("does not point to a file", text)
|
||||||
|
self.assertIn("CI status: project CI_STATUS_SCRIPT", text)
|
||||||
|
self.assertNotIn("curl: found", text)
|
||||||
|
self.assertNotIn("Gitea API", text)
|
||||||
|
|
||||||
|
def test_ready_ci_passes_with_executable_status_script(self) -> None:
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
project = Path(temp)
|
||||||
|
init_repo(project)
|
||||||
|
write_contract_makefile(project)
|
||||||
|
(project / "Dockerfile").write_text("FROM scratch\n", encoding="utf-8")
|
||||||
|
script = project / ".pouch" / "builder" / "ci-status.sh"
|
||||||
|
script.parent.mkdir(parents=True)
|
||||||
|
script.write_text("#!/bin/sh\necho STATUS=empty\n", encoding="utf-8")
|
||||||
|
script.chmod(0o755)
|
||||||
|
code, text = run_check(
|
||||||
|
project,
|
||||||
|
"--ready",
|
||||||
|
env={"BUILD_VENUE": "ci"},
|
||||||
|
which={"docker": "/usr/bin/docker", "curl": None, "gh": None, "glab": None},
|
||||||
|
)
|
||||||
self.assertEqual(code, 0, text)
|
self.assertEqual(code, 0, text)
|
||||||
self.assertIn("curl: found", text)
|
self.assertIn("[PASS] 15. CI_STATUS_SCRIPT 可执行", text)
|
||||||
self.assertIn("Gitea API: /api/v1", text)
|
self.assertIn("CI status: project CI_STATUS_SCRIPT", text)
|
||||||
self.assertNotIn("glab: ", text)
|
|
||||||
|
def test_ready_local_skips_status_script(self) -> None:
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
project = Path(temp)
|
||||||
|
init_repo(project)
|
||||||
|
write_contract_makefile(project)
|
||||||
|
code, text = run_check(
|
||||||
|
project, "--ready", which={"dpkg-deb": "/usr/bin/dpkg-deb"}
|
||||||
|
)
|
||||||
|
self.assertEqual(code, 0, text)
|
||||||
|
self.assertIn("[SKIP] 15. CI_STATUS_SCRIPT 可执行", text)
|
||||||
|
self.assertIn("BUILD_VENUE is not ci", text)
|
||||||
|
|
||||||
def test_user_makefile_does_not_satisfy_contract(self) -> None:
|
def test_user_makefile_does_not_satisfy_contract(self) -> None:
|
||||||
with tempfile.TemporaryDirectory() as temp:
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
|||||||
+127
-155
@@ -1,6 +1,5 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
import os
|
||||||
import subprocess
|
import subprocess
|
||||||
import tempfile
|
import tempfile
|
||||||
@@ -9,7 +8,10 @@ from pathlib import Path
|
|||||||
|
|
||||||
|
|
||||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
SCRIPT = REPO_ROOT / "skills" / "builder" / "scripts" / "wait_ci.sh"
|
SCRIPTS = REPO_ROOT / "skills" / "builder" / "scripts"
|
||||||
|
WAIT_CI = SCRIPTS / "wait_ci.sh"
|
||||||
|
INIT_CI = SCRIPTS / "init_ci_status.sh"
|
||||||
|
TEMPLATES = REPO_ROOT / "skills" / "builder" / "templates" / "ci-status"
|
||||||
|
|
||||||
|
|
||||||
def git(cwd: Path, *args: str) -> str:
|
def git(cwd: Path, *args: str) -> str:
|
||||||
@@ -33,7 +35,7 @@ def git(cwd: Path, *args: str) -> str:
|
|||||||
|
|
||||||
|
|
||||||
class BuilderCiTests(unittest.TestCase):
|
class BuilderCiTests(unittest.TestCase):
|
||||||
def make_repo(self, root: Path, origin: str) -> str:
|
def make_repo(self, root: Path, origin: str = "https://gitea.example.test/acme/widget.git") -> str:
|
||||||
git(root, "init", "-b", "main")
|
git(root, "init", "-b", "main")
|
||||||
git(root, "config", "user.email", "test@example.com")
|
git(root, "config", "user.email", "test@example.com")
|
||||||
git(root, "config", "user.name", "Test")
|
git(root, "config", "user.name", "Test")
|
||||||
@@ -43,6 +45,27 @@ class BuilderCiTests(unittest.TestCase):
|
|||||||
git(root, "remote", "add", "origin", origin)
|
git(root, "remote", "add", "origin", origin)
|
||||||
return git(root, "rev-parse", "HEAD")
|
return git(root, "rev-parse", "HEAD")
|
||||||
|
|
||||||
|
def write_makefile(self, root: Path, venue: str = "ci") -> None:
|
||||||
|
(root / "makefile.builder").write_text(
|
||||||
|
"BUILD_VENUE ?= "
|
||||||
|
+ venue
|
||||||
|
+ "\n"
|
||||||
|
"CI_STATUS_SCRIPT ?= .pouch/builder/ci-status.sh\n"
|
||||||
|
"print-%:\n"
|
||||||
|
"\t@echo $($*)\n"
|
||||||
|
"venue:\n"
|
||||||
|
"\t@echo $(BUILD_VENUE)\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
def write_status_script(self, root: Path, body: str, executable: bool = True) -> Path:
|
||||||
|
path = root / ".pouch" / "builder" / "ci-status.sh"
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text("#!/bin/sh\n" + body + "\n", encoding="utf-8")
|
||||||
|
if executable:
|
||||||
|
path.chmod(0o755)
|
||||||
|
return path
|
||||||
|
|
||||||
def fake_command(self, bin_dir: Path, name: str, body: str) -> None:
|
def fake_command(self, bin_dir: Path, name: str, body: str) -> None:
|
||||||
command = bin_dir / name
|
command = bin_dir / name
|
||||||
command.write_text(f"#!/bin/sh\n{body}\n", encoding="utf-8")
|
command.write_text(f"#!/bin/sh\n{body}\n", encoding="utf-8")
|
||||||
@@ -52,20 +75,22 @@ class BuilderCiTests(unittest.TestCase):
|
|||||||
self,
|
self,
|
||||||
root: Path,
|
root: Path,
|
||||||
sha: str,
|
sha: str,
|
||||||
bin_dir: Path,
|
bin_dir: Path | None = None,
|
||||||
extra_env: dict[str, str] | None = None,
|
extra_env: dict[str, str] | None = None,
|
||||||
) -> subprocess.CompletedProcess[str]:
|
) -> subprocess.CompletedProcess[str]:
|
||||||
|
path = os.environ["PATH"]
|
||||||
|
if bin_dir is not None:
|
||||||
|
path = f"{bin_dir}{os.pathsep}{path}"
|
||||||
env = {
|
env = {
|
||||||
**os.environ,
|
**os.environ,
|
||||||
"PATH": f"{bin_dir}{os.pathsep}{os.environ['PATH']}",
|
"PATH": path,
|
||||||
"BUILDER_CI_POLL_SEC": "0",
|
"BUILDER_CI_POLL_SEC": "0",
|
||||||
"BUILDER_CI_APPEAR_SEC": "0",
|
"BUILDER_CI_APPEAR_SEC": "0",
|
||||||
"BUILDER_CI_FORGE": "gitea",
|
|
||||||
}
|
}
|
||||||
if extra_env:
|
if extra_env:
|
||||||
env.update(extra_env)
|
env.update(extra_env)
|
||||||
return subprocess.run(
|
return subprocess.run(
|
||||||
["bash", str(SCRIPT), "--sha", sha, "--root", str(root)],
|
["bash", str(WAIT_CI), "--sha", sha, "--root", str(root)],
|
||||||
cwd=root,
|
cwd=root,
|
||||||
env=env,
|
env=env,
|
||||||
text=True,
|
text=True,
|
||||||
@@ -73,176 +98,123 @@ class BuilderCiTests(unittest.TestCase):
|
|||||||
check=False,
|
check=False,
|
||||||
)
|
)
|
||||||
|
|
||||||
def test_gitea_success_reads_exact_run_and_jobs(self) -> None:
|
def test_init_copies_each_kind_template(self) -> None:
|
||||||
|
for kind in ("tea", "glab", "gh", "curl"):
|
||||||
|
with self.subTest(kind=kind), tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
result = subprocess.run(
|
||||||
|
["bash", str(INIT_CI), "--root", str(root), "--kind", kind],
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
dest = root / ".pouch" / "builder" / "ci-status.sh"
|
||||||
|
self.assertEqual(result.returncode, 0, result.stderr)
|
||||||
|
self.assertEqual(result.stdout.strip(), str(dest))
|
||||||
|
self.assertTrue(dest.is_file())
|
||||||
|
self.assertTrue(os.access(dest, os.X_OK))
|
||||||
|
self.assertEqual(
|
||||||
|
dest.read_text(encoding="utf-8"),
|
||||||
|
(TEMPLATES / f"{kind}.sh").read_text(encoding="utf-8"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_init_rejects_unknown_kind(self) -> None:
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
result = subprocess.run(
|
||||||
|
["bash", str(INIT_CI), "--root", temp, "--kind", "jenkins"],
|
||||||
|
text=True,
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
self.assertEqual(result.returncode, 2)
|
||||||
|
self.assertIn("--kind must be tea, glab, gh, or curl", result.stderr)
|
||||||
|
|
||||||
|
def test_success_reads_project_script_not_forge_cli(self) -> None:
|
||||||
with tempfile.TemporaryDirectory() as temp:
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
root = Path(temp)
|
root = Path(temp)
|
||||||
sha = self.make_repo(root, "https://gitea.example.test/acme/widget.git")
|
sha = self.make_repo(root)
|
||||||
|
self.write_makefile(root)
|
||||||
|
self.write_status_script(
|
||||||
|
root,
|
||||||
|
'echo "STATUS=success ID=12 PIPELINE=https://gitea.example.test/run/12 JOBS=build=success"',
|
||||||
|
)
|
||||||
bin_dir = root / "bin"
|
bin_dir = root / "bin"
|
||||||
bin_dir.mkdir()
|
bin_dir.mkdir()
|
||||||
self.fake_command(
|
for name in ("curl", "tea", "glab", "gh"):
|
||||||
bin_dir,
|
self.fake_command(bin_dir, name, "exit 9")
|
||||||
"curl",
|
result = self.run_wait(root, sha, bin_dir)
|
||||||
"""
|
|
||||||
case "$*" in
|
|
||||||
*"/jobs?"*) cat "$BUILDER_CI_FIXTURE/jobs.json" ;;
|
|
||||||
*"/actions/runs?"*) cat "$BUILDER_CI_FIXTURE/runs.json" ;;
|
|
||||||
*) exit 9 ;;
|
|
||||||
esac
|
|
||||||
""",
|
|
||||||
)
|
|
||||||
fixture = root / "fixture"
|
|
||||||
fixture.mkdir()
|
|
||||||
(fixture / "runs.json").write_text(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"workflow_runs": [
|
|
||||||
{
|
|
||||||
"id": 9,
|
|
||||||
"head_sha": "0" * 40,
|
|
||||||
"status": "completed",
|
|
||||||
"conclusion": "success",
|
|
||||||
"html_url": "https://gitea.example.test/wrong",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"id": 12,
|
|
||||||
"head_sha": sha,
|
|
||||||
"status": "completed",
|
|
||||||
"conclusion": "success",
|
|
||||||
"html_url": "https://gitea.example.test/run/12",
|
|
||||||
},
|
|
||||||
]
|
|
||||||
}
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(fixture / "jobs.json").write_text(
|
|
||||||
json.dumps(
|
|
||||||
{"jobs": [{"id": 1, "name": "build", "head_sha": sha, "status": "completed", "conclusion": "success"}]}
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
result = self.run_wait(
|
|
||||||
root,
|
|
||||||
sha,
|
|
||||||
bin_dir,
|
|
||||||
{"BUILDER_CI_FIXTURE": str(fixture)},
|
|
||||||
)
|
|
||||||
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
|
||||||
self.assertIn("OK pipeline=https://gitea.example.test/run/12", result.stdout)
|
self.assertIn("OK pipeline=https://gitea.example.test/run/12", result.stdout)
|
||||||
self.assertIn("id=12", result.stdout)
|
self.assertIn("id=12", result.stdout)
|
||||||
self.assertIn("jobs=build=success", result.stdout)
|
self.assertIn("jobs=build=success", result.stdout)
|
||||||
self.assertNotIn("wrong", result.stdout + result.stderr)
|
|
||||||
|
|
||||||
def test_gitea_failed_job_is_failure(self) -> None:
|
def test_failed_status_is_failure(self) -> None:
|
||||||
with tempfile.TemporaryDirectory() as temp:
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
root = Path(temp)
|
root = Path(temp)
|
||||||
sha = self.make_repo(root, "git@gitea.example.test:acme/widget.git")
|
sha = self.make_repo(root)
|
||||||
bin_dir = root / "bin"
|
self.write_makefile(root)
|
||||||
bin_dir.mkdir()
|
self.write_status_script(
|
||||||
fixture = root / "fixture"
|
|
||||||
fixture.mkdir()
|
|
||||||
self.fake_command(
|
|
||||||
bin_dir,
|
|
||||||
"curl",
|
|
||||||
"""
|
|
||||||
case "$*" in
|
|
||||||
*"/jobs?"*) cat "$BUILDER_CI_FIXTURE/jobs.json" ;;
|
|
||||||
*"/actions/runs?"*) cat "$BUILDER_CI_FIXTURE/runs.json" ;;
|
|
||||||
*) exit 9 ;;
|
|
||||||
esac
|
|
||||||
""",
|
|
||||||
)
|
|
||||||
(fixture / "runs.json").write_text(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"workflow_runs": [
|
|
||||||
{
|
|
||||||
"id": 21,
|
|
||||||
"head_sha": sha,
|
|
||||||
"status": "completed",
|
|
||||||
"conclusion": "failure",
|
|
||||||
"html_url": "https://gitea.example.test/run/21",
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
(fixture / "jobs.json").write_text(
|
|
||||||
json.dumps(
|
|
||||||
{"jobs": [{"id": 2, "name": "test", "head_sha": sha, "status": "completed", "conclusion": "failure"}]}
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
result = self.run_wait(
|
|
||||||
root,
|
root,
|
||||||
sha,
|
'echo "STATUS=failed ID=21 PIPELINE=https://gitea.example.test/run/21 JOBS=test=failure"',
|
||||||
bin_dir,
|
|
||||||
{"BUILDER_CI_FIXTURE": str(fixture)},
|
|
||||||
)
|
)
|
||||||
|
result = self.run_wait(root, sha)
|
||||||
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
|
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
|
||||||
self.assertIn("reason=job_failed", result.stderr)
|
self.assertIn("reason=job_failed", result.stderr)
|
||||||
self.assertIn("test=failure", result.stderr)
|
self.assertIn("test=failure", result.stderr)
|
||||||
|
|
||||||
def test_gitea_does_not_treat_other_sha_as_pipeline(self) -> None:
|
def test_empty_status_is_no_pipeline(self) -> None:
|
||||||
with tempfile.TemporaryDirectory() as temp:
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
root = Path(temp)
|
root = Path(temp)
|
||||||
sha = self.make_repo(root, "https://gitea.example.test/acme/widget.git")
|
sha = self.make_repo(root)
|
||||||
bin_dir = root / "bin"
|
self.write_makefile(root)
|
||||||
bin_dir.mkdir()
|
self.write_status_script(root, 'echo "STATUS=empty"')
|
||||||
fixture = root / "fixture"
|
result = self.run_wait(root, sha)
|
||||||
fixture.mkdir()
|
|
||||||
self.fake_command(
|
|
||||||
bin_dir,
|
|
||||||
"curl",
|
|
||||||
'cat "$BUILDER_CI_FIXTURE/runs.json"',
|
|
||||||
)
|
|
||||||
(fixture / "runs.json").write_text(
|
|
||||||
json.dumps(
|
|
||||||
{
|
|
||||||
"workflow_runs": [
|
|
||||||
{
|
|
||||||
"id": 99,
|
|
||||||
"head_sha": "f" * 40,
|
|
||||||
"status": "completed",
|
|
||||||
"conclusion": "success",
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
),
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
result = self.run_wait(
|
|
||||||
root,
|
|
||||||
sha,
|
|
||||||
bin_dir,
|
|
||||||
{"BUILDER_CI_FIXTURE": str(fixture)},
|
|
||||||
)
|
|
||||||
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
|
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
|
||||||
self.assertIn("reason=no_pipeline", result.stderr)
|
self.assertIn("reason=no_pipeline", result.stderr)
|
||||||
|
|
||||||
def test_github_and_gitlab_branches_still_use_their_cli(self) -> None:
|
def test_missing_script_is_usage_error(self) -> None:
|
||||||
for forge, command, output in (
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
("github", "gh", "printf '%s\\n' '[]'"),
|
root = Path(temp)
|
||||||
("gitlab", "glab", "printf '%s\\n' '[]'"),
|
sha = self.make_repo(root)
|
||||||
):
|
self.write_makefile(root)
|
||||||
with self.subTest(forge=forge), tempfile.TemporaryDirectory() as temp:
|
result = self.run_wait(root, sha)
|
||||||
root = Path(temp)
|
self.assertEqual(result.returncode, 2, result.stdout + result.stderr)
|
||||||
sha = self.make_repo(root, f"https://{forge}.example.test/acme/widget.git")
|
self.assertIn("CI status script missing", result.stderr)
|
||||||
bin_dir = root / "bin"
|
|
||||||
bin_dir.mkdir()
|
def test_non_executable_script_is_usage_error(self) -> None:
|
||||||
self.fake_command(bin_dir, command, output)
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
result = self.run_wait(
|
root = Path(temp)
|
||||||
root,
|
sha = self.make_repo(root)
|
||||||
sha,
|
self.write_makefile(root)
|
||||||
bin_dir,
|
self.write_status_script(root, 'echo "STATUS=success"', executable=False)
|
||||||
{
|
result = self.run_wait(root, sha)
|
||||||
"BUILDER_CI_FORGE": forge,
|
self.assertEqual(result.returncode, 2, result.stdout + result.stderr)
|
||||||
"BUILDER_CI_APPEAR_SEC": "0",
|
self.assertIn("not executable", result.stderr)
|
||||||
},
|
|
||||||
)
|
def test_relative_script_outside_builder_dir_is_rejected(self) -> None:
|
||||||
self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
self.assertIn("reason=no_pipeline", result.stderr)
|
root = Path(temp)
|
||||||
|
sha = self.make_repo(root)
|
||||||
|
self.write_makefile(root)
|
||||||
|
rogue = root / "ci-status.sh"
|
||||||
|
rogue.write_text('#!/bin/sh\necho STATUS=success\n', encoding="utf-8")
|
||||||
|
rogue.chmod(0o755)
|
||||||
|
result = self.run_wait(
|
||||||
|
root, sha, extra_env={"CI_STATUS_SCRIPT": "ci-status.sh"}
|
||||||
|
)
|
||||||
|
self.assertEqual(result.returncode, 2, result.stdout + result.stderr)
|
||||||
|
self.assertIn("must be under .pouch/builder/", result.stderr)
|
||||||
|
|
||||||
|
def test_script_exit_3_is_cannot_query(self) -> None:
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
sha = self.make_repo(root)
|
||||||
|
self.write_makefile(root)
|
||||||
|
self.write_status_script(root, 'echo "Error: not logged in" >&2; exit 3')
|
||||||
|
result = self.run_wait(root, sha)
|
||||||
|
self.assertEqual(result.returncode, 3, result.stdout + result.stderr)
|
||||||
|
self.assertIn("CI status script exited 3", result.stderr)
|
||||||
|
self.assertIn("not logged in", result.stderr)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1]
|
|||||||
CASES = REPO_ROOT / "skills" / "tester" / "scripts" / "generate_cases_csv.py"
|
CASES = REPO_ROOT / "skills" / "tester" / "scripts" / "generate_cases_csv.py"
|
||||||
RUNS = REPO_ROOT / "skills" / "tester" / "scripts" / "generate_run_csv.py"
|
RUNS = REPO_ROOT / "skills" / "tester" / "scripts" / "generate_run_csv.py"
|
||||||
sys.path.insert(0, str(RUNS.parent))
|
sys.path.insert(0, str(RUNS.parent))
|
||||||
|
from failure_feedback import attach_feedback
|
||||||
|
|
||||||
|
|
||||||
def load_module(name: str, path: Path):
|
def load_module(name: str, path: Path):
|
||||||
@@ -29,6 +30,7 @@ class TesterRunRecordTests(unittest.TestCase):
|
|||||||
root = Path(temp)
|
root = Path(temp)
|
||||||
cases_dir = root / ".pouch" / "tester" / "cases"
|
cases_dir = root / ".pouch" / "tester" / "cases"
|
||||||
cases_dir.mkdir(parents=True)
|
cases_dir.mkdir(parents=True)
|
||||||
|
(root / ".pouch" / "tester" / "runs").mkdir()
|
||||||
(root / ".pouch" / "tester" / "catalog.yaml").write_text(
|
(root / ".pouch" / "tester" / "catalog.yaml").write_text(
|
||||||
"categories:\n - id: smoke\n title: Smoke\n",
|
"categories:\n - id: smoke\n title: Smoke\n",
|
||||||
encoding="utf-8",
|
encoding="utf-8",
|
||||||
@@ -51,6 +53,63 @@ class TesterRunRecordTests(unittest.TestCase):
|
|||||||
self.assertNotEqual(first, second)
|
self.assertNotEqual(first, second)
|
||||||
self.assertRegex(first, r"^\d{8}T\d{6}Z-smoke-[0-9a-f]{6}$")
|
self.assertRegex(first, r"^\d{8}T\d{6}Z-smoke-[0-9a-f]{6}$")
|
||||||
|
|
||||||
|
def test_run_csv_preserves_all_results_and_structured_issue_fields(self) -> None:
|
||||||
|
runs = load_module("tester_run_csv_issues", RUNS)
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
cases_dir = root / ".pouch" / "tester" / "cases"
|
||||||
|
cases_dir.mkdir(parents=True)
|
||||||
|
(root / ".pouch" / "tester" / "runs").mkdir()
|
||||||
|
(root / ".pouch" / "tester" / "catalog.yaml").write_text(
|
||||||
|
"categories:\n - id: smoke\n title: Smoke\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
(cases_dir / "smoke.yaml").write_text(
|
||||||
|
"category: smoke\ncases:\n"
|
||||||
|
" - id: TC-1\n title: One\n what: one\n lifecycle: active\n"
|
||||||
|
" tag: smoke\n risk: readonly\n how: {steps: [open]}\n"
|
||||||
|
" expected: [{kind: visible-text, value: ready}]\n"
|
||||||
|
" - id: TC-2\n title: Two\n what: two\n lifecycle: active\n"
|
||||||
|
" tag: smoke\n risk: readonly\n how: {steps: [open]}\n"
|
||||||
|
" expected: [{kind: visible-text, value: ready}]\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
run = attach_feedback(
|
||||||
|
{
|
||||||
|
"runId": "issues",
|
||||||
|
"results": [
|
||||||
|
{
|
||||||
|
"id": "TC-1",
|
||||||
|
"result": "fail",
|
||||||
|
"issue": {
|
||||||
|
"expected": "ready",
|
||||||
|
"observed": "blank",
|
||||||
|
"artifact_hash": "sha256:abc",
|
||||||
|
"environment": "Android tester",
|
||||||
|
"evidence_path": "evidence/1.png",
|
||||||
|
"reproduction": "open",
|
||||||
|
"next_step": "targeted rerun",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{"id": "TC-2", "result": "not-run", "blocked_by": ["TC-1"]},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
trigger="pr",
|
||||||
|
target="PR-1",
|
||||||
|
related_issue="ISSUE-1",
|
||||||
|
)
|
||||||
|
run_path = root / ".pouch" / "tester" / "runs" / "issues.yaml"
|
||||||
|
import yaml
|
||||||
|
run_path.write_text(yaml.safe_dump(run, sort_keys=False), encoding="utf-8")
|
||||||
|
out = runs.export_run_csv(root, run_path=run_path)
|
||||||
|
with out.open(encoding="utf-8-sig", newline="") as handle:
|
||||||
|
rows = list(csv.DictReader(handle))
|
||||||
|
self.assertEqual([row["id"] for row in rows], ["TC-1", "TC-2"])
|
||||||
|
self.assertEqual(rows[0]["issue_type"], "product")
|
||||||
|
self.assertEqual(rows[0]["expected"], "ready")
|
||||||
|
self.assertEqual(rows[0]["artifact_hash"], "sha256:abc")
|
||||||
|
self.assertEqual(rows[1]["blocked_by"], "TC-1")
|
||||||
|
self.assertEqual(run["feedback"]["route"], "pr-update-issue-dispatcher-reassign")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
Reference in New Issue
Block a user