From 3a757e33768baa4cf6c52026399803c4896363e3 Mon Sep 17 00:00:00 2001 From: laily Date: Fri, 16 Oct 2026 14:33:55 +0800 Subject: [PATCH] fix(tester): assume mobile capabilities in preflight --- skills/test-preflight/README.md | 7 +- skills/test-preflight/SKILL.md | 16 +- skills/test-preflight/references/schema.md | 37 +--- skills/test-preflight/scripts/preflight.py | 162 +----------------- skills/test-preflight/tests/test_preflight.py | 12 +- skills/tester/README.md | 4 +- skills/tester/SKILL.md | 2 +- skills/tester/references/concepts.md | 2 +- skills/tester/references/run.md | 4 +- skills/tester/references/schema.md | 6 +- 10 files changed, 35 insertions(+), 217 deletions(-) diff --git a/skills/test-preflight/README.md b/skills/test-preflight/README.md index 88e49a3..7a97ef8 100644 --- a/skills/test-preflight/README.md +++ b/skills/test-preflight/README.md @@ -2,10 +2,9 @@ Reusable preflight for tester matrices. It supports read-only `verify` and test-environment-only `prepare`, and writes per-check YAML/CSV evidence for -artifacts, tester nodes, devices, network, services, mobile album/photo/pinyin -capabilities, and fixture/cleanup contracts. Capability checks run on the actual -Android/iOS tester node; authorization environment variables are not accepted as -a substitute. +artifacts, tester nodes, devices, network, services, and fixture/cleanup contracts. +This project assumes mobile album/photo/IME capabilities; journey cases exercise them +and preflight does not gate them or read authorization environment variables. Run the self-tests with: diff --git a/skills/test-preflight/SKILL.md b/skills/test-preflight/SKILL.md index 6f0eae3..9bcb18e 100644 --- a/skills/test-preflight/SKILL.md +++ b/skills/test-preflight/SKILL.md @@ -2,8 +2,8 @@ name: test-preflight description: >- Verify or prepare a mobile/integration test environment from a declared matrix: - artifacts, tester bindings, HOST/PUBLIC_HOST, service probes, mobile capability - probes, and fixture/cleanup contracts. Use before tester runs; it is not a deployment, + artifacts, tester bindings, HOST/PUBLIC_HOST, service probes, and fixture/cleanup + contracts. Use before tester runs; it is not a deployment, build, or business-data skill. --- @@ -41,19 +41,17 @@ Mac or that a deployment happened. Overall status is `ready`, `blocked`, or `environment-fail`: -- `blocked` means a required declaration, artifact, binding, runner, capability, fixture, +- `blocked` means a required declaration, artifact, binding, runner, fixture, cleanup contract, or authorization is absent/unsafe. - `environment-fail` means a declared environment exists but DNS/TCP/HTTP, device/toolchain, installation, launch, or deployment execution failed. - `ready` means every recorded check passed. A ready preflight permits the caller to continue with `integration`; this skill does not run test cases. -For every mobile matrix row, preflight requires real tester-node capability evidence for -`album_picker`, `photo_permission` (including a usable test image), and `ime_pinyin` -(composition and candidate selection). Android evidence must exercise the Android picker, -permission state and real Chinese IME composition; iOS evidence must exercise the corresponding -Photos picker/permission and text-input capabilities on the iOS node. `AUTHORIZE_ALBUM` and -`IME_PINYIN` are never accepted as substitutes for these probes. +This project assumes the client has the required album picker, photo permission, and Chinese +IME capabilities. Preflight does not probe or gate these capabilities, and does not read +`AUTHORIZE_ALBUM` or `IME_PINYIN`. Their real behavior is exercised by journey cases; a failure +there is a product test failure, not an environment block. Do not treat a preflight report as a product result. A preflight failure means no client session may be created; the tester owns `not-run` propagation and must write diff --git a/skills/test-preflight/references/schema.md b/skills/test-preflight/references/schema.md index 9096a6d..c6c912c 100644 --- a/skills/test-preflight/references/schema.md +++ b/skills/test-preflight/references/schema.md @@ -18,10 +18,9 @@ fixtures: cleanup: required: true steps: [delete-run-data] +# Optional project metadata. It is not a preflight gate. capabilities: - # These baseline checks are mandatory for mobile rows and cannot be disabled - # with AUTHORIZE_ALBUM or IME_PINYIN. - required: [album_picker, photo_permission, ime_pinyin] + assumed: [album_picker, photo_permission, ime_pinyin] matrix: - platform: android tester: @@ -52,34 +51,12 @@ needs non-empty `references`; a required cleanup needs non-empty `steps` or `run`. For a read-only run, use `required: false` with a non-empty `reason`. The skill never executes these values. -## Mobile capability contract +## Client capabilities -Every Android/iOS matrix row checks these capabilities on the actual tester node: - -| capability | required evidence | -|---|---| -| `album_picker` | `checks.picker`: the real platform album/Photos picker opened and returned a selectable asset | -| `photo_permission` | `checks.permission` and `checks.test_image`: permission state was exercised and the declared test image was available | -| `ime_pinyin` | `checks.composition` and `checks.candidates`: real Chinese pinyin composition and candidate selection succeeded | - -The default list is always applied; `capabilities.required` can add project-specific -checks but cannot remove these three. A contract runner receives one request per -capability: - -```json -{ - "operation": "capability_probe", - "platform": "android", - "node": "tester.android", - "capability": "ime_pinyin", - "required_checks": ["composition", "candidates"] -} -``` - -It must return every requested check with `status`, `observed`, and (when useful) -`expected`/`remediation`. Missing evidence is `blocked`; a real probe failure is -`environment-fail`. Environment variables such as `AUTHORIZE_ALBUM` and -`IME_PINYIN` are not capability evidence. +The project assumes the mobile test nodes provide the album picker, photo permission, +test image, and Chinese IME. These are not preflight declarations or gates. Journey +cases exercise the real UI; failures are recorded as product failures. `AUTHORIZE_ALBUM` +and `IME_PINYIN` are not read. ## Tester-node contract diff --git a/skills/test-preflight/scripts/preflight.py b/skills/test-preflight/scripts/preflight.py index 66dbbd5..9931153 100644 --- a/skills/test-preflight/scripts/preflight.py +++ b/skills/test-preflight/scripts/preflight.py @@ -31,17 +31,6 @@ except ImportError as exc: # pragma: no cover PLATFORMS = {"android", "ios"} STATUSES = {"ready", "blocked", "environment-fail"} -CAPABILITY_CHECKS = { - "album_picker": ("picker",), - "photo_permission": ("permission", "test_image"), - "ime_pinyin": ("composition", "candidates"), -} -CAPABILITY_EXPECTED = { - "album_picker": "real platform album picker opens and returns a selectable asset", - "photo_permission": "photo permission flow is observable and the declared test image is available", - "ime_pinyin": "real Chinese pinyin composition produces and selects candidate words", -} -DEFAULT_CAPABILITIES = tuple(CAPABILITY_CHECKS) EVIDENCE_FIELDS = [ "check_id", "mode", "platform", "node", "target", "observed", "expected", "status", "remediation", "timestamp", @@ -172,27 +161,6 @@ def validate_shape(config: Mapping[str, Any], evidence: Evidence) -> bool: if not isinstance(config.get("host"), str) or not config["host"].strip(): evidence.add("config-host", target="HOST", expected="non-empty host URL", status="blocked", remediation="提供 HOST") ok = False - capabilities = config.get("capabilities") - if capabilities is not None and not isinstance(capabilities, (list, dict)): - evidence.add( - "config-capabilities", - target="capabilities", - observed=capabilities, - expected="list or mapping with required list", - status="blocked", - remediation="声明 capabilities.required；不能用 AUTHORIZE_ALBUM/IME_PINYIN 代替能力检查", - ) - ok = False - elif isinstance(capabilities, dict) and capabilities.get("required") is not None and not isinstance(capabilities.get("required"), list): - evidence.add( - "config-capabilities-required", - target="capabilities.required", - observed=capabilities.get("required"), - expected="list of capability ids", - status="blocked", - remediation="将 capabilities.required 写成列表", - ) - ok = False probes = config.get("service_probes") if not isinstance(probes, list) or not probes: evidence.add("config-service-probes", target="service_probes", expected="at least one probe", status="blocked", remediation="声明服务 URL 探针") @@ -454,122 +422,6 @@ def check_fixture_cleanup(evidence: Evidence, config: Mapping[str, Any], platfor return ok -def capability_ids(config: Mapping[str, Any]) -> list[str]: - """Return mandatory mobile capabilities. - - The three baseline capabilities are never disabled by an authorization - environment variable. A config may add more checks, but it cannot remove - the album/photo/pinyin checks required for real client journeys. - """ - raw = config.get("capabilities") - requested: Any = raw.get("required") if isinstance(raw, dict) else raw - ids = list(DEFAULT_CAPABILITIES) - if isinstance(requested, list): - for item in requested: - name = text(item).strip() - if name and name not in ids: - ids.append(name) - return ids - - -def check_capabilities( - evidence: Evidence, - config: Mapping[str, Any], - row: Mapping[str, Any], - platform_name: str, - node: str, - index: int, - *, - platform_ok: bool, -) -> bool: - """Run real capability probes on the resolved tester node. - - A contract runner must return each required sub-check. We intentionally do - not infer album, permission, image, or IME behavior from adb/xcrun alone: - the adapter must exercise the real picker/permission/UI composition path. - """ - tester = row.get("tester") or {} - runner = tester.get("runner") if isinstance(tester, dict) else None - is_local = node in {"local", "localhost", "127.0.0.1"} - capabilities_ok = True - for capability in capability_ids(config): - required = CAPABILITY_CHECKS.get(capability, ("capability",)) - prefix = f"{platform_name}-{index}-capability-{capability}" - if not platform_ok: - for check in required: - evidence.add( - f"{prefix}-{check}", - platform=platform_name, - node=node, - target=capability, - observed="platform prerequisite failed", - expected=CAPABILITY_EXPECTED.get(capability, capability), - status="environment-fail", - remediation="先修复平台/设备闸门，再由 tester 节点重跑真实能力探针", - ) - capabilities_ok = False - continue - if not (isinstance(runner, dict) and runner.get("kind") == "contract"): - for check in required: - evidence.add( - f"{prefix}-{check}", - platform=platform_name, - node=node, - target=capability, - observed="no real capability runner", - expected=CAPABILITY_EXPECTED.get(capability, capability), - status="blocked", - remediation="配置 tester 节点的 contract runner，真实打开 picker/权限 UI 并完成拼音 composition/candidate 检查", - ) - capabilities_ok = False - continue - - status, observed, result = runner_call( - runner, - { - "operation": "capability_probe", - "platform": platform_name, - "node": node, - "capability": capability, - "required_checks": list(required), - "artifact": (row.get("artifact") or {}).get("path"), - }, - ) - checks = result.get("checks") if isinstance(result, dict) else None - if not isinstance(checks, dict): - checks = {} - for check in required: - detail = checks.get(check) - if detail is None: - evidence.add( - f"{prefix}-{check}", - platform=platform_name, - node=node, - target=capability, - observed=observed or "runner omitted required evidence", - expected=CAPABILITY_EXPECTED.get(capability, capability), - status="blocked", - remediation=f"runner 必须返回 checks.{check}，且证据来自真实 {platform_name} 能力操作", - ) - capabilities_ok = False - continue - detail = detail if isinstance(detail, dict) else {"observed": detail} - detail_status = status_from(detail.get("status"), status) - evidence.add( - f"{prefix}-{check}", - platform=platform_name, - node=node, - target=text(detail.get("target", capability)), - observed=detail.get("observed", observed), - expected=detail.get("expected", CAPABILITY_EXPECTED.get(capability, capability)), - status=detail_status, - remediation=text(detail.get("remediation", "")), - ) - if detail_status != "ready": - capabilities_ok = False - return capabilities_ok - - def run(config: Mapping[str, Any], mode: str) -> Evidence: evidence = Evidence(mode) shape_ok = validate_shape(config, evidence) @@ -627,16 +479,10 @@ def run(config: Mapping[str, Any], mode: str) -> Evidence: evidence.add(f"{platform_name}-{index}-platform-runner", platform=platform_name, node=node, target=node, expected="node runner contract", status="blocked", remediation="远端 tester 必须由 agent-runtime 提供可执行 runner") platform_ok = False network_ok = check_network(evidence, config, row, platform_name, node, index) - capabilities_ok = check_capabilities( - evidence, - config, - row, - platform_name, - node, - index, - platform_ok=platform_ok, - ) - row_ok = artifact_ok and fixture_ok and platform_ok and network_ok and capabilities_ok + # This isolated project assumes the client capabilities used by its + # journeys. Their real behavior is a product assertion, not a preflight + # environment gate. + row_ok = artifact_ok and fixture_ok and platform_ok and network_ok if mode == "prepare": artifact = row.get("artifact") or {} identity = artifact.get("package" if platform_name == "android" else "bundle_id") diff --git a/skills/test-preflight/tests/test_preflight.py b/skills/test-preflight/tests/test_preflight.py index c67a60f..a8a2ccf 100644 --- a/skills/test-preflight/tests/test_preflight.py +++ b/skills/test-preflight/tests/test_preflight.py @@ -140,7 +140,7 @@ class PreflightTests(unittest.TestCase): self.assertTrue(any(row["check_id"].endswith("-fixture") and row["status"] == "blocked" for row in result.rows)) self.assertTrue(any(row["check_id"].endswith("-cleanup") and row["status"] == "blocked" for row in result.rows)) - def test_mobile_capabilities_are_mandatory_and_missing_evidence_blocks(self): + def test_mobile_capabilities_are_assumed_and_not_a_preflight_gate(self): data = self.config() config_path = self.write_config(data) env = os.environ.copy() @@ -151,12 +151,9 @@ class PreflightTests(unittest.TestCase): "--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "capabilities.csv"), ], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env) report = __import__("yaml").safe_load(output.read_text(encoding="utf-8")) - self.assertEqual(process.returncode, 1) - self.assertEqual(report["status"], "blocked") - capability_rows = [row for row in report["checks"] if "capability-album_picker" in row["check_id"]] - self.assertTrue(capability_rows) - self.assertTrue(all(row["status"] == "blocked" for row in capability_rows)) - self.assertTrue(any("photo_permission" in row["check_id"] for row in report["checks"])) + self.assertEqual(process.returncode, 0) + self.assertEqual(report["status"], "ready") + self.assertFalse(any("capability-" in row["check_id"] for row in report["checks"])) def test_host_unreachable_from_actual_runner_is_environment_fail(self): data = self.config() @@ -221,6 +218,7 @@ class PreflightTests(unittest.TestCase): self.assertEqual(result.overall(), "ready") self.assertNotIn("install", calls) self.assertNotIn("start", calls) + self.assertNotIn("capability_probe", calls) if __name__ == "__main__": diff --git a/skills/tester/README.md b/skills/tester/README.md index 38a7d9d..3e1d78f 100644 --- a/skills/tester/README.md +++ b/skills/tester/README.md @@ -31,8 +31,8 @@ `standard` 只跑最低档 `readonly`。隔离、可重置的 Writes 类环境在 catalog 设置 `execution_profile: isolated-full` 后，所有 active 自动化用例都进入执行，不因 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或 - `AUTHORIZE_PASSWORD` skipped；risk 原值仍保留，必须用 test-preflight 的真实能力检查和 - cleanup/rollback 保证安全。 + `AUTHORIZE_PASSWORD` skipped；risk 原值仍保留。设备、产物、服务和 cleanup/rollback + 由 test-preflight 保证；客户端相册、权限和中文输入能力默认具备，由 journey 用例验证。 ## 使用示例 diff --git a/skills/tester/SKILL.md b/skills/tester/SKILL.md index c610f46..7d966fb 100644 --- a/skills/tester/SKILL.md +++ b/skills/tester/SKILL.md @@ -74,7 +74,7 @@ tester agent 配置。 - `local.yaml.example` 的 HOST 默认是 `127.0.0.1`；如果目标不是本机，必须在 `local.yaml` 或环境变量中显式覆盖。`local.yaml` 不存在且没有环境变量时仍不猜 HOST，停下并记 environment-fail。 -- 默认执行 profile 是 `standard`：风险上限来自 catalog 的 `default_max_risk`，超出上限的条目标 `skipped`，禁止执行。若 catalog 声明 `execution_profile: isolated-full`，只允许用于隔离且可重置的测试环境：所有 active 自动化用例都进入执行矩阵，不因 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或 `AUTHORIZE_PASSWORD` skipped；原有 `risk` 值必须保留并进入报告，安全边界改由 test-preflight 能力探针和 cleanup/rollback 契约保证。 +- 默认执行 profile 是 `standard`：风险上限来自 catalog 的 `default_max_risk`，超出上限的条目标 `skipped`，禁止执行。若 catalog 声明 `execution_profile: isolated-full`，只允许用于隔离且可重置的测试环境：所有 active 自动化用例都进入执行矩阵，不因 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或 `AUTHORIZE_PASSWORD` skipped；原有 `risk` 值必须保留并进入报告。设备、产物、服务和清理契约由 test-preflight 检查；相册、照片权限/测试图片和中文拼音能力在本项目中默认具备，由 journey 用例真实验证。 - `risk` 只能是 `readonly` / `mutating` / `destructive`，默认只跑 `readonly`。`tag=smoke` 只能配 `risk=readonly`。规则见 [levels.md](references/levels.md)。 - `driver=browser` 必须开真实页面；`driver=mobile` 必须挂模拟器 session。二者都不得改成只打 API。 - `automation=assisted|manual` 需要人在场：无人值守时记 `not-run`，不要自己代跑人工步骤。isolated-full 取消的是风险/授权过滤，不会伪造人工操作。见 [concepts.md](references/concepts.md)。 diff --git a/skills/tester/references/concepts.md b/skills/tester/references/concepts.md index a3a5531..f571cb0 100644 --- a/skills/tester/references/concepts.md +++ b/skills/tester/references/concepts.md @@ -84,7 +84,7 @@ catalog（整个测试集） - 与 `gate` 正交：`risk` 说「影响多大」，`gate` 说「碰哪个子系统」。点动 vs 焊接这种细分授权用 `gate`，不要再造 risk 名。 - `tag=smoke` 只能配 `risk=readonly`：冒烟测试不许留下变化。 - 隔离可重置环境可设置 `execution_profile: isolated-full` 跑所有 active 用例；安全前置由 - test-preflight capability 与 cleanup/rollback 契约保证，原始 risk 仍必须报告。 + test-preflight 的设备、服务与 cleanup/rollback 契约保证，原始 risk 仍必须报告；客户端能力由 journey 用例验证。 ### 3.2 `automation`：为什么是 `automated / assisted / manual` diff --git a/skills/tester/references/run.md b/skills/tester/references/run.md index 9b8cfac..8e07999 100644 --- a/skills/tester/references/run.md +++ b/skills/tester/references/run.md @@ -141,8 +141,8 @@ preflight 通过后，先按 catalog 顺序执行 `integration`，再按客户 - `HOST` 未设（环境变量 / `local.yaml` 都没有）：全部停，environment-fail。 - `standard`：命中条的 `risk` 在 catalog `authorize` 里有对应变量时，变量不是 `true` 则该条 `skipped`；有 `gate` 时还要对应变量为 `true`，`assume` 不成立同样 `skipped`。即使用户点了 id 也不执行。 -- `isolated-full`：不读取 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM`、`AUTHORIZE_PASSWORD` 等人工开关；相册、照片权限/测试图片和中文拼音真实 composition/candidate 能力必须由 test-preflight 证明。 -- standard 中授权/假设不成立是**主动不跑**（`skipped`）；isolated-full 的能力缺失是 preflight blocked/environment-fail，并为全部 testcase 写 `not-run` + `blocked_by: [test-preflight]`。 +- `isolated-full`：不读取 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM`、`AUTHORIZE_PASSWORD` 等人工开关；相册、照片权限/测试图片和中文拼音能力默认具备，由 journey 用例真实验证，失败记为产品失败。 +- standard 中授权/假设不成立是**主动不跑**（`skipped`）；isolated-full 不因客户端能力授权变量停止，设备、产物、服务或清理前置失败时才为全部 testcase 写 `not-run` + `blocked_by: [test-preflight]`。 - `status_probe` 是**分类级前置条件**（业界对应 pytest 的 session fixture / Robot 的 Suite Setup）。用 `select` 返回的 `status_probe` 解析生效值，不要自己读 catalog 猜。返回的 `probe.url` 已用 `vars` 展开，直接请求即可；`select` 同批返回 `vars`： | `categories[].status_probe` | 生效探测 | diff --git a/skills/tester/references/schema.md b/skills/tester/references/schema.md index 4f33a49..338f867 100644 --- a/skills/tester/references/schema.md +++ b/skills/tester/references/schema.md @@ -25,7 +25,7 @@ runs/*.csv 同一次运行的成绩单；generate_run_csv.py 从对 | `categories` | `{id, title, status_probe?}` 列表，id 与 `cases/.yaml` 对应；列表顺序即 `select` / 执行顺序。`status_probe` 见下 | | `risks` | 从低到高，只能是 `readonly` → `mutating` → `destructive` 的一段，且以 `readonly` 开头。含义见 [levels.md](levels.md) | | `default_max_risk` | 未指定时的运行上限，必须是 `risks` 里的值 | -| `execution_profile` | `standard`（默认）或 `isolated-full`。后者只用于隔离可重置环境，保留 risk 语义但不按 risk/authorize/gate 过滤 active 用例；安全边界由 preflight capability 与 cleanup/rollback 契约承担 | +| `execution_profile` | `standard`（默认）或 `isolated-full`。后者只用于隔离可重置环境，保留 risk 语义但不按 risk/authorize/gate 过滤 active 用例；设备、服务和 cleanup/rollback 契约仍由 preflight 检查，客户端能力由 journey 真实验证 | | `vars` | 可替换变量，如 `API_BASE: http://${HOST}:8095` | | `artifacts` | 可选的移动产物身份契约。按 `android` / `ios` 声明 `required` 和 `package` / `bundle_id`；每轮文件路径或下载地址由调用方提供，不提交临时产物路径。 | | `authorize` | 风险名 → 环境变量名。该档执行前变量必须为 `true`（`readonly` 不需要） | @@ -38,8 +38,8 @@ runs/*.csv 同一次运行的成绩单；generate_run_csv.py 从对 | `integration_suites` | 可选的集成套件声明；标准值为 `api-contract`、`authorization`、`build-contract` | `isolated-full` 不得把风险字段改成 `readonly`，也不得把 `AUTHORIZE_ALBUM`、 -`AUTHORIZE_PASSWORD`、`IME_PINYIN` 等变量当成能力门禁。移动用例必须通过 test-preflight -的真实相册 picker、照片权限/测试图片和中文拼音 composition/candidate 检查。 +`AUTHORIZE_PASSWORD`、`IME_PINYIN` 等变量当成门禁。移动用例必须通过 test-preflight +的设备、产物、服务和网络检查；相册、权限和输入法能力默认具备，交由 journey 用例验证。 `local.yaml` 与环境变量覆盖 `HOST` 和授权开关。模板中的 `HOST` 默认是 `127.0.0.1`；远程目标必须显式覆盖。查找顺序：环境变量 > `local.yaml` >