fix(tester): assume mobile capabilities in preflight

This commit is contained in:
2026-10-16 14:33:55 +08:00
parent 5b9a693fda
commit 3a757e3376
10 changed files with 35 additions and 217 deletions
+3 -4
View File
@@ -2,10 +2,9 @@
Reusable preflight for tester matrices. It supports read-only `verify` and
test-environment-only `prepare`, and writes per-check YAML/CSV evidence for
artifacts, tester nodes, devices, network, services, mobile album/photo/pinyin
capabilities, and fixture/cleanup contracts. Capability checks run on the actual
Android/iOS tester node; authorization environment variables are not accepted as
a substitute.
artifacts, tester nodes, devices, network, services, and fixture/cleanup contracts.
This project assumes mobile album/photo/IME capabilities; journey cases exercise them
and preflight does not gate them or read authorization environment variables.
Run the self-tests with:
+7 -9
View File
@@ -2,8 +2,8 @@
name: test-preflight
description: >-
Verify or prepare a mobile/integration test environment from a declared matrix:
artifacts, tester bindings, HOST/PUBLIC_HOST, service probes, mobile capability
probes, and fixture/cleanup contracts. Use before tester runs; it is not a deployment,
artifacts, tester bindings, HOST/PUBLIC_HOST, service probes, and fixture/cleanup
contracts. Use before tester runs; it is not a deployment,
build, or business-data skill.
---
@@ -41,19 +41,17 @@ Mac or that a deployment happened.
Overall status is `ready`, `blocked`, or `environment-fail`:
- `blocked` means a required declaration, artifact, binding, runner, capability, fixture,
- `blocked` means a required declaration, artifact, binding, runner, fixture,
cleanup contract, or authorization is absent/unsafe.
- `environment-fail` means a declared environment exists but DNS/TCP/HTTP,
device/toolchain, installation, launch, or deployment execution failed.
- `ready` means every recorded check passed. A ready preflight permits the
caller to continue with `integration`; this skill does not run test cases.
For every mobile matrix row, preflight requires real tester-node capability evidence for
`album_picker`, `photo_permission` (including a usable test image), and `ime_pinyin`
(composition and candidate selection). Android evidence must exercise the Android picker,
permission state and real Chinese IME composition; iOS evidence must exercise the corresponding
Photos picker/permission and text-input capabilities on the iOS node. `AUTHORIZE_ALBUM` and
`IME_PINYIN` are never accepted as substitutes for these probes.
This project assumes the client has the required album picker, photo permission, and Chinese
IME capabilities. Preflight does not probe or gate these capabilities, and does not read
`AUTHORIZE_ALBUM` or `IME_PINYIN`. Their real behavior is exercised by journey cases; a failure
there is a product test failure, not an environment block.
Do not treat a preflight report as a product result. A preflight failure means
no client session may be created; the tester owns `not-run` propagation and must write
+7 -30
View File
@@ -18,10 +18,9 @@ fixtures:
cleanup:
required: true
steps: [delete-run-data]
# Optional project metadata. It is not a preflight gate.
capabilities:
# These baseline checks are mandatory for mobile rows and cannot be disabled
# with AUTHORIZE_ALBUM or IME_PINYIN.
required: [album_picker, photo_permission, ime_pinyin]
assumed: [album_picker, photo_permission, ime_pinyin]
matrix:
- platform: android
tester:
@@ -52,34 +51,12 @@ needs non-empty `references`; a required cleanup needs non-empty `steps` or
`run`. For a read-only run, use `required: false` with a non-empty `reason`.
The skill never executes these values.
## Mobile capability contract
## Client capabilities
Every Android/iOS matrix row checks these capabilities on the actual tester node:
| capability | required evidence |
|---|---|
| `album_picker` | `checks.picker`: the real platform album/Photos picker opened and returned a selectable asset |
| `photo_permission` | `checks.permission` and `checks.test_image`: permission state was exercised and the declared test image was available |
| `ime_pinyin` | `checks.composition` and `checks.candidates`: real Chinese pinyin composition and candidate selection succeeded |
The default list is always applied; `capabilities.required` can add project-specific
checks but cannot remove these three. A contract runner receives one request per
capability:
```json
{
"operation": "capability_probe",
"platform": "android",
"node": "tester.android",
"capability": "ime_pinyin",
"required_checks": ["composition", "candidates"]
}
```
It must return every requested check with `status`, `observed`, and (when useful)
`expected`/`remediation`. Missing evidence is `blocked`; a real probe failure is
`environment-fail`. Environment variables such as `AUTHORIZE_ALBUM` and
`IME_PINYIN` are not capability evidence.
The project assumes the mobile test nodes provide the album picker, photo permission,
test image, and Chinese IME. These are not preflight declarations or gates. Journey
cases exercise the real UI; failures are recorded as product failures. `AUTHORIZE_ALBUM`
and `IME_PINYIN` are not read.
## Tester-node contract
+4 -158
View File
@@ -31,17 +31,6 @@ except ImportError as exc: # pragma: no cover
PLATFORMS = {"android", "ios"}
STATUSES = {"ready", "blocked", "environment-fail"}
CAPABILITY_CHECKS = {
"album_picker": ("picker",),
"photo_permission": ("permission", "test_image"),
"ime_pinyin": ("composition", "candidates"),
}
CAPABILITY_EXPECTED = {
"album_picker": "real platform album picker opens and returns a selectable asset",
"photo_permission": "photo permission flow is observable and the declared test image is available",
"ime_pinyin": "real Chinese pinyin composition produces and selects candidate words",
}
DEFAULT_CAPABILITIES = tuple(CAPABILITY_CHECKS)
EVIDENCE_FIELDS = [
"check_id", "mode", "platform", "node", "target", "observed", "expected",
"status", "remediation", "timestamp",
@@ -172,27 +161,6 @@ def validate_shape(config: Mapping[str, Any], evidence: Evidence) -> bool:
if not isinstance(config.get("host"), str) or not config["host"].strip():
evidence.add("config-host", target="HOST", expected="non-empty host URL", status="blocked", remediation="提供 HOST")
ok = False
capabilities = config.get("capabilities")
if capabilities is not None and not isinstance(capabilities, (list, dict)):
evidence.add(
"config-capabilities",
target="capabilities",
observed=capabilities,
expected="list or mapping with required list",
status="blocked",
remediation="声明 capabilities.required;不能用 AUTHORIZE_ALBUM/IME_PINYIN 代替能力检查",
)
ok = False
elif isinstance(capabilities, dict) and capabilities.get("required") is not None and not isinstance(capabilities.get("required"), list):
evidence.add(
"config-capabilities-required",
target="capabilities.required",
observed=capabilities.get("required"),
expected="list of capability ids",
status="blocked",
remediation="将 capabilities.required 写成列表",
)
ok = False
probes = config.get("service_probes")
if not isinstance(probes, list) or not probes:
evidence.add("config-service-probes", target="service_probes", expected="at least one probe", status="blocked", remediation="声明服务 URL 探针")
@@ -454,122 +422,6 @@ def check_fixture_cleanup(evidence: Evidence, config: Mapping[str, Any], platfor
return ok
def capability_ids(config: Mapping[str, Any]) -> list[str]:
"""Return mandatory mobile capabilities.
The three baseline capabilities are never disabled by an authorization
environment variable. A config may add more checks, but it cannot remove
the album/photo/pinyin checks required for real client journeys.
"""
raw = config.get("capabilities")
requested: Any = raw.get("required") if isinstance(raw, dict) else raw
ids = list(DEFAULT_CAPABILITIES)
if isinstance(requested, list):
for item in requested:
name = text(item).strip()
if name and name not in ids:
ids.append(name)
return ids
def check_capabilities(
evidence: Evidence,
config: Mapping[str, Any],
row: Mapping[str, Any],
platform_name: str,
node: str,
index: int,
*,
platform_ok: bool,
) -> bool:
"""Run real capability probes on the resolved tester node.
A contract runner must return each required sub-check. We intentionally do
not infer album, permission, image, or IME behavior from adb/xcrun alone:
the adapter must exercise the real picker/permission/UI composition path.
"""
tester = row.get("tester") or {}
runner = tester.get("runner") if isinstance(tester, dict) else None
is_local = node in {"local", "localhost", "127.0.0.1"}
capabilities_ok = True
for capability in capability_ids(config):
required = CAPABILITY_CHECKS.get(capability, ("capability",))
prefix = f"{platform_name}-{index}-capability-{capability}"
if not platform_ok:
for check in required:
evidence.add(
f"{prefix}-{check}",
platform=platform_name,
node=node,
target=capability,
observed="platform prerequisite failed",
expected=CAPABILITY_EXPECTED.get(capability, capability),
status="environment-fail",
remediation="先修复平台/设备闸门,再由 tester 节点重跑真实能力探针",
)
capabilities_ok = False
continue
if not (isinstance(runner, dict) and runner.get("kind") == "contract"):
for check in required:
evidence.add(
f"{prefix}-{check}",
platform=platform_name,
node=node,
target=capability,
observed="no real capability runner",
expected=CAPABILITY_EXPECTED.get(capability, capability),
status="blocked",
remediation="配置 tester 节点的 contract runner,真实打开 picker/权限 UI 并完成拼音 composition/candidate 检查",
)
capabilities_ok = False
continue
status, observed, result = runner_call(
runner,
{
"operation": "capability_probe",
"platform": platform_name,
"node": node,
"capability": capability,
"required_checks": list(required),
"artifact": (row.get("artifact") or {}).get("path"),
},
)
checks = result.get("checks") if isinstance(result, dict) else None
if not isinstance(checks, dict):
checks = {}
for check in required:
detail = checks.get(check)
if detail is None:
evidence.add(
f"{prefix}-{check}",
platform=platform_name,
node=node,
target=capability,
observed=observed or "runner omitted required evidence",
expected=CAPABILITY_EXPECTED.get(capability, capability),
status="blocked",
remediation=f"runner 必须返回 checks.{check},且证据来自真实 {platform_name} 能力操作",
)
capabilities_ok = False
continue
detail = detail if isinstance(detail, dict) else {"observed": detail}
detail_status = status_from(detail.get("status"), status)
evidence.add(
f"{prefix}-{check}",
platform=platform_name,
node=node,
target=text(detail.get("target", capability)),
observed=detail.get("observed", observed),
expected=detail.get("expected", CAPABILITY_EXPECTED.get(capability, capability)),
status=detail_status,
remediation=text(detail.get("remediation", "")),
)
if detail_status != "ready":
capabilities_ok = False
return capabilities_ok
def run(config: Mapping[str, Any], mode: str) -> Evidence:
evidence = Evidence(mode)
shape_ok = validate_shape(config, evidence)
@@ -627,16 +479,10 @@ def run(config: Mapping[str, Any], mode: str) -> Evidence:
evidence.add(f"{platform_name}-{index}-platform-runner", platform=platform_name, node=node, target=node, expected="node runner contract", status="blocked", remediation="远端 tester 必须由 agent-runtime 提供可执行 runner")
platform_ok = False
network_ok = check_network(evidence, config, row, platform_name, node, index)
capabilities_ok = check_capabilities(
evidence,
config,
row,
platform_name,
node,
index,
platform_ok=platform_ok,
)
row_ok = artifact_ok and fixture_ok and platform_ok and network_ok and capabilities_ok
# This isolated project assumes the client capabilities used by its
# journeys. Their real behavior is a product assertion, not a preflight
# environment gate.
row_ok = artifact_ok and fixture_ok and platform_ok and network_ok
if mode == "prepare":
artifact = row.get("artifact") or {}
identity = artifact.get("package" if platform_name == "android" else "bundle_id")
@@ -140,7 +140,7 @@ class PreflightTests(unittest.TestCase):
self.assertTrue(any(row["check_id"].endswith("-fixture") and row["status"] == "blocked" for row in result.rows))
self.assertTrue(any(row["check_id"].endswith("-cleanup") and row["status"] == "blocked" for row in result.rows))
def test_mobile_capabilities_are_mandatory_and_missing_evidence_blocks(self):
def test_mobile_capabilities_are_assumed_and_not_a_preflight_gate(self):
data = self.config()
config_path = self.write_config(data)
env = os.environ.copy()
@@ -151,12 +151,9 @@ class PreflightTests(unittest.TestCase):
"--config", str(config_path), "--yaml", str(output), "--csv", str(self.root / "capabilities.csv"),
], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
report = __import__("yaml").safe_load(output.read_text(encoding="utf-8"))
self.assertEqual(process.returncode, 1)
self.assertEqual(report["status"], "blocked")
capability_rows = [row for row in report["checks"] if "capability-album_picker" in row["check_id"]]
self.assertTrue(capability_rows)
self.assertTrue(all(row["status"] == "blocked" for row in capability_rows))
self.assertTrue(any("photo_permission" in row["check_id"] for row in report["checks"]))
self.assertEqual(process.returncode, 0)
self.assertEqual(report["status"], "ready")
self.assertFalse(any("capability-" in row["check_id"] for row in report["checks"]))
def test_host_unreachable_from_actual_runner_is_environment_fail(self):
data = self.config()
@@ -221,6 +218,7 @@ class PreflightTests(unittest.TestCase):
self.assertEqual(result.overall(), "ready")
self.assertNotIn("install", calls)
self.assertNotIn("start", calls)
self.assertNotIn("capability_probe", calls)
if __name__ == "__main__":
+2 -2
View File
@@ -31,8 +31,8 @@
`standard` 只跑最低档 `readonly`。隔离、可重置的 Writes 类环境在 catalog 设置
`execution_profile: isolated-full` 后,所有 active 自动化用例都进入执行,不因
`AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或
`AUTHORIZE_PASSWORD` skipped;risk 原值仍保留,必须用 test-preflight 的真实能力检查和
cleanup/rollback 保证安全。
`AUTHORIZE_PASSWORD` skipped;risk 原值仍保留。设备、产物、服务和 cleanup/rollback
由 test-preflight 保证;客户端相册、权限和中文输入能力默认具备,由 journey 用例验证。
## 使用示例
+1 -1
View File
@@ -74,7 +74,7 @@ tester agent 配置。
- `local.yaml.example` 的 HOST 默认是 `127.0.0.1`;如果目标不是本机,必须在
`local.yaml` 或环境变量中显式覆盖。`local.yaml` 不存在且没有环境变量时仍不猜
HOST,停下并记 environment-fail。
- 默认执行 profile 是 `standard`:风险上限来自 catalog 的 `default_max_risk`,超出上限的条目标 `skipped`,禁止执行。若 catalog 声明 `execution_profile: isolated-full`,只允许用于隔离且可重置的测试环境:所有 active 自动化用例都进入执行矩阵,不因 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或 `AUTHORIZE_PASSWORD` skipped;原有 `risk` 值必须保留并进入报告,安全边界改由 test-preflight 能力探针和 cleanup/rollback 契约保证。
- 默认执行 profile 是 `standard`:风险上限来自 catalog 的 `default_max_risk`,超出上限的条目标 `skipped`,禁止执行。若 catalog 声明 `execution_profile: isolated-full`,只允许用于隔离且可重置的测试环境:所有 active 自动化用例都进入执行矩阵,不因 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM` 或 `AUTHORIZE_PASSWORD` skipped;原有 `risk` 值必须保留并进入报告。设备、产物、服务和清理契约由 test-preflight 检查;相册、照片权限/测试图片和中文拼音能力在本项目中默认具备,由 journey 用例真实验证。
- `risk` 只能是 `readonly` / `mutating` / `destructive`,默认只跑 `readonly`。`tag=smoke` 只能配 `risk=readonly`。规则见 [levels.md](references/levels.md)。
- `driver=browser` 必须开真实页面;`driver=mobile` 必须挂模拟器 session。二者都不得改成只打 API。
- `automation=assisted|manual` 需要人在场:无人值守时记 `not-run`,不要自己代跑人工步骤。isolated-full 取消的是风险/授权过滤,不会伪造人工操作。见 [concepts.md](references/concepts.md)。
+1 -1
View File
@@ -84,7 +84,7 @@ catalog(整个测试集)
- 与 `gate` 正交:`risk` 说「影响多大」,`gate` 说「碰哪个子系统」。点动 vs 焊接这种细分授权用 `gate`,不要再造 risk 名。
- `tag=smoke` 只能配 `risk=readonly`:冒烟测试不许留下变化。
- 隔离可重置环境可设置 `execution_profile: isolated-full` 跑所有 active 用例;安全前置由
test-preflight capability 与 cleanup/rollback 契约保证,原始 risk 仍必须报告。
test-preflight 的设备、服务与 cleanup/rollback 契约保证,原始 risk 仍必须报告;客户端能力由 journey 用例验证。
### 3.2 `automation`:为什么是 `automated / assisted / manual`
+2 -2
View File
@@ -141,8 +141,8 @@ preflight 通过后,先按 catalog 顺序执行 `integration`,再按客户
- `HOST` 未设(环境变量 / `local.yaml` 都没有):全部停,environment-fail。
- `standard`:命中条的 `risk` 在 catalog `authorize` 里有对应变量时,变量不是 `true` 则该条 `skipped`;有 `gate` 时还要对应变量为 `true`,`assume` 不成立同样 `skipped`。即使用户点了 id 也不执行。
- `isolated-full`:不读取 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM`、`AUTHORIZE_PASSWORD` 等人工开关;相册、照片权限/测试图片和中文拼音真实 composition/candidate 能力必须由 test-preflight 证明。
- standard 中授权/假设不成立是**主动不跑**(`skipped`);isolated-full 的能力缺失是 preflight blocked/environment-fail,并为全部 testcase 写 `not-run` + `blocked_by: [test-preflight]`。
- `isolated-full`:不读取 `AUTHORIZE_MUTATING`、`AUTHORIZE_DESTRUCTIVE`、`AUTHORIZE_ALBUM`、`AUTHORIZE_PASSWORD` 等人工开关;相册、照片权限/测试图片和中文拼音能力默认具备,由 journey 用例真实验证,失败记为产品失败。
- standard 中授权/假设不成立是**主动不跑**(`skipped`);isolated-full 不因客户端能力授权变量停止,设备、产物、服务或清理前置失败时才为全部 testcase 写 `not-run` + `blocked_by: [test-preflight]`。
- `status_probe` 是**分类级前置条件**(业界对应 pytest 的 session fixture / Robot 的 Suite Setup)。用 `select` 返回的 `status_probe` 解析生效值,不要自己读 catalog 猜。返回的 `probe.url` 已用 `vars` 展开,直接请求即可;`select` 同批返回 `vars`:
| `categories[].status_probe` | 生效探测 |
+3 -3
View File
@@ -25,7 +25,7 @@ runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对
| `categories` | `{id, title, status_probe?}` 列表,id 与 `cases/<id小写>.yaml` 对应;列表顺序即 `select` / 执行顺序。`status_probe` 见下 |
| `risks` | 从低到高,只能是 `readonly` → `mutating` → `destructive` 的一段,且以 `readonly` 开头。含义见 [levels.md](levels.md) |
| `default_max_risk` | 未指定时的运行上限,必须是 `risks` 里的值 |
| `execution_profile` | `standard`(默认)或 `isolated-full`。后者只用于隔离可重置环境,保留 risk 语义但不按 risk/authorize/gate 过滤 active 用例;安全边界由 preflight capability 与 cleanup/rollback 契约承担 |
| `execution_profile` | `standard`(默认)或 `isolated-full`。后者只用于隔离可重置环境,保留 risk 语义但不按 risk/authorize/gate 过滤 active 用例;设备、服务和 cleanup/rollback 契约仍由 preflight 检查,客户端能力由 journey 真实验证 |
| `vars` | 可替换变量,如 `API_BASE: http://${HOST}:8095` |
| `artifacts` | 可选的移动产物身份契约。按 `android` / `ios` 声明 `required` 和 `package` / `bundle_id`;每轮文件路径或下载地址由调用方提供,不提交临时产物路径。 |
| `authorize` | 风险名 → 环境变量名。该档执行前变量必须为 `true`(`readonly` 不需要) |
@@ -38,8 +38,8 @@ runs/*.csv 同一次运行的成绩单;generate_run_csv.py 从对
| `integration_suites` | 可选的集成套件声明;标准值为 `api-contract`、`authorization`、`build-contract` |
`isolated-full` 不得把风险字段改成 `readonly`,也不得把 `AUTHORIZE_ALBUM`、
`AUTHORIZE_PASSWORD`、`IME_PINYIN` 等变量当成能力门禁。移动用例必须通过 test-preflight
的真实相册 picker、照片权限/测试图片和中文拼音 composition/candidate 检查。
`AUTHORIZE_PASSWORD`、`IME_PINYIN` 等变量当成门禁。移动用例必须通过 test-preflight
的设备、产物、服务和网络检查;相册、权限和输入法能力默认具备,交由 journey 用例验证。
`local.yaml` 与环境变量覆盖 `HOST` 和授权开关。模板中的 `HOST` 默认是
`127.0.0.1`;远程目标必须显式覆盖。查找顺序:环境变量 > `local.yaml` >