feat: add deployer
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
# deployer
|
||||
|
||||
把「一堆 VPS / NAS 上的 Docker 服务」变成一个 Git 仓库就能管的事:仓库里只放服务配置(数据),
|
||||
部署、同步、升级的方法和脚本全部由这个 skill 自带,换台电脑、换个项目都能直接用。
|
||||
|
||||
支持两种用法:
|
||||
|
||||
- **独立配置中心**:一个专门的 Git 仓库管所有机器的所有服务(如 app00)
|
||||
- **项目内环境**:在普通项目里放 `.skiff/deployer/{prod,test,dev}/`,
|
||||
把这个项目的生产/测试/开发环境也用同一套流程部署
|
||||
|
||||
## 什么时候使用
|
||||
|
||||
- 想用一套固定流程把本地改好的 Docker Compose 配置发到某台服务器
|
||||
- 要升级某个服务的镜像版本、重启服务、看远程容器状态和日志
|
||||
- 新加一个服务、把服务从一台机器挪到另一台、或下线旧服务
|
||||
- 想给当前项目加 prod/test/dev 三套远程环境并随时部署其中一套
|
||||
- 需要一张「哪台机器跑哪些服务」的清单
|
||||
|
||||
## 使用前准备
|
||||
|
||||
- 本机装有 Python 3、`rsync`、`ssh`
|
||||
- 目标机器装好 Docker + Docker Compose v2
|
||||
- `~/.ssh/config` 里为每台机器配好 Host 别名,且能免密(或 agent)登录
|
||||
- 知道每个服务的运行时数据放在哪(这些目录不能被同步覆盖)
|
||||
|
||||
## 使用示例
|
||||
|
||||
```text
|
||||
# 独立配置中心
|
||||
帮我把 vyyo1/naiveproxy 的配置改完部署上去
|
||||
升级 vora3/gpt-load 的镜像版本
|
||||
列一下现在所有服务和各自在哪台机器上
|
||||
新增一个服务 uptime 到 vora3,先帮我建好目录结构
|
||||
vhom1 上那个 naiveproxy 为什么 sync 失败?
|
||||
|
||||
# 项目内环境
|
||||
给这个项目建好 .skiff/deployer,prod 和 test 分别放到两台机器上
|
||||
把 test 环境重新部署一下
|
||||
prod 的 compose 加个 redis,改完发上去
|
||||
```
|
||||
|
||||
## Agent 会做什么
|
||||
|
||||
1. 读服务/环境目录(及共享的父目录)的 `_config.yaml`,确定目标机器和远程路径;
|
||||
项目内布局从 `.skiff/deployer/` 自动发现,无需额外配置
|
||||
2. 用 skill 自带脚本把本地目录同步到远程(rsync,自动排除 `data/`、`_data/`)
|
||||
3. 在远程执行对应的 `docker compose` 操作(启动 / 重建 / 升级 / 重启)
|
||||
4. 同步后查看容器状态和日志确认生效
|
||||
5. 只针对你指定的那一个服务操作,不会批量动整台机器
|
||||
|
||||
项目内布局下,远程目录名自动带上项目前缀(如 `my-project-prod`),
|
||||
避免同一台机器上多个项目的同名环境互相覆盖;需要固定名字时在 `_config.yaml` 写 `name:`。
|
||||
|
||||
重要边界:同步使用 `--delete`,远程多余的文件会被删除;数据库、证书等运行时数据
|
||||
必须放在排除目录或远程绝对路径挂载。涉及删除数据卷、清理远程文件的操作会先向你确认。
|
||||
|
||||
## 如何判断完成
|
||||
|
||||
- 脚本输出显示同步完成、远程命令执行成功
|
||||
- `ps` 显示容器 Up、`logs` 无报错;升级后镜像 tag 与配置一致
|
||||
- 域名/端口类服务能 curl 通
|
||||
@@ -0,0 +1,200 @@
|
||||
---
|
||||
name: deployer
|
||||
description: >-
|
||||
管理多 VPS / NAS 的 Docker Compose 配置中心:仓库只存服务数据(compose.yaml、静态配置),
|
||||
部署方法与脚本由本 skill 提供。当用户要求部署、同步、升级、重启远程 Docker 服务,
|
||||
新增/迁移/下线服务,梳理节点与服务清单,或提到 make sync/deploy/upgrade/TGT、_config.yaml、
|
||||
rsync 同步、tar over SSH、Synology NAS 部署失败时使用。
|
||||
---
|
||||
|
||||
# deployer:多机 Compose 配置中心
|
||||
|
||||
仓库 = 数据(各机器的 `compose.yaml` 与静态配置);方法 = 本 skill 的脚本与规范。
|
||||
本地改配置 → skill 脚本同步到对应 SSH 节点 → 远程 `docker compose` 应用。
|
||||
|
||||
---
|
||||
|
||||
## 何时使用
|
||||
|
||||
- 部署 / 同步 / 升级 / 重启某个远程 Docker Compose 服务
|
||||
- 新增、迁移、下线一个服务;梳理「哪台机器跑什么」
|
||||
- sync 失败排查、证书丢失、改了配置不生效等运维问题
|
||||
- 提到 `make deploy TGT=...`、`TGT=`、`_config.yaml`、rsync/tar 同步
|
||||
|
||||
## 不适用
|
||||
|
||||
- 单机 docker 日常使用(无多机同步诉求)
|
||||
- K8s / Nomad 等编排系统
|
||||
- CI/CD 流水线构建发布(本流程是 push 式运维,不是流水线)
|
||||
|
||||
---
|
||||
|
||||
## 核心模型(先读懂再动手)
|
||||
|
||||
- **仓库只放数据**:`compose.yaml`、Caddyfile、Traefik 动态配置等静态配置进 Git;
|
||||
运行时数据(证书、数据库、上传文件)永不进 Git,也永不参与同步范围。
|
||||
- **每个可部署服务目录必须有 `compose.yaml`**,且能解析出目标节点 `node`
|
||||
(来自该目录、部署根或祖先目录的 `_config.yaml`,或父目录名恰为 SSH Host 别名)。
|
||||
- **`node` 即 SSH Host 别名**(`~/.ssh/config`),支持 `user@host` 形式。
|
||||
- `unused/` 下不参与自动发现与部署。
|
||||
|
||||
### 两种布局
|
||||
|
||||
**A. 独立配置中心仓库**(如 app00):仓库根即部署根,
|
||||
`DEPLOYER_ROOT=/path/to/repo` 指定后按仓库内相对路径操作:
|
||||
|
||||
```
|
||||
repo/
|
||||
├── _config.yaml # 可选,全局默认
|
||||
├── vyyo1/_config.yaml # node: vyyo1(主机目录)
|
||||
│ └── naiveproxy/ # 服务目录:compose.yaml + 可选 _config.yaml
|
||||
└── unused/
|
||||
```
|
||||
|
||||
远程目录名 = 目录末级名:`vyyo1/naiveproxy` → `/opt/app/naiveproxy`。
|
||||
|
||||
**B. 项目内环境布局**:项目根放 `.skiff/deployer/{prod,test,dev}/`,
|
||||
每个环境一个目录。从项目内任意位置运行脚本即自动发现(也可用 `DEPLOYER_ROOT`
|
||||
显式指定),无需环境变量:
|
||||
|
||||
```
|
||||
my-project/
|
||||
├── src/ ... # 项目本体
|
||||
└── .skiff/deployer/
|
||||
├── _config.yaml # 三个环境共享默认(node/base_path 等)
|
||||
├── prod/
|
||||
│ ├── compose.yaml # 生产 compose 与配置
|
||||
│ └── _config.yaml # 环境级覆盖
|
||||
├── test/compose.yaml
|
||||
└── dev/compose.yaml
|
||||
```
|
||||
|
||||
项目模式下远程目录名自动加项目前缀 `{git仓库名}-{env}`
|
||||
(如 `my-project-prod`),防止同主机多项目的同名环境互相覆盖;
|
||||
`_config.yaml` 写 `name:` 可显式指定。
|
||||
|
||||
## 步骤
|
||||
|
||||
### 0. 定位部署根
|
||||
|
||||
skill 目录下的 `scripts/deploy/` 是通用部署工具链(lib/sync/remote/list),
|
||||
不依赖具体项目路径。部署根按以下顺序解析:
|
||||
|
||||
1. 环境变量 `DEPLOYER_ROOT` 显式指定(独立配置中心仓库用这个)
|
||||
2. 从当前目录向上找 `.skiff/deployer/`(项目内环境布局自动发现)
|
||||
3. skill 安装位置兜底(仅用于查看,没有可部署服务)
|
||||
|
||||
```bash
|
||||
# <skill-dir> = 本 SKILL.md 所在目录,先解析出来记下
|
||||
# 布局 A:显式指定仓库根
|
||||
export DEPLOYER_ROOT=/path/to/your/compose-repo
|
||||
python3 <skill-dir>/scripts/deploy/list.py
|
||||
|
||||
# 布局 B:在项目内直接跑即可(cwd 在项目里)
|
||||
python3 <skill-dir>/scripts/deploy/list.py
|
||||
```
|
||||
|
||||
### 1. 摸底:列出服务与节点
|
||||
|
||||
上一步的 `list.py` 输出全部服务与节点分布;新增环境/服务后重跑确认被发现。
|
||||
项目布局下 `prod/test/dev` 各显示为 `{项目名}-{env}`。
|
||||
|
||||
### 2. 解析单个服务
|
||||
|
||||
```bash
|
||||
# 查看 node、远程路径、排除规则(sync.py 干跑会打印这些信息)
|
||||
python3 <skill-dir>/scripts/deploy/sync.py <service-path>
|
||||
```
|
||||
|
||||
或直接读服务目录及祖先的 `_config.yaml`。
|
||||
|
||||
### 3. 命令选择(语义严格区分)
|
||||
|
||||
| 意图 | 命令 |
|
||||
|------|------|
|
||||
| 只同步文件,不动容器 | `sync.py <svc>` |
|
||||
| 应用 compose/配置变更 | `sync.py <svc> && remote.py <svc> up` |
|
||||
| 改配置后强制重建 | `remote.py <svc> recreate`(配合前置 sync) |
|
||||
| 镜像 tag 变更升级 | `sync.py <svc> && remote.py <svc> upgrade` |
|
||||
| 仅重启,不同步文件 | `remote.py <svc> restart` |
|
||||
| 排查 | `remote.py <svc> ps` / `remote.py <svc> logs` |
|
||||
|
||||
### 4. 项目侧 Makefile(可选薄封装)
|
||||
|
||||
若项目有 Makefile 封装,命令形如 `make deploy TGT=<服务路径>`。
|
||||
没有 Makefile 时直接调 python 脚本即可,不要新建封装层。
|
||||
|
||||
### 5. 新增服务 / 环境 checklist
|
||||
|
||||
独立仓库布局:
|
||||
|
||||
1. 在合适分类目录创建服务文件夹,写 `compose.yaml`
|
||||
2. 在服务目录或祖先目录放 `_config.yaml`(至少能解析出 `node`)
|
||||
3. 有运行时目录 → 加进 `sync_exclude`
|
||||
4. 远程首次建目录:`ssh <node> "mkdir -p <base_path>/<name>"`
|
||||
5. 首次部署:sync + up
|
||||
6. 验证:ps + logs,必要时 curl/ssh 检查端口
|
||||
|
||||
项目环境布局:
|
||||
|
||||
1. 项目根建 `.skiff/deployer/{env}/`(env 通常为 prod/test/dev)
|
||||
2. 每个环境写 `compose.yaml`;三个环境共享的 node/base_path 放
|
||||
`.skiff/deployer/_config.yaml`
|
||||
3. 环境有差异(不同主机、不同排除项)→ 在该环境的 `_config.yaml` 覆盖
|
||||
4. 同名冲突或需要固定远程目录名 → `_config.yaml` 写 `name:`
|
||||
5. 首次部署前确认目标主机的远程目录不存在旧内容(rsync `--delete` 会清掉)
|
||||
|
||||
### 6. 下线服务
|
||||
|
||||
独立仓库布局:配置移入 `unused/`(自动脱离发现体系),远程按需手动清理:
|
||||
`ssh <node> "cd <base_path>/<name> && docker compose down"`,数据卷按需保留或删除。
|
||||
项目环境布局:删除对应 `.skiff/deployer/{env}/` 目录即可脱离发现体系,远程清理同上。
|
||||
|
||||
---
|
||||
|
||||
## 注意事项
|
||||
|
||||
- **禁止节点级批量操作**:所有 sync/up/recreate/upgrade/restart 必须按单服务执行。
|
||||
批量升级风险过高,逐个来。
|
||||
- **rsync 带 `--delete`**:远程多余文件会被删除。运行时数据必须放在
|
||||
默认排除的 `data/`、`_data/`,或 compose 挂载的远程绝对路径
|
||||
(如 `/data01/docker/<svc>/`),否则会被清掉。
|
||||
- **镜像固定 tag**,不用 `:latest` 漂移;成对升级的服务(如 proxy 客户端/服务端)要同步升。
|
||||
- **密钥**:优先放远程 `.env` 或环境变量,不要提交新密钥进 Git。
|
||||
- **Git 安全**:不 `--force` 推送、不硬 reset,除非用户明确要求。
|
||||
- **NAS / Synology 特例**:部分 NAS 的 SSH 用户禁用 rsync 协议(Permission denied)。
|
||||
表现是 sync 报错但 ssh 正常。处理顺序:
|
||||
1. 该节点 `_config.yaml` 写真实 `base_path`(如 `/volume1/docker`,避开符号链接路径)
|
||||
2. 仍失败则手动 tar over SSH 推送:
|
||||
|
||||
```bash
|
||||
tar czf - -C <服务目录> . --exclude='data' --exclude='_data' \
|
||||
| ssh <node> "mkdir -p <base_path>/<name> && cd <base_path>/<name> && tar xzf -"
|
||||
ssh <node> "cd <base_path>/<name> && /usr/local/bin/docker compose up -d"
|
||||
```
|
||||
|
||||
tar 不会删除远程多余文件;需清理旧文件时手动 SSH 删除。
|
||||
3. Synology 上 docker 路径可能是 `/usr/local/bin/docker`
|
||||
|
||||
## 验证
|
||||
|
||||
- `list.py` 输出全部服务与节点分布,数量与预期一致
|
||||
- 每次 sync/deploy 后 `remote.py <svc> ps` 容器 Up、`logs` 无报错
|
||||
- 升级后额外确认镜像 tag 与 compose.yaml 一致
|
||||
- 改 Traefik/Caddy 路由后 curl 对应域名验证生效
|
||||
|
||||
## scripts/
|
||||
|
||||
|
||||
| 文件 | 用途 |
|
||||
|------|------|
|
||||
| `scripts/deploy/lib.py` | 解析服务目录、合并继承 `_config.yaml`、SSH/rsync 参数构造 |
|
||||
| `scripts/deploy/sync.py` | rsync -avz --delete 同步;无 rsync 时 tar over SSH 兜底 |
|
||||
| `scripts/deploy/remote.py` | SSH 远程 docker compose:up/recreate/restart/upgrade/ps/logs |
|
||||
| `scripts/deploy/list.py` | 扫描全部可部署服务 |
|
||||
|
||||
## references/
|
||||
|
||||
| 文件 | 用途 |
|
||||
|------|------|
|
||||
| `references/config-reference.md` | `_config.yaml` 字段完整说明与继承合并规则 |
|
||||
@@ -0,0 +1,137 @@
|
||||
# `_config.yaml` 配置参考
|
||||
|
||||
`_config.yaml` 供 skill 部署脚本解析,决定同步目标与排除规则。可放在**服务目录、部署根或其任意祖先目录**;子目录中的字段覆盖父目录(继承合并)。
|
||||
|
||||
## 放置位置(两种布局)
|
||||
|
||||
| 布局 | 部署根 | `_config.yaml` 典型位置 |
|
||||
|------|--------|------------------------|
|
||||
| 独立配置中心仓库(`DEPLOYER_ROOT` 指向) | 仓库根 | 主机目录 `vyyo1/_config.yaml`、服务目录 |
|
||||
| 项目内环境 `.skiff/deployer/{env}/` | `.skiff/deployer/` | 根级共享默认、各环境目录覆盖 |
|
||||
|
||||
项目布局示例:
|
||||
|
||||
```yaml
|
||||
# .skiff/deployer/_config.yaml — 三个环境共享
|
||||
node: my-vps
|
||||
base_path: /srv/apps
|
||||
|
||||
# .skiff/deployer/prod/_config.yaml — 仅生产环境差异
|
||||
node: prod-vps # 覆盖父级
|
||||
name: my-project-api # 可选,覆盖默认的 {项目名}-{env}
|
||||
```
|
||||
|
||||
## 字段一览
|
||||
|
||||
| 字段 | 必填 | 可继承 | 说明 |
|
||||
|------|------|--------|------|
|
||||
| `node` | 是 | 是 | SSH 目标主机别名 |
|
||||
| `port` | 否 | 是 | SSH 端口,默认 22 |
|
||||
| `identity_file` | 否 | 是 | SSH 私钥路径 |
|
||||
| `base_path` | 否 | 是 | 远程部署根目录,默认 `/opt/app` |
|
||||
| `sync_exclude` | 否 | 是 | rsync 排除目录列表 |
|
||||
| `name` | 否 | 是 | 远程目录名覆盖(见下) |
|
||||
|
||||
## name(可选)
|
||||
|
||||
远程目录名默认取服务/环境文件夹最后一级名称。**项目内环境布局**默认改为
|
||||
`{git仓库名}-{env}`(如 `my-project-prod`),防止同主机多项目的同名环境互相覆盖。
|
||||
写 `name:` 显式指定最终远程目录名(不含 base_path 部分)。
|
||||
|
||||
## node(必填)
|
||||
|
||||
对应 `~/.ssh/config` 中的 `Host` 别名,支持带用户形式:
|
||||
|
||||
```yaml
|
||||
node: deploy@prod
|
||||
```
|
||||
|
||||
独立仓库布局默认远程目录名 = 服务文件夹最后一级名称:`vyyo1/naiveproxy` → `{base_path}/naiveproxy`;
|
||||
项目环境布局见上文 name 一节。
|
||||
兜底规则:合并后仍无 `node` 时,若**直接父目录名**是 SSH Host 别名则作为 node。
|
||||
|
||||
|
||||
## port(可选)
|
||||
|
||||
```yaml
|
||||
node: edge
|
||||
port: 2222
|
||||
```
|
||||
|
||||
sync 通过 `rsync -e "ssh -p PORT"` 连接,remote 使用 `ssh -p PORT`。
|
||||
与 `~/.ssh/config` 的 `Port` 同时存在时以 `_config.yaml` 为准(命令行覆盖 config)。
|
||||
|
||||
## identity_file(可选)
|
||||
|
||||
```yaml
|
||||
node: edge
|
||||
identity_file: ~/.ssh/deploy_ed25519
|
||||
```
|
||||
|
||||
须为绝对路径或 `~` 开头;私钥不进 Git。带 passphrase 的密钥需事先加入 ssh-agent,脚本不支持交互输入。
|
||||
|
||||
## base_path(可选)
|
||||
|
||||
```yaml
|
||||
node: nas
|
||||
base_path: /volume1/docker # 默认路径是符号链接的设备写真实路径
|
||||
```
|
||||
|
||||
实际同步目标 `{base_path}/{服务名}`。
|
||||
|
||||
## sync_exclude(可选)
|
||||
|
||||
rsync 排除的目录名列表(仅目录名)。默认已排除 `data`、`_data`。
|
||||
子级重新定义时**整体替换**父级列表(非追加):
|
||||
|
||||
```yaml
|
||||
sync_exclude:
|
||||
- data
|
||||
- _data
|
||||
- logs
|
||||
```
|
||||
|
||||
运行时数据放这些目录或 compose 挂载的远程绝对路径(如 `/data01/docker/<svc>/`),避免被 `--delete` 清掉。
|
||||
|
||||
---
|
||||
|
||||
## 继承合并规则
|
||||
|
||||
1. 收集**部署根自身**的 `_config.yaml`(全局默认),再收集部署根到服务目录路径上的所有 `_config.yaml`
|
||||
2. 按祖先 → 子顺序合并;同名字段后者覆盖前者;`sync_exclude` 整体替换
|
||||
3. 无任何配置且父目录名为 SSH Host 别名 → 兜底为 node
|
||||
|
||||
## 配置组织建议
|
||||
|
||||
| 场景 | 建议 |
|
||||
|------|------|
|
||||
| 同一 SSH 主机多个服务 | 主机目录写一份 `node`/`base_path`,子服务免重复 |
|
||||
| 各服务目标不同 | 服务目录单独写 `_config.yaml` |
|
||||
| 个别覆盖 | 子目录只写差异字段 |
|
||||
| 项目三环境同主机 | `.skiff/deployer/_config.yaml` 写共享 node,各环境只放差异 |
|
||||
| 项目环境分属不同主机 | 各环境 `_config.yaml` 分别写 `node` |
|
||||
|
||||
## 示例
|
||||
|
||||
```yaml
|
||||
# hosts/nas/_config.yaml — 该节点下所有服务共享
|
||||
node: nas
|
||||
base_path: /volume1/docker
|
||||
|
||||
# hosts/web1/_config.yaml
|
||||
node: web1
|
||||
port: 2222
|
||||
identity_file: ~/.ssh/web1_ed25519
|
||||
|
||||
# hosts/web2/special/_config.yaml — 覆盖父级的 node
|
||||
node: deploy@web2-backup
|
||||
```
|
||||
|
||||
## 可部署服务判定
|
||||
|
||||
同时满足才被 `list.py` 发现:
|
||||
|
||||
- 目录内有 `compose.yaml`
|
||||
- 能通过继承或兜底解析出 `node`
|
||||
- 独立仓库布局:路径中不含 `unused/` 段
|
||||
- 项目环境布局:位于部署根 `.skiff/deployer/` 内(其外的项目文件不扫描)
|
||||
Executable
+334
@@ -0,0 +1,334 @@
|
||||
#!/usr/bin/env python3
|
||||
"""deployer 部署工具公共模块:解析部署根、服务目录与 _config.yaml。
|
||||
|
||||
支持两种布局:
|
||||
1. 独立配置中心仓库(DEPLOYER_ROOT 指向,或 skill 安装位置)
|
||||
2. 项目内环境目录 .skiff/deployer/{prod,test,dev}/(从 CWD 自动发现)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
DEFAULT_BASE_PATH = "/opt/app"
|
||||
DEFAULT_SYNC_EXCLUDES = ("data", "_data")
|
||||
_SKILL_DIR = Path(__file__).resolve().parent.parent # scripts/
|
||||
PROJECT_ROOT: Path | None = None
|
||||
|
||||
|
||||
def _find_project_root() -> Path:
|
||||
"""部署根:DEPLOYER_ROOT > 从 CWD 向上找 .skiff/deployer > skill 安装位置。"""
|
||||
env = os.environ.get("DEPLOYER_ROOT", "").strip()
|
||||
if env:
|
||||
p = Path(env).expanduser().resolve()
|
||||
if not p.is_dir():
|
||||
print(f"错误: DEPLOYER_ROOT 不是目录: {p}")
|
||||
sys.exit(1)
|
||||
return p
|
||||
cur = Path.cwd()
|
||||
while True:
|
||||
cand = cur / ".skiff" / "deployer"
|
||||
if cand.is_dir():
|
||||
return cand
|
||||
if cur == cur.parent:
|
||||
break
|
||||
cur = cur.parent
|
||||
return _SKILL_DIR.parent
|
||||
|
||||
|
||||
def project_root() -> Path:
|
||||
global PROJECT_ROOT
|
||||
if PROJECT_ROOT is None:
|
||||
PROJECT_ROOT = _find_project_root()
|
||||
return PROJECT_ROOT
|
||||
|
||||
|
||||
def in_project_layout(root: Path | None = None) -> bool:
|
||||
"""部署根是否为某项目内的 .skiff/deployer/。"""
|
||||
root = root or project_root()
|
||||
return root.name == "deployer" and root.parent.name == ".skiff"
|
||||
|
||||
|
||||
def project_display_name(root: Path | None = None) -> str:
|
||||
"""项目名:git 仓库名优先,否则 .skiff 的父目录名。"""
|
||||
root = root or project_root()
|
||||
anchor = root.parent.parent if in_project_layout(root) else root
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
["git", "-C", str(anchor), "rev-parse", "--show-toplevel"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if proc.returncode == 0:
|
||||
return Path(proc.stdout.strip()).name
|
||||
except OSError:
|
||||
pass
|
||||
return anchor.name
|
||||
|
||||
|
||||
def load_config(config_path: os.PathLike | str, *, required: bool = True) -> dict:
|
||||
"""解析精简版 _config.yaml(仅支持本项目使用的字段)。"""
|
||||
config: dict = {}
|
||||
current_list_key: str | None = None
|
||||
|
||||
try:
|
||||
with open(config_path, encoding="utf-8") as f:
|
||||
for raw_line in f:
|
||||
line = raw_line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
|
||||
if line.startswith("- ") and current_list_key:
|
||||
config.setdefault(current_list_key, []).append(line[2:].strip())
|
||||
continue
|
||||
|
||||
current_list_key = None
|
||||
if ":" not in line:
|
||||
continue
|
||||
|
||||
key, value = line.split(":", 1)
|
||||
key = key.strip()
|
||||
value = value.strip()
|
||||
if not value:
|
||||
current_list_key = key
|
||||
config.setdefault(key, [])
|
||||
continue
|
||||
|
||||
config[key] = value
|
||||
except FileNotFoundError:
|
||||
if required:
|
||||
print(f"错误: 配置文件不存在: {config_path}")
|
||||
sys.exit(1)
|
||||
return {}
|
||||
|
||||
return config
|
||||
|
||||
|
||||
def config_paths_for_service(service_dir: str) -> list[Path]:
|
||||
"""收集部署根自身及服务目录各层 _config.yaml(祖先在前,服务目录在后)。
|
||||
|
||||
部署根的 _config.yaml(如 .skiff/deployer/_config.yaml)作为全局默认,
|
||||
对所有环境/服务生效。
|
||||
"""
|
||||
root = project_root()
|
||||
rel = Path(service_dir)
|
||||
paths: list[Path] = []
|
||||
root_config = root / "_config.yaml"
|
||||
if root_config.is_file():
|
||||
paths.append(root_config)
|
||||
for depth in range(1, len(rel.parts) + 1):
|
||||
config_path = root / Path(*rel.parts[:depth]) / "_config.yaml"
|
||||
if config_path.is_file() and config_path != root_config:
|
||||
paths.append(config_path)
|
||||
return paths
|
||||
|
||||
|
||||
def merge_service_config(service_dir: str) -> tuple[dict, list[Path]]:
|
||||
"""合并服务目录及其祖先的 _config.yaml,子级覆盖父级。"""
|
||||
merged: dict = {}
|
||||
sources = config_paths_for_service(service_dir)
|
||||
for config_path in sources:
|
||||
merged.update(load_config(config_path, required=True))
|
||||
return merged, sources
|
||||
|
||||
|
||||
def ssh_config_hosts() -> set[str]:
|
||||
"""读取 ~/.ssh/config 中的 Host 别名(不含通配符)。"""
|
||||
global _SSH_HOSTS
|
||||
if _SSH_HOSTS is not None:
|
||||
return _SSH_HOSTS
|
||||
|
||||
hosts: set[str] = set()
|
||||
config_path = Path.home() / ".ssh" / "config"
|
||||
if config_path.is_file():
|
||||
for raw_line in config_path.read_text(encoding="utf-8").splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
if not line.lower().startswith("host "):
|
||||
continue
|
||||
for host in line.split()[1:]:
|
||||
if "*" in host or "?" in host or "!" in host:
|
||||
continue
|
||||
hosts.add(host)
|
||||
|
||||
_SSH_HOSTS = hosts
|
||||
return hosts
|
||||
|
||||
|
||||
_SSH_HOSTS: set[str] | None = None
|
||||
|
||||
|
||||
def parse_port(raw: str, *, strict: bool = True) -> int | None:
|
||||
"""解析 SSH 端口,无效时 strict 模式下退出。"""
|
||||
try:
|
||||
port = int(raw)
|
||||
except ValueError:
|
||||
if strict:
|
||||
print(f"错误: port 必须是整数: {raw!r}")
|
||||
sys.exit(1)
|
||||
return None
|
||||
if not 1 <= port <= 65535:
|
||||
if strict:
|
||||
print(f"错误: port 超出有效范围 1-65535: {port}")
|
||||
sys.exit(1)
|
||||
return None
|
||||
return port
|
||||
|
||||
|
||||
def resolve_identity_file(raw: str, *, strict: bool = True) -> str | None:
|
||||
"""解析 SSH 私钥路径(expanduser,须为绝对路径且文件存在)。"""
|
||||
path = os.path.expanduser(raw)
|
||||
if not os.path.isabs(path):
|
||||
if strict:
|
||||
print(f"错误: identity_file 必须是绝对路径或 ~ 开头: {raw!r}")
|
||||
sys.exit(1)
|
||||
return None
|
||||
if not os.path.isfile(path):
|
||||
if strict:
|
||||
print(f"错误: identity_file 不存在: {path}")
|
||||
sys.exit(1)
|
||||
return None
|
||||
return path
|
||||
|
||||
|
||||
def ssh_base_args(info: dict) -> list[str]:
|
||||
"""ssh 可执行文件及 -p / -i 等选项(不含 host 与 remote command)。"""
|
||||
args = ["ssh"]
|
||||
port = info.get("port")
|
||||
if port is not None:
|
||||
args.extend(["-p", str(port)])
|
||||
identity_file = info.get("identity_file")
|
||||
if identity_file:
|
||||
args.extend(["-i", identity_file])
|
||||
return args
|
||||
|
||||
|
||||
def ssh_cmd(info: dict, remote_command: str) -> list[str]:
|
||||
"""构建 ssh 命令行(含可选 -p / -i)。"""
|
||||
return [*ssh_base_args(info), info["node"], remote_command]
|
||||
|
||||
|
||||
def rsync_ssh_args(info: dict) -> list[str]:
|
||||
"""rsync 需自定义 ssh 时通过 -e 传入 port / identity_file。"""
|
||||
base = ssh_base_args(info)
|
||||
if len(base) == 1:
|
||||
return []
|
||||
return ["-e", shlex.join(base)]
|
||||
|
||||
|
||||
def node_from_parent_dir(service_dir: str) -> str | None:
|
||||
"""父目录名若是 SSH Host 别名,则作为 node。"""
|
||||
parent = Path(service_dir.rstrip("/")).parent.name
|
||||
if not parent or parent == ".":
|
||||
return None
|
||||
if parent in ssh_config_hosts():
|
||||
return parent
|
||||
return None
|
||||
|
||||
|
||||
def service_info(service_dir: str, *, strict: bool = True) -> dict | None:
|
||||
"""解析单个服务目录,返回 node、远程路径等信息。"""
|
||||
root = project_root()
|
||||
abs_dir = (root / service_dir).resolve()
|
||||
if not abs_dir.is_dir():
|
||||
if strict:
|
||||
print(f"错误: 服务目录不存在: {service_dir}")
|
||||
sys.exit(1)
|
||||
return None
|
||||
|
||||
env_name = abs_dir.name
|
||||
rel_dir = str(abs_dir.relative_to(root))
|
||||
config, sources = merge_service_config(rel_dir)
|
||||
|
||||
# 远程目录名:_config.yaml 的 name 显式覆盖;项目布局默认 {项目名}-{env}
|
||||
# 防止同主机上多个项目的 prod/test 相互覆盖
|
||||
if "name" in config:
|
||||
name = str(config["name"])
|
||||
elif in_project_layout(root):
|
||||
name = f"{project_display_name(root)}-{env_name}"
|
||||
if strict:
|
||||
print(f"提示: 项目模式,远程名自动加前缀: {name!r}(_config.yaml 写 name: 可覆盖)")
|
||||
else:
|
||||
name = env_name
|
||||
|
||||
node = config.get("node")
|
||||
if not node:
|
||||
node = node_from_parent_dir(rel_dir)
|
||||
if node and strict and not config:
|
||||
print(f"提示: 未找到 _config.yaml,使用父目录 SSH 主机 {node!r}")
|
||||
|
||||
if not node:
|
||||
if strict:
|
||||
print(f"错误: 无法解析 node: {service_dir}")
|
||||
print(" 请在服务目录或其祖先目录添加 _config.yaml,或确保父目录是 SSH 主机别名")
|
||||
sys.exit(1)
|
||||
return None
|
||||
|
||||
base_path = config.get("base_path", DEFAULT_BASE_PATH)
|
||||
excludes = config.get("sync_exclude") or list(DEFAULT_SYNC_EXCLUDES)
|
||||
remote_dir = f"{base_path}/{name}"
|
||||
port = parse_port(config["port"], strict=strict) if "port" in config else None
|
||||
identity_file = (
|
||||
resolve_identity_file(config["identity_file"], strict=strict)
|
||||
if "identity_file" in config
|
||||
else None
|
||||
)
|
||||
|
||||
info = {
|
||||
"service_dir": str(abs_dir),
|
||||
"name": name,
|
||||
"node": node,
|
||||
"port": port,
|
||||
"identity_file": identity_file,
|
||||
"base_path": base_path,
|
||||
"remote_dir": remote_dir,
|
||||
"sync_exclude": excludes,
|
||||
"config_sources": [str(p.relative_to(root)) for p in sources],
|
||||
}
|
||||
local_config = abs_dir / "_config.yaml"
|
||||
if strict and sources and not local_config.is_file():
|
||||
rel_sources = info["config_sources"]
|
||||
if len(rel_sources) == 1:
|
||||
print(f"提示: 使用继承配置 {rel_sources[0]!r}")
|
||||
else:
|
||||
print(f"提示: 使用继承配置 {' -> '.join(rel_sources)!r}")
|
||||
|
||||
return info
|
||||
|
||||
|
||||
def is_deployable_dir(path: Path, root: Path) -> bool:
|
||||
if "unused" in path.parts or "__pycache__" in path.parts:
|
||||
return False
|
||||
if ".skiff" in path.parts and root.name != "deployer":
|
||||
return False
|
||||
if not (path / "compose.yaml").is_file():
|
||||
return False
|
||||
try:
|
||||
path.relative_to(root)
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def discover_services(node: str | None = None) -> list[dict]:
|
||||
"""扫描仓库内所有带 compose.yaml 且能解析 node 的服务。"""
|
||||
root = project_root()
|
||||
services: list[dict] = []
|
||||
|
||||
for compose_path in sorted(root.glob("**/compose.yaml")):
|
||||
service_dir = compose_path.parent
|
||||
if not is_deployable_dir(service_dir, root):
|
||||
continue
|
||||
rel = str(service_dir.relative_to(root))
|
||||
info = service_info(rel, strict=False)
|
||||
if info is None:
|
||||
continue
|
||||
if node is None or info["node"] == node:
|
||||
services.append(info)
|
||||
|
||||
return services
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env python3
|
||||
"""列出仓库内所有可部署服务及其目标节点。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
|
||||
from lib import discover_services, project_root
|
||||
|
||||
|
||||
def main() -> int:
|
||||
root = project_root()
|
||||
os.chdir(root)
|
||||
services = discover_services()
|
||||
if not services:
|
||||
print("未找到任何可部署服务(需 compose.yaml 且能解析 node)")
|
||||
return 0
|
||||
|
||||
by_node: dict[str, list[str]] = defaultdict(list)
|
||||
for info in services:
|
||||
rel = os.path.relpath(info["service_dir"], root)
|
||||
by_node[info["node"]].append(rel)
|
||||
|
||||
print(f"共 {len(services)} 个服务,分布在 {len(by_node)} 个节点:\n")
|
||||
for node in sorted(by_node):
|
||||
print(f"[{node}]")
|
||||
for service in by_node[node]:
|
||||
print(f" - {service}")
|
||||
print()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+79
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
在远程节点上执行 docker compose 操作。
|
||||
|
||||
用法:
|
||||
python remote.py <服务目录> <命令>
|
||||
|
||||
命令:
|
||||
up 启动/更新容器(不拉镜像)
|
||||
recreate 强制重建容器(up -d --force-recreate,改配置后用)
|
||||
restart 重启 compose 内所有服务
|
||||
upgrade 拉取镜像并重建容器
|
||||
ps 查看容器状态
|
||||
logs 查看最近日志(非 follow)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from lib import project_root, service_info, ssh_cmd
|
||||
|
||||
REMOTE_COMMANDS = {
|
||||
"up": "docker compose up -d",
|
||||
"recreate": "docker compose up -d --force-recreate",
|
||||
"restart": "docker compose restart",
|
||||
"upgrade": "docker compose pull && docker compose up -d",
|
||||
"ps": "docker compose ps",
|
||||
"logs": "docker compose logs --tail=100",
|
||||
}
|
||||
|
||||
|
||||
def run_ssh(info: dict, compose_command: str) -> int:
|
||||
remote_dir = info["remote_dir"]
|
||||
remote_shell = f"cd {remote_dir} && {compose_command}"
|
||||
cmd = ssh_cmd(info, remote_shell)
|
||||
endpoint = info["node"]
|
||||
if info.get("port") is not None:
|
||||
endpoint = f"{endpoint}:{info['port']}"
|
||||
print(f"远程执行: {endpoint}:{remote_dir}")
|
||||
print(f"$ {compose_command}")
|
||||
print("-" * 60)
|
||||
try:
|
||||
return subprocess.run(cmd, check=False).returncode
|
||||
except FileNotFoundError:
|
||||
print("错误: ssh 命令未找到")
|
||||
return 1
|
||||
|
||||
|
||||
def run_for_service(service_dir: str, action: str) -> int:
|
||||
info = service_info(service_dir)
|
||||
compose_command = REMOTE_COMMANDS[action]
|
||||
endpoint = info["node"]
|
||||
if info.get("port") is not None:
|
||||
endpoint = f"{endpoint}:{info['port']}"
|
||||
print(f"服务: {service_dir} -> {endpoint}:{info['remote_dir']}")
|
||||
return run_ssh(info, compose_command)
|
||||
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="在远程节点执行 docker compose 操作")
|
||||
parser.add_argument("target", help="服务目录,例如 hosts/web1/myapp")
|
||||
parser.add_argument(
|
||||
"action",
|
||||
choices=sorted(REMOTE_COMMANDS),
|
||||
help="远程操作",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
os.chdir(project_root())
|
||||
return run_for_service(args.target, args.action)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+145
@@ -0,0 +1,145 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
目录同步脚本
|
||||
将指定目录同步到远程机器
|
||||
|
||||
用法:
|
||||
python sync.py <目录路径>
|
||||
例如: python sync.py hosts/web1/myapp
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import shlex
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from lib import DEFAULT_BASE_PATH, project_root, rsync_ssh_args, service_info, ssh_cmd
|
||||
|
||||
|
||||
def sync_directory(info: dict) -> int:
|
||||
if shutil.which("rsync"):
|
||||
return _sync_rsync(info)
|
||||
print("本机没有 rsync,改用 tar over SSH(不会删除远程多余文件)")
|
||||
return _sync_tar(info)
|
||||
|
||||
|
||||
def _sync_rsync(info: dict) -> int:
|
||||
source_dir = info["service_dir"]
|
||||
node = info["node"]
|
||||
remote_path = f"{node}:{info['remote_dir']}"
|
||||
|
||||
rsync_cmd = [
|
||||
"rsync",
|
||||
"-avz",
|
||||
"--delete",
|
||||
*rsync_ssh_args(info),
|
||||
]
|
||||
for item in info["sync_exclude"]:
|
||||
rsync_cmd.append(f"--exclude={item}/")
|
||||
|
||||
rsync_cmd.extend([f"{source_dir}/", remote_path])
|
||||
|
||||
print(f"正在同步 {source_dir} 到 {remote_path}")
|
||||
if info["sync_exclude"]:
|
||||
print(f"排除目录: {', '.join(info['sync_exclude'])}")
|
||||
print(f"执行命令: {' '.join(rsync_cmd)}")
|
||||
print("-" * 60)
|
||||
|
||||
try:
|
||||
subprocess.run(rsync_cmd, check=True)
|
||||
print("-" * 60)
|
||||
print("同步完成!")
|
||||
return 0
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"错误: rsync 执行失败,退出码: {e.returncode}")
|
||||
return 1
|
||||
except FileNotFoundError:
|
||||
print("错误: rsync 命令未找到,请确保已安装 rsync")
|
||||
return 1
|
||||
|
||||
|
||||
def _sync_tar(info: dict) -> int:
|
||||
source_dir = info["service_dir"]
|
||||
remote_dir = info["remote_dir"]
|
||||
tar_cmd = ["tar", "czf", "-", "-C", source_dir]
|
||||
for item in info["sync_exclude"]:
|
||||
tar_cmd.append(f"--exclude={item}")
|
||||
tar_cmd.append(".")
|
||||
|
||||
remote_shell = (
|
||||
f"mkdir -p {shlex.quote(remote_dir)} && "
|
||||
f"tar xzf - -C {shlex.quote(remote_dir)}"
|
||||
)
|
||||
ssh = ssh_cmd(info, remote_shell)
|
||||
print(f"正在同步 {source_dir} 到 {info['node']}:{remote_dir}")
|
||||
if info["sync_exclude"]:
|
||||
print(f"排除目录: {', '.join(info['sync_exclude'])}")
|
||||
print(f"执行命令: tar | {' '.join(ssh)}")
|
||||
print("-" * 60)
|
||||
|
||||
tar = subprocess.Popen(tar_cmd, stdout=subprocess.PIPE)
|
||||
try:
|
||||
completed = subprocess.run(ssh, stdin=tar.stdout, check=False)
|
||||
finally:
|
||||
if tar.stdout:
|
||||
tar.stdout.close()
|
||||
tar.wait()
|
||||
if tar.returncode:
|
||||
print(f"错误: tar 打包失败,退出码: {tar.returncode}")
|
||||
return tar.returncode
|
||||
if completed.returncode:
|
||||
print(f"错误: 远程 tar 解包失败,退出码: {completed.returncode}")
|
||||
return completed.returncode
|
||||
print("-" * 60)
|
||||
print("同步完成!")
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="将指定目录同步到远程机器",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
示例:
|
||||
python sync.py hosts/web1/myapp
|
||||
python sync.py infra/traefik --base-path /opt/app
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"directory",
|
||||
help="要同步的目录路径(相对项目根,例如: hosts/web1/myapp)",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--base-path",
|
||||
default=DEFAULT_BASE_PATH,
|
||||
help=f"远程基础路径(默认: {DEFAULT_BASE_PATH})",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
info = service_info(args.directory)
|
||||
if args.base_path != DEFAULT_BASE_PATH:
|
||||
name = info["name"]
|
||||
info["base_path"] = args.base_path
|
||||
info["remote_dir"] = f"{args.base_path}/{name}"
|
||||
|
||||
print(f"目标节点: {info['node']}")
|
||||
if info.get("port") is not None:
|
||||
print(f"SSH 端口: {info['port']}")
|
||||
if info.get("identity_file"):
|
||||
print(f"SSH 密钥: {info['identity_file']}")
|
||||
print(f"源目录: {args.directory}")
|
||||
print(f"远程路径: {info['remote_dir']}")
|
||||
print()
|
||||
|
||||
return sync_directory(info)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user