jeka/kiauh
1
0
Fork 0
mirror of https://github.com/dw-0/kiauh.git synced 2026-08-03 12:57:53 +05:00
Code Issues Packages Projects Releases Wiki Activity

feat(tests): implement live VM acceptance test strategy (TODO-006 to TODO-010)

Browse Source
This commit is contained in:
dw-0
2026-07-04 13:38:52 +02:00
parent 7b5522ac94
commit 1d1bac1524
25 changed files with 926 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/live-testing.md
+75
View File
@@ -0,0 +1,75 @@
# Live System Testing
Live tests run KIAUH workflows against a real Debian 12 QEMU/KVM VM. They are
isolated from the local developer machine by design.
## Safety Rules
- Live tests NEVER run on the local machine.
- They require `KIAUH_LIVE_ALLOW=1`.
- The target host must be explicitly set via `KIAUH_LIVE_TARGET_HOST` and match
the VM in the inventory.
- Local hostnames, loopback addresses, and the current hostname are blocked.
## Prepare a VM
1. Create a Debian 12 QEMU/KVM VM.
2. Create a user with passwordless sudo.
3. Install an SSH key for that user.
4. Install KIAUH on the VM (e.g. clone this repository).
5. Create a clean snapshot named `clean`:
```bash
virsh snapshot-create-as debian12-kiauh clean
```
## Inventory
Edit `kiauh/live/inventory.yaml` or point to a custom file:
```yaml
vms:
- name: debian12-kiauh
host: 192.168.122.10
user: kiauh
key_file: ~/.ssh/kiauh_vm
os: debian-12
domain: debian12-kiauh
snapshot: clean
```
## Run Live Tests
```bash
export KIAUH_LIVE_ALLOW=1
export KIAUH_LIVE_TARGET_HOST=192.168.122.10
pytest -m live
```
Each scenario reverts the VM to the clean snapshot first, so scenarios are
independent.
## Add a Scenario
Create a YAML file in `kiauh/live/scenarios/`:
```yaml
name: Install Klipper on Debian 12
vm: debian12-kiauh
os: debian-12
steps:
- command: ["kiauh", "install", "klipper", "--count", "1"]
timeout: 600
expected:
- type: service
name: klipper.service
state: running
```
Supported assertion types: `service`, `file`, `package`, `port`, `command`.
## Troubleshooting
- `UnsafeTargetError`: check `KIAUH_LIVE_ALLOW` and `KIAUH_LIVE_TARGET_HOST`.
- `InventoryError`: check the inventory YAML path and format.
- `LiveRunnerError` during snapshot revert: ensure `virsh` works and the domain
and snapshot names match the inventory.
docs/prd/PRD-002-live-vm-test-strategy.md
+74
View File
@@ -0,0 +1,74 @@
## ⚠️ Working on this PRD
Do NOT implement this PRD directly. It has been broken into sequential tasks.
Work through the tasks below in order.
## Task Index
| # | Task | Todo | Blocked by | Status |
|---|------|------|------------|--------|
| 1/5 | VM inventory, SSH fixture, and safety guards | TODO-006 | — | 🔄 open |
| 2/5 | Scenario loader and Klipper install scenario | TODO-007 | TODO-006 | ⏳ blocked |
| 3/5 | Remove Klipper and Moonraker scenarios | TODO-008 | TODO-007 | ⏳ blocked |
| 4/5 | Mainsail and Fluidd install scenarios | TODO-009 | TODO-008 | ⏳ blocked |
| 5/5 | Backup, restore, and update scenarios | TODO-010 | TODO-009 | ⏳ blocked |
Start with: **TODO-006** (PRD #2 - Task 1/5: VM inventory, SSH fixture, and safety guards)
---
# PRD #2: Isolated live-system acceptance test strategy on Debian 12 VM
**Tags:** `prd`, `prd-2`
## Problem Statement
- Pytest unit tests cannot validate real package installs, systemd services, git clones, and OS-specific behavior.
- Running workflow tests on a local developer machine risks destroying the environment.
- Need reproducible, isolated acceptance tests with explicit expected outcomes.
## Solution
- Acceptance tests run only on a pre-built Debian 12 QEMU/KVM VM.
- Test harness connects via SSH; never executes on the local host.
- YAML scenarios define workflows and expected outcomes.
- VM snapshot reverted before every scenario.
- Multi-layer safety prevents local execution.
## User Stories
1. As a maintainer, I want install/remove Klipper workflow tested on a real VM, so I know the installer still works.
2. As a maintainer, I want install/remove Moonraker workflow tested, so API stack compatibility is verified.
3. As a maintainer, I want Mainsail/Fluidd install workflow tested, so web client setup works end-to-end.
4. As a maintainer, I want backup/restore workflow tested, so user data survives the cycle.
5. As a maintainer, I want tests parameterized by VM inventory, so future Ubuntu/Debian versions can be added without code changes.
6. As a maintainer, I want local execution blocked by multiple guards, so the developer machine is never modified.
7. As a CI operator, I want scenario results to show expected vs actual outcome, so failures are actionable.
## Implementation Decisions
- VM inventory config supplies host/IP, SSH user/key, OS family. No auto-provisioning; base images are prepared in advance.
- Test harness: pytest + SSH fixture + Testinfra assertions. Commands are routed through SSH; assertions use Testinfra modules for service/file/package/port state.
- Scenario schema YAML: `name`, `os`, `steps` (commands/options), `expected` (assertions for service running, file exists, package installed, port reachable, process present).
- Snapshot reset: revert VM overlay before every scenario. Scenarios must be independent.
- Safety guards: require `KIAUH_LIVE_TARGET_HOST`; abort if value is `localhost`, `127.*`, or matches current hostname; abort if target resolves to a local interface; verify SSH host key differs from local; optional explicit confirmation prompt.
- Workflow priority: (1) install/remove Klipper; (2) install/remove Moonraker; (3) install Mainsail/Fluidd; (4) backup/restore; (5) update flows.
- Test user on VM has passwordless sudo; VM has internet access; long installs use timeouts.
- Scenario runner exposes expected outcome per step; failure shows command, expected assertion, and actual result.
## Testing Decisions
- Acceptance tests verify observable system state, not internal functions.
- Each scenario defines exact pre-state (clean snapshot) and post-state assertions.
- Flaky network commands are retried with timeout; failures attach relevant VM logs.
- Live suite is marked with a `live` pytest marker and excluded from the default `pytest` run.
## Out of Scope
- Running live tests on the local machine or bare metal.
- Auto-provisioning or building VM images.
- Testing every extension in the first iteration; only core workflows.
## Further Notes
- Future OS matrix (Ubuntu 22.04/24.04) is enabled by adding inventory entries and matching base images; no harness changes needed.
docs/todo/TODO-006.md
+31
View File
@@ -0,0 +1,31 @@
# PRD #2 - Task 1/5: VM inventory, SSH fixture, and safety guards
**Tags:** `task`, `prd-2`
## Parent PRD
PRD #2: Isolated live-system acceptance test strategy on Debian 12 VM (`docs/prd/PRD-002-live-vm-test-strategy.md`)
## What to build
Create the VM inventory config schema (host/IP, SSH user/key, OS family). Implement the SSH connection fixture and the multi-layer safety guards that abort if the target could be the local machine. Write tests for the guards without executing any workflow.
## Acceptance criteria
- [ ] Inventory config schema documented and validated.
- [ ] SSH fixture connects only when target is explicitly allowed.
- [ ] Guards block `localhost`, `127.*`, current hostname, local interfaces, and unknown SSH host keys.
- [ ] Guard tests run on the local machine and prove the blocks work.
## Blocked by
None — can start immediately.
## Next task
- TODO-007 (PRD #2 - Task 2/5: Scenario loader and Klipper install scenario)
## User stories addressed
- User story 5
- User story 6
docs/todo/TODO-007.md
+32
View File
@@ -0,0 +1,32 @@
# PRD #2 - Task 2/5: Scenario loader and Klipper install scenario
**Tags:** `task`, `prd-2`
## Parent PRD
PRD #2: Isolated live-system acceptance test strategy on Debian 12 VM (`docs/prd/PRD-002-live-vm-test-strategy.md`)
## What to build
Implement the YAML scenario loader and the `live` pytest marker. Write the first end-to-end scenario: install Klipper on the Debian 12 VM, define expected outcomes (service file, env file, folders), and run it with snapshot revert before the scenario.
## Acceptance criteria
- [ ] YAML scenario loader parses `name`, `os`, `steps`, and `expected` assertions.
- [ ] `pytest -m live` runs only live scenarios; default run skips them.
- [ ] Klipper install scenario runs on the VM and passes.
- [ ] Snapshot revert happens before the scenario.
- [ ] Expected outcomes include file, service, and folder assertions.
## Blocked by
- TODO-006 (PRD #2 - Task 1/5: VM inventory, SSH fixture, and safety guards)
## Next task
- TODO-008 (PRD #2 - Task 3/5: Remove Klipper and Moonraker scenarios)
## User stories addressed
- User story 1
- User story 7
docs/todo/TODO-008.md
+30
View File
@@ -0,0 +1,30 @@
# PRD #2 - Task 3/5: Remove Klipper and Moonraker scenarios
**Tags:** `task`, `prd-2`
## Parent PRD
PRD #2: Isolated live-system acceptance test strategy on Debian 12 VM (`docs/prd/PRD-002-live-vm-test-strategy.md`)
## What to build
Add remove-Klipper and install/remove-Moonraker scenarios. Each scenario starts from a clean snapshot. Capture relevant VM logs when a scenario fails to make debugging actionable.
## Acceptance criteria
- [ ] Remove Klipper scenario runs and verifies service/files are gone.
- [ ] Install Moonraker scenario runs and verifies service/config/log files.
- [ ] Remove Moonraker scenario runs and verifies cleanup.
- [ ] Failure output includes tail of installer/service logs.
## Blocked by
- TODO-007 (PRD #2 - Task 2/5: Scenario loader and Klipper install scenario)
## Next task
- TODO-009 (PRD #2 - Task 4/5: Mainsail and Fluidd install scenarios)
## User stories addressed
- User story 2
docs/todo/TODO-009.md
+30
View File
@@ -0,0 +1,30 @@
# PRD #2 - Task 4/5: Mainsail and Fluidd install scenarios
**Tags:** `task`, `prd-2`
## Parent PRD
PRD #2: Isolated live-system acceptance test strategy on Debian 12 VM (`docs/prd/PRD-002-live-vm-test-strategy.md`)
## What to build
Add install scenarios for Mainsail and Fluidd web clients. Assert that the static files are deployed and the reverse-proxy/service config is in place. Harden the snapshot-revert fixture so it runs reliably before every scenario.
## Acceptance criteria
- [ ] Mainsail install scenario passes on the VM.
- [ ] Fluidd install scenario passes on the VM.
- [ ] Expected outcomes check webroot directory and reverse-proxy config.
- [ ] Snapshot revert fixture is robust (wait for SSH, error on revert failure).
## Blocked by
- TODO-008 (PRD #2 - Task 3/5: Remove Klipper and Moonraker scenarios)
## Next task
- TODO-010 (PRD #2 - Task 5/5: Backup, restore, and update scenarios)
## User stories addressed
- User story 3
docs/todo/TODO-010.md
+32
View File
@@ -0,0 +1,32 @@
# PRD #2 - Task 5/5: Backup, restore, and update scenarios
**Tags:** `task`, `prd-2`
## Parent PRD
PRD #2: Isolated live-system acceptance test strategy on Debian 12 VM (`docs/prd/PRD-002-live-vm-test-strategy.md`)
## What to build
Add backup/restore and update scenarios. Document the live test runbook: how to prepare the VM, set inventory, run scenarios, and read results. Ensure the whole live suite can be executed in one command.
## Acceptance criteria
- [ ] Backup scenario creates an archive with expected content.
- [ ] Restore scenario returns config files to expected state.
- [ ] Update scenario changes a component version/config observable on the VM.
- [ ] Runbook `docs/live-testing.md` covers VM setup, inventory, execution, and troubleshooting.
- [ ] Full `pytest -m live` run completes end-to-end.
## Blocked by
- TODO-009 (PRD #2 - Task 4/5: Mainsail and Fluidd install scenarios)
## Next task
None — this is the last task.
## User stories addressed
- User story 4
- User story 7