squeezelite-esp32/documentation/HARDWARE_TEST_MATRIX.md

Hardware Test Matrix

Date: 2026-02-12
Scope: Platform-specific stability and release validation for squeezelite-esp32.

Purpose

Define a practical hardware-focused test strategy that complements host/unit tests and acknowledges embedded uncertainty under real-world load.

This matrix is designed to:

• maximize platform stability
• prioritize high-impact failures first
• provide clear release gates per hardware profile

Use together with:

• documentation/TESTING_CHARTER.md
• documentation/CONTRACT_TEST_TEMPLATE.md

Platforms

Mapped to build profiles used in this repo:

• i2s -> build-scripts/I2S-4MFlash-sdkconfig.defaults
• muse -> build-scripts/Muse-sdkconfig.defaults
• squeezeamp -> build-scripts/SqueezeAmp-sdkconfig.defaults

Priority Levels

• P0: release blocking, must pass
• P1: strong confidence tests, expected to pass before release
• P2: extended coverage/soak, required for nightly and milestone validation

Execution Tiers

• PR smoke: fast subset (P0 only, selected P1)
• Nightly: full P0 + P1 and selected P2
• Pre-release: full matrix, all P0/P1, mandatory P2 soak and fault-injection

Core Hardware Matrix

ID	Area	Test Path	Priority	Platforms	Pass Criteria
HW-BOOT-001	Boot	Cold boot to operational state	P0	all	boots successfully within target time budget; no panic/reset loop
HW-BOOT-002	Boot	Warm reboot (esp_restart) loop x50	P0	all	no stuck boot, no boot partition confusion, stable counters
HW-BOOT-003	Boot identity	Platform profile and key GPIO map sanity	P0	all	expected FW_PLATFORM_NAME, expected peripheral init paths
HW-STOR-001	Storage/NVS	NVS read/write/reset cycle	P0	all	persisted values survive reboot, reset restores defaults
HW-STOR-002	Storage/NVS	Corrupt/partial NVS handling	P1	all	graceful recovery path, no crash, defaults restored or fallback used
HW-STOR-003	SPIFFS	SPIFFS mount + defaults file read	P0	all	mount succeeds, required files loaded, clear error if missing
HW-NET-001	Wi-Fi	AP connect + DHCP + DNS	P0	all	connected state reached, IP assigned, DNS query success
HW-NET-002	Wi-Fi resilience	AP loss/recovery reconnect	P0	all	reconnect within SLA, no task deadlock or watchdog
HW-NET-003	Service discovery	mDNS announce/discover	P1	all	service visible on LAN; no crash if mDNS unavailable
HW-NET-004	Ethernet	Link up/down + DHCP + traffic	P1	ethernet-capable	state transitions handled cleanly; no memory growth trend
HW-AUD-001	Audio output	Start/stop playback, no signal path errors	P0	all	successful init and playback lifecycle, no panic
HW-AUD-002	Audio behavior	Format/rate transitions during playback	P1	all	transitions without hard fault; bounded glitch behavior
HW-AUD-003	Audio resilience	Underrun/rebuffer recovery	P1	all	playback recovers, no runaway CPU or deadlock
HW-AUD-004	Controls	Volume/mute/jack/speaker path checks	P1	platform-specific	control actions reflected correctly in output state
HW-UI-001	Inputs	Button/rotary/IR event mapping	P1	platform-specific	expected commands dispatched; no ghost/repeat storm
HW-UI-002	Display	Display init + update loop	P1	display-capable	stable render/update path, no crash under repeated updates
HW-PWR-001	Battery/charger	Battery telemetry and status logic	P1	battery-capable	values in expected range; no invalid-state loop
HW-BT-001	Bluetooth	Pair/connect/disconnect cycles	P1	bt-enabled	stable lifecycle across repeated cycles
HW-BT-002	Bluetooth resilience	BT stack restart/recovery	P2	bt-enabled	stack recovers without reboot or hard fault
HW-OTA-001	OTA	Happy-path OTA update	P0	all	new image boots and reports expected version
HW-OTA-002	OTA failure	Interrupted OTA (network/power cut)	P0	all	safe fallback/rollback path works; device recoverable
HW-OTA-003	Recovery partition	Recovery boot and exit path	P0	all	recovery mode entry/exit consistent and deterministic
HW-PWRF-001	Power fault	Brownout/power-cut during write/OTA	P0	all	no bricking; deterministic recovery behavior
HW-SOAK-001	Soak	12h playback + periodic reconnect	P2	all	no crash/reset; memory trend within threshold
HW-SOAK-002	Soak	24h mixed load (stream/control/network churn)	P2	all	no regressions in responsiveness and stability

Platform-Specific Focus

i2s (generic baseline)

• Emphasize:
  • audio path stability under varied sample rates
  • generic GPIO/peripheral defaults
  • OTA + recovery baseline behavior
• Required:
  • all P0
  • HW-AUD-002, HW-AUD-003, HW-SOAK-001

muse (portable/battery-centric)

• Emphasize:
  • battery telemetry, charger events, low-power edge conditions
  • UI/display responsiveness under battery/network churn
  • Wi-Fi reconnect after sleep-like or low-power transitions
• Required:
  • all P0
  • HW-PWR-001, HW-UI-001, HW-UI-002, HW-SOAK-001

squeezeamp (amp/control-centric)

• Emphasize:
  • amplifier-related controls, output routing, jack/speaker transitions
  • thermal/long-play stability proxies (extended playback)
  • network resilience during active playback
• Required:
  • all P0
  • HW-AUD-004, HW-NET-002, HW-SOAK-001, HW-SOAK-002

Release Gates

A release is eligible only if:

• every target platform passes all P0 tests
• no unresolved regression in previously passing P1 tests
• at least one P2 soak run per platform completed within release window
• OTA/recovery tests (HW-OTA-001/002/003) pass on all platforms

Observability Requirements During Tests

Collect at minimum:

• reset reason, panic reason, boot count, recovery boot count
• heap free/minimum and largest block trend
• task stack high-water marks for critical tasks
• network reconnect counters and durations
• OTA stage/result events

A test can be marked "pass with warning" only for non-P0 tests and only with a filed follow-up issue and owner.

Failure Triage Rules

• P0 fail: immediate release block
• repeated P1 fail on same platform: treat as release risk, escalate to block unless waived
• any soak fail with crash/reset: open defect with logs and reproduction steps before next release candidate

Suggested First Implementation Wave

1. Automate all P0 checks for each platform profile.
2. Add nightly P1 network/audio/UI subset.
3. Add 12h soak (HW-SOAK-001) with telemetry capture and trend thresholds.
4. Expand to full pre-release matrix once stable signal quality is established.