85 lines
2.5 KiB
Markdown
85 lines
2.5 KiB
Markdown
# Test Strategy Requirements
|
|
|
|
## Purpose
|
|
Define mandatory test execution policy, suite composition, and quality gates for preventing regressions.
|
|
|
|
## Scope
|
|
This document covers:
|
|
|
|
- Test levels and ownership
|
|
- Required suites by pipeline stage
|
|
- Compatibility-harness isolation/reset strategy
|
|
- Pass/fail and release gates
|
|
|
|
## Test Levels
|
|
- Unit tests: pure logic and transformation behavior
|
|
- Integration tests: plugin + compatibility-harness DB/APIs
|
|
- End-to-end tests: UI/API/CalDAV/ICS end-to-end flows
|
|
- Smoke tests: fast high-signal regression checks
|
|
|
|
Reference documents:
|
|
|
|
- `tests/calendar_entries.md`
|
|
- `tests/e2e_test_cases.md`
|
|
- `tests/api_test_cases.md`
|
|
- `tests/smoke_tests.md`
|
|
- `tests/lifecycle_test_cases.md`
|
|
- `tests/user_ui_caldav_1hr_spec.md`
|
|
- `requirements/architecture.md`
|
|
|
|
## Execution Policy
|
|
|
|
### Per Commit / PR
|
|
Must run:
|
|
|
|
- lint/static checks (when available)
|
|
- smoke suite
|
|
- changed-area unit/integration tests
|
|
|
|
### Nightly
|
|
Must run:
|
|
|
|
- full unit + integration suite
|
|
- full E2E suite
|
|
- full API suite (if API enabled)
|
|
- CalDAV and ICS interoperability checks
|
|
|
|
### Pre-Release
|
|
Must run:
|
|
|
|
- full nightly suite
|
|
- upgrade/migration tests
|
|
- rollback sanity checks
|
|
- packaging/install/uninstall validation
|
|
- 60-minute user functional sweep (UI + CalDAV) using `tests/user_ui_caldav_1hr_spec.md`
|
|
|
|
## Harness Determinism
|
|
- Tests must run against isolated compatibility-harness-managed state.
|
|
- Reset to known baseline before suite groups.
|
|
- No test may depend on leftover state from prior runs.
|
|
- Default CI runs must not depend on external systems.
|
|
|
|
## Flake Management
|
|
- Flaky tests must be tagged and tracked with issue references.
|
|
- Repeatedly flaky tests cannot remain in required gates without mitigation.
|
|
|
|
## Coverage Expectations
|
|
- CRUD flows for single and recurring events are mandatory coverage.
|
|
- Recurrence exception behavior (single occurrence delete without split) is mandatory coverage.
|
|
- Authn/Authz paths for admin, API, and CalDAV roles are mandatory coverage.
|
|
- Error-model contract coverage is mandatory for API endpoints.
|
|
|
|
## Pass/Fail Gates
|
|
- Any required suite failure blocks merge/release as applicable.
|
|
- Failures must capture logs/artifacts sufficient for debugging.
|
|
|
|
## Reporting
|
|
- CI should publish suite summary with:
|
|
- passed/failed/skipped counts
|
|
- runtime
|
|
- links to failure artifacts
|
|
|
|
## Change Management
|
|
- New features must include updated tests and requirement-traceable cases.
|
|
- Requirement updates that change behavior must update corresponding test docs and automation.
|