calendar-plugin/requirements/test_strategy.md

Test Strategy Requirements

Purpose

Define mandatory test execution policy, suite composition, and quality gates for preventing regressions.

Scope

This document covers:

• Test levels and ownership
• Required suites by pipeline stage
• Compatibility-harness isolation/reset strategy
• Pass/fail and release gates

Test Levels

• Unit tests: pure logic and transformation behavior
• Integration tests: plugin + compatibility-harness DB/APIs
• End-to-end tests: UI/API/CalDAV/ICS end-to-end flows
• Smoke tests: fast high-signal regression checks

Reference documents:

• tests/calendar_entries.md
• tests/e2e_test_cases.md
• tests/api_test_cases.md
• tests/smoke_tests.md
• tests/lifecycle_test_cases.md
• tests/user_ui_caldav_1hr_spec.md
• requirements/architecture.md

Execution Policy

Per Commit / PR

Must run:

• lint/static checks (when available)
• smoke suite
• changed-area unit/integration tests

Nightly

Must run:

• full unit + integration suite
• full E2E suite
• full API suite (if API enabled)
• CalDAV and ICS interoperability checks

Pre-Release

Must run:

• full nightly suite
• upgrade/migration tests
• rollback sanity checks
• packaging/install/uninstall validation
• 60-minute user functional sweep (UI + CalDAV) using tests/user_ui_caldav_1hr_spec.md

Harness Determinism

• Tests must run against isolated compatibility-harness-managed state.
• Reset to known baseline before suite groups.
• No test may depend on leftover state from prior runs.
• Default CI runs must not depend on external systems.

Flake Management

• Flaky tests must be tagged and tracked with issue references.
• Repeatedly flaky tests cannot remain in required gates without mitigation.

Coverage Expectations

• CRUD flows for single and recurring events are mandatory coverage.
• Recurrence exception behavior (single occurrence delete without split) is mandatory coverage.
• Authn/Authz paths for admin, API, and CalDAV roles are mandatory coverage.
• Error-model contract coverage is mandatory for API endpoints.
• CalDAV client-compatibility regressions are mandatory coverage, including:
  • collection GET availability probe compatibility
  • collection REPORT handling with and without trailing slash
  • sync-collection no-change stability (current token -> zero change entries)

Pass/Fail Gates

• Any required suite failure blocks merge/release as applicable.
• Failures must capture logs/artifacts sufficient for debugging.

Reporting

• CI should publish suite summary with:
  • passed/failed/skipped counts
  • runtime
  • links to failure artifacts

Change Management

• New features must include updated tests and requirement-traceable cases.
• Requirement updates that change behavior must update corresponding test docs and automation.