adrians
/
feca-mailshots-plugin
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
feca-mailshots-plugin/requirements/mailshot.md

408 lines
20 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# 1 Mailshot
Use and comply with `requirements/ui_design.md`.
## 1.0 Environment Baseline
* All writable mailshot tables and mailshot asset binaries are stored in the database named by `.env` variable `MAILSHOTS_REMOTE_MYSQL_DB`.
* This includes `mailshots`, `mailshot_queries`, `mailshot_attachments`, `mailshot_last_run`, and any mailshot asset tables created for PDF/image assets.
* In this document, references in the form `MAILSHOTS_REMOTE_MYSQL_DB.<table>` are normative.
* Recipient/source data reads used to resolve mailshot recipients and DSL source fields must use `.env` variable `MEMBERS_REMOTE_MYSQL_DB`.
* SMTP/IMAP credentials are sourced from WordPress-side per-user mailshot credential storage.
* If user credentials are missing, send/run actions must be blocked and the UI must direct users to the WordPress mailshot credentials page.
## 1.1 Purpose
Provide a Mailshot feature to:
* manage mailshot definitions
* preview recipient data and template rendering
* test mailshots safely
* execute full mailshot runs
## 1.2 Data Model and Dependencies
* Use `MAILSHOTS_REMOTE_MYSQL_DB.mailshots` as the source of mailshot definitions.
* Use `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries` as the source of recipient/data-source queries.
* Send mail using per-user SMTP/IMAP credentials from WordPress-side mailshot credential storage.
* If user credentials are missing, block send/run and direct user to the WordPress mailshot credentials page.
* Copy successfully sent messages to configured IMAP `Sent` folder.
### 1.2.1 Schema Reference: `MAILSHOTS_REMOTE_MYSQL_DB.mailshots`
| Column | Type | Null | Key | Notes |
| -------------------------- | ---------------- | ---- | --- | ----------------------------------------------------------------------------------- |
| `id` | `int(11)` | NO | PK | Auto-increment |
| `Purpose` | `varchar(100)` | YES | | Human-readable mailshot purpose |
| `DataSource` | `varchar(100)` | NO | | Data-source name matching `mailshot_queries.name` |
| `CC` | `varchar(100)` | YES | | Optional CC list |
| `BCC` | `varchar(100)` | YES | | Optional BCC list |
| `Subject` | `longtext` | NO | | Message subject template |
| `Message` | `longtext` | NO | | Message body template |
| `PDFAttachment` | `longtext` | YES | | Optional PDF attachment html template |
| `AttachmentNames` | `longtext` | YES | | JSON array of named entries from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_attachments` |
| `PDFFilenameDerivedFrom` | `varchar(128)` | YES | | Optional data-source field used as base filename for generated PDFs |
| `ReplyTo` | `varchar(100)` | YES | | Optional reply-to address |
### 1.2.2 Schema Reference: `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`
| Column | Type | Null | Key | Notes |
| ------------ | ---------------- | ---- | ------ | -------------------------------------------------------- |
| `ID` | `int(11)` | NO | PK | Auto-increment |
| `name` | `varchar(100)` | NO | UNIQUE | Data-source name used by `mailshots.DataSource` |
| `dsl_text` | `longtext` | YES | | DSL sentence (source of truth) |
| `sql` | `longtext` | NO | | Legacy column retained; compile target not user-authored |
## 1.3 Navigation and Pages
Under `Mailshot`, provide child pages:
* `Mailshots`
* `Assets`
* `Mailshot Recipients Preview`
* `Mailshot Test`
* `Run Mailshot`
* `Download PDF`
Under `Mailshot > Assets`, provide:
* `Mailshot Data Source > Data Sources`
* `PDF Assets`
* `Attachments`
## 1.3.1 Attachments Data Model
Use `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_attachments` as reusable uploaded attachments with:
* `id`
* `name` (unique)
* `file_name`
* `mime_type`
* `file_bytes`
* `created_at`
* `updated_at`
Persist selected named attachments on each mailshot in `MAILSHOTS_REMOTE_MYSQL_DB.mailshots.AttachmentNames` (JSON array of attachment names).
## 1.4 Mailshots Page (`MAILSHOTS_REMOTE_MYSQL_DB.mailshots` CRUD)
### 1.4.1 Data Pane
* Display rows from `MAILSHOTS_REMOTE_MYSQL_DB.mailshots`.
* In the list/table view, hide internal `id` and display user-facing columns (`Purpose`, `DataSource`, `Subject`).
* Sort the mailshot list by `Purpose` (ascending).
* Provide editing for all mutable fields.
* Highlight unsaved changed fields.
* Use a separate pane for editing the fields.  Note that the Message field is multiline html-formatted and should be edited using an html edit control.  The design of the page should maximise room for this field
* Provide a separate html edit control for `PDFAttachment` (optional template) using the same html editing features as `Message`.
* Message and PDF Attachment html editors should open in overlay panes to maximise editing room.
* Provide `PDF Filename Derived From` dropdown populated from available data-source fields for the selected `DataSource`.
* Persist `PDF Filename Derived From` as `MAILSHOTS_REMOTE_MYSQL_DB.mailshots.PDFFilenameDerivedFrom`.
* The Data Pane also include a table of mailshots,  width limited to the page width,  with truncation of fields.
* Keep editing responsive; do not trigger per-keystroke save/reload.
* Data-source pulldowns must be sorted by data-source `name` (ascending).
* Show available data-source fields in the editor and allow one-click token insertion into `Subject`, `Message`, and `PDFAttachment`.
* Token insertion must generate Twig tokens using `{{ ... }}` syntax (not `m_...` syntax).
* Field insert token format default is `{{ field_name }}` where `field_name` is a canonical safe field token derived from the selected source field.
* `Subject`, `Message`, and `PDFAttachment` are always rendered through Twig at render/send time.
* In editor pane, provide `Attachments` dropdown (initially blank) sourced from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_attachments.name`.
* Provide `Add selected attachment` button to append selected item into mailshot attachment list.
* Allow removing previously selected attachments from the mailshot attachment list.
### 1.4.2 Action Pane
* Provide `Save` button. Enable only when unsaved changes exist.
* Provide `Discard Changes` button. Enable only when unsaved changes exist.
* Provide `New` button to add a new `MAILSHOTS_REMOTE_MYSQL_DB.mailshots` row.
* Provide `Delete` button. Enable only when one row is selected.
* On `Delete`, require confirmation before delete is applied.
### 1.4.3 Validation
* Require `DataSource` to match an existing `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries.name`.
* Prevent save when `Subject` or `Message` are blank.
* Display validation errors in the information pane.
## 1.5 Template Engine (Twig)
* Use Twig as the template engine for mailshot rendering.
* All template tokens and expressions use Twig syntax.
* Template context is the selected recipient row (plus approved helper functions/filters).
* Default undefined-variable behavior is non-fatal and renders as empty string while recording a warning.
* Default escaping policy is:
* `Subject`: unescaped text output.
* `Message`: unescaped only when explicitly requested in template (`| raw`), otherwise escaped.
* `PDFAttachment`: unescaped only when explicitly requested in template (`| raw`), otherwise escaped.
* Control-flow tags (`{% if %}`, `{% for %}`, `{% set %}`) are allowed in `Message` and `PDFAttachment`, but disallowed in `Subject` by default.
## Mailshot Data Sources Page
This is defined in `requirements/mailshot_data_source.md`.
## 1.6 Mailshot Recipients Preview Page
### 1.6.1 Selection and Query Execution
* Provide control to select a data source from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`.
* On preview request, execute the selected data-source query and display resulting recipient rows.
### 1.6.2 Preview Table
* Display columns from a deterministic field contract derived from all DSL-cited sources, plus any additional source-native columns present in preview output.
* The deterministic field contract must include all allowed fields for each cited source (built-in or custom).
* Provide row count and sample-limit indicator in the information pane.
* Provide generalized recipient sorting controls (`Sort by` and sort direction) and apply them to the displayed preview rows.
* If query returns zero rows, display zero-result message (no error state).
* Provide `Download spreadsheet` action.
* On `Download spreadsheet`, generate `recipients.xlsx` from rows currently shown in the data pane and download it.
* In `recipients.xlsx`, format the header row in bold.
* In `recipients.xlsx`, auto-size each column width to fit header/data content.
### 1.6.3 Safety Rules
* Preview does not send email.
* Preview does not persist changes to recipient-source tables.
## 1.7 Mailshot Test Page
### 1.7.1 Test Setup
* Provide control to select a mailshot.
* Provide control to select one preview/test recipient row from the resolved query output.
* Provide `Test email address` input for explicit destination override, defaulting to configured `.env` value `MAILSHOT_TEST_TO_DEFAULT` when set.
* If `MAILSHOT_TEST_TO_DEFAULT` is unset/blank, initialize `Test email address` as blank.
* Rendered Message control must display the html message according to the html formatting.
* Provide a rendered `PDFAttachment` html preview pane below the rendered message preview pane.
* The rendered Message and rendered PDF Attachment preview panes should default to equal heights and support drag-resize via a divider.
### 1.7.2 Test Mode: Render Only (No Send)
* Provide action `Render Test (No Send)`.
* Perform full Twig template rendering for selected row.
* Display rendered `Subject`, `Message`, and `PDFAttachment`.
* Do not send SMTP mail.
* Do not write IMAP `Sent` copy.
### 1.7.3 Test Mode: Send Test Email
* Provide action `Send Test Email`.
* Require non-blank `Test email address`.
* When `Test email address` is blank, block send and show validation error in the information pane.
* Use selected test row only for template rendering context.
* Send exactly one email to `Test email address` (not to original recipient address).
* Copy sent test email to IMAP `Sent`.
* Display success/failure status and timestamp.
* Keep rendered `Subject`, `Message`, and `PDFAttachment` visible after send.
## 1.8 Run Mailshot Page
### 1.8.1 Pre-Run Validation
* Require selected mailshot.
* Require resolved recipient query.
* Block run when recipient query returns zero recipients.
* Display validation failures and do not start run when validation fails.
### 1.8.2 Execution Semantics
* On `Run Mailshot`, execute recipient query and iterate recipients.
* For each recipient row:
* perform Twig template rendering
* send one email to the recipient destination email
* copy successful sends to IMAP `Sent`
* Continue processing remaining recipients after per-recipient failures.
### 1.8.3 Run Results
* Display live or final counters:
* attempted
* sent
* failed
* skipped (if applicable)
* Provide downloadable or copyable run log in the UI.
* Include per-recipient failure reason where available.
### 1.8.4 Last Run Retry Table
* Persist the outcome of the most recent run only in `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run`.
* Clear `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run` when starting a new `Run Mailshot` execution.
* Each row in `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run` must store:
* `mailshot_id`
* `recipient_key` (stable non-email identifier)
* `recipient_key_field` (source field used to derive key)
* `recipient_email_last` (most recent resolved destination email)
* `status`
* `error_message` and/or `warning_message`
* `attempt_count`
* `last_attempt_at`
* Mailshots page must display the latest rows for the selected mailshot from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run`.
* Mailshots page must provide:
* `Retry Failed Sends` button to retry all failed rows from the selected mailshot.
* per-row `Retry` button to retry one failed row by recipient key.
* Retry must resolve recipient data from the current data source and current template values at retry time (not snapshot payload from original run).
* Email address must not be used as the retry key.
## 1.9 Download PDF Page
### 1.9.1 Setup and Validation
* Provide control to select a mailshot.
* Require selected mailshot.
* Require resolved recipient query.
* Block generation when recipient query returns zero recipients.
* Block generation when selected mailshot has blank `PDFAttachment` template.
### 1.9.2 Output Options
* Provide action to generate and download individual recipient PDFs as a single `.zip`.
* Provide action to generate and download a single merged PDF containing all recipient PDFs.
* Use html-to-pdf conversion of rendered `PDFAttachment` content per recipient row.
### 1.9.3 Results and Logging
* Display generation success/failure status in the UI.
* Generate downloadable files directly (no per-row log table required on this page).
## 1.10 Mailshot Queries Page
Provide a CRUD page for the `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries` table.
### 1.10.1 DSL Built-in Sources
Built-in DSL sources include:
* `contacts`
* `accounts`
* `renewals`
`renewals` is a first-class source and must be joinable to:
* `accounts` via `renewals.account_id = accounts.ID`
* `contacts` through accounts (`renewals.account_id = contacts.Accountid`)
### 1.10.2 DSL Renewal and Membership Filters
DSL filters include:
* `selected-renewal`
* `pending-renewal`
* `fen1-contact`
* `fen2-contact`
* `member-or-affiliate-or-parish-council`
`selected-renewal` uses saved field values (`renewals.selected = true`).
`pending-renewal` filters rows to `renewals.status = 'pending'`.
## 2.0 No-Effect and Error Handling
* If `Save` is clicked with no changes, no effect.
* If `Discard Changes` is clicked with no changes, no effect.
* If a selected mailshot references a missing `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries` row, block preview/test/run and display error.
* If SMTP send fails for one recipient, record error and continue with next recipient.
* If IMAP copy fails after successful SMTP send, mark warning for that recipient.
* For any DSL sentence, all fields from all cited sources must be available for token insertion and template rendering; missing values may render empty, but fields must not disappear from mailshot field availability.
* Twig template syntax errors must not crash the page or run; show clear error identifying mailshot and template field (`Subject`, `Message`, or `PDFAttachment`).
* Default behavior on template syntax error is fail-fast for that recipient (skip send for recipient, continue run for others) and record error.
* Retry actions must update `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run` rows in-place (incrementing `attempt_count` and updating status/messages/timestamp).
### 2.0.1 IMAP Sent-Copy Contract
* Use per-user configured IMAP sent-folder path from WordPress mailshot credentials.
* IMAP folder names must be encoded/decoded using IMAP UTF-7 rules.
* For each successful SMTP send, attempt a single IMAP append for sent-copy persistence.
* IMAP append failure must not change SMTP send outcome; classify as `warning`, not `failed`.
* IMAP append retries are not automatic in run loop; retry occurs only via explicit operator retry action.
* Sent-copy operations must be idempotent per attempt id; duplicate append for the same attempt id is not allowed.
## 2.1 PDF Assets
### 2.1.1 Purpose
Provide reusable image assets that can be inserted into `PDFAttachment` html templates using Twig syntax and resolved at PDF render time.
### 2.1.2 Navigation and Page
Under `Mailshot`, provide page `PDF Assets`.
### 2.1.3 Data Model
Provide CRUD storage for PDF assets with at least:
* `id` (primary key)
* `name` (unique asset identifier used in token marker)
* `file_name` (original upload filename)
* `mime_type`
* `file_bytes` (or storage reference)
* `width_mm`
* `height_mm`
* `justification` (`left`, `right`, `in-place`)
* `created_at`
* `updated_at`
### 2.1.4 PDF Assets Page Behavior
* List assets in a table.
* Support `Create`, `Read`, `Update`, `Delete`.
* Upload graphic file during create/update.
* Edit dimensions in millimetres.
* Edit justification (`left`, `right`, `in-place`).
* Validate file type and size constraints.
### 2.1.5 Mailshots PDF Editor Integration
* In the `PDF Attachment` editor overlay, add an insert-asset action with dropdown of available assets.
* On insert, place Twig token syntax at cursor position.
* Default insert format is `{{ pdf_asset('asset_name') }}`.
* Asset names used in inserted tokens must be normalized to safe token characters.
### 2.1.6 Rendering Behavior
* On rendering `PDFAttachment` html (Mailshot Test preview/send, Run Mailshot, Download PDF), resolve each inserted PDF asset Twig token to rendered image html for that asset.
* Render image using configured dimensions (`width_mm`, `height_mm`) and configured justification.
* If template references missing asset, do not fail entire run; record warning and render explicit missing-asset placeholder html.
* Missing asset placeholder default is `[missing pdf asset: <asset_name>]`.
### 2.1.7 PDF Filename Derivation
* For each rendered recipient pdf (download and email attachment), if `PDFFilenameDerivedFrom` is set, use row value from that field as base filename.
* Sanitize filename to safe filename characters.
* If `PDFFilenameDerivedFrom` is blank or field value resolves blank, use fallback naming strategy.
* Ensure filename uniqueness within a generated batch (append numeric suffix as needed).
## 2.2 PDF Assets Decisions
* Asset binaries are stored in DB table (`LONGBLOB`) for portability and backup simplicity.
* Allowed upload formats: `png`, `jpg/jpeg`, `svg`.
* Scope/ownership: global asset library, editor-capable users (WordPress `edit_pages`) may CRUD.
* Asset naming: unique case-insensitive `name`; renaming does not auto-migrate existing asset insertions in templates and should warn user.
* Marker syntax is removed. PDF asset insertion uses Twig syntax.
* Helper name is `pdf_asset(name)` as a global Twig helper, not a filter.
* Dimension/aspect behavior: fixed width+height in mm with `object-fit: contain` (no crop/stretch distortion).
* Justification semantics: `in-place` renders inline at marker location; `left` and `right` render block-aligned elements on their own line (no text-wrap behavior).
* Upload limits: max file size `2MB`, max image dimensions `4000x4000`.
* Missing asset token behavior: continue processing, emit warning, render explicit missing-asset placeholder.
* Delete behavior for in-use assets: block delete when asset helper usage is referenced by any `PDFAttachment` template and show referencing mailshot count.
## 2.3 Mailshot Attachments
### 2.3.1 Page and Layout
Under `Mailshot > Assets`, provide page `Attachments`.
* Data pane: table of attachments sorted by `name`.
* CRUD pane (right): edit `name` and uploaded file.
* Provide actions: `Save`, `New`, `Delete`.
### 2.3.2 CRUD and Validation
* `name` is required and unique.
* Create requires uploaded file bytes.
* Update allows metadata-only edits without replacing file.
* Delete must be blocked while referenced by any `MAILSHOTS_REMOTE_MYSQL_DB.mailshots.AttachmentNames`.
### 2.3.3 Send/Run Behavior
* On `Mailshot Test > Send Test Email`, include all selected named attachments.
* On `Run Mailshot`, include all selected named attachments for each recipient email.
* If a selected attachment name is missing, continue send/run and record warning.
Reference in New Issue View Git Blame Copy Permalink