# 1 Mailshot

Use and comply with `requirements/ui_design.md`.

## 1.0 Environment Baseline

* All writable mailshot tables and mailshot asset binaries are stored in the database named by `.env` variable `MAILSHOTS_REMOTE_MYSQL_DB`.
* This includes `mailshots`, `mailshot_queries`, `mailshot_attachments`, `mailshot_last_run`, and any mailshot asset tables created for PDF/image assets.
* In this document, references in the form `MAILSHOTS_REMOTE_MYSQL_DB.<table>` are normative.
* Recipient/source data reads used to resolve mailshot recipients and DSL source fields must use `.env` variable `MEMBERS_REMOTE_MYSQL_DB`.
* SMTP/IMAP credentials are sourced from WordPress-side per-user mailshot credential storage.
* If user credentials are missing, send/run actions must be blocked and the UI must direct users to the WordPress mailshot credentials page.

## 1.1 Purpose

Provide a Mailshot feature to:

* manage mailshot definitions
* preview recipient data and template rendering
* test mailshots safely
* execute full mailshot runs

## 1.2 Data Model and Dependencies

* Use `MAILSHOTS_REMOTE_MYSQL_DB.mailshots` as the source of mailshot definitions.
* Use `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries` as the source of recipient/data-source queries.
* Send mail using per-user SMTP/IMAP credentials from WordPress-side mailshot credential storage.
* If user credentials are missing, block send/run and direct user to the WordPress mailshot credentials page.
* Copy successfully sent messages to configured IMAP `Sent` folder.

### 1.2.1 Schema Reference: `MAILSHOTS_REMOTE_MYSQL_DB.mailshots`

| Column                     | Type             | Null | Key | Notes                                                                               |
| -------------------------- | ---------------- | ---- | --- | ----------------------------------------------------------------------------------- |
| `id`                     | `int(11)`      | NO   | PK  | Auto-increment                                                                      |
| `Purpose`                | `varchar(100)` | YES  |     | Human-readable mailshot purpose                                                     |
| `DataSource`             | `varchar(100)` | NO   |     | Data-source name matching `mailshot_queries.name`                                 |
| `CC`                     | `varchar(100)` | YES  |     | Optional CC list                                                                    |
| `BCC`                    | `varchar(100)` | YES  |     | Optional BCC list                                                                   |
| `Subject`                | `longtext`     | NO   |     | Message subject template                                                            |
| `Message`                | `longtext`     | NO   |     | Message body template                                                               |
| `PDFAttachment`          | `longtext`     | YES  |     | Optional PDF attachment html template                                               |
| `AttachmentNames`        | `longtext`     | YES  |     | JSON array of named entries from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_attachments` |
| `PDFFilenameDerivedFrom` | `varchar(128)` | YES  |     | Optional data-source field used as base filename for generated PDFs                 |
| `ReplyTo`                | `varchar(100)` | YES  |     | Optional reply-to address                                                           |
| `RecipientEmailField`    | `varchar(255)` | YES  |     | Optional selected data-source field used as recipient email destination             |

### 1.2.2 Schema Reference: `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`

| Column        | Type           | Null | Key    | Notes                                                    |
| ------------- | -------------- | ---- | ------ | -------------------------------------------------------- |
| `ID`          | `int(11)`      | NO   | PK     | Auto-increment                                           |
| `name`        | `varchar(100)` | NO   | UNIQUE | Data-source name used by `mailshots.DataSource`          |
| `dsl_text`    | `longtext`     | YES  |        | DSL sentence (source of truth)                           |
| `dsl_version` | `varchar(16)`  | NO   |        | DSL version marker; defaults to `v1`                     |
| `sql`         | `longtext`     | NO   |        | Legacy column retained; compile target not user-authored |

## 1.3 Navigation and Pages

Under `Mailshot`, provide child pages:

* `Mailshots`
* `Assets`
* `Mailshot Recipients Preview`
* `Mailshot Test`
* `Run Mailshot`
* `Download PDF`

Under `Mailshot > Assets`, provide:

* `Mailshot Data Source > Data Sources`
* `PDF Assets`
* `Attachments`

## 1.3.1 Attachments Data Model

Use `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_attachments` as reusable uploaded attachments with:

* `id`
* `name` (unique)
* `file_name`
* `mime_type`
* `file_bytes`
* `created_at`
* `updated_at`

Persist selected named attachments on each mailshot in `MAILSHOTS_REMOTE_MYSQL_DB.mailshots.AttachmentNames` (JSON array of attachment names).

## 1.4 Mailshots Page (`MAILSHOTS_REMOTE_MYSQL_DB.mailshots` CRUD)

### 1.4.1 Data Pane

* Display rows from `MAILSHOTS_REMOTE_MYSQL_DB.mailshots`.
* In the list/table view, hide internal `id` and display user-facing columns (`Purpose`, `DataSource`, `Subject`).
* Provide `Sort by` and direction controls for the list/table view.
* Sortable list columns are `Purpose`, `DataSource`, and `Subject`; the `Actions` column is not sortable.
* Default list sort is `Purpose` ascending.
* Preserve selected sort when editing, saving, duplicating, or deleting from the mailshot list.
* Provide editing for all mutable fields.
* Track whether the edit form has unsaved changes and warn before closing/quitting an edit with unsaved changes.
* Use a separate pane for editing the fields.  Note that the Message field is multiline html-formatted and should be edited using an html edit control.  The design of the page should maximise room for this field
* Provide a separate html edit control for `PDFAttachment` (optional template) using the same html editing features as `Message`.
* Message and PDF Attachment html editors should open in overlay panes to maximise editing room.
* Provide `PDF Filename Derived From` dropdown populated from available data-source fields for the selected `DataSource`.
* Persist `PDF Filename Derived From` as `MAILSHOTS_REMOTE_MYSQL_DB.mailshots.PDFFilenameDerivedFrom`.
* Provide `Recipient Email Field` dropdown populated from available data-source fields for the selected `DataSource`.
* Persist `Recipient Email Field` as `MAILSHOTS_REMOTE_MYSQL_DB.mailshots.RecipientEmailField`.
* If `RecipientEmailField` is blank, show an inline warning that the mailshot cannot be used to send a mailshot; PDF-only workflows may leave it blank.
* The Data Pane also include a table of mailshots,  width limited to the page width,  with truncation of fields.
* Keep editing responsive; do not trigger per-keystroke save/reload.
* Data-source pulldowns must be sorted by data-source `name` (ascending).
* Show available data-source fields in the editor and allow one-click token insertion into `Subject`, `Message`, and `PDFAttachment`.
* Token insertion must generate Twig tokens using `{{ ... }}` syntax (not `m_...` syntax).
* Field insert token format default is `{{ field_name }}` where `field_name` is a canonical safe field token derived from the selected source field.
* `Subject`, `Message`, and `PDFAttachment` are always rendered through Twig at render/send time.
* In editor pane, provide `Attachments` dropdown (initially blank) sourced from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_attachments.name`.
* Provide `Add selected attachment` button to append selected item into mailshot attachment list.
* Allow removing previously selected attachments from the mailshot attachment list.

### 1.4.2 Action Pane

* Provide `Save` button.
* Provide `Quit`/close action for leaving the edit pane; if unsaved changes exist, require confirmation before closing without saving.
* Provide `New` button to add a new `MAILSHOTS_REMOTE_MYSQL_DB.mailshots` row.
* Provide per-row `Delete` button.
* On per-row `Delete`, require confirmation before delete is applied.
* Provide per-row `Duplicate` button in the list/table.
* `Duplicate` must copy all mailshot definition fields into a new row and assign a unique purpose:
  * first choice: `Copy of <original purpose>`
  * subsequent collisions: `Copy of <original purpose> (2)`, `Copy of <original purpose> (3)`, and so on
  * blank original purpose fallback: `Copy of Mailshot #<source id>`
  * resulting purpose must fit the `Purpose varchar(100)` column.
* After successful duplicate, redirect to the edit modal for the newly-created copy.

### 1.4.3 Validation

* Require `DataSource` to match an existing `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries.name`.
* Prevent save when `Subject` is blank.
* Allow `Message` to be blank for PDF-only workflows.
* If `RecipientEmailField` is non-blank, it must be one of the available fields for the selected `DataSource`.
* Display validation errors in the information pane.

## 1.5 Template Engine (Twig)

* Use Twig as the template engine for mailshot rendering.
* All template tokens and expressions use Twig syntax.
* Template context is the selected recipient row (plus approved helper functions/filters).
* Twig uses strict variables. Undefined variables or invalid expressions must fail rendering with a clear error.
* Default escaping policy is:
  * Twig autoescape is `html` for `Subject`, `Message`, and `PDFAttachment`.
  * Rendered `Subject` is HTML-entity-decoded before use as an email subject.
  * Template authors may use Twig filters such as `raw` where unescaped HTML output is required.
* Control-flow tags (`{% if %}`, `{% for %}`, `{% set %}`) are supported by Twig in all template fields.
* Template context must include both original query field keys (for example `pages.Page`) and canonical safe token keys (for example `pages_page`).
* Boolean-like fields whose field name starts with `is`, `has`, or `can`, or whose field name is `selected`, `active`, `enabled`, `disabled`, or `lapsed`, should render as `Yes`/`No` when their values are recognizably boolean-ish.

## Mailshot Data Sources Page

This is defined in `requirements/mailshot_data_source.md`.

## 1.6 Mailshot Recipients Preview Page

### 1.6.1 Selection and Query Execution

* Provide control to select a data source from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`.
* On preview request, execute the selected data-source query and display resulting recipient rows.

### 1.6.2 Preview Table

* Display columns from a deterministic field contract derived from all DSL-cited sources, plus any additional source-native columns present in preview output.
* The deterministic field contract must include all allowed fields for each cited source (built-in or custom).
* Provide row count and sample-limit indicator in the information pane.
* Provide per-row checkbox selection in the recipients table.
* Provide a header-level checkbox to select/deselect all currently visible rows.
* Display count of currently selected rows in the information pane/status line.
* Persist row selection state for the current browser session only (no database storage).
* Session-persisted selection must survive leaving and re-entering the page in the same browser session.
* Provide generalized recipient sorting controls (`Sort by` and sort direction) and apply them to the displayed preview rows.
* If query returns zero rows, display zero-result message (no error state).
* Provide `Download spreadsheet` action.
* On `Download spreadsheet`, generate `recipients.xlsx` from rows currently shown in the data pane and download it.
* In `recipients.xlsx`, format the header row in bold.
* In `recipients.xlsx`, auto-size each column width to fit header/data content.

### 1.6.3 Safety Rules

* Preview does not send email.
* Preview does not persist changes to recipient-source tables.

## 1.7 Mailshot Test Page

### 1.7.1 Test Setup

* Provide control to select a mailshot.
* Provide control to select one preview/test recipient row from the resolved query output.
* Recipient-row selector must also provide an `All recipients` option.
* Provide `Test email address` input for explicit destination override, defaulting to configured `.env` value `MAILSHOT_TEST_TO_DEFAULT` when set.
* If `MAILSHOT_TEST_TO_DEFAULT` is unset/blank, initialize `Test email address` as blank.
* After successful render-only test, provide an `Open Render Preview` action.
* Render preview opens in an overlay/modal and displays rendered `Message` and rendered `PDFAttachment` side by side in sandboxed iframe previews.
* Render preview includes rendered `Subject` as modal context.

### 1.7.2 Test Mode: Render Only (No Send)

* Provide action `Render Test (No Send)`.
* Perform full Twig template rendering for selected row.
* Display rendered `Subject`, `Message`, and `PDFAttachment` through the render preview modal.
* Render-only UI must not store the full rendered html payload in WordPress options; large rendered payloads should be recomputed after redirect or passed by another size-safe mechanism.
* Do not send SMTP mail.
* Do not write IMAP `Sent` copy.

### 1.7.3 Test Mode: Send Test Email

* Provide action `Send Test Email`.
* Require non-blank `Test email address`.
* When `Test email address` is blank, block send and show validation error in the information pane.
* Use selected test row only for template rendering context.
* Send exactly one email to `Test email address` (not to original recipient address).
* When `All recipients` is selected, send one rendered test email per recipient row, all to `Test email address`.
* Confirmation prompt text must be context-aware:
  * one-row mode: `Send one test email to the entered address?`
  * all-recipient mode: `Send <n> emails to the entered address?`
* `Render Test (No Send)` must require a specific recipient row (not `All recipients`).
* Copy sent test email to IMAP `Sent`.
* Display success/failure status and timestamp.
* Send-test UI must keep the user on the Mailshot Test page and display same-pane feedback. If the server returns non-JSON or an HTTP error, show an inline error rather than navigating to a WordPress critical-error page.
* Apply the configured Download PDF memory limit, when valid, before render/send test actions that may render PDF attachments.

## 1.8 Run Mailshot Page

### 1.8.1 Pre-Run Validation

* Require selected mailshot.
* Require resolved recipient query.
* Block run when recipient query returns zero recipients.
* Display validation failures and do not start run when validation fails.

### 1.8.2 Execution Semantics

* On `Run Mailshot`, execute recipient query and iterate recipients.
* For each recipient row:
  * perform Twig template rendering
  * send one email to the recipient destination email
  * copy successful sends to IMAP `Sent`
* Continue processing remaining recipients after per-recipient failures.
* Provide action `Run Mailshot to Selected Rows` to run only rows selected in `Review Recipients`.
* `Run Mailshot to Selected Rows` must use session-persisted selected recipient keys from `Review Recipients` (same browser session, no database storage).
* If no selected rows are available for the selected mailshot data source, block action with explicit validation message.
* Run page must include hint text: `Select rows for mailshot in Review Recipients page.` with a link to the Review Recipients admin page.

### 1.8.3 Run Results

* Display live or final counters:
  * attempted
  * sent
  * failed
  * skipped (if applicable)
* Provide downloadable or copyable run log in the UI.
* Include per-recipient failure reason where available.

### 1.8.4 Last Run Retry Table

* Persist the outcome of the most recent run only in `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run`.
* Clear `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run` when starting a new `Run Mailshot` execution.
* Each row in `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run` must store:
  * `mailshot_id`
  * `recipient_key` (stable non-email identifier)
  * `recipient_key_field` (source field used to derive key)
  * `recipient_email_last` (most recent resolved destination email)
  * `status`
  * `error_message` and/or `warning_message`
  * `attempt_count`
  * `last_attempt_at`
* Run Mailshot page must display the latest rows for the selected mailshot from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run`.
* Run Mailshot page must provide:
  * `Retry Failed Sends` button to retry all failed rows from the selected mailshot.
  * per-row `Retry` button to retry one failed row by recipient key.
* Retry must resolve recipient data from the current data source and current template values at retry time (not snapshot payload from original run).
* Email address must not be used as the retry key.

## 1.9 Download PDF Page

### 1.9.1 Setup and Validation

* Provide control to select a mailshot.
* Require selected mailshot.
* Require resolved recipient query.
* Block generation when recipient query returns zero recipients.
* Block generation when selected mailshot has blank `PDFAttachment` template.
* Merged PDF generation requires Ghostscript (`gs`) and PHP `exec()` availability.
* ZIP generation uses PHP `ZipArchive` when available; if unavailable, it requires a `zip` CLI binary and PHP `exec()` availability.

### 1.9.2 Output Options

* Provide action to generate and download individual recipient PDFs as a single `.zip`.
* Provide action to generate and download a single merged PDF containing all recipient PDFs.
* Use html-to-pdf conversion of rendered `PDFAttachment` content per recipient row.
* Downloaded merged PDF filename must be based on the selected mailshot `Purpose`, sanitized to safe filename characters, with suffix `_merged.pdf`.
* Downloaded ZIP filename must be based on the selected mailshot `Purpose`, sanitized to safe filename characters, with suffix `_pdfs.zip`.
* If `Purpose` is blank or cannot be loaded, use fallback base name `mailshot_<id>`.
* PDF rendering must use Dompdf with default font `DejaVu Sans` and font subsetting enabled.
* Before Dompdf rendering, normalize contact-symbol characters that commonly fail in PDF fonts:
  * `📞` and `☎` render as a `☎` symbol pinned to `DejaVu Sans`
  * `📧` and `✉` render as a `✉` symbol pinned to `DejaVu Sans`
* Template authors should still prefer PDF-safe symbols (`☎`, `✉`) over color emoji symbols.

### 1.9.3 Results and Logging

* Display generation success/failure status in the UI.
* Generate downloadable files directly (no per-row log table required on this page).

## 1.10 Mailshot Queries Page

Provide a CRUD page for the `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries` table.

### 1.10.1 DSL Built-in Sources

Built-in DSL sources include:

* `contacts`
* `accounts`
* `renewals`

`renewals` is a first-class source and must be joinable to:

* `accounts` via `renewals.account_id = accounts.ID`
* `contacts` through accounts (`renewals.account_id = contacts.Accountid`)

### 1.10.2 DSL Renewal and Membership Filters

DSL filters include:

* `selected-renewal`
* `pending-renewal`
* `fen1-contact`
* `primary-contact`
* `member-or-affiliate-or-parish-council`

`selected-renewal` uses saved field values (`renewals.selected = true`).
`pending-renewal` filters rows to `renewals.status = 'pending'`.

## 2.0 No-Effect and Error Handling

* If `Save` is clicked with no changes, no effect.
* If `Quit`/close is clicked with no changes, close without confirmation.
* If a selected mailshot references a missing `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries` row, block preview/test/run and display error.
* If SMTP send fails for one recipient, record error and continue with next recipient.
* If IMAP copy fails after successful SMTP send, mark warning for that recipient.
* For any DSL sentence, all fields from all cited sources must be available for token insertion and template rendering; missing values may render empty, but fields must not disappear from mailshot field availability.
* Twig template syntax errors must not crash the page or run; show clear error identifying mailshot and template field (`Subject`, `Message`, or `PDFAttachment`).
* Default behavior on template syntax error is fail-fast for that recipient (skip send for recipient, continue run for others) and record error.
* Retry actions must update `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run` rows in-place (incrementing `attempt_count` and updating status/messages/timestamp).

### 2.0.1 IMAP Sent-Copy Contract

* Use per-user configured IMAP sent-folder path from WordPress mailshot credentials.
* IMAP folder names must be encoded/decoded using IMAP UTF-7 rules.
* For each successful SMTP send, attempt a single IMAP append for sent-copy persistence.
* IMAP append failure must not change SMTP send outcome; classify as `warning`, not `failed`.
* IMAP append retries are not automatic in run loop; retry occurs only via explicit operator retry action.
* Sent-copy operations must be idempotent per attempt id; duplicate append for the same attempt id is not allowed.

## 2.1 PDF Assets

### 2.1.1 Purpose

Provide reusable image assets that can be inserted into `PDFAttachment` html templates using Twig syntax and resolved at PDF render time.

### 2.1.2 Navigation and Page

Under `Mailshot`, provide page `PDF Assets`.

### 2.1.3 Data Model

Provide CRUD storage for PDF assets with at least:

* `id` (primary key)
* `name` (unique asset identifier used in token marker)
* `file_name` (original upload filename)
* `mime_type`
* `file_bytes` (or storage reference)
* `width_mm`
* `height_mm`
* `justification` (`left`, `right`, `in-place`)
* `created_at`
* `updated_at`

### 2.1.4 PDF Assets Page Behavior

* List assets in a table.
* Support `Create`, `Read`, `Update`, `Delete`.
* Create/update requires image bytes, provided either by file upload or by direct base64 entry.
* Edit dimensions in millimetres.
* Edit justification (`left`, `right`, `in-place`).
* Validate name, filename-derived or explicit MIME type, positive dimensions, allowed justification, and valid base64 image bytes.

### 2.1.5 Mailshots PDF Editor Integration

* In the `PDF Attachment` editor overlay, add an insert-asset action with dropdown of available assets.
* On insert, place Twig token syntax at cursor position.
* Default insert format is `{{ pdf_asset('asset_name') }}`.
* Asset names used in inserted tokens must be escaped safely as Twig string literals; the saved asset name itself is not renamed during insertion.

### 2.1.6 Rendering Behavior

* On rendering `PDFAttachment` html (Mailshot Test preview/send, Run Mailshot, Download PDF), resolve each inserted PDF asset Twig token to rendered image html for that asset.
* Render image using configured dimensions (`width_mm`, `height_mm`) and configured justification.
* If template references a missing asset, rendering must fail with a clear error naming the missing asset.
* If `pdf_asset()` is unavailable, the asset has no readable bytes, dimensions are invalid, or justification is invalid, rendering must fail with a clear error.
* PNG assets require either GD PNG support or ImageMagick `convert` for runtimes where Dompdf cannot consume the PNG bytes directly; otherwise rendering must fail with a clear error.

### 2.1.7 PDF Filename Derivation

* For each rendered recipient pdf (download and email attachment), if `PDFFilenameDerivedFrom` is set, use row value from that field as base filename.
* Sanitize filename to safe filename characters.
* If `PDFFilenameDerivedFrom` is blank or field value resolves blank, use fallback naming strategy.
* Ensure filename uniqueness within a generated batch (append numeric suffix as needed).

## 2.2 PDF Assets Decisions

* Asset binaries are stored in DB table (`LONGBLOB`) for portability and backup simplicity.
* Supported filename-derived MIME types are `jpg/jpeg`, `png`, `gif`, `webp`, `svg`, `bmp`, `tif`, and `tiff`.
* Scope/ownership: global asset library, editor-capable users (WordPress `edit_pages`) may CRUD.
* Asset naming: unique case-insensitive `name`; renaming does not auto-migrate existing asset insertions in templates and should warn user.
* Marker syntax is removed. PDF asset insertion uses Twig syntax.
* Helper name is `pdf_asset(name)` as a global Twig helper, not a filter.
* Dimension/aspect behavior: fixed width and height in millimetres are emitted as inline image style.
* Justification semantics: `in-place` renders inline at marker location; `left` renders a block-aligned element on its own line; `right` floats the image right so surrounding text can fill the space to its left.
* Missing asset token behavior: fail rendering with a clear error.
* Delete behavior: delete the asset row by id; current implementation does not block deletion for in-use assets.

## 2.3 Mailshot Attachments

### 2.3.1 Page and Layout

Under `Mailshot > Assets`, provide page `Attachments`.

* Data pane: table of attachments sorted by `name`.
* CRUD pane (right): edit `name` and uploaded file.
* Provide actions: `Save`, `New`, `Delete`.

### 2.3.2 CRUD and Validation

* `name` is required and unique.
* Create/update requires file bytes, provided either by file upload or by direct base64 entry.
* Supported filename-derived MIME types are `txt`, `csv`, `pdf`, `doc`, `docx`, `odt`, `rtf`, `html`/`htm`, `json`, `xml`, `jpg`/`jpeg`, `png`, `gif`, `webp`, `svg`, and `zip`.
* Delete behavior: delete the attachment row by id; current implementation does not block deletion for in-use attachments.

### 2.3.3 Send/Run Behavior

* On `Mailshot Test > Send Test Email`, include all selected named attachments.
* On `Run Mailshot`, include all selected named attachments for each recipient email.
* If a selected attachment name is missing, continue send/run and record warning.