# 1 Mailshot

Use and comply with `requirements/ui_design.md`.

## 1.0 Environment Baseline

* All writable mailshot tables and mailshot asset binaries are stored in the database named by `.env` variable `MAILSHOTS_REMOTE_MYSQL_DB`.
* This includes `mailshots`, `mailshot_queries`, `mailshot_attachments`, `mailshot_last_run`, and any mailshot asset tables created for PDF/image assets.
* In this document, references in the form `MAILSHOTS_REMOTE_MYSQL_DB.<table>` are normative.
* Recipient/source data reads used to resolve mailshot recipients and DSL source fields must use `.env` variable `MEMBERS_REMOTE_MYSQL_DB`.
* SMTP/IMAP credentials are sourced from WordPress-side per-user mailshot credential storage.
* If user credentials are missing, send/run actions must be blocked and the UI must direct users to the WordPress mailshot credentials page.

## 1.1 Purpose

Provide a Mailshot feature to:

* manage mailshot definitions
* preview recipient data and template rendering
* test mailshots safely
* execute full mailshot runs

## 1.2 Data Model and Dependencies

* Use `MAILSHOTS_REMOTE_MYSQL_DB.mailshots` as the source of mailshot definitions.
* Use `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries` as the source of recipient/data-source queries.
* Send mail using per-user SMTP/IMAP credentials from WordPress-side mailshot credential storage.
* If user credentials are missing, block send/run and direct user to the WordPress mailshot credentials page.
* Copy successfully sent messages to configured IMAP `Sent` folder.

### 1.2.1 Schema Reference: `MAILSHOTS_REMOTE_MYSQL_DB.mailshots`

| Column                     | Type             | Null | Key | Notes                                                                               |
| -------------------------- | ---------------- | ---- | --- | ----------------------------------------------------------------------------------- |
| `id`                     | `int(11)`      | NO   | PK  | Auto-increment                                                                      |
| `Purpose`                | `varchar(100)` | YES  |     | Human-readable mailshot purpose                                                     |
| `DataSource`             | `varchar(100)` | NO   |     | Data-source name matching `mailshot_queries.name`                                 |
| `CC`                     | `varchar(100)` | YES  |     | Optional CC list                                                                    |
| `BCC`                    | `varchar(100)` | YES  |     | Optional BCC list                                                                   |
| `Subject`                | `longtext`     | NO   |     | Message subject template                                                            |
| `Message`                | `longtext`     | NO   |     | Message body template                                                               |
| `PDFAttachment`          | `longtext`     | YES  |     | Optional PDF attachment html template                                               |
| `AttachmentNames`        | `longtext`     | YES  |     | JSON array of named entries from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_attachments` |
| `PDFFilenameDerivedFrom` | `varchar(128)` | YES  |     | Optional data-source field used as base filename for generated PDFs                 |
| `ReplyTo`                | `varchar(100)` | YES  |     | Optional reply-to address                                                           |

### 1.2.2 Schema Reference: `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`

| Column       | Type             | Null | Key    | Notes                                                    |
| ------------ | ---------------- | ---- | ------ | -------------------------------------------------------- |
| `ID`       | `int(11)`      | NO   | PK     | Auto-increment                                           |
| `name`     | `varchar(100)` | NO   | UNIQUE | Data-source name used by `mailshots.DataSource`        |
| `dsl_text` | `longtext`     | YES  |        | DSL sentence (source of truth)                           |
| `sql`      | `longtext`     | NO   |        | Legacy column retained; compile target not user-authored |

## 1.3 Navigation and Pages

Under `Mailshot`, provide child pages:

* `Mailshots`
* `Assets`
* `Mailshot Recipients Preview`
* `Mailshot Test`
* `Run Mailshot`
* `Download PDF`

Under `Mailshot > Assets`, provide:

* `Mailshot Data Source > Data Sources`
* `PDF Assets`
* `Attachments`

## 1.3.1 Attachments Data Model

Use `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_attachments` as reusable uploaded attachments with:

* `id`
* `name` (unique)
* `file_name`
* `mime_type`
* `file_bytes`
* `created_at`
* `updated_at`

Persist selected named attachments on each mailshot in `MAILSHOTS_REMOTE_MYSQL_DB.mailshots.AttachmentNames` (JSON array of attachment names).

## 1.4 Mailshots Page (`MAILSHOTS_REMOTE_MYSQL_DB.mailshots` CRUD)

### 1.4.1 Data Pane

* Display rows from `MAILSHOTS_REMOTE_MYSQL_DB.mailshots`.
* In the list/table view, hide internal `id` and display user-facing columns (`Purpose`, `DataSource`, `Subject`).
* Sort the mailshot list by `Purpose` (ascending).
* Provide editing for all mutable fields.
* Highlight unsaved changed fields.
* Use a separate pane for editing the fields.  Note that the Message field is multiline html-formatted and should be edited using an html edit control.  The design of the page should maximise room for this field
* Provide a separate html edit control for `PDFAttachment` (optional template) using the same html editing features as `Message`.
* Message and PDF Attachment html editors should open in overlay panes to maximise editing room.
* Provide `PDF Filename Derived From` dropdown populated from available data-source fields for the selected `DataSource`.
* Persist `PDF Filename Derived From` as `MAILSHOTS_REMOTE_MYSQL_DB.mailshots.PDFFilenameDerivedFrom`.
* The Data Pane also include a table of mailshots,  width limited to the page width,  with truncation of fields.
* Keep editing responsive; do not trigger per-keystroke save/reload.
* Data-source pulldowns must be sorted by data-source `name` (ascending).
* Show available data-source fields in the editor and allow one-click token insertion into `Subject`, `Message`, and `PDFAttachment`.
* Token insertion must generate Twig tokens using `{{ ... }}` syntax (not `m_...` syntax).
* Field insert token format default is `{{ field_name }}` where `field_name` is a canonical safe field token derived from the selected source field.
* `Subject`, `Message`, and `PDFAttachment` are always rendered through Twig at render/send time.
* In editor pane, provide `Attachments` dropdown (initially blank) sourced from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_attachments.name`.
* Provide `Add selected attachment` button to append selected item into mailshot attachment list.
* Allow removing previously selected attachments from the mailshot attachment list.

### 1.4.2 Action Pane

* Provide `Save` button. Enable only when unsaved changes exist.
* Provide `Discard Changes` button. Enable only when unsaved changes exist.
* Provide `New` button to add a new `MAILSHOTS_REMOTE_MYSQL_DB.mailshots` row.
* Provide `Delete` button. Enable only when one row is selected.
* On `Delete`, require confirmation before delete is applied.

### 1.4.3 Validation

* Require `DataSource` to match an existing `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries.name`.
* Prevent save when `Subject` or `Message` are blank.
* Display validation errors in the information pane.

## 1.5 Template Engine (Twig)

* Use Twig as the template engine for mailshot rendering.
* All template tokens and expressions use Twig syntax.
* Template context is the selected recipient row (plus approved helper functions/filters).
* Default undefined-variable behavior is non-fatal and renders as empty string while recording a warning.
* Default escaping policy is:
  * `Subject`: unescaped text output.
  * `Message`: unescaped only when explicitly requested in template (`| raw`), otherwise escaped.
  * `PDFAttachment`: unescaped only when explicitly requested in template (`| raw`), otherwise escaped.
* Control-flow tags (`{% if %}`, `{% for %}`, `{% set %}`) are allowed in `Message` and `PDFAttachment`, but disallowed in `Subject` by default.

## Mailshot Data Sources Page

This is defined in `requirements/mailshot_data_source.md`.

## 1.6 Mailshot Recipients Preview Page

### 1.6.1 Selection and Query Execution

* Provide control to select a data source from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`.
* On preview request, execute the selected data-source query and display resulting recipient rows.

### 1.6.2 Preview Table

* Display columns from a deterministic field contract derived from all DSL-cited sources, plus any additional source-native columns present in preview output.
* The deterministic field contract must include all allowed fields for each cited source (built-in or custom).
* Provide row count and sample-limit indicator in the information pane.
* Provide generalized recipient sorting controls (`Sort by` and sort direction) and apply them to the displayed preview rows.
* If query returns zero rows, display zero-result message (no error state).
* Provide `Download spreadsheet` action.
* On `Download spreadsheet`, generate `recipients.xlsx` from rows currently shown in the data pane and download it.
* In `recipients.xlsx`, format the header row in bold.
* In `recipients.xlsx`, auto-size each column width to fit header/data content.

### 1.6.3 Safety Rules

* Preview does not send email.
* Preview does not persist changes to recipient-source tables.

## 1.7 Mailshot Test Page

### 1.7.1 Test Setup

* Provide control to select a mailshot.
* Provide control to select one preview/test recipient row from the resolved query output.
* Provide `Test email address` input for explicit destination override, defaulting to configured `.env` value `MAILSHOT_TEST_TO_DEFAULT` when set.
* If `MAILSHOT_TEST_TO_DEFAULT` is unset/blank, initialize `Test email address` as blank.
* Rendered Message control must display the html message according to the html formatting.
* Provide a rendered `PDFAttachment` html preview pane below the rendered message preview pane.
* The rendered Message and rendered PDF Attachment preview panes should default to equal heights and support drag-resize via a divider.

### 1.7.2 Test Mode: Render Only (No Send)

* Provide action `Render Test (No Send)`.
* Perform full Twig template rendering for selected row.
* Display rendered `Subject`, `Message`, and `PDFAttachment`.
* Do not send SMTP mail.
* Do not write IMAP `Sent` copy.

### 1.7.3 Test Mode: Send Test Email

* Provide action `Send Test Email`.
* Require non-blank `Test email address`.
* When `Test email address` is blank, block send and show validation error in the information pane.
* Use selected test row only for template rendering context.
* Send exactly one email to `Test email address` (not to original recipient address).
* Copy sent test email to IMAP `Sent`.
* Display success/failure status and timestamp.
* Keep rendered `Subject`, `Message`, and `PDFAttachment` visible after send.

## 1.8 Run Mailshot Page

### 1.8.1 Pre-Run Validation

* Require selected mailshot.
* Require resolved recipient query.
* Block run when recipient query returns zero recipients.
* Display validation failures and do not start run when validation fails.

### 1.8.2 Execution Semantics

* On `Run Mailshot`, execute recipient query and iterate recipients.
* For each recipient row:
  * perform Twig template rendering
  * send one email to the recipient destination email
  * copy successful sends to IMAP `Sent`
* Continue processing remaining recipients after per-recipient failures.

### 1.8.3 Run Results

* Display live or final counters:
  * attempted
  * sent
  * failed
  * skipped (if applicable)
* Provide downloadable or copyable run log in the UI.
* Include per-recipient failure reason where available.

### 1.8.4 Last Run Retry Table

* Persist the outcome of the most recent run only in `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run`.
* Clear `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run` when starting a new `Run Mailshot` execution.
* Each row in `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run` must store:
  * `mailshot_id`
  * `recipient_key` (stable non-email identifier)
  * `recipient_key_field` (source field used to derive key)
  * `recipient_email_last` (most recent resolved destination email)
  * `status`
  * `error_message` and/or `warning_message`
  * `attempt_count`
  * `last_attempt_at`
* Mailshots page must display the latest rows for the selected mailshot from `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run`.
* Mailshots page must provide:
  * `Retry Failed Sends` button to retry all failed rows from the selected mailshot.
  * per-row `Retry` button to retry one failed row by recipient key.
* Retry must resolve recipient data from the current data source and current template values at retry time (not snapshot payload from original run).
* Email address must not be used as the retry key.

## 1.9 Download PDF Page

### 1.9.1 Setup and Validation

* Provide control to select a mailshot.
* Require selected mailshot.
* Require resolved recipient query.
* Block generation when recipient query returns zero recipients.
* Block generation when selected mailshot has blank `PDFAttachment` template.

### 1.9.2 Output Options

* Provide action to generate and download individual recipient PDFs as a single `.zip`.
* Provide action to generate and download a single merged PDF containing all recipient PDFs.
* Use html-to-pdf conversion of rendered `PDFAttachment` content per recipient row.

### 1.9.3 Results and Logging

* Display generation success/failure status in the UI.
* Generate downloadable files directly (no per-row log table required on this page).

## 1.10 Mailshot Queries Page

Provide a CRUD page for the `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries` table.

### 1.10.1 DSL Built-in Sources

Built-in DSL sources include:

* `contacts`
* `accounts`
* `renewals`

`renewals` is a first-class source and must be joinable to:

* `accounts` via `renewals.account_id = accounts.ID`
* `contacts` through accounts (`renewals.account_id = contacts.Accountid`)

### 1.10.2 DSL Renewal and Membership Filters

DSL filters include:

* `selected-renewal`
* `pending-renewal`
* `fen1-contact`
* `fen2-contact`
* `member-or-affiliate-or-parish-council`

`selected-renewal` uses saved field values (`renewals.selected = true`).
`pending-renewal` filters rows to `renewals.status = 'pending'`.

## 2.0 No-Effect and Error Handling

* If `Save` is clicked with no changes, no effect.
* If `Discard Changes` is clicked with no changes, no effect.
* If a selected mailshot references a missing `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries` row, block preview/test/run and display error.
* If SMTP send fails for one recipient, record error and continue with next recipient.
* If IMAP copy fails after successful SMTP send, mark warning for that recipient.
* For any DSL sentence, all fields from all cited sources must be available for token insertion and template rendering; missing values may render empty, but fields must not disappear from mailshot field availability.
* Twig template syntax errors must not crash the page or run; show clear error identifying mailshot and template field (`Subject`, `Message`, or `PDFAttachment`).
* Default behavior on template syntax error is fail-fast for that recipient (skip send for recipient, continue run for others) and record error.
* Retry actions must update `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_last_run` rows in-place (incrementing `attempt_count` and updating status/messages/timestamp).

### 2.0.1 IMAP Sent-Copy Contract

* Use per-user configured IMAP sent-folder path from WordPress mailshot credentials.
* IMAP folder names must be encoded/decoded using IMAP UTF-7 rules.
* For each successful SMTP send, attempt a single IMAP append for sent-copy persistence.
* IMAP append failure must not change SMTP send outcome; classify as `warning`, not `failed`.
* IMAP append retries are not automatic in run loop; retry occurs only via explicit operator retry action.
* Sent-copy operations must be idempotent per attempt id; duplicate append for the same attempt id is not allowed.

## 2.1 PDF Assets

### 2.1.1 Purpose

Provide reusable image assets that can be inserted into `PDFAttachment` html templates using Twig syntax and resolved at PDF render time.

### 2.1.2 Navigation and Page

Under `Mailshot`, provide page `PDF Assets`.

### 2.1.3 Data Model

Provide CRUD storage for PDF assets with at least:

* `id` (primary key)
* `name` (unique asset identifier used in token marker)
* `file_name` (original upload filename)
* `mime_type`
* `file_bytes` (or storage reference)
* `width_mm`
* `height_mm`
* `justification` (`left`, `right`, `in-place`)
* `created_at`
* `updated_at`

### 2.1.4 PDF Assets Page Behavior

* List assets in a table.
* Support `Create`, `Read`, `Update`, `Delete`.
* Upload graphic file during create/update.
* Edit dimensions in millimetres.
* Edit justification (`left`, `right`, `in-place`).
* Validate file type and size constraints.

### 2.1.5 Mailshots PDF Editor Integration

* In the `PDF Attachment` editor overlay, add an insert-asset action with dropdown of available assets.
* On insert, place Twig token syntax at cursor position.
* Default insert format is `{{ pdf_asset('asset_name') }}`.
* Asset names used in inserted tokens must be normalized to safe token characters.

### 2.1.6 Rendering Behavior

* On rendering `PDFAttachment` html (Mailshot Test preview/send, Run Mailshot, Download PDF), resolve each inserted PDF asset Twig token to rendered image html for that asset.
* Render image using configured dimensions (`width_mm`, `height_mm`) and configured justification.
* If template references missing asset, do not fail entire run; record warning and render explicit missing-asset placeholder html.
* Missing asset placeholder default is `[missing pdf asset: <asset_name>]`.

### 2.1.7 PDF Filename Derivation

* For each rendered recipient pdf (download and email attachment), if `PDFFilenameDerivedFrom` is set, use row value from that field as base filename.
* Sanitize filename to safe filename characters.
* If `PDFFilenameDerivedFrom` is blank or field value resolves blank, use fallback naming strategy.
* Ensure filename uniqueness within a generated batch (append numeric suffix as needed).

## 2.2 PDF Assets Decisions

* Asset binaries are stored in DB table (`LONGBLOB`) for portability and backup simplicity.
* Allowed upload formats: `png`, `jpg/jpeg`, `svg`.
* Scope/ownership: global asset library, all authenticated users may CRUD.
* Asset naming: unique case-insensitive `name`; renaming does not auto-migrate existing asset insertions in templates and should warn user.
* Marker syntax is removed. PDF asset insertion uses Twig syntax.
* Helper name is `pdf_asset(name)` as a global Twig helper, not a filter.
* Dimension/aspect behavior: fixed width+height in mm with `object-fit: contain` (no crop/stretch distortion).
* Justification semantics: `in-place` renders inline at marker location; `left` and `right` render block-aligned elements on their own line (no text-wrap behavior).
* Upload limits: max file size `2MB`, max image dimensions `4000x4000`.
* Missing asset token behavior: continue processing, emit warning, render explicit missing-asset placeholder.
* Delete behavior for in-use assets: block delete when asset helper usage is referenced by any `PDFAttachment` template and show referencing mailshot count.

## 2.3 Mailshot Attachments

### 2.3.1 Page and Layout

Under `Mailshot > Assets`, provide page `Attachments`.

* Data pane: table of attachments sorted by `name`.
* CRUD pane (right): edit `name` and uploaded file.
* Provide actions: `Save`, `New`, `Delete`.

### 2.3.2 CRUD and Validation

* `name` is required and unique.
* Create requires uploaded file bytes.
* Update allows metadata-only edits without replacing file.
* Delete must be blocked while referenced by any `MAILSHOTS_REMOTE_MYSQL_DB.mailshots.AttachmentNames`.

### 2.3.3 Send/Run Behavior

* On `Mailshot Test > Send Test Email`, include all selected named attachments.
* On `Run Mailshot`, include all selected named attachments for each recipient email.
* If a selected attachment name is missing, continue send/run and record warning.