# Mailshot Data Source page

## Purpose

Document requirements for a safe, non-admin-friendly mailshot data source definition model.

## Environment baseline for this specification

`members` as a database name means the value of `MEMBERS_REMOTE_MYSQL_DB`.

For this specification:

* Data-source definition persistence (create/update/delete/list of `mailshot_queries`) uses `MAILSHOTS_REMOTE_MYSQL_DB`.
* Source-data query execution (built-in/custom recipient sources) uses `MEMBERS_REMOTE_MYSQL_DB`.

## Problem with prior approach

Creating a new data source currently requires:

* knowledge of MySQL schema details, or
* admin-level access to create DB views/queries.

This blocks non-expert and non-admin users.

## Direction

Use a strict, constrained query language (DSL) for mailshot audience selection.

The DSL text is stored as the data-source definition in `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`, then parsed and compiled into safe SQL executed against approved source-data tables in `MEMBERS_REMOTE_MYSQL_DB`.

## Goals

* Allow non-admin users to define useful audiences.
* Prevent arbitrary SQL.
* Keep execution efficient (push filtering to SQL, avoid large unnecessary joins).
* Make queries explainable and previewable.

## Example requirements (in natural language)

* All contacts with name like "smith".
* All contacts joined to member accounts where contact FEN1 is true.
* All renewal rows marked as selected.
* All pending renewals.
* All accounts where name is not in an exclusion table.

## Strict grammar (v1.5)

Use lowercase keywords only.

### EBNF

```ebnf
query              = source_expr , [ where_clause ] ;

source_expr        = source_term , { source_op , source_term } ;
source_term        = source_ref | "(" , source_expr , ")" ;
source_op          = "and" ;

source_ref         = built_in_source | table_ref ;
built_in_source    = "contacts" | "accounts" | "renewals" ;
table_ref          = ident_part , "." , ident_part ;
ident_part         = identifier | quoted_identifier ;

where_clause       = "where" , predicate_expr ;
predicate_expr     = predicate_term , { bool_op , predicate_term } ;
predicate_term     = [ "not" ] , predicate_atom
                   | "(" , predicate_expr , ")" ;
bool_op            = "and" ;

predicate_atom     = filter_expr
                   | field_ref , compare_op , value
                   | field_ref , ("=" | "!=") , field_ref
                   | field_ref , [ "not" ] , "in" , "(" , field_ref , ")"
                   | field_ref , "in" , "(" , value_list , ")" ;

filter_expr        = filter_name
                   | filter_name , "(" , value_list , ")" ;

filter_name        = "selected-renewal"
                   | "pending-renewal"
                   | "fen1-contact"
                   | "fen2-contact"
                   | "member-or-affiliate-or-parish-council" ;

field_ref          = source_ref , "." , field_name ;
field_name         = identifier | quoted_identifier ;
compare_op         = "=" | "!=" | "contains" | "starts-with" | "ends-with" ;

value_list         = value , { "," , value } ;
value              = quoted_string | number | "true" | "false" ;

identifier         = letter , { letter | digit | "_" } ;
quoted_identifier  = "`" , { character - "`" } , "`" ;
quoted_string      = "'" , { character - "'" } , "'" ;
number             = digit , { digit } ;
letter             = "a"…"z" | "A"…"Z" ;
digit              = "0"…"9" ;
```

## Semantics (v1.5)

* `built_in_source` maps to a predefined table alias.
* For `accounts`, the compiler must implicitly left-join account picklists so additional readable virtual fields are available:
  * `accounts.type` from `picklist_account_type.value`
  * `accounts.public_location` from `picklist_public_location.value`
  * `accounts.account_sector` from `picklist_sector.value`
* `table_ref` maps directly to a table in the remote members database (`schema.table`).
* For `table_ref`, validation must confirm the table exists.
* Query result shape must be source-faithful and deterministic per DSL:
  * each cited source contributes its full allowed field set to an expected-field contract;
  * preview and template-token resolution must preserve availability of every field in that contract, even when raw SQL row keys are sparse;
  * source-native columns from query output may also be present;
  * no unrelated synthetic recipient projection columns are allowed.
* If the resolved source does not expose an email field, validation must return a warning:
  * `Data Source does not have an email field, it cannot be used for a Mailshot`
  * The sentence is still valid and can be saved and previewed.
* `and` between sources means relational intersection using configured join paths (not email-only matching).
* For custom sources, an explicit field equality predicate can provide join semantics.
* `where` applies after source composition.
* `not` negates only the next predicate/group.
* Predefined predicates are `selected-renewal`, `pending-renewal`, `fen1-contact`, `fen2-contact`, and `member-or-affiliate-or-parish-council`.
* `renewals` is a built-in source mapped to membership renewal rows.
* `pending-renewal` applies only when source set includes `renewals` and means `renewals.status = 'pending'`.
* `selected-renewal` applies only when source set includes `renewals` and means `renewals.selected = true`.
* `member-or-affiliate-or-parish-council` applies to account data and means:
  * `accounts.type` is `Member` or `Affiliate`, or
  * `accounts.name` contains `Parish Council`.
* `field_ref` is restricted by whitelist per built-in source.
* `field_ref` is also supported for selected custom sources when table metadata is available.
* Field-to-field comparisons are supported with `=` and `!=` only.
* Set-membership against another field source is supported via:
  * `field_ref in (field_ref)`
  * `field_ref not in (field_ref)`
* For `in(field_ref)` / `not in(field_ref)`, the RHS source is treated as a reference source (subquery semantics), not a joined source.
* `contains`, `starts-with`, and `ends-with` require a literal right-hand side value (not a field reference).
* In v1.5, mixing built-in sources and custom sources in the same sentence is not supported; validation must fail with a clear error.
* For multi-custom-source sentences, all selected custom sources must be connected by explicit `=` field-to-field predicates (graph-connected join semantics), otherwise validation fails.
* `in(field_ref)` / `not in(field_ref)` must compile as subquery membership; the RHS source is a reference source and does not need to appear in `source_expr`.
* Canonical naming in this document uses exact field names as defined in database metadata for the selected source. The implementation must reject invalid names and must not add aliasing/normalization fallback.

### Join-path model (required)

The compiler must use an explicit join graph per source pair. Example v1 join paths:

* `contacts` -> `accounts`: `contacts.Accountid = accounts.ID`
* `accounts` -> `contacts`: `accounts.ID = contacts.Accountid`
* `renewals` -> `accounts`: `renewals.account_id = accounts.ID`
* `accounts` -> `renewals`: `accounts.ID = renewals.account_id`
* `renewals` -> `contacts`: `renewals.account_id = contacts.Accountid`
* `contacts` -> `renewals`: `contacts.Accountid = renewals.account_id`

If no approved join path exists between two sources for `and`, parsing/validation must fail with a clear error.

No implicit join behavior is allowed:

* Selecting a source such as `accounts` must not silently join `contacts` (or any other source).
* Output columns must come only from sources explicitly requested in the sentence.

## Example valid queries

* `contacts`
* `contacts and accounts where fen1-contact`
* `contacts and accounts where fen2-contact`
* `accounts and contacts where contacts.Accountid = accounts.ID`
* `contacts where contacts.Last contains 'smith'`
* `contacts and accounts where accounts.Type = 'Member' and contacts.FENContact1 = true`
* `accounts where member-or-affiliate-or-parish-council`
* `renewals where pending-renewal`
* `renewals where selected-renewal`
* `renewals and accounts and contacts where pending-renewal`
* `accounts where accounts.Name not in (members.ExcludedAccounts.ExcludedAccount)`
* `members.mailshot_test`

## Example invalid queries

* `select * from members` (raw SQL not allowed)
* `contacts where drop table` (unknown tokens)
* `contacts where accounts.Name = 'x'` (invalid if `accounts` is not included in source expression)
* `contacts or accounts` (`or` is not supported; use `and` only)
* `accounts where pending-renewal` (invalid: filter requires `renewals` source)
* `accounts where fen1-contact` (invalid: filter requires `contacts` source)
* `renewals where member-or-affiliate-or-parish-council` (invalid: filter requires `accounts` source)
* `accounts and contacts where contacts.Accountid contains accounts.ID` (invalid: field-to-field supports only `=`/`!=`)
* `accounts where accounts.Name contains members.ExcludedAccounts.ExcludedAccount` (invalid: `contains` requires a literal RHS)

## Compilation and performance requirements

* Parse DSL to AST.
* Validate AST against allowed sources, fields, and operators.
* Validate all `and` source combinations against approved join graph metadata.
* Do not perform implicit source joins during compilation.
* Compile AST to parameterized SQL only (no string-concatenated SQL).
* Push filters into SQL `WHERE` / join conditions.
* Preview must not assume fixed contact/account projection columns.
* Compiler/runtime must enforce deterministic source-field availability from DSL-cited sources.
* Preview column derivation must include:
  * all fields from the DSL expected-field contract, and
  * union of keys from returned rows (not just first row keys), to avoid hiding sparse/source-native columns.
* For mailshot execution, email remains the only required field.
* Provide preview endpoints: count + sample rows before execution.

## Storage model

Store:

* `dsl_text` (editable source).
* `dsl_text` is the single source of truth for execution, preview, and validation.
* No SQL fallback path is permitted when `dsl_text` is missing/empty; this must be treated as an error.

Optional:

* normalized AST JSON for diagnostics/explain.
* `sql` retained as blank/deprecated for backward compatibility when the legacy column exists, and never executed.

## User interface requirements

Provide a UI to view, create, update, validate, and preview DSL sentences.

### Mailshot data source page

Add a dedicated page "Data Sources" under FECA Mailshots admin page for managing data source sentences.

Top regions:

* Context pane:
  * page title `Mailshot Data Sources`
* Information pane:
  * validation errors, save status, preview status
* Statistics pane:
  * total saved data sources
  * selected data source id/name

### Data source list (view)

Provide a table/list showing existing data sources with columns:

* `ID`
* `Name`
* `DSL sentence`
* `Updated at`

List behavior:

* single-row selection
* text filter by `Name` or sentence content
* sort by `Name` and `Updated at`

Selection to persist between page loads.

### Editor (create/update sentence)

Provide an editor panel with:

* `Name` input (required)
* multi-line `DSL sentence` input (required)
* read-only `Last validated status`

Actions:

* `New`:
  * clear editor and start create mode
* `Save`:
  * create or update current record
  * enabled only when form is valid and has unsaved changes
* `Discard changes`:
  * revert editor to last saved state
* `Delete`:
  * confirmation required
  * blocked when the data source is referenced by one or more mailshots

### Validation UX

Provide explicit validation before save and on-demand:

* `Validate sentence` button
* parse + semantic validation against:
  * grammar
  * source/field/operator whitelist
  * join-path rules

On validation result:

* show `Valid` / `Invalid`
* show `Warnings` separately from errors
* for invalid:
  * show user-friendly error message
  * include line/column when available
  * do not save invalid sentence
* for warning-only cases (for example missing email field):
  * allow save
  * allow preview
  * indicate the data source cannot be used for sending a mailshot

### Preview UX

Provide `Preview recipients` action for selected sentence:

* show recipient count
* show sample rows (for example first 50)
* derive displayed columns from the deterministic expected-field contract plus returned row shape
* do not assume fixed contact/account projection columns
* show execution time
* do not execute/send mailshot from this page

### DSL Sentence Builder (required)

Provide a `Build DSL` action in the data-source editor.

On click, open a modal/overlay builder that allows users to construct full valid DSL without memorising grammar keywords, filter names, or field names.

Builder layout:

* Top sub-pane: `Data Sources`
* Bottom sub-pane: `Constraints / Filters`
* Footer: generated DSL preview + actions

#### Top sub-pane: Data Sources

Provide source selection controls for:

* `contacts`
* `accounts`
* `renewals`
* custom table source (schema/table selector or validated text input)

Behavior:

* multiple source selection is combined using `and`
* order of selected sources is preserved in generated sentence
* selecting an invalid source combination is blocked in UI with clear message

#### Bottom sub-pane: Constraints / Filters

Provide controls to add one or more constraint rows.

Each row supports one of:

* predefined filter
* field comparison
* grouped expression with `not`

Predefined filter control:

* dropdown label uses user-friendly text, not raw DSL token
* examples:
  * `Renewal is selected` -> `selected-renewal`
  * `Renewal is pending` -> `pending-renewal`
  * `Contact is FEN1` -> `fen1-contact`
  * `Contact is FEN2` -> `fen2-contact`
  * `Account is member/affiliate/parish council` -> `member-or-affiliate-or-parish-council`

Field comparison row:

* source dropdown (restricted to selected sources)
* field dropdown (restricted to fields allowed for selected source)
* operator dropdown (`=`, `!=`, `contains`, `starts-with`, `ends-with`, `in`)
* RHS mode dropdown:
  * literal value input (typed by expected value type), or
  * source+field selector for field-to-field comparisons and set-membership (`in`)
* for `in`, provide multi-value token input

Logical composition:

* all rows combine with `and` in v1.5
* each row can be negated via checkbox (`not`)
* optional group rows allow nested bracketed expressions

#### Generated DSL + Sync

Footer must show:

* read-only generated DSL sentence (live update)
* validation status
* parse/semantic errors with line/column mapped to relevant builder controls

Actions:

* `Apply` writes generated DSL to the editor text area
* `Cancel` closes overlay with no change
* `Reset` clears builder selections

Sync rules:

* opening builder from existing DSL pre-populates controls when sentence is parseable
* if existing DSL is not parseable, show message and allow user to start from clean builder
* direct text edits remain supported; builder is not the only editing path

#### Safety and usability requirements

* UI must prevent creating syntactically invalid DSL states
* UI must prevent selecting filters incompatible with current source set
* custom table sources must be validated before Apply
* generated DSL must always conform to strict grammar in this spec

### API expectations for UI

The UI expects backend endpoints for:

* list data sources
* get one data source
* validate sentence
* create data source
* update data source
* delete data source
* preview sentence (count + sample)
* list schemas for custom source selection
* list tables for selected schema
* list fields for selected source (built-in or custom)

WordPress REST route contract:

* `GET /wp-json/mailshots/v1/data-sources`
* `GET /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/validate`
* `POST /wp-json/mailshots/v1/data-sources`
* `PUT /wp-json/mailshots/v1/data-sources/{id}`
* `DELETE /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/preview`
* `POST /wp-json/mailshots/v1/data-sources/{id}/preview`
* `GET /wp-json/mailshots/v1/data-sources/schemas`
* `GET /wp-json/mailshots/v1/data-sources/tables?schema=...`
* `GET /wp-json/mailshots/v1/data-sources/source-fields?source=...`

Auth/capability requirements:

* all routes require authenticated WordPress users;
* in this plugin, editor-capable operational access is the baseline:
  * create/update/delete/read/preview/validate all require WordPress `edit_pages` (editor or administrator);
* plugin Setup/settings remains administrator-only under WordPress `manage_options` (defined in top-level policy).

All endpoints must enforce server-side validation even if client validates first.

## Additional functional requirements (in scope)

* Mailshot Data Source preview, Mailshot Recipients Preview, and Mailshot Test preview must all display source-faithful columns using the same result-shape rule.
* DSL Builder must support custom sources via schema + table selection (not text-only entry), with manual text fallback only when metadata APIs are unavailable.
* After a custom source is added in Builder, it must be immediately available in:
  * source dropdowns,
  * LHS field dropdowns,
  * RHS field dropdowns (for field comparisons).
* Builder must support `field not in (other_source.other_field)` generation using RHS source + field controls (not value-type literal controls).
* Builder prefill must parse and restore field references where source names are qualified (for example `members.ExcludedAccounts.ExcludedAccount`).
* Validation errors must be explicit for unsupported/ineffective predicates; no silent no-op predicate behavior.

## Implementation details

This section is normative guidance for regenerating the feature from scratch.

### 1. Data model

* Table: `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`
* Required columns:
  * `ID` (PK)
  * `name`
  * `dsl_text`
  * `updated_at` (recommended)
* `dsl_text` is the only executable representation.
* Persist `sql` as blank/deprecated if column still exists for backward compatibility; do not execute it.

### 2. Backend architecture

Build a strict pipeline:

1. Tokenize DSL text.
2. Parse to AST.
3. Validate AST (grammar + semantics).
4. Compile AST to parameterized SQL and parameters.
5. Execute SQL for preview/send paths.

Keep parser/validator/compiler in a dedicated module (for example `mailshotDsl.js`), and route handlers thin.

### 3. Parser/AST contract

* AST should separate `source_expr` and `where`.
* Predicates should distinguish:
  * named filters,
  * compare literal,
  * compare field-to-field,
  * `in(value_list)`,
  * `in(field_ref)` / `not in(field_ref)`.
* Preserve token line/column for diagnostics.

### 4. Validation rules (server-side)

* Validate source compatibility via explicit join-path metadata.
* Validate filter applicability against selected sources.
* Validate field refs against per-source field whitelist:
  * built-ins: static map
  * custom: metadata-provider abstraction (not direct SQL dependency in validator logic)
* Enforce:
  * field-to-field only `=` / `!=`
  * literal RHS only for `contains` / `starts-with` / `ends-with`
* Return warnings (not errors) for no-email-source sentences.
* Metadata-provider minimum DB grants must be documented and kept minimal for the read-only DB user.

### 5. SQL compilation strategy

Use parameterized SQL only.

* Built-in-only source expressions:
  * compile from fixed source metadata (table + alias + base where).
  * compile approved joins only.
* Custom-only source expressions:
  * single source: `SELECT * FROM schema.table WHERE ...`
  * multi source: require explicit `=` field join connectivity and compile with aliased joins.
* `in(field_ref)` / `not in(field_ref)`:
  * compile as `EXISTS` / `NOT EXISTS` subqueries.

### 6. Preview/result-shape rules

* Normalize row values to JSON-safe scalars.
* Enforce expected-field contract from DSL-cited sources.
* Keep source-native returned columns available.
* It is allowed to expose normalized source-field aliases required to satisfy the expected-field contract.
* Do not add unrelated synthetic recipient projection columns.
* Derive display columns from expected-field contract plus union of keys in returned rows.
* Prune-all-empty columns only if explicitly configured and documented; default should preserve source-faithful shape.

### 7. API surface

Implement at minimum:

* `GET /wp-json/mailshots/v1/data-sources`
* `GET /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/validate`
* `POST /wp-json/mailshots/v1/data-sources`
* `PUT /wp-json/mailshots/v1/data-sources/{id}`
* `DELETE /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/preview`
* `POST /wp-json/mailshots/v1/data-sources/{id}/preview`
* `GET /wp-json/mailshots/v1/data-sources/schemas`
* `GET /wp-json/mailshots/v1/data-sources/tables?schema=...`
* `GET /wp-json/mailshots/v1/data-sources/source-fields?source=...`

All endpoints must validate server-side and return structured errors:
`{ ok:false, error, line?, column? }`.

### 8. UI generation guidance

Mailshot Data Source page:

* left: data source list with select/filter/sort.
* right: editor + validation + preview.
* include `Build DSL` modal.

Builder modal:

* source selection section (built-in + schema/table custom add).
* constraints section with row editor (filter/comparison).
* responsive row layout that wraps controls on narrow widths.
* RHS field mode must expose source + field selectors.
* show generated DSL live; validate with debounce.
* `Apply` writes DSL text; `Cancel` leaves editor unchanged.

### 9. Safety and regression test guidance

Minimum automated tests:

* parser positive/negative cases (line/column assertions).
* semantic validation failures (join path, filter applicability, field whitelist).
* compile output shape for:
  * built-in joins,
  * custom joins,
  * `in(field_ref)` and `not in(field_ref)`.
* preview shape:
  * includes all expected fields from cited sources, and
  * does not introduce unrelated synthetic recipient projection columns.
* builder prefill for qualified source field refs.
* no-op predicate prevention (must error, not silently ignore).