feca-mailshots-plugin/requirements/mailshot_data_source.md

# Mailshot Data Source page

## Purpose

Document requirements for a safe, non-admin-friendly mailshot data source definition model.

## Environment baseline for this specification

`members` as a database name means the value of `MEMBERS_REMOTE_MYSQL_DB`.
`fen` as a database name means the value of `FEN_REMOTE_MYSQL_DB`.

For this specification:

* Data-source definition persistence (create/update/delete/list of `mailshot_queries`) uses `MAILSHOTS_REMOTE_MYSQL_DB`.
* Source-data query execution for membership built-in/custom recipient sources uses `MEMBERS_REMOTE_MYSQL_DB`.
* Source-data query execution for FEN editorial, advertising, and invoice built-in sources uses `FEN_REMOTE_MYSQL_DB`.

## Problem with prior approach

Creating a new data source currently requires:

* knowledge of MySQL schema details, or
* admin-level access to create DB views/queries.

This blocks non-expert and non-admin users.

## Direction

Use a strict, constrained query language (DSL) for mailshot audience selection.

The DSL text is stored as the data-source definition in `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`, then parsed and compiled into safe SQL executed against approved source-data tables in `MEMBERS_REMOTE_MYSQL_DB` and `FEN_REMOTE_MYSQL_DB`.

## Goals

* Allow non-admin users to define useful audiences.
* Prevent arbitrary SQL.
* Keep execution efficient (push filtering to SQL, avoid large unnecessary joins).
* Make queries explainable and previewable.

## Example requirements (in natural language)

* All contacts with name like "smith".
* All contacts joined to member accounts where contact FEN1 is true.
* All renewal rows marked as selected.
* All pending renewals.
* All accounts where name is not in an exclusion table.

## Strict grammar (v1.6)

Use lowercase keywords only.

### EBNF

```ebnf
query              = source_expr , [ where_clause ] ;

source_expr        = source_term , { source_op , source_term } ;
source_term        = source_ref | "(" , source_expr , ")" ;
source_op          = "and" ;

source_ref         = built_in_source | table_ref ;
built_in_source    = "contacts"
                   | "accounts"
                   | "renewals"
                   | "advertisers"
                   | "ads"
                   | "pages"
                   | "articles"
                   | "issues"
                   | "invoices" ;
table_ref          = ident_part , "." , ident_part ;
ident_part         = identifier | quoted_identifier ;

where_clause       = "where" , predicate_expr ;
predicate_expr     = predicate_term , { bool_op , predicate_term } ;
predicate_term     = [ "not" ] , predicate_atom
                   | "(" , predicate_expr , ")" ;
bool_op            = "and" ;

predicate_atom     = filter_expr
                   | field_ref , compare_op , value
                   | field_ref , ("=" | "!=") , field_ref
                   | field_ref , [ "not" ] , "in" , "(" , field_ref , ")"
                   | field_ref , "in" , "(" , value_list , ")" ;

filter_expr        = filter_name
                   | filter_name , "(" , value_list , ")" ;

filter_name        = "selected-renewal"
                   | "pending-renewal"
                   | "selected"
                   | "page-in-issue"
                   | "ad-in-issue"
                   | "pending-invoice"
                   | "selected-invoice"
                   | "invoice-ids"
                   | "fen1-contact"
                   | "primary-contact"
                   | "member-or-affiliate-or-parish-council" ;

field_ref          = source_ref , "." , field_name ;
field_name         = identifier | quoted_identifier ;
compare_op         = "=" | "!=" | "contains" | "starts-with" | "ends-with" ;

value_list         = value , { "," , value } ;
value              = quoted_string | number | "true" | "false" ;

identifier         = letter , { letter | digit | "_" } ;
quoted_identifier  = "`" , { character - "`" } , "`" ;
quoted_string      = "'" , { character - "'" } , "'" ;
number             = digit , { digit } ;
letter             = "a"…"z" | "A"…"Z" ;
digit              = "0"…"9" ;
```

## Semantics (v1.6)

* `built_in_source` maps to a predefined table alias.
* For `accounts`, the compiler must implicitly left-join account picklists so additional readable virtual fields are available:
  * `accounts.type` from `picklist_account_type.value`
  * `accounts.public_location` from `picklist_public_location.value`
  * `accounts.account_sector` from `picklist_sector.value`
* `table_ref` maps directly to a table in the remote members database (`schema.table`).
* For `table_ref`, validation must confirm the table exists.
* Query result shape must be source-faithful and deterministic per DSL:
  * each cited source contributes its full allowed field set to an expected-field contract;
  * preview and template-token resolution must preserve availability of every field in that contract, even when raw SQL row keys are sparse;
  * source-native columns from query output may also be present;
  * no unrelated synthetic recipient projection columns are allowed.
* If the resolved source does not expose an email field, validation must return a warning:
  * `Data Source does not have an email field, it cannot be used for a Mailshot`
  * The sentence is still valid and can be saved and previewed.
* `and` between sources means relational intersection using configured join paths (not email-only matching).
* For custom sources, an explicit field equality predicate can provide join semantics.
* `where` applies after source composition.
* `not` negates only the next predicate/group.
* Predefined predicates are `selected-renewal`, `pending-renewal`, `selected`, `page-in-issue`, `ad-in-issue`, `pending-invoice`, `selected-invoice`, `invoice-ids`, `fen1-contact`, `primary-contact`, and `member-or-affiliate-or-parish-council`.
* `renewals` is a built-in source mapped to membership renewal rows.
* `pending-renewal` applies only when source set includes `renewals` and means `renewals.status = 'pending'`.
* `selected-renewal` applies only when source set includes `renewals` and means `renewals.selected = true`.
* `primary-contact` applies only when source set includes `contacts` and means `contacts.is_contact_1` is truthy.
* FEN editorial, advertising, and invoice data must be exposed as built-in sources, not custom table references, using the source names below.
* These built-in sources are derived from the sibling `../feca2-app/server/src/lib/mailshotDsl.js` implementation, except that `articles` must now be promoted to a first-class built-in source. In the sibling implementation, `Articles` is used by an issue-scoped article filter but is not listed as a standalone built-in source.
* FEN built-in source table mappings:
  * `advertisers` maps to `fen.advertisers`.
  * `ads` maps to `fen.ads`.
  * `pages` maps to `fen.Pages`.
  * `articles` maps to `fen.Articles`.
  * `issues` maps to `fen.Issues`.
  * `invoices` maps to `fen.invoices`.
* Required FEN source fields and canonical token aliases:
  * `advertisers`: `name` / `advertisername` from `AdvertiserName`.
  * `ads`: `id` from `ID`, `advertiser` from `Advertiser`, `adsize` / `size` from `AdSize`, `price` from `Price`, `issue` from the related page `Issue`, `pageid` from `PageID`, `state` from `State`, `notes` from `Notes`.
  * `pages`: `id` from `ID`, `issue` from `Issue`, `page` from `Page`.
  * `articles`: `id` from `ID`, `pageid` from `PageID`, `articlenumber` / `article` from `ArticleNumber`, `content` from `Content`, `membername` from `MemberName`, `author` from `Author`, `dcn` from `DCN`, `articlewords` from `ArticleWords`, `otherwords` from `OtherWords`, `othercontent` / `other` from `OtherContent`.
  * `issues`: `id` / `issue` from `ID`, `issuemonths` from `IssueMonths`, `description` from `Description`.
  * `invoices`: `id`, `issue` / `issue_id`, `ad_id`, `invoice_number`, `invoice_date`, `due_date`, `invoice_page`, `invoice_size`, `invoice_price`, `status`, `payment_date`, `amount_paid`, `payment_method`, `payment_reference`, `notes`, `created_at`, `updated_at`.
* Required FEN physical columns:
  * `fen.Issues`: `ID`, `CopyDate`, `PublicationDate`, `IssueMonths`, `Description`.
  * `fen.Pages`: `ID`, `Issue`, `Page`, `PageSizeName`, `Content`.
  * `fen.Articles`: `ID`, `PageID`, `ArticleNumber`, `Content`, `MemberName`, `Author`, `DCN`, `ArticleWords`, `OtherWords`, `OtherContent`.
  * `fen.ads`: `ID`, `PageID`, `AdSize`, `Advertiser`, `Price`, `State`, `Notes`.
  * `fen.advertisers`: `Entry ID`, `AdvertiserName`, `title`, `contact_name`, `address_1`, `address_2`, `town`, `post_code`, `Description`, `IsLapsed?`, `Home Phone`, `Phone`, `Email`, `Selected`.
  * `fen.invoices`: `id`, `issue_id`, `ad_id`, `invoice_number`, `invoice_date`, `due_date`, `invoice_page`, `invoice_size`, `invoice_price`, `status`, `payment_date`, `amount_paid`, `payment_method`, `payment_reference`, `notes`, `created_at`, `updated_at`.
* `selected` applies only when source set includes `advertisers` and means `advertisers.Selected` is truthy.
* `page-in-issue(<issue>)` accepts exactly one numeric issue ID. It applies only when the source set includes `pages` or `articles`, and means the page row, or the page joined from the article row, belongs to that issue.
* `ad-in-issue(<issue>)` accepts exactly one numeric issue ID. It applies only when the source set includes one of `advertisers`, `ads`, `pages`, `issues`, or `invoices`, and means the row has an ad in that issue.
* `pending-invoice` applies only when source set includes `invoices` and means `invoices.status = 'pending'`.
* `selected-invoice` applies only when source set includes `invoices` and means the invoice ID is in the runtime selected invoice ID list.
* `invoice-ids(...)` applies only when source set includes `invoices`; it accepts one or more numeric invoice IDs.
* `member-or-affiliate-or-parish-council` applies to account data and means:
  * `accounts.type` is `Member` or `Affiliate`, or
  * `accounts.name` contains `Parish Council`.
* `field_ref` is restricted by whitelist per built-in source.
* `field_ref` is also supported for selected custom sources when table metadata is available.
* Field-to-field comparisons are supported with `=` and `!=` only.
* Set-membership against another field source is supported via:
  * `field_ref in (field_ref)`
  * `field_ref not in (field_ref)`
* For `in(field_ref)` / `not in(field_ref)`, the RHS source is treated as a reference source (subquery semantics), not a joined source.
* `contains`, `starts-with`, and `ends-with` require a literal right-hand side value (not a field reference).
* In v1.6, mixing built-in sources and custom sources in the same sentence is not supported; validation must fail with a clear error.
* For multi-custom-source sentences, all selected custom sources must be connected by explicit `=` field-to-field predicates (graph-connected join semantics), otherwise validation fails.
* `in(field_ref)` / `not in(field_ref)` must compile as subquery membership; the RHS source is a reference source and does not need to appear in `source_expr`.
* Canonical naming for built-in sources uses the token aliases listed in this document.
* Canonical naming for custom table references uses exact field names as defined in database metadata for the selected source. The implementation must reject invalid custom field names and must not add aliasing/normalization fallback.

### Join-path model (required)

The compiler must use an explicit join graph per source pair. Example v1 join paths:

* `contacts` -> `accounts`: `contacts.Accountid = accounts.ID`
* `accounts` -> `contacts`: `accounts.ID = contacts.Accountid`
* `renewals` -> `accounts`: `renewals.account_id = accounts.ID`
* `accounts` -> `renewals`: `accounts.ID = renewals.account_id`
* `renewals` -> `contacts`: `renewals.account_id = contacts.Accountid`
* `contacts` -> `renewals`: `contacts.Accountid = renewals.account_id`
* `articles` -> `pages`: `articles.PageID = pages.ID`
* `pages` -> `articles`: `pages.ID = articles.PageID`
* `pages` -> `issues`: `pages.Issue = issues.ID`
* `issues` -> `pages`: `issues.ID = pages.Issue`
* `ads` -> `pages`: `ads.PageID = pages.ID`
* `pages` -> `ads`: `pages.ID = ads.PageID`
* `ads` -> `advertisers`: normalized `ads.Advertiser = advertisers.AdvertiserName`
* `advertisers` -> `ads`: normalized `advertisers.AdvertiserName = ads.Advertiser`
* `invoices` -> `ads`: `invoices.ad_id = ads.ID`
* `ads` -> `invoices`: `ads.ID = invoices.ad_id`
* `invoices` -> `issues`: `invoices.issue_id = issues.ID`
* `issues` -> `invoices`: `issues.ID = invoices.issue_id`
* `invoices` -> `pages`: `invoices.issue_id = pages.Issue`
* `pages` -> `invoices`: `pages.Issue = invoices.issue_id`

If no approved join path exists between two sources for `and`, parsing/validation must fail with a clear error.

No implicit join behavior is allowed:

* Selecting a source such as `accounts` must not silently join `contacts` (or any other source).
* Output columns must come only from sources explicitly requested in the sentence.

## Example valid queries

* `contacts`
* `contacts and accounts where fen1-contact`
* `contacts and accounts where primary-contact`
* `accounts and contacts where contacts.Accountid = accounts.ID`
* `contacts where contacts.Last contains 'smith'`
* `contacts and accounts where accounts.Type = 'Member' and contacts.FENContact1 = true`
* `accounts where member-or-affiliate-or-parish-council`
* `renewals where pending-renewal`
* `renewals where selected-renewal`
* `renewals and accounts and contacts where pending-renewal`
* `advertisers where selected`
* `ads and pages where ad-in-issue(202605)`
* `articles and pages where page-in-issue(202605)`
* `invoices where pending-invoice`
* `invoices and ads and advertisers where invoice-ids(101, 102)`
* `accounts where accounts.Name not in (members.ExcludedAccounts.ExcludedAccount)`
* `members.mailshot_test`

## Example invalid queries

* `select * from members` (raw SQL not allowed)
* `contacts where drop table` (unknown tokens)
* `contacts where accounts.Name = 'x'` (invalid if `accounts` is not included in source expression)
* `contacts or accounts` (`or` is not supported; use `and` only)
* `accounts where pending-renewal` (invalid: filter requires `renewals` source)
* `accounts where fen1-contact` (invalid: filter requires `contacts` source)
* `accounts where primary-contact` (invalid: filter requires `contacts` source)
* `ads where selected` (invalid: filter requires `advertisers` source)
* `advertisers where pending-invoice` (invalid: filter requires `invoices` source)
* `pages where ad-in-issue` (invalid: `ad-in-issue` requires exactly one numeric issue ID)
* `ads where page-in-issue(202605)` (invalid: `page-in-issue` requires `pages` or `articles` source)
* `invoices where invoice-ids('abc')` (invalid: `invoice-ids` accepts numeric invoice IDs only)
* `renewals where member-or-affiliate-or-parish-council` (invalid: filter requires `accounts` source)
* `accounts and contacts where contacts.Accountid contains accounts.ID` (invalid: field-to-field supports only `=`/`!=`)
* `accounts where accounts.Name contains members.ExcludedAccounts.ExcludedAccount` (invalid: `contains` requires a literal RHS)

## Compilation and performance requirements

* Parse DSL to AST.
* Validate AST against allowed sources, fields, and operators.
* Validate all `and` source combinations against approved join graph metadata.
* Do not perform implicit source joins during compilation.
* Compile AST to parameterized SQL only (no string-concatenated SQL).
* Push filters into SQL `WHERE` / join conditions.
* Preview must not assume fixed contact/account projection columns.
* Compiler/runtime must enforce deterministic source-field availability from DSL-cited sources.
* Preview column derivation must include:
  * all fields from the DSL expected-field contract, and
  * union of keys from returned rows (not just first row keys), to avoid hiding sparse/source-native columns.
* For mailshot execution, email remains the only required field.
* Provide preview endpoints: count + sample rows before execution.

## Storage model

Store:

* `dsl_text` (editable source).
* `dsl_text` is the single source of truth for execution, preview, and validation.
* No SQL fallback path is permitted when `dsl_text` is missing/empty; this must be treated as an error.

Optional:

* normalized AST JSON for diagnostics/explain.
* `sql` retained as blank/deprecated for backward compatibility when the legacy column exists, and never executed.

## User interface requirements

Provide a UI to view, create, update, validate, and preview DSL sentences.

### Mailshot data source page

Add a dedicated page "Data Sources" under FECA Mailshots admin page for managing data source sentences.

Top regions:

* Context pane:
  * page title `Mailshot Data Sources`
* Information pane:
  * validation errors, save status, preview status
* Statistics pane:
  * total saved data sources
  * selected data source id/name

### Data source list (view)

Provide a table/list showing existing data sources with columns:

* `ID`
* `Name`
* `DSL sentence`
* `Updated at`

List behavior:

* single-row selection
* text filter by `Name` or sentence content
* sort by `Name` and `Updated at`

Selection to persist between page loads.

### Editor (create/update sentence)

Provide an editor panel with:

* `Name` input (required)
* multi-line `DSL sentence` input (required)
* read-only `Last validated status`

Actions:

* `New`:
  * clear editor and start create mode
* `Save`:
  * create or update current record
  * enabled only when form is valid and has unsaved changes
* `Discard changes`:
  * revert editor to last saved state
* `Delete`:
  * confirmation required
  * blocked when the data source is referenced by one or more mailshots

### Validation UX

Provide explicit validation before save and on-demand:

* `Validate sentence` button
* parse + semantic validation against:
  * grammar
  * source/field/operator whitelist
  * join-path rules

On validation result:

* show `Valid` / `Invalid`
* show `Warnings` separately from errors
* for invalid:
  * show user-friendly error message
  * include line/column when available
  * do not save invalid sentence
* for warning-only cases (for example missing email field):
  * allow save
  * allow preview
  * indicate the data source cannot be used for sending a mailshot

### Preview UX

Provide `Preview recipients` action for selected sentence:

* show recipient count
* show sample rows (for example first 50)
* derive displayed columns from the deterministic expected-field contract plus returned row shape
* do not assume fixed contact/account projection columns
* show execution time
* do not execute/send mailshot from this page

### DSL Sentence Builder (required)

Provide a `Build DSL` action in the data-source editor.

On click, open a modal/overlay builder that allows users to construct full valid DSL without memorising grammar keywords, filter names, or field names.

Builder layout:

* Top sub-pane: `Data Sources`
* Bottom sub-pane: `Constraints / Filters`
* Footer: generated DSL preview + actions

#### Top sub-pane: Data Sources

Provide source selection controls for:

* `contacts`
* `accounts`
* `renewals`
* custom table source (schema/table selector or validated text input)

Behavior:

* multiple source selection is combined using `and`
* order of selected sources is preserved in generated sentence
* selecting an invalid source combination is blocked in UI with clear message

#### Bottom sub-pane: Constraints / Filters

Provide controls to add one or more constraint rows.

Each row supports one of:

* predefined filter
* field comparison
* grouped expression with `not`

Predefined filter control:

* dropdown label uses user-friendly text, not raw DSL token
* examples:
  * `Renewal is selected` -> `selected-renewal`
  * `Renewal is pending` -> `pending-renewal`
  * `Advertiser is selected` -> `selected`
  * `Page is in issue` -> `page-in-issue(...)`
  * `Advertiser has ad in issue` -> `ad-in-issue(...)`
  * `Invoice is pending` -> `pending-invoice`
  * `Invoice is selected` -> `selected-invoice`
  * `Invoice ID is one of` -> `invoice-ids(...)`
  * `Contact is FEN1` -> `fen1-contact`
  * `Contact is primary` -> `primary-contact`
  * `Account is member/affiliate/parish council` -> `member-or-affiliate-or-parish-council`

Field comparison row:

* source dropdown (restricted to selected sources)
* field dropdown (restricted to fields allowed for selected source)
* operator dropdown (`=`, `!=`, `contains`, `starts-with`, `ends-with`, `in`)
* RHS mode dropdown:
  * literal value input (typed by expected value type), or
  * source+field selector for field-to-field comparisons and set-membership (`in`)
* for `in`, provide multi-value token input

Logical composition:

* all rows combine with `and` in v1.6
* each row can be negated via checkbox (`not`)
* optional group rows allow nested bracketed expressions

#### Generated DSL + Sync

Footer must show:

* read-only generated DSL sentence (live update)
* validation status
* parse/semantic errors with line/column mapped to relevant builder controls

Actions:

* `Apply` writes generated DSL to the editor text area
* `Cancel` closes overlay with no change
* `Reset` clears builder selections

Sync rules:

* opening builder from existing DSL pre-populates controls when sentence is parseable
* if existing DSL is not parseable, show message and allow user to start from clean builder
* direct text edits remain supported; builder is not the only editing path

#### Safety and usability requirements

* UI must prevent creating syntactically invalid DSL states
* UI must prevent selecting filters incompatible with current source set
* custom table sources must be validated before Apply
* generated DSL must always conform to strict grammar in this spec

### API expectations for UI

The UI expects backend endpoints for:

* list data sources
* get one data source
* validate sentence
* create data source
* update data source
* delete data source
* preview sentence (count + sample)
* list schemas for custom source selection
* list tables for selected schema
* list fields for selected source (built-in or custom)

WordPress REST route contract:

* `GET /wp-json/mailshots/v1/data-sources`
* `GET /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/validate`
* `POST /wp-json/mailshots/v1/data-sources`
* `PUT /wp-json/mailshots/v1/data-sources/{id}`
* `DELETE /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/preview`
* `POST /wp-json/mailshots/v1/data-sources/{id}/preview`
* `GET /wp-json/mailshots/v1/data-sources/schemas`
* `GET /wp-json/mailshots/v1/data-sources/tables?schema=...`
* `GET /wp-json/mailshots/v1/data-sources/source-fields?source=...`

Auth/capability requirements:

* all routes require authenticated WordPress users;
* in this plugin, editor-capable operational access is the baseline:
  * create/update/delete/read/preview/validate all require WordPress `edit_pages` (editor or administrator);
* plugin Setup/settings remains administrator-only under WordPress `manage_options` (defined in top-level policy).

All endpoints must enforce server-side validation even if client validates first.

## Additional functional requirements (in scope)

* Mailshot Data Source preview, Mailshot Recipients Preview, and Mailshot Test preview must all display source-faithful columns using the same result-shape rule.
* DSL Builder must support custom sources via schema + table selection (not text-only entry), with manual text fallback only when metadata APIs are unavailable.
* After a custom source is added in Builder, it must be immediately available in:
  * source dropdowns,
  * LHS field dropdowns,
  * RHS field dropdowns (for field comparisons).
* Builder must support `field not in (other_source.other_field)` generation using RHS source + field controls (not value-type literal controls).
* Builder prefill must parse and restore field references where source names are qualified (for example `members.ExcludedAccounts.ExcludedAccount`).
* Validation errors must be explicit for unsupported/ineffective predicates; no silent no-op predicate behavior.

## Implementation details

This section is normative guidance for regenerating the feature from scratch.

### 1. Data model

* Table: `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`
* Required columns:
  * `ID` (PK)
  * `name`
  * `dsl_text`
  * `updated_at` (recommended)
* `dsl_text` is the only executable representation.
* Persist `sql` as blank/deprecated if column still exists for backward compatibility; do not execute it.

### 2. Backend architecture

Build a strict pipeline:

1. Tokenize DSL text.
2. Parse to AST.
3. Validate AST (grammar + semantics).
4. Compile AST to parameterized SQL and parameters.
5. Execute SQL for preview/send paths.

Keep parser/validator/compiler in a dedicated module (for example `mailshotDsl.js`), and route handlers thin.

### 3. Parser/AST contract

* AST should separate `source_expr` and `where`.
* Predicates should distinguish:
  * named filters,
  * compare literal,
  * compare field-to-field,
  * `in(value_list)`,
  * `in(field_ref)` / `not in(field_ref)`.
* Preserve token line/column for diagnostics.

### 4. Validation rules (server-side)

* Validate source compatibility via explicit join-path metadata.
* Validate filter applicability against selected sources.
* Validate field refs against per-source field whitelist:
  * built-ins: static map
  * custom: metadata-provider abstraction (not direct SQL dependency in validator logic)
* Enforce:
  * field-to-field only `=` / `!=`
  * literal RHS only for `contains` / `starts-with` / `ends-with`
* Return warnings (not errors) for no-email-source sentences.
* Metadata-provider minimum DB grants must be documented and kept minimal for the read-only DB user.

### 5. SQL compilation strategy

Use parameterized SQL only.

* Built-in-only source expressions:
  * compile from fixed source metadata (table + alias + base where).
  * compile approved joins only.
* Custom-only source expressions:
  * single source: `SELECT * FROM schema.table WHERE ...`
  * multi source: require explicit `=` field join connectivity and compile with aliased joins.
* `in(field_ref)` / `not in(field_ref)`:
  * compile as `EXISTS` / `NOT EXISTS` subqueries.

### 6. Preview/result-shape rules

* Normalize row values to JSON-safe scalars.
* Enforce expected-field contract from DSL-cited sources.
* Keep source-native returned columns available.
* It is allowed to expose normalized source-field aliases required to satisfy the expected-field contract.
* Do not add unrelated synthetic recipient projection columns.
* Derive display columns from expected-field contract plus union of keys in returned rows.
* Prune-all-empty columns only if explicitly configured and documented; default should preserve source-faithful shape.

### 7. API surface

Implement at minimum:

* `GET /wp-json/mailshots/v1/data-sources`
* `GET /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/validate`
* `POST /wp-json/mailshots/v1/data-sources`
* `PUT /wp-json/mailshots/v1/data-sources/{id}`
* `DELETE /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/preview`
* `POST /wp-json/mailshots/v1/data-sources/{id}/preview`
* `GET /wp-json/mailshots/v1/data-sources/schemas`
* `GET /wp-json/mailshots/v1/data-sources/tables?schema=...`
* `GET /wp-json/mailshots/v1/data-sources/source-fields?source=...`

All endpoints must validate server-side and return structured errors:
`{ ok:false, error, line?, column? }`.

### 8. UI generation guidance

Mailshot Data Source page:

* left: data source list with select/filter/sort.
* right: editor + validation + preview.
* include `Build DSL` modal.

Builder modal:

* source selection section (built-in + schema/table custom add).
* constraints section with row editor (filter/comparison).
* responsive row layout that wraps controls on narrow widths.
* RHS field mode must expose source + field selectors.
* show generated DSL live; validate with debounce.
* `Apply` writes DSL text; `Cancel` leaves editor unchanged.

### 9. Safety and regression test guidance

Minimum automated tests:

* parser positive/negative cases (line/column assertions).
* semantic validation failures (join path, filter applicability, field whitelist).
* compile output shape for:
  * built-in joins,
  * custom joins,
  * `in(field_ref)` and `not in(field_ref)`.
* preview shape:
  * includes all expected fields from cited sources, and
  * does not introduce unrelated synthetic recipient projection columns.
* builder prefill for qualified source field refs.
* no-op predicate prevention (must error, not silently ignore).