adrians
/
feca-mailshots-plugin
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
feca-mailshots-plugin/requirements/mailshot_data_source.md

565 lines
22 KiB
Markdown
Raw Blame History

# Mailshot Data Source page
## Purpose
Document requirements for a safe, non-admin-friendly mailshot data source definition model.
## Environment baseline for this specification
`members` as a database name means the value of `MEMBERS_REMOTE_MYSQL_DB`.
For this specification:
* Data-source definition persistence (create/update/delete/list of `mailshot_queries`) uses `MAILSHOTS_REMOTE_MYSQL_DB`.
* Source-data query execution (built-in/custom recipient sources) uses `MEMBERS_REMOTE_MYSQL_DB`.
## Problem with prior approach
Creating a new data source currently requires:
* knowledge of MySQL schema details, or
* admin-level access to create DB views/queries.
This blocks non-expert and non-admin users.
## Direction
Use a strict, constrained query language (DSL) for mailshot audience selection.
The DSL text is stored as the data-source definition in `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`, then parsed and compiled into safe SQL executed against approved source-data tables in `MEMBERS_REMOTE_MYSQL_DB`.
## Goals
* Allow non-admin users to define useful audiences.
* Prevent arbitrary SQL.
* Keep execution efficient (push filtering to SQL, avoid large unnecessary joins).
* Make queries explainable and previewable.
## Example requirements (in natural language)
* All contacts with name like "smith".
* All contacts joined to member accounts where contact FEN1 is true.
* All renewal rows marked as selected.
* All pending renewals.
* All accounts where name is not in an exclusion table.
## Strict grammar (v1.5)
Use lowercase keywords only.
### EBNF
```ebnf
query = source_expr , [ where_clause ] ;
source_expr = source_term , { source_op , source_term } ;
source_term = source_ref | "(" , source_expr , ")" ;
source_op = "and" ;
source_ref = built_in_source | table_ref ;
built_in_source = "contacts" | "accounts" | "renewals" ;
table_ref = ident_part , "." , ident_part ;
ident_part = identifier | quoted_identifier ;
where_clause = "where" , predicate_expr ;
predicate_expr = predicate_term , { bool_op , predicate_term } ;
predicate_term = [ "not" ] , predicate_atom
| "(" , predicate_expr , ")" ;
bool_op = "and" ;
predicate_atom = filter_expr
| field_ref , compare_op , value
| field_ref , ("=" | "!=") , field_ref
| field_ref , [ "not" ] , "in" , "(" , field_ref , ")"
| field_ref , "in" , "(" , value_list , ")" ;
filter_expr = filter_name
| filter_name , "(" , value_list , ")" ;
filter_name = "selected-renewal"
| "pending-renewal"
| "fen1-contact"
| "fen2-contact"
| "member-or-affiliate-or-parish-council" ;
field_ref = source_ref , "." , field_name ;
field_name = identifier | quoted_identifier ;
compare_op = "=" | "!=" | "contains" | "starts-with" | "ends-with" ;
value_list = value , { "," , value } ;
value = quoted_string | number | "true" | "false" ;
identifier = letter , { letter | digit | "_" } ;
quoted_identifier = "`" , { character - "`" } , "`" ;
quoted_string = "'" , { character - "'" } , "'" ;
number = digit , { digit } ;
letter = "a"…"z" | "A"…"Z" ;
digit = "0"…"9" ;
```
## Semantics (v1.5)
* `built_in_source` maps to a predefined table alias.
* `table_ref` maps directly to a table in the remote members database (`schema.table`).
* For `table_ref`, validation must confirm the table exists.
* Query result shape must be source-faithful and deterministic per DSL:
* each cited source contributes its full allowed field set to an expected-field contract;
* preview and template-token resolution must preserve availability of every field in that contract, even when raw SQL row keys are sparse;
* source-native columns from query output may also be present;
* no unrelated synthetic recipient projection columns are allowed.
* If the resolved source does not expose an email field, validation must return a warning:
* `Data Source does not have an email field, it cannot be used for a Mailshot`
* The sentence is still valid and can be saved and previewed.
* `and` between sources means relational intersection using configured join paths (not email-only matching).
* For custom sources, an explicit field equality predicate can provide join semantics.
* `where` applies after source composition.
* `not` negates only the next predicate/group.
* Predefined predicates are `selected-renewal`, `pending-renewal`, `fen1-contact`, `fen2-contact`, and `member-or-affiliate-or-parish-council`.
* `renewals` is a built-in source mapped to membership renewal rows.
* `pending-renewal` applies only when source set includes `renewals` and means `renewals.status = 'pending'`.
* `selected-renewal` applies only when source set includes `renewals` and means `renewals.selected = true`.
* `member-or-affiliate-or-parish-council` applies to account data and means:
* `accounts.type` is `Member` or `Affiliate`, or
* `accounts.name` contains `Parish Council`.
* `field_ref` is restricted by whitelist per built-in source.
* `field_ref` is also supported for selected custom sources when table metadata is available.
* Field-to-field comparisons are supported with `=` and `!=` only.
* Set-membership against another field source is supported via:
* `field_ref in (field_ref)`
* `field_ref not in (field_ref)`
* For `in(field_ref)` / `not in(field_ref)`, the RHS source is treated as a reference source (subquery semantics), not a joined source.
* `contains`, `starts-with`, and `ends-with` require a literal right-hand side value (not a field reference).
* In v1.5, mixing built-in sources and custom sources in the same sentence is not supported; validation must fail with a clear error.
* For multi-custom-source sentences, all selected custom sources must be connected by explicit `=` field-to-field predicates (graph-connected join semantics), otherwise validation fails.
* `in(field_ref)` / `not in(field_ref)` must compile as subquery membership; the RHS source is a reference source and does not need to appear in `source_expr`.
* Canonical naming in this document uses exact field names as defined in database metadata for the selected source. The implementation must reject invalid names and must not add aliasing/normalization fallback.
### Join-path model (required)
The compiler must use an explicit join graph per source pair. Example v1 join paths:
* `contacts` -> `accounts`: `contacts.Accountid = accounts.ID`
* `accounts` -> `contacts`: `accounts.ID = contacts.Accountid`
* `renewals` -> `accounts`: `renewals.account_id = accounts.ID`
* `accounts` -> `renewals`: `accounts.ID = renewals.account_id`
* `renewals` -> `contacts`: `renewals.account_id = contacts.Accountid`
* `contacts` -> `renewals`: `contacts.Accountid = renewals.account_id`
If no approved join path exists between two sources for `and`, parsing/validation must fail with a clear error.
No implicit join behavior is allowed:
* Selecting a source such as `accounts` must not silently join `contacts` (or any other source).
* Output columns must come only from sources explicitly requested in the sentence.
## Example valid queries
* `contacts`
* `contacts and accounts where fen1-contact`
* `contacts and accounts where fen2-contact`
* `accounts and contacts where contacts.Accountid = accounts.ID`
* `contacts where contacts.Last contains 'smith'`
* `contacts and accounts where accounts.Type = 'Member' and contacts.FENContact1 = true`
* `accounts where member-or-affiliate-or-parish-council`
* `renewals where pending-renewal`
* `renewals where selected-renewal`
* `renewals and accounts and contacts where pending-renewal`
* `accounts where accounts.Name not in (members.ExcludedAccounts.ExcludedAccount)`
* `members.mailshot_test`
## Example invalid queries
* `select * from members` (raw SQL not allowed)
* `contacts where drop table` (unknown tokens)
* `contacts where accounts.Name = 'x'` (invalid if `accounts` is not included in source expression)
* `contacts or accounts` (`or` is not supported; use `and` only)
* `accounts where pending-renewal` (invalid: filter requires `renewals` source)
* `accounts where fen1-contact` (invalid: filter requires `contacts` source)
* `renewals where member-or-affiliate-or-parish-council` (invalid: filter requires `accounts` source)
* `accounts and contacts where contacts.Accountid contains accounts.ID` (invalid: field-to-field supports only `=`/`!=`)
* `accounts where accounts.Name contains members.ExcludedAccounts.ExcludedAccount` (invalid: `contains` requires a literal RHS)
## Compilation and performance requirements
* Parse DSL to AST.
* Validate AST against allowed sources, fields, and operators.
* Validate all `and` source combinations against approved join graph metadata.
* Do not perform implicit source joins during compilation.
* Compile AST to parameterized SQL only (no string-concatenated SQL).
* Push filters into SQL `WHERE` / join conditions.
* Preview must not assume fixed contact/account projection columns.
* Compiler/runtime must enforce deterministic source-field availability from DSL-cited sources.
* Preview column derivation must include:
* all fields from the DSL expected-field contract, and
* union of keys from returned rows (not just first row keys), to avoid hiding sparse/source-native columns.
* For mailshot execution, email remains the only required field.
* Provide preview endpoints: count + sample rows before execution.
## Storage model
Store:
* `dsl_text` (editable source).
* `dsl_text` is the single source of truth for execution, preview, and validation.
* No SQL fallback path is permitted when `dsl_text` is missing/empty; this must be treated as an error.
Optional:
* normalized AST JSON for diagnostics/explain.
* `sql` retained as blank/deprecated for backward compatibility when the legacy column exists, and never executed.
## User interface requirements
Provide a UI to view, create, update, validate, and preview DSL sentences.
### Mailshot data source page
Add a dedicated page "Data Sources" under FECA Mailshots admin page for managing data source sentences.
Top regions:
* Context pane:
* page title `Mailshot Data Sources`
* Information pane:
* validation errors, save status, preview status
* Statistics pane:
* total saved data sources
* selected data source id/name
### Data source list (view)
Provide a table/list showing existing data sources with columns:
* `ID`
* `Name`
* `DSL sentence`
* `Updated at`
List behavior:
* single-row selection
* text filter by `Name` or sentence content
* sort by `Name` and `Updated at`
Selection to persist between page loads.
### Editor (create/update sentence)
Provide an editor panel with:
* `Name` input (required)
* multi-line `DSL sentence` input (required)
* read-only `Last validated status`
Actions:
* `New`:
* clear editor and start create mode
* `Save`:
* create or update current record
* enabled only when form is valid and has unsaved changes
* `Discard changes`:
* revert editor to last saved state
* `Delete`:
* confirmation required
* blocked when the data source is referenced by one or more mailshots
### Validation UX
Provide explicit validation before save and on-demand:
* `Validate sentence` button
* parse + semantic validation against:
* grammar
* source/field/operator whitelist
* join-path rules
On validation result:
* show `Valid` / `Invalid`
* show `Warnings` separately from errors
* for invalid:
* show user-friendly error message
* include line/column when available
* do not save invalid sentence
* for warning-only cases (for example missing email field):
* allow save
* allow preview
* indicate the data source cannot be used for sending a mailshot
### Preview UX
Provide `Preview recipients` action for selected sentence:
* show recipient count
* show sample rows (for example first 50)
* derive displayed columns from the deterministic expected-field contract plus returned row shape
* do not assume fixed contact/account projection columns
* show execution time
* do not execute/send mailshot from this page
### DSL Sentence Builder (required)
Provide a `Build DSL` action in the data-source editor.
On click, open a modal/overlay builder that allows users to construct full valid DSL without memorising grammar keywords, filter names, or field names.
Builder layout:
* Top sub-pane: `Data Sources`
* Bottom sub-pane: `Constraints / Filters`
* Footer: generated DSL preview + actions
#### Top sub-pane: Data Sources
Provide source selection controls for:
* `contacts`
* `accounts`
* `renewals`
* custom table source (schema/table selector or validated text input)
Behavior:
* multiple source selection is combined using `and`
* order of selected sources is preserved in generated sentence
* selecting an invalid source combination is blocked in UI with clear message
#### Bottom sub-pane: Constraints / Filters
Provide controls to add one or more constraint rows.
Each row supports one of:
* predefined filter
* field comparison
* grouped expression with `not`
Predefined filter control:
* dropdown label uses user-friendly text, not raw DSL token
* examples:
* `Renewal is selected` -> `selected-renewal`
* `Renewal is pending` -> `pending-renewal`
* `Contact is FEN1` -> `fen1-contact`
* `Contact is FEN2` -> `fen2-contact`
* `Account is member/affiliate/parish council` -> `member-or-affiliate-or-parish-council`
Field comparison row:
* source dropdown (restricted to selected sources)
* field dropdown (restricted to fields allowed for selected source)
* operator dropdown (`=`, `!=`, `contains`, `starts-with`, `ends-with`, `in`)
* RHS mode dropdown:
* literal value input (typed by expected value type), or
* source+field selector for field-to-field comparisons and set-membership (`in`)
* for `in`, provide multi-value token input
Logical composition:
* all rows combine with `and` in v1.5
* each row can be negated via checkbox (`not`)
* optional group rows allow nested bracketed expressions
#### Generated DSL + Sync
Footer must show:
* read-only generated DSL sentence (live update)
* validation status
* parse/semantic errors with line/column mapped to relevant builder controls
Actions:
* `Apply` writes generated DSL to the editor text area
* `Cancel` closes overlay with no change
* `Reset` clears builder selections
Sync rules:
* opening builder from existing DSL pre-populates controls when sentence is parseable
* if existing DSL is not parseable, show message and allow user to start from clean builder
* direct text edits remain supported; builder is not the only editing path
#### Safety and usability requirements
* UI must prevent creating syntactically invalid DSL states
* UI must prevent selecting filters incompatible with current source set
* custom table sources must be validated before Apply
* generated DSL must always conform to strict grammar in this spec
### API expectations for UI
The UI expects backend endpoints for:
* list data sources
* get one data source
* validate sentence
* create data source
* update data source
* delete data source
* preview sentence (count + sample)
* list schemas for custom source selection
* list tables for selected schema
* list fields for selected source (built-in or custom)
WordPress REST route contract:
* `GET /wp-json/mailshots/v1/data-sources`
* `GET /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/validate`
* `POST /wp-json/mailshots/v1/data-sources`
* `PUT /wp-json/mailshots/v1/data-sources/{id}`
* `DELETE /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/preview`
* `POST /wp-json/mailshots/v1/data-sources/{id}/preview`
* `GET /wp-json/mailshots/v1/data-sources/schemas`
* `GET /wp-json/mailshots/v1/data-sources/tables?schema=...`
* `GET /wp-json/mailshots/v1/data-sources/source-fields?source=...`
Auth/capability requirements:
* all routes require authenticated WordPress users;
* create/update/delete routes require a mailshot management capability (for example `manage_mailshots`);
* read/preview/validate routes require a mailshot read capability (for example `read_mailshots`).
All endpoints must enforce server-side validation even if client validates first.
## Additional functional requirements (in scope)
* Mailshot Data Source preview, Mailshot Recipients Preview, and Mailshot Test preview must all display source-faithful columns using the same result-shape rule.
* DSL Builder must support custom sources via schema + table selection (not text-only entry), with manual text fallback only when metadata APIs are unavailable.
* After a custom source is added in Builder, it must be immediately available in:
* source dropdowns,
* LHS field dropdowns,
* RHS field dropdowns (for field comparisons).
* Builder must support `field not in (other_source.other_field)` generation using RHS source + field controls (not value-type literal controls).
* Builder prefill must parse and restore field references where source names are qualified (for example `members.ExcludedAccounts.ExcludedAccount`).
* Validation errors must be explicit for unsupported/ineffective predicates; no silent no-op predicate behavior.
## Implementation details
This section is normative guidance for regenerating the feature from scratch.
### 1. Data model
* Table: `MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries`
* Required columns:
* `ID` (PK)
* `name`
* `dsl_text`
* `updated_at` (recommended)
* `dsl_text` is the only executable representation.
* Persist `sql` as blank/deprecated if column still exists for backward compatibility; do not execute it.
### 2. Backend architecture
Build a strict pipeline:
1. Tokenize DSL text.
2. Parse to AST.
3. Validate AST (grammar + semantics).
4. Compile AST to parameterized SQL and parameters.
5. Execute SQL for preview/send paths.
Keep parser/validator/compiler in a dedicated module (for example `mailshotDsl.js`), and route handlers thin.
### 3. Parser/AST contract
* AST should separate `source_expr` and `where`.
* Predicates should distinguish:
* named filters,
* compare literal,
* compare field-to-field,
* `in(value_list)`,
* `in(field_ref)` / `not in(field_ref)`.
* Preserve token line/column for diagnostics.
### 4. Validation rules (server-side)
* Validate source compatibility via explicit join-path metadata.
* Validate filter applicability against selected sources.
* Validate field refs against per-source field whitelist:
* built-ins: static map
* custom: metadata-provider abstraction (not direct SQL dependency in validator logic)
* Enforce:
* field-to-field only `=` / `!=`
* literal RHS only for `contains` / `starts-with` / `ends-with`
* Return warnings (not errors) for no-email-source sentences.
* Metadata-provider minimum DB grants must be documented and kept minimal for the read-only DB user.
### 5. SQL compilation strategy
Use parameterized SQL only.
* Built-in-only source expressions:
* compile from fixed source metadata (table + alias + base where).
* compile approved joins only.
* Custom-only source expressions:
* single source: `SELECT * FROM schema.table WHERE ...`
* multi source: require explicit `=` field join connectivity and compile with aliased joins.
* `in(field_ref)` / `not in(field_ref)`:
* compile as `EXISTS` / `NOT EXISTS` subqueries.
### 6. Preview/result-shape rules
* Normalize row values to JSON-safe scalars.
* Enforce expected-field contract from DSL-cited sources.
* Keep source-native returned columns available.
* It is allowed to expose normalized source-field aliases required to satisfy the expected-field contract.
* Do not add unrelated synthetic recipient projection columns.
* Derive display columns from expected-field contract plus union of keys in returned rows.
* Prune-all-empty columns only if explicitly configured and documented; default should preserve source-faithful shape.
### 7. API surface
Implement at minimum:
* `GET /wp-json/mailshots/v1/data-sources`
* `GET /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/validate`
* `POST /wp-json/mailshots/v1/data-sources`
* `PUT /wp-json/mailshots/v1/data-sources/{id}`
* `DELETE /wp-json/mailshots/v1/data-sources/{id}`
* `POST /wp-json/mailshots/v1/data-sources/preview`
* `POST /wp-json/mailshots/v1/data-sources/{id}/preview`
* `GET /wp-json/mailshots/v1/data-sources/schemas`
* `GET /wp-json/mailshots/v1/data-sources/tables?schema=...`
* `GET /wp-json/mailshots/v1/data-sources/source-fields?source=...`
All endpoints must validate server-side and return structured errors:
`{ ok:false, error, line?, column? }`.
### 8. UI generation guidance
Mailshot Data Source page:
* left: data source list with select/filter/sort.
* right: editor + validation + preview.
* include `Build DSL` modal.
Builder modal:
* source selection section (built-in + schema/table custom add).
* constraints section with row editor (filter/comparison).
* responsive row layout that wraps controls on narrow widths.
* RHS field mode must expose source + field selectors.
* show generated DSL live; validate with debounce.
* `Apply` writes DSL text; `Cancel` leaves editor unchanged.
### 9. Safety and regression test guidance
Minimum automated tests:
* parser positive/negative cases (line/column assertions).
* semantic validation failures (join path, filter applicability, field whitelist).
* compile output shape for:
* built-in joins,
* custom joins,
* `in(field_ref)` and `not in(field_ref)`.
* preview shape:
* includes all expected fields from cited sources, and
* does not introduce unrelated synthetic recipient projection columns.
* builder prefill for qualified source field refs.
* no-op predicate prevention (must error, not silently ignore).
Reference in New Issue View Git Blame Copy Permalink