adrians
/
feca-mailshots-plugin
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
feca-mailshots-plugin/requirements/mailshot_data_source.md

22 KiB
Raw Blame History

Mailshot Data Source page

Purpose

Document requirements for a safe, non-admin-friendly mailshot data source definition model.

Environment baseline for this specification

members as a database name means the value of MEMBERS_REMOTE_MYSQL_DB.

For this specification:

  • Data-source definition persistence (create/update/delete/list of mailshot_queries) uses MAILSHOTS_REMOTE_MYSQL_DB.
  • Source-data query execution (built-in/custom recipient sources) uses MEMBERS_REMOTE_MYSQL_DB.

Problem with prior approach

Creating a new data source currently requires:

  • knowledge of MySQL schema details, or
  • admin-level access to create DB views/queries.

This blocks non-expert and non-admin users.

Direction

Use a strict, constrained query language (DSL) for mailshot audience selection.

The DSL text is stored as the data-source definition in MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries, then parsed and compiled into safe SQL executed against approved source-data tables in MEMBERS_REMOTE_MYSQL_DB.

Goals

  • Allow non-admin users to define useful audiences.
  • Prevent arbitrary SQL.
  • Keep execution efficient (push filtering to SQL, avoid large unnecessary joins).
  • Make queries explainable and previewable.

Example requirements (in natural language)

  • All contacts with name like "smith".
  • All contacts joined to member accounts where contact FEN1 is true.
  • All renewal rows marked as selected.
  • All pending renewals.
  • All accounts where name is not in an exclusion table.

Strict grammar (v1.5)

Use lowercase keywords only.

EBNF

query              = source_expr , [ where_clause ] ;

source_expr        = source_term , { source_op , source_term } ;
source_term        = source_ref | "(" , source_expr , ")" ;
source_op          = "and" ;

source_ref         = built_in_source | table_ref ;
built_in_source    = "contacts" | "accounts" | "renewals" ;
table_ref          = ident_part , "." , ident_part ;
ident_part         = identifier | quoted_identifier ;

where_clause       = "where" , predicate_expr ;
predicate_expr     = predicate_term , { bool_op , predicate_term } ;
predicate_term     = [ "not" ] , predicate_atom
                   | "(" , predicate_expr , ")" ;
bool_op            = "and" ;

predicate_atom     = filter_expr
                   | field_ref , compare_op , value
                   | field_ref , ("=" | "!=") , field_ref
                   | field_ref , [ "not" ] , "in" , "(" , field_ref , ")"
                   | field_ref , "in" , "(" , value_list , ")" ;

filter_expr        = filter_name
                   | filter_name , "(" , value_list , ")" ;

filter_name        = "selected-renewal"
                   | "pending-renewal"
                   | "fen1-contact"
                   | "fen2-contact"
                   | "member-or-affiliate-or-parish-council" ;

field_ref          = source_ref , "." , field_name ;
field_name         = identifier | quoted_identifier ;
compare_op         = "=" | "!=" | "contains" | "starts-with" | "ends-with" ;

value_list         = value , { "," , value } ;
value              = quoted_string | number | "true" | "false" ;

identifier         = letter , { letter | digit | "_" } ;
quoted_identifier  = "`" , { character - "`" } , "`" ;
quoted_string      = "'" , { character - "'" } , "'" ;
number             = digit , { digit } ;
letter             = "a"…"z" | "A"…"Z" ;
digit              = "0"…"9" ;

Semantics (v1.5)

  • built_in_source maps to a predefined table alias.
  • table_ref maps directly to a table in the remote members database (schema.table).
  • For table_ref, validation must confirm the table exists.
  • Query result shape must be source-faithful and deterministic per DSL:
    • each cited source contributes its full allowed field set to an expected-field contract;
    • preview and template-token resolution must preserve availability of every field in that contract, even when raw SQL row keys are sparse;
    • source-native columns from query output may also be present;
    • no unrelated synthetic recipient projection columns are allowed.
  • If the resolved source does not expose an email field, validation must return a warning:
    • Data Source does not have an email field, it cannot be used for a Mailshot
    • The sentence is still valid and can be saved and previewed.
  • and between sources means relational intersection using configured join paths (not email-only matching).
  • For custom sources, an explicit field equality predicate can provide join semantics.
  • where applies after source composition.
  • not negates only the next predicate/group.
  • Predefined predicates are selected-renewal, pending-renewal, fen1-contact, fen2-contact, and member-or-affiliate-or-parish-council.
  • renewals is a built-in source mapped to membership renewal rows.
  • pending-renewal applies only when source set includes renewals and means renewals.status = 'pending'.
  • selected-renewal applies only when source set includes renewals and means renewals.selected = true.
  • member-or-affiliate-or-parish-council applies to account data and means:
    • accounts.type is Member or Affiliate, or
    • accounts.name contains Parish Council.
  • field_ref is restricted by whitelist per built-in source.
  • field_ref is also supported for selected custom sources when table metadata is available.
  • Field-to-field comparisons are supported with = and != only.
  • Set-membership against another field source is supported via:
    • field_ref in (field_ref)
    • field_ref not in (field_ref)
  • For in(field_ref) / not in(field_ref), the RHS source is treated as a reference source (subquery semantics), not a joined source.
  • contains, starts-with, and ends-with require a literal right-hand side value (not a field reference).
  • In v1.5, mixing built-in sources and custom sources in the same sentence is not supported; validation must fail with a clear error.
  • For multi-custom-source sentences, all selected custom sources must be connected by explicit = field-to-field predicates (graph-connected join semantics), otherwise validation fails.
  • in(field_ref) / not in(field_ref) must compile as subquery membership; the RHS source is a reference source and does not need to appear in source_expr.
  • Canonical naming in this document uses exact field names as defined in database metadata for the selected source. The implementation must reject invalid names and must not add aliasing/normalization fallback.

Join-path model (required)

The compiler must use an explicit join graph per source pair. Example v1 join paths:

  • contacts -> accounts: contacts.Accountid = accounts.ID
  • accounts -> contacts: accounts.ID = contacts.Accountid
  • renewals -> accounts: renewals.account_id = accounts.ID
  • accounts -> renewals: accounts.ID = renewals.account_id
  • renewals -> contacts: renewals.account_id = contacts.Accountid
  • contacts -> renewals: contacts.Accountid = renewals.account_id

If no approved join path exists between two sources for and, parsing/validation must fail with a clear error.

No implicit join behavior is allowed:

  • Selecting a source such as accounts must not silently join contacts (or any other source).
  • Output columns must come only from sources explicitly requested in the sentence.

Example valid queries

  • contacts
  • contacts and accounts where fen1-contact
  • contacts and accounts where fen2-contact
  • accounts and contacts where contacts.Accountid = accounts.ID
  • contacts where contacts.Last contains 'smith'
  • contacts and accounts where accounts.Type = 'Member' and contacts.FENContact1 = true
  • accounts where member-or-affiliate-or-parish-council
  • renewals where pending-renewal
  • renewals where selected-renewal
  • renewals and accounts and contacts where pending-renewal
  • accounts where accounts.Name not in (members.ExcludedAccounts.ExcludedAccount)
  • members.mailshot_test

Example invalid queries

  • select * from members (raw SQL not allowed)
  • contacts where drop table (unknown tokens)
  • contacts where accounts.Name = 'x' (invalid if accounts is not included in source expression)
  • contacts or accounts (or is not supported; use and only)
  • accounts where pending-renewal (invalid: filter requires renewals source)
  • accounts where fen1-contact (invalid: filter requires contacts source)
  • renewals where member-or-affiliate-or-parish-council (invalid: filter requires accounts source)
  • accounts and contacts where contacts.Accountid contains accounts.ID (invalid: field-to-field supports only =/!=)
  • accounts where accounts.Name contains members.ExcludedAccounts.ExcludedAccount (invalid: contains requires a literal RHS)

Compilation and performance requirements

  • Parse DSL to AST.
  • Validate AST against allowed sources, fields, and operators.
  • Validate all and source combinations against approved join graph metadata.
  • Do not perform implicit source joins during compilation.
  • Compile AST to parameterized SQL only (no string-concatenated SQL).
  • Push filters into SQL WHERE / join conditions.
  • Preview must not assume fixed contact/account projection columns.
  • Compiler/runtime must enforce deterministic source-field availability from DSL-cited sources.
  • Preview column derivation must include:
    • all fields from the DSL expected-field contract, and
    • union of keys from returned rows (not just first row keys), to avoid hiding sparse/source-native columns.
  • For mailshot execution, email remains the only required field.
  • Provide preview endpoints: count + sample rows before execution.

Storage model

Store:

  • dsl_text (editable source).
  • dsl_text is the single source of truth for execution, preview, and validation.
  • No SQL fallback path is permitted when dsl_text is missing/empty; this must be treated as an error.

Optional:

  • normalized AST JSON for diagnostics/explain.
  • sql retained as blank/deprecated for backward compatibility when the legacy column exists, and never executed.

User interface requirements

Provide a UI to view, create, update, validate, and preview DSL sentences.

Mailshot data source page

Add a dedicated page "Data Sources" under FECA Mailshots admin page for managing data source sentences.

Top regions:

  • Context pane:
    • page title Mailshot Data Sources
  • Information pane:
    • validation errors, save status, preview status
  • Statistics pane:
    • total saved data sources
    • selected data source id/name

Data source list (view)

Provide a table/list showing existing data sources with columns:

  • ID
  • Name
  • DSL sentence
  • Updated at

List behavior:

  • single-row selection
  • text filter by Name or sentence content
  • sort by Name and Updated at

Selection to persist between page loads.

Editor (create/update sentence)

Provide an editor panel with:

  • Name input (required)
  • multi-line DSL sentence input (required)
  • read-only Last validated status

Actions:

  • New:
    • clear editor and start create mode
  • Save:
    • create or update current record
    • enabled only when form is valid and has unsaved changes
  • Discard changes:
    • revert editor to last saved state
  • Delete:
    • confirmation required
    • blocked when the data source is referenced by one or more mailshots

Validation UX

Provide explicit validation before save and on-demand:

  • Validate sentence button
  • parse + semantic validation against:
    • grammar
    • source/field/operator whitelist
    • join-path rules

On validation result:

  • show Valid / Invalid
  • show Warnings separately from errors
  • for invalid:
    • show user-friendly error message
    • include line/column when available
    • do not save invalid sentence
  • for warning-only cases (for example missing email field):
    • allow save
    • allow preview
    • indicate the data source cannot be used for sending a mailshot

Preview UX

Provide Preview recipients action for selected sentence:

  • show recipient count
  • show sample rows (for example first 50)
  • derive displayed columns from the deterministic expected-field contract plus returned row shape
  • do not assume fixed contact/account projection columns
  • show execution time
  • do not execute/send mailshot from this page

DSL Sentence Builder (required)

Provide a Build DSL action in the data-source editor.

On click, open a modal/overlay builder that allows users to construct full valid DSL without memorising grammar keywords, filter names, or field names.

Builder layout:

  • Top sub-pane: Data Sources
  • Bottom sub-pane: Constraints / Filters
  • Footer: generated DSL preview + actions

Top sub-pane: Data Sources

Provide source selection controls for:

  • contacts
  • accounts
  • renewals
  • custom table source (schema/table selector or validated text input)

Behavior:

  • multiple source selection is combined using and
  • order of selected sources is preserved in generated sentence
  • selecting an invalid source combination is blocked in UI with clear message

Bottom sub-pane: Constraints / Filters

Provide controls to add one or more constraint rows.

Each row supports one of:

  • predefined filter
  • field comparison
  • grouped expression with not

Predefined filter control:

  • dropdown label uses user-friendly text, not raw DSL token
  • examples:
    • Renewal is selected -> selected-renewal
    • Renewal is pending -> pending-renewal
    • Contact is FEN1 -> fen1-contact
    • Contact is FEN2 -> fen2-contact
    • Account is member/affiliate/parish council -> member-or-affiliate-or-parish-council

Field comparison row:

  • source dropdown (restricted to selected sources)
  • field dropdown (restricted to fields allowed for selected source)
  • operator dropdown (=, !=, contains, starts-with, ends-with, in)
  • RHS mode dropdown:
    • literal value input (typed by expected value type), or
    • source+field selector for field-to-field comparisons and set-membership (in)
  • for in, provide multi-value token input

Logical composition:

  • all rows combine with and in v1.5
  • each row can be negated via checkbox (not)
  • optional group rows allow nested bracketed expressions

Generated DSL + Sync

Footer must show:

  • read-only generated DSL sentence (live update)
  • validation status
  • parse/semantic errors with line/column mapped to relevant builder controls

Actions:

  • Apply writes generated DSL to the editor text area
  • Cancel closes overlay with no change
  • Reset clears builder selections

Sync rules:

  • opening builder from existing DSL pre-populates controls when sentence is parseable
  • if existing DSL is not parseable, show message and allow user to start from clean builder
  • direct text edits remain supported; builder is not the only editing path

Safety and usability requirements

  • UI must prevent creating syntactically invalid DSL states
  • UI must prevent selecting filters incompatible with current source set
  • custom table sources must be validated before Apply
  • generated DSL must always conform to strict grammar in this spec

API expectations for UI

The UI expects backend endpoints for:

  • list data sources
  • get one data source
  • validate sentence
  • create data source
  • update data source
  • delete data source
  • preview sentence (count + sample)
  • list schemas for custom source selection
  • list tables for selected schema
  • list fields for selected source (built-in or custom)

WordPress REST route contract:

  • GET /wp-json/mailshots/v1/data-sources
  • GET /wp-json/mailshots/v1/data-sources/{id}
  • POST /wp-json/mailshots/v1/data-sources/validate
  • POST /wp-json/mailshots/v1/data-sources
  • PUT /wp-json/mailshots/v1/data-sources/{id}
  • DELETE /wp-json/mailshots/v1/data-sources/{id}
  • POST /wp-json/mailshots/v1/data-sources/preview
  • POST /wp-json/mailshots/v1/data-sources/{id}/preview
  • GET /wp-json/mailshots/v1/data-sources/schemas
  • GET /wp-json/mailshots/v1/data-sources/tables?schema=...
  • GET /wp-json/mailshots/v1/data-sources/source-fields?source=...

Auth/capability requirements:

  • all routes require authenticated WordPress users;
  • create/update/delete routes require a mailshot management capability (for example manage_mailshots);
  • read/preview/validate routes require a mailshot read capability (for example read_mailshots).

All endpoints must enforce server-side validation even if client validates first.

Additional functional requirements (in scope)

  • Mailshot Data Source preview, Mailshot Recipients Preview, and Mailshot Test preview must all display source-faithful columns using the same result-shape rule.
  • DSL Builder must support custom sources via schema + table selection (not text-only entry), with manual text fallback only when metadata APIs are unavailable.
  • After a custom source is added in Builder, it must be immediately available in:
    • source dropdowns,
    • LHS field dropdowns,
    • RHS field dropdowns (for field comparisons).
  • Builder must support field not in (other_source.other_field) generation using RHS source + field controls (not value-type literal controls).
  • Builder prefill must parse and restore field references where source names are qualified (for example members.ExcludedAccounts.ExcludedAccount).
  • Validation errors must be explicit for unsupported/ineffective predicates; no silent no-op predicate behavior.

Implementation details

This section is normative guidance for regenerating the feature from scratch.

1. Data model

  • Table: MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries
  • Required columns:
    • ID (PK)
    • name
    • dsl_text
    • updated_at (recommended)
  • dsl_text is the only executable representation.
  • Persist sql as blank/deprecated if column still exists for backward compatibility; do not execute it.

2. Backend architecture

Build a strict pipeline:

  1. Tokenize DSL text.
  2. Parse to AST.
  3. Validate AST (grammar + semantics).
  4. Compile AST to parameterized SQL and parameters.
  5. Execute SQL for preview/send paths.

Keep parser/validator/compiler in a dedicated module (for example mailshotDsl.js), and route handlers thin.

3. Parser/AST contract

  • AST should separate source_expr and where.
  • Predicates should distinguish:
    • named filters,
    • compare literal,
    • compare field-to-field,
    • in(value_list),
    • in(field_ref) / not in(field_ref).
  • Preserve token line/column for diagnostics.

4. Validation rules (server-side)

  • Validate source compatibility via explicit join-path metadata.
  • Validate filter applicability against selected sources.
  • Validate field refs against per-source field whitelist:
    • built-ins: static map
    • custom: metadata-provider abstraction (not direct SQL dependency in validator logic)
  • Enforce:
    • field-to-field only = / !=
    • literal RHS only for contains / starts-with / ends-with
  • Return warnings (not errors) for no-email-source sentences.
  • Metadata-provider minimum DB grants must be documented and kept minimal for the read-only DB user.

5. SQL compilation strategy

Use parameterized SQL only.

  • Built-in-only source expressions:
    • compile from fixed source metadata (table + alias + base where).
    • compile approved joins only.
  • Custom-only source expressions:
    • single source: SELECT * FROM schema.table WHERE ...
    • multi source: require explicit = field join connectivity and compile with aliased joins.
  • in(field_ref) / not in(field_ref):
    • compile as EXISTS / NOT EXISTS subqueries.

6. Preview/result-shape rules

  • Normalize row values to JSON-safe scalars.
  • Enforce expected-field contract from DSL-cited sources.
  • Keep source-native returned columns available.
  • It is allowed to expose normalized source-field aliases required to satisfy the expected-field contract.
  • Do not add unrelated synthetic recipient projection columns.
  • Derive display columns from expected-field contract plus union of keys in returned rows.
  • Prune-all-empty columns only if explicitly configured and documented; default should preserve source-faithful shape.

7. API surface

Implement at minimum:

  • GET /wp-json/mailshots/v1/data-sources
  • GET /wp-json/mailshots/v1/data-sources/{id}
  • POST /wp-json/mailshots/v1/data-sources/validate
  • POST /wp-json/mailshots/v1/data-sources
  • PUT /wp-json/mailshots/v1/data-sources/{id}
  • DELETE /wp-json/mailshots/v1/data-sources/{id}
  • POST /wp-json/mailshots/v1/data-sources/preview
  • POST /wp-json/mailshots/v1/data-sources/{id}/preview
  • GET /wp-json/mailshots/v1/data-sources/schemas
  • GET /wp-json/mailshots/v1/data-sources/tables?schema=...
  • GET /wp-json/mailshots/v1/data-sources/source-fields?source=...

All endpoints must validate server-side and return structured errors: { ok:false, error, line?, column? }.

8. UI generation guidance

Mailshot Data Source page:

  • left: data source list with select/filter/sort.
  • right: editor + validation + preview.
  • include Build DSL modal.

Builder modal:

  • source selection section (built-in + schema/table custom add).
  • constraints section with row editor (filter/comparison).
  • responsive row layout that wraps controls on narrow widths.
  • RHS field mode must expose source + field selectors.
  • show generated DSL live; validate with debounce.
  • Apply writes DSL text; Cancel leaves editor unchanged.

9. Safety and regression test guidance

Minimum automated tests:

  • parser positive/negative cases (line/column assertions).
  • semantic validation failures (join path, filter applicability, field whitelist).
  • compile output shape for:
    • built-in joins,
    • custom joins,
    • in(field_ref) and not in(field_ref).
  • preview shape:
    • includes all expected fields from cited sources, and
    • does not introduce unrelated synthetic recipient projection columns.
  • builder prefill for qualified source field refs.
  • no-op predicate prevention (must error, not silently ignore).