adrians
/
feca-mailshots-plugin
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
feca-mailshots-plugin/requirements/mailshot_data_source.md

28 KiB
Raw Blame History

Mailshot Data Source page

Purpose

Document requirements for a safe, non-admin-friendly mailshot data source definition model.

Environment baseline for this specification

members as a database name means the value of MEMBERS_REMOTE_MYSQL_DB. fen as a database name means the value of FEN_REMOTE_MYSQL_DB.

For this specification:

  • Data-source definition persistence (create/update/delete/list of mailshot_queries) uses MAILSHOTS_REMOTE_MYSQL_DB.
  • Source-data query execution for membership built-in/custom recipient sources uses MEMBERS_REMOTE_MYSQL_DB.
  • Source-data query execution for FEN editorial, advertising, and invoice built-in sources uses FEN_REMOTE_MYSQL_DB.

Problem with prior approach

Creating a new data source currently requires:

  • knowledge of MySQL schema details, or
  • admin-level access to create DB views/queries.

This blocks non-expert and non-admin users.

Direction

Use a strict, constrained query language (DSL) for mailshot audience selection.

The DSL text is stored as the data-source definition in MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries, then parsed and compiled into safe SQL executed against approved source-data tables in MEMBERS_REMOTE_MYSQL_DB and FEN_REMOTE_MYSQL_DB.

Goals

  • Allow non-admin users to define useful audiences.
  • Prevent arbitrary SQL.
  • Keep execution efficient (push filtering to SQL, avoid large unnecessary joins).
  • Make queries explainable and previewable.

Example requirements (in natural language)

  • All contacts with name like "smith".
  • All contacts joined to member accounts where contact FEN1 is true.
  • All renewal rows marked as selected.
  • All pending renewals.
  • All accounts where name is not in an exclusion table.

Strict grammar (v1.6)

Use lowercase keywords only.

EBNF

query              = source_expr , [ where_clause ] ;

source_expr        = source_term , { source_op , source_term } ;
source_term        = source_ref | "(" , source_expr , ")" ;
source_op          = "and" ;

source_ref         = built_in_source | table_ref ;
built_in_source    = "contacts"
                   | "accounts"
                   | "renewals"
                   | "advertisers"
                   | "ads"
                   | "pages"
                   | "articles"
                   | "issues"
                   | "invoices" ;
table_ref          = ident_part , "." , ident_part ;
ident_part         = identifier | quoted_identifier ;

where_clause       = "where" , predicate_expr ;
predicate_expr     = predicate_term , { bool_op , predicate_term } ;
predicate_term     = [ "not" ] , predicate_atom
                   | "(" , predicate_expr , ")" ;
bool_op            = "and" ;

predicate_atom     = filter_expr
                   | field_ref , compare_op , value
                   | field_ref , ("=" | "!=") , field_ref
                   | field_ref , [ "not" ] , "in" , "(" , field_ref , ")"
                   | field_ref , "in" , "(" , value_list , ")" ;

filter_expr        = filter_name
                   | filter_name , "(" , value_list , ")" ;

filter_name        = "selected-renewal"
                   | "pending-renewal"
                   | "selected"
                   | "page-in-issue"
                   | "ad-in-issue"
                   | "pending-invoice"
                   | "selected-invoice"
                   | "invoice-ids"
                   | "fen1-contact"
                   | "primary-contact"
                   | "member-or-affiliate-or-parish-council" ;

field_ref          = source_ref , "." , field_name ;
field_name         = identifier | quoted_identifier ;
compare_op         = "=" | "!=" | "contains" | "starts-with" | "ends-with" ;

value_list         = value , { "," , value } ;
value              = quoted_string | number | "true" | "false" ;

identifier         = letter , { letter | digit | "_" } ;
quoted_identifier  = "`" , { character - "`" } , "`" ;
quoted_string      = "'" , { character - "'" } , "'" ;
number             = digit , { digit } ;
letter             = "a"…"z" | "A"…"Z" ;
digit              = "0"…"9" ;

Semantics (v1.6)

  • built_in_source maps to a predefined table alias.
  • For accounts, the compiler must implicitly left-join account picklists so additional readable virtual fields are available:
    • accounts.type from picklist_account_type.value
    • accounts.public_location from picklist_public_location.value
    • accounts.account_sector from picklist_sector.value
  • table_ref maps directly to a table in the remote members database (schema.table).
  • For table_ref, validation must confirm the table exists.
  • Query result shape must be source-faithful and deterministic per DSL:
    • each cited source contributes its full allowed field set to an expected-field contract;
    • preview and template-token resolution must preserve availability of every field in that contract, even when raw SQL row keys are sparse;
    • source-native columns from query output may also be present;
    • no unrelated synthetic recipient projection columns are allowed.
  • If the resolved source does not expose an email field, validation must return a warning:
    • Data Source does not have an email field, it cannot be used for a Mailshot
    • The sentence is still valid and can be saved and previewed.
  • and between sources means relational intersection using configured join paths (not email-only matching).
  • For custom sources, an explicit field equality predicate can provide join semantics.
  • where applies after source composition.
  • not negates only the next predicate/group.
  • Predefined predicates are selected-renewal, pending-renewal, selected, page-in-issue, ad-in-issue, pending-invoice, selected-invoice, invoice-ids, fen1-contact, primary-contact, and member-or-affiliate-or-parish-council.
  • renewals is a built-in source mapped to membership renewal rows.
  • pending-renewal applies only when source set includes renewals and means renewals.status = 'pending'.
  • selected-renewal applies only when source set includes renewals and means renewals.selected = true.
  • primary-contact applies only when source set includes contacts and means contacts.is_contact_1 is truthy.
  • FEN editorial, advertising, and invoice data must be exposed as built-in sources, not custom table references, using the source names below.
  • These built-in sources are derived from the sibling ../feca2-app/server/src/lib/mailshotDsl.js implementation, except that articles must now be promoted to a first-class built-in source. In the sibling implementation, Articles is used by an issue-scoped article filter but is not listed as a standalone built-in source.
  • FEN built-in source table mappings:
    • advertisers maps to fen.advertisers.
    • ads maps to fen.ads.
    • pages maps to fen.Pages.
    • articles maps to fen.Articles.
    • issues maps to fen.Issues.
    • invoices maps to fen.invoices.
  • Required FEN source fields and canonical token aliases:
    • advertisers: name / advertisername from AdvertiserName.
    • ads: id from ID, advertiser from Advertiser, adsize / size from AdSize, price from Price, issue from the related page Issue, pageid from PageID, state from State, notes from Notes.
    • pages: id from ID, issue from Issue, page from Page.
    • articles: id from ID, pageid from PageID, articlenumber / article from ArticleNumber, content from Content, membername from MemberName, author from Author, dcn from DCN, articlewords from ArticleWords, otherwords from OtherWords, othercontent / other from OtherContent.
    • issues: id / issue from ID, issuemonths from IssueMonths, description from Description.
    • invoices: id, issue / issue_id, ad_id, invoice_number, invoice_date, due_date, invoice_page, invoice_size, invoice_price, status, payment_date, amount_paid, payment_method, payment_reference, notes, created_at, updated_at.
  • Required FEN physical columns:
    • fen.Issues: ID, CopyDate, PublicationDate, IssueMonths, Description.
    • fen.Pages: ID, Issue, Page, PageSizeName, Content.
    • fen.Articles: ID, PageID, ArticleNumber, Content, MemberName, Author, DCN, ArticleWords, OtherWords, OtherContent.
    • fen.ads: ID, PageID, AdSize, Advertiser, Price, State, Notes.
    • fen.advertisers: Entry ID, AdvertiserName, title, contact_name, address_1, address_2, town, post_code, Description, IsLapsed?, Home Phone, Phone, Email, Selected.
    • fen.invoices: id, issue_id, ad_id, invoice_number, invoice_date, due_date, invoice_page, invoice_size, invoice_price, status, payment_date, amount_paid, payment_method, payment_reference, notes, created_at, updated_at.
  • selected applies only when source set includes advertisers and means advertisers.Selected is truthy.
  • page-in-issue(<issue>) accepts exactly one numeric issue ID. It applies only when the source set includes pages or articles, and means the page row, or the page joined from the article row, belongs to that issue.
  • ad-in-issue(<issue>) accepts exactly one numeric issue ID. It applies only when the source set includes one of advertisers, ads, pages, issues, or invoices, and means the row has an ad in that issue.
  • pending-invoice applies only when source set includes invoices and means invoices.status = 'pending'.
  • selected-invoice applies only when source set includes invoices and means the invoice ID is in the runtime selected invoice ID list.
  • invoice-ids(...) applies only when source set includes invoices; it accepts one or more numeric invoice IDs.
  • member-or-affiliate-or-parish-council applies to account data and means:
    • accounts.type is Member or Affiliate, or
    • accounts.name contains Parish Council.
  • field_ref is restricted by whitelist per built-in source.
  • field_ref is also supported for selected custom sources when table metadata is available.
  • Field-to-field comparisons are supported with = and != only.
  • Set-membership against another field source is supported via:
    • field_ref in (field_ref)
    • field_ref not in (field_ref)
  • For in(field_ref) / not in(field_ref), the RHS source is treated as a reference source (subquery semantics), not a joined source.
  • contains, starts-with, and ends-with require a literal right-hand side value (not a field reference).
  • In v1.6, mixing built-in sources and custom sources in the same sentence is not supported; validation must fail with a clear error.
  • For multi-custom-source sentences, all selected custom sources must be connected by explicit = field-to-field predicates (graph-connected join semantics), otherwise validation fails.
  • in(field_ref) / not in(field_ref) must compile as subquery membership; the RHS source is a reference source and does not need to appear in source_expr.
  • Canonical naming for built-in sources uses the token aliases listed in this document.
  • Canonical naming for custom table references uses exact field names as defined in database metadata for the selected source. The implementation must reject invalid custom field names and must not add aliasing/normalization fallback.

Join-path model (required)

The compiler must use an explicit join graph per source pair. Example v1 join paths:

  • contacts -> accounts: contacts.Accountid = accounts.ID
  • accounts -> contacts: accounts.ID = contacts.Accountid
  • renewals -> accounts: renewals.account_id = accounts.ID
  • accounts -> renewals: accounts.ID = renewals.account_id
  • renewals -> contacts: renewals.account_id = contacts.Accountid
  • contacts -> renewals: contacts.Accountid = renewals.account_id
  • articles -> pages: articles.PageID = pages.ID
  • pages -> articles: pages.ID = articles.PageID
  • pages -> issues: pages.Issue = issues.ID
  • issues -> pages: issues.ID = pages.Issue
  • ads -> pages: ads.PageID = pages.ID
  • pages -> ads: pages.ID = ads.PageID
  • ads -> advertisers: normalized ads.Advertiser = advertisers.AdvertiserName
  • advertisers -> ads: normalized advertisers.AdvertiserName = ads.Advertiser
  • invoices -> ads: invoices.ad_id = ads.ID
  • ads -> invoices: ads.ID = invoices.ad_id
  • invoices -> issues: invoices.issue_id = issues.ID
  • issues -> invoices: issues.ID = invoices.issue_id
  • invoices -> pages: invoices.issue_id = pages.Issue
  • pages -> invoices: pages.Issue = invoices.issue_id

If no approved join path exists between two sources for and, parsing/validation must fail with a clear error.

No implicit join behavior is allowed:

  • Selecting a source such as accounts must not silently join contacts (or any other source).
  • Output columns must come only from sources explicitly requested in the sentence.

Example valid queries

  • contacts
  • contacts and accounts where fen1-contact
  • contacts and accounts where primary-contact
  • accounts and contacts where contacts.Accountid = accounts.ID
  • contacts where contacts.Last contains 'smith'
  • contacts and accounts where accounts.Type = 'Member' and contacts.FENContact1 = true
  • accounts where member-or-affiliate-or-parish-council
  • renewals where pending-renewal
  • renewals where selected-renewal
  • renewals and accounts and contacts where pending-renewal
  • advertisers where selected
  • ads and pages where ad-in-issue(202605)
  • articles and pages where page-in-issue(202605)
  • invoices where pending-invoice
  • invoices and ads and advertisers where invoice-ids(101, 102)
  • accounts where accounts.Name not in (members.ExcludedAccounts.ExcludedAccount)
  • members.mailshot_test

Example invalid queries

  • select * from members (raw SQL not allowed)
  • contacts where drop table (unknown tokens)
  • contacts where accounts.Name = 'x' (invalid if accounts is not included in source expression)
  • contacts or accounts (or is not supported; use and only)
  • accounts where pending-renewal (invalid: filter requires renewals source)
  • accounts where fen1-contact (invalid: filter requires contacts source)
  • accounts where primary-contact (invalid: filter requires contacts source)
  • ads where selected (invalid: filter requires advertisers source)
  • advertisers where pending-invoice (invalid: filter requires invoices source)
  • pages where ad-in-issue (invalid: ad-in-issue requires exactly one numeric issue ID)
  • ads where page-in-issue(202605) (invalid: page-in-issue requires pages or articles source)
  • invoices where invoice-ids('abc') (invalid: invoice-ids accepts numeric invoice IDs only)
  • renewals where member-or-affiliate-or-parish-council (invalid: filter requires accounts source)
  • accounts and contacts where contacts.Accountid contains accounts.ID (invalid: field-to-field supports only =/!=)
  • accounts where accounts.Name contains members.ExcludedAccounts.ExcludedAccount (invalid: contains requires a literal RHS)

Compilation and performance requirements

  • Parse DSL to AST.
  • Validate AST against allowed sources, fields, and operators.
  • Validate all and source combinations against approved join graph metadata.
  • Do not perform implicit source joins during compilation.
  • Compile AST to parameterized SQL only (no string-concatenated SQL).
  • Push filters into SQL WHERE / join conditions.
  • Preview must not assume fixed contact/account projection columns.
  • Compiler/runtime must enforce deterministic source-field availability from DSL-cited sources.
  • Preview column derivation must include:
    • all fields from the DSL expected-field contract, and
    • union of keys from returned rows (not just first row keys), to avoid hiding sparse/source-native columns.
  • For mailshot execution, email remains the only required field.
  • Provide preview endpoints: count + sample rows before execution.

Storage model

Store:

  • dsl_text (editable source).
  • dsl_text is the single source of truth for execution, preview, and validation.
  • No SQL fallback path is permitted when dsl_text is missing/empty; this must be treated as an error.

Optional:

  • normalized AST JSON for diagnostics/explain.
  • sql retained as blank/deprecated for backward compatibility when the legacy column exists, and never executed.

User interface requirements

Provide a UI to view, create, update, validate, and preview DSL sentences.

Mailshot data source page

Add a dedicated page "Data Sources" under FECA Mailshots admin page for managing data source sentences.

Top regions:

  • Context pane:
    • page title Mailshot Data Sources
  • Information pane:
    • validation errors, save status, preview status
  • Statistics pane:
    • total saved data sources
    • selected data source id/name

Data source list (view)

Provide a table/list showing existing data sources with columns:

  • ID
  • Name
  • DSL sentence
  • Updated at

List behavior:

  • single-row selection
  • text filter by Name or sentence content
  • sort by Name and Updated at

Selection to persist between page loads.

Editor (create/update sentence)

Provide an editor panel with:

  • Name input (required)
  • multi-line DSL sentence input (required)
  • read-only Last validated status

Actions:

  • New:
    • clear editor and start create mode
  • Save:
    • create or update current record
    • enabled only when form is valid and has unsaved changes
  • Discard changes:
    • revert editor to last saved state
  • Delete:
    • confirmation required
    • blocked when the data source is referenced by one or more mailshots

Validation UX

Provide explicit validation before save and on-demand:

  • Validate sentence button
  • parse + semantic validation against:
    • grammar
    • source/field/operator whitelist
    • join-path rules

On validation result:

  • show Valid / Invalid
  • show Warnings separately from errors
  • for invalid:
    • show user-friendly error message
    • include line/column when available
    • do not save invalid sentence
  • for warning-only cases (for example missing email field):
    • allow save
    • allow preview
    • indicate the data source cannot be used for sending a mailshot

Preview UX

Provide Preview recipients action for selected sentence:

  • show recipient count
  • show sample rows (for example first 50)
  • derive displayed columns from the deterministic expected-field contract plus returned row shape
  • do not assume fixed contact/account projection columns
  • show execution time
  • do not execute/send mailshot from this page

DSL Sentence Builder (required)

Provide a Build DSL action in the data-source editor.

On click, open a modal/overlay builder that allows users to construct full valid DSL without memorising grammar keywords, filter names, or field names.

Builder layout:

  • Top sub-pane: Data Sources
  • Bottom sub-pane: Constraints / Filters
  • Footer: generated DSL preview + actions

Top sub-pane: Data Sources

Provide source selection controls for:

  • contacts
  • accounts
  • renewals
  • custom table source (schema/table selector or validated text input)

Behavior:

  • multiple source selection is combined using and
  • order of selected sources is preserved in generated sentence
  • selecting an invalid source combination is blocked in UI with clear message

Bottom sub-pane: Constraints / Filters

Provide controls to add one or more constraint rows.

Each row supports one of:

  • predefined filter
  • field comparison
  • grouped expression with not

Predefined filter control:

  • dropdown label uses user-friendly text, not raw DSL token
  • examples:
    • Renewal is selected -> selected-renewal
    • Renewal is pending -> pending-renewal
    • Advertiser is selected -> selected
    • Page is in issue -> page-in-issue(...)
    • Advertiser has ad in issue -> ad-in-issue(...)
    • Invoice is pending -> pending-invoice
    • Invoice is selected -> selected-invoice
    • Invoice ID is one of -> invoice-ids(...)
    • Contact is FEN1 -> fen1-contact
    • Contact is primary -> primary-contact
    • Account is member/affiliate/parish council -> member-or-affiliate-or-parish-council

Field comparison row:

  • source dropdown (restricted to selected sources)
  • field dropdown (restricted to fields allowed for selected source)
  • operator dropdown (=, !=, contains, starts-with, ends-with, in)
  • RHS mode dropdown:
    • literal value input (typed by expected value type), or
    • source+field selector for field-to-field comparisons and set-membership (in)
  • for in, provide multi-value token input

Logical composition:

  • all rows combine with and in v1.6
  • each row can be negated via checkbox (not)
  • optional group rows allow nested bracketed expressions

Generated DSL + Sync

Footer must show:

  • read-only generated DSL sentence (live update)
  • validation status
  • parse/semantic errors with line/column mapped to relevant builder controls

Actions:

  • Apply writes generated DSL to the editor text area
  • Cancel closes overlay with no change
  • Reset clears builder selections

Sync rules:

  • opening builder from existing DSL pre-populates controls when sentence is parseable
  • if existing DSL is not parseable, show message and allow user to start from clean builder
  • direct text edits remain supported; builder is not the only editing path

Safety and usability requirements

  • UI must prevent creating syntactically invalid DSL states
  • UI must prevent selecting filters incompatible with current source set
  • custom table sources must be validated before Apply
  • generated DSL must always conform to strict grammar in this spec

API expectations for UI

The UI expects backend endpoints for:

  • list data sources
  • get one data source
  • validate sentence
  • create data source
  • update data source
  • delete data source
  • preview sentence (count + sample)
  • list schemas for custom source selection
  • list tables for selected schema
  • list fields for selected source (built-in or custom)

WordPress REST route contract:

  • GET /wp-json/mailshots/v1/data-sources
  • GET /wp-json/mailshots/v1/data-sources/{id}
  • POST /wp-json/mailshots/v1/data-sources/validate
  • POST /wp-json/mailshots/v1/data-sources
  • PUT /wp-json/mailshots/v1/data-sources/{id}
  • DELETE /wp-json/mailshots/v1/data-sources/{id}
  • POST /wp-json/mailshots/v1/data-sources/preview
  • POST /wp-json/mailshots/v1/data-sources/{id}/preview
  • GET /wp-json/mailshots/v1/data-sources/schemas
  • GET /wp-json/mailshots/v1/data-sources/tables?schema=...
  • GET /wp-json/mailshots/v1/data-sources/source-fields?source=...

Auth/capability requirements:

  • all routes require authenticated WordPress users;
  • in this plugin, editor-capable operational access is the baseline:
    • create/update/delete/read/preview/validate all require WordPress edit_pages (editor or administrator);
  • plugin Setup/settings remains administrator-only under WordPress manage_options (defined in top-level policy).

All endpoints must enforce server-side validation even if client validates first.

Additional functional requirements (in scope)

  • Mailshot Data Source preview, Mailshot Recipients Preview, and Mailshot Test preview must all display source-faithful columns using the same result-shape rule.
  • DSL Builder must support custom sources via schema + table selection (not text-only entry), with manual text fallback only when metadata APIs are unavailable.
  • After a custom source is added in Builder, it must be immediately available in:
    • source dropdowns,
    • LHS field dropdowns,
    • RHS field dropdowns (for field comparisons).
  • Builder must support field not in (other_source.other_field) generation using RHS source + field controls (not value-type literal controls).
  • Builder prefill must parse and restore field references where source names are qualified (for example members.ExcludedAccounts.ExcludedAccount).
  • Validation errors must be explicit for unsupported/ineffective predicates; no silent no-op predicate behavior.

Implementation details

This section is normative guidance for regenerating the feature from scratch.

1. Data model

  • Table: MAILSHOTS_REMOTE_MYSQL_DB.mailshot_queries
  • Required columns:
    • ID (PK)
    • name
    • dsl_text
    • updated_at (recommended)
  • dsl_text is the only executable representation.
  • Persist sql as blank/deprecated if column still exists for backward compatibility; do not execute it.

2. Backend architecture

Build a strict pipeline:

  1. Tokenize DSL text.
  2. Parse to AST.
  3. Validate AST (grammar + semantics).
  4. Compile AST to parameterized SQL and parameters.
  5. Execute SQL for preview/send paths.

Keep parser/validator/compiler in a dedicated module (for example mailshotDsl.js), and route handlers thin.

3. Parser/AST contract

  • AST should separate source_expr and where.
  • Predicates should distinguish:
    • named filters,
    • compare literal,
    • compare field-to-field,
    • in(value_list),
    • in(field_ref) / not in(field_ref).
  • Preserve token line/column for diagnostics.

4. Validation rules (server-side)

  • Validate source compatibility via explicit join-path metadata.
  • Validate filter applicability against selected sources.
  • Validate field refs against per-source field whitelist:
    • built-ins: static map
    • custom: metadata-provider abstraction (not direct SQL dependency in validator logic)
  • Enforce:
    • field-to-field only = / !=
    • literal RHS only for contains / starts-with / ends-with
  • Return warnings (not errors) for no-email-source sentences.
  • Metadata-provider minimum DB grants must be documented and kept minimal for the read-only DB user.

5. SQL compilation strategy

Use parameterized SQL only.

  • Built-in-only source expressions:
    • compile from fixed source metadata (table + alias + base where).
    • compile approved joins only.
  • Custom-only source expressions:
    • single source: SELECT * FROM schema.table WHERE ...
    • multi source: require explicit = field join connectivity and compile with aliased joins.
  • in(field_ref) / not in(field_ref):
    • compile as EXISTS / NOT EXISTS subqueries.

6. Preview/result-shape rules

  • Normalize row values to JSON-safe scalars.
  • Enforce expected-field contract from DSL-cited sources.
  • Keep source-native returned columns available.
  • It is allowed to expose normalized source-field aliases required to satisfy the expected-field contract.
  • Do not add unrelated synthetic recipient projection columns.
  • Derive display columns from expected-field contract plus union of keys in returned rows.
  • Prune-all-empty columns only if explicitly configured and documented; default should preserve source-faithful shape.

7. API surface

Implement at minimum:

  • GET /wp-json/mailshots/v1/data-sources
  • GET /wp-json/mailshots/v1/data-sources/{id}
  • POST /wp-json/mailshots/v1/data-sources/validate
  • POST /wp-json/mailshots/v1/data-sources
  • PUT /wp-json/mailshots/v1/data-sources/{id}
  • DELETE /wp-json/mailshots/v1/data-sources/{id}
  • POST /wp-json/mailshots/v1/data-sources/preview
  • POST /wp-json/mailshots/v1/data-sources/{id}/preview
  • GET /wp-json/mailshots/v1/data-sources/schemas
  • GET /wp-json/mailshots/v1/data-sources/tables?schema=...
  • GET /wp-json/mailshots/v1/data-sources/source-fields?source=...

All endpoints must validate server-side and return structured errors: { ok:false, error, line?, column? }.

8. UI generation guidance

Mailshot Data Source page:

  • left: data source list with select/filter/sort.
  • right: editor + validation + preview.
  • include Build DSL modal.

Builder modal:

  • source selection section (built-in + schema/table custom add).
  • constraints section with row editor (filter/comparison).
  • responsive row layout that wraps controls on narrow widths.
  • RHS field mode must expose source + field selectors.
  • show generated DSL live; validate with debounce.
  • Apply writes DSL text; Cancel leaves editor unchanged.

9. Safety and regression test guidance

Minimum automated tests:

  • parser positive/negative cases (line/column assertions).
  • semantic validation failures (join path, filter applicability, field whitelist).
  • compile output shape for:
    • built-in joins,
    • custom joins,
    • in(field_ref) and not in(field_ref).
  • preview shape:
    • includes all expected fields from cited sources, and
    • does not introduce unrelated synthetic recipient projection columns.
  • builder prefill for qualified source field refs.
  • no-op predicate prevention (must error, not silently ignore).