Qwiratry DatabaseStorage Location Tree

Plan/Spec for tree-document tables: projecting a live Table out of a parsed document tree (JSON, XML, YAML, HTML, …) via Qwiratry tree navigation, then selecting rows with ordinary Relation / Catalog walk.

Status: Tree Location MVP shipped for JSON / YAML / XML (checklist §16.1–§16.3 — core Tree + Format dists). §16.4.1–§16.4.8 landed (TreeWithPath / :path, path-transformer, write-path / :table-action, merge molds with :context, multi-site :table-name-selector, sync API without mutation auto-flush; Format JSON/YAML/XML re-runs). Remaining follow-on (HTML) tracked under Post-SQL §F.7 / plan §16.5. Decisions Q1–Q31 are under §18 (§17 has no open items for this cut). Projection selectors are Blocks / Callables (Q16b).

Related sources: Qwiratry-TaOP-Basics.md §4.1–§4.2 (Location vs Format vs DatabaseStorage; document-to-rows / rows-to-document hooks; Tree under Location); Post-SQL-TaOP-Improvements.md §F.7 (implementation-order home for this work); current Qwiratry::DatabaseStorage::Location::Tree (core) and …::Location::JSON (Format-JSON dist).

Question IDs are stable: Q1, Q2, … stay the same when an item moves from §Open Questions into §Decisions.

1. Goals

1. Stop treating “document root = array of row objects” as the JSON (or any tree Format) table shape.

2. Introduce Qwiratry::DatabaseStorage::Location::Tree in the core raku-Qwiratry repo (not in Format-JSON or any other Format dist). It owns document → table projection for tree-shaped payloads. Format dists only ship thin subclasses that default :format.

3. Resolve each projection stage with Qwiratry tree navigation only (same operators / evaluators / navigators as document query — no JSONPath, XPath-only dialect, or Format-private path language). Callers may omit useTable projection arguments when the Format’s Location::* subclass supplies a default (§8).

4. Project row data through further tree-navigation stages (row collection under the table site; cell → field mapping) so HTML-like nesting (<table> / <tbody> / <tr> / cells) is expressible without Format-private hacks. Which candidate nodes count as data rows is the row-selector’s job (or a later ordinary query over the loaded table) — not a separate Location filter stage.

5. Keep in-memory table → row selection on the existing Relation / Catalog / WalkCursor path after projection into the row buffer.

6. Flush with merge: write projected row data back through the same stages into the document tree, then Format-render and Location-write.

7. Keep Format and blob Location general; JSON / XML / YAML / HTML differ by Format, thin Location::* subclasses (default :format), and which projection defaults those subclasses advertise (§8). (HTML here is only an example of that flexibility — not a commitment to ship Location::HTML in this cut.)

2. Non-goals

• Inventing a second path language for DatabaseStorage.

• Putting table-shape assumptions in Format::JSON (or future XML / YAML / HTML Formats).

• Making blob Location::* (File, HTTP) grow table APIs.

• SQL / CSV / Memory backends (unchanged by this plan).

• Using DatabaseStorage for documents that are not opened as tables — plain Format Parse + tree query remains the path for free-form trees.

• Streaming pull of document slices (Streaming-v2 / Post-SQL §F.8 Format pull); first cut may keep whole-document load/flush.

• Guessing HTML header vs body conventions when the caller did not encode that in the row-selector (or handle it with ordinary query after load).

• A separate row-filter projection stage — redundant with row-selector and with outside query operators (see §18).

3. Problem with the current cut

Qwiratry::DatabaseStorage::Location::JSON (Format-JSON dist) implements document-to-rows / rows-to-document as:

• parse root is a Positional of Associatives → those are the rows; or

• parse root is a single Associative → one-row table.

That falsely assumes the top of the document is the table. Real documents nest tables under keys (or deeper structure). The same mistake would recur for XML, YAML, and HTML if each Format subclass hard-coded a root shape.

Even after selecting a table site, row data is often not “the site’s immediate children as Associatives.” HTML is the clearest example:

• The table site may be a <table> node.

• Data rows may live under <tbody> (or be mixed with <thead> / <tfoot>).

• Header vs data <tr> distinctions belong in the row-selector (e.g. only tbody/tr, or only rows that match a navigation predicate the shared query stack already supports), and/or in ordinary query after load — not in a fourth Location-private filter stage.

• Cells must map to Relation fields (ordered <td> → columns, or named projection).

Basics §4.2 already stated the intended split: Location + Format stay general; DatabaseStorage hooks pick the array/object paths that become rows. This plan makes that normative, navigable, and multi-stage.

4. Layering (unchanged DQ2)

Projection runs on the already-parsed Mu tree (Associative / Positional / Format-specific nodes the Tree::Navigator understands). Formats must not grow “table mode.”

5. Two outer steps

5.1. Document → in-memory table (this plan)

Open/load runs the projection pipeline (§6) from the parsed document into @rows (and field metadata).

5.2. In-memory table → row (existing)

After projection, row selection is ordinary TaOP/Qwiratry table work: WalkCursor, Catalog child-results from a Database LHS, filters, Positional / Iterator scan, etc. Projection selectors do not encode “which live row the user wants next”; chaining “open the HTML table, then pick a data row” is (§6) then (§5.2), not a Location-private query string.

6. Projection pipeline (document → rows)

All stages use only Qwiratry tree navigation / matching (navigators, Query::Evaluator::Navigation, Query::Selector, and the same predicate shapes document query already uses — Q8). No Format-private CSS/XPath dialect beside what the shared stack already provides.

Stages run in order on load; flush walks them in reverse for merge (§9).

There is no separate row-filter stage. Narrowing which nodes become rows is done by the row-selector, or by ordinary query operators after the table is loaded (§5.2 / §18).

6.1. Table site (table root)

Select the table root — the node returned by :table-selector — from the document origin (Q1 / Q14).

Examples:

• JSON default: Block/query that yields the document parse root as the table root (today’s array-or-object root).

• Override: child key people under the document root.

• Future HTML default (illustrative): a sensible <table> site policy — not implemented in this cut.

Each open must end up with a resolved table-selector: caller argument, else Format Location::* default, else error (§8).

Parameter name (Q3a): :table-selector — a Block / Callable (Q16b) that returns the table root. An optional :table-name-selector (Q15, also a Block) may further narrow multiple hits (Q2).

6.2. Row collection

From the table root, select the row nodes that will be projected (:row-selector is relative to that root; it may still use ancestor axes to leave it — Q16a).

The table root is often not the row list:

• HTML (when a Format exists): from <table>, navigate to tbody then tr (skipping <thead> by not selecting it), or whatever navigation predicate already fits — caller and/or Format default chooses. Prefer encoding “data rows only” here rather than loading header rows and filtering later, when the document shape allows.

• JSON default: identity relative to the table root when that root is already the array of row objects (preserves today’s behaviour via §8 defaults).

Resolved the same way as §6.1 (caller → Format default → error).

Parameter name (Q3a): :row-selector — a Block / Callable (Q8 / Q16b) evaluated with origin = table root (Q16a).

Result: an ordered list of row nodes (not yet Relation field maps). Further row subsetting after load uses ordinary table/query operators (§5.2), not a Location row-filter argument.

6.3. Cell / field projection (and optional header)

Database.useTable opens a front-end Table and a backend storage table (under Tree: a DatabaseStorage::Location::Tree table). That backend table performs the Tree→Table step when it hydrates (load / rehydrate): run the selectors, map row nodes to row values, and establish fields.

Two column-naming modes (Q14 / Q17):

1. No :header-selector (hash-like): :row-selector yields row nodes; :field-projection (Block/query) selects fields from each row and assigns keys, returning hashes (column → value).

2. With :header-selector (array-like): run :header-selector before :row-selector; both return arrays that are spliced into hashes (header array → keys; each row array → values). In this mode key assignment comes from the header, not from :field-projection.

Format defaults may supply the usual case (JSON/YAML: hash-like object keys → fields, no header-selector). Callers override for positional / header-driven trees (Q10).

Examples:

• Hash-like (no header-selector): JSON/YAML object/mapping → column→value hashes from :field-projection.

• Array-like + :header-selector: header query → key array; row-selector → value arrays; splice into hashes.

• HTML <tr> (future): often header-selector + positional row cells.

Parameter names (Q3a): :field-projection; optional :header-selector.

6.4. Into the row buffer

The backend Tree table feeds projected rows into fill_from_aoh (or successor) using the column-name policy resolved at open (Q10 / Q17).

6.5. Pipeline sketch

document root
    --(table-selector)----->  table root             # e.g. <table> / JSON root
    --(header-selector?)--->  header key array       # optional (array mode)
    --(row-selector)------->  row nodes | value arrs # nodes (hash) / arrays (w/ header)
    --(field-projection)--->  column→value hashes    # hash mode only (key assign)
    -----------------------> @!rows + fields         # array mode: splice header×row

Illustrative useTable (API names per Q3a):

# JSON: Format defaults preserve today’s root array/object → rows
my $u = $db.useTable(name => 'people', action => 'can-create');

# Override when the table is nested under a key
my $t = $db.useTable(
	name => 'roster',
	action => 'use',
	table-selector => -> $doc { $doc<people> },  # → table root
);

# Same idea via query AST: evaluate-selector runs Query::Runtime from
# the document origin. C<$_> is a build-time placeholder (undefined
# subject → origin at select). Blocks remain the preferred Tree style
# (**Q16b**); OperatorBase is still accepted.
# use Qwiratry::Query::Slang;  # query-ast, ⪪
my $t-ast = $db.useTable(
	name => 'roster',
	action => 'use',
	table-selector => query-ast({ $_ ⪪ <people> }),
);

# Future HTML (illustrative only — not in this cut): Format may supply
# sensible defaults; caller overrides when needed
# my $h = $html-db.useTable(name => 'report', action => 'use');
# my $h2 = $html-db.useTable(..., row-selector => …);

# XML with blank Format defaults: omitting projection args errors
# my $x = $xml-db.useTable(name => 'items', action => 'use');  # error
# my $x = $xml-db.useTable(
#	name => 'items', action => 'use',
#	table-selector => …, row-selector => …, field-projection => …,
# );

7. Class shape

# raku-Qwiratry (core)
Qwiratry::DatabaseStorage::Location
└── Qwiratry::DatabaseStorage::Location::Tree   # common projection + merge
        # lib/Qwiratry/DatabaseStorage/Location/Tree.rakumod

# Format dists (thin Location::* subclasses — not the Tree base)
…::Location::JSON    # raku-Qwiratry-Format-JSON;  format => 'JSON'
…::Location::XML     # raku-Qwiratry-Format-XML;   format => 'XML'
…::Location::YAML    # raku-Qwiratry-Format-YAML;  format => 'YAML'
…::Location::HTML    # later Format dist if/when it exists; format => 'HTML'

Location::Tree (core raku-Qwiratry only)

• Lives in raku-Qwiratry, beside DatabaseStorage::Location — never in Format-JSON or another Format sibling.

• Resolve and store per-table projection policies (§6.1–§6.3) via §8 (caller argument, else Format default) and open-time navigation options (Q1 / Q3a / Q3b).

• Evaluate every stage via the shared tree navigation / predicate stack.

• Implement load: parse → pipeline → row buffer + fields.

• Implement flush: obtain document tree → navigate to table site / row collection → merge rows via the caller/Format :transformer (molds) → render → write (§9 / Q18).

• Reject open/load when a stage is still unresolved after §8, or when a stage yields a non-conforming shape (Q2 / Q10).

Format-dist subclasses (Location::JSON, later XML/YAML/HTML)

• Ship in their Format repo; subclass core Location::Tree.

• Default :format (and file-extension if needed).

• Provide projection defaults for the three useTable stages (§8) — or deliberately provide none (blank), so omission errors until the caller passes them (XML intent).

• Must not fork the Tree pipeline; defaults are ordinary selector / projection values built from shared navigation primitives.

• May ship named helpers for common overrides — not a private path DSL and not a Location row-filter API.

• May document Format-specific navigator quirks.

CSVdemo / non-tree Location backends stay on plain DatabaseStorage::Location (or their own subclasses). Tree projection (§6–§8) applies only to Location::Tree and its descendants.

8. useTable projection arguments and Format defaults

For DatabaseStorage::Location::Tree (and subclasses), each of the three projection stages must be resolved before load/flush:

Resolution order (per stage):

1. Value passed to useTable (or equivalent open API) by the caller.

2. Else the default advertised by the concrete Format Location::* subclass (JSON / XML / YAML / …).

3. Else error — core Tree does not invent a universal default.

Illustrative Format policies (flexibility; not all shipped in this cut):

Core Location::Tree itself supplies no stage defaults. Only Format subclasses may. Callers may always override a Format default by passing the argument explicitly.

Table name remains the TaOP/front-end name (and typically the on-disk basename under table-file-for); projection arguments are independent of the name unless the caller chooses to align them.

There is no separate Projection type (Q3 / §18). The opened Table is the projection of the document through the resolved selectors; useTable takes ordinary named arguments (Q3a: :table-selector, :table-name-selector, :row-selector, :header-selector, :field-projection, :transformer, :path-transformer, optional :table-write-path) plus Format-advertised defaults per stage (Q3b / Q19). The :transformer carries the molds used on flush/merge inverse (Q18). Write-path derivation for auto-creation is Q26.

9. Merge-on-flush

Tree-backed tables usually share a document with non-table structure (sibling keys, <thead>, captions, etc.). Flush MUST:

1. Obtain the current full document tree per Q5 / Q20a (Walkable: retained in-memory tree; .flush() / .rehydrate() / .check-fresh()).

2. Re-navigate with the same table-site and row-collection selectors.

3. Reconcile the in-memory @rows with the document’s selected row nodes using the caller-supplied :transformer (molds on that Transformer — Q18), i.e. the inverse of the hydrate field projection. Nodes outside the row-selector result (e.g. <thead> rows never selected) stay untouched unless Q11 says otherwise.

4. Format Render the whole document; Location Destination overwrite (existing ensure-parent / create semantics).

Empty data-row set writes an empty collection at the row-collection site (Q6), without deleting unselected sibling structure by default.

Creating a new file with can-create / ensure still allowed; in-document missing-node behaviour (table-site ancestors vs under-table path to rows) is §10 (Q7).

10. Destination operators and in-document path creation

(Q7 — likely behaviour; refine if DestinationMode names differ in code.)

Blob Location already ensures parent filesystem paths on write. This section is about missing nodes inside the document tree, split by which selector they sit under:

• Table action governs the table-selector site (the top node that selector names) and any ancestors needed to reach that site from the document origin.

• DestinationMode governs intermediate nodes between that table site and the row nodes (the path the row-selector walks under the table).

10.1. Table action → table-selector site / ancestors

10.2. DestinationMode → path from table site to rows

Unselected siblings (nodes outside the row-selector result) stay untouched on these writes unless DestinationMode is overwrite (Q11). Only ensure (table action) may auto-create ancestors / the table site; only ensure-present (DestinationMode) may auto-create under-table intermediates. Other faces require existing structure or create only the table site itself when its ancestors are already present (§10.1 create / can-create).

The intent: table actions decide whether the table node may appear; DestinationModes decide whether structure under that table toward the rows may be grown (ensure-*) or must already exist.

10.3. Auto-creation via EMPTY mold (path Transformer)

§10.1 / §10.2 decide whether missing nodes may be created. How a missing node is materialised should not be hard-coded hashes / arrays in core (KeyPath’s current leaf-seed / {} spine is an interim). Prefer the same mold/Transformer machinery as flush/merge (Q18), in the spirit of XSLT/Stratego construction and filesystem mkdir -p policy:

1. Path walk stays in Tree (e.g. KeyPath or equivalent). When a step is missing and §10 allows creation, Tree asks a path-creation Transformer for the new node.

2. That Transformer exposes a named mold EMPTY (parallel to the existing TOP convention for transform entry). The mold’s result is the node to insert at that step.

3. Input / context for EMPTY: the mold is invoked with the step name (path segment) as the transform source / topic, plus enough $*CONTEXT to distinguish roles — at least step, path-so-far, and whether this step is an ancestor, table-site, or row-collection leaf (so JSON can choose {} vs [], XML can name the element, etc.).

4. Format defaults: each Format Location::* (JSON, YAML, XML, later HTML) may advertise a default path-creation Transformer whose EMPTY mold builds sensible empty nodes for that Format. Callers may pass a different Transformer when useful.

5. Keep merge and path-creation separate in the API. Flush/merge still uses :transformer (Q18). Path auto-creation should take its own parameter (working name :path-transformer / empty-transformer) so merge molds and EMPTY do not have to share one object — though a single Transformer may define both if the caller wants that.

6. Policy stays outside EMPTY. Table action / DestinationMode gates still run in Tree; EMPTY never decides whether creation is allowed.

Analogues: filesystem ensure (mkdir -p vs plain mkdir) for §10 flags; XSLT / Stratego “build the node” for EMPTY; not XPath/Tree-sitter (query-only).

10.3.1. Write-path descriptor and derivation (Q26)

Location builds the :path descriptor passed to path-transformer($root, :path(...)) / TreeWithPath when a write needs in-document path auto-creation. Interim wire format remains what normalize-path already accepts (step-name list, list of %( step, role?, … ) hashes, or Hash with steps); present/missing and parent linkage are filled during the walk (Q27), not necessarily up front.

Who builds it: Location (not the caller’s Block). Preferred source:

1. Lower :table-selector when it is a query AST (OperatorBase / query-ast {…}) within write-path limits (below).

2. Else use optional :table-write-path (same path shapes; caller- supplied when the selector cannot be lowered).

3. Else, if a write requires path auto-creation → error. Reads / hydrate may still use a full :table-selector (Block or rich query).

Write-path axis limits (first cut): only single-step child (⪪) and parent (⪫) with a plain name selector. No descendant/ancestor, no filters/predicates, no non-name wildcards. Further axes later: Post-SQL §G / §F.12.4.

Step roles (for EMPTY / $*CONTEXT, not policy):

:row-selector for hydrate may still be a Block or richer query; only a lowerable AST (or an explicit path argument if added later) feeds under-table ensure steps (§10.2).

10.4. Path-constrained tree walk (TreeWithPath)

§10.3’s interim shape — Location/KeyPath walks the path and hand-invokes EMPTY — works, but bypasses the normal Transformer + Walker pipeline. TreeRewrite and ordinary document Transformers walk an existing tree; missing path steps have no node to match, so rewrite alone cannot create them.

Idea: still start from a tree node as the transform root (document or table-site origin — same as usual), but use a Walker that does not walk the whole tree. It only follows a specified path of steps under that root. Auto-creation is then “transform along those steps” rather than a special-case ensure loop.

10.4.1. Goal

When §10 allows creation of missing nodes, Tree supplies:

• a tree root (the node to walk from), and

• a path (ordered steps + roles / policy flags)

to the path-creation Transformer. Walker Qwiratry::Walker::Implementation::TreeWithPath yields step frames along that path only; molds (especially EMPTY) run through ordinary APPLY / $*CONTEXT, not an ad-hoc invoke-empty-mold.

10.4.2. Proposed pieces

1. Qwiratry::Walker::Implementation::TreeWithPathTree walker whose root is a tree node (Hash, XML element, …), as for other document walkers. Traversal is restricted to the supplied path: follow step₁ under root, then step₂ under that child, and so on — no sibling / whole-document fan-out. Step following reuses existing tree Navigators for “child named X” under each parent (Q23). Selected for path-constrained transform only (e.g. via :path / Q21), not Query AST planning (Q22).

2. Path descriptor (alongside the tree root)Carries at least:

   • ordered steps (keys / segment names) from the tree root,

   • per-step role (ancestor / table-site / row-collection),

   • whether each step is already present vs missing under the current parent,

   • enough parent linkage to attach a newly created node.

3. Step frame as mold topicEach visit exposes something molds can bind (topic and/or $*CONTEXT): step, path-so-far, role, parent, missing, … For a present step the topic may be the existing child node; for a missing step the topic/context still carries the step name so EMPTY can build a node. EMPTY’s result is the node to insert (same as §10.3).

4. Attachment after EMPTY (Q24 / Q28 / Q31)After the mold returns, the walker invokes attach during the walk under the current parent; how to attach lives on the tree Navigator (attach-child, write-side dual of tree-child-results). Strategy is not required for this cut. Associative face (JSON object keys; XML attributes) → key assign by step name; Positional face (JSON arrays; XML element children) → append. Treat attach as a subset of tree rewriting. Location does not wait for a post-transform step→node batch. Attribute-vs-child (or key-vs-index) disambiguation stays with Navigator / selector / Format; XML element names come from EMPTY, then append onto the parent’s children.

5. API (Q25 / Q21 / Q29)Keep :path-transformer separate from merge :transformer (§10.3). Invoke with the tree root as the positional input, the path as :path (Q21), and the table-path operational mode as :table-action (Q29), e.g. path-transformer($tree-root, :path($path), :table-action('use')). Presence of :path selects TreeWithPath. Success always returns the walk root; policy / mode violations throw (no Bool probe return). Callers may still pass a custom path Transformer; Formats keep default EMPTY molds.

6. Policy outside molds; modes on the walker (Q29)Location owns §10 policy: which table action / DestinationMode applies, when to invoke the path walk, and how exceptions surface. TreeWithPath implements :table-action mechanics for the table write path (ancestors + table-site — §10.1):

   :table-action	Behaviour	Success	Failure
   use	Probe only — follow path, do not create	return $root if every step present	throw if any step missing
   ensure	Create missing steps via EMPTY / attach	return $root (mutated)	throw on hard failures (no mold, unsupported parent, …)
   create	Per §10.1 (site must be absent; ancestors present — no auto-ancestors)	return $root	throw when existence rules violated
   can-create	Per §10.1	return $root	throw when policy violated

   Under-table row-collection ensure remains gated by DestinationMode (§10.2), not table action — Location may run a separate path walk (or segment) for that. EMPTY never decides policy. Strategy does not participate in policy for this cut.

10.4.3. Why not full-document TreeRewrite alone?

Ordinary TreeRewrite matches and replaces existing nodes; a missing key has no node to replace. Path creation still mutates the live tree during the walk (Q24): attach EMPTY’s result under the parent — a narrow, path-ordered form of tree rewriting, not a whole-document rewrite and not a post-pass Location merge of step→node pairs. TreeWithPath makes “visit these steps under this root” first-class and keeps EMPTY as ordinary mold application.

10.4.4. Relation to interim §10.3 / checklist §16.4

• Landed (§16.4.1 / Post-SQL §F.7.5): TreeWithPath + Transformer :path / TRANSFORM-PATH; Navigator step follow; Associative attach on missing steps; unit coverage in t/unit/tree-with-path.rakutest. No hand-rolled KeyPath.ensure loop was required for this cut.

• Landed (§16.4.2 / Post-SQL §F.7.6): core $.path-transformer, useTable / projection-param wiring, associative-path-transformer, ensure-document-path (t/unit/path-transformer.rakutest).

• Landed (§16.4.3 / Post-SQL §F.7.7): Format default EMPTY / :path-transformer on JSON / YAML / XML Location subclasses.

• Landed (§16.4.4 / Post-SQL §F.7.8): derive-write-path, :table-write-path, :table-action on path walk, Positional/XML attach; flush invokes path ensure when derivable.

• Landed (§16.4.5 / Post-SQL §F.7.9): merge Transformer molds see :context / $*CONTEXT<rows>; Format JSON/YAML default molds.

• Landed (§16.4.6 / Post-SQL §F.7.10): multi-site :table-name-selector — candidate-list origin, List/slip filters, useTable hydrate (t/unit/table-name-selector.rakutest).

• Landed (§16.4.7 / Post-SQL §F.7.11): .check-fresh / .rehydrate / .flush with :throw-on-conflict (Q20); Tree mutations mark dirty and do not auto-flush (t/unit/tree-sync-api.rakutest).

• Landed (§16.4.8 / Post-SQL §F.7.12): Format JSON / YAML / XML Location + conformance suites re-run; Location tests call explicit .flush after mutations (no Tree auto-flush).

• Next: §16.5 HTML (optional later).

• Related checklist items still unchecked: §16.5 HTML.

• JSON/YAML Associative keys first; XML step→element naming via the same EMPTY context later.

• Path-walk design Qs for this cut are closed (Q21–Q31, §18).

11. Shape contracts per stage

• Table site: normally exactly one node. Unexpected shapes at any projection stage error in general (Q2). If the table-selector returns multiple sites, an optional table-name selector may disambiguate (e.g. tables tagged with names); without it, multiple hits still error (Q2 / Q12).

• Row collection: ordered list of row nodes from evaluating the :row-selector Block / Callable (Q8 / Q16b).

• Field projection: each row node → row data and a column-name source (hash-like keys vs array-like / header / explicit list), applied by the backend Tree table on hydrate (Q10).

Non-conforming stage outputs fail closed with a clear error at open/load (and at flush if the site vanished or changed type).

12. Cross-Format sameness

JSON, XML, YAML, HTML (and similar tree Formats) share Location::Tree. Differences belong in:

• Format Parse/Render (text ↔ Mu tree),

• Tree::Navigator specialisations for Format-specific node types,

• optional thin Location::* subclasses that only default :format / extension,

• optional named helpers for common row-selectors / field projections built from shared navigation primitives (JSON object identity, HTML tbody/tr, …).

No per-Format fork of the pipeline algorithm or a second selector language.

13. Migration of current Location::JSON

1. Add Location::Tree in core raku-Qwiratry (lib/Qwiratry/DatabaseStorage/Location/Tree.rakumod); put shared projection + merge + §8 resolution there — not under Format-JSON.

2. In Format-JSON, rebase Location::JSON onto core Location::Tree as a thin subclass.

3. Move today’s root array/object → rows behaviour into JSON Format defaults for the three stages (so existing useTable(name => …) demos keep working without new arguments).

4. Allow caller overrides for nested tables (e.g. table-selector under a key) and document that path.

5. Update Format-JSON t/03-location-json.rakutest and pod examples (default path + override path).

6. Note in core and Format-JSON Changelogs.

HTML remains out of scope for this cut; when a Format-HTML dist exists it can subclass Tree and supply sensible defaults per §8 without changing core.

14. Migration of Location::XML (raku-Qwiratry-Format-XML)

The Format dist already exists (Format::XML, Tree::Navigator::XML). This section is only about adding tree-table Location support on top of it (after core Location::Tree lands — not required to block the JSON rebase).

1. Add thin Qwiratry::DatabaseStorage::Location::XML in raku-Qwiratry-Format-XML, subclassing core Location::Tree; default :format / extension only as needed.

2. Projection defaults: none (blank). Omitting any unresolved stage at useTable must error; callers always pass table-site, row, and field-projection for XML tables until/unless this policy is revisited.

3. Reuse existing XML Format Parse/Render and Tree::Navigator::XML; do not invent an XML-only path language.

4. Tests: open without projection args → error; open with full caller projection → load/flush/merge; document that XML does not guess row/table shape.

5. Changelog + Extending notes in Format-XML; cross-link this plan §8 / §14.

15. Migration of Location::YAML (raku-Qwiratry-Format-YAML)

The Format dist already exists (Format::YAML). This section is only about adding tree-table Location support on top of it (after core Location::Tree lands — not required to block the JSON rebase).

1. Add thin Qwiratry::DatabaseStorage::Location::YAML in raku-Qwiratry-Format-YAML, subclassing core Location::Tree; default :format / extension only as needed.

2. Projection defaults: same as JSON (Q13) — root site; identity rows; object/mapping keys → fields — so YAML documents that already look like JSON tables open with useTable(name => …) alone.

3. Same Tree pipeline and §8 resolution as JSON/XML; YAML-specific work is wiring Format Parse/Render (+ navigator if needed).

4. Tests: omit-args works (JSON-like defaults); caller-override nested paths; flush/merge round-trip.

5. Changelog + Extending notes in Format-YAML; cross-link this plan §8 / §15.

16. Implementation checklist

16.1. MVP / JSON

• 1.1. Resolve any remaining open Q* items; record under §18 (Q14–Q20 closed).

• 1.2. Add Qwiratry::DatabaseStorage::Location::Tree in core raku-Qwiratry (lib/Qwiratry/DatabaseStorage/Location/Tree.rakumod) with the three-stage pipeline, §8 resolution (caller → Format default → error), tree-navigation evaluation, merge flush — do not place this class in Format-JSON.

• 1.3. In Format-JSON, rebase Qwiratry::DatabaseStorage::Location::JSON onto core Tree; advertise today’s root/identity/object-key behaviour as Format defaults.

• 1.4. Wire useTable / Table open path so any stage still unresolved after §8 fails clearly.

• 1.5. Tests: JSON omit-args (defaults); JSON nested override; unresolved stage error path (e.g. mock / XML-style blank defaults); flush preserves unselected structure.

• 1.6. Docs: this plan Decisions; Basics §4.2 cross-link if still stale; Post-SQL §F.7; Format-JSON examples/Changelog.

16.2. YAML

• 2.1. Add Qwiratry::DatabaseStorage::Location::YAML to existing raku-Qwiratry-Format-YAML (§15; JSON-like defaults per Q13).

16.3. XML

• 3.1. Add Qwiratry::DatabaseStorage::Location::XML to existing raku-Qwiratry-Format-XML (§14, blank defaults).

16.4. Things that should've been in 16.1

Tracked under Post-SQL §F.7 follow-ons as well (see that file for the matching F.7 item numbers):

• 4.1. Path-constrained tree walk (§10.4): Qwiratry::Walker::Implementation::TreeWithPath; tree node remains the walk root; invoke via :path / TRANSFORM-PATH (Q21); transform-only (Q22); step following via Navigators (Q23); missing-step frames → ordinary APPLY; Associative attach under parents (Q24); no interim KeyPath.ensure loop (t/unit/tree-with-path.rakutest; Post-SQL §F.7.5).

• 4.2. Path auto-creation — core (§10.3 / Q25): Location::Tree $.path-transformer attribute; projection-param / useTable wiring; associative-path-transformer (EMPTY-named mold → {} / [] by role); ensure-document-path invokes Transformer :path. §10 policy unchanged (t/unit/path-transformer.rakutest; Post-SQL §F.7.6).

• 4.3. Path auto-creation — Format / other dists (§10.3): JSON / YAML default associative-path-transformer; XML EMPTY → XML::Element named from the path step; Format location tests cover defaults (Post-SQL §F.7.7). No Test-dist helper needed for this cut.

• 4.4. In-document missing-node path creation (§10 / Q7 / Q26– Q29 / Q31): derive-write-path / :table-write-path; TreeWithPath :table-action + Positional/XML append attach; flush wires path ensure when a write path is available (Post-SQL §F.7.8). No separate KeyPath class.

• 4.5. Flush/merge via Transformer molds (Q18): transform forwards :context to inline APPLY; JSON / YAML §8 default :transformer is a mold over $*CONTEXT<rows> (Callable still accepted) (Post-SQL §F.7.9).

• 4.6. Multi-site :table-name-selector end-to-end (Q2 / Q12 / Q15): select-table-root disambiguates candidate lists; errors when still ambiguous / empty; project-to-rows / useTable coverage in t/unit/table-name-selector.rakutest (Post-SQL §F.7.10).

• 4.7. Sync API polish / coverage: .check-fresh / .rehydrate / .flush with :throw-on-conflict (Q20); Tree mutations mark dirty and no longer auto-flush (t/unit/tree-sync-api.rakutest) (Post-SQL §F.7.11).

• 4.8. Re-run tests on JSON, YAML, and XML to ensure they all still pass; Location suites updated for explicit .flush after Tree mutations (Post-SQL §F.7.12).

16.5. HTML

• 5.1. Optional later: HTML Format + Qwiratry::DatabaseStorage::Location::HTML with sensible defaults — not part of the JSON / YAML / XML cut (Post-SQL §F.7.13).

17. Open Questions

Answered items move to §18; IDs stay stable. Q1–Q31 are decided (§18). No open items remain for this plan’s current cut.

18. Decisions

Answered question IDs stay stable; they live only here (not also under §17).

• Q1. Alternate navigation origins (other than the Format parse root) are allowed only if that origin is reversible — the same path can be used to read and to write/merge. If an origin would be read-only (or otherwise not round-trippable through Tree flush), do not offer it; table-site navigation then starts at the parse root only.

• Q2. Unexpected projection-stage shapes error in general. Special case: if the table-selector returns multiple table sites, allow an optional table-name selector so name-tagged tables can be chosen by name; if that optional selector is omitted (or still ambiguous), error (refines Q12).

• Q3. Do not introduce a separate Projection class or :projection bundle object. The Table from useTable is the projection.

• Q3a. useTable projection argument names are :table-selector, :row-selector, :field-projection, optional :table-name-selector (Q15), optional :header-selector (Q17), :transformer (Q18 — the Transformer that holds the molds for flush/merge inverse), :path-transformer (Q25 — path auto-creation / EMPTY, separate from merge), and optional :table-write-path (Q26 — explicit write path when the selector cannot be lowered).

• Q3b. No separate “defaults advertising” API. Format Location::* subclasses implement their §8 defaults in ordinary code; document those defaults in the Format dist (and this plan’s §8 table). Core still resolves caller → Format default → error.

• Q4 / §8. Callers need not always pass root/identity for JSON. The three projection stages allow silent Format defaults. Resolution is caller argument → Format Location::* default → error. Core Qwiratry::DatabaseStorage::Location::Tree has no universal defaults. Qwiratry::DatabaseStorage::Location::JSON keeps today’s root array/object behaviour as defaults; Qwiratry::DatabaseStorage::Location::XML ships blank defaults (omit → error); Qwiratry::DatabaseStorage::Location::HTML may later ship sensible defaults (not this cut); Qwiratry::DatabaseStorage::Location::YAML matches JSON (Q13).

• Q5. Flush coherence depends on I/O face:

  • Walkable (whole-document) load: keep the parsed document tree in memory across mutations; use .flush() / .rehydrate() / .check-fresh() (Q20a / Q20b).

  • Push/pull streaming faces: re-read and re-parse when rehydrating (no long-lived whole-document tree assumed).

• Q6. Empty data-row flush writes an empty collection at the row-collection site (does not delete the site / unselected siblings by default). Disk↔memory sync is .flush() / .rehydrate() / .check-fresh() (Q20a / Q20b), not a single .refresh().

• Q7. In-document missing-node behaviour is split — see §10: table actions govern the table-selector site / ancestors (§10.1; only ensure auto-creates ancestors); DestinationModes govern intermediates between table site and rows (§10.2; insert/overwrite/update/delete require existing path; ensure-present / ensure-absent as written there).

• Q8. :row-selector is a Block / Callable from the table root (Q16a / Q16b). It may use ordinary Raku and tree navigation inside the Block; there is no separate Location filter language and no query-ast requirement for Tree stages.

• Q9. Do not add a row-filter (or row-acceptance) projection stage. Choosing which nodes are rows is either:

  1. the row-selector (tree navigation / match from the table site), or

  2. ordinary query operators on the loaded table (§5.2). A third Location-only filter would duplicate both.

• Q10. Database.useTable creates the front-end Table and the backend storage table; under Tree, that backend table does the Tree→Table hydrate. Column naming / value shaping is via :field-projection and optional :header-selector (Q14 / Q17).

• Q11. Nodes outside the row-selector result stay untouched on flush, unless DestinationMode is overwrite (which replaces the selected collection’s contents — still not a license to rewrite unrelated siblings outside that collection unless overwrite’s replace semantics imply it for that site alone).

• Q12. Multiple table-site hits: error unless disambiguated via the optional table-name selector in Q2.

• Q13. Qwiratry::DatabaseStorage::Location::YAML projection defaults are the same as JSON (root site; identity rows; mapping/object keys → fields).

• Q14. Terminology: table root = node selected by :table-selector (a Block/query that returns the table root). :row-selector selects rows relative to that table root. :field-projection selects fields from each row and assigns keys (hash mode; see Q17 for header/array mode). For JSON/YAML omit-args defaults, the document parse root is the table root (and the default row/field Blocks match today’s root array/object → hash rows behaviour). Concrete default values are those Blocks/queries — not empty Blocks or Format-private sentinels.

• Q15. Optional table-name selector parameter is :table-name-selector, and like the other selectors it is a Block / Callable (Q16b).

• Q16a. On hydrate, the query starting point (origin) for :row-selector is fixed to the table root. The selection may still leave that site via axes such as ancestor and walk other parts of the document.

• Q16b. Projection selectors are Blocks / Callables (evaluate at hydrate with the document / table-root origin). Do not prefer query-ast {…} for Tree Location stages.

• Q17. Column naming: optional :header-selector is the extra cue beside :field-projection. If present, run it before :row-selector; both return arrays spliced into column→value hashes (header → keys; row arrays → values). If absent, :field-projection returns hashes (column → value) already — that fully specifies hash-like naming with no further cue.

• Q18. Flush/merge inverse of field projection is done with a Qwiratry Transformer whose molds perform the rewrite (not an ad-hoc Location-only rewriter). The caller passes in that Transformer (molds live on it) via :transformer / constructor; Format subclasses may supply a §8 default so JSON omit-args still round-trips. Tree does not invent molds behind the caller’s back.

• Q19. §8 defaults are attributes on the Tree / Format subclass, filled by Format subclasses for their usual defaults, and overridable by constructor / useTable parameters (caller wins per §8).

• Q20a. Disk sync API on Tree (abstract) / concrete backend tables:

  • .flush() — write memory → storage; returns True if anything changed.

  • .rehydrate() — load storage → memory; returns True if anything changed.

  • .check-fresh() — compare which of memory vs storage is more recent (does not itself load/write). Implementations live on the concrete classes.

• Q20b. Stale / conflict handling: both .flush() and .rehydrate() take optional :throw-on-conflict (default True). When that flag is true, the method must run the stale / freshness check (same logic as .check-fresh()) before writing/loading, and throw on conflict. When false, skip the throw (proceed even if storage and memory disagree). .check-fresh() remains the standalone compare API (no load/write).

• Q21. Path descriptor is a named transform option :path alongside the positional tree-root input: path-transformer($tree-root, :path($path)) (same via .transform(...)). CALL-ME / transform should route :path like :context / :streaming / :mode. Presence of :path selects TreeWithPath and the path-constrained walk; creation still happens only via ordinary mold APPLY (e.g. EMPTY) plus walk-time attach (Q24), not magic inside every Transformer. Policy (§10.1 / §10.2) remains outside molds.

• Q22. TreeWithPath is transform-only for the first cut: used when path auto-creation runs (:path / Q21), not wired into Query AST planning / query walkers.

• Q23. Path step following reuses existing tree Navigators for “child named X” (and format-specific equivalents) under each parent — same child-lookup rules as ordinary document query / tree walk; do not specialise private step logic inside TreeWithPath when a Navigator already covers the parent node type.

• Q24. After EMPTY returns a node for a missing path step, the walker attaches it during the walk under the current parent (via Navigator attach-child — Q28). This is a subset of tree rewriting (grow a missing child), not a post-transform Location batch of step→node pairs (§10.4). Strategy not required for this cut.

• Q25. Path-creation Transformer parameter name is fixed as :path-transformer (separate from merge :transformer / Q18).

• Q26. Write-path descriptor / derivation (§10.3.1):

  • Shape: Location builds a :path value in the interim TreeWithPath.normalize-path forms (step names and/or step hashes with roles); walk fills present/missing / parent (Q27).

  • Builder: Location. Prefer lower :table-selector when it is query AST within limits; else optional :table-write-path; else error on writes that need path auto-creation (hydrate may still use a Block / rich selector).

  • Limits (first cut): single-step child / parent only; plain name selectors. Expand later per Post-SQL §G / §F.12.4.

  • Roles: table-site = last table-path step; ancestor = preceding table-path steps; row-collection = last step of a writable :row-selector path when available, else same as table-site. EMPTY consumes roles via $*CONTEXT (Q27 / Q30); policy stays outside molds.

• Q27. Path-walk mold topic / context (TRANSFORM-PATH):

  • Missing steps: run ordinary APPLY; topic is the step frame Hash; $*CONTEXT is that frame (optionally merged with caller :context). Frame fields include at least step, role, parent, missing, path-so-far (and node when present).

  • Present steps: follow via Navigator only — do not run molds.

  • APPLY accepts / forwards :context into Mold.execute.

• Q28. Attach of an EMPTY (or other mold) result for a missing step is invoked by the walker during the walk (Q24 — TreeWithPath.attach-child delegates). The Navigator owns the attach implementation (Navigator::Base.attach-child, Format overrides such as XML element append) — write-side dual of tree-child-results / the same lookup used by follow-child. Not Tree::Replace.replace-node. Strategy not required for this cut. How Associative vs Positional faces attach → Q31.

• Q29. §10 policy vs path-walk mechanics (§10.4.2):

  • Location owns policy (which table action / DestinationMode applies, when to invoke the walk, how errors surface).

  • TreeWithPath takes :table-action and implements table-path mechanics: use = probe only (no create; throw if any step missing; return $root if complete); ensure / create / can-create per §10.1 (create or existence checks; success returns $root; violations throw). One success return type — the walk root — not Bool.

  • DestinationMode gates under-table steps (§10.2) separately.

  • EMPTY and Strategy do not decide policy.

• Q30. Path-creation molds are selected by mold name EMPTY, parallel to TOP. There may be multiple EMPTY molds on the same Transformer, distinguished by different when clauses (e.g. by role or Format-specific shape). Ordinary mold ordering / first-match still applies among those name peers.

• Q31. Navigator attach-child by parent face (§10.4.2 / Q28):

  • Associative (JSON object keys; XML attributes) → $parent{$step} = $child (key assign by step name) — Navigator::Base.

  • Positional (JSON arrays) → push; XML element children → append (Navigator::XML override). XML needs no third attach API: attributes vs children are the two faces above. Navigator / selector / Format choose the face for a step; XML element names come from EMPTY, then Positional-face append. On a pure Positional JSON array, the step name may be unused at attach time (push/append only).

19. References

• Qwiratry-TaOP-Basics.md §4.1–§4.2

• Post-SQL-TaOP-Improvements.md §F.7; write-path axis expansion later §G / §F.12.4

• Code (core raku-Qwiratry): lib/Qwiratry/DatabaseStorage/Location.rakumod, lib/Qwiratry/DatabaseStorage/Location/Tree.rakumod (:path-transformer / §16.4.2), lib/Qwiratry/Walker/Implementation/TreeWithPath.rakumod (§16.4.1), lib/Qwiratry/Transformer.rakumod (:path / TRANSFORM-PATH), lib/Qwiratry/Tree/Navigator*.rakumod, lib/Qwiratry/Query/Evaluator/Navigation.rakumod, lib/Qwiratry/Query/Selector.rakumod, t/unit/tree-with-path.rakutest, t/unit/path-transformer.rakutest

• Format-JSON (thin subclass only): lib/Qwiratry/DatabaseStorage/Location/JSON.rakumod, t/03-location-json.rakutest