Post-SQL TaOP Improvements

Follow-on work after Aggregate Functions, Textish Formats, TaOP Basics, SQL-Features deferrals, TaOP-Plan leftovers, and related SQL / TaOP landing. Items here were deferred from those implementation cuts.

A. From Qwiratry-AggregateFunctions-Plan.md

Moved from that plan’s former §K.2–§K.5. Cross-links to Aggregate Functions sections (e.g. §J.4, §A.6, §I, §H) still mean that document.

A.1. Low-hanging Aggregate Functions

(was §K.2)

Catalogue entries that fit the existing AggregateFunction / MovingFrameAggregateFunction + aggsub surface: ordinary row / window-* / optional frame-*, extractors in @.parameters, no FILTER / DISTINCT, no ordered-set / hypothetical-set syntax, no COMBINEFUNC, no whole-tree rewrite. Skip nulls like Postgres unless noted. Already shipped: Sum, Avg, Count (COUNT(*)), Rank, DenseRank.

A.1.1. General-purpose folds (aggreduce / broadcast in aggmap)

(was §K.2.1)

• Count over an expression (COUNT(expr) — count non-null)

• Min / Max

• AnyValue (any_value)

• BoolAnd / BoolOr (and Every as alias of BoolAnd)

• BitAnd / BitOr / BitXor

• StringAgg / ArrayAgg / XmlAgg — concatenation in input / sort order only; in-aggregate ORDER BY is not this list

• JsonAgg / JsonbAgg / JsonObjectAgg (and strict variants that skip nulls) — same ordering caveat as above

A.1.2. Statistics (one or two Numeric extractors; finalize in CALL-ME / RESULT)

(was §K.2.2)

• VarPop / VarSamp / StddevPop / StddevSamp (and aliases Variance / Stddev)

• Corr / CovarPop / CovarSamp

• RegrCount / RegrAvgx / RegrAvgy / RegrSxx / RegrSyy / RegrSxy / RegrSlope / RegrIntercept / RegrR2

A.1.3. Window-oriented (aggmap; Rank-style or framed)

(was §K.2.3)

• RowNumber

• Lag / Lead (use Driver $.previous / $.next or the window buffer)

• FirstValue / LastValue / NthValue (need frame)

• Ntile / PercentRank / CumeDist (need window cardinality + counter)

A.1.4. Out of this bucket

(was §K.2.4)

Need other work: FILTER (Aggregate Functions Plan §J.4.1) / DISTINCT (§A.6) forms; mode / percentile_* / hypothetical-set; in-aggregate ORDER BY; SORTOP-only planner wins; SSPACE / MSSPACE / FINALFUNC_MODIFY (and related CREATE AGGREGATE metadata); range / multirange aggregates unless Raku grows matching types. Parallel / combine hooks: §A.2 below.

A.2. Parallel Processing of Queries

(was §K.3)

Postgres partial / parallel aggregation: workers fold their own chunk, then merge states. Map the CREATE AGGREGATE hooks onto Qwiratry when (and if) the Driver or a SQL Location runs aggregates in parallel:

• COMBINEFUNC — merge two transition states of the same aggregate into one (required for partial aggregation). Authors of associative aggregates (Sum, Count) can supply this; non-associative ones stay sequential.

• SERIALFUNC / DESERIALFUNC — turn opaque / internal-like state into bytes (or a transferable Raku value) and back, so state can cross threads, processes, or remote shards. Needed when state is not a plain shareable Raku object.

• PARALLEL — today $.parallel is stub-only (Aggregate Functions Plan §I). Use it as metadata (safe / restricted / unsafe) for pushdown and for whether the in-Raku Driver may split a window. Wire it when combine (and serialize, if needed) exist — the attribute alone does nothing.

• Prefer starting with in-process chunking + COMBINEFUNC for catalogue aggregates that are naturally associative; add serialize/deserialize when state must leave the process (SQL workers, distributed Locations).

A.3. TaOP Relation mutation APIs

(was §K.4)

Not sure we'll do this, but maybe.

Aggregate work today is pipeline-shaped: aggreduce / aggmap (and ⌿ / ⇶) consume an item stream and emit results. They do not mutate a live TaOP Relation / Table in place. Later, if TaOP wants OOP-style table methods:

• Decide whether mutation means replace-columns / replace-relation, or only convenience wrappers that still return a new Relation.

• Keep a single evaluation path: methods should call aggreduce / aggmap (or the operators), not a parallel Driver.

• Align naming and semantics with Qwiratry-TaOP-Plan.md / Basics (row sinks, FieldMode) so aggregates do not invent a second mutation story.

• Do not block Aggregate Functions catalogue or rewrite work on this — it is a TaOP front-end concern once Relations are stable.

A.4. Other later items

(was §K.5)

• Indirect / computed callees (my $c = Sum; $c.({.amount})).

• Split the transform out of AggregateFunctions.rakumod if the file grows unwieldy.

• Nested aggregates inside extractor Blocks (body nesting is Aggregate Functions Plan §J.4; extractors are still not scanned).

B. From Qwiratry-TaOP-Textish.md

Textish §9 steps 1–13 are complete. Remaining follow-ups from that plan (§B.1 / §B.5 done under Parity §F.1):

B.1. Consider: public multi-phase formatter API (replicates TOP)

Done (Post-SQL §F.1.1 / Qwiratry-TaOP-Parity.md).

TOP formatters expose prepare-table / output-header / output-row / output-footer and accumulate $.output; Database::Storage.format drives that sequence. Qwiratry exposes the same phase methods on Qwiratry::Format::Textish::RenderWalkable with one-shot render-walkable as a thin driver. Pull/streaming remains the better large-table path long term (§B.2 / §F.8).

B.2. Optional streaming (§9.14 / §7.5)

ParsePull / RenderPull for HalfHuman and WithBorders when the Streaming proposal makes pull useful; today both Formats are walkable-only.

B.3. Richer Associative cell stringify (§6.5)

Something better than .raku for nested Associatives in Textish render (shared !cell-string); not a §9.1–13 blocker.

B.4. Other textish Formats

HTML (tree-table Location — §I), Spreadsheet / .ods, Pod6, and richer JSON (beyond Format-JSON). See TaOP Plan §5.4; not part of Textish v1.

B.5. Stronger HalfHuman round-trip

Done for emit-side space → _ (Post-SQL §F.1.2 / Parity Q3): HalfHuman Render option :spaces-to-underscores (default on). Still does not paper over Parse option misuse (e.g. headers without :headers). Further dialect round-trip hardening stays optional.

B.6. Body-cell / per-cell formatting

(Textish §8.2; was Post-SQL F.1.3; backlog §F.8.1)

Today :header-alignment aligns header cells only; body keeps Int-right / else-left. Out of scope for Textish v1.

Old TOP has no :header-alignment / :body-alignment options (body = Int-right / else-left; WithBorders has a TODO to centre headers). Qwiratry already went beyond TOP with header-only :header-alignment (defaults: HalfHuman left-int-right; WithBorders centre). Not TOP parity.

Short-term shape (choose one): (a) leave header-only as today; (b) extend :header-alignment to body too; (c) add a separate :body-alignment (or wait for per-cell).

Long-term: each cell may carry its own formatting when useful; Formats must still format raw values when none is provided.

Notify Tim on any golden string changes from this work.

B.7. Consider: SourceOperator returning an Iterator

Today evaluate always uses read-walkable (whole payload: typically Str / path shapes) and does not return an Iterator. Pull is composed by ParseOperator when both sides support it. Revisit only if that split proves awkward; prerequisites already rejected bare Source → Iterator for now.

C. From Qwiratry-TaOP-Basics.md

Basics first-cut work is done (Memory, Destination, Location::JSON, CSVdemo, Database-as-LHS, Format-CSV / DatabaseStorage::CSV, Textish Formats). Tree-document tables (§F.7): core Location::Tree + Format-JSON / Format-YAML / Format-XML Location::* are implemented (DatabaseStorage-Location-Tree.md §16.1–§16.3). Remaining from Basics (excluding items already shipped whose §7 text is stale):

C.1. WalkCursor backends (Basics §6)

• SQL DatabaseStorage cursor mapping: FETCH / MOVE onto fetch-* / move-*.

• Metanoia Walkable-as-cursor-factory (deferred in Cursor proposal).

• Mapping TaOP’s Cursor::Storage::Postgres onto WalkCursor (0-based Raku vs 1-based SQL adaptors).

C.2. Sibling / later (Basics §7 step 7, §4.5)

• Qwiratry::DatabaseStorage::SQL in raku-Qwiratry-DatabaseStorage-SQL (DBIish + SQL::Abstract).

• More Format pull; full Format Supply push (push is a Basics Non-goal / deferred).

C.3. Planned front-end (Basics Goals / §3.2 / DQ1)

• View class that does Relation (Table already does).

• Query output as a Relation without being a Table (DQ1 “MAY”).

C.4. Low priority

• DQ10 — QueryIterator.of advertising Tuple (default Mu is fine for now).

D. From SQL-Features-Plan.md

Moved from that plan’s deferred language follow-ups: §B.3 / §C (paging and transactions), plus Destination deferrals from §A.2 / §A.4.5. Section A (File/HTTP Destination) and §B.2 (aggregates) stay in SQL-Features / Aggregate Functions plans.

D.1. Paging

(was SQL-Features §B.3)

Can probably just be done with array element ranges.

D.2. Transactions / Atoms

(was SQL-Features §C / §C.1)

Suitable model: TaOP's planned atom control-flow form (TOP::ControlFlow) — a block that binds KEEP / UNDO to commit / rollback on one or more transaction-capable handles.

D.2.1. Language shape (sketch)

(was §C.1.1)

atom $location [, $location2, …] {
	# destination / pipeline writes against those locations
}

• atom is ordinary block control flow, not a Destination glyph.

• Arguments are Location (or later Relation / Database) handles that advertise a transaction capability.

• On normal exit: KEEP → commit each listed handle.

• On failure / unwind: UNDO → rollback each listed handle.

• Writes outside an atom stay in direct mode (no transaction), matching TaOP's note for database actions outside atom.

D.2.2. Fit with Destination

(was §C.1.2)

• Destination operators inside atom participate in the open transaction(s) when the target Location supports it.

• File (and most blob HTTP) typically lack multi-op transactions; atom on a non-capable Location should fail clearly (capability / LocationError), not pretend atomicity.

• Full usefulness lands with Table / database Locations (later); this section still defines the language so Destination CrUD and TaOP Relation share one story.

D.2.3. Deferral notes

(was §C.1.3)

• Specify atom in the language / slang plan; do not require implementing transactional File or HTTP in the Destination operator cut (SQL-Features §A.4.5).

• Nested atom, savepoints, and multi-Location two-phase commit: deferred.

D.3. Table / Schema Destination backends

(was SQL-Features §A.4.5)

§A.4.3 maps Destination operators onto Table (row sinks) and Schema (DDL). Language meaning is specified; only File and HTTP backends shipped in the Destination cut. Implement table row sinks and schema DDL against those columns when Table / Database Locations are ready.

D.4. RETURNING-style Destination results

(was SQL-Features §A.4.5)

Today Destination evaluation is payload pass-through: the write happens, then the upstream payload continues down the pipeline. Later: optional RETURNING-shaped results from Destination operators (language + backends that can supply them).

D.5. Row locking

(was SQL-Features §A.2 Non-goal)

Row locking (FOR UPDATE, …) deferred from the first Destination / language cut. Revisit when SQL / Table Locations need lock clauses on reads.

E. From Qwiratry-TaOP-Plan.md

Moved from that plan’s remaining Immediate Next Steps (§11 items 10–11), §12 Future plans, and other still-open design notes (§5.4–5.5, §7–§9, §10.4). Steps 1–9 of §11 are done (or partial with notes there).

E.1. Immediate next steps (remaining)

(was TaOP-Plan §11 items 10–11)

• E.1.1. Pull the Qwiratry-specific parts of Implementation::Loader (and possibly Singleton) that were imported into Qwiratry::Implementation::Loader back out into a separate module — keep the existing Qwiratry::Implementation::Loader modules as well; do not move the whole thing.

• E.1.2. raku-Qwiratry-DatabaseStorage-SQL with DBIish + SQL::Abstract.

E.2. Future plans

(was TaOP-Plan §12)

Ideas that should not block the first TOP-on-Qwiratry slice (Memory + HalfHuman

• Postgres-only SQL).

E.2.1. Red-style multi-dialect SQL encoding

(was §12.1)

When we support more than Postgres (SQLite, MySQL, …), follow Red ORM’s architecture: keep a driver-neutral AST of the query intent; each DB Location/driver translates that AST to its dialect and runs it (Red: ResultSeq → Red::AST → Red::Driver.translate).

Qwiratry hook (likely Walker / Plan for table+SQL origins): in-memory Walkers execute the Query AST directly; SQL Location drivers translate. Shared SQL encoding can stay near core; dialect quirks stay in Location packages. Until then, Postgres may emit Postgres SQL directly.

E.2.2. Other later items (already noted elsewhere)

(was §12.2)

• Additional DBIish Locations (SQLite, MySQL) — Accept / when someone needs them.

• Spreadsheet / .ods, HTML, richer tree formats — TaOP matrix “Make/Accept” (also §E.3.1 / §B.4).

• Full streaming adoption per Streaming-Qwiratry-Proposal.rakudoc.

• Predicate pushdown once multi-dialect translate exists.

E.2.3. SQL Features

(was §12.3)

SQL-Features Plan §B.3 / §C (and related Destination deferrals) now live in Post-SQL §D. Aggregate Functions absorption is done; leftovers in Post-SQL §A. Spec archive: old-design-docs/Qwiratry-AggregateFunctions-Plan.md, old-design-docs/SQL-Features-Plan.md.

E.3. Other still-open notes from the plan

E.3.1. Later formats (accept / make)

(was TaOP-Plan §5.4)

HTML, Spreadsheet / .ods, Pod6, richer JSON. Consume existing raku-Qwiratry-Format-JSON rather than reimplement; tree-shaped formats use Navigator. Overlaps Textish follow-ups in §B.

E.3.2. Public multi-phase Format API

(was TaOP-Plan §5.5)

Done for Textish (§B.1 / §F.1.1). TOP’s prepare-table / output-header / output-row / output-footer vs Qwiratry one-shot render: one-shot drives the phases on Textish::RenderWalkable.

E.3.3. Deferred TOP front-end concepts

(was TaOP-Plan §7)

View, TupleSet, Section, Lot, DataDictionary — design hooks only until Table + DatabaseStorage solidified. View / query-as-Relation also in §C.3.

E.3.4. SQL WalkCursor / Iterators

(was TaOP-Plan §8)

SQL DatabaseStorage should expose cursors as WalkCursor / Iterators (dialects via DBIish + SQL::Abstract). Depends on §E.1.2; cursor mapping detail also in §C.1.

E.3.5. Migrate / retire Raku-TOP

(was TaOP-Plan §9 sequencing step 7)

Migrate TaOP tests / demos; retire or thin-wrap old Raku-TOP once Qwiratry siblings cover the same ground.

E.3.6. FieldMode Overflow on SQL → JSON/JSONB

(was TaOP-Plan §10.4)

Support Automatic, Error, and Overflow as far as each backend allows; do not silently degrade Overflow on swap. Needs §E.1.2.

F. Suggested Implementation Order

Numbered checklist of work from this document. Priority order:

1. TOP parity — behaviour the old TOP code already had that Qwiratry still lacks (not greenfield TOP concepts).

2. Core hygiene — independent cleanups that do not need SQL (postponed; see archived spec).

3. SQL backend (MVP) — Postgres-shaped DatabaseStorage and related TOP Storage behaviour, plus multi-dialect follow-through.

4. TOP parity finalisation — migrate / retire Raku-TOP once parity + SQL MVP land.

5. New TOP features — greenfield TOP-shaped work in Qwiratry (after SQL MVP).

6. New SQL features — DB-backend extras beyond the SQL MVP.

7. Tree-document tables — Location::Tree + Format Location::* (JSON / YAML / XML).

8. Other new features — Format-specific, streaming, migration, etc.

9. Aggregate Functions — catalogue and related work after the SQL sibling.

10. Research — XSLT / Spreadsheet function surveys.

11. Metanoias and Post-Metanoias — larger architectural rethinks.

12. Things to consider — optional / deferred cleanups (not blocking the order above).

Items are numbered section.item (e.g. first point in §F.1 is 1.1; first in §F.12 is 12.1). Wrapped lines hang past the checkbox and number (see 3.1). Cross-refs point at sections above. “Consider” / “Maybe” items stay optional.

F.1. TOP parity (highest — old TOP had these)

Spec: Qwiratry-TaOP-Parity.md.

• 1.1. Consider: public multi-phase formatter API (TOP prepare-table / output-header / output-row / output-footer; one-shot drives phases) (§B.1; also §E.3.2).

• 1.2. Stronger HalfHuman round-trip where dialect allows (§B.5).

F.2. Core hygiene

Spec: Core-Hygiene.md.Status: All parts postponed until later. Items live under §F.5 (paging) and §F.12 (loader / SourceOperator / DQ10).

F.3. SQL backend (MVP)

Old TOP had Postgres Storage / cursors / FieldMode; keep the sibling MVP tight. Multi-dialect is explicitly in-scope even though it extends past a single Postgres port.

• 3.1. raku-Qwiratry-DatabaseStorage-SQL (DBIish + SQL::Abstract) (§E.1.2; also §C.2).

• 3.2. SQL WalkCursor / Iterators; FETCH/MOVE mapping; Postgres cursor adaptors (§C.1; also §E.3.4).

• 3.3. FieldMode Overflow on SQL → JSON/JSONB (§E.3.6).

• 3.4. Table Destination backends (row sinks per §D.3 Table column; Schema DDL may wait for §F.6 if not needed for MVP).

• 3.5. Red-style multi-dialect SQL encoding (§E.2.1).

• 3.6. Additional DBIish Locations (SQLite, MySQL) (§E.2.2).

F.4. TOP parity finalisation

After §F.1 and the SQL MVP (§F.3) are in good shape.

• 4.1. Migrate TaOP tests/demos; retire or thin-wrap Raku-TOP (§E.3.5).

F.5. New TOP features (in Qwiratry; after SQL MVP)

Greenfield TOP-shaped language / front-end work that was not in the old TOP code.

• 5.1. View (does Relation) (§C.3; also §E.3.3).

• 5.2. Richer Associative cell stringify (TOP TODO / Textish §6.5) (§B.3).

• 5.3. Section, Lot, DataDictionary (and TupleSet if not done as §5.8) (§E.3.3).

• 5.4. Paging via array element ranges or equivalent (§D.1).

• 5.5. atom slang + transaction capability (§D.2 / §D.2.1–§D.2.3).

• 5.6. Nested atom / savepoints / multi-Location 2PC (§D.2.3).

• 5.7. Maybe: Query output as a Relation without being a Table (§C.3; Parity §7).

• 5.8. Maybe: TupleSet (§E.3.3; Parity §7 — never implemented in old TOP).

F.6. New SQL features (DB backend; after SQL MVP)

Extras that belong on the SQL / DatabaseStorage sibling rather than core Qwiratry.

• 6.1. Schema Destination / DDL side of §D.3 (if deferred from §F.3).

• 6.2. RETURNING-style Destination results (§D.4).

• 6.3. Row locking (FOR UPDATE, …) (§D.5).

• 6.4. Predicate pushdown once multi-dialect translate exists (§E.2.2).

F.7. Tree-document tables

Spec: DatabaseStorage-Location-Tree.md. Projection stages use tree navigation with Format defaults (caller → Format default → error) and merge-on-flush.

Shipped (plan §16.1–§16.3):

• 7.1. Core DatabaseStorage::Location::Tree in raku-Qwiratry (three-stage pipeline, §8 resolution, merge-on-flush) (DatabaseStorage-Location-Tree.md §16.1.2 / §16.1.4).

• 7.2. Format-JSON thin Location::JSON with root/identity/object-key §8 defaults (§16.1.3 / §16.1.5).

• 7.3. Format-YAML thin Location::YAML with JSON-like §8 defaults (§16.2).

• 7.4. Format-XML thin Location::XML with blank §8 defaults (§16.3).

Follow-ons from the Tree plan (not in the JSON/YAML/XML first cut): (numbers align with plan §16.4 where noted)

• 7.5. Path-constrained tree walk (plan §10.4 / §16.4.1): Qwiratry::Walker::Implementation::TreeWithPath; tree node is still the walk root; invoke as $transformer($tree-root, :path($path)) / TRANSFORM-PATH (Q21); transform-only (Q22); step following via tree Navigators (Q23); missing-step frames drive ordinary APPLY + Associative attach (Q24); t/unit/tree-with-path.rakutest.

• 7.6. Path auto-creation — core (plan §10.3 / §16.4.2): Location::Tree $.path-transformer, projection/useTable wiring, associative-path-transformer, ensure-document-path; t/unit/path-transformer.rakutest.

• 7.7. Path auto-creation — Format / other dists (plan §10.3 / §16.4.3): JSON / YAML default associative-path-transformer; XML EMPTY → XML::Element named from the path step; Format location tests cover defaults (no Test-dist helper).

• 7.8. In-document missing-node path creation (plan §10 / Q7 / §16.4.4 / Q26–Q29 / Q31): derive-write-path / :table-write-path; TreeWithPath :table-action + Positional/XML append; flush path-ensure when derivable. Under-table DestinationMode gating remains for later polish.

• 7.9. Flush/merge inverse via Qwiratry Transformer molds (Q18 / plan §16.4.5): transform forwards :context to inline APPLY; JSON / YAML default :transformer molds use $*CONTEXT<rows> (Callable overrides still work).

• 7.10. Multi-site :table-name-selector end-to-end (Q2 / Q12 / Q15 / plan §16.4.6): select-table-root candidate-list origin + List/slip filters; t/unit/table-name-selector.rakutest (including useTable hydrate).

• 7.11. Sync API polish / coverage: .check-fresh / .rehydrate / .flush with :throw-on-conflict (Q20a / Q20b / plan §16.4.7); Tree mutations no longer auto-flush; t/unit/tree-sync-api.rakutest.

• 7.12. Re-run JSON / YAML / XML tests after path-walk / path-transformer / sync work (plan §16.4.8); Location tests use explicit .flush (Tree no longer auto-flushes).

• 7.13. Optional later: HTML Format + Navigator + Location::HTML with sensible <table> / body-row / cell defaults (§I).

Streaming whole-document vs push/pull rehydrate (Q5) stays under §F.8 streaming items.

F.8. Other new features (formats, streaming, …)

• 8.1. Body-cell / per-cell formatting (§B.6).

• 8.2. Later Formats: HTML (§I), Spreadsheet/.ods, Pod6, richer JSON (§B.4; also §E.3.1 / §E.2.2).

• 8.3. HalfHuman / WithBorders ParsePull / RenderPull (§B.2).

• 8.4. More Format pull; Format Supply push (§C.2; Streaming-v2).

• 8.5. Full streaming adoption per Streaming proposals (§E.2.2).

F.9. Aggregate Functions (after SQL backend)

• 9.1. General-purpose folds: Count(expr), Min/Max, AnyValue, BoolAnd/BoolOr/Every, bit aggregates, string/array/xml/json aggs (§A.1.1).

• 9.2. Statistics folds: variance/stddev, corr/covar, regr_* (§A.1.2).

• 9.3. Window-oriented catalogue: RowNumber, Lag/Lead, FirstValue/LastValue/NthValue, Ntile/PercentRank/CumeDist (§A.1.3).

• 9.4. Out-of-bucket aggregate forms when ready: FILTER / DISTINCT, ordered-set / hypothetical-set, in-aggregate ORDER BY, CREATE AGGREGATE metadata hooks (§A.1.4; also §A.2 for parallel).

• 9.5. Indirect / computed aggregate callees (§A.4).

• 9.6. Split transform out of AggregateFunctions.rakumod if needed (§A.4).

• 9.7. Nested aggregates inside extractor Blocks (§A.4).

• 9.8. Consider: TaOP Relation mutation APIs wrapping aggreduce/aggmap (§A.3).

• 9.9. Parallel aggregation: COMBINEFUNC, then SERIALFUNC/DESERIALFUNC, wire $.parallel (§A.2).

F.10. Research

• 10.1. More XSLT features including functions (research).

• 10.2. Spreadsheet Functions (Research).

F.11. Metanoias and Post-Metanoias

• 11.1. Metanoia Walkable-as-cursor-factory (§C.1).

F.12. Things to consider

Deferred from postponed §F.2 (see archived Core-Hygiene.md). Paging moved to §F.5.4.

• 12.1. Extract the relevant parts of Implementation::Loader / Singleton to a separate module again; keep the existing Qwiratry::Implementation::Loader modules (§E.1.1).

• 12.2. Consider (after Streaming v2): SourceOperator returning an Iterator (§B.7).

• 12.3. DQ10: QueryIterator.of advertising Tuple (§C.4).

• 12.4. Expand Tree Location write-path axis lowering beyond child / parent (§G.1).

• 12.5. Wire TreeWithPath into Query AST planning / query walkers (§G.2).

• 12.6. Strategy participation in path-walk attach (§H).

G. Tree Location Improvements (future)

G.1. Tree Location write-path axes (future)

Related: DatabaseStorage-Location-Tree.md §10.3–§10.4 / Q26 (derive :path from :table-selector / :row-selector query AST, or optional :table-write-path).

First cut (landed with Q26): lower only single-step navigation axes — child (⪪) and parent (⪫) — with plain name selectors. No descendant/ancestor, no filters / predicates / wildcards that are not a simple step name. If the selector exceeds those limits and no :table-write-path is provided, writes that need path auto-creation error (reads may still use the full selector).

Later (this section): consider expanding the allowed write-path axis set (e.g. limited descendant forms, or other single-step axes) once child/parent lowering and :table-write-path are solid. Track implementation under §F.12.4.

G.2. Wire TreeWithPath into Query AST

Related: DatabaseStorage-Location-Tree.md §10.4 / Q21–Q23; Post-SQL §F.7.5.

First cut (landed): Qwiratry::Walker::Implementation::TreeWithPath is transform-only (Q22). It runs when a Transformer is invoked with :path / TRANSFORM-PATH (Q21) — path auto-creation for Tree Location (:path-transformer, :table-action). Step following reuses tree Navigators (Q23). It is not selected by Query AST planning or ordinary query walkers.

Why deferred: path-constrained walk and document query fan-out are different traversal models. Query walkers visit siblings / axes under a match; TreeWithPath follows one ordered path and yields step frames (present vs missing) for mold APPLY. Mixing them early would blur “query over an existing tree” with “grow missing steps on a write path.”

Later (this section): consider wiring TreeWithPath (or an equivalent path-constrained walker) into Query AST planning / query walkers when a query is already a linear child/parent chain (or other lowerable path shape). Candidates:

• planning that chooses TreeWithPath when the AST is within write-path axis limits (§G.1), instead of a full-tree walker;

• hydrate / read paths that share the same path walker as flush ensure (one lowering, two faces);

• query-time “ensure along this path” without going through Location’s :path-transformer constructor.

Keep Q21 invocation shape ($root + :path) and Navigator step follow. Do not invent a second path language. Policy for DatabaseStorage writes stays on Location / :table-action (Q29) even if Query starts using the walker. Track implementation under §F.12.5.

H. Strategy for Attach

Related: DatabaseStorage-Location-Tree.md §10.4.2 / Q24 / Q28 / Q31; Qwiratry::Strategy hooks; Navigator attach-child.

First cut (landed): after EMPTY (or another mold) returns a node for a missing path step, the walker attaches it during the walk under the current parent (Q24). TreeWithPath.attach-child delegates to the tree Navigator (Navigator::Base.attach-child, Format overrides such as XML element append) — write-side dual of tree-child-results / follow-child (Q28). Face rules (Q31): Associative → key assign by step name; Positional → push/append. This is a subset of tree rewriting (grow a missing child), not a post-transform Location batch of step→node pairs, and not Tree::Replace.replace-node.

Strategy is not required for this cut. EMPTY and Strategy do not decide §10 policy (:table-action / DestinationMode stay on Location / walker mechanics — Q29). Ordinary document walkers may still run Strategy hooks (before / on-match / after); path-walk attach does not consult them today.

Why deferred: path attach is a narrow, ordered mutation at a known parent+step. Full Strategy / TreeRewrite targets matching and replacing existing nodes (and traversal control via ControlSignal). Wiring Strategy into attach early risks conflating “create missing child” with “rewrite matched subtree,” and could let Strategy override ensure/use policy that Location must own.

Later (this section): consider whether Strategy should participate in path-walk attach, for example:

• optional Strategy hooks around attach (approve / skip / substitute the mold result) without moving policy out of Location;

• unifying grow-missing-child with a controlled rewrite face so document Transformers and path ensure share one attach story;

• Format-specific attach behaviour that is awkward on Navigator alone but fits a Strategy override.

Any design must keep Q29 intact: Strategy must not choose use vs ensure vs DestinationMode. Track implementation under §F.12.6.

I. HTML Format / Navigation / Location

Status: optional later — after JSON / YAML / XML Tree Location (§F.7.1– §F.7.12). Track checklist under §F.7.13 (and Format discovery under §F.8.2).

Spec home for HTML: this section. It carries forward the HTML-relevant parts of the former Tree Location plan so that work can proceed without that document. Core Location::Tree (projection stages, merge-on-flush, path ensure, sync API) is already shipped; HTML only adds Format + Navigator + a thin Location::HTML subclass with defaults.

I.1. Goal

Ship an HTML tree Format and open HTML documents as live TaOP tables via the existing Tree Location pipeline:

1. Qwiratry::Format::HTML (sibling dist) — Parse / Render between HTML text and a Mu tree Navigators understand.

2. Qwiratry::Tree::Navigator::HTML — element / attribute child lookup and attach-child (same faces as XML: Associative attributes, Positional element children).

3. Qwiratry::DatabaseStorage::Location::HTML — thin subclass of core Location::Tree; default :format / extension; advertise sensible <table> / body-row / cell projection defaults so useTable(name => …) can open without every selector when the document matches those conventions.

Do not fork the Tree projection algorithm or invent an HTML-only path language (no CSS/XPath dialect beside what shared Query / Navigators already provide).

I.2. Layering (unchanged)

Formats must not grow “table mode.” Projection runs on the already-parsed Mu tree.

Class shape:

# core raku-Qwiratry
Qwiratry::DatabaseStorage::Location::Tree

# sibling Format-HTML dist (future)
…::Format::HTML
…::Tree::Navigator::HTML
…::DatabaseStorage::Location::HTML   # format => 'HTML'; §I.4 defaults

I.3. Why HTML needs multi-stage projection

Even after selecting a table site, row data is often not “the site’s immediate children as Associatives.” HTML is the clearest case:

• The table site may be a <table> node.

• Data rows may live under <tbody> (or be mixed with <thead> / <tfoot>).

• Header vs data <tr> distinctions belong in the row-selector (e.g. only tbody/tr, or only rows that match a navigation predicate the shared query stack already supports), and/or in ordinary query after load — not in a fourth Location-private filter stage.

• Cells must map to Relation fields (ordered <td> / <th> → columns, usually via :header-selector + positional cells).

Chaining “open the HTML table, then pick a data row” is projection then ordinary table/query — not a Location-private query string.

I.4. Projection defaults

Core Location::Tree supplies no universal stage defaults. Resolution per stage remains: caller argument → Format Location::* default → error.

Location::HTML should advertise defaults so omit-args open can work for common HTML tables:

Column-naming mode: HTML defaults should use array-like + :header-selector (header array → keys; each row array → values). Hash mode (no header-selector; :field-projection returns hashes) remains available via caller override for unusual documents.

Callers may always override any default. If a document does not match the defaults (no <table>, no tbody, irregular cells), omit-args must error or the caller must pass selectors — do not silently guess.

Illustrative open:

# Format defaults: open a conventional HTML table by name alone
my $h = $html-db.useTable(name => 'report', action => 'use');

# Override row collection when data rows are not under tbody
my $h2 = $html-db.useTable(
	name => 'report',
	action => 'use',
	row-selector => -> $table { … },   # Block / Callable preferred
);

useTable projection argument names (already on Tree): :table-selector, :table-name-selector, :row-selector, :header-selector, :field-projection, :transformer (flush/merge molds), :path-transformer (EMPTY path creation), optional :table-write-path.

I.5. Non-goals (HTML-specific)

• Guessing header vs body conventions when the caller did not encode that in the row-selector (or handle it with ordinary query after load).

• A separate Location row-filter stage — redundant with row-selector and with outside query operators.

• Putting table-shape assumptions in Format::HTML itself (Parse/Render stay general; defaults live on Location::HTML).

• Shipping HTML as part of the JSON / YAML / XML first cut (already done without HTML).

I.6. Pipeline sketch (HTML)

HTML document root
    --(table-selector)----->  <table> root
    --(header-selector)---->  header key array     # thead / th cells
    --(row-selector)------->  data <tr> nodes      # e.g. tbody/tr
    --(field-projection)--->  cell value arrays    # then splice header×row
    -----------------------> @!rows + fields

I.7. Merge-on-flush

Tree-backed HTML tables share a document with non-table structure (<thead>, captions, sibling elements, scripts, …). Flush must:

1. Keep / rehydrate the full document tree (Walkable sync: .flush / .rehydrate / .check-fresh).

2. Re-navigate with the same table-site and row-collection selectors.

3. Reconcile @rows into the selected row nodes via :transformer molds (inverse of hydrate field projection). Nodes outside the row-selector result (e.g. <thead> rows never selected) stay untouched unless DestinationMode overwrite applies to that collection site alone.

4. Format Render the whole document; Location Destination write.

Empty data-row flush writes an empty collection at the row-collection site (does not delete <thead> / caption / unselected siblings by default).

I.8. Path auto-creation (EMPTY)

When table action / DestinationMode allows creating missing in-document nodes, HTML follows the same path-walk as other Tree Formats (:path-transformer, mold name EMPTY, TreeWithPath + Navigator attach-child).

Location::HTML may advertise a default path-creation Transformer whose EMPTY molds build sensible empty HTML nodes by role / step name (e.g. <table>, <tbody>, <tr>, <td> / <th>), using $*CONTEXT roles (ancestor / table-site / row-collection) the same way JSON chooses {} vs [] and XML names elements. Policy (use / ensure / create) stays outside EMPTY. Write-path axis limits remain as in §G.1 until expanded.

I.9. Navigator notes

• Prefer reusing XML-like element / attribute navigation patterns; specialise only where HTML differs (void elements, case folding of tag names if the parse tree requires it, tbody insertion quirks if the HTML parser normalises structure).

• Associative face → attributes; Positional face → element children (attach-child append). No third attach API.

• Named helpers for common selectors (tbody/tr, header row) may ship in the Format dist — built from shared navigation primitives, not a private DSL.

I.10. Implementation checklist

• I.10.1. Sibling dist raku-Qwiratry-Format-HTML (or agreed name): Format::HTML Parse / Render; META6; basic round-trip tests.

• I.10.2. Tree::Navigator::HTML (child lookup + attach-child); unit coverage for element / attribute faces.

• I.10.3. Thin DatabaseStorage::Location::HTML subclassing core Location::Tree; default :format / extension.

• I.10.4. Implement §I.4 projection defaults + document them in the Format dist (caller → default → error).

• I.10.5. Default :path-transformer / EMPTY for HTML empty nodes (§I.8); Location tests for ensure / create along a table path.

• I.10.6. Default or documented :transformer molds for flush/merge of header + body cells; prove <thead> survives body-only flush.

• I.10.7. Conformance via Qwiratry::Test where applicable; explicit .flush after Tree mutations (no auto-flush).

• I.10.8. Changelog + Extending notes; wire discovery through Qwiratry::Format / Location backends.

I.11. Decisions that apply (no separate HTML Q list yet)

Taken from the Tree Location cut; restate here so this section stands alone:

• Projection selectors are Blocks / Callables (preferred over query-ast for Tree stages); optional :table-name-selector when multiple table sites hit; still error if ambiguous / empty.

• No separate Projection class or Location row-filter stage.

• Column naming: optional :header-selector before :row-selector; both return arrays spliced into hashes — the usual HTML path.

• Flush/merge inverse uses a caller/Format :transformer (molds); path auto-creation uses separate :path-transformer (EMPTY).

• Core Tree has no universal defaults; only Format subclasses may.

• Sync API: .flush / .rehydrate / .check-fresh with :throw-on-conflict (default True); mutations mark dirty and do not auto-flush.

Open HTML-only product choices (resolve when implementing §I.10):

• Exact default for “which <table>” when several exist and no name selector is passed (error vs first table vs id matching useTable name).

• Whether missing <tbody> still selects direct <tr> children of <table> (parser-normalised documents often insert tbody).

• Header cell source when both <thead> and a first body row of <th> exist.

• Which HTML parser / DOM Mu type the Format commits to (must be stable for Navigator + tests).