Qwiratry TaOP Basics

Normative sketch for the TaOP layer on Qwiratry: Relation, Memory, Postgres, Tables (and related front-end objects). These are not Qwiratry core prerequisites.

Depends on: Qwiratry-TaOP-Prerequisites.md (streaming Location I/O, TypedIterator, QueryIterator does TypedIterator, WalkCursor tree contract).Parent plan: Qwiratry-TaOP-Plan.md (§4.1–4.2 and following).

Status: In progress — Memory, Destination, Location, CSVdemo, and Database-as-LHS navigation (DQ17) done; Location::JSON in Format::JSON. Next: more tests / later siblings. Prerequisites complete for TypedIterator / query match streams. Scope: TaOP Relation vs Table (preserve front-end API); Qwiratry::DatabaseStorage kinds (Memory, Location-backed, CSVdemo, CSV / SQL siblings); anonymous swappable Tables; Destination row sinks and Parse ElementType on structured .of (DQ15); for structured .of; WalkCursor backends. Blob Location remains URI I/O; Format stays encoding.

1. Goals

1. Relation / Table as distinct TaOP types: Relation is the shared relational role (Table, View, and — planned — query output); Table is the stored-table front-end with Storage actions, fields, FieldMode, mutation.

2. Preserve TaOP front-end APIs for Table (including Relation-inherited surface), Field, and Database as in TaOP.rakumod — not drop-in backend compatibility, but the object/method face users call.

3. Memory DatabaseStorage as the default lightweight backend for anonymous Tables.

4. Anonymous, swappable Tables (Table.new without Database; later DatabaseStorage backend swap with capability checks).

5. Capability-typed backends so blob-only Locations do not pretend to be tables (and vice versa), with clear failures on capability swap.

6. Qwiratry::DatabaseStorage hierarchy (not “Location absorbs tables”) — see §4 / DQ2:

   • Qwiratry::DatabaseStorage abstract base — TaOP Storage-shaped backend surface (§4.3), including optional streaming pull/sink

   • Qwiratry::DatabaseStorage::Location — Location (+ optional Format) as transport; subclasses (e.g. …::Location::JSON) map document trees ↔ rows

   • Qwiratry::DatabaseStorage::Memory — in-process default

   • Qwiratry::DatabaseStorage::CSVdemo — basic demo only (not full CSV)

   • Qwiratry::DatabaseStorage::CSV — full CSV via Text::CSV in sibling raku-Qwiratry-DatabaseStorage-CSV (later)

   • Qwiratry::DatabaseStorage::SQL — DBIish + SQL::Abstract in sibling raku-Qwiratry-DatabaseStorage-SQL (later); dialects via those libs, not a custom encoder

2. Non-goals

• Replacing the TaOP object/method API with pipelines only (pipelines remain complementary; plan §10.2).

• Custom / Red-style multi-dialect SQL encoding in Qwiratry (plan §12). Dialects are DBIish + SQL::Abstract’s job; extend those if more features are needed.

• Graduating CSVdemo Format into production CSV; Qwiratry::DatabaseStorage::CSVdemo is a basic demo only. Full CSV lives in raku-Qwiratry-DatabaseStorage-CSV (Qwiratry::DatabaseStorage::CSV + Text::CSV).

• Re-implementing WalkCursor tree proof (done in prerequisites / Cursor proposal).

• Full Format Supply push I/O (deferred; not required to start Basics).

• Changing SourceOperator to return a pull Iterator (rejected in prerequisites; walkable evaluate stays).

• Representing Tables as Location URIs (DQ2).

• A Location::Base::Relation capability face (withdrawn; that name collided with TaOP Relation — see DQ1 / §3.2).

3. Pipelines and table objects

3.1 Pipeline face — Source / Destination

Defined for Qwiratry I/O operators in the prerequisites. SourceOperator.evaluate stays walkable-only (does not return an Iterator). Blob Locations expose text via walkable/pull. Tables expose rows via DatabaseStorage scan / row pull → TypedIterator, composed into pipelines as Basics defines.

Reading a table must not mean “serialize the whole table as one string.” It means start a pull of rows (Table / DatabaseStorage scan).

Parse/Render apply when the pipeline still deals with encoded text. When the upstream stream’s .of is structured (row-like), Parse must throw (X::Qwiratry::IO::ElementType — DQ15); select a string field first if text parsing is wanted. Soft-skip is withdrawn for now. Destination on DatabaseStorage uses a row sink; on a blob-only Location, structured .of is an error (already the text I/O rule in core — do not .Str-dump rows).

3.1.1 Structured .of / row sinks (Basics work)

Moved here from prerequisites (not needed until DatabaseStorage exists):

• Choose row stream .of type when implementing Memory (Associative interim vs Tuple once it exists — DQ8; there is no separate Row type). Done: Tuple (Hash::Ordered) for rows; interim .of advertises Associative.

• Destination row sink API on DatabaseStorage (write-row / close or equivalent); wire Destination operators when .of / elements are rows.

• ParseOperator: if .of is structured / row-like and a text Format was requested, throw (ElementType or similar) — DQ15; select a string field first if parsing is wanted.

• End-to-end: Memory Table pull → query → Memory Destination row sink without Format dump. Done: t/unit/taop-basics-step6.rakutest (blob Destination also rejects structured rows via ElementType — no Str dump).

3.2 Object face — Table / Database / Relation

• Relation — TaOP shared front-end role for table-like sets. Table does Relation; planned View does Relation; query output MAY be a Relation without being a Table (DQ1).

• Table / Field / Database — preserve TOP.rakumod-facing APIs; backends are DatabaseStorage, not a requirement to keep TaOP Storage class names.

• DatabaseStorage — backend hierarchy (§4). Not the same as Relation.

• Withdrawn: earlier Basics sketch of Qwiratry::Location::Base::Relation (a Location “live table handle” face). That was meant as “ask a Location for use-table,” but it collided with TaOP Relation and fought DQ2. Table handles come from DatabaseStorage / Database.useTable, not from Location.

Pipelines and objects are complementary. Table mutation / Destination row sinks SHOULD share underlying row I/O (TypedIterator, prerequisites §5.3).

Source ↗  →  (query / transform)  →  Destination ↘   # pipeline; files AND tables
$table = Table.new;                                  # Memory DatabaseStorage
$db ⪪ <countries>                                    # Database LHS → table child
# Location-backed example (concept):
# Qwiratry::DatabaseStorage::Location::JSON + Location::File + Format::JSON

3.3 What “Source / Destination with a table” means (normative)

4. DatabaseStorage; Location for blobs

4.1 Design rule (see DQ2)

Do not require a Table to be a Location URI. Location-backed tables compose Location (and Format) inside Qwiratry::DatabaseStorage::Location subclasses.

Withdrawn: Location::Base::Relation as a Location capability. TaOP Relation stays the front-end role only.

4.2 Qwiratry::DatabaseStorage kinds

Qwiratry::DatabaseStorage                          # abstract; full table backend
├── Qwiratry::DatabaseStorage::Memory              # default anonymous Tables
├── Qwiratry::DatabaseStorage::Location            # abstract; Location as transport
│   └── Qwiratry::DatabaseStorage::Location::JSON  # Loc + Format::JSON → rows
│       # (ships in raku-Qwiratry-Format-JSON)
├── Qwiratry::DatabaseStorage::CSVdemo             # basic demo only — not full CSV
├── Qwiratry::DatabaseStorage::CSV                 # dist: raku-Qwiratry-DatabaseStorage-CSV
│       # Text::CSV; full CSV spec (later)
└── Qwiratry::DatabaseStorage::SQL                 # dist: raku-Qwiratry-DatabaseStorage-SQL
        # DBIish + SQL::Abstract (multi-dialect via those libs)

Qwiratry::DatabaseStorage::Location hooks (normative intent): overridable hooks that turn a Location (+ optional Format) payload into row/column traffic and back. A JSON subclass overrides hooks to pick the array/object paths that become rows and fields — so Location::File + Format::JSON stay general, while DatabaseStorage owns table projection. These hooks are in addition to the broader backend surface in §4.3 — not a substitute for it.

Database / discovery: TaOP-facing Database selects a Qwiratry::DatabaseStorage::* kind (this is the Storage hierarchy; no separate TableStorage name). useTable returns a Table backed by that storage.

SQL dialects: Qwiratry::DatabaseStorage::SQL does not invent a dialect layer. Use DBIish + SQL::Abstract; extend those ecosystems when more SQL features are needed.

4.3 Backend contract (more than streaming)

Preserve TaOP-facing Table / Database / Field / Relation method surface where practical. Streaming row pull/sink is necessary but not sufficient.

TaOP's Database::Storage / Table::Storage already do far more than iterate rows: table actions, field metadata, FieldMode/makeTuple, Positional rows, Associative fields, add-row, format/parse delegation, existence checks, etc. Qwiratry backends must support the same kinds of work so the front-end APIs keep working — exact method names may differ; the things-it-can-do set should track TaOP Storage.

Illustrative sketch (not a complete API):

role Qwiratry::DatabaseStorage {
    # Database-level (TaOP Database::Storage-shaped)
    method useTable(Table :$table, Str :$action = 'use', *%params --> Mu) { ... }

    # Per-table surface (TaOP Table::Storage-shaped) — may live on the object
    # returned/bound by useTable, or on the same class for single-table kinds:
    #   fields / field-indices / add-field
    #   FieldMode install + makeTuple
    #   Positional row access and/or Iterator / TypedIterator scan
    #   add-row / fill_from_aoh / mutation
    #   exists / table-actions (create|alter|use|can-create|ensure)
    #   format / parse (delegate to Format factories; plan §10.2)
    #   DestinationMode-aware row sink (pipeline face; DQ3)
    #   optional WalkCursor
    method storage-capabilities(--> Associative) { ... }
}

role Qwiratry::DatabaseStorage::Location does Qwiratry::DatabaseStorage {
    # holds Location (+ optional Format); subclasses override mapping hooks
    method document-to-rows(Mu $document --> Mu) { ... }
    method rows-to-document(Mu $rows --> Mu) { ... }
}

Table (front-end) MUST provide at minimum:

4.4 Capability advertising and failures

4.4.1 Advertised keys

Suggested keys (illustrative) on DatabaseStorage and/or Location:

Backend swap (Memory → SQL, etc.) MUST fail explicitly if a required capability is missing — no silent FieldMode degradation.

4.4.2 Capability-failure exceptions

Reuse X::Qwiratry::IO::LocationError (+ NotFound / AlreadyExists) for identity failures on Destination modes. Do not overload those for “this backend cannot be a table.”

4.4.3 Structured stream on blob Destination

4.5 Built-in expectations (first slices)

4.6 Documentation updates

• Adding-Sources-Destinations — when pipelines meet tables vs blobs.

• Location base pod — Walkable / pull/sink only (no Relation face).

• DatabaseStorage / Table docs — hooks, FieldMode, navigation as LHS/RHS.

• Cross-link Streaming proposal and prerequisites TypedIterator.

4.7 Acceptance criteria (DatabaseStorage)

• Memory useTable supports at least use / can-create / ensure.

• Qwiratry::DatabaseStorage::Location abstract hooks usable by a JSON subclass with File + Format (…::Location::JSON in Qwiratry::Format::JSON).

• Qwiratry::DatabaseStorage::CSVdemo demo path exists (not full CSV).

• Backend surface covers TaOP Storage-shaped needs (§4.3), not only row streams. Done (Memory first cut): fields / FieldMode / makeTuple / Positional / row-iterator / add-row / table actions / DestinationMode sink / WalkCursor / storage-capabilities — see t/unit/taop-acceptance-4.7-5.4.rakutest. SQL-sized surface remains §7 step 7.

• File alone as useTable fails clearly.

• Table usable as navigation LHS/RHS; Database LHS yields tables as children.

• TaOP-facing Table / Field / Database / Relation APIs preserved in spirit. Done (first cut): bare exports; Table does Relation; Tuple rows; Field column view; Table.format / Table.parse share Format factories (t/unit/taop-acceptance-4.7-5.4.rakutest).

• Memory (and later SQL) Walkable scan uses WalkCursor on Axis.preceding-following where advertised.

5. Anonymous, swappable Tables

5.1 Design rule

my $t = Table.new;   # lightweight Memory DatabaseStorage by default
# later, preferably only the declaration changes:
my $t = Table.new(:name<countries>, :storage-type<CSVdemo>, :location($dir), ...);
# or Location::JSON (Format-JSON dist) / SQL (sibling dist, §7 step 7)
# — not “the table IS a Location URI” (DQ2 / DQ4)

• Database optional for single-table / anonymous use.

• Named Database for multi-table stores and navigation roots ($db ⪪ <table>).

• No TaOP unnamed Memory singleton as the primary model.

• Preserve TaOP-facing Table / Field / Database / Relation APIs (Goal 2).

Namespace follows packaging option C in the plan; this spec requires behaviour.

5.2 Front-end responsibilities

• Optional name / Database metadata.

• Delegates to DatabaseStorage (§4); row I/O consistent with Destination sinks (§3).

• TOP-like methods (add-row, fields, parse / format, … per plan §10.2): parse/format share Format factories with operators; Location only when talking to blobs (or via Qwiratry::DatabaseStorage::Location composition).

• Backend swap preserves API; capability mismatches fail per §4.4.

5.3 First-class query / walk / schema origin

5.4 Acceptance criteria (anonymous Tables)

• Anonymous Memory Table.new without Database/name.

• TaOP-style row/field access; FieldMode Automatic on Memory.

• Usable as Walker/query origin without Catalog wrap.

• Schema.discover (or adapter) accepts it.

• Backend swap to Location-backed DatabaseStorage works; bare blob Location fails clearly. Done: Memory → CSVdemo same front-end API; File → WrongFace (t/unit/taop-acceptance-4.7-5.4.rakutest). SQL swap is §7 step 7; Location::JSON covered in Format-JSON t/03-location-json.rakutest.

• Documented sketch: anonymous Memory → redeclare with CSVdemo / Location::JSON storage (§5.1; exercised in t/unit/taop-acceptance-4.7-5.4.rakutest).

6. WalkCursor backends (TOP)

Tree contract is done in core (Cursor-Proposal.md, prerequisites §5.6). Still open here:

• Memory / loaded-table WalkCursor on Axis.preceding-following (ordered row sequence) — required in Memory first cut (DQ19).

• SQL DatabaseStorage cursor mapping FETCH / MOVE onto fetch-* / move-*.

• Metanoia Walkable-as-cursor-factory (deferred in Cursor proposal).

• Mapping TaOP's Cursor::Storage::Postgres onto WalkCursor (0-based Raku vs 1-based SQL adaptors).

(Who mints: Walker / Plan.cursor is the query entry; Memory Table/storage provides the cursor — DQ19.)

7. Implementation order

After prerequisites (streaming, TypedIterator text path, WalkCursor tree, QueryIterator does TypedIterator):

• 1. Qwiratry::DatabaseStorage abstract + Memory + minimal TaOP-facing Table / Relation / Field / Database; pick row .of (DQ8); Memory WalkCursor when needed. Done (first cut).

• 2. Destination row sink + Parse ElementType for structured .of (§3.1.1 / DQ15); honour DestinationMode (DQ3). Done (Memory + File paths).

• 3. Qwiratry::DatabaseStorage::Location hooks + JSON subclass (File + Format::JSON) in raku-Qwiratry-Format-JSON. Done (first cut).

• 4. Qwiratry::DatabaseStorage::CSVdemo basic demo.

• 5. Database-as-LHS navigation via Walker::Implementation::Table (DQ17; does not require Schema.discover). Extend Schema.discover / Catalog when a Catalog root is needed.

• 6. Tests: bare File ≠ table; Memory pull/sink rows; capability failures; no Str dump. Done: t/unit/taop-basics-step6.rakutest (plus earlier memory-taop / location-taop / csvdemo-taop / database-lhs-nav; JSON persistence in Format-JSON t/03-location-json.rakutest). Destination blob path rejects structured content (ElementType) instead of Str-dumping via File ~$content.

• 7. Partial: raku-Qwiratry-Format-CSV (Format::CSV + DatabaseStorage::CSV via Text::CSV) — see Qwiratry-CSV-Plan.md. Still later: raku-Qwiratry-DatabaseStorage-SQL; HalfHuman / WithBorders (Textish); more Format pull. Full Supply push deferred.

Do not ship DatabaseStorage while table reads would still be forced through walkable-Str-only dumps.

8. Open Questions

(none numbered right now — see §9)

9. Decided Questions

• DQ1 (was OQ1) — Relation vs Table. Different. Keep both. Relation = shared front-end role; Table = stored table. Do not use Location::Base::Relation (withdrawn).

• DQ2 (was OQ2) — Table identity / URI + DatabaseStorage. Tables are not Location URIs. Navigation LHS/RHS + Destination/mutation. Qwiratry::DatabaseStorage::* is the Storage hierarchy (not TableStorage): Memory; …::Location (+ JSON); CSVdemo; CSV in raku-Qwiratry-DatabaseStorage-CSV; SQL in raku-Qwiratry-DatabaseStorage-SQL. TaOP-facing Database selects a Qwiratry::DatabaseStorage::* kind. Parent plan updated to match (DQ18).

• DQ2b — SQL dialects. Multi-dialect SQL is DBIish + SQL::Abstract’s responsibility. Do not build a Qwiratry-local Red-style encoder; extend those libraries when more features are needed.

• DQ3 (was OQ3) — Destination write policy. Language + File/HTTP done (DestinationMode + slang ops). DatabaseStorage row sinks honour the same family.

• DQ4 (was OQ4) — Memory URI. Constructor-only Memory DatabaseStorage; no memory:// table URI. Location appears only inside Qwiratry::DatabaseStorage::Location.

• DQ5 (was OQ5) — Database. TaOP-facing front-end selecting Qwiratry::DatabaseStorage::* / useTable + navigation root — not a Location grouping.

• DQ6 (was OQ6) — Exceptions. §4.4.2–4.4.3 sketch.

• DQ7 (was OQ7) — FieldMode loading site. Follow TOP: load on DatabaseStorage / per-table storage construction (TOP: Table::Storage construction calls load-library(type => "TOP::FieldMode::$field-mode", …) and stores $!field-mode-object used by makeTuple). Not on Database alone; not a Format.

• DQ8 (was OQ8) — Row stream .of type. No separate Row type — prefer Tuple when it exists; Associative interim OK. Decide at Memory implementation.

• DQ9 (was OQ9) — Format pull vs CSV. Structured pull is enough to start. …::CSVdemo is demo-only; full CSV is raku-Qwiratry-DatabaseStorage-CSV + Text::CSV. Format chunk streaming is not a blocker.

• DQ10 (was OQ10) — QueryIterator .of. If easy, advertise Tuple (once it exists); default Mu is fine. Not high priority.

• DQ11 — Backend surface vs streaming. Qwiratry::DatabaseStorage is not only a streaming API. Backends must support the TaOP Storage-shaped things-it-can-do set (§4.3). Method names follow DQ12 (TOP-shaped).

• DQ12 (was OQ12) — DatabaseStorage method surface. Prefer (a) port TaOP Database::Storage / Table::Storage method names largely as-is (useTable, makeTuple, add-row, add-field, Positional/Associative, format/parse, …). First Memory cut implements that surface as needed for anonymous Table + FieldMode + basic mutation/scan; grow only when tests require it.

• DQ13 (was OQ13) — Two-level storage like TaOP. Keep the split: database-level storage (useTable) and per-table storage bound to the front-end Table. Under Qwiratry, mirror TaOP's pairing — e.g. Qwiratry::DatabaseStorage::Memory (database driver) + a per-table class such as Qwiratry::DatabaseStorage::Memory::Table (or equivalently named Table::Storage::Memory re-exported from the Qwiratry dist). Do not revive a separate TableStorage product hierarchy name.

• DQ16 (was OQ16) — Bare front-end names. Keep short names users type: Table.new(...), Database, Relation, Field, Tuple (no Qwiratry::Table required for normal use). Modules live under the Qwiratry distribution; replace TOP-prefixed helpers where relevant (TOP::Core → Qwiratry equivalent). FieldMode: Qwiratry::…::FieldMode::* (or keep discoverable short path), not forcing TOP:: in user code.

• DQ17 (was OQ17) — Database-as-LHS navigation. Use / extend Walker::Implementation::Table (its job is table navigation; this is the first major table functionality). Anonymous Tables remain valid origins without a Database. TOP has no Schema.discover: Memory storage keeps a %tables map and useTable by name; there is no FK/catalog discovery service. Qwiratry Schema.discover / Catalog stays the place to recognise a Database (named tables) or a lone Table when building a Catalog — extend it when wiring Database LHS, rather than inventing a TaOP-style discovery API.

• DQ18 (was OQ18) — Parent plan sync. Done: Qwiratry-TaOP-Plan.md updated to Qwiratry::DatabaseStorage (not Location-absorbs-Storage).

• DQ19 (was OQ19) — WalkCursor in Memory first cut. Yes — implement Memory WalkCursor early so later backends can copy the pattern. Clarifying the rest of the old question: query entry stays Walker / Plan.cursor (callers do not normally WalkCursor.new by hand); Memory storage/Table must provide a cursor (factory / specialization) so Plan.cursor can obtain one for preceding-following tests. Iterator/TypedIterator scan alone is not enough for that first cut.

• DQ14.3.1 (was OQ14.3.1) — Destination → Table API shape. (c) — DestinationOperator talks to the Table front-end; Table implements DestinationMode by delegating to a storage row-sink (and may implement add-row via that sink). Same shape for Memory first cut; SQL can share it.

• DQ14.1 (was OQ14.1) — Destination vs table-action gap table. Withdrawn as irrelevant. Table actions = structure; Destination = data. No alignment table.

• DQ14.3.2 (was OQ14.3.2) — Overwrite on a table. Overwrite (⮷) is truncate-all then insert the stream. Upsert-by-key is Ensure Present (⮷⊕), not Overwrite.

• DQ14.3.3 — Insert: container vs content. Normative mapping is SQL-Features-Plan.md §A.4.3: File/HTTP Insert is container-wise (create path/resource only if absent). Table Insert is content-wise / row-wise (insert each row only when its identity is absent; fail per conflicting row). Do not treat “table must be empty” as Table Insert. Optional @.key-fields names identity columns; when unset, Memory uses the full row Associative as identity (interim).

• DQ14.2 (was OQ14.2) — Re-open with a different table action (Raku-TOP behaviour). Intended: yes — each Table.new / useTable passes :action into Table::Storage TWEAK, which re-checks existence and may set create/alter flags (create errors if present; alter/use error if absent; can-create/ensure create if missing; alter/ensure may alter schema). Memory today: Database::Storage::Memory.useTable returns the cached %tables{$name} on a second open and does not re-run action handling — so a later different :action is ignored. (Table::Storage::Memory.raw-exists always returns False, with a TODO to consult database.tables.) Postgres today: useTable always constructs a new Table::Storage::Postgres with the new :action (replacing the cached object), so a second open does re-apply action logic (create/alter still incomplete stubs). Qwiratry lean: honour re-open with a new :action (Postgres-shaped); do not copy Memory’s silent cache short-circuit. Document that Memory must re-evaluate actions on reopen (or drop the early return).

• DQ15 (was OQ15) — Parse/Render on mismatched shape. Throw (for now), e.g. X::Qwiratry::IO::ElementType, when Parse is asked to run a text Format on an already row-structured stream (or structured .of). If the user wants text parsed, they select the relevant string field first. Same spirit for Render when the subject shape does not match what the Format expects — throw rather than soft-skip. Soft-skip can be revisited later.

10. References

• Qwiratry-TaOP-Plan.md — especially §3–4, §6–8, §10 (aligned with Basics DQ2 / DQ18: DatabaseStorage for tables)

• Qwiratry-TaOP-Prerequisites.md

• SQL-Features-Plan.md — Destination modes (DQ3)

• Cursor-Proposal.md

• lib/TypedIterator.rakumod, lib/Qwiratry/QueryIterator.rakumod

• lib/Qwiratry/Query/Slang/Ops.rakumod, lib/Qwiratry/Operator/IO.rakumod

• Streaming-Qwiratry-Proposal.md

• Adding-Sources-Destinations

• lib/Qwiratry/Table/Schema.rakumod, Table.rakumod (Catalog — not TaOP Table)

• Raku TOP: ~/src/TOP/Raku-TOP/main — especially TOP.rakumod, Database/Storage.rakumod (FieldMode load), FieldMode plugins

• DataOrientedProgramming.xmi (Relation ⊃ Table / View / …)