Qwiratry TOP Basics

Normative sketch for the TOP layer on Qwiratry: Relation, Memory, Postgres, Tables (and related front-end objects). These are not Qwiratry core prerequisites.

Depends on: Qwiratry-TOP-Prerequisites.md (streaming Location I/O, TypedIterator, QueryIterator does TypedIterator, WalkCursor tree contract).Parent plan: Qwiratry-TOP-Plan.md (§4.1–4.2 and following).

Status: Not started as product work. Prerequisites complete for TypedIterator / query match streams. Do not implement Relation/Table while table Sources would still dump to Str. Scope: Location Relation face; Memory / CSV / Postgres Locations; anonymous swappable Tables; sharing row I/O with Source/Destination; WalkCursor backends; Destination row sinks and Parse skip for structured .of; choosing row .of type when table Locations land.

1. Goals

1. Relation / use-table as the TOP object face on Location backends: live Table handle, actions, fields, FieldMode, mutation — sharing row read/write with Source/Destination.

2. Memory Location as the default lightweight backend for anonymous Tables.

3. Postgres Location (sibling package) with row Source/Destination + Relation; map SQL cursors onto WalkCursor where useful.

4. Anonymous, swappable Tables (Table.new without Database; later :location swap with capability checks).

5. Capability-typed backends so blob-only Locations do not pretend to be tables (and vice versa), with clear failures on location swap.

6. No Qwiratry::Storage::* — Storage responsibilities move into upgraded Location + thin TOP Table front-end.

2. Non-goals

• Replacing the TOP object/method API with pipelines only (pipelines remain complementary; plan §10.2).

• Multi-dialect / Red-style SQL encoding (plan §12) — near term Postgres only.

• Graduating CSVdemo into a production Format (CSV tables are Qwiratry::Location::CSV).

• Re-implementing WalkCursor tree proof (done in prerequisites / Cursor proposal).

• Full Format Supply push adaptors (deferred; not required to start Basics).

• Changing SourceOperator to return a pull Iterator (rejected in prerequisites; walkable evaluate stays).

3. Two faces of the same Location

3.1 Pipeline face — Source / Destination

Defined for Qwiratry adaptors in the prerequisites. SourceOperator.evaluate stays walkable-only (does not return an Iterator); table Locations expose rows via read-pull → TypedIterator (and Relation handles), composed into pipelines as Basics defines.

Source on a table location must not mean “serialize the whole table as one string.” It means start a pull of rows (Location read-pull / Relation scan).

Parse/Render apply when the pipeline still deals with encoded text. When the upstream stream’s .of is structured (row-like), Parse must be skipped (or fail clearly if a text Parse was requested). Destination on a table-capable Location uses a row sink; on a blob-only Location, structured .of is an error (already the text-adaptor rule in core — do not .Str-dump rows).

3.1.1 Structured .of / row sinks (Basics work)

Moved here from prerequisites (not needed until table Locations exist):

• Choose row stream .of type when implementing Memory (Associative interim vs dedicated Row / Tuple once those types exist).

• Destination row sink API on table-capable Locations (write-row / close or equivalent); wire DestinationOperator when .of / elements are rows.

• ParseOperator: if .of is structured, do not parse-pull / join as text.

• End-to-end: Memory Source pull → query → Memory Destination row sink without Format dump.

3.2 Object face — Relation / use-table

Qwiratry::Location::Base::Relation (name negotiable) binds a live Table for the TOP object/method API:

• table actions (create / alter / use / can-create / ensure)

• fields, FieldMode, makeTuple-level shaping

• long-lived handle for $table.add-row, field access, Walker origin, etc.

Relation is not a replacement for Source/Destination and not “the only way to talk to tables.” It is the object-handle face. The same backend module SHOULD back both faces and share the underlying row iterator / write path (TypedIterator, prerequisites §5.3).

Source ↗  →  (query / transform)  →  Destination ↘   # pipeline; files AND tables
$table = Table.new(:location<…>)                     # object face; same Location backend

3.3 What “Source with a table” means (normative)

4. Relation / upgrade Location

4.1 Design rule

Do not add Qwiratry::Storage::*. Extend Location so a backend may implement any subset of:

• Source / Destination (blob and/or structured pull via TypedIterator)

• Relation for live Table handles

Format remains “how is it encoded?” for HalfHuman, WithBorders, demo CSVdemo, etc. CSV tables are Qwiratry::Location::CSV (sibling package).

4.2 Discovery and naming

• Keep Qwiratry::Location as the sole Location discovery/dispatch factory.

• URI scheme → backend mapping unchanged in spirit (file://, postgres://, later CSV / memory:// or in-process constructor).

• Backend module Qwiratry::Location::<Name> may provide ::Source, ::Destination, ::Relation (and pull capabilities on those classes).

• Factory asks for an operation type; missing capability → clear error.

• Walkable bases for Source / Destination stay as today; Relation is an additional discoverable type or capability.

4.3 Relation capability contract

role Qwiratry::Location::Base::Relation {
    method use-table(
        :$table,
        Str :$action = 'use',       # create | alter | use | can-create | ensure
        :%fields,
        Str :$field-mode = 'Automatic',
        Str :$overflow-field-name,
        *%params
        --> Mu                      # table handle
    ) { ... }

    method relation-capabilities(--> Associative) { ... }
}

Table handle MUST provide at minimum:

4.4 Capability advertising

Suggested keys (illustrative):

Location swap (Memory → Postgres, etc.) MUST fail explicitly if a required capability is missing — no silent FieldMode degradation.

4.5 Built-in expectations (first slices)

4.6 Documentation updates

• Adding-Sources-Destinations — Source/Destination on tables; row vs blob; Relation; capabilities.

• Location base pod — Walkable, Pull/sink, Relation.

• Cross-link Streaming proposal and prerequisites TypedIterator.

• WalkCursor Extending notes already cover the tree landing; Relation docs should point Memory/SQL cursors at the same contract.

4.7 Acceptance criteria (Relation)

• Relation discoverable without a Storage namespace.

• Memory use-table supports at least use / can-create / ensure.

• Row Iterator from Relation handle matches structured Source pull for the same backend (shared path; same .of).

• File as Relation fails clearly; Memory as blob-only Source may be absent.

• Extending docs updated.

• Memory (and later Postgres) Walkable scan uses WalkCursor on Axis.preceding-following where random/relative row motion is advertised.

5. Anonymous, swappable Tables

5.1 Design rule

my $t = Table.new;   # lightweight Memory Location by default; name/Database optional
# later, preferably only the declaration changes:
my $t = Table.new(:location('postgres://...'), :name<countries>, ...);

• Database optional for single-table / anonymous use.

• Named Database for multi-table stores / URI grouping.

• No TOP unnamed Memory singleton as the primary model.

Namespace follows packaging option C in the plan; this spec requires behaviour.

5.2 Front-end responsibilities

• Optional name / Database metadata.

• Delegates storage to Relation handle (§4); row I/O consistent with Source/Destination on that location (§3).

• TOP-like methods across backends (add-row, fields, parse / format, … per plan §10.2), sharing Format/Location factories with operators.

• Location swap preserves API; capability mismatches fail per §4.4.

5.3 First-class query / walk / schema origin

5.4 Acceptance criteria (anonymous Tables)

• Anonymous Memory Table.new without Database/name.

• TOP-style row/field access; FieldMode Automatic on Memory.

• Usable as Walker/query origin without Catalog wrap.

• Schema.discover (or adapter) accepts it.

• :location to relation-capable backend works; blob-only location fails clearly.

• Documented sketch: anonymous Memory → redeclare with Postgres location.

6. WalkCursor backends (TOP)

Tree contract is done in core (Cursor-Proposal.md, prerequisites §5.6). Still open here:

• Memory / loaded-table WalkCursor on Axis.preceding-following (ordered row sequence).

• Postgres (or SQL) Location cursor mapping FETCH / MOVE onto fetch-* / move-*.

• Metanoia Walkable-as-cursor-factory (deferred in Cursor proposal).

• Whether Relation handles mint WalkCursor directly or only via Walker / Plan.cursor.

• Mapping TOP’s Cursor::Storage::Postgres onto WalkCursor (0-based Raku vs 1-based SQL adaptors).

7. Implementation order

After prerequisites (streaming, TypedIterator text path, WalkCursor tree, QueryIterator does TypedIterator):

1. Relation base + Memory Relation sharing row I/O with structured pull; pick row .of; Memory WalkCursor on Axis.preceding-following when needed.

2. Destination row sink + Parse skip for structured .of (§3.1.1).

3. Minimal Table front-end anonymous-first (§5).

4. Walker / Schema.discover hooks for lone Table.

5. Tests: File≠Relation; Memory pull yields rows; Destination writes rows; capability failures; no Str dump.

6. Plan sequencing: HalfHuman, CSV Location, Postgres + SQL WalkCursor; more Format pull backends. Full Supply push remains deferred.

Do not ship Relation + Table while table Sources would still be forced through walkable-Str-only dumps.

8. Open points

• Final name for Relation vs Table.

• How URI encodes table identity (postgres://db#table vs path vs params).

• Destination write policy: append vs replace vs upsert (per backend capability).

• Memory URI vs constructor-only.

• Database as Location grouping vs front-end object vs both.

• Exact exception types for capability failures / structured-on-blob Destination.

• FieldMode plugin loading site.

• Which type object row streams advertise in .of before Tuple lands (Associative vs dedicated Row) — decide when Memory Location is implemented.

• How far Format pull must go before CSV Location (structured pull is enough to start CSV Location).

• Whether Tree/Table QueryIterators override .of beyond default Mu.

9. References

• Qwiratry-TOP-Plan.md — especially §3–4, §6–8, §10

• Qwiratry-TOP-Prerequisites.md

• Cursor-Proposal.md

• lib/TypedIterator.rakumod, lib/Qwiratry/QueryIterator.rakumod (QueryIterator → TypedIterator in prerequisites)

• Streaming-Qwiratry-Proposal.md

• Adding-Sources-Destinations

• lib/Qwiratry/Table/Schema.rakumod, Table.rakumod (existing Catalog — not the TOP Table front-end)

• Raku TOP reference tree: ~/src/TOP/Raku-TOP/main (read-only)