Rebuilding Raku TOP on Qwiratry

Working plan for rewriting Raku TOP against Qwiratry. Reference tree for this plan: ~/src/TOP/Raku-TOP/main (read-only; do not edit it from here).

This is a design note, not a commitment to package layout or API names. We will refine it as we implement.

Prerequisites spec (normative for core I/O foundations): Qwiratry-TaOP-Prerequisites.md.TaOP layer (Relation, Memory, Postgres, Tables): Qwiratry-TaOP-Basics.md.

1. Why rebuild

TOP is Table-Oriented Programming: Database, Table/Relation, Field, Tuple, plus:

• Parsers / Formatters for tabular text (CSV, HalfHuman, WithBorders)

• Storage backends (Memory, CSV stub, Postgres via DBIish)

• Field modes (Automatic / Error / Overflow) that decide what happens when a row has columns not in the schema

Qwiratry already owns the broader DOP pipeline: Query AST, Walkers, Strategies, Transformers/Molds, Formats, Locations, and table navigation via Qwiratry::Table::Catalog / Schema. TaOP should become the relational object front-end and storage story on that pipeline, not a parallel mini-framework.

Today’s Qwiratry Table catalog is about FK-aware navigation over row collections. TaOP's Table is a richer mutable relation with schema, storage actions, and format hooks. Both should coexist: Catalog for walking; TaOP-style Table for owning rows and backends.

2. What TaOP has today (inventory)

2.1 Core object model (lib/raku/TOP.rakumod)

Table actions (create / alter / use / can-create / ensure) control whether storage must exist, may alter schema, and whether fields are supplied. Keep these; they are one of TaOP’s useful distinctions from “just open a file”.

2.2 Formats as Parse / Format (TaOP naming)

In TOP, Table::Storage.format / parse load formatter/parser classes and drive them against the live table. In Qwiratry, that responsibility splits cleanly into Format plugins plus optional Location I/O.

2.3 Storage

Table::Storage always installs a FieldMode and implements makeTuple / add-field / format / parse.

2.4 Field modes

Entry point: vet-for-tuple before constructing a Tuple.

3. Qwiratry mapping

3.1 Separation Qwiratry already got right

TOP blurred three concerns that Qwiratry should keep distinct:

1. Representation (CSV / HalfHuman / borders) → Qwiratry Format (Parse / Render)

2. Where the data lives and how we talk to it (file, HTTP, Memory, Postgres, …) → Qwiratry Location

3. Relational front-end (Database, Table, Field, Tuple, FieldMode) → TaOP layer on top of Location + Format

Today’s Location contract is only whole-value text I/O (read → Str, write(content)). That is enough for simple File/HTTP blobs, but not for live tables or for Source/Destination on tables. The plan is to upgrade Location so the same backend family exposes blob and/or row pipeline I/O plus optional Relation handles — not a parallel Storage hierarchy. Streaming-shaped I/O comes first (Qwiratry-TaOP-Prerequisites.md); Relation/Tables are TaOP Basics (Qwiratry-TaOP-Basics.md). See §4.1.

3.2 Proposed layers

  Query slang / Walkers / Catalog navigation
                 ↑
         TaOP relation surface
      (Database, Table, Field, Tuple, FieldMode)
                 ↑
     Location backends (upgraded; see §4.1)
   (Memory | CSV Location | Postgres | later SQLite, .ods, …)
                 ↑
        Formats (when serialising to/from text)

I/O pipelines stay (payload may be text or rows, depending on Location capability):

Source ↗ → Parse? → (query / transform) → Render? → Destination ↘

Table-native work stays:

$t = Table.new(...)  → Location (row-capable) ←→ Walker / Catalog / Feed operators
# or Database.useTable when a named multi-table store is wanted

3.3 Name mapping (TaOP → Qwiratry)

Exact distribution follows packaging option C; see §9.

4. What changes Qwiratry needs

Changes to make in Qwiratry before (or as the foundation of) the TaOP rebuild, so the architectural model is right. Items here are Qwiratry-core concerns; TaOP formats and backends then plug into them.

4.1 Locations for blobs; Qwiratry::DatabaseStorage for tables

Normative detail: Qwiratry-TaOP-Basics.md (TaOP layer). Core I/O foundations: Qwiratry-TaOP-Prerequisites.md.

Supersedes earlier “Location absorbs Storage”: Basics DQ2 / DQ18. Blob I/O stays on Qwiratry::Location::* (File, HTTP, …). Live tables use Qwiratry::DatabaseStorage::* (Memory, Location-backed, CSVdemo, CSV/SQL siblings) — TOP-shaped Database::Storage / Table::Storage two-level split (Basics DQ13). Tables are navigable objects, not Location URIs.

Today’s Location contract is whole-value / streaming text I/O. That is right for File/HTTP blobs. It is not the table backend.

Decision: separate Storage hierarchy for tables.

• Location — URI/path Source/Destination for blobs (and optional text pull/sink)

• Qwiratry::DatabaseStorage — table backends: useTable, fields, FieldMode, row scan/mutation, optional WalkCursor; Destination row writes honour DestinationMode when pipelines target tables (Basics)

• Format — encoding (HalfHuman, WithBorders, CSVdemo, JSON, …)

Streaming-shaped Location I/O remains a prerequisite for blob pipelines (Qwiratry-TaOP-Prerequisites.md). Table row streams live on DatabaseStorage / Table handles (Qwiratry-TaOP-Basics.md).

Why this fits

• Location stays “bytes/text at a URI”; DatabaseStorage owns relational storage.

• TableStorage::Location / DatabaseStorage::Location subclasses may compose Location + Format (e.g. JSON file → rows) without making the Table itself a URI.

• Source/Destination operators still work for files; table Destinations map onto table mutation / sinks (Basics OQ14).

• Aligns with TaOP front-ends (Table, Database, Field, Relation) and bare names (Table.new) — Basics DQ16.

Risks to watch

• Do not conflate table actions (can-create, …) with Destination modes (Basics OQ14).

• I/O operators must not .Str-dump rows onto blob Locations.

• Keep Format separate; full CSV tables are raku-Qwiratry-DatabaseStorage-CSV, not graduated CSVdemo.

4.2 Anonymous, swappable Tables as first-class values

Normative detail: Qwiratry-TaOP-Basics.md §5. Prerequisites: Qwiratry-TaOP-Prerequisites.md.

Support tables that live in ordinary variables with no Database and no name required:

my $t = Table.new;                    # lightweight Memory DatabaseStorage by default
# ... later, ideally only the declaration (or storage/database args) changes:
my $t = Table.new(:database($db), :name<countries>, ...);

Walkers, Catalog discovery, and Format render must accept a lone Table / Relation as an origin, not only a Catalog of named tables. Verify and close gaps in Table::Schema.discover / table Walker so anonymous tables are first-class. Database-as-LHS navigation goes through Walker::Implementation::Table (Basics DQ17).

Capability gaps across backends are expected (Memory ≠ SQL). Prefer a capability check so unsupported features fail clearly when the backend is swapped, rather than silently degrading.

4.3 Soft prerequisites (nice before heavy backends)

• Row streaming + Destination on tables — required before treating Table/DatabaseStorage as done; see prerequisites (TypedIterator) and Qwiratry-TaOP-Basics.md. Format chunk streaming can trail.

• raku-Qwiratry-Format-CSV (option C): Format::CSV + DatabaseStorage::CSV via Text::CSV. Do not graduate core CSVdemo. WithBorders is a separate Format sibling that depends on Format::CSV — see Textish.

4.4 Explicitly not required as Qwiratry rewrites

• Format / Location discovery factories — keep for blobs/encodings.

• Tree Navigators / Match / RakuAST — unrelated to TaOP table surface.

• Query slang syntax for in-memory table walking — already the right shape; richer DatabaseStorage backends and (later) SQL pushdown extend it.

• Custom multi-dialect SQL encoding (Red-style) — deferred; dialects via DBIish + SQL::Abstract (Basics DQ2b). Short term still one SQL sibling package.

5. Formats to rebuild

Follow docs/rakudoc/Extending/Adding-Formats.rakudoc: modules under Qwiratry::Format::* with ::Parse / ::Render, discovered by Qwiratry::Format.

5.1 HalfHuman (core)

Normative implementation detail: Qwiratry-TaOP-Textish.md.

• Parse: whitespace-split lines → ordered field values; when used with a typed Table, add-row / FieldMode decides naming (Automatic → A, B, … or existing headers if we add a header line convention).

• Render: port TOP::Formatter::HalfHuman (prepare-table width scan → header/row sprintf). Options: :show-headers.

• Portability note: HalfHuman is TaOP's “Unix command output” dialect; good core citizen, few deps.

5.2 WithBorders (raku-Qwiratry-Format-WithBorders)

Normative implementation detail: Qwiratry-TaOP-Textish.md.CSV prerequisite: Qwiratry-CSV-Plan.md.

• Render only (no parser in TaOP today). Dist: raku-Qwiratry-Format-WithBorders (not core; not Format-CSV). Depends on Format::CSV for the box catalog. HalfHuman remains core.

• Options: :show-headers, :outer-line-type, :inner-line-type (Light / Double / Heavy).

• Box-drawing character table: load BoxDrawingCharacters.csv via Qwiratry::Format::CSV into a Table — no static glyph hash. Do not use Format::CSVdemo.

5.3 CSV is Format + DatabaseStorage (not a graduated CSVdemo)

Normative detail: Qwiratry-CSV-Plan.md.

• Do not graduate CSVdemo. Keep Qwiratry::Format::CSVdemo as the small built-in demo Format for I/O pipeline experiments. Qwiratry::DatabaseStorage::CSVdemo is a separate basic table demo backend.

• Production CSV: sibling raku-Qwiratry-Format-CSV provides Qwiratry::Format::CSV (Text::CSV Parse/Render) and Qwiratry::DatabaseStorage::CSV (Location-backed tables).

• Pipelines that only need a tiny demo may keep Format CSVdemo; real CSV and WithBorders catalog load use Format::CSV.

• Prefer streamable row access on DatabaseStorage::CSV; materialise when necessary (§10.3).

5.4 Later formats (accept / make)

From TaOP's matrix: HTML, Spreadsheet, Pod6, richer JSON. JSON already lives in raku-Qwiratry-Format-JSON; TaOP rebuild should consume that package rather than reimplement. Tree-shaped formats use existing Navigator extension point. Follow-up: Post-SQL-TaOP-Improvements.md §E.3.1.

5.5 Formatter protocol differences

TOP formatters are stateful multi-phase: prepare-table, output-header, output-row, output-footer, accumulate $.output.

Qwiratry renderers are one-shot: render($data, :%options --> Str).

Port strategy:

1. First implementation: whole-table render matching current Format base (simple, matches CSVdemo/JSON).

2. Optionally keep an internal phased helper for WithBorders/HalfHuman (scan widths, emit borders) inside render.

3. When streaming lands (Streaming-Qwiratry-Proposal.rakudoc), expose row iterators for large tables without changing the discovery model.

Follow-up (public phase API Consider): Post-SQL-TaOP-Improvements.md §E.3.2 / §B.

6. DatabaseStorage backends: Memory, CSV, SQL

TaOP's Database::Storage::* / Table::Storage::* become Qwiratry::DatabaseStorage::* (Basics DQ2 / DQ13) — not upgraded Locations. Formats stay for encodings that are not themselves table stores (HalfHuman, WithBorders Render, demo Format CSVdemo). Full CSV tables are a DatabaseStorage sibling package (§6.3).

6.1 What DatabaseStorage must provide

TaOP Storage needed (port method surface largely as-is — Basics DQ12):

• create / open / alter tables (useTable actions)

• Positional rows, Associative fields

• FieldMode-aware makeTuple

• SQL cursor / potential key-based pagination (SQL sibling)

• type mapping (Raku ↔ SQL) where relevant

• optional WalkCursor (Memory first cut includes it — Basics DQ19)

Pipeline Destination modes on tables are a separate concern from table actions (Basics OQ14). Blob Locations stay URI Source/Destination only.

Short term: implement Memory first; CSV and SQL as sibling packages (§6.3–6.4). Dialects via DBIish + SQL::Abstract (Basics DQ2b).

6.2 Memory

Port Database::Storage::Memory / Table::Storage::Memory first as in-process Qwiratry::DatabaseStorage::Memory (constructor-only; no memory:// table URI).

• Anonymous tables in variables are the default happy path. A table need not be named or attached to a Database.

• Named Databases remain useful for multi-table stores and navigation roots ($db ⪪ <table>).

• Avoid TaOP's unnamed Memory singleton as the main model.

• Full in-memory rows; no blob I/O.

• Primary target for tests, FieldMode, and Memory WalkCursor.

• Should work with Table::Schema / Catalog and as a lone Walker origin (see §4.2); navigation via Walker::Implementation::Table (Basics DQ17).

6.3 CSV DatabaseStorage module

Normative detail: Qwiratry-CSV-Plan.md.

CSV tables are a dedicated DatabaseStorage backend (sibling package under option C):

• Package: raku-Qwiratry-Format-CSV → Qwiratry::Format::CSV and Qwiratry::DatabaseStorage::CSV (Text::CSV)

• Database ≈ directory (or explicit file list)

• Table ≈ one CSV file

• Actions: use / ensure / create map to file existence and header writing

• Reads: prefer streamable row iteration; materialise when necessary (§10.3)

• WithBorders (separate dist) uses Format::CSV for its box catalog — see Textish

Qwiratry::DatabaseStorage::CSVdemo in core is a basic demo only — not full CSV. Current TaOP Database::Storage::CSV is a stub; rewrite rather than salvage.

6.4 SQL DatabaseStorage (sibling)

Package: raku-Qwiratry-DatabaseStorage-SQL → Qwiratry::DatabaseStorage::SQL (DBIish + SQL::Abstract; multi-dialect via those libs — Basics DQ2b).

Connection strings may appear on Database construction; individual Tables are still not Location URIs.

6.5 Capability matrix (target)

7. Field modes and similar objects

Keep FieldMode as first-class plugins:

• Load via the same discovery style as Format/Location (Qwiratry::…::FieldMode::*).

• Install on DatabaseStorage / per-table storage at construction (:field-mode, :overflow-field-name) — Basics DQ7.

• Call from makeTuple only; parsers/formatters should not reimplement column invention.

Objects we expect to keep in a similar shape:

Deferred TaOP concepts (still from README object model): View, TupleSet, Section, Lot, DataDictionary — design hooks only until Table + DatabaseStorage solidify. Follow-up: Post-SQL-TaOP-Improvements.md §E.3.3.

8. Integration with query, walkers, streaming

• Memory / loaded CSV tables should be excellent Walker::Implementation::Table citizens: rows as Associatives/Tuples, Catalog for multi-table / Database roots (Basics DQ17).

• SQL DatabaseStorage: expose cursors as WalkCursor / Iterators; dialects via DBIish + SQL::Abstract. Follow-up: Post-SQL-TaOP-Improvements.md §E.3.4 / §E.1.2.

• Blob I/O path remains valid: file://data.txt → Format Parse → query → Render → File Destination.

• Keep the TaOP object/method API (Table, Field, Database, including parse / format) as a first-class interface; pipelines are complementary. Methods must share Format factories with operators (§10.2).

9. Packaging and migration

Decision: option C (hybrid). Core gets bare Table/Database/Relation/Field + FieldMode + Memory DatabaseStorage + HalfHuman + CSVdemo; heavier pieces (raku-Qwiratry-Format-CSV, raku-Qwiratry-Format-WithBorders, raku-Qwiratry-DatabaseStorage-SQL) live as siblings like Qwiratry-Format-JSON / Qwiratry-Location-HTTP. Leave Format CSVdemo as a demo Format — do not graduate it.

Suggested sequencing (editable):

1. Design freeze of Basics + this plan with you.

2. Prerequisites already landed (TypedIterator, streaming Location blob I/O, WalkCursor tree).

3. FieldMode + Memory DatabaseStorage + Table/Database/Field/Tuple (anonymous tables first; Memory WalkCursor — Basics DQ19).

4. raku-Qwiratry-Format-CSV — Format::CSV + DatabaseStorage::CSV (Qwiratry-CSV-Plan.md).

5. Formats: HalfHuman in core; WithBorders in raku-Qwiratry-Format-WithBorders (Qwiratry-TaOP-Textish.md).

6. raku-Qwiratry-DatabaseStorage-SQL (DBIish + SQL::Abstract) — Post-SQL-TaOP-Improvements.md §E.1.2.

7. Migrate TaOP tests / demos; retire or thin-wrap old Raku-TOP — Post-SQL-TaOP-Improvements.md §E.3.5. Compatibility: prefer clean Qwiratry-era module paths with bare front-end exports; no requirement for drop-in Database::Storage::* module names unless a shim helps.

10. Decisions

1. Anonymous tables. Allow tables that just live in variables (lightweight Memory DatabaseStorage by default). Naming or attaching a Database should ideally change only the declaration. Do not require a Database for every Table; do not revive TaOP's unnamed Memory singleton as the main model. Capability checks on backend swap (§4.2, §6.2).

2. TaOP object/method API stays; bare names. Table.new(...), Database, Field, Relation (Basics DQ16). Pipelines complementary. Object methods share Format factories with operators (§2.1–2.2, §5.5, §8).

3. CSV DatabaseStorage: streamable preferred. Prefer streamable row access for Qwiratry::DatabaseStorage::CSV; materialise when necessary. See §5.3 / §6.3.

4. FieldMode Overflow on SQL → JSON/JSONB. Support Automatic, Error, and Overflow as far as each backend allows. Do not silently degrade Overflow on swap (§4.2). Follow-up: Post-SQL-TaOP-Improvements.md §E.3.6.

5. SQL dialects. DBIish + SQL::Abstract (Basics DQ2b); no Qwiratry-local Red-style encoder. Multi-dialect enrichment: Post-SQL-TaOP-Improvements.md §E.2.1.

6. Package split. Option C (hybrid). See §9.

7. Tuple / column order — reading note. There is no “insertion-ordered Hash” in the Raku language today. Ordinary hashes deliberately do not preserve order; since Rakudo 2018.05, iteration order is randomised per invocation. Read:

   • Hashes and maps — documents that key/value order cannot be relied on.

   • class Hash — same guarantee of randomised order.

   • Hash::Ordered (what TaOP already uses for Tuple) — ordered Associative; also discussed for core as is Hash::Ordered in rakudo#5779.

   Practical choices are still Hash::Ordered (or equivalent) and/or explicit field order on the Table. Default lean: keep Hash::Ordered for Tuple unless we later adopt a core ordered-hash form.

8. DatabaseStorage for tables; Location for blobs. Supersedes “Location absorbs Storage.” See §4.1; Basics DQ2 / DQ18.

9. CSV sibling is raku-Qwiratry-Format-CSV (Format::CSV + DatabaseStorage::CSV). Do not graduate CSVdemo. WithBorders is raku-Qwiratry-Format-WithBorders (Textish). Specs: Qwiratry-CSV-Plan.md, Qwiratry-TaOP-Textish.md.

11. Immediate next steps (when we start coding)

• 1. Qwiratry-TaOP-Basics.md: Memory DatabaseStorage + bare Table/Database/Relation/Field; shared row paths; Memory WalkCursor.

• 2. Skeleton: FieldMode + Memory + minimal Table (anonymous-first) / optional Database.

• 3. Port a few Memory + FieldMode tests from Raku-TOP.

• 4. Destination row writes for tables (resolve Basics OQ14) + Parse skip policy (OQ15). (OQ14/OQ15 → DQ14/DQ15: DestinationMode row sinks; Parse throws ElementType.)

• 5. raku-Qwiratry-Format-CSV — Format::CSV + DatabaseStorage::CSV. Spec: Qwiratry-CSV-Plan.md.

• 6. Add HalfHuman (core) + WithBorders (raku-Qwiratry-Format-WithBorders); wire File → Parse → Table → Render example. Spec: Qwiratry-TaOP-Textish.md. HalfHuman + WithBorders Render landed; Textish §9 steps 1–13 done (ASCII :block-name, shared :header-alignment + prepare/sprintf via Qwiratry::Format::Textish::RenderWalkable, File pipeline tests); follow-up §9.14 remains (optional streaming). Design notes: Textish §8; decision TQ6 in Textish §12.

• 7. Pull Aggregate Functions stuff into Qwiratry (including aggmap and aggreduce). Spec: Qwiratry-AggregateFunctions-Plan.md. Done (Driver, catalogue first cut, Deduplication, body rewrite). Further catalogue / parallel / Relation-mutation follow-ups: Post-SQL-TaOP-Improvements.md §A.

• 8. Wire ⌿ → aggreduce; add ⇶ → aggmap (ordinary map + optional windowing); sort out unicode map/grep slang (AggregateFunctions Plan WQ5 / WQ7 / WQ8).

• 9. Extract remaining features from this plan and the SQL-Features-Plan.md and Qwiratry-TaOP-Textish.md and put them into some kind of TODO list (or maybe Qwiratry-TaOP-Advanced.md) Done for the main deferrals — collected in Post-SQL-TaOP-Improvements.md (§A Aggregate, §B Textish, §C Basics, §D SQL-Features, §E this plan’s remaining next steps + former §12).

• 10. Update Implementation::Loader… — moved to Post-SQL-TaOP-Improvements.md §E.1.1.

• 11. raku-Qwiratry-DatabaseStorage-SQL… — moved to Post-SQL-TaOP-Improvements.md §E.1.2.

12. Future plans

Moved to Post-SQL-TaOP-Improvements.md §E.2 (including former §12.1 Red-style multi-dialect encoding, §12.2 other later items, §12.3 SQL Features pointer).

Source trees consulted: Raku-TOP/main (TOP, FieldMode, Formatter, Parser, Database::Storage::{Memory,CSV,Postgres}); Qwiratry core (Format, Location, Table/Schema, Operator::IO, Streaming proposal, Developing TODO).