Qwiratry TaOP Textish Formats

Spec for porting human-readable text-table Formats from Raku-TOP: HalfHuman (core) and WithBorders (sibling dist). Not production CSV / Spreadsheet / HTML / Pod6 / rich JSON.

Parent plan: Qwiratry-TaOP-Plan.md (§5, §11 steps 5–6).Basics (Table / Format factories): Qwiratry-TaOP-Basics.md.CSV prerequisite for WithBorders: Qwiratry-CSV-Plan.md (in the Format-CSV sibling).Extension guide: Adding-Formats.rakudoc.

Status: Textish §9 steps 1–13 complete (HalfHuman + WithBorders + ASCII :block-name + shared :header-alignment / prepare helpers via CQwiratry::Format::Textish::RenderWalkable + File pipeline tests). Follow-up §9.14 remains (optional streaming). Plan §11 step 6 done.

1. Goals

1. HalfHuman as a first-class core Format (Parse + Render) — TaOP’s whitespace-aligned “Unix command output” dialect.

2. WithBorders as a Format (Render only) in raku-Qwiratry-Format-WithBorders — Unicode box-drawing CLI tables; depends on Format::CSV for the box catalog.

3. Usable from pipelines and from Table.parse / Table.format where the Formats are installed (shared Qwiratry::Format.make — plan §10.2).

4. Preserve TOP’s useful ideas (width scan, type-aware alignment, phased emit inside one-shot render, data-driven box glyphs) without copying TOP’s Storage-owned formatter protocol as a public Qwiratry API.

5. Keep CSVdemo as the demo CSV Format; WithBorders uses Format::CSV, not CSVdemo (Qwiratry-CSV-Plan.md).

2. Non-goals

• Implementing Format::CSV / DatabaseStorage::CSV (CSV Plan).

• HTML, Spreadsheet / .ods, Pod6, richer JSON (Plan §5.4; JSON already in raku-Qwiratry-Format-JSON).

• Changing SourceOperator to return Iterators; File → Parse stays walkable unless both sides advertise pull.

• A public multi-phase formatter API on Format base (prepare-table / output-row / …). Phasing stays internal to each Render implementation (Plan §5.5).

• Perfect HalfHuman round-trip guarantees (TOP itself is imperfect — see §5.3).

3. Packaging

Core does not depend on Format-CSV or Format-WithBorders.

META6 (WithBorders dist): provides for WithBorders; resource BoxDrawingCharacters.csv; depends on Qwiratry + Format-CSV sibling (or whatever provides Qwiratry::Format::CSV).

4. Source map (Raku-TOP)

Consult these when implementing (ideas and golden behaviour):

There is no TOP::Parser::WithBorders. CSV parser used by WithBorders’ catalog load is TOP::Parser::CSV — Qwiratry equivalent is Format::CSV (CSV Plan).

5. HalfHuman

5.1 Dialect rules (from TOP pod)

Port these as documentation and soft conventions (not all are hard-enforced by TOP today):

• Columns are whitespace-separated.

• Cells ideally contain no spaces (suggest _).

• Prefer a visible placeholder over a completely blank column: Render emits - for undefined cells (TQ3; TOP’s pod suggested this but its Render used '').

5.2 Render — Qwiratry::Format::HalfHuman::Render

Contract: render-walkable(Mu $data, *%options --> Str) (optional RenderPull later).

Input shape: Positional of Associative rows (Table.list, AoH, or Seq). Field order preference:

1. Ordered names from options (:fields) if provided, else

2. Table / first-row key order when keys are ordered (Tuple / Hash::Ordered), else

3. Sorted keys (document that unordered Hash loses column order).

Options (TOP-compatible):

Internal algorithm (TOP phases folded into one-shot render — Plan §5.5):

1. prepare — scan rows; per column track max display width and type ladder Nil → Int → Num → Str (TOP’s widening rule).

2. Build a space-joined sprintf format: Int columns right-aligned (%Ns), others left (%-Ns).

3. header — if :show-headers, emit field names with alignment from :header-alignment (HalfHuman default left-int-right — same as body: Int columns right, others left; see §8.2).

4. rows — emit each row in field order; undefined → - (TQ3).

5. Return accumulated Str.

Known TOP quirk — improve: HalfHuman’s prepare scan does not fold header name widths into column maxes (WithBorders does). Prefer WithBorders’ behaviour: if showing headers, seed widths from header lengths so long names do not misalign.

5.3 Parse — Qwiratry::Format::HalfHuman::Parse

Contract: parse-walkable(Str $input --> Mu) → structured rows for pipelines; Table.parse then fills via fill_from_aoh / add-row.

Algorithm (port TOP TWEAK logic; Format must not require a live Table):

• Split into lines; skip blank lines (TQ2; TOP did not skip — blank lines became degenerate rows via split(/\s+/)).

• Each line: split on /\s+/, then valuate tokens (<< $_ >> or equivalent) so IntStr / NumStr appear where possible.

• Emit an array of lists (positional field values), matching TOP’s $!table.add-row(@fields) (TQ1). Table.parse / FieldMode Automatic then invents A, B, C, … when filling a fresh Table. Pipelines that need Associative rows can zip against known headers or letter names after parse.

Headers (Qwiratry extension):

TOP Parse has no header handling (first line is data — fits ls -l). Keep default False for TOP compatibility; document that Render with :show-headers then Parse without :headers treats the header as a data row.

TOP’s :filename / :command / :handle: those are Location / process Source concerns, not Format. HalfHuman Parse takes Str (or later pull chunks). Example shape:

my $t = Table.new;
$t.parse(qx{ls -l /usr}.out.slurp, 'HalfHuman');   # conceptual
# or: File Source → Parse HalfHuman → query → …

5.4 Round-trip expectations

• Render → Parse without :headers after Render with headers: lossy (header becomes a row) unless caller sets :headers.

• Spaces inside cells: lossy (split cannot recover).

• Prefer documenting dialect limits over inventing quoting in v1.

6. WithBorders

6.1 Role

Render only — Qwiratry::Format::WithBorders::Render. No Parse in TOP; none required for first cut (ASCII-art table parse is a later “accept” if anyone needs it).

6.2 Options (TOP-compatible)

6.3 Internal algorithm (port TOP)

Same prepare → header → rows → footer structure as HalfHuman, plus borders:

1. prepare — seed %maxes from header names when showing headers; scan rows with get-cell-string; build row sprintf using outer/inner vertical box characters as separators.

2. header — top border (Corner Top-* + T Top); optional header cells; separator (T Left/Right + Cross Orthogonal) using inner horizontal weight where TOP does.

3. rows — body lines via the vertical-separated format.

4. footer — bottom border (Corner Bottom-*).

Cell stringify (get-cell-string): Associative → .raku; defined otherwise stringify; undefined → - (same placeholder as HalfHuman TQ3). (TOP used '' and had a TODO for JSON-ish later — out of scope for v1 unless trivial.)

Alignment: same Int right / else left rule as HalfHuman.

6.4 Box-drawing catalog (data-driven)

Do not hardcode a glyph hash of corners/tees.

1. Ship resources/BoxDrawingCharacters.csv in the WithBorders dist (TOP’s catalog; columns include Char, Block Name, Overall Shape, Overall Position, Primary Weight, Secondary Weight, Style, Primary Shape, …).

2. On first WithBorders use (lazy singleton), load the resource into an in-memory Table via Qwiratry operator slang:

   • ⮳ $path ↱ "CSV" ⮷ $table — File Source → Qwiratry::Format::CSV Parse → Memory table Destination (Qwiratry-CSV-Plan.md)

3. get-character(:$containment, :$shape, :$position) ports TOP’s grep, filtered by :block-name (default Box; see §8.1 for ASCII + fallbacks):

   • Block Name = option value, Style = Solid for Box (for ASCII, Semi-Solid — match catalog rows),

   • weights from outer/inner options (ASCII weight mapping in §8.1),

   • For shape => 'T', secondary weight = inner line type (mixed joins).

4. Warn if more than one Char matches; return the character string. If none match, apply §8.1 fallbacks (do not hardcode a glyph hash of corners/tees as the primary path).

Do not use Format::CSVdemo for this path.

6.5 TOP TODOs worth tracking

From WithBorders source (carry as Qwiratry TODOs, not blockers). ASCII :block-name and shared :header-alignment are designed in §8 and scheduled as §9 steps 10 and 11; the third stays unscheduled:

• ASCII box-drawing via :block-name<ASCII> + catalog fallbacks — §9.10 / §8.1.

• Header alignment (left-int-right / left / right / centre) in core for HalfHuman + WithBorders (defaults HalfHuman=left-int-right, WithBorders=centre) — §9.11 / §8.2.

• Richer cell stringify than .raku for nested Associatives.

7. Qwiratry integration

7.1 Format module layout

# core
lib/Qwiratry/Format/Textish/RenderWalkable.rakumod
  Qwiratry::Format::Textish::RenderWalkable  is Base::RenderWalkable  (role; not a plugin)
lib/Qwiratry/Format/HalfHuman.rakumod
  Qwiratry::Format::HalfHuman::Parse  does ParseWalkable
  Qwiratry::Format::HalfHuman::Render does Textish::RenderWalkable

# raku-Qwiratry-Format-WithBorders
lib/Qwiratry/Format/WithBorders.rakumod
  Qwiratry::Format::WithBorders::Render does Textish::RenderWalkable
resources/BoxDrawingCharacters.csv

Follow Adding-Formats: META6 provides, discovery via Qwiratry::Format.make(:type, :format).

7.2 Shared factories (Basics / Plan §10.2)

Already landed:

• Table.format($name, *%options) → Format Render → Str

• Table.parse($text, $name, *%options) → Format Parse → replace rows

Textish Formats must work through those and through ParseOperator / RenderOperator without a second code path.

7.3 Pipeline example (Plan §11.5)

Target demo / test (§9.13) — landed as C<t/integration/textish-file-pipeline.rakutest>:

# File of HalfHuman text → Parse(:headers) → filter → Render HalfHuman or
# WithBorders → File Destination (text path; not row sink)

C forwards C<%.options> to C (so slang/AST can pass HalfHuman C<:headers>). WithBorders is exercised when the sibling C paths are present; otherwise that subtest skips.

Blob Destination still rejects structured .of (Basics step 6); Render to Str then write, or use table Destination when staying in rows.

7.4 Shared Textish RenderWalkable role

Header alignment (§8.2) and shared prepare/sprintf / field-order helpers live in core as CQwiratry::Format::Textish::RenderWalkable — a role that C CFormat::Base::RenderWalkable. HalfHuman and WithBorders Renders C it. The four-segment name keeps Format discovery from treating it as a plugin (discovery only loads three-segment CQwiratry::Format::* names).

§9.12 decision: fold the duplicated helpers into this role (not a sibling C). Border/box catalog logic stays WithBorders-only.

7.5 Streaming

First cut: walkable whole-value only (matches CSVdemo). Optional ParsePull / RenderPull can follow Streaming proposal once useful (§9.14); not required to ship HalfHuman / WithBorders.

8. Follow-up design (ASCII borders & header alignment)

TOP left both features as TODOs with no algorithm. This section is the proposed design for §9.10–11. Confirm or amend in §12 (Decisions) when implementing.

8.1 ASCII WithBorders (:block-name) (§9.10)

Option: :block-name — catalog column Block Name. Default 'Box' (Unicode box-drawing, current behaviour). 'ASCII' selects the ASCII art rows already in BoxDrawingCharacters.csv.

Keeps terminology aligned with the catalog (prefer this over a parallel “encoding” / “border-style” name).

Box path (unchanged): get-character greps Block Name = Box, Style = Solid, weights from :outer-line-type / :inner-line-type (Light / Double / Heavy).

ASCII path — Table lookup first: same get-character shape/position / containment API, but filter Block Name = ASCII and Style = Semi-Solid. Map line weights onto catalog Primary Weight / Secondary Weight:

Catalog today has Straight Horizontal/Vertical, Cross Orthogonal (+), Cross Diagonal (X), and Diagonal Forward/Backward (/ \) — not a full Corner / T grid. Prefer lookup over a hardcoded glyph hash; fill gaps with fallbacks.

Fallback chain (when the primary grep returns no rows):

1. Straight Horizontal / Vertical — already present; no fallback needed.

2. Cross Orthogonal (and T junctions) — fall back to Cross Orthogonal + if a T-shaped query misses.

3. Corner — fall back to diagonal straights (visual ASCII corners):

   • Top-Left, Bottom-Right → Diagonal Forward (/)

   • Top-Right, Bottom-Left → Diagonal Backward (\)

4. If still empty — last resort + for junctions, - / | for straights (document; should be rare once diagonals exist).

Warn if a lookup returns more than one Char (existing Unicode rule). Fallbacks should still be deterministic (at most one chosen Char).

Tests: golden for a small AoH with :block-name<ASCII>, default Double/Light weights, plus Light/Light; assert corners use / \ (or + only where the fallback chain says so). Keep Box goldens as the default path.

8.2 Header alignment (core, shared) (§9.11)

Scope: both HalfHuman and WithBorders. Implement the alignment helpers in core as role CQwiratry::Format::Textish::RenderWalkable (C) from the start so both Formats C the same role. Each Format sets its own default when the option is omitted.

Defaults:

Option: :header-alignment — string (case-insensitive normalize OK):

When :show-headers:

• Build header output with the chosen alignment (separate header sprintf or pad helper — not necessarily the body format string).

• Body rows keep Int-right / else-left regardless of :header-alignment.

• WithBorders border / separator lines unchanged (still based on %maxes).

Out of scope for v1: changing body-cell alignment via this option.

Tests: HalfHuman default golden stays left-int-right; WithBorders default goldens use centre (update existing Double/Light goldens if headers were left-aligned). Each Format also covers the other mode explicitly (:header-alignment<centre> on HalfHuman; :header-alignment<left-int-right> on WithBorders). Spot-check left / right. Body numeric columns stay right-aligned under centre.

9. Implementation order

• 1. Qwiratry::Format::HalfHuman in core — Render (widths + headers + sprintf).

• 2. HalfHuman Parse (whitespace lines → rows; :headers optional).

• 3. Tests: Render golden strings; Parse of Unix-ish lines; Table.format / Table.parse; pipeline File→Parse→Render.

• 4. Port golden cases from TOP xt/02-formatters-parsers.rakutest where they still apply (HalfHuman / ls path).

• 5. Scaffold raku-Qwiratry-Format-WithBorders (depends on Format-CSV).

• 6. Copy BoxDrawingCharacters.csv into WithBorders resources/.

• 7. Qwiratry::Format::WithBorders Render + lazy box-char Table load via Format::CSV.

• 8. Tests: default Double/Light borders match TOP-ish goldens; option matrix Light/Heavy; Table.format('WithBorders', …).

• 9. Docs: Changelog; mark Plan §11.6 done; cross-link Extending + this file.

• 10. ASCII WithBorders via :block-name<ASCII> + catalog fallbacks (see §8.1).

• 11. Shared core :header-alignment for HalfHuman and WithBorders (values left-int-right / left / right / centre; defaults HalfHuman=left-int-right, WithBorders=centre; see §8.2).

• 12. Shared prepare/sprintf / field-order helpers live on CQwiratry::Format::Textish::RenderWalkable (no sibling C; see §7.4 / §12 TQ6).

• 13. File pipeline tests (§7.3): File Source → Parse → filter → Render HalfHuman or WithBorders → File Destination (C<t/integration/textish-file-pipeline.rakutest>).

• 14. Optional streaming later (§7.5): ParsePull / RenderPull when the Streaming proposal makes them useful.

10. Acceptance criteria

• Core discovers HalfHuman; WithBorders dist discovers WithBorders when installed (with Format::CSV available).

• Table.format('HalfHuman') works from core; .format('WithBorders', …) works when Format-WithBorders (+ Format-CSV) are installed.

• Table.parse($hh-text, 'HalfHuman') fills rows; FieldMode Automatic works.

• Pipeline Parse/Render with HalfHuman round-trips simple tables when Parse uses :headers matching Render’s header line (document limits).

• File Source → Parse → filter → Render → File Destination for HalfHuman (and WithBorders when the sibling dist is on the lib path) — C<t/integration/textish-file-pipeline.rakutest> (§9.13).

• WithBorders loads glyphs from its resource CSV via a Table parsed with Format::CSV (not CSVdemo; not a static hash).

• Core does not depend on Text::CSV, Format-CSV, or Format-WithBorders.

• CSVdemo / DatabaseStorage::CSVdemo behaviour unchanged.

11. Open questions

(none numbered right now — see §12)

12. Decisions

Closed questions keep their TQ numbers (no separate TD renumbering).

• TQ1 — Parse row shape. What TOP does: TOP::Parser::HalfHuman splits each line on whitespace, valuates tokens (<< $_ >>), then calls $!table.add-row(@fields) with a positional list — not letter-keyed hashes. Column names come from FieldMode / existing fields (Automatic → A, B, …). Decision: Format::HalfHuman::Parse returns an array of lists of values (TOP-shaped). Table.parse / FieldMode name columns; with :headers, body rows may be AoH keyed by the header line.

• TQ2 — Blank-line policy. What TOP does: iterates $handle.lines with no blank skip; empty/whitespace-only lines still go through split / add-row (degenerate rows). Decision: skip blank lines (improve on TOP).

• TQ3 — Dash placeholder. What TOP does: Formatter pod suggests - for missing values, but add-row maps undefined cells to ''. Decision: Render emits - for undefined cells (shared CTextish::RenderWalkable C). Associative cells use C<.raku>.

• TQ4 — WithBorders dist. Dist is raku-Qwiratry-Format-WithBorders (separate from Format-CSV). Depends on Format::CSV for the box catalog; core stays free of both siblings. HalfHuman ships in core (no Text::CSV dep).

• TQ5 — Plan §11 order. CSV Plan (§11.5) before WithBorders; HalfHuman may land in core in parallel once Format APIs are clear.

• Box catalog via Format::CSV. Data-driven Table load; not CSVdemo; not a static glyph hash. (Packaging decision; not a numbered TQ.)

• TQ6 — Shared Textish helpers (§9.12). After WithBorders, prepare/sprintf / field-order / width-type scan / cell stringify duplicated between HalfHuman and WithBorders. Decision: fold into CQwiratry::Format::Textish::RenderWalkable (same role as header-alignment); do not introduce a sibling C. WithBorders keeps border/box-catalog methods locally. Associative cells stringify via C<.raku> for both Formats.

13. References

• Qwiratry-CSV-Plan.md — Format::CSV + DatabaseStorage::CSV

• Qwiratry-TaOP-Plan.md — §5 Formats, §5.5 protocol, §9 packaging, §11 next steps

• Qwiratry-TaOP-Basics.md — Table.format/parse, DQ15

• Adding-Formats.rakudoc — Format extension guide (HalfHuman, WithBorders sibling, discovery)

• raku-Qwiratry-Format-WithBorders sibling dist — t/01-withborders-behavior.rakutest

• Raku-TOP: TOP::Formatter::{HalfHuman,WithBorders}, TOP::Parser::HalfHuman, TOP::Parser::CSV, resources/BoxDrawingCharacters.csv, xt/02-formatters-parsers.rakutest

• Existing demos: lib/Qwiratry/Format/{CSVdemo,JSONdemo,NDJSONdemo}.rakumod