Streaming Qwiratry Proposal - v2

Follow-on design notes: walkable / push / pull, Raku mappings, and push streaming.

Continues from Streaming-Qwiratry-Proposal.md (which covers the original pull streaming proposal and the Streaming Rename map).

Streaming Thoughts

We should have two language models (stream/walk), and three APIs (walkable/push/pull).

Transformers

Transformers should support attributes; these can fulfil the role that accumulators fill in XSLT. They probably already do, but we should check. Transformers should support multicontext; see below.

Molds

Molds should support the following characteristics:

• Context node (/ or something)

• Sibling Limit: This is how far it's permitted to look at previous/next siblings before throwing an error. This is to help control memory consumption (for streaming behaviour). These are preceding-limit and following-limit.

  • Default: Inf. This is Walkable.

  • 0: This is like Streaming mode in XSLT

  • 1: This allows the immediately preceding/following sibling to be accessed, but no further (following-limit)

  • n: Any other number can be put here; I haven't seen a use for it, but someone might have one someday.

  • Note that nodes can always access ancestors, descendants, and attributes -- the above restrictions are for siblings.

• Multicontext:

  • Molds can be marked as multi-input or multi-output. This is to facilitate the processing of multiple streams in parallel.

  • If a mold/transformer marks itself multi-input, then we need to handle this as multiple contexts somehow; not sure how we do this; probably with extra parameters to the mold. .

• Questions

  • How do we integrate molds (or maybe mold protos) with the walkable/push/pull capabilities/APIs

The smart thing to do as a developer is to have a mold that's in streaming mode, and then it calls out to other molds (with different names) that are not in streaming mode.

Operators

We should be looking at having the walkable/push/pull APIs being capabilities, rather than roles; we should probably have the base one just throw exceptions, and then work from there. Also, there should be a fourth capability, "streaming", which is true if push or pull is true.

Future thoughts

• Consider allowing markers on loops, etc, to indicate that the loop items should be processed in order (cf. xsl:iterate vs. xsl:for-each) * Possibly also worthwhile being able to mark that things can be done in parallel in groups

Raku Push, Pull, and Walkable

A map of existing Raku features onto the three sectors used elsewhere in this document. Empty cells mean that feature is not really that model.

• Push — producer drives (Supply / SAX-ish).

• Pull — consumer drives (pull-one / StAX-ish).

• Walkable — whole value in hand, then navigate (DOM-ish).

Grammars are split into input vs output rows below (walkable input; push-like actions / walkable Match).

Keep for now: Queues, Grammars (input), Grammars (output). Other rows are still under discussion.

Model converters

How to convert from one model (source / column) into another (destination / row). Cells name what Raku already provides.

Qwiratry should make (helpers / typed bridges beyond bare Raku):

• A reusable Pull → Push pump that turns an Iterator (especially text-chunk iterators) into a Supply via Supplier, with clear done / error behaviour.

• A reusable Push → Pull adapter that exposes a Supply as an Iterator (likely Channel-backed), so existing pull parse/render paths can consume push sources without rewriting the query core.

• Thin result roles or types that mark text-chunk streams vs record/value streams, so destination/parse do not confuse a Seq of hashes with an Iterator of Str chunks.

• Operator-level conversion policy: when adjacent I/O stages disagree (push vs pull vs walkable), choose a named converter instead of ad-hoc ~~ Iterator checks.

• Optional Push → Walkable / Walkable → Push helpers aimed at adaptor boundaries (whole document ↔ Supply), so format authors are not inventing accumulation each time.

Push Streaming for Qwiratry

This section sketches a possible push-capable extension for Qwiratry. The current 0.10.0 implementation is fundamentally walkable + pull, with one push-ish edge already present in destination sinks.

The obvious Raku building blocks are Supply and Supplier:

• Supply is the push stream observed by consumers.

• Supplier is the producer handle that emits values into a Supply.

That suggests the following rough shape for a future push API family:

Some observations:

• DestinationPush is the least speculative piece, because current sinks already accept pushed chunks via write-chunk.

• SourcePush is the natural fit for sockets, callbacks, HTTP bodies, and protocol handlers that already arrive as events.

• ParsePush and RenderPush would probably work best as Supply-to-Supply adaptors, though internally they might still bridge through pull logic in a first implementation.

Bridging is likely to matter more than purity:

• Push -> Pull: buffer with Channel, or expose a pull Iterator view over arriving push events.

• Pull -> Push: pump pull-one results into a Supplier.emit loop.

• Push -> Walkable: accumulate all pushed values, then hand over one materialized structure.

• Walkable -> Push: emit one value, or emit a sequence derived from the whole value.

This means Qwiratry does not necessarily need a fully separate push-only query engine. A practical first step could be:

1. keep the current pull query core;

2. add push-capable source / destination / maybe parse / render surfaces;

3. bridge push into pull at the edge using Supply/Supplier plus a queue or pump;

4. only later decide whether native push query execution is worthwhile.

Open question: whether push should be expressed primarily as Supply, as Supplier-backed sinks/sources, or as both. My current guess is:

• expose Supply at public stream boundaries;

• use Supplier internally when Qwiratry itself is the producer;

• keep sink objects where explicit lifecycle (write-chunk, close) matters.

Data Models

Two Raku stream shapes that matter for push vs pull:

Supplier  ────emit────▶  Supply  ─────tap────▶ Consumer
Iterable  ──iterator──▶ Iterator ──pull-one──▶ Consumer