Qwiratry — Qwiratry-AggregateFunctions-Plan

Qwiratry Aggregate Functions Plan

Status: Writing.Role: Superseding implementation spec for Aggregate Functions inside Qwiratry. The standalone raku-AggregateFunctions/Specification.md remains historical reference only (DQ6 / WQ6) — do not edit that repo for this work.

Goal: Declare aggregate functions as classes, run them with aggreduce (collapsing / Postgres GROUP BY) and aggmap (row-preserving / Postgres WINDOW), drive item streams from Qwiratry Walkers / Iterators (not a private AggregateWalk), wire existing ReduceOperator (⌿) to aggreduce, and add ⇶ (map / optional WINDOW) that calls aggmap — ordinary per-item map by default, windowing when requested; no separate GroupByOperator / γ.

Companion docs:

Requirements: RAKUDO_RAKUAST=1 (body rewrite is RakuAST-only). use 6.e.PREVIEW is allowed for the rewrite hook.

Non-goals for the first landing:

Will do later:

  • SQL text generation

Will probably do later:

  • Query pushdown to SQL (backend)

Could do later:

  • Whole-tree nested body rewrite (done: §J.4 / §H.3)

  • FILTER on aggregate arguments (still later: §J.4.1)

  • Parallel processing of queries (including COMBINEFUNC / SERIALFUNC / DESERIALFUNC)

  • PARALLEL -- currently attribute only, unused

Probably not:

  • SORTOP (not unless there are performance problems)

  • Ordered-set / hypothetical-set aggregates

  • SSPACE / MSSPACE

  • TaOP Relation mutation APIs

Unsure:

  • FINALFUNC_MODIFY / MFINALFUNC_MODIFY

  • FINALFUNC_EXTRA as its own mechanism

Non-goals forever:

  • A separate GroupByOperator / γ (collapse goes through ⌿ → aggreduce)

  • Keeping a separate legacy MapOperator (».)

  • A sub sum that clashes with Raku’s builtin

  • unit statements (use if necessary, but prefer class/role declarations instead)

  • Editing or shimming raku-AggregateFunctions

  • Wrapping input items in a cursor type

  • Frames in the first Walker cut if aggmap without frames + Rank already prove the handover

Source of truth for SQL behaviour is Postgres. Method names are kebab-case (window-start, frame-enter, …).

Question ID scheme: Stable IDs from the AggregateFunctions Spec open/answered lists there are preserved here as AF-Qn. Qwiratry-integration decisions use WQ1–WQ8. The same AF number never appears as both open and decided.

A. Qwiratry integration

A.1. Motivation

raku-AggregateFunctions already defines the roles, catalogue stubs (Sum, Rank), drivers (aggreduce / aggmap), and a CHECK-time body rewrite. What it does not reuse is Qwiratry’s traversal stack — hidden AggregateWalk partitions and visits items itself.

Primary goal: keep the aggregate-function API; replace AggregateWalk with Qwiratry::Aggregate::Driver over an Iterator (Walker → QueryIterator when walkable; Iterable .iterator otherwise — WQ1).

Secondary goals:

  1. Route collapse through existing ReduceOperator (⌿) → aggreduce (no new GroupBy / γ node).

  2. Add ⇶ whose evaluator calls aggmap: ordinary per-item map when no windowing is requested; row-preserving WINDOW when window / sort / frame / aggregate columns are present (WQ8). MapOperator / ». is retired; ⇶ is the sole map operator.

  3. Later: SQL Location Walker pushdown of ⌿ / ⇶ without changing how authors declare Sum / Rank.

A.2. Module layout

Standalone todayQwiratry landing
AggregateFunctionQwiratry::AggregateFunction
MovingFrameAggregateFunctionQwiratry::MovingFrameAggregateFunction
AggregateFunctions (Sum, Rank, drivers, rewrite)Split under Qwiratry::Aggregate::… + export from Qwiratry

Suggested files:

lib/Qwiratry/AggregateFunction.rakumod
lib/Qwiratry/MovingFrameAggregateFunction.rakumod
lib/Qwiratry/Aggregate/Sum.rakumod
lib/Qwiratry/Aggregate/Rank.rakumod
lib/Qwiratry/Aggregate/Driver.rakumod
lib/Qwiratry/Aggregate/BodyRewrite.rakumod
lib/Qwiratry/Operator/MapReduce.rakumod   # extend ReduceOperator; add ⇶

Roles stay separately use-able and are not re-exported merely by loading the catalogue. AggregateFunction (role) ≠ driver/catalogue surface.

Entrypoint (WQ4): use Qwiratry; installs the body rewrite and exports &aggreduce, &aggmap, Sum, Rank, ….

A.3. Driver and Walkers

Walkers own traversal; aggregate instances own state. The Driver pulls walk items (table rows, XML elements, AST nodes, prior matches, … — not a Table-only row type).

Subject ──Walker (or Iterable.iterator)──▶ item stream
              │
              ├─ ⌿  / aggreduce ──▶ collapsed items
              └─ ⇶ / aggmap ──▶ mapped items
                                      (window annotate when requested)
  1. Façade accepts Iterable / walkable subject.

  2. Item stream: Walker → QueryIterator if walkable; else .iterator (WQ1).

  3. Driver takes Iterator (QueryIterator does Iterator); window-split, sort, lifecycle — not private tree/table navigation.

  4. Aggregate “previous” / frame history: Driver loop locals + window buffer (copy onto $.previous / …). Not Walker fetch-prior (walk order ≠ sorted window order).

Concrete Walkers must not grow private Sum / Rank logic; yield items and let the Driver run, or opt into pushdown via can-pushdown-aggregate / pushdown-aggregate (capability aggregation.pushdown; §H step 11). Table / Tree advertise aggregation enabled with pushdown False.

Strategy.finish / FinishResult are unrelated unless a Strategy explicitly collects matches then calls the Driver.

A.4. Language surface (operators)

Survey / Features PlanOperatorDriver
Partition + Aggregation (GROUP BY)existing ReduceOperator (⌿)aggreduce
Map + optional WINDOW⇶ (AggMapOperator)aggmap
Deduplication⑂ (DeduplicationOperator)none (not a fold; §A.6 / §H.3)

Decision (WQ5 / WQ7 / WQ8): Do not add GroupByOperator / γ. ⌿’s evaluator calls aggreduce. ⇶ (U+21F6) calls aggmap and covers both:

  1. Normal map — no window / sort / frame, and body has no aggregate columns → one output item per input item.

  2. Windowing — when window and/or sort / frame / Sum. / Rank. (etc.) are requested → row-preserving partition / frame walk (§E / §F).

HAVING remains σ after ⌿. Glyph ⇶ matches Operators.rakudoc §4 (WQ5). Legacy ». / MapOperator is removed.

Plain Raku aggreduce / aggmap remain available; the operators are the Query AST / slang entrypoints onto the same Driver.

A.5. First-landing cuts vs later

AreaFirst landing?
Roles + Sum + body rewrite + aggreduceyes
MovingFrameAggregateFunction + Rank + aggmap without framesyes
Sum in aggmapwhole-window broadcast (WQ3 / was AF-Q13)
Frames + running / sliding Sum / Avgyes (§F / §H step 10)
⌿ → aggreduce; ⇶ → aggmap (map + optional window; WQ7 / WQ8)after Driver
aggsub phaserslater (§G)
DeduplicationOperator / DISTINCTthird round (§A.6 / §H.3)
Whole-tree body walkyes (§J.4 / §H.3)
FILTER on aggregate argumentslater (§J.4.1)

A.6. DeduplicationOperator

Postgres DISTINCT / relational-algebra duplicate elimination: remove duplicate rows or values (deduplication). Not a fold (not ⌿) and not GROUP BY collapse. Survey: old-design-docs/SQL-Grouping-Spitballing.md (historical δ / “Unique”). Chosen glyph: ⑂ (U+2442 OCR FORK) — fork / converge metaphor for dedupe; not RA δ; not universe ⊙ (XNOR). Implementation order: §H.3.

  • Add DeduplicationOperator (⑂) as a first-class pipeline operator: same subject → item stream shape as other transformers; emit one item per distinct equivalence class (first wins, order-preserving).

  • Postfix ⑂ (no RHS): whole item (eqv).

  • Infix + Block (sort-shaped, like ⇅):

    • 1-arg key extractor → compare keys with cmp; Same ⇒ duplicate (DISTINCT ON-style).

    • 2-arg comparator returning Order → Same ⇒ duplicate.

  • Do not use ⑂ * (Raku Whatever-currying).

  • Keep it separate from aggreduce / aggmap: authors compose ⑂ then ⌿ (or ⑂ after map) rather than overloading collapse.

  • SUM(DISTINCT …) / COUNT(DISTINCT …) can wait on ⑂ (dedupe then fold) or on catalogue-aware distinct modes later; do not invent a second Driver path only for distinct aggregates.

  • SQL pushdown (§K.1) should map ⑂ → SELECT DISTINCT / DISTINCT ON when the Location supports it.

  • FILTER (WHERE …) on aggregate arguments is a different feature (row predicate into one aggregate); see §J.4.1, not Deduplication.

B. Terminology

  • A class that does AggregateFunction is a reusable aggregate function. An instance is one in-progress evaluation (one window, or one column in one aggmap call).

  • aggreduce / aggmap (via Aggregate::Driver) sort, split into windows, call lifecycle methods, then evaluate body. aggmap without windowing args is ordinary map: evaluate body once per item, no partition / frame / aggregate lifecycle (WQ8).

Window vs frame

Window = set of items sharing the same window key (window => { … }). Same categorisation as SQL GROUP BY / PARTITION BY: keys eqv. Postgres also uses “window” for the whole OVER (…) clause; here window is the keyed set only. sort = order inside that window; frame (§F) = which items inside the window count for this output item.

TermMeaning here
WindowItems with the same window key. Optional; omit ⇒ one window over all input.
FrameSubset of the current window for the current output item (ROWS / RANGE / GROUPS). aggmap only.
window parameterBlock that extracts the key.
$.window-labelCurrent key value on the instance.

What moves: the frame, not the window. Postgres “moving-aggregate mode” is a moving frame start (MINVFUNC).

  • aggreduce — walk the whole window, emit once (collapse). No frame.

  • aggmap — walk the window (or its frame), emit once per input item.

Parameters (both): window, sort (1-arg easy path; 2-arg like Raku sort), body, frame (aggmap only).

Hash items: use .<region> (not .region) unless items are objects.

C. Reference implementation state (standalone)

Historical note: describes raku-AggregateFunctions at the time of the standalone spec. In Qwiratry, land under Qwiratry:: paths in §A.2; there is no AggregateWalk — use Qwiratry::Aggregate::Driver.

C.1. Standalone tree (for reference)

lib/AggregateFunction.rakumod (→ lib/Qwiratry/AggregateFunction.rakumod)

  • No unit. Role AggregateFunction[::StateType].

  • Enums: ModificationType, ParallelState (Safe Restricted Unsafe).

  • Lifecycle kebab-case: window-start / window-end. row still a required stub.

  • $.state, @.parameters, $.current, $.previous, $.next, $.counter are is rw.

lib/MovingFrameAggregateFunction.rakumod (→ lib/Qwiratry/MovingFrameAggregateFunction.rakumod)

  • use AggregateFunction; (→ use Qwiratry::AggregateFunction;). Listed in META6.json provides.

  • Frame lifecycle: frame-start / frame-enter / frame-exit / frame-end; helpers trigger-frame-start / trigger-frame-enter (window key from the driver, not from the item; no set-value).

lib/AggregateFunctions.rakumod (→ Qwiratry Aggregate catalogue + Driver + rewrite)

  • class Sum does AggregateFunction[Numeric] (skip nulls; empty → Nil). Stub class Rank. Exported stub aggmap / aggreduce. Body rewrite / aggreduce walk is §D.3 / §J.2.

t/00-use.rakutest checks use of the three modules (adapt paths under Qwiratry).

C.2. Shared plumbing

  1. use, not unit. Authors use Qwiratry::AggregateFunction; and use Qwiratry::MovingFrameAggregateFunction; separately. Do not export those roles from AggregateFunctions. List each in META6.json provides so installs resolve. Filename already matches the use name. is export on the role only if use does not make it visible otherwise.

  2. Export from AggregateFunctions: aggreduce, aggmap, and the callable classes (Sum, Rank, later Count / Avg / DenseRank). No sub sum. Replace the fez description.

  3. is rw on attributes that aggreduce / aggmap assign: $.state, @.parameters, $.current, $.frame-state, $.window-label, and any $.previous / $.next / $.counter kept on the aggregate function.

  4. Kebab-case lifecycle: window-start, window-end, frame-start, frame-end, frame-enter, frame-exit, trigger-frame-start, trigger-frame-enter.

  5. Items stay ordinary. Do not add newpartition / window-value / set-value / counter / previous on items. The driver keeps $previous / $next / $counter as loop locals and copies them onto the instance ($.previous / …). No cursor type (AF-Q1).

  6. Tests: keep t/00-use.rakutest. Sum and Rank live in lib/Qwiratry/Aggregate/… (catalogue).

Rename map (old lib/ / spitball → this spec). Window-level fold is window-*; moving-frame extras are frame-*. Old window_start / window_end on MovingWindowAggregateFunction were MINITCOND / MFINALFUNC (now frame-*). Old run_start / run_end on AggregateFunction were INITCOND / FINALFUNC (now window-*).

OldNewConcept
MovingWindowAggregateFunctionMovingFrameAggregateFunctionthe role
run_start / run-startwindow-startINITCOND; once per window
run_end / run-endwindow-endFINALFUNC; once per window
$.window_state$.frame-stateMSTYPE
window_start (moving role)frame-startMINITCOND
window_enterframe-enterMSFUNC
window_exitframe-exitMINVFUNC
window_end (moving role)frame-endMFINALFUNC; optional
trigger_window_starttrigger-frame-starthelper
trigger_window_entertrigger-frame-enterhelper
parameter window(keep)key extractor Block
$.window_label$.window-labelcurrent key value
row(keep)SFUNC

D. aggreduce

Postgres analogue: SELECT … agg() … FROM t GROUP BY keys — collapses each group to (usually) one row.

HAVING is not a parameter: filter with ordinary grep.

D.1. AggregateFunction role

Postgres analogue: ordinary CREATE AGGREGATE (no moving-frame extras). One state value per evaluation; one result after the last row.

Loading the role

Do not add unit. use Qwiratry::AggregateFunction; loads lib/Qwiratry/AggregateFunction.rakumod by name. Add a provides entry. MovingFrameAggregateFunction.rakumod already uses it; AggregateFunctions.rakumod must too.

Intended semantics

RakuPostgresWhen
StateTypeSTYPEcompile time
$.stateinternalmutated during a run
@.parameterscall argsset when the instance is made
$.currentcurrent itemraw item; each row
window-startINITCONDstart of a window
rowSFUNCeach row; receives arg values
window-endFINALFUNCend of window; return value
$.sortSORTOPstub only (Nil)
$.parallelPARALLELstub only

Notes kept out of the table:

  • row updates $.state. It does not have to return the user-visible value.

  • Read path: because state lives on the object, we do not need Postgres’s extra FINALFUNC / MFINALFUNC as separate required methods. CALL-ME (instance) reads $.state / $.frame-state and returns the user-visible value (Avg: divide here; Sum: return the number). Default window-end / frame-end can be aliases of that, or omitted in favour of CALL-ME only.

  • Default window-start may leave $.state uninitialized. Sum must not treat empty as 0.

  • row is stub ({...}) and must be implemented. Signature: method row(*@values) — @values are this row’s evaluated argument expressions, not the Callables.

  • allows-zero-arg: default False on the role. Rank / Count override to True (RANK() / COUNT(*) take no value expression). The AST transform uses this when Name.() has an empty argument list (Sum.() vs Rank.()).

window-start / window-end are once per window (window key), not per row.

  • aggreduce: for each window: window-start → row for every item → read once (CALL-ME / window-end) → one output hash.

  • aggmap: window-start or frame-start when the window key changes (and frame-start again if a moving frame is restarted empty). Not every output row. window-end only if this window was a whole-window fold to broadcast; otherwise the per-row read is CALL-ME after enter/exit. New window → start again, not an extra end hook besides that optional window-end.

Optional on the role: $.previous / $.next (raw items) that aggreduce/aggmap set. Useful for Rank peers. Do not require items to grow those fields. $.next is optional if the loop can peek without storing it.

Callable (no extra sub)

AggregateFunction does Callable. Read of the current result is instance CALL-ME (stand-in for FINALFUNC / MFINALFUNC). Do not call CALL-ME on the type object. The body rewrite (§J.2) constructs one instance per aggregate column in a hygienic lexical and rewrites the column expression to $lexical(). Evaluated body only sees those instances via CALL-ME. Two Sum.({.amount}) columns are two lexicals → two instances.

Result extraction

After the last row of an aggreduce window, read via CALL-ME (or window-end if kept as an alias). Do not clobber $.state in the read path (aggmap may still enter more rows). aggreduce may use a fresh instance per window.

Tests

  • use Qwiratry::AggregateFunction; then class Stub does AggregateFunction[Numeric] { method row(*@) {} }.

  • Missing row fails when driven.

  • Default window-end returns $.state.

D.2. Sum (Postgres SUM)

Postgres sum(expression): sum of non-null values. Put class Sum in lib/Qwiratry/Aggregate/… (catalogue). Do not add sub sum (Raku already has &sum). In body, write Sum.({.amount}) (see §J.2).

Intended results

InputPostgresThis module
1, 2, 366
empty (no rows)NULLNil
all nullNULLNil
1, null, 233
single 555
  • Skip nulls. After a real value, later nulls do not reset to Nil.

  • Empty / all-null → Nil, not 0.

  • Numeric this cut. No SUM(DISTINCT …) / FILTER.

  • Sum.() with no expression is a compile-time error from the AST transform (allows-zero-arg is false; Postgres SUM() is a syntax error). Sum() without the postfix dot is a Type::Coercion — sorry, write Sum.({…}). Hand-rolled Sum.new driven with no extractors is a runtime throw as a backstop.

  • Moving-window Sum / Avg are §F (landed). Count when the two-column test needs it.

Class sketch

aggreduce sets $_, evaluates {.amount}, then calls row with that number:

class Sum does AggregateFunction[Numeric] does Callable {
    method window-start() { $.state = Nil }

    method row(*@values) {
        my $value = @values[0];
        $.state = ($.state // 0) + $value if $value.defined;
    }

    method CALL-ME() {
        # instance only (transform constructs instances; see §J.2)
        # return $.state
    }
}

row does not evaluate {.amount} itself. @.parameters still holds the Blocks so aggreduce knows what to evaluate. StateType must allow Nil, or use a “seen any” flag.

Tests

Hand-roll window-start / row / window-end first, then via aggreduce:

  • [1, 2, 3] → 6

  • empty → Nil

  • null among numbers ignored

  • all null → Nil

  • two Sum.({.amount}) columns do not share state

  • Sum.() throws

D.3. aggreduce / Driver collapse

Stub: sub aggreduce(%parameters, @values) {}.

Call shape

@sales.aggreduce(%(
    window => { .<region> },         # omit ⇒ one window
    sort      => { .<sale-date> },   # 1-arg; optional this cut
    body      => -> {
        region => .<region>,
        total  => Sum.({.<amount>}),
        n      => Count.(),
    },
))

Descending or other special orders use a 2-arg Block, same as Raku sort:

sort => { $^b.<amount> <=> $^a.<amount> }

Export a sub and a method @values.aggreduce(%parameters). Test the method form the spitball used.

What body is

With commas, a pointy block of fatarrows is a List of Pairs. Hashify it so later |%_ works.

-> { … } has no implicit $_ parameter. Set the topic, then call with zero args:

$_ = $item;
my %out = %parameters<body>().Hash;

Around the body (ordinary Raku)

Per item / window, aggreduce does not leave row to the body’s Sum.({.amount}) call:

  1. $_ = $item; $.current = $item (and $.previous if we keep it on the role).

  2. For each registered instance: evaluate its argument Blocks (with that $_); row(|@results).

  3. $_ = $item again if needed; evaluate body. $__AGGFN_N() CALL-ME returns the already updated result.

collect-aggregate-functions was a placeholder; inline registration via the body rewrite (§J.2).

window-start must run before the first row, so instances must exist before the item loop. The transform creates them in hygienic lexicals and binds $*AGGREGATE-FUNCTION-INSTANCES for that call. aggreduce / aggmap read that dynamic; they do not scan body at runtime.

CHECK / BEGIN / COMPILE on this module do not see inside a Block the caller stored in a hash.

No cursor on items

aggreduce can group with loop locals only (key from window Block, eqv for “same window”). It does not need .previous / .counter / .window-value / .set-value on items.

Rank later still needs a previous item and a window counter: attributes on AggregateFunction / Rank, set by aggmap from loop locals (AF-Q1).

Algorithm

  1. Split by window (or one window if omitted). Window order: first-seen. Key equality: eqv (not ===). A List key would break under ===.

  2. Register from $*AGGREGATE-FUNCTION-INSTANCES (one instance per rewritten column). Missing dynamic ⇒ the caller was not rewritten (RAKUDO_RAKUAST=1 / use Qwiratry); throw.

  3. Reject functions that only make sense as a per-row window walk (Rank is invalid in SQL GROUP BY without a subquery). Sum is valid.

  4. Each window: window-start; each item: evaluate args, row(|@values); one output hash. Prefer SQL: output columns are window keys or aggregate functions.

  5. Cardinality = number of windows. No rows and window present ⇒ zero rows. No rows and window omitted ⇒ one row (SELECT sum(x) FROM t with no rows). Empty Sum is Nil.

Tests

  • window => { .<region> } with Sum.({.<amount>}).

  • No window: one row; empty input → one row, total => Nil.

  • Rank.() in aggreduce throws once Rank exists.

  • Output length = distinct keys under eqv.

  • Window keys appear in the result.

  • Method vs sub.

E. aggmap

Postgres analogue (windowing mode): SELECT … window_function() OVER (PARTITION BY … ORDER BY …) — row-preserving. Frames: §F.

Dual mode (WQ8): aggmap / ⇶ is also ordinary map. With no window / sort / frame and no aggregate columns in body, emit one transformed item per input item — ordinary map cardinality, without aggregate lifecycle. Windowing turns on when those args / Name.() columns are present.

E.1. MovingFrameAggregateFunction

The windows are the same as aggreduce. aggreduce / aggmap do the walk. Roles only supply callbacks those two call.

No middle WindowAggregateFunction. AggregateFunction is enough for a fold of a window (window-start / row / window-end): aggreduce emits once; aggmap can write that value on every row, or keep calling row/frame-enter as the frame end moves (rows only enter).

MovingFrameAggregateFunction is the moving-frame role (frame-exit / MINVFUNC). The window is fixed. Expanding ROWS UNBOUNDED PRECEDING … CURRENT ROW never exits. aggreduce never exits. Rank never exits (ignores frame; one row per item).

The full rename map is in §C.2. Window-level fold is window-*; moving-frame extras are frame-*.

row vs CALL-ME (object state; no required frame-end):

  • row / frame-enter / frame-exit — write the object’s state.

  • Read is CALL-ME on the instance (what body sees). Same method for aggreduce (once per window) and aggmap (once per output row). For Sum, that is $.state. For Avg, derive sum/count here. Postgres needed a separate FINALFUNC because sfunc returns the next state value; we mutate in place, so we do not need a third required method.

  • window-end / frame-end — optional aliases of that read. First cut: CALL-ME is enough. Keep the names when a phaser slang wants WINDOWLAST / FRAMEFINAL / RESULT.

window-start / window-end once per window, not per row (see §D.1). frame-start as well when the window key changes, and again if the moving frame is restarted empty.

Default frame-enter can still be self.row(|@values) on the moving role so sliding Sum shares row with collapse.

MovingFrameAggregateFunction API

Postgres: MSFUNC, MINVFUNC, MSTYPE, MINITCOND, MFINALFUNC. This is the role API; frame which items is §F.

RakuPostgresWhen
$.frame-stateMSTYPEslides with the frame
frame-startMINITCONDnew window or restart after empty frame
frame-enterMSFUNCrow enters; updates frame state
frame-exitMINVFUNCrow leaves; updates frame state
frame-endMFINALFUNCoptional; default = CALL-ME (read)

Read for this output row is CALL-ME, not a required frame-end. End of window is window-end only for aggreduce / whole-window broadcast (once per window).

Defaults:

  • frame-enter → self.row(|@values) (args already evaluated, same as §D.3).

  • frame-exit → default throws X::StubCode = unavailable; aggmap treats that as “restart the frame” (AF-Q16). A bare {...} on the role would be a Raku required method (every class must override), so the role supplies a concrete throwing body with the same detection. Do not override with an empty frame-exit {} (looks like a successful no-op undo). Real inverses override with a working body. Rank keeps the default; aggmap must not call frame-exit for Rank anyway.

  • frame-end → $.frame-state.

  • frame-start → no-op {} (override when there is a MINITCOND; same idea as window-start).

There is no middle role. MovingFrameAggregateFunction adds frame-state / enter / exit / end on top of AggregateFunction.

Do not override these

trigger-frame-start: set $.window-label from the loop-local window key, not $current.window_label(). Then frame-start.

trigger-frame-enter: frame-enter. Do not $.current.set-value($.frame-state). body builds the output hash.

Tests

  • use Qwiratry::MovingFrameAggregateFunction; composes with Int, Int.

  • Default frame-enter calls row.

  • Default frame-end returns $.frame-state.

  • Default frame-start is a no-op.

  • Default frame-exit throws X::StubCode ⇒ unavailable ⇒ aggmap restarts the frame (AF-Q16).

Sliding Sum: §F.

E.2. Rank (Postgres RANK)

Postgres rank() → bigint: rank with gaps = row_number of the first row of the peer group.

Peers: window ORDER BY keys only (eqv on those keys). Not a full-row compare, not SELECT DISTINCT. No sort ⇒ every row in the window is a peer ⇒ every rank is 1. A new window key restarts numbering. Ignores frame.

DENSE_RANK / ROW_NUMBER only as needed for the spitball (dense_rank).

Intended results

SituationRANK()
First row in window1
Consecutive peers on ORDER BY keyssame rank
After a peer group of size k that started at nn + k

Example, one window, scores 90, 80, 80, 70 ordered DESC: ranks 1, 2, 2, 4.

  • DENSE_RANK: 1, 2, 2, 3

  • ROW_NUMBER: 1, 2, 3, 4

1-arg sort => { .<amount> } is ASC (Raku cmp). DESC is the 2-arg form.

Class in AggregateFunctions.rakumod

Callable class Rank (capital R, parallel to Sum; no sub rank). does MovingFrameAggregateFunction; keeps default frame-exit (unused). aggmap must not call frame-exit for Rank. Peer detection and read path: AF-Q14 / AF-Q15 (§M.2).

Fixes to the spitball class rank:

  • method row(*@values) — Rank has no SQL arguments; ignore @values.

  • Previous item: $.previous on the aggregate function (or a loop local passed in), not $item.previous.

  • Peers: eqv on sort keys, not $prev.id != $_.id.

  • Counter: 1-based index in the window, kept by aggmap (loop local or $.counter on the instance), not $item.counter. Non-peer: $.frame-state = $counter; peer: leave it.

  • After frame-start, first enter must become 1, not stay 0.

  • No newpartition on items. Window change: not $key eqv $prev-key in aggmap.

Tests

  • Unique keys, 1-arg sort: 1..N in that order.

  • Same with 2-arg DESC: who gets 1 flips.

  • Ties: same rank, then a gap.

  • window => { .<region> }: restart at 1.

  • Empty input: empty output.

  • Two rows that differ in an unsorted column but match sort keys share a rank.

E.3. aggmap / Driver map

Normal map (no windowing):

@array.aggmap(%(
    body => -> { name => .<name>.uc, amount => .<amount> },
))
# or, once the operator exists:  $relation ⇶ { … }

Windowing (aggregates / partition):

@array.aggmap(%(
    window    => { .<region> },
    sort      => { .<amount> },      # 1-arg; easy
    body      => -> {
        region            => .<region>,
        salesperson       => .<salesperson>,
        sale-date         => .<sale-date>,
        amount            => .<amount>,
        region_rank       => Rank.(),
        region_dense_rank => DenseRank.(),
    },
))

Same cardinality as the input. Hashify body. No frame yet (§F). Until frames (§F): ranking sees every item in order; plain Sum broadcasts one whole-window value onto every item (WQ3 / AF-Q13).

Same sub + method export as aggreduce.

Sort then windows

Match Raku sort:

  • 1-arg { .<amount> }: Schwartzian key, then cmp. “Same key” is $a cmp $b == Same (treat as eqv, not ===).

  • 2-arg: only when 1-arg cannot express the order.

Sort so windows are contiguous (window key, then sort). New window: window key !eqv previous. window omitted ⇒ one window. sort omitted ⇒ keep input order inside the window.

A 1-arg window Block that returns a List must use eqv.

body

Same rules as aggreduce body. Fields that are not aggregate functions (region, salesperson, …) are just ordinary Raku in that block (.<region>). No extra “pass-through” pass.

Do not assign $item = $agfunc.frame-state.

Per-row loop without frames

Drop stub bugs: window-start every row; does instead of ~~; methods on items; replacing the item with window state. Keep $current / $previous / $next as loop locals (as the stub comment already wants).

sort (window key, then user sort)
register instances from $*AGGREGATE-FUNCTION-INSTANCES
for each item:
    if window key !eqv previous:
        $counter = 0; trigger-frame-start (or window-start)
    $counter++
    set $.current / $.previous / $.counter on instances if those attrs exist
    evaluate arg Blocks; frame-enter / row(|@values)
    $_ = $item; hashify body()     # CALL-ME returns current values
    yield output hash

Tests

Spitball rank example: PARTITION BY region + order by amount. Output length = input. window omitted; empty list; single row; method vs sub. Framed running sum: §F.

F. Frames

The second and third chained aggmaps. Rank still ignores frame.

@array.aggmap(%(
    window    => { .<region> },      # omit for company running total
    sort      => { .<sale-date> },   # 1-arg
    frame     => …,
    body      => -> {
        |%_,
        company_to_date => Sum.({.<amount>}),
        # region_avg_7d => Avg({.<amount>})
    },
))

(avvg in the spitball is avg.)

F.1. Frame parameter

frame is a hash passed to aggmap only. It selects which rows in the current window count for each output row. Defaults and SQL analogues:

DefaultWhenPostgres analogue
whole windowno sortframe = entire PARTITION BY set
range / unbounded preceding … currentsort present, frame omittedRANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW

Keys

KeyTypeMeaning
typerows | range | groupHow bounds are measured (Postgres ROWS / RANGE / GROUPS).
startsee belowLower bound (integer, sentinel, or omitted if bounds is a Range).
endsee belowUpper bound.
boundsRangeAbsolute integer start/end together. Implies start-type / end-type are absolute. rows / group only.
start-typerelative | absoluteHow to interpret start. Default relative. Ignored when bounds is set.
end-typerelative | absoluteHow to interpret end. Default relative. Ignored when bounds is set.
exclude-currentBoolDrop the current row from the frame.
exclude-tiesBoolDrop ordering peers of the current row. Does not drop the current row unless exclude-current is also true.

exclude-* together map to Postgres EXCLUDE (neither / current row / ties / group).

type and the type of start / end

start and end are not always the order-key type. They depend on type:

typestart / end areExample
rowsInteger row offsets (or relative sentinels; see below)relative -1 … 0 ⇒ one preceding row through current
groupInteger peer-group offsetsrelative -1 … 0 ⇒ previous peer group through current group
rangeDelta on the order key — same kind as sort producesDuration for dates (-6 days … 0); numeric difference for numbers

For range, bounds are applied by subtracting/adding on the sorted order key (Postgres RANGE). For rows / group, bounds are counts in the ordered row list or peer-group list.

relative vs absolute

Relative (default): offset from the current row in the ordered window. Integers and named sentinels only — not a Raku Range (a Range has no notion of “current row”).

  • 0 = current row (or current peer group when type is group).

  • Negative = preceding; positive = following.

  • Matches Postgres n PRECEDING / CURRENT ROW / n FOLLOWING.

  • Unbounded / current as strings: unbounded-preceding, unbounded-following, current-row.

Absolute: index into the current window in that window’s sort order. Same shape as Postgres cursor MOVE ABSOLUTE n, not SQL frame n PRECEDING. Not a CREATE AGGREGATE feature; aggmap interprets it when sliding the frame.

Absolute valueMeaning
0Postgres UNBOUNDED PRECEDING (start of the window; first row)
Positive nnth row from the window start, 0-based (1 = second row)
-1Postgres UNBOUNDED FOLLOWING (end of the window; last row)
Negative n other than -1from the window end (-2 = second-to-last)

So 0..-1 is the whole window. First included row is 0; last included row is -1.

Storing absolute bounds as a Raku Range

A Range is absolute only, and only a shorthand for integer start/end when type is rows or group. Put it in bounds; that implies start-type and end-type are absolute. Do not put a Range in start / end.

  • 0..-1 — whole window (UNBOUNDED PRECEDING … UNBOUNDED FOLLOWING).

  • 0..4 — first through fifth row.

  • 1..-2 — second row through second-to-last.

Do not use a Range for relative frames. Relative uses start / end integers or the named sentinels (unbounded-preceding, current-row, unbounded-following). Prefer separate start / end when start-type / end-type differ, or when type is range with Duration bounds.

Examples

Running sum (company-wide, no window key) — relative:

frame => %( type => 'rows', start => 'unbounded-preceding', end => 'current-row' )

Sliding sum, one row back through current — relative:

frame => %( type => 'rows', start => -1, end => 0 )

Seven-day average by region (order key is date) — relative:

frame => %( type => 'range', start => -6 days, end => 0 )

Whole window as an absolute Range:

frame => %( type => 'rows', bounds => 0..-1 )   # start-type/end-type implied absolute

Absolute slice — second through fifth row of the window:

frame => %( type => 'rows', start => 1, end => 4, start-type => 'absolute', end-type => 'absolute' )
# or: bounds => 1..4

Exclude current row and its sort-key peers — relative:

frame => %( type => 'rows', start => -1, end => 1, exclude-current => True, exclude-ties => True )

F.2. How aggmap should slide

For each output row:

  1. Window key !eqv: trigger-frame-start (do not frame-exit the old window).

  2. frame-exit each row that left the frame (oldest first). If frame-exit is unavailable ⇒ restart.

  3. frame-enter each row that entered (evaluate that row’s args, then enter).

  4. Evaluate body / frame-end for the current item.

Empty frame ⇒ restart, do not frame-exit the last row. Moving end only needs frame-enter. frame-exit is for a moving start.

F.3. Sum in moving-frame mode

Same class Sum:

  • frame-start: Nil, like window-start.

  • frame-enter: row(|@values) (add, skip nulls).

  • frame-exit: subtract (or restart if Num is painful; tests prefer Int / Rat).

  • frame-end: $.frame-state (Nil if the frame has no non-null values).

Avg is Sum/Count over the same frame.

F.4. Tests

  • Sliding Sum, ROWS BETWEEN 1 PRECEDING AND CURRENT ROW; restart path matches.

  • Running sum: no window key, ROWS UNBOUNDED PRECEDING … CURRENT ROW.

  • Range: window by region, 6 days preceding … current.

  • Rank.() beside a framed Sum ignores the frame.

  • Empty / all-null frame: Sum is Nil.

G. Phasers (aggsub) — later

Later definition spelling, not the first cut. Classes + methods remain the implementation. aggreduce/aggmap still do the walk and invoke the phasers. This is not a replacement for calling enter/exit from aggmap.

Sketch:

aggsub Rank() {
    WINDOWFIRST { $.frame-state = 0 }
    if ! self.is-duplicate($previous, $current) {
        $.frame-state = $current.counter;
    }
}

That slang compiles to a class. Possible map:

PhaserMethodWhen
WINDOWFIRSTwindow-startstart of a window
WINDOWLASTwindow-endend of a window (collapse result)
unphased bodyroweach item / each enter
FRAMEFIRSTframe-startfresh frame state (new window or restart)
FRAMEADDframe-enterrow entered the frame
FRAMESUBTRACTframe-exitrow left; only moving frame start
FRAMEFINALframe-endvalue for this output row (read/CALL-ME)
RESULTCALL-MEaggregate result for the body (optional)

Colonpair :!aggreduce ⇒ allows-aggreduce False (ranking-style). Empty signature ⇒ allows-zero-arg.

FRAMEFINAL maps to frame-end (MFINALFUNC): it runs every output row after enter/exit, returning this aggregate’s value for the row — not once at the end of the window (WINDOWLAST / window-end already covers that). Default is still “current $.frame-state” / CALL-ME; the phaser is for non-trivial finalize (e.g. Avg divide). Avoid FRAMELAST: LAST suggests once-at-end, which mismatches per-row emit. RESULT overrides CALL-ME when stored state is not already the body-visible value (e.g. packed (sum, count)).

Useful as a later spelling of class Rank does ….

H. Implementation orders

H.1 Basics Implementation

  • 1. Roles under Qwiratry::; kebab-case lifecycle; is rw; no unit; roles not re-exported from the catalogue (§C.2 / §A.2).

  • 2. AggregateFunction + Callable + row(*@values) (§D.1).

  • 3. class Sum in the Qwiratry catalogue, no sub sum (§D.2).

  • 4. Body rewrite (§J.2) installed from use Qwiratry (WQ4) + aggreduce / Aggregate::Driver collapse over Iterator (WQ1); args then row (§D.3).

  • 5. Wire façade: walkable → Walker/QueryIterator; Iterable → .iterator.

  • 6. MovingFrameAggregateFunction defaults — no middle role (§E.1).

  • 7. Rank on ORDER BY keys + eqv + counter not on items (§E.2); resolve AF-Q14 / AF-Q15 while implementing.

  • 8. aggmap with 1-arg sort + rank example; Sum broadcasts (WQ3) (§E.3).

  • 9. Wire ReduceOperator (⌿) → aggreduce; add ⇶ → aggmap (WQ7 / WQ8 dual mode).

  • 10. Frames + running / sliding Sum / Avg (§F); frame-exit unavailable per AF-Q16 (X::StubCode → restart).

  • 11. Capability / pushdown hooks on Table / future SQL Walkers.

Align with TaOP Plan §11 items 7–8 after Textish / CSV. Do not modify raku-AggregateFunctions (WQ6).

Note: AF-Q14 / AF-Q15 / AF-Q16 are decided (§M.2). Remaining checklist items use those decisions (especially frames / AF-Q16 in step 10).

H.2 Advanced Implementation

  • 12. §G — aggsub phasers (definition slang for aggregate classes).

  • 13. Rewrite existing aggregate functions (eg. Sum, Rank) to use aggsub

  • 14. Catalogue Count and DenseRank (parallel to Sum / Rank).

  • 15. §N testing note (c) — File → Parse → Table pipeline integration for aggreduce / aggmap.

H.3. Third Round Implementation

  • 16. §A.6 — DeduplicationOperator (⑂, U+2442): sort-shaped 1-arg / 2-arg Block or postfix whole-item; compose with ⌿ / ⇶; not a second Driver. Pushdown mapping when §K.1 exists.

  • 17. §J.4 — whole-tree body walk (nested blocks / if / when / ternary); always register nested aggregates; FILTER remains later (§J.4.1).

I. Postgres CREATE AGGREGATE checklist

FeatureThis cut
SFUNC / STYPE / FINALFUNC / INITCONDyes (row, StateType, window-end, window-start)
arguments@.parameters hold Blocks; row gets evaluated values
MSFUNC / MINVFUNC / MSTYPE / MFINALFUNC / MINITCONDyes (§E.1 / §F)
PARALLEL on the objectattribute only, unused
SSPACE / MSSPACE / FINALFUNC_MODIFY / MFINALFUNC_MODIFYno
COMBINEFUNC / SERIALFUNC / DESERIALFUNCno
SORTOP / ordered-set ORDER BY / HYPOTHETICALno
FINALFUNC_EXTRA as its own hookno; CALL-ME / window-end may read @.parameters

J. Discovery / body rewrite

aggreduce / aggmap must register one instance per aggregate column before the first window-start / row. Find anything whose type does AggregateFunction (including MovingFrameAggregateFunction), not a hardcoded Sum / Rank name list. Two Sum.({.amount}) columns must not share an instance.

CHECK / BEGIN / COMPILE on this module never see inside a body Block the caller stuffed into a hash. Chosen path (Q4): a post-parse AST transform that rewrites the Block in the importing compilation unit so registration is explicit in the compiled code.

Not an option: evaluate body once as a “discover pass.” That extra run can fire pass-through fields and any other side effects. Not chosen: walk RakuAST of the runtime Block (it has none — see Probe).

J.1. RakuAST walk (not chosen)

Inspect $body.AST (or equivalent) without evaluating body. Walk call nodes; if the callee’s type does AggregateFunction, construct/register an instance (argument Blocks into @.parameters).

HelpsHurts
No dummy run of body (no extra .<region> / side effects).Relies on RakuAST for a Block that arrived as a hash value; may be missing or incomplete.
Works with empty input (aggreduce with no window still emits one row).Indirect / computed callees (my $c = Sum; $c({.amount})) are easy to miss.
Column identity can come from AST position (two identical Sum.({.amount}) stay distinct).Nested Blocks, conditionals, and macros are more work.

J.2. Post-parse AST transform (chosen; installed by use Qwiratry)

Installed invisibly from use Qwiratry (Q6). After normal parsing, find each aggreduce / aggmap call (sub and method form) in the importing compilation unit and rewrite that call’s body so registration is explicit in the compiled code. This is a RakuAST transform, not a new grammar.

HelpsHurts
Can make body look exactly like the spitball and still be typed.Bigger tool; separate from §G aggsub (that slang is for defining aggregate classes, not for scanning body).
Handles empty input if rewrite is compile-time.Separate from §G aggsub (defining classes, not scanning body).
Rewrites Rank.() / Sum.({…}) (ApplyPostfix + Call::Term) to instance $var() without type-object CALL-ME.

Hook. Prefer attaching a post-parse visitor to the importing compilation unit from EXPORT / use (use 6.e.PREVIEW allowed). No custom grammar or actions unless that hook cannot be installed. Fall back to Slangify only if needed to get a statement-list / compunit callback. Require RAKUDO_RAKUAST=1. Keep the transform inline in lib/Qwiratry/Aggregate/… (catalogue) unless the file becomes unreadable.

Walk depth. Fatarrow column values are walked for nested C / C / ternary / blocks so nested C<Sum.()> / C<Rank.()> still register (§J.4). Extractor Blocks on C<Name.({…})> are not scanned for nested aggregates (those stay on C<@.parameters>).

Documented spelling: Name.()

In body, aggregate columns are calls with an explicit .() postfix:

WriteMeaning
Rank.() / Count.()zero-arg (allows-zero-arg)
Sum.({.<amount>})extractor Block stored on @.parameters

Probe (Rakudo 2026.07). Rank.() is ApplyPostfix (Type::Simple + Call::Term with empty ArgList) — a real call, not Type::Coercion. Sum.({.<amount>}) is the same node type with a Block argument. Runtime: Rank.() does invoke type-object CALL-ME, so the transform still rewrites to $__AGGFN_N() (Q8).

Do not write Sum.(.<amount>) (no Block): that is also ApplyPostfix, but the argument is Term::TopicCall evaluated immediately on $_, not captured as an extractor. Compile-time sorry: extractor must be a {…} Block.

Rank() / Sum() without the postfix dot remain Type::Coercion (Rank(Any)). Compile-time sorry with a hint to write Rank.() / Sum.({…}). Int() is ordinary coercion — leave it alone. Sum({.<amount>}) without the extra dot is also ApplyPostfix (same as Sum.({.<amount>})); accept it, but document the .() form.

Body rewrite (Q5)

For each top-level aggregate column in body:

  1. Find the node — a RakuAST::ApplyPostfix whose operand is a type name and whose postfix is Call::Term (Name.() / Name.({…}) / Name({…})).

  2. Resolve the type — look up the name; keep it only if the class does AggregateFunction (not a hardcoded name list). Other Type.() / Type() nodes are ordinary Raku — leave them alone.

  3. Zero-arg vs extractor — empty ArgList is valid only if allows-zero-arg (default False; Rank / Count true). Sum.() is a compile-time sorry (Postgres SUM() is a syntax error). Non-empty args must be Block(s); copy them onto @.parameters. Sum.(.<amount>) (topic call, not a Block) is a sorry.

  4. Inject a hygienic lexical — one per column, counter incremented per aggreduce / aggmap call site (e.g. $__AGGFN_0001, $__AGGFN_0002). Names are transform-generated (not user source), so they do not collide with $r, $rank, or other caller lexicals. Initialize with Type.new (and @.parameters when the source had a {…} extractor).

  5. Rewrite the column expression to $__AGGFN_N() — instance CALL-ME returns the current value after row / frame hooks have run. Zero-arg: Rank.() → $__AGGFN_0001(). One-arg: Sum.({.<amount>}) → $__AGGFN_0002() (extractor already on the instance; body does not pass it again).

  6. Wire instances via $*AGGREGATE-FUNCTION-INSTANCES. Wrap the rewritten call so the hygienic lexicals and the dynamic are in the same scope, bound before aggreduce / aggmap runs:

{
    my $__AGGFN_0001 = Sum.new: parameters => ({.<amount>},);
    my $__AGGFN_0002 = Count.new;
    my $*AGGREGATE-FUNCTION-INSTANCES = ($__AGGFN_0001, $__AGGFN_0002);
    @sales.aggreduce(%(
        body => -> {
            total => $__AGGFN_0001(),
            n     => $__AGGFN_0002(),
        },
    ));
}

aggreduce / aggmap read $*AGGREGATE-FUNCTION-INSTANCES (a List of instances) and call window-start / row on those. body only reads via $__AGGFN_N(). Nested aggreduce / aggmap nest the dynamic. An empty list is valid (no aggregate columns). If the dynamic is undefined, the call was not rewritten — throw (missing RAKUDO_RAKUAST=1 or use Qwiratry).

User-facing source is Rank.() / Sum.({.<amount>}). Compiled code never relies on empty Type() coercion or type-object CALL-ME.

J.3. How discovery was chosen (AF-Q4)

This is an empirical question, not a preference vote. Probe the real call shape (body as a Block inside a hash passed to aggreduce / aggmap):

  1. Does that Block have a usable AST ($body.AST or equivalent)?

  2. Can a walk find Sum.({.<amount>}) and Rank.() (type does AggregateFunction), including two identical Sum columns as two instances?

  3. Can we register before any row, including empty input (no dummy eval of body)?

If yes to all three for the first-cut examples → 9.1. If the AST is missing, incomplete, or cannot resolve the callee type → 9.2 post-parse transform. Do not add a discover pass later to dodge a transform.

Probe (Rakudo 2026.07 / 6.d)

Call shape: aggreduce(%( body => -> { … } )).

  1. No. A runtime Block has no .AST. @!compstuff is VMNull. .raku is -> { #(Block|…) ... }. CHECK $*CUexists only in the **caller’s** compilation unit, not insideaggreduce`.

  2. Not from that Block. Str.AST of similar source works as a separate parse: Sum({ 1 }) and Sum.({ 1 }) are ApplyPostfix (type + Call::Term); Rank.() is the same node with an empty ArgList. Rank() without the postfix dot is not a call: the grammar makes Type() a Type::Coercion (Rank(Any)). Runtime: Rank() does not invoke CALL-ME; Rank.() and Sum.({.x}) do (type-object CALL-ME — hence the rewrite). Sum.(.<amount>) is also a call, but the argument is a topic call evaluated immediately, not a Block.

  3. Not via 9.1 without a compile-time rewrite: we never get the caller’s AST at aggreduce runtime.

Chosen: §J.2 post-parse AST transform (fails test 1 of the decision procedure). Documented spelling is Name.(); the transform rewrites those ApplyPostfix nodes to hygienic-lexical $var() calls and binds $*AGGREGATE-FUNCTION-INSTANCES (§J.2 / AF-Q5).

J.4. Whole-tree body walk

Rewrite walks each top-level fatarrow value recursively (nested blocks, if / when, ternary, and other expressions) so total => $flag ?? Sum.({.<amount>}) !! 0 still registers Sum. Extractor Blocks on Name.({…}) are not scanned for further aggregates (Post-SQL-TaOP-Improvements.md §A.4).

J.4.1. Postgres conditional aggregates in SELECT

Every aggregate in the SELECT list is always computed for the group. There is no “sometimes this column is an aggregate.”

  • CASE WHEN … THEN SUM(x) ELSE 0 END still runs SUM; the CASE wraps the result after aggregation.

  • A non-grouped row expression mixed with an aggregate in CASE is an error (must appear in GROUP BY or be used in an aggregate function).

  • Conditional rows into an aggregate is FILTER (WHERE …), which remains later — not a nested if around the column.

Nested aggregates found by the whole-tree walk are always registered (drive row every item). if / CASE / ternary in body only affects what CALL-ME contributes to the output hash, not whether the instance exists.

K. Things to add later

Not this cut. Do not block §D.3 on these.

K.1. Separate SQL Source module

  • SQL Text Generation (as a separate module, via SQL::Abstract)

  • Real SQL pushdown (Location Walker that implements can-pushdown-aggregate / pushdown-aggregate; hooks already exist §H.1.11).

K.2–K.5 — moved

Former §K.2 (Low-hanging Aggregate Functions), §K.3 (Parallel Processing of Queries), §K.4 (TaOP Relation mutation APIs), and §K.5 (Other later items) now live in Post-SQL-TaOP-Improvements.md §A (as §A.1–§A.4).

L. Open Questions

Numbers are stable IDs. AF-Qn = AggregateFunctions Spec IDs (never reuse a number as both open and decided). WQn = Qwiratry-integration decisions (WQ1–WQ8).

L.1. Open (from AggregateFunctions Spec)

(none — AF-Q14 / AF-Q15 decided in §M.2 while implementing Rank.)

L.2. No further WQ open items

WQ1–WQ8 are decided (§M.1).

M. Decided questions

M.1. Qwiratry integration (WQ1–WQ8)

WQ1. Factory path for bare List / Seq. Option B: Driver consumes an Iterator. Walkable subjects go Walker → QueryIterator; bare Iterable subjects use .iterator. No Sequence Walker in the first cut. Aggregate “previous” / frame history stay Driver-side (loop locals + window buffer), not Walker fetch-prior.

WQ2. agggrep. Typo for aggreduce in TaOP Plan §11.7–8. No third aggregate driver; filter-after-aggregate is σ / ordinary grep after aggreduce / aggmap.

WQ3. Sum in aggmap before frames (was AggregateFunctions AF-Q13). Until frames (§F), plain Sum in aggmap broadcasts the whole-window total onto every item in that window (Postgres whole-partition default). Running / sliding sums wait for §F.

AF-Q13 original options: (a) treat early aggmap Sum as running; (b) broadcast whole-window in early aggmap, defer running to frames; (c) separate RunningSum. Chose (b).

WQ4. Package entrypoint. Available via use Qwiratry; (aggreduce / aggmap, body rewrite, catalogue classes). Keep AggregateFunction as the role module (Qwiratry::AggregateFunction) for authors of custom aggregate functions; do not confuse that role with the driver/catalogue surface.

WQ5. Map operator glyph. ⇶ (U+21F6, THREE RIGHTWARDS ARROWS), as in Operators.rakudoc §4. Sole map operator (AggMapOperator); legacy ». / MapOperator removed. Collapse side already has ⌿ — no new glyph for GROUP BY.

WQ6. Standalone raku-AggregateFunctions. Leave that repo alone (never released). Use its Specification and implementation as reference while porting into Qwiratry. No re-export shim and no cutover edits there.

WQ7. Operators → Driver. ReduceOperator (⌿) calls aggreduce — no separate GroupByOperator / γ. ⇶ (AggMapOperator) calls aggmap. Legacy MapOperator / ». removed.

WQ8. aggmap / ⇶ dual mode. Ordinary map when no windowing is requested (no window / sort / frame, no aggregate columns in body): one output item per input item. Windowing applies when those args / Name.() columns are present (§E / §F).

M.2. From AggregateFunctions Spec (AF-Q1–AF-Q13)

Stable IDs from AggregateFunctions Spec §12:

  1. No cursor on items. Walk state is loop locals and attributes on the aggregate-function instance ($.previous / $.next / $.counter). Items stay ordinary hashes/objects. window-value: loop-local key / $.window-label. set-value: body builds the output hash; never $.current.set-value(...).

  2. window is the key parameter. Matches frame and the spitball. Postgres: the frame moves; the keyed window (item set) does not. Reopen only if window => { .<region> } feels too much like the whole OVER (…) spec rather than PARTITION BY.

  3. Middle role: no. Walks live in aggreduce/aggmap (Driver). frame-exit only when the frame start moves. AggregateFunction + MovingFrameAggregateFunction is enough.

  4. Discovery: post-parse AST transform (§J.2). A runtime body Block has no RakuAST; do not evaluate body as a discover pass. After normal parsing, find aggreduce / aggmap in the importing compilation unit and rewrite body. Find Name.() / Name.({…}) (ApplyPostfix + Call::Term) whose type does AggregateFunction, not a name list. Name() without the postfix dot is Type::Coercion — sorry, write Name.(). No custom grammar unless the hook cannot be installed otherwise (then Slangify). Require RAKUDO_RAKUAST=1. use 6.e.PREVIEW allowed. Install from use Qwiratry.

  5. Documented spelling is Name.(). Rank.() / Count.() / Sum.({.<amount>}) parse as ApplyPostfix + Call::Term. The transform injects a hygienic lexical per column and rewrites to $__AGGFN_N(). Rank() without the dot is coercion — compile-time sorry with a hint. Sum.(.<amount>) evaluates the topic immediately — sorry. After rewrite, body only reads instance CALL-ME. Sum.() (allows-zero-arg false) is a compile-time sorry.

  6. Transform is invisible after use Qwiratry. Ordinary @rows.aggreduce(%( body => -> { total => Sum.({.<amount>}) } )) — no extra keywords. Requires RAKUDO_RAKUAST=1. If attaching a post-parse walk from use fails, try Slangify; if that also fails, revisit.

  7. One instance per call site. Each aggregate column in body becomes one hygienic lexical and one instance. Two identical Sum.({.<amount>}) columns are two lexicals → two instances. Evaluated body reads via $__AGGFN_N() → instance CALL-ME. The driver sees those instances via $*AGGREGATE-FUNCTION-INSTANCES (or rewrite __aggfn wiring).

  8. No type-object CALL-ME. Construction is transform-generated hygienic lexicals (Type.new + @.parameters). body only sees instances via $lexical().

  9. Extractor Blocks live on the instance. {.<amount>} is an argument to Sum, stored as @.parameters. Driver sets topic, evaluates those Blocks, then row(|@values). body only reads CALL-ME after row.

  10. Empty input cardinality matches Postgres. No items and no window ⇒ one result (SELECT sum(x) FROM t → one row, NULL). No items with window ⇒ zero results. Empty Sum is Nil (NULL).

  11. Wire instances with $*AGGREGATE-FUNCTION-INSTANCES. The rewrite wraps each aggreduce / aggmap call so hygienic lexicals and the dynamic are bound before the sub runs. Nested calls nest the dynamic. Undefined dynamic ⇒ not rewritten; throw.

  12. body walk is whole-tree under fatarrow values (§J.4). Nested if / when / ternary / blocks register aggregates; extractor Blocks are not scanned. Non-AggregateFunction Type() / Type.() is left alone.

  13. Sum in aggmap before frames — decided as WQ3 (broadcast).

  14. No peer-start / peer-end. Peer detection stays on Rank: the Driver copies the call’s sort Block onto $.sort, sets $.previous / $.current / $.counter (1-based window index), then calls row. Rank.row compares sort keys (eqv for 1-arg; Same for 2-arg). No sort ⇒ every row is a peer. Rejected: new role hooks for peer boundaries.

  15. Rank overrides CALL-ME (and frame-end) to return $.frame-state. Fold state ($.state / STYPE) and frame state ($.frame-state / MSTYPE) stay separate slots. Body reads CALL-ME; no copy into $.state.

  16. frame-exit unavailable = throw X::StubCode. When aggmap needs an inverse and frame-exit is unavailable, restart the frame. Detection: the MovingFrameAggregateFunction default throws X::StubCode (message frame-exit unavailable). A bare {...} on the role would be a Raku required method, so the role uses a concrete throwing body with the same signal. Stub/default has no side effects before the throw, so try/catch is equivalent to “do not call”. Do not use an empty frame-exit {} (successful no-op looks like a working undo). Classes with a real MINVFUNC override frame-exit. Rank keeps the default; aggmap must not call frame-exit for Rank anyway. Rejected: explicit has-frame-exit flag; “empty body means unavailable”.

N. Testing

  • Port standalone tests into t/unit/aggregate-*.rakutest and/or t/spec/Operators/ once ⌿ → aggreduce and ⇶ → aggmap exist.

  • Integration: aggreduce / aggmap over (a) bare List, (b) Table / Tree Walker item streams, (c) later File → Parse → Table pipeline matches.

  • Body rewrite: Sum.() sorry; Rank() coercion sorry; two Sum.({…}) columns do not share state; missing rewrite throws clearly.

  • Empty input cardinality (AF-Q10): no window → one result; with window → zero results.

  • Broadcast Sum in aggmap before frames (WQ3).

  • aggmap / ⇶ as ordinary map with no windowing args (WQ8).

O. Success criteria

  1. Sum / aggreduce tests pass under Qwiratry modules with no AggregateWalk class.

  2. The collapse driver iterates items from an Iterator — QueryIterator from a Walker when the subject is walkable, otherwise Iterable .iterator (WQ1).

  3. A Table (or Tree) subject can be aggregated without copying partition logic into that Walker.

  4. ReduceOperator (⌿) evaluates via aggreduce; ⇶ evaluates via aggmap (WQ7). That path supports ordinary map and optional windowing (WQ8). Legacy MapOperator / ». is removed.

  5. This document is the implementation spec; standalone AggregateFunctions Spec is reference only (WQ6). Spec cross-links from SQL-Features-Plan §B.2 and TaOP Plan §11.7 point here.

Qwiratry v0.11.0

Declarative query and data-walking architecture for Raku, with transformers, molds, walkers, I/O pipelines, and aggregate functions.

Authors

  • Tim Nelson

License

Dependencies

SlangifyImplementation::Loader:ver<0.0.9+>Glob::Grammar

Test Dependencies

Provides

  • Qwiratry
  • Qwiratry::Aggregate::AggSub
  • Qwiratry::Aggregate::AggSub::Compiler
  • Qwiratry::Aggregate::Avg
  • Qwiratry::Aggregate::BodyRewrite
  • Qwiratry::Aggregate::Count
  • Qwiratry::Aggregate::DenseRank
  • Qwiratry::Aggregate::Driver
  • Qwiratry::Aggregate::Rank
  • Qwiratry::Aggregate::Sum
  • Qwiratry::AggregateFunction
  • Qwiratry::Context
  • Qwiratry::DatabaseStorage
  • Qwiratry::DatabaseStorage::CSVdemo
  • Qwiratry::DatabaseStorage::Location
  • Qwiratry::DatabaseStorage::Location::Tree
  • Qwiratry::DatabaseStorage::Memory
  • Qwiratry::FieldMode
  • Qwiratry::Format
  • Qwiratry::Format::Base
  • Qwiratry::Format::CSVdemo
  • Qwiratry::Format::HalfHuman
  • Qwiratry::Format::JSONdemo
  • Qwiratry::Format::NDJSONdemo
  • Qwiratry::Format::Textish::RenderWalkable
  • Qwiratry::Implementation::Loader
  • Qwiratry::Location
  • Qwiratry::Location::Base
  • Qwiratry::Location::File
  • Qwiratry::Mold
  • Qwiratry::Mold::Compiler
  • Qwiratry::Mold::Registry
  • Qwiratry::Mold::Slang
  • Qwiratry::MovingFrameAggregateFunction
  • Qwiratry::Operator::Capability
  • Qwiratry::Operator::IO
  • Qwiratry::Operator::MapReduce
  • Qwiratry::Operator::Navigation
  • Qwiratry::Operator::Set
  • Qwiratry::Query::Evaluator::Eager
  • Qwiratry::Query::Evaluator::Filter
  • Qwiratry::Query::Evaluator::Join
  • Qwiratry::Query::Evaluator::Lazy
  • Qwiratry::Query::Evaluator::MapReduce
  • Qwiratry::Query::Evaluator::Navigation
  • Qwiratry::Query::Evaluator::Relational
  • Qwiratry::Query::Evaluator::Row
  • Qwiratry::Query::Evaluator::Set
  • Qwiratry::Query::Evaluator::Union
  • Qwiratry::Query::Extract
  • Qwiratry::Query::NamedJoins
  • Qwiratry::Query::RelationCommon
  • Qwiratry::Query::Runtime
  • Qwiratry::Query::Selector
  • Qwiratry::Query::Slang
  • Qwiratry::Query::Slang::Ops
  • Qwiratry::Query::Slang::Topic
  • Qwiratry::Query::Specificity
  • Qwiratry::Query::Topic
  • Qwiratry::QueryCursor
  • Qwiratry::QueryIterator
  • Qwiratry::QueryMatch
  • Qwiratry::Setup
  • Qwiratry::Singleton
  • Qwiratry::Strategy
  • Qwiratry::Strategy::ControlSignal
  • Qwiratry::Strategy::FinishResult
  • Qwiratry::Strategy::RewriteSpec
  • Qwiratry::Strategy::Traversal
  • Qwiratry::Suggest
  • Qwiratry::TaOP
  • Qwiratry::Table
  • Qwiratry::Table::Schema
  • Qwiratry::Transformer
  • Qwiratry::Transformer::Copy
  • Qwiratry::Transformer::TreeRewrite
  • Qwiratry::Tree::Navigator
  • Qwiratry::Tree::Navigator::Base
  • Qwiratry::Tree::Navigator::Filesystem
  • Qwiratry::Tree::Navigator::Match
  • Qwiratry::Tree::Navigator::RakuAST
  • Qwiratry::Tree::Replace
  • Qwiratry::Walker
  • Qwiratry::Walker::Capabilities
  • Qwiratry::Walker::Factory
  • Qwiratry::Walker::Implementation::Table
  • Qwiratry::Walker::Implementation::Tree
  • Qwiratry::Walker::Implementation::TreeWithPath
  • Qwiratry::Walker::Master
  • Qwiratry::Walker::Providing
  • TypedIterator
  • WalkCursor
  • X::Qwiratry

The Camelia image is copyright 2009 by Larry Wall. "Raku" is a trademark of the Yet Another Society. All rights reserved.

Built with Podlite — the markup and publishing tools behind this site.