Qwiratry — Qwiratry-AggregateFunctions-Plan
Qwiratry Aggregate Functions Plan
Status: Writing.Role: Superseding implementation spec for Aggregate Functions inside Qwiratry. The standalone raku-AggregateFunctions/Specification.md remains historical reference only (DQ6 / WQ6) — do not edit that repo for this work.
Goal: Declare aggregate functions as classes, run them with aggreduce
(collapsing / Postgres GROUP BY) and aggmap (row-preserving / Postgres
WINDOW), drive item streams from Qwiratry Walkers / Iterators (not a
private AggregateWalk), wire existing ReduceOperator (⌿) to
aggreduce, and add ⇶ (map / optional WINDOW) that calls
aggmap — ordinary per-item map by default, windowing when requested;
no separate GroupByOperator / γ.
Companion docs:
Historical standalone design (reference): raku-AggregateFunctions/Specification.md
Survey / naming spitball: old-design-docs/SQL-Grouping-Spitballing.md
Language redesign (v2 grouping/window stubs): SQL-Features-Plan.md §B.2
TaOP next steps (item 7): Qwiratry-TaOP-Plan.md §11
Walker extension surface: ../Extending/Adding-Walkers.rakudoc
Requirements: RAKUDO_RAKUAST=1 (body rewrite is RakuAST-only).
use 6.e.PREVIEW is allowed for the rewrite hook.
Non-goals for the first landing:
Will do later:
SQL text generation
Will probably do later:
Query pushdown to SQL (backend)
Could do later:
Whole-tree nested
bodyrewrite (done: §J.4 / §H.3)FILTERon aggregate arguments (still later: §J.4.1)Parallel processing of queries (including
COMBINEFUNC/SERIALFUNC/DESERIALFUNC)PARALLEL -- currently attribute only, unused
Probably not:
SORTOP(not unless there are performance problems)Ordered-set / hypothetical-set aggregates
SSPACE/MSSPACETaOP Relation mutation APIs
Unsure:
FINALFUNC_MODIFY/MFINALFUNC_MODIFYFINALFUNC_EXTRAas its own mechanism
Non-goals forever:
A separate
GroupByOperator/ γ (collapse goes through ⌿ →aggreduce)Keeping a separate legacy
MapOperator(».)A
sub sumthat clashes with Raku’s builtinunitstatements (use if necessary, but prefer class/role declarations instead)Editing or shimming
raku-AggregateFunctionsWrapping input items in a cursor type
Frames in the first Walker cut if
aggmapwithout frames +Rankalready prove the handover
Source of truth for SQL behaviour is Postgres. Method names are kebab-case
(window-start, frame-enter, …).
Question ID scheme: Stable IDs from the AggregateFunctions Spec open/answered lists there are preserved here as AF-Qn. Qwiratry-integration decisions use WQ1–WQ8. The same AF number never appears as both open and decided.
A. Qwiratry integration
A.1. Motivation
raku-AggregateFunctions already defines the roles, catalogue stubs (Sum,
Rank), drivers (aggreduce / aggmap), and a CHECK-time body rewrite. What
it does not reuse is Qwiratry’s traversal stack — hidden AggregateWalk
partitions and visits items itself.
Primary goal: keep the aggregate-function API; replace AggregateWalk
with Qwiratry::Aggregate::Driver over an Iterator (Walker →
QueryIterator when walkable; Iterable .iterator otherwise — WQ1).
Secondary goals:
Route collapse through existing
ReduceOperator(⌿) →aggreduce(no new GroupBy / γ node).Add ⇶ whose evaluator calls
aggmap: ordinary per-item map when no windowing is requested; row-preserving WINDOW whenwindow/sort/frame/ aggregate columns are present (WQ8).MapOperator/».is retired; ⇶ is the sole map operator.Later: SQL Location Walker pushdown of ⌿ / ⇶ without changing how authors declare
Sum/Rank.
A.2. Module layout
| Standalone today | Qwiratry landing |
|---|---|
AggregateFunction | Qwiratry::AggregateFunction |
MovingFrameAggregateFunction | Qwiratry::MovingFrameAggregateFunction |
AggregateFunctions (Sum, Rank, drivers, rewrite) | Split under Qwiratry::Aggregate::… + export from Qwiratry |
Suggested files:
lib/Qwiratry/AggregateFunction.rakumod
lib/Qwiratry/MovingFrameAggregateFunction.rakumod
lib/Qwiratry/Aggregate/Sum.rakumod
lib/Qwiratry/Aggregate/Rank.rakumod
lib/Qwiratry/Aggregate/Driver.rakumod
lib/Qwiratry/Aggregate/BodyRewrite.rakumod
lib/Qwiratry/Operator/MapReduce.rakumod # extend ReduceOperator; add ⇶Roles stay separately use-able and are not re-exported merely by loading
the catalogue. AggregateFunction (role) ≠ driver/catalogue surface.
Entrypoint (WQ4): use Qwiratry; installs the body rewrite and exports
&aggreduce, &aggmap, Sum, Rank, ….
A.3. Driver and Walkers
Walkers own traversal; aggregate instances own state. The Driver pulls walk items (table rows, XML elements, AST nodes, prior matches, … — not a Table-only row type).
Subject ──Walker (or Iterable.iterator)──▶ item stream
│
├─ ⌿ / aggreduce ──▶ collapsed items
└─ ⇶ / aggmap ──▶ mapped items
(window annotate when requested)Façade accepts Iterable / walkable subject.
Item stream: Walker →
QueryIteratorif walkable; else.iterator(WQ1).Driver takes
Iterator(QueryIteratordoesIterator); window-split, sort, lifecycle — not private tree/table navigation.Aggregate “previous” / frame history: Driver loop locals + window buffer (copy onto
$.previous/ …). Not Walkerfetch-prior(walk order ≠ sorted window order).
Concrete Walkers must not grow private Sum / Rank logic; yield items
and let the Driver run, or opt into pushdown via
can-pushdown-aggregate / pushdown-aggregate (capability
aggregation.pushdown; §H step 11). Table / Tree advertise
aggregation enabled with pushdown False.
Strategy.finish / FinishResult are unrelated unless a Strategy explicitly
collects matches then calls the Driver.
A.4. Language surface (operators)
| Survey / Features Plan | Operator | Driver |
|---|---|---|
| Partition + Aggregation (GROUP BY) | existing ReduceOperator (⌿) | aggreduce |
| Map + optional WINDOW | ⇶ (AggMapOperator) | aggmap |
| Deduplication | ⑂ (DeduplicationOperator) | none (not a fold; §A.6 / §H.3) |
Decision (WQ5 / WQ7 / WQ8): Do not add GroupByOperator / γ. ⌿’s
evaluator calls aggreduce. ⇶ (U+21F6) calls aggmap and covers
both:
Normal map — no
window/sort/frame, andbodyhas no aggregate columns → one output item per input item.Windowing — when
windowand/orsort/frame/Sum./Rank.(etc.) are requested → row-preserving partition / frame walk (§E / §F).
HAVING remains σ after ⌿. Glyph ⇶ matches Operators.rakudoc §4
(WQ5). Legacy ». / MapOperator is removed.
Plain Raku aggreduce / aggmap remain available; the operators are the Query
AST / slang entrypoints onto the same Driver.
A.5. First-landing cuts vs later
| Area | First landing? |
|---|---|
Roles + Sum + body rewrite + aggreduce | yes |
MovingFrameAggregateFunction + Rank + aggmap without frames | yes |
Sum in aggmap | whole-window broadcast (WQ3 / was AF-Q13) |
Frames + running / sliding Sum / Avg | yes (§F / §H step 10) |
⌿ → aggreduce; ⇶ → aggmap (map + optional window; WQ7 / WQ8) | after Driver |
aggsub phasers | later (§G) |
DeduplicationOperator / DISTINCT | third round (§A.6 / §H.3) |
| Whole-tree body walk | yes (§J.4 / §H.3) |
FILTER on aggregate arguments | later (§J.4.1) |
A.6. DeduplicationOperator
Postgres DISTINCT / relational-algebra duplicate elimination: remove
duplicate rows or values (deduplication). Not a fold (not ⌿) and not
GROUP BY collapse. Survey:
old-design-docs/SQL-Grouping-Spitballing.md
(historical δ / “Unique”). Chosen glyph: ⑂ (U+2442 OCR FORK) — fork /
converge metaphor for dedupe; not RA δ; not universe ⊙ (XNOR). Implementation
order: §H.3.
Add
DeduplicationOperator(⑂) as a first-class pipeline operator: same subject → item stream shape as other transformers; emit one item per distinct equivalence class (first wins, order-preserving).Postfix ⑂ (no RHS): whole item (
eqv).Infix + Block (sort-shaped, like
⇅):1-arg key extractor → compare keys with
cmp;Same⇒ duplicate (DISTINCT ON-style).2-arg comparator returning
Order→Same⇒ duplicate.
Do not use
⑂ *(Raku Whatever-currying).Keep it separate from
aggreduce/aggmap: authors compose ⑂ then ⌿ (or ⑂ after map) rather than overloading collapse.SUM(DISTINCT …)/COUNT(DISTINCT …)can wait on ⑂ (dedupe then fold) or on catalogue-aware distinct modes later; do not invent a second Driver path only for distinct aggregates.SQL pushdown (§K.1) should map ⑂ →
SELECT DISTINCT/DISTINCT ONwhen the Location supports it.FILTER (WHERE …)on aggregate arguments is a different feature (row predicate into one aggregate); see §J.4.1, not Deduplication.
B. Terminology
A class that
does AggregateFunctionis a reusable aggregate function. An instance is one in-progress evaluation (one window, or one column in oneaggmapcall).aggreduce/aggmap(viaAggregate::Driver) sort, split into windows, call lifecycle methods, then evaluatebody.aggmapwithout windowing args is ordinary map: evaluatebodyonce per item, no partition / frame / aggregate lifecycle (WQ8).
Window vs frame
Window = set of items sharing the same window key (window => { … }).
Same categorisation as SQL GROUP BY / PARTITION BY: keys eqv.
Postgres also uses “window” for the whole OVER (…) clause; here window is
the keyed set only. sort = order inside that window; frame (§F) =
which items inside the window count for this output item.
| Term | Meaning here |
|---|---|
| Window | Items with the same window key. Optional; omit ⇒ one window over all input. |
| Frame | Subset of the current window for the current output item (ROWS / RANGE / GROUPS). aggmap only. |
window parameter | Block that extracts the key. |
$.window-label | Current key value on the instance. |
What moves: the frame, not the window. Postgres “moving-aggregate mode”
is a moving frame start (MINVFUNC).
aggreduce— walk the whole window, emit once (collapse). Noframe.aggmap— walk the window (or itsframe), emit once per input item.
Parameters (both): window, sort (1-arg easy path; 2-arg like Raku
sort), body, frame (aggmap only).
Hash items: use .<region> (not .region) unless items are objects.
C. Reference implementation state (standalone)
Historical note: describes raku-AggregateFunctions at the time of the
standalone spec. In Qwiratry, land under Qwiratry:: paths in §A.2; there is
no AggregateWalk — use Qwiratry::Aggregate::Driver.
C.1. Standalone tree (for reference)
lib/AggregateFunction.rakumod (→ lib/Qwiratry/AggregateFunction.rakumod)
No
unit. RoleAggregateFunction[::StateType].Enums:
ModificationType,ParallelState(Safe Restricted Unsafe).Lifecycle kebab-case:
window-start/window-end.rowstill a required stub.$.state,@.parameters,$.current,$.previous,$.next,$.counterareis rw.
lib/MovingFrameAggregateFunction.rakumod (→ lib/Qwiratry/MovingFrameAggregateFunction.rakumod)
use AggregateFunction;(→use Qwiratry::AggregateFunction;). Listed inMETA6.jsonprovides.Frame lifecycle:
frame-start/frame-enter/frame-exit/frame-end; helperstrigger-frame-start/trigger-frame-enter(window key from the driver, not from the item; noset-value).
lib/AggregateFunctions.rakumod (→ Qwiratry Aggregate catalogue + Driver + rewrite)
class Sum does AggregateFunction[Numeric](skip nulls; empty →Nil). Stubclass Rank. Exported stubaggmap/aggreduce. Body rewrite /aggreducewalk is §D.3 / §J.2.
t/00-use.rakutest checks use of the three modules (adapt paths under Qwiratry).
C.2. Shared plumbing
use, notunit. Authorsuse Qwiratry::AggregateFunction;anduse Qwiratry::MovingFrameAggregateFunction;separately. Do not export those roles fromAggregateFunctions. List each inMETA6.jsonprovidesso installs resolve. Filename already matches theusename.is exporton the role only ifusedoes not make it visible otherwise.Export from
AggregateFunctions:aggreduce,aggmap, and the callable classes (Sum,Rank, laterCount/Avg/DenseRank). Nosub sum. Replace the fez description.is rwon attributes thataggreduce/aggmapassign:$.state,@.parameters,$.current,$.frame-state,$.window-label, and any$.previous/$.next/$.counterkept on the aggregate function.Kebab-case lifecycle:
window-start,window-end,frame-start,frame-end,frame-enter,frame-exit,trigger-frame-start,trigger-frame-enter.Items stay ordinary. Do not add
newpartition/window-value/set-value/counter/previouson items. The driver keeps$previous/$next/$counteras loop locals and copies them onto the instance ($.previous/ …). No cursor type (AF-Q1).Tests: keep
t/00-use.rakutest.SumandRanklive inlib/Qwiratry/Aggregate/…(catalogue).
Rename map (old lib/ / spitball → this spec). Window-level fold is window-*; moving-frame extras are frame-*. Old window_start / window_end on MovingWindowAggregateFunction were MINITCOND / MFINALFUNC (now frame-*). Old run_start / run_end on AggregateFunction were INITCOND / FINALFUNC (now window-*).
| Old | New | Concept |
|---|---|---|
MovingWindowAggregateFunction | MovingFrameAggregateFunction | the role |
run_start / run-start | window-start | INITCOND; once per window |
run_end / run-end | window-end | FINALFUNC; once per window |
$.window_state | $.frame-state | MSTYPE |
window_start (moving role) | frame-start | MINITCOND |
window_enter | frame-enter | MSFUNC |
window_exit | frame-exit | MINVFUNC |
window_end (moving role) | frame-end | MFINALFUNC; optional |
trigger_window_start | trigger-frame-start | helper |
trigger_window_enter | trigger-frame-enter | helper |
parameter window | (keep) | key extractor Block |
$.window_label | $.window-label | current key value |
row | (keep) | SFUNC |
D. aggreduce
Postgres analogue: SELECT … agg() … FROM t GROUP BY keys — collapses each group to (usually) one row.
HAVING is not a parameter: filter with ordinary grep.
D.1. AggregateFunction role
Postgres analogue: ordinary CREATE AGGREGATE (no moving-frame extras). One state value per evaluation; one result after the last row.
Loading the role
Do not add unit. use Qwiratry::AggregateFunction; loads lib/Qwiratry/AggregateFunction.rakumod by name. Add a provides entry. MovingFrameAggregateFunction.rakumod already uses it; AggregateFunctions.rakumod must too.
Intended semantics
| Raku | Postgres | When |
|---|---|---|
StateType | STYPE | compile time |
$.state | internal | mutated during a run |
@.parameters | call args | set when the instance is made |
$.current | current item | raw item; each row |
window-start | INITCOND | start of a window |
row | SFUNC | each row; receives arg values |
window-end | FINALFUNC | end of window; return value |
$.sort | SORTOP | stub only (Nil) |
$.parallel | PARALLEL | stub only |
Notes kept out of the table:
rowupdates$.state. It does not have to return the user-visible value.Read path: because state lives on the object, we do not need Postgres’s extra
FINALFUNC/MFINALFUNCas separate required methods.CALL-ME(instance) reads$.state/$.frame-stateand returns the user-visible value (Avg: divide here;Sum: return the number). Defaultwindow-end/frame-endcan be aliases of that, or omitted in favour ofCALL-MEonly.Default
window-startmay leave$.stateuninitialized.Summust not treat empty as0.rowis stub ({...}) and must be implemented. Signature:method row(*@values)—@valuesare this row’s evaluated argument expressions, not the Callables.allows-zero-arg: defaultFalseon the role.Rank/Countoverride toTrue(RANK()/COUNT(*)take no value expression). The AST transform uses this whenName.()has an empty argument list (Sum.()vsRank.()).
window-start / window-end are once per window (window key), not per row.
aggreduce: for each window:window-start→rowfor every item → read once (CALL-ME/window-end) → one output hash.aggmap:window-startorframe-startwhen thewindowkey changes (andframe-startagain if a moving frame is restarted empty). Not every output row.window-endonly if this window was a whole-window fold to broadcast; otherwise the per-row read isCALL-MEafter enter/exit. New window → start again, not an extra end hook besides that optionalwindow-end.
Optional on the role: $.previous / $.next (raw items) that aggreduce/aggmap set. Useful for Rank peers. Do not require items to grow those fields. $.next is optional if the loop can peek without storing it.
Callable (no extra sub)
AggregateFunction does Callable. Read of the current result is instance CALL-ME (stand-in for FINALFUNC / MFINALFUNC). Do not call CALL-ME on the type object. The body rewrite (§J.2) constructs one instance per aggregate column in a hygienic lexical and rewrites the column expression to $lexical(). Evaluated body only sees those instances via CALL-ME. Two Sum.({.amount}) columns are two lexicals → two instances.
Result extraction
After the last row of an aggreduce window, read via CALL-ME (or window-end if kept as an alias). Do not clobber $.state in the read path (aggmap may still enter more rows). aggreduce may use a fresh instance per window.
Tests
use Qwiratry::AggregateFunction;thenclass Stub does AggregateFunction[Numeric] { method row(*@) {} }.Missing
rowfails when driven.Default
window-endreturns$.state.
D.2. Sum (Postgres SUM)
Postgres sum(expression): sum of non-null values. Put class Sum in lib/Qwiratry/Aggregate/… (catalogue). Do not add sub sum (Raku already has &sum). In body, write Sum.({.amount}) (see §J.2).
Intended results
| Input | Postgres | This module |
|---|---|---|
1, 2, 3 | 6 | 6 |
| empty (no rows) | NULL | Nil |
| all null | NULL | Nil |
1, null, 2 | 3 | 3 |
single 5 | 5 | 5 |
Skip nulls. After a real value, later nulls do not reset to
Nil.Empty / all-null →
Nil, not0.Numericthis cut. NoSUM(DISTINCT …)/FILTER.Sum.()with no expression is a compile-time error from the AST transform (allows-zero-argis false; PostgresSUM()is a syntax error).Sum()without the postfix dot is aType::Coercion— sorry, writeSum.({…}). Hand-rolledSum.newdriven with no extractors is a runtime throw as a backstop.Moving-window
Sum/Avgare §F (landed).Countwhen the two-column test needs it.
Class sketch
aggreduce sets $_, evaluates {.amount}, then calls row with that number:
class Sum does AggregateFunction[Numeric] does Callable {
method window-start() { $.state = Nil }
method row(*@values) {
my $value = @values[0];
$.state = ($.state // 0) + $value if $value.defined;
}
method CALL-ME() {
# instance only (transform constructs instances; see §J.2)
# return $.state
}
}row does not evaluate {.amount} itself. @.parameters still holds the Blocks so aggreduce knows what to evaluate. StateType must allow Nil, or use a “seen any” flag.
Tests
Hand-roll window-start / row / window-end first, then via aggreduce:
[1, 2, 3]→6empty →
Nilnull among numbers ignored
all null →
Niltwo
Sum.({.amount})columns do not share stateSum.()throws
D.3. aggreduce / Driver collapse
Stub: sub aggreduce(%parameters, @values) {}.
Call shape
@sales.aggreduce(%(
window => { .<region> }, # omit ⇒ one window
sort => { .<sale-date> }, # 1-arg; optional this cut
body => -> {
region => .<region>,
total => Sum.({.<amount>}),
n => Count.(),
},
))Descending or other special orders use a 2-arg Block, same as Raku sort:
sort => { $^b.<amount> <=> $^a.<amount> }Export a sub and a method @values.aggreduce(%parameters). Test the method form the spitball used.
What body is
With commas, a pointy block of fatarrows is a List of Pairs. Hashify it so later |%_ works.
-> { … } has no implicit $_ parameter. Set the topic, then call with zero args:
$_ = $item;
my %out = %parameters<body>().Hash;Around the body (ordinary Raku)
Per item / window, aggreduce does not leave row to the body’s Sum.({.amount}) call:
$_ = $item;$.current = $item(and$.previousif we keep it on the role).For each registered instance: evaluate its argument Blocks (with that
$_);row(|@results).$_ = $itemagain if needed; evaluatebody.$__AGGFN_N()CALL-MEreturns the already updated result.
collect-aggregate-functions was a placeholder; inline registration via the body rewrite (§J.2).
window-start must run before the first row, so instances must exist before the item loop. The transform creates them in hygienic lexicals and binds $*AGGREGATE-FUNCTION-INSTANCES for that call. aggreduce / aggmap read that dynamic; they do not scan body at runtime.
CHECK / BEGIN / COMPILE on this module do not see inside a Block the caller stored in a hash.
No cursor on items
aggreduce can group with loop locals only (key from window Block, eqv for “same window”). It does not need .previous / .counter / .window-value / .set-value on items.
Rank later still needs a previous item and a window counter: attributes on AggregateFunction / Rank, set by aggmap from loop locals (AF-Q1).
Algorithm
Split by
window(or one window if omitted). Window order: first-seen. Key equality:eqv(not===). AListkey would break under===.Register from
$*AGGREGATE-FUNCTION-INSTANCES(one instance per rewritten column). Missing dynamic ⇒ the caller was not rewritten (RAKUDO_RAKUAST=1/use Qwiratry); throw.Reject functions that only make sense as a per-row window walk (
Rankis invalid in SQLGROUP BYwithout a subquery).Sumis valid.Each window:
window-start; each item: evaluate args,row(|@values); one output hash. Prefer SQL: output columns are window keys or aggregate functions.Cardinality = number of windows. No rows and
windowpresent ⇒ zero rows. No rows andwindowomitted ⇒ one row (SELECT sum(x) FROM twith no rows). EmptySumisNil.
Tests
window => { .<region> }withSum.({.<amount>}).No
window: one row; empty input → one row,total => Nil.Rank.()inaggreducethrows onceRankexists.Output length = distinct keys under
eqv.Window keys appear in the result.
Method vs sub.
E. aggmap
Postgres analogue (windowing mode): SELECT … window_function() OVER
(PARTITION BY … ORDER BY …) — row-preserving. Frames: §F.
Dual mode (WQ8): aggmap / ⇶ is also ordinary map. With no
window / sort / frame and no aggregate columns in body, emit one
transformed item per input item — ordinary map cardinality, without
aggregate lifecycle. Windowing turns on when those args / Name.() columns
are present.
E.1. MovingFrameAggregateFunction
The windows are the same as aggreduce. aggreduce / aggmap do the walk. Roles only supply callbacks those two call.
No middle WindowAggregateFunction. AggregateFunction is enough for a fold of a window (window-start / row / window-end): aggreduce emits once; aggmap can write that value on every row, or keep calling row/frame-enter as the frame end moves (rows only enter).
MovingFrameAggregateFunction is the moving-frame role (frame-exit / MINVFUNC). The window is fixed. Expanding ROWS UNBOUNDED PRECEDING … CURRENT ROW never exits. aggreduce never exits. Rank never exits (ignores frame; one row per item).
The full rename map is in §C.2. Window-level fold is window-*; moving-frame extras are frame-*.
row vs CALL-ME (object state; no required frame-end):
row/frame-enter/frame-exit— write the object’s state.Read is
CALL-MEon the instance (whatbodysees). Same method foraggreduce(once per window) andaggmap(once per output row). ForSum, that is$.state. ForAvg, derivesum/counthere. Postgres needed a separateFINALFUNCbecausesfuncreturns the next state value; we mutate in place, so we do not need a third required method.window-end/frame-end— optional aliases of that read. First cut:CALL-MEis enough. Keep the names when a phaser slang wantsWINDOWLAST/FRAMEFINAL/RESULT.
window-start / window-end once per window, not per row (see §D.1). frame-start as well when the window key changes, and again if the moving frame is restarted empty.
Default frame-enter can still be self.row(|@values) on the moving role so sliding Sum shares row with collapse.
MovingFrameAggregateFunction API
Postgres: MSFUNC, MINVFUNC, MSTYPE, MINITCOND, MFINALFUNC. This is the role API; frame which items is §F.
| Raku | Postgres | When |
|---|---|---|
$.frame-state | MSTYPE | slides with the frame |
frame-start | MINITCOND | new window or restart after empty frame |
frame-enter | MSFUNC | row enters; updates frame state |
frame-exit | MINVFUNC | row leaves; updates frame state |
frame-end | MFINALFUNC | optional; default = CALL-ME (read) |
Read for this output row is CALL-ME, not a required frame-end. End of window is window-end only for aggreduce / whole-window broadcast (once per window).
Defaults:
frame-enter→self.row(|@values)(args already evaluated, same as §D.3).frame-exit→ default throwsX::StubCode= unavailable;aggmaptreats that as “restart the frame” (AF-Q16). A bare{...}on the role would be a Raku required method (every class must override), so the role supplies a concrete throwing body with the same detection. Do not override with an emptyframe-exit {}(looks like a successful no-op undo). Real inverses override with a working body.Rankkeeps the default;aggmapmust not callframe-exitforRankanyway.frame-end→$.frame-state.frame-start→ no-op{}(override when there is aMINITCOND; same idea aswindow-start).
There is no middle role. MovingFrameAggregateFunction adds frame-state / enter / exit / end on top of AggregateFunction.
Do not override these
trigger-frame-start: set $.window-label from the loop-local window key, not $current.window_label(). Then frame-start.
trigger-frame-enter: frame-enter. Do not $.current.set-value($.frame-state). body builds the output hash.
Tests
use Qwiratry::MovingFrameAggregateFunction;composes withInt, Int.Default
frame-entercallsrow.Default
frame-endreturns$.frame-state.Default
frame-startis a no-op.Default
frame-exitthrowsX::StubCode⇒ unavailable ⇒aggmaprestarts the frame (AF-Q16).
Sliding Sum: §F.
E.2. Rank (Postgres RANK)
Postgres rank() → bigint: rank with gaps = row_number of the first row of the peer group.
Peers: window ORDER BY keys only (eqv on those keys). Not a full-row compare, not SELECT DISTINCT. No sort ⇒ every row in the window is a peer ⇒ every rank is 1. A new window key restarts numbering. Ignores frame.
DENSE_RANK / ROW_NUMBER only as needed for the spitball (dense_rank).
Intended results
| Situation | RANK() |
|---|---|
| First row in window | 1 |
Consecutive peers on ORDER BY keys | same rank |
| After a peer group of size k that started at n | n + k |
Example, one window, scores 90, 80, 80, 70 ordered DESC: ranks 1, 2, 2, 4.
DENSE_RANK:1, 2, 2, 3ROW_NUMBER:1, 2, 3, 4
1-arg sort => { .<amount> } is ASC (Raku cmp). DESC is the 2-arg form.
Class in AggregateFunctions.rakumod
Callable class Rank (capital R, parallel to Sum; no sub rank). does MovingFrameAggregateFunction; keeps default frame-exit (unused). aggmap must not call frame-exit for Rank. Peer detection and read path: AF-Q14 / AF-Q15 (§M.2).
Fixes to the spitball class rank:
method row(*@values)—Rankhas no SQL arguments; ignore@values.Previous item:
$.previouson the aggregate function (or a loop local passed in), not$item.previous.Peers:
eqvon sort keys, not$prev.id != $_.id.Counter: 1-based index in the window, kept by
aggmap(loop local or$.counteron the instance), not$item.counter. Non-peer:$.frame-state = $counter; peer: leave it.After
frame-start, first enter must become1, not stay0.No
newpartitionon items. Window change:not $key eqv $prev-keyinaggmap.
Tests
Unique keys, 1-arg sort:
1..Nin that order.Same with 2-arg DESC: who gets
1flips.Ties: same rank, then a gap.
window => { .<region> }: restart at1.Empty input: empty output.
Two rows that differ in an unsorted column but match sort keys share a rank.
E.3. aggmap / Driver map
Normal map (no windowing):
@array.aggmap(%(
body => -> { name => .<name>.uc, amount => .<amount> },
))
# or, once the operator exists: $relation ⇶ { … }Windowing (aggregates / partition):
@array.aggmap(%(
window => { .<region> },
sort => { .<amount> }, # 1-arg; easy
body => -> {
region => .<region>,
salesperson => .<salesperson>,
sale-date => .<sale-date>,
amount => .<amount>,
region_rank => Rank.(),
region_dense_rank => DenseRank.(),
},
))Same cardinality as the input. Hashify body. No frame yet (§F). Until frames (§F): ranking sees every item in order; plain Sum broadcasts one whole-window value onto every item (WQ3 / AF-Q13).
Same sub + method export as aggreduce.
Sort then windows
Match Raku sort:
1-arg
{ .<amount> }: Schwartzian key, thencmp. “Same key” is$a cmp $b == Same(treat aseqv, not===).2-arg: only when 1-arg cannot express the order.
Sort so windows are contiguous (window key, then sort). New window: window key !eqv previous. window omitted ⇒ one window. sort omitted ⇒ keep input order inside the window.
A 1-arg window Block that returns a List must use eqv.
body
Same rules as aggreduce body. Fields that are not aggregate functions (region, salesperson, …) are just ordinary Raku in that block (.<region>). No extra “pass-through” pass.
Do not assign $item = $agfunc.frame-state.
Per-row loop without frames
Drop stub bugs: window-start every row; does instead of ~~; methods on items; replacing the item with window state. Keep $current / $previous / $next as loop locals (as the stub comment already wants).
sort (window key, then user sort)
register instances from $*AGGREGATE-FUNCTION-INSTANCES
for each item:
if window key !eqv previous:
$counter = 0; trigger-frame-start (or window-start)
$counter++
set $.current / $.previous / $.counter on instances if those attrs exist
evaluate arg Blocks; frame-enter / row(|@values)
$_ = $item; hashify body() # CALL-ME returns current values
yield output hashTests
Spitball rank example: PARTITION BY region + order by amount. Output length = input. window omitted; empty list; single row; method vs sub. Framed running sum: §F.
F. Frames
The second and third chained aggmaps. Rank still ignores frame.
@array.aggmap(%(
window => { .<region> }, # omit for company running total
sort => { .<sale-date> }, # 1-arg
frame => …,
body => -> {
|%_,
company_to_date => Sum.({.<amount>}),
# region_avg_7d => Avg({.<amount>})
},
))(avvg in the spitball is avg.)
F.1. Frame parameter
frame is a hash passed to aggmap only. It selects which rows in the current window count for each output row. Defaults and SQL analogues:
| Default | When | Postgres analogue |
|---|---|---|
| whole window | no sort | frame = entire PARTITION BY set |
range / unbounded preceding … current | sort present, frame omitted | RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW |
Keys
| Key | Type | Meaning |
|---|---|---|
type | rows | range | group | How bounds are measured (Postgres ROWS / RANGE / GROUPS). |
start | see below | Lower bound (integer, sentinel, or omitted if bounds is a Range). |
end | see below | Upper bound. |
bounds | Range | Absolute integer start/end together. Implies start-type / end-type are absolute. rows / group only. |
start-type | relative | absolute | How to interpret start. Default relative. Ignored when bounds is set. |
end-type | relative | absolute | How to interpret end. Default relative. Ignored when bounds is set. |
exclude-current | Bool | Drop the current row from the frame. |
exclude-ties | Bool | Drop ordering peers of the current row. Does not drop the current row unless exclude-current is also true. |
exclude-* together map to Postgres EXCLUDE (neither / current row / ties / group).
type and the type of start / end
start and end are not always the order-key type. They depend on type:
type | start / end are | Example |
|---|---|---|
rows | Integer row offsets (or relative sentinels; see below) | relative -1 … 0 ⇒ one preceding row through current |
group | Integer peer-group offsets | relative -1 … 0 ⇒ previous peer group through current group |
range | Delta on the order key — same kind as sort produces | Duration for dates (-6 days … 0); numeric difference for numbers |
For range, bounds are applied by subtracting/adding on the sorted order key (Postgres RANGE). For rows / group, bounds are counts in the ordered row list or peer-group list.
relative vs absolute
Relative (default): offset from the current row in the ordered window. Integers and named sentinels only — not a Raku Range (a Range has no notion of “current row”).
0= current row (or current peer group whentypeisgroup).Negative = preceding; positive = following.
Matches Postgres
n PRECEDING/CURRENT ROW/n FOLLOWING.Unbounded / current as strings:
unbounded-preceding,unbounded-following,current-row.
Absolute: index into the current window in that window’s sort order. Same shape as Postgres cursor MOVE ABSOLUTE n, not SQL frame n PRECEDING. Not a CREATE AGGREGATE feature; aggmap interprets it when sliding the frame.
| Absolute value | Meaning |
|---|---|
0 | Postgres UNBOUNDED PRECEDING (start of the window; first row) |
Positive n | nth row from the window start, 0-based (1 = second row) |
-1 | Postgres UNBOUNDED FOLLOWING (end of the window; last row) |
Negative n other than -1 | from the window end (-2 = second-to-last) |
So 0..-1 is the whole window. First included row is 0; last included row is -1.
Storing absolute bounds as a Raku Range
A Range is absolute only, and only a shorthand for integer start/end when type is rows or group. Put it in bounds; that implies start-type and end-type are absolute. Do not put a Range in start / end.
0..-1— whole window (UNBOUNDED PRECEDING…UNBOUNDED FOLLOWING).0..4— first through fifth row.1..-2— second row through second-to-last.
Do not use a Range for relative frames. Relative uses start / end integers or the named sentinels (unbounded-preceding, current-row, unbounded-following). Prefer separate start / end when start-type / end-type differ, or when type is range with Duration bounds.
Examples
Running sum (company-wide, no window key) — relative:
frame => %( type => 'rows', start => 'unbounded-preceding', end => 'current-row' )Sliding sum, one row back through current — relative:
frame => %( type => 'rows', start => -1, end => 0 )Seven-day average by region (order key is date) — relative:
frame => %( type => 'range', start => -6 days, end => 0 )Whole window as an absolute Range:
frame => %( type => 'rows', bounds => 0..-1 ) # start-type/end-type implied absoluteAbsolute slice — second through fifth row of the window:
frame => %( type => 'rows', start => 1, end => 4, start-type => 'absolute', end-type => 'absolute' )
# or: bounds => 1..4Exclude current row and its sort-key peers — relative:
frame => %( type => 'rows', start => -1, end => 1, exclude-current => True, exclude-ties => True )F.2. How aggmap should slide
For each output row:
Window key
!eqv:trigger-frame-start(do notframe-exitthe old window).frame-exiteach row that left the frame (oldest first). Ifframe-exitis unavailable ⇒ restart.frame-entereach row that entered (evaluate that row’s args, then enter).Evaluate
body/frame-endfor the current item.
Empty frame ⇒ restart, do not frame-exit the last row. Moving end only needs frame-enter. frame-exit is for a moving start.
F.3. Sum in moving-frame mode
Same class Sum:
frame-start:Nil, likewindow-start.frame-enter:row(|@values)(add, skip nulls).frame-exit: subtract (or restart ifNumis painful; tests preferInt/Rat).frame-end:$.frame-state(Nilif the frame has no non-null values).
Avg is Sum/Count over the same frame.
F.4. Tests
Sliding
Sum,ROWS BETWEEN 1 PRECEDING AND CURRENT ROW; restart path matches.Running sum: no
windowkey,ROWS UNBOUNDED PRECEDING … CURRENT ROW.Range:
windowby region, 6 days preceding … current.Rank.()beside a framedSumignores the frame.Empty / all-null frame:
SumisNil.
G. Phasers (aggsub) — later
Later definition spelling, not the first cut. Classes + methods remain the implementation. aggreduce/aggmap still do the walk and invoke the phasers. This is not a replacement for calling enter/exit from aggmap.
Sketch:
aggsub Rank() {
WINDOWFIRST { $.frame-state = 0 }
if ! self.is-duplicate($previous, $current) {
$.frame-state = $current.counter;
}
}That slang compiles to a class. Possible map:
| Phaser | Method | When |
|---|---|---|
WINDOWFIRST | window-start | start of a window |
WINDOWLAST | window-end | end of a window (collapse result) |
| unphased body | row | each item / each enter |
FRAMEFIRST | frame-start | fresh frame state (new window or restart) |
FRAMEADD | frame-enter | row entered the frame |
FRAMESUBTRACT | frame-exit | row left; only moving frame start |
FRAMEFINAL | frame-end | value for this output row (read/CALL-ME) |
RESULT | CALL-ME | aggregate result for the body (optional) |
Colonpair :!aggreduce ⇒ allows-aggreduce False (ranking-style). Empty
signature ⇒ allows-zero-arg.
FRAMEFINAL maps to frame-end (MFINALFUNC): it runs every output row after enter/exit, returning this aggregate’s value for the row — not once at the end of the window (WINDOWLAST / window-end already covers that). Default is still “current $.frame-state” / CALL-ME; the phaser is for non-trivial finalize (e.g. Avg divide). Avoid FRAMELAST: LAST suggests once-at-end, which mismatches per-row emit. RESULT overrides CALL-ME when stored state is not already the body-visible value (e.g. packed (sum, count)).
Useful as a later spelling of class Rank does ….
H. Implementation orders
H.1 Basics Implementation
1. Roles under
Qwiratry::; kebab-case lifecycle;is rw; nounit; roles not re-exported from the catalogue (§C.2 / §A.2).2.
AggregateFunction+Callable+row(*@values)(§D.1).3.
class Sumin the Qwiratry catalogue, nosub sum(§D.2).4. Body rewrite (§J.2) installed from
use Qwiratry(WQ4) +aggreduce/Aggregate::Drivercollapse overIterator(WQ1); args thenrow(§D.3).5. Wire façade: walkable → Walker/
QueryIterator; Iterable →.iterator.6.
MovingFrameAggregateFunctiondefaults — no middle role (§E.1).7.
Rankon ORDER BY keys +eqv+ counter not on items (§E.2); resolve AF-Q14 / AF-Q15 while implementing.8.
aggmapwith 1-argsort+ rank example;Sumbroadcasts (WQ3) (§E.3).9. Wire
ReduceOperator(⌿) →aggreduce; add ⇶ →aggmap(WQ7 / WQ8 dual mode).10. Frames + running / sliding
Sum/Avg(§F);frame-exitunavailable per AF-Q16 (X::StubCode→ restart).11. Capability / pushdown hooks on Table / future SQL Walkers.
Align with TaOP Plan §11 items 7–8 after Textish / CSV. Do not modify
raku-AggregateFunctions (WQ6).
Note: AF-Q14 / AF-Q15 / AF-Q16 are decided (§M.2). Remaining checklist items use those decisions (especially frames / AF-Q16 in step 10).
H.2 Advanced Implementation
12. §G —
aggsubphasers (definition slang for aggregate classes).13. Rewrite existing aggregate functions (eg. Sum, Rank) to use aggsub
14. Catalogue
CountandDenseRank(parallel toSum/Rank).15. §N testing note (c) — File → Parse → Table pipeline integration for
aggreduce/aggmap.
H.3. Third Round Implementation
16. §A.6 —
DeduplicationOperator(⑂, U+2442): sort-shaped 1-arg / 2-arg Block or postfix whole-item; compose with ⌿ / ⇶; not a second Driver. Pushdown mapping when §K.1 exists.17. §J.4 — whole-tree
bodywalk (nested blocks /if/when/ ternary); always register nested aggregates;FILTERremains later (§J.4.1).
I. Postgres CREATE AGGREGATE checklist
| Feature | This cut |
|---|---|
SFUNC / STYPE / FINALFUNC / INITCOND | yes (row, StateType, window-end, window-start) |
| arguments | @.parameters hold Blocks; row gets evaluated values |
MSFUNC / MINVFUNC / MSTYPE / MFINALFUNC / MINITCOND | yes (§E.1 / §F) |
PARALLEL on the object | attribute only, unused |
SSPACE / MSSPACE / FINALFUNC_MODIFY / MFINALFUNC_MODIFY | no |
COMBINEFUNC / SERIALFUNC / DESERIALFUNC | no |
SORTOP / ordered-set ORDER BY / HYPOTHETICAL | no |
FINALFUNC_EXTRA as its own hook | no; CALL-ME / window-end may read @.parameters |
J. Discovery / body rewrite
aggreduce / aggmap must register one instance per aggregate column before the first window-start / row. Find anything whose type does AggregateFunction (including MovingFrameAggregateFunction), not a hardcoded Sum / Rank name list. Two Sum.({.amount}) columns must not share an instance.
CHECK / BEGIN / COMPILE on this module never see inside a body Block the caller stuffed into a hash. Chosen path (Q4): a post-parse AST transform that rewrites the Block in the importing compilation unit so registration is explicit in the compiled code.
Not an option: evaluate body once as a “discover pass.” That extra run can fire pass-through fields and any other side effects. Not chosen: walk RakuAST of the runtime Block (it has none — see Probe).
J.1. RakuAST walk (not chosen)
Inspect $body.AST (or equivalent) without evaluating body. Walk call nodes; if the callee’s type does AggregateFunction, construct/register an instance (argument Blocks into @.parameters).
| Helps | Hurts |
|---|---|
No dummy run of body (no extra .<region> / side effects). | Relies on RakuAST for a Block that arrived as a hash value; may be missing or incomplete. |
Works with empty input (aggreduce with no window still emits one row). | Indirect / computed callees (my $c = Sum; $c({.amount})) are easy to miss. |
Column identity can come from AST position (two identical Sum.({.amount}) stay distinct). | Nested Blocks, conditionals, and macros are more work. |
J.2. Post-parse AST transform (chosen; installed by use Qwiratry)
Installed invisibly from use Qwiratry (Q6). After normal parsing, find each aggreduce / aggmap call (sub and method form) in the importing compilation unit and rewrite that call’s body so registration is explicit in the compiled code. This is a RakuAST transform, not a new grammar.
| Helps | Hurts |
|---|---|
Can make body look exactly like the spitball and still be typed. | Bigger tool; separate from §G aggsub (that slang is for defining aggregate classes, not for scanning body). |
| Handles empty input if rewrite is compile-time. | Separate from §G aggsub (defining classes, not scanning body). |
Rewrites Rank.() / Sum.({…}) (ApplyPostfix + Call::Term) to instance $var() without type-object CALL-ME. |
Hook. Prefer attaching a post-parse visitor to the importing compilation unit from EXPORT / use (use 6.e.PREVIEW allowed). No custom grammar or actions unless that hook cannot be installed. Fall back to Slangify only if needed to get a statement-list / compunit callback. Require RAKUDO_RAKUAST=1. Keep the transform inline in lib/Qwiratry/Aggregate/… (catalogue) unless the file becomes unreadable.
Walk depth. Fatarrow column values are walked for nested C / C / ternary / blocks so nested C<Sum.()> / C<Rank.()> still register (§J.4). Extractor Blocks on C<Name.({…})> are not scanned for nested aggregates (those stay on C<@.parameters>).
Documented spelling: Name.()
In body, aggregate columns are calls with an explicit .() postfix:
| Write | Meaning |
|---|---|
Rank.() / Count.() | zero-arg (allows-zero-arg) |
Sum.({.<amount>}) | extractor Block stored on @.parameters |
Probe (Rakudo 2026.07). Rank.() is ApplyPostfix (Type::Simple + Call::Term with empty ArgList) — a real call, not Type::Coercion. Sum.({.<amount>}) is the same node type with a Block argument. Runtime: Rank.() does invoke type-object CALL-ME, so the transform still rewrites to $__AGGFN_N() (Q8).
Do not write Sum.(.<amount>) (no Block): that is also ApplyPostfix, but the argument is Term::TopicCall evaluated immediately on $_, not captured as an extractor. Compile-time sorry: extractor must be a {…} Block.
Rank() / Sum() without the postfix dot remain Type::Coercion (Rank(Any)). Compile-time sorry with a hint to write Rank.() / Sum.({…}). Int() is ordinary coercion — leave it alone. Sum({.<amount>}) without the extra dot is also ApplyPostfix (same as Sum.({.<amount>})); accept it, but document the .() form.
Body rewrite (Q5)
For each top-level aggregate column in body:
Find the node — a
RakuAST::ApplyPostfixwhose operand is a type name and whose postfix isCall::Term(Name.()/Name.({…})/Name({…})).Resolve the type — look up the name; keep it only if the class
does AggregateFunction(not a hardcoded name list). OtherType.()/Type()nodes are ordinary Raku — leave them alone.Zero-arg vs extractor — empty
ArgListis valid only ifallows-zero-arg(defaultFalse;Rank/Counttrue).Sum.()is a compile-time sorry (PostgresSUM()is a syntax error). Non-empty args must be Block(s); copy them onto@.parameters.Sum.(.<amount>)(topic call, not a Block) is a sorry.Inject a hygienic lexical — one per column, counter incremented per
aggreduce/aggmapcall site (e.g.$__AGGFN_0001,$__AGGFN_0002). Names are transform-generated (not user source), so they do not collide with$r,$rank, or other caller lexicals. Initialize withType.new(and@.parameterswhen the source had a{…}extractor).Rewrite the column expression to
$__AGGFN_N()— instanceCALL-MEreturns the current value afterrow/ frame hooks have run. Zero-arg:Rank.()→$__AGGFN_0001(). One-arg:Sum.({.<amount>})→$__AGGFN_0002()(extractor already on the instance;bodydoes not pass it again).Wire instances via
$*AGGREGATE-FUNCTION-INSTANCES. Wrap the rewritten call so the hygienic lexicals and the dynamic are in the same scope, bound beforeaggreduce/aggmapruns:
{
my $__AGGFN_0001 = Sum.new: parameters => ({.<amount>},);
my $__AGGFN_0002 = Count.new;
my $*AGGREGATE-FUNCTION-INSTANCES = ($__AGGFN_0001, $__AGGFN_0002);
@sales.aggreduce(%(
body => -> {
total => $__AGGFN_0001(),
n => $__AGGFN_0002(),
},
));
}aggreduce / aggmap read $*AGGREGATE-FUNCTION-INSTANCES (a List of instances) and call window-start / row on those. body only reads via $__AGGFN_N(). Nested aggreduce / aggmap nest the dynamic. An empty list is valid (no aggregate columns). If the dynamic is undefined, the call was not rewritten — throw (missing RAKUDO_RAKUAST=1 or use Qwiratry).
User-facing source is Rank.() / Sum.({.<amount>}). Compiled code never relies on empty Type() coercion or type-object CALL-ME.
J.3. How discovery was chosen (AF-Q4)
This is an empirical question, not a preference vote. Probe the real call shape (body as a Block inside a hash passed to aggreduce / aggmap):
Does that Block have a usable AST (
$body.ASTor equivalent)?Can a walk find
Sum.({.<amount>})andRank.()(typedoes AggregateFunction), including two identicalSumcolumns as two instances?Can we register before any
row, including empty input (no dummy eval ofbody)?
If yes to all three for the first-cut examples → 9.1. If the AST is missing, incomplete, or cannot resolve the callee type → 9.2 post-parse transform. Do not add a discover pass later to dodge a transform.
Probe (Rakudo 2026.07 / 6.d)
Call shape: aggreduce(%( body => -> { … } )).
No. A runtime
Blockhas no.AST.@!compstuffisVMNull..rakuis-> { #(Block|…) ... }.CHECK $*CUexists only in the **caller’s** compilation unit, not insideaggreduce`.Not from that Block.
Str.ASTof similar source works as a separate parse:Sum({ 1 })andSum.({ 1 })areApplyPostfix(type +Call::Term);Rank.()is the same node with an emptyArgList.Rank()without the postfix dot is not a call: the grammar makesType()aType::Coercion(Rank(Any)). Runtime:Rank()does not invokeCALL-ME;Rank.()andSum.({.x})do (type-objectCALL-ME— hence the rewrite).Sum.(.<amount>)is also a call, but the argument is a topic call evaluated immediately, not a Block.Not via 9.1 without a compile-time rewrite: we never get the caller’s AST at
aggreduceruntime.
Chosen: §J.2 post-parse AST transform (fails test 1 of the decision procedure). Documented spelling is Name.(); the transform rewrites those ApplyPostfix nodes to hygienic-lexical $var() calls and binds $*AGGREGATE-FUNCTION-INSTANCES (§J.2 / AF-Q5).
J.4. Whole-tree body walk
Rewrite walks each top-level fatarrow value recursively (nested blocks,
if / when, ternary, and other expressions) so
total => $flag ?? Sum.({.<amount>}) !! 0 still registers Sum. Extractor
Blocks on Name.({…}) are not scanned for further aggregates
(Post-SQL-TaOP-Improvements.md §A.4).
J.4.1. Postgres conditional aggregates in SELECT
Every aggregate in the SELECT list is always computed for the group.
There is no “sometimes this column is an aggregate.”
CASE WHEN … THEN SUM(x) ELSE 0 ENDstill runsSUM; theCASEwraps the result after aggregation.A non-grouped row expression mixed with an aggregate in
CASEis an error (must appear in GROUP BY or be used in an aggregate function).Conditional rows into an aggregate is
FILTER (WHERE …), which remains later — not a nestedifaround the column.
Nested aggregates found by the whole-tree walk are always registered
(drive row every item). if / CASE / ternary in body only affects what
CALL-ME contributes to the output hash, not whether the instance exists.
K. Things to add later
Not this cut. Do not block §D.3 on these.
K.1. Separate SQL Source module
SQL Text Generation (as a separate module, via SQL::Abstract)
Real SQL pushdown (Location Walker that implements
can-pushdown-aggregate/pushdown-aggregate; hooks already exist §H.1.11).
K.2–K.5 — moved
Former §K.2 (Low-hanging Aggregate Functions), §K.3 (Parallel Processing of Queries), §K.4 (TaOP Relation mutation APIs), and §K.5 (Other later items) now live in Post-SQL-TaOP-Improvements.md §A (as §A.1–§A.4).
L. Open Questions
Numbers are stable IDs. AF-Qn = AggregateFunctions Spec IDs (never reuse a number as both open and decided). WQn = Qwiratry-integration decisions (WQ1–WQ8).
L.1. Open (from AggregateFunctions Spec)
(none — AF-Q14 / AF-Q15 decided in §M.2 while implementing Rank.)
L.2. No further WQ open items
WQ1–WQ8 are decided (§M.1).
M. Decided questions
M.1. Qwiratry integration (WQ1–WQ8)
WQ1. Factory path for bare List / Seq. Option B: Driver consumes an
Iterator. Walkable subjects go Walker → QueryIterator; bare Iterable
subjects use .iterator. No Sequence Walker in the first cut. Aggregate
“previous” / frame history stay Driver-side (loop locals + window buffer),
not Walker fetch-prior.
WQ2. agggrep. Typo for aggreduce in TaOP Plan §11.7–8. No third
aggregate driver; filter-after-aggregate is σ / ordinary grep after
aggreduce / aggmap.
WQ3. Sum in aggmap before frames (was AggregateFunctions AF-Q13).
Until frames (§F), plain Sum in aggmap broadcasts the whole-window
total onto every item in that window (Postgres whole-partition default).
Running / sliding sums wait for §F.
AF-Q13 original options: (a) treat early aggmap Sum as running; (b)
broadcast whole-window in early aggmap, defer running to frames; (c)
separate RunningSum. Chose (b).
WQ4. Package entrypoint. Available via use Qwiratry; (aggreduce /
aggmap, body rewrite, catalogue classes). Keep AggregateFunction as the
role module (Qwiratry::AggregateFunction) for authors of custom aggregate
functions; do not confuse that role with the driver/catalogue surface.
WQ5. Map operator glyph. ⇶ (U+21F6, THREE RIGHTWARDS ARROWS),
as in Operators.rakudoc §4. Sole map operator (AggMapOperator);
legacy ». / MapOperator removed. Collapse side already has ⌿ — no
new glyph for GROUP BY.
WQ6. Standalone raku-AggregateFunctions. Leave that repo alone (never
released). Use its Specification and implementation as reference while
porting into Qwiratry. No re-export shim and no cutover edits there.
WQ7. Operators → Driver. ReduceOperator (⌿) calls aggreduce — no
separate GroupByOperator / γ. ⇶ (AggMapOperator) calls aggmap.
Legacy MapOperator / ». removed.
WQ8. aggmap / ⇶ dual mode. Ordinary map when no windowing is
requested (no window / sort / frame, no aggregate columns in
body): one output item per input item. Windowing applies when those
args / Name.() columns are present (§E / §F).
M.2. From AggregateFunctions Spec (AF-Q1–AF-Q13)
Stable IDs from AggregateFunctions Spec §12:
No cursor on items. Walk state is loop locals and attributes on the aggregate-function instance (
$.previous/$.next/$.counter). Items stay ordinary hashes/objects.window-value: loop-local key /$.window-label.set-value:bodybuilds the output hash; never$.current.set-value(...).windowis the key parameter. Matchesframeand the spitball. Postgres: the frame moves; the keyed window (item set) does not. Reopen only ifwindow => { .<region> }feels too much like the wholeOVER (…)spec rather thanPARTITION BY.Middle role: no. Walks live in
aggreduce/aggmap(Driver).frame-exitonly when the frame start moves.AggregateFunction+MovingFrameAggregateFunctionis enough.Discovery: post-parse AST transform (§J.2). A runtime
bodyBlock has no RakuAST; do not evaluatebodyas a discover pass. After normal parsing, findaggreduce/aggmapin the importing compilation unit and rewritebody. FindName.()/Name.({…})(ApplyPostfix+Call::Term) whose typedoes AggregateFunction, not a name list.Name()without the postfix dot isType::Coercion— sorry, writeName.(). No custom grammar unless the hook cannot be installed otherwise (thenSlangify). RequireRAKUDO_RAKUAST=1.use 6.e.PREVIEWallowed. Install fromuse Qwiratry.Documented spelling is
Name.().Rank.()/Count.()/Sum.({.<amount>})parse asApplyPostfix+Call::Term. The transform injects a hygienic lexical per column and rewrites to$__AGGFN_N().Rank()without the dot is coercion — compile-time sorry with a hint.Sum.(.<amount>)evaluates the topic immediately — sorry. After rewrite,bodyonly reads instanceCALL-ME.Sum.()(allows-zero-argfalse) is a compile-time sorry.Transform is invisible after
use Qwiratry. Ordinary@rows.aggreduce(%( body => -> { total => Sum.({.<amount>}) } ))— no extra keywords. RequiresRAKUDO_RAKUAST=1. If attaching a post-parse walk fromusefails, trySlangify; if that also fails, revisit.One instance per call site. Each aggregate column in
bodybecomes one hygienic lexical and one instance. Two identicalSum.({.<amount>})columns are two lexicals → two instances. Evaluatedbodyreads via$__AGGFN_N()→ instanceCALL-ME. The driver sees those instances via$*AGGREGATE-FUNCTION-INSTANCES(or rewrite__aggfnwiring).No type-object
CALL-ME. Construction is transform-generated hygienic lexicals (Type.new+@.parameters).bodyonly sees instances via$lexical().Extractor Blocks live on the instance.
{.<amount>}is an argument toSum, stored as@.parameters. Driver sets topic, evaluates those Blocks, thenrow(|@values).bodyonly readsCALL-MEafterrow.Empty input cardinality matches Postgres. No items and no
window⇒ one result (SELECT sum(x) FROM t→ one row,NULL). No items withwindow⇒ zero results. EmptySumisNil(NULL).Wire instances with
$*AGGREGATE-FUNCTION-INSTANCES. The rewrite wraps eachaggreduce/aggmapcall so hygienic lexicals and the dynamic are bound before the sub runs. Nested calls nest the dynamic. Undefined dynamic ⇒ not rewritten; throw.bodywalk is whole-tree under fatarrow values (§J.4). Nestedif/when/ ternary / blocks register aggregates; extractor Blocks are not scanned. Non-AggregateFunctionType()/Type.()is left alone.Sum in
aggmapbefore frames — decided as WQ3 (broadcast).No
peer-start/peer-end. Peer detection stays onRank: the Driver copies the call’ssortBlock onto$.sort, sets$.previous/$.current/$.counter(1-based window index), then callsrow.Rank.rowcompares sort keys (eqvfor 1-arg;Samefor 2-arg). No sort ⇒ every row is a peer. Rejected: new role hooks for peer boundaries.RankoverridesCALL-ME(andframe-end) to return$.frame-state. Fold state ($.state/STYPE) and frame state ($.frame-state/MSTYPE) stay separate slots. Body readsCALL-ME; no copy into$.state.frame-exitunavailable = throwX::StubCode. Whenaggmapneeds an inverse andframe-exitis unavailable, restart the frame. Detection: theMovingFrameAggregateFunctiondefault throwsX::StubCode(messageframe-exit unavailable). A bare{...}on the role would be a Raku required method, so the role uses a concrete throwing body with the same signal. Stub/default has no side effects before the throw, so try/catch is equivalent to “do not call”. Do not use an emptyframe-exit {}(successful no-op looks like a working undo). Classes with a realMINVFUNCoverrideframe-exit.Rankkeeps the default;aggmapmust not callframe-exitforRankanyway. Rejected: explicithas-frame-exitflag; “empty body means unavailable”.
N. Testing
Port standalone tests into
t/unit/aggregate-*.rakutestand/ort/spec/Operators/once ⌿ →aggreduceand ⇶ →aggmapexist.Integration:
aggreduce/aggmapover (a) bare List, (b) Table / Tree Walker item streams, (c) later File → Parse → Table pipeline matches.Body rewrite:
Sum.()sorry;Rank()coercion sorry; twoSum.({…})columns do not share state; missing rewrite throws clearly.Empty input cardinality (AF-Q10): no
window→ one result; withwindow→ zero results.Broadcast
Suminaggmapbefore frames (WQ3).aggmap/ ⇶ as ordinary map with no windowing args (WQ8).
O. Success criteria
Sum/aggreducetests pass under Qwiratry modules with noAggregateWalkclass.The collapse driver iterates items from an
Iterator—QueryIteratorfrom a Walker when the subject is walkable, otherwise Iterable.iterator(WQ1).A Table (or Tree) subject can be aggregated without copying partition logic into that Walker.
ReduceOperator(⌿) evaluates viaaggreduce; ⇶ evaluates viaaggmap(WQ7). That path supports ordinary map and optional windowing (WQ8). LegacyMapOperator/».is removed.This document is the implementation spec; standalone AggregateFunctions Spec is reference only (WQ6). Spec cross-links from SQL-Features-Plan §B.2 and TaOP Plan §11.7 point here.