Architecture & deep dive

Minimal usage is in ../README.md; quick start in ./quickstart.en.md.

本文档由 README 拆分而来，保留全部设计细节；仅位置迁移，未删减。

Architecture

bin/raku-pm.raku          Thin entry shell (just forwards args to RakuPM::CLI)
        │
        ▼
RakuPM::CLI               CLI body: command dispatch + every command implementation
        │                 (the 900+ lines that used to live in bin)
        │                 ※ It is a module for one reason only — precomp: bin
        │                   scripts get NO precompilation and are fully recompiled
        │                   on every invocation (~0.75s measured); with the body in
        │                   a module only the thin shell is compiled, cutting
        │                   startup by 0.6~0.8s (0.79.0)
        │
        ▼
RakuPM::Client            orchestrator (à la Zef::Client — coordinates only)
        │
        ├── RakuPM::Repositories    repository *list*: persisted config + aliases + env
        │       │                   (repositories.json decides which indexes are mounted)
        │       ▼
        ├── RakuPM::Resolver       dependency resolver (recursion + conflicts + topo sort)
        │       │                  filters by "does this version really provide the module?", then ranks
        │       │
        │       ├── RakuPM::Repository (role)   repository abstraction (pluggable)
        │       │       │                    └ mixes in Repository::Matching:
        │       │       │                      the single implementation of
        │       │       │                      "keyword ↔ distribution" matching
        │       │       │                      (shared by search/store/installed/install)
        │       │       ├── Repository::Local        local directory repo
        │       │       └── Repository::Ecosystem    remote JSON ecosystem (zef/rea/cpan…)
        │       │
        │       ├── RakuPM::Version     semver + constraint matching
        │       │
        │       └── RakuPM::Spec        install-spec parsing (Foo:ver<1.2.3> / [email protected])
        │                               zero-dependency: shared by install and the query commands
        │
        ├── RakuPM::Store           package store (multi-version + content digest)
        │
        ├── RakuPM::Installer       installer (store → CUR::Installation's site/)
        │                           + transaction/generations + bin wrappers
        │   ├── RakuPM::InstallOptions  carrier for the install-chain options
        │   │                           (collapses 7+ boolean flags into one object)
        │   ├── RakuPM::Ledger          the ONLY reader/writer of installed.json
        │   ├── RakuPM::Fs              recursive delete / copy / walk / size
        │   │                           (single implementation point)
        │   ├── RakuPM::Installer::Generations     format & lifecycle of generations/
        │   │                           (id allocation / before / meta / manifest / current / prune)
        │   └── RakuPM::Installer::ShellTemplates  the three platform shell templates
        │                               (bash / cmd .bat / PowerShell .ps1 — pure string functions)
        │
        ├── RakuPM::Prefix          the ONLY source of the *install prefix*
        │                           default prefix / prefix resolution /
        │                           "is this the default prefix?" / promote default.
        │                           Zero-dependency — the CLI must know the prefix
        │                           BEFORE requiring Client (cold-start path).
        │
        ├── RakuPM::Platform        the ONLY source of platform differences (H9)
        │                           OS detection / path-list separator / executable
        │                           suffixes / MSYS paths / process killing / library
        │                           search paths / md5 command & output format.
        │                           Every function reads $*DISTRO at call time, so tests
        │                           can shadow it (`my $*DISTRO = Distro.new(:name('linux'))`)
        │                           and really exercise all three platform branches anywhere.
        │
        ├── RakuPM::NativeLib       system library (:from<native>) probing
        │                           logical name → per-OS file names + search paths + install command
        │                           (all platform knowledge delegated to RakuPM::Platform)
        │
        ├── RakuPM::Cleaner         cache cleanup (`clean` / `gc`): decides which
        │                           versions are still in use; dry-run by default
        │
        └── RakuPM::Lock            lockfile (exact versions + digests + native-libs,
                                    guarantees reproducibility)

(this one is NOT in the Client tree — the CLI calls it directly, and it has
 nothing to do with the install target)
RakuPM::Author            producer side (authoring): works on a distribution directory
        └── refresh               rebuild META6.json's provides from disk

RakuPM::Client itself is split by responsibility into five roles (SelfManager / Tester / Git / Query / Flusher) mixed in via does, so the orchestrator doesn't grow into a thousand-line god class. External behavior is unchanged.

Startup time: measured breakdown, and what we evaluated and did not do (0.79.1)

How fast startup is, why, and which optimisations were evaluated and rejected — recorded here so the argument does not have to be re-litigated. Measured on one machine (raku -Ilib -e …, best of 7 runs):

Key finding: loading a precomp module costs "one fixed charge + a tiny per-module increment", independent of module size. RakuPM::Installer (1205 lines) and RakuPM::Net (a tiny zero-dependency module) take almost the same time to load alone (0.395s vs 0.342s) — the "big module = slow" intuition does not hold here; what is actually expensive is the module-loading machinery itself. This is also the real reason the 0.79.0 refactor worked: it did not save "lines of code", it moved bin from impossible to cache to cacheable.

Where slow commands really spend their time: index loading (0.84.2 measurement + fix)

The section above explains the ~0.4s every command pays; it does not explain why search takes 12 seconds. Measured in 0.84.2 (same machine, best of 3–5):

Breakdown (rea alone): refresh 5.86s ≈ 2.4s parsing (wasted) + 2.4s parsing (useful)

• 0.44s index building + misc — while the actual matching takes only 0.36s (a keyword with zero hits costs the same as one with many, so the algorithm was never the bottleneck). The bottleneck is that every command slurps and parses the 40 MB of indexes (rea 23.6 + zef 12.6 + cpan 2.6 MB).

The fix: Ecosystem.!read-fetched-at used to slurp + from-json the entire index just to read one _rakupm_t timestamp — and !load-rows then parsed the same file all over again. It now scans only the first and last 4 KB (the timestamp always sits within the first or last few bytes: rea/cpan at offset 5, zef 24 bytes from the end — both key orders are in use, hence both ends), decoded as latin-1 (truncated multi-byte sequences cannot make it throw).

What still costs (recorded honestly, not done for now): of the remaining ~6.6s, about 4.2s is "parsing 40 MB of JSON" itself — a necessary cost, since %!by-nv / %!provides are built from full rows. The only way further down is to change the cache shape: have refresh also write a "query-sized slim index" (keeping just name / version / auth / description, i.e. what search and info need — an estimated 40 MB → a few MB). That costs a new cache format, invalidation logic and migration of old caches, and buys about 3.5s — evaluated and not done: complexity traded for time, which conflicts with the "bicycle, not motorbike" principle.

Another hotspot: verify's content check (0.84.3 → 0.86.0, three iterations)

Of verify's 19.3s, 17.6s goes into per-file MD5 (33 installed entries / 296 files / 2.1 MB). The way this got fixed is itself a "measure first" case study — each step overturned the previous step's premise:

1. 0.84.3: repaired a silently-dead fast path. There used to be a layer "files ≥64KB go to the system md5 command" (certutil -hashfile / md5sum / md5 -q). Measurement showed it had never worked on Chinese Windows: certutil prints GBK (MD5 \xb5\xc4 … 哈希:), $proc.out.slurp(:close) decodes as UTF-8, throws "Malformed UTF-8", and the surrounding try swallowed it → empty output → fall back to pure Raku. So every big file paid for a doomed process launch (0.25s) plus the slow pure-Raku hash (331KB measured at 2.07s) — slower than pure Raku alone. Reading bytes (:bin) and decoding as latin-1 fixed it: 331KB file 2.5s → 0.25s, verify 19.3s → 13.5s.

2. 0.85.0: hash less (manifest sidecar). With parallelism ruled out (below), the move was to hash only when content actually changed: each store entry's .files.json (a sidecar like .digest / build.json — not counted in the fingerprint, never copied into an install prefix) records every file's (size, mtime, md5); unchanged files reuse the old digest. verify: ~13s first time, 1.8s afterwards. Trade-off stated plainly: a modification that also restores size and mtime is not detected; for unconditional full hashing set RAKUPM_VERIFY_NOCACHE=1.

3. 0.86.0: change the implementation (native binding) — the real answer. Both earlier steps still argued inside the premise "pure-Raku MD5 runs at 0.16 MB/s". The root fix is to hand CPU-bound work to native code: NativeCall-bind the system libcrypto MD5(), measured at 0.246s for the same files (44× faster than pure Raku, with RFC vectors bit-identical to the pure-Raku path). verify with forced full hashing: 11.7s → 2.86s — by then the remaining cost is mostly ~1s of CLI/Client startup plus file IO, no longer the hashing. Library names are probed across platforms (libcrypto / libcrypto.so.3 / .so.1.1 / libcrypto.dylib …), validated with an RFC vector on first use, and fall back to pure Raku. The "system command" layer was then deleted — with a native binding it is pure overhead, while the platform special cases it carried (three output formats, per-OS command names) and the GBK-class trap were real cost. One path fewer = one class of platform-only bugs fewer.

Parallelism: measured useless, recorded so nobody retries it. With 296 files read into memory and only the MD5 computed, 8 threads buy just 1.18x (10.97s → 9.27s, results bit-identical to serial) — pure-Raku hashing does not truly parallelise on MoarVM. (Contrast: parsing the index JSONs in parallel does help (4.32s → 3.31s) because that path involves IO.) Multiple processes are no better: each raku subprocess costs ~1s to start, and 8 of them eat the entire gain. Conclusion: for CPU-bound work there are only two real ways out — a native implementation, or computing less (manifests / caches).

Evaluated and not done: splitting / lazifying the heavy RakuPM::Client graph

The idea: keep read-only commands (list / installed / which / env / lock / outdated / doctor) from loading Installer / Builder / Tester. Verdict: not done, for three reasons:

1. The ceiling is only ~80ms (the 0.231s → 0.292s gap), and it only matters for pure query commands; for install (seconds to minutes) it is noise at the 0.1% level.

2. Cost and payoff are asymmetric: Client's attributes carry compile-time type constraints (has RakuPM::Installer $.installer;), it news Installer / Builder / Store / Lock in TWEAK, and it is composed of 5 roles. Actually keeping read commands from loading those would require dropping the type constraints and switching to lazy accessors — i.e. giving up the compile-time guarantees of the most safety-critical class in the project (whole-tree transactional install).

3. It is a different kind of change from 0.79.0: that one had an architectural payoff (bin can never get precomp); this one has none — Client is already lazily required (CLI:513) and already precomp'd, so splitting it only breaks something already cheap into smaller pieces.

One suspicious item ruled out too: CLI's top-level use JSON::Fast (used exactly once, by raku-pm version to read META6.json) looks like it could be dropped, but RakuPM::Spec already does use JSON::Fast — the transitive import is unavoidable, and the measured marginal cost is just 0.020s.

What remains — raku start plus the first module load — is a fixed, unavoidable cost.

HTTP layer: single backend (curl)

All network traffic goes through the RakuPM::HTTP facade; a backend implements the RakuPM::HTTP::Backend role:

As of 0.82.0 there is a single backend. It used to ship a pure-Raku HTTP::Tiny fallback plus an auto-fallback order and a pluggable registry (RAKUPM_HTTP_BACKEND / register-backend); all of that is gone — the "pluggability" had little value for a single-machine teaching tool and a large maintenance surface, and post-form / post-json (upload / login) already shell out to curl directly, so collapsing onto one curl path is simply simpler. When a test needs a stand-in, inject one via the :backend constructor argument (see t/http-debug.t).

Two traps already hit in production; avoid them when writing / changing a backend:

• Pass request headers to .get(), not to .new(). HTTP::Tinyish spells the constructor option default-headers; .new(headers => …) is silently dropped. This once meant If-None-Match was never actually sent, so the ETag/304 conditional refresh did nothing and every TTL expiry re-downloaded 10–18 MB.

• The timeout option is timeout for Tinyish (not max-time). Getting it wrong means no timeout at all: a stalled connection hangs forever (identical on Windows and WSL — it is not a platform issue).

The install pipeline

resolve → fetch → build → store → test → install → lock

1. resolve: look up which distributions can provide the target module, pick the highest version matching the constraint, expand dependencies into a tree, then topologically sort into a dependency-first, bottom-up order.

2. fetch: local repos point straight at a directory; ecosystem repos build a URL from the index's path, HTTP-download the tarball and unpack it into cache/dist/<dist>/<version>/.

3. build: if the dist ships a Build.pm (convention: a class Build whose build method returns truthy) or its META6 declares a "builder" field (e.g. Distribution::Builder::MakeFromJSON), a raku subprocess runs the build from the dist root; a false result fails the install. build-depends (the builder class is usually a separate ecosystem package) are resolved and installed before the build; --no-build skips the stage entirely. Native packages typically compile their C sources into resources/libraries/*.so here.

4. store: the source directory is admitted (MD5 content digest); one download can be deployed repeatedly. Build-generated resources are copied into the store here, which is what makes %?RESOURCES work after install.

5. test: run t/*.t and t/*.rakutest against the source directory (-I lib -I inst#<target>/site, dependencies already installed). Any failing case aborts the install; --no-test skips it.

6. install: hand the distribution to CompUnit::Repository::Installation, which writes it into target/site/. It defaults to :precompile, so Rakudo manages precompilation — the early approach of shelling out to raku -M<mod> to precompile into a flat lib/ is gone (slower and it carried no version metadata).

7. lock: write exact versions + digests.

8. generation (atomic install): steps 3–7 above are wrapped in a single transaction — if any package fails, the target is restored to exactly what it was before the run; only if everything succeeds is a generation recorded, which you can return to with raku-pm rollback. See Atomic install and generation rollback below.

A git install runs the same pipeline. Only steps 1 and 2 differ for the target package: resolve treats the META6.json depends as a set of roots (see below), and fetch skips downloading in favour of the clone directory.

Atomic install and generation rollback

An install either takes effect completely or not at all — never a half-installed target:

raku-pm install A          # A depends on B → order [B, A]
  B installed into the CUR ✓
  A's tests fail ✗
  → rollback: B is removed too; the target is back to its previous state

raku-pm generations     # list all generations
raku-pm rollback        # go back one generation
raku-pm rollback 000003 # go back to a specific generation

Why generations are cheap: CompUnit::Repository::Installation supports multiple coexisting versions, so installing a new version never erases an older one. "Rolling back" therefore means removing the versions this run installed from the CUR and restoring installed.json to its pre-transaction snapshot — the older versions were never gone. A generation is thus a few KB of JSON, not a copy of a hundreds-of-MB site/.

Layout:

target/
├── site/                    the live CUR::Installation
├── installed.json           current ledger (a generation's manifest is its snapshot)
└── generations/
    ├── current              current generation id
    ├── 000001/
    │   ├── before.json      ledger snapshot before the transaction (rollback truth)
    │   ├── manifest.json    ledger after commit (what this generation looks like)
    │   └── meta.json        timestamp, what was installed
    └── 000002/…

The 10 most recent generations are kept; the current one is never pruned.

Limits (stated honestly):

• build / test run against the source directory and never touch the target, so they are already atomic. The transaction protects the "activate into the target CUR + write the ledger + place bins" part.

• Rolling back to a generation whose sources have since been removed from the store by raku-pm clean warns and skips (it cannot be reinstalled). Don't gc aggressively if you might need to roll back.

• When a same-version --force reinstall is rolled back, the copy in the CUR is removed; if the store still has that version it is reinstalled automatically, so you never end up with "the ledger says it's installed but it can't be loaded".

Reverse dependencies and reclamation

Dependencies are declared as module names ("depends": { "Foo::Bar" => "1.0" }) while the ledger keys on distribution names; provides bridges the two, giving a dependency graph. A reverse dependency is just its reversed edge.

raku-pm rdepends Foo          # who depends on Foo (a module name works too)
raku-pm uninstall Foo         # refuses while something still depends on it
raku-pm uninstall Foo --recursive   # remove its dependents too
raku-pm uninstall Foo --force       # force (leaves broken dependencies; your call)
raku-pm autoremove            # reclaim packages nothing depends on any more
raku-pm autoremove --dry      # list only, change nothing

After uninstall, packages that nothing depends on any more are reported, e.g. after removing Top:

以下 1 个包不再被任何已装包依赖（当初是作为依赖装进来的）：
  · Mid 1.0
  raku-pm autoremove        可以把它们一起清掉

Why it never deletes blindly: every ledger entry carries a reason —

Entries from older ledgers without reason are treated as explicit — better to reclaim nothing than to silently delete something the user installed. This is the same idea as apt's auto/manual marker: without it, "auto reclamation" is just "deleting things".

autoremove loops until it converges: removing Mid can turn Leaf (only depended on by Mid) into a new orphan, which is then handled too.

Verify: content integrity + install consistency

raku-pm verify              # both sections
raku-pm verify --content    # content integrity only (old behaviour)
raku-pm verify --fix        # also repair what can be repaired

1. Content integrity — has the source in the store been modified? Covers every version in versions[], not just the active one.

2. Install consistency — do the ledger / CUR / store / resources agree? This section was added later because section 1 alone gives false confidence: in the GDBM incident the CUR was already half-broken (site/dist/<id> gone) while the store copy was perfectly intact, so section 1 still reported "all fine".

Build traces: the build.json sidecar (since 0.44.0)

Every store entry now carries a build.json recording what the build phase actually did: mode (built / skipped-no-build / not-needed / not-recorded), success or failure, which raku was used (the compiler release, handy for diagnosing precomp/ABI mismatch), and the declared resources that were never produced.

This closes the honest limit stated below. Declaring only resources that exist on disk is a necessary trade-off in Store (declaring a phantom resource makes CUR::Installation fail), so "the build never succeeded → the resource was never produced" used to be silently erased from the store's META6.json; the metadata looked perfectly clean and nothing could be checked. build.json keeps the erased evidence verbatim, which is what finally makes verify able to report it.

It is deliberately excluded from the content digest (skipped alongside .digest in !fingerprint): it is a record of how the entry was produced, not distribution content, and it carries a timestamp — including it would make every recomputation drift and raise false "content was modified" alarms.

What remains undetectable: a resource that no META6 ever declared (upstream packaging never listed it). In that case there is no record of what should exist; we don't pretend otherwise.

Repositories: a configurable array

The Raku world has more than one ecosystem index, and their coverage differs a lot (measured 2026-09):

922 distributions exist only in REA and not in the zef index, so a single index cannot install everything — repositories have to be an array.

Config precedence (high → low):

1. $RAKUPM_TARGET/repositories.json (maintained by repos add/remove, persisted)

2. RAKUPM_ECOSYSTEM env var — comma/semicolon separated, each entry an URL or alias

3. default: the zef index + the local $RAKUPM_REPO directory

One easy trap: RAKUPM_REPO is an invocation-level setting (it's "which directory to look for local packages this time"), so it overrides the local path stored in repositories.json. RAKUPM_ECOSYSTEM by contrast is a fallback for persistent config and only applies when no config file exists. Otherwise RAKUPM_REPO=./vendor raku-pm install Foo would silently reuse the path stored last time, which looks like "I put the package there and it still can't find it".

raku -Ilib bin/raku-pm.raku repos                 # list configured repos
raku -Ilib bin/raku-pm.raku repos add rea          # add the ecosystem archive (alias)
raku -Ilib bin/raku-pm.raku repos add https://example.com/eco.json   # or a full URL
raku -Ilib bin/raku-pm.raku repos remove rea

# or point at several indexes without touching disk
RAKUPM_ECOSYSTEM='https://360.zef.pm,rea' raku -Ilib bin/raku-pm.raku install ADT

When one index cannot be fetched (since 0.49.5: degrade + warn, never abort the whole command): the resolver asks every repository, so a single unreachable index used to break the entire command — even when the answer was sitting in a reachable repository (a local directory repo, or the zef index). Both cases now continue:

Either way one ⚠ line names the index and the URL. So if a "module not found" appears right below a few index ⚠ lines, that means "it may exist, I just cannot see it right now" — fix the network/proxy and retry, or raku-pm repos remove the index you don't need. It never degrades silently: an extra warning line beats letting you believe the package really does not exist. (A common case in mainland China: cpan / rea on raw.githubusercontent.com are unreachable while the primary index 360.zef.pm works fine — you can still install anything that index has.)

Local directory repos (no index file needed)

One directory holding one subdirectory per distribution, each with META6.json + lib/:

~/my-dists/
├── HTTP-Client/{META6.json, lib/...}
└── Web-App/{META6.json, lib/...}

# a directory is detected as a local repo (name defaults to the basename,
# override with --name=; the absolute path is stored)
raku -Ilib bin/raku-pm.raku repos add ~/my-dists
# fill it: fetch sources (download only, no install), or drop/symlink dirs yourself
raku -Ilib bin/raku-pm.raku fetch HTTP::Client --to=~/my-dists
raku -Ilib bin/raku-pm.raku repos remove ~/my-dists     # by path or by name

Afterwards install / search / dependency resolution all consult this repo.

Two things worth keeping apart:

• A local repo is a source of sources, not installed packages — installed ones live in $RAKUPM_TARGET/site/;

• A single package you are developing does not need a repo: just raku-pm install ./my-pkg (its dependencies are installed automatically). A repo pays off for several local packages you want to install by name and have resolved as dependencies.

Install prefix

--target=<path> sets the install prefix (one prefix = one self-contained package environment). It overrides the RAKUPM_TARGET env var and defaults to ~/.raku-pm. It works before or after the command — both positions are equivalent:

raku -Ilib bin/raku-pm.raku install Foo --target=./vendor
raku -Ilib bin/raku-pm.raku --target=./vendor install Foo    # same thing

A leading ~ is expanded: --target=~/myenv means myenv in your home directory, not a literal directory named ~ under the current one. (Rakudo itself does not expand ~ inside an IO::Path; raku-pm does that at the entry point.)

The prefix is self-consistent: the wrapper in <prefix>/bin/ passes its own prefix to the CLI explicitly — whichever prefix's wrapper you invoke, that's the prefix you operate on. Same semantics as <venv>/bin/python.

Whether to write the global site/bin: after installing, raku-pm copies the wrapper into <rakudo>/share/perl6/site/bin (the directory zef uses, already on your PATH) so the command works immediately. That default applies only to the default prefix; a custom prefix never writes there — the global site/bin is shared machine-wide, so a local install writing into it pollutes the global environment, and multiple prefixes would overwrite each other:

With a custom prefix, the install prints exactly what you need to configure:

export RAKULIB="inst#<prefix>/site"     # module search path (the inst# prefix is required)
export PATH="<prefix>/bin:$PATH"        # command search path (put it first)

raku-pm env prints these two lines for the current prefix.

Why bin must come first on PATH: across directories it is PATH order that decides, and only within a single directory does PATHEXT get to compare extensions. So putting <prefix>/bin first beats a <name>.exe sitting in zef's directory — no need to touch other tools' files.

Recommended: write nothing outside the prefix (PATH mode)

The default above copies the wrapper into the global site/bin (zef-style). If you'd rather have raku-pm leave files only inside its own directory, switch the default prefix to PATH mode too:

# 1) put the prefix's own bin first on PATH (add to your shell profile)
export RAKULIB="inst#$HOME/.raku-pm/site"
export PATH="$HOME/.raku-pm/bin:$PATH"      # order matters: before rakudo's site/bin

# 2) from now on, never write to the global site/bin (self-upgrade honours it too)
raku-pm self-upgrade --no-promote-bin

# 3) optional: remove the three copies previously written to the global site/bin
#    (<rakudo>/share/perl6/site/bin/raku-pm, raku-pm.bat, raku-pm.ps1).
#    Only do this after step 1 works (`raku-pm env` shows [0] = <prefix>/bin),
#    otherwise you end up with no entry point at all.

Three upsides: (1) nothing lives outside the prefix, so upgrading or reinstalling rakudo cannot take it away; (2) you never modify files left behind by another package manager (the 0.48.0 "entry check" rename was only a remedy); (3) undoing it is just a PATH edit, with no leftovers. The cost: you configure those two lines once — raku-pm env prints them for the current prefix.

Multiple projects / versions side by side: one prefix is one isolated environment:

raku-pm --target=./projA/.raku-env install Foo    # project A's own environment
raku-pm --target=./projB/.raku-env install Foo    # project B's — versions may differ

A custom prefix is in PATH mode by default (it writes no global site/bin), and the wrapper in <prefix>/bin/ passes its own prefix to the CLI — whichever prefix's wrapper you invoke, that's the prefix you operate on.

⚠ A global RAKULIB is a trap when developing raku-pm itself

Exporting RAKULIB from your shell profile (as suggested above) is right for consuming packages — plain raku can then use Foo too. But note the side effect: plain raku will also load raku-pm's own copy of RakuPM from <prefix>/site. If that copy is old, running tests or scripts inside the repo makes you think you are testing the working tree when you are actually testing the stale copy. The symptom is that old bugs get reported as failures of the current code, sending you in the wrong direction entirely (0.49.3 cost a long debugging session for exactly this; it affects t/ and xt/ files that do not use lib).

Three habits that keep you safe:

• Run tests only via raku tools/run-suite.raku: it passes -I<worktree>/lib to every test and self-checks that the chain head really is the worktree, aborting if not (no false green). Besides running tests it carries three static gates: named-argument check, docs discipline (version consistency across three places / provides completeness / bin / Changes) and personal-trace check (nothing that ships in the package may contain the local user name or an absolute path — this class of leak recurs in new disguises, so it is pinned as a gate; --docs-only runs it in seconds);

• Run a single file as raku -Ilib t/<x>.t (-I always comes before RAKULIB);

• To see which copy plain raku would use: raku-pm which RakuPM::Version (it also lists the shadowed ones);

• raku-pm now self-checks on startup too: when multiple distinct RakuPM versions appear on the load chain it prints ⚠ multiple RakuPM versions detected on the load chain to stderr, warning that the loaded copy may not be the one you expect (since 0.54.1; a pure warning, zero-cost, does not change loading behavior).

To clean up stale RakuPM copies in the prefix (use the command, do not delete files by hand):

raku-pm installed RakuPM              # versions in the prefix; * = the one `use` picks
raku-pm self-upgrade                  # make the prefix current (highest version is preferred)
raku-pm uninstall RakuPM --version=0.36.14   # then shed the old ones you no longer want

Output:

已配置的仓库（共 2 个，按优先级从上到下）：
  [0] zef   远程索引  https://360.zef.pm
  [1] local 本地目录  .

Search tags each hit with the index it came from:

==> 拉取生态索引 [zef]：https://360.zef.pm
匹配到 1 个发行版：
  · [zef] NativeLibs 0.0.9  (zef:raku-community-modules)
    Native libraries utilities

Field differences between indexes

Index records are largely isomorphic (name / version / provides / depends), but two differences matter:

• Tarball URL: 360.zef.pm gives a relative path (needs the host prepended); REA and cpan give an absolute source-url. !tarball-url prefers source-url.

• auth: some REA records have no top-level auth; the identity sits inside the dist string (ADT:ver<0.5>:auth<github:timo>). !auth-of falls back to parsing that.

Both the index cache and the unpack directories are namespaced by repository name: rea and cpan both live on raw.githubusercontent.com (host-based cache names would clobber each other), and the same distribution+version may exist in several indexes (so unpack results are isolated per repository).

Local directory layout

$RAKUPM_TARGET/                   default ~/.raku-pm
├── cache/                        ecosystem indexes + downloaded tarballs & unpack dirs
│   ├── index-360.zef.pm.json
│   └── dist/<dist>/<version>/
├── store/                        home of "downloaded" packages
│   └── <dist>/<version>/         multiple versions coexist
│       ├── META.json             raku-pm's own normalised metadata
│       ├── META6.json            Raku standard metadata (the only one CUR reads)
│       ├── lib/                  sources
│       ├── resources/            resources (native .so/.dll build output lives here)
│       ├── build.json            build-trace sidecar (since 0.44.0; not in the digest)
│       └── .digest               content digest (MD5, tamper detection)
├── site/                         ★ module search path: CompUnit::Repository::Installation
│   ├── precomp/                  precompilation, managed by Rakudo
│   └── dist/                     per-distribution metadata (multiple versions coexist)
├── installed.json                raku-pm's own installed ledger
└── raku-pm.lock                  lockfile

Why two layers, store and site (à la cargo's registry/cache + src):

• store/ is the cache — one download, many deployments; reinstalling after an uninstall never touches the network again;

• site/ is the runtime, handed to Raku's own CompUnit::Repository::Installation (the same thing zef installs into), which resolves multiple versions from metadata.

Earlier versions used a flat lib/*.rakumod directory here. That was wrong — a flat directory carries no version metadata at all, so use Foo:ver<1.1.0> silently loads whatever is there. See the next section.

Path-safety invariant: sanitise foreign strings before they enter a path (0.87.0)

Every <dist>/<version> above is a path component. Both values come from the package's own META6.json / the ecosystem index row — i.e. they are controlled by the publisher, and are therefore untrusted input.

Before the fix, Store only did subst('::','-') (the old private helper was named !safe-name, but it only handled the module separator, not path-structure characters), and the $version on that same path was not handled at all. So "name": "../../../../../../tmp/PWNED" wrote an entire store entry outside the target — "publish one package, write files anywhere on the user's machine".

The invariant (do not route around it):

1. Anything that joins a foreign string into a path must first pass it through RakuPM::Fs.path-component — the project's single sanitiser. Internally it splits input into two families that are deliberately handled differently (the line was drawn explicitly in 0.87.4 — do not conflate them):

   • Traversal family (/, \, exactly . / .., control characters, trailing dots) → die outright. They either escape the target directory or make two distinct names land in the same place. Silently rewriting ../x to x would make unrelated distributions collide in one directory — worse than refusing.

   • Windows-reserved family (* ? " < > |) → percent-encode (* → %2A … with % itself encoded as %25 so the mapping stays injective). These cannot escape the target; they merely make the filename illegal on Windows. The old behaviour produced Invalid argument on Windows (retried 3×, then rolled back, with an error that revealed nothing) and a file literally named * elsewhere (which a shell will glob-expand). Measured on the live index, the affected entries are the 109 whose version is the * placeholder (? " < > | % occur 0 times) — encoding makes them installable on Windows too (verified end-to-end: use works afterwards). Why not "replace with _": that collides with the real _; injectivity is the one hard constraint here.

   • :: and a lone : map to - (the latter is the Windows ADS separator).

2. Before writing to disk, assert containment with RakuPM::Fs.assert-under. That is the second line of defence: entry points multiply as the code evolves, and any one that forgets the sanitiser turns into a loud error instead of a silent out-of-bounds write.

3. The seven sites that need it today: ① Store.path-for (name + version); ② Client/Tester test log dirs; ③ the Repository/Ecosystem download cache (index-row controlled, and it runs before the store); ④ the Installer temp filename; ⑤ Author archive names / publish destination; ⑥ Client/Git.!fetch-source's fetch --to output directory; ⑦ Client/Git.!git-cache-key's git cache directory name (added in 0.88.6 — the only site that joined a whole path verbatim: the address can come from the CLI, from self-upgrade --from, or from a malicious distribution's META6.json git-deps, the last one entirely invisible to the user). (There is one further redundant site, Author.new, whose module name is already regex-validated — kept so that relaxing that regex later cannot silently open a traversal.) A new path is a new sanitisation site.

   ★ assert-under is inherently blind to .. traversal (measured, documented in 0.88.6): it compares literal prefixes ($child.absolute starting with $root.absolute ~ sep), and a path containing .. literally is underneath. On top of that Raku's IO::Path does not collapse .. (.absolute and .cleanup both keep it verbatim; the kernel is what resolves it) — so such checks look perfectly fine. Conclusion: sanitisation must happen when the path is built; a later assertion cannot repair it. Item 2 above is only a backstop against someone bypassing the sanitiser. ⑦ is implemented as stack-style normalisation (.. pops one level, an out-of-range .. is dropped) rather than "die on sight of ..": .. is illegal in a remote URL but perfectly legitimate in a local path (self-upgrade --from ../repo), so a blanket die would kill valid usage.

Regression guard: t/path-safety.t — including an end-to-end assertion that the escape destinations the old implementation would have produced do not exist, plus a positive control (a legal name really does install) so the negative assertions cannot pass vacuously.

Separately: traversal via archive member names (a tar containing ../x) is not ours to defend — GNU tar and the BSD tar shipped with Windows both reject embedded .. and strip leading / and drive letters (verified empirically). That relies on external defaults, which is exactly why the sanitiser above covers the paths we build; the two are complementary. That premise is pinned by xt/tar-member-escape.t (mutation check: adding -P to the extraction command makes the file land outside the destination and the test goes red on the spot) — precisely because it is "an external tool's default behaviour" does someone need to watch it: adding -P so that some package installs would not be caught by any other test.

Version management

• Multiple versions coexist: store/ can hold both JSON/1.0.0 and JSON/2.0.0; after installing into site/, both are reachable via use and you pick with :ver<>.

• Pin a version: install JSON --version=1.0.0 (downgrades too).

• Upgrade: upgrade JSON moves to the highest version in the repositories. upgrade with no package name upgrades all installed packages (the zef upgrade / apt upgrade convention).

• Move to a specific version: upgrade JSON --version=1.0.0.

• Downgrade caveat (important): Raku's use JSON without a version constraint always loads the highest version — that is Rakudo's CUR::Installation behaviour and raku-pm cannot change it. So installing an older version does not switch use over by default; it just adds another version alongside, and the command tells you so. To make the old version actually take effect, either run upgrade JSON --version=1.0.0 --only (removes the higher ones first), or write use JSON:ver<1.0.0> in your code.

• Uninstall one version: uninstall JSON --version=1.0.0 drops just that version and keeps the rest; without --version it removes the whole package (all versions).

• Pre-release ordering (since 0.87.3): follows semver 2.0.0 §11 — 1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 (numeric identifiers sort below alphanumeric ones; numeric identifiers compare numerically rather than lexically; a release outranks any pre-release of the same core; +build is ignored). There is exactly one implementation of this order: RakuPM::Version.key (with !pre-key building the pre-release segment key), and cmp / sort-versions / pick-best all go through it. Do not go back to folding the pre-release into a boolean: then alpha/beta/rc1 all share one key, cmp reports Same, and pick-best returns an answer that depends on input order — no error, no crash, just silently wrong. Measured on the live ecosystem, Hey would degrade from the correct 1.0.0-beta.9 to 1.0.0-beta.2.

• Uninstall success is decided, not announced (since 0.87.3): uninstall goes by what was actually removed from the CUR (Installer.uninstall returns the list of removed versions). All four "nothing was removed" cases — package not installed, version doesn't exist, range matched nothing, refused because of reverse dependencies — now error out with exit code 1 instead of printing "uninstalled". Scripts judge by exit code, and a false success is worse than no message at all.

• No implicit self-targeting: uninstall never defaults to raku-pm itself — write uninstall RakuPM explicitly, so a missing argument can't delete your package manager.

• Managing raku-pm itself:

  • self-upgrade [--from=<git-url>] [--from-dir=<dir>] [--dry] — upgrade raku-pm itself. Unlike upgrade RakuPM, it does not depend on raku-pm being recorded in our own ledger (it's often zef-installed, outside our ledger), and it does not require RakuPM to be published to any ecosystem index (it isn't — looking it up in the repos would always fail). Source priority:

    1. --from <url>: pull source from the given git remote (overrides everything);

    2. if the running copy lives in a git checkout (dev/self-run) → git pull that repo + reinstall from local source;

    3. otherwise pull from the default upstream https://gitee.com/skyter10086/raku-pm.git. Requires Git on PATH. --dry resolves only, without actually replacing anything. Before upgrading it compares versions: if the local copy is already up to date (same version AND same git commit) it skips the download/install entirely; if the version matches but the git commit differs (upstream force-pushed / rewrote history, so content changed) it reinstalls anyway. --force skips all checks and reinstalls.

  • self-remove [--dry] — uninstall raku-pm itself, doing more than uninstall RakuPM: ① remove the distribution from CUR::Installation; ② delete store/RakuPM/ (per-version source snapshots, usually the bulk of the size); ③ delete the raku-pm / raku-pm.bat wrappers under target/bin/; ④ drop the RakuPM entries from the ledger and lockfile. --dry only reports what would be deleted.

  • Caveat: if zef also installed raku-pm, that copy is outside our management — after self-remove, the raku-pm command may still exist (running zef's copy). Clean it up with zef uninstall RakuPM.

• Coexistence ≠ conflict: with no version given, use gets the highest one; with :ver<>, Rakudo matches exactly. "Highest" here means real RakuPM::Version.cmp — 0.13.0 > 0.9.2. Sorting them as strings (the obvious .sort) puts "0.13.0" < "0.9.2" because '1' < '9', and a freshly-installed 0.9.2 ends up as the active version over a later-installed 0.13.0. installed-versions uses RakuPM::Version.cmp so the highest is always the highest.

Multiple versions and use Foo:ver<1.2.3>

Raku supports use Module:ver<1.2.3>, but having the files on disk is not enough — Raku has to know which version each copy is before it can match.

Why a flat lib/ is wrong (and how quietly it's wrong)

If installing just means copying *.rakumod into a flat target/lib/, that directory has no version metadata whatsoever. Raku's CompUnit::Repository::FileSystem derives module names from file paths (lib/Foo/Bar.rakumod → Foo::Bar) but cannot derive versions, so a :ver<> constraint is satisfied unconditionally:

target/lib actually holds 2.0.0
> use MultiVer:ver<1.1.0>
我是 2.0.0          ← no error, just the wrong version

This is the worst kind of failure: it doesn't error, it just gives you the wrong thing.

The right way: CompUnit::Repository::Installation

It's what zef installs into (site / home / vendor are all such repositories). It stores metadata per distribution, keeps multiple versions of the same module side by side, and use Foo:ver<1.1.0> is resolved by Rakudo itself — which errors out clearly when nothing matches. raku-pm now installs into $RAKUPM_TARGET/site/:

my $cur  = CompUnit::Repository::Installation.new(prefix => $target.add('site'));
my $dist = Distribution::Path.new($store-dir, :meta-file($store-dir.add('META6.json')));
$cur.install($dist, :force);       # :precompile defaults on; Rakudo handles precomp

And so:

$ raku-pm install Concurrent::Stack --version=1.1
$ raku-pm install Concurrent::Stack            # and 1.3

$ raku-pm installed
    Concurrent::Stack 1.1
  * Concurrent::Stack 1.3
（* = 不指定版本时 use 会拿到的那个）

$ raku -I"inst#$HOME/.raku-pm/site" -e 'use Concurrent::Stack:ver<1.1>; ...'
$ raku -I"inst#$HOME/.raku-pm/site" -e 'use Concurrent::Stack:ver<9.9>; ...'
===SORRY!=== Could not find Concurrent::Stack:ver<9.9> in: ...

Two details that matter

1. The inst# prefix is mandatory. Without it, Rakudo treats site/ as an ordinary file-system repository and all the version metadata goes unread. Let raku-pm env generate it for you:

$ raku-pm env
# bash / zsh / Git Bash：
export RAKULIB="inst#C:\Users\...\.raku-pm\site"
# PowerShell：
$env:RAKULIB = "inst#C:\Users\...\.raku-pm\site"

2. The store needs an extra standard META6.json. CUR::Installation reads nothing else, and its provides is module => relative path — one thing more than our own META.json carries. The wrinkle: RakuPM::Distribution.provides only kept the module names; the paths were dropped while parsing the ecosystem index. So at admission time we scan lib/ and reconstruct them (Foo::Bar → lib/Foo/Bar.rakumod; note .rakudoc is docs, not a module, so it is excluded), using forward slashes throughout (Windows backslashes make Distribution::Path build bogus paths). Modules declared in provides with no file behind them are skipped — including them would just make the install fail.

Three extensions count as module sources: .rakumod / .pm6 / .pm. That is not our own invention — Rakudo itself loads .pm as a Raku module (it prints a deprecated notice; the same works through an inst# repository), and .pm was the mainstream extension in the early Perl 6 era. This list is defined in exactly one place (@MODULE-EXTS in RakuPM::Distribution, read elsewhere via module-extensions) — do not inline it anywhere: it used to be written out twice (in provides-with-paths and !walk-source-files), both times missing .pm, so every distribution from the .pm era installed unusably.

3. Missing provides falls back to the disk (0.87.1). provides is an optional field in META6, and a fair number of ecosystem entries omit it. A full index scan on this machine (25117 rows, 146 without provides / 53 distributions) is quite telling:

So the entries missing provides are precisely the .pm-era distributions — both root causes (no declaration, .pm unrecognised) hit the same victims. Missing provides plus writing out an empty standard META6 gave you a ghost install:

$ raku-pm install Automata::Cellular  → ✓ done (rc=0)
$ raku-pm installed                   → it is listed
$ raku -e 'use Automata::Cellular'    → Could not find …
$ raku-pm verify                      → ✓ no inconsistencies   ← no clue at all

So admission now goes through Distribution.provides-from-disk: if there is a declaration, use it (keeping only entries whose files really exist); if the declaration is empty, or none of the declared modules exists on disk, reconstruct from lib/. lib/ is the distribution's own content and deserves more trust than a missing or corrupted metadata field. If reconstruction still yields nothing while lib/ does contain files, Store.put refuses admission (better to install nothing than to write an entry that use can never reach) — and that check runs before anything is written, so a refusal does not even create a directory.

Both directions now live in RakuPM::Distribution: provides-with-paths (module name → path), scan-provides (scan lib/ → module names) and provides-from-disk (the fallback above). They used to be a private Store method (and only the first direction existed). After the move, the consumer side (admission writing META6.json) and the producer side (refresh / check) share one implementation.

Finding ghosts already on the machine: check #6 of verify / doctor (issue kind ghost-provides) — "the store's standard META6.json declares no module at all while lib/ does contain module sources". Those cannot be repaired (--fix only back-fills the ledger); reinstalling is the fix (a reinstall goes through the fallback above). The regression guard is t/ghost-provides.t, including a real install plus an external raku process proving use works, and three mutations (drop the fallback / drop .pm / re-inline the extension list).

Coexisting with zef: two versions of the same module

zef and raku-pm install into separate places (zef into ~/.raku and rakudo's site/; raku-pm into $RAKUPM_TARGET/site/). Raku's multi-version design means they coexist naturally without overwriting each other — two versions of the same module can live side by side, picked precisely with :ver<>.

The repository chain order (measured):

entries in RAKULIB (inst#<target>/site ← raku-pm's packages live here)
  → ~/.raku                (zef home)
  → rakudo's site / vendor / core

So with RAKULIB exported (raku-pm env generates it), a script sees:

Without RAKULIB raku-pm's packages are invisible and everything goes through zef as before.

To diagnose, raku-pm version Foo lists the versions installed by both managers and marks the "active" one — what use Foo actually loads when no version is specified.

The direct tool: raku-pm which (0.48.0). "First hit on the chain wins" sounds simple, but your machine usually has more than two repositories — this one has three, measured:

24 distributions exist in more than one repository (and HTTP::Tiny, JSON::Fast, MIME::Base64 even differ in version). There's also a home-beats-site effect: without RAKULIB, use JSON::Fast resolves to home's 0.20.1, not site's 0.19.

So don't estimate — ask:

raku-pm which JSON::Fast     # where this module actually loads from, and which version; lists shadowed copies
raku-pm which                # cross-repo conflict overview (which packages live in several repos, at what versions)

which takes a module name (e.g. JSON::Fast), not a distribution name. Without an argument it lists all conflicts by repository-chain index — lower index wins.

One thing that's easy to conflate: package name collisions and command name collisions are different problems. The above is the former (one module in several repositories). The latter is having multiple raku-pm executables on PATH: Windows resolves via PATHEXT, where .EXE outranks .BAT, so a leftover <rakudo>/site/bin/raku-pm.exe from the zef era shadows the bash/.bat wrapper raku-pm writes — the symptom being "the upgrade succeeded but raku-pm version still reports the old version". Since 0.48.0 install and upgrade run an entry check: it computes which file would actually be executed and, if it isn't one raku-pm wrote, renames it to <name>.zef-old (reversible) and says so. raku-pm env also lists the candidates in system resolution order and marks the one that really runs.

Design note: why there is no separate "local package index". Three layers, each with a job:

1. installed.json (the ledger) — the local index of installed packages: name, all versions, provides, digest, and the source directory (self-installs and git clones record where they came from). Dependency resolution queries it (O(1)); installed / version display it;

2. site/ is itself a CUR::Installation — Rakudo maintains an incremental index there (short/ + dist/), and use resolution uses it directly; no scanning needed;

3. store/ — the source cache; any installed package's full source can be retrieved from it (reinstalls never re-download).

"Walk the directories every time" is neither necessary nor sufficient: scanning only tells you what's in a directory, while dependency resolution needs to know what is installed and whether versions satisfy — that is the ledger's job. There is additionally an explicit local: repository (a RAKUPM_REPO directory laid out as repo/<name>/META6.json) for the "a bunch of local packages as a repo" scenario, scanned on demand.

Ledger self-heal. site/ and installed.json are kept in sync by two triggers: raku-pm installed runs a reconcile pass over the chain on entry, and the install-chain skip branch adopts any pre-existing own-site dist that is missing from the ledger before saying "already installed". Both make self-installed packages show up in installed even when the original record write was missed (e.g. interrupted installs, ledger written by old code).

Lockfile

raku-pm.lock records exact versions + content digests:

{
  "version": 1,
  "entries": [
    { "name": "JSON", "version": "2.0.0", "auth": "demo:raku",
      "digest": "d47078e2...", "depends": [] }
  ]
}

• A normal install updates the lock automatically;

• --locked turns on strict mode: any mismatch aborts. Good for CI.

Committable lockfile (0.54.0)

The lockfile is written to raku-pm.lock in the current working directory by default (Cargo.lock-style), so it can be committed alongside the project for reproducible installs ("same versions on a different machine or a later date"):

cd my-project
raku-pm install JSON            # pins exact versions into ./raku-pm.lock
git add -f raku-pm.lock         # commit it (-f: .gitignore ignores raku-pm.lock by default)
# teammate, after cloning:
raku-pm install --locked        # strictly reuses locked versions; missing/drifted lock aborts

• The lock describes the project's dependency closure and is independent of the install location (--target): even if packages go into a separate prefix like .raku-env, the lock stays in the project root — you commit the lock, not the prefix.

• --lock-file=<path> relocates the lock anywhere (CI often pins it to a fixed repo path). install / reinstall / self-upgrade / lock all accept it.

• raku-pm lock shows the current lock contents.

• Note: .gitignore ignores raku-pm.lock by default (to avoid dev leftovers); use git add -f, or add !raku-pm.lock to your project's .gitignore to commit it.

Concurrency: a cross-process file lock

One install is a long chain of writes — read raku-pm.lock, resolve, write to the store, admit into site, write the lockfile back. Two raku-pm processes running at once step on each other:

• both writing the same version into the store → half a package, digest mismatch, verify fails forever after;

• both writing raku-pm.lock → the later write wipes the earlier one wholesale;

• both promoting bin wrappers → a wrapper pointing at a path that isn't finished yet.

So every write command (install / uninstall / upgrade / fetch / self-upgrade / repos add|remove|update) takes a cross-process file lock first, and any other raku-pm queues up:

$ raku-pm install Foo          # while another terminal installs something else
⏳ 另一个 raku-pm 正在写入 /home/user/.raku-pm，等待它结束…

Details (all in RakuPM::FileLock):

• flock, not "lock file exists": when a process is killed, Ctrl+C'd, or the machine restarts, the OS reclaims the lock — no stale lock to clean by hand. The lock file is target/.raku-pm.run.lock; it holds no state. Who/what/when lives in a sidecar .owner file instead, because Windows' LockFileEx is a mandatory lock — while it's held, even reading the locked file is refused.

• Timeout, not an infinite wait: 600s by default (RAKUPM_LOCK_TIMEOUT); on timeout it fails loudly and says who is holding the lock.

• Re-entrant within one process: install calls install/upgrade internally, so a reference count is kept — only the outermost release really unlocks, otherwise a nested release would hand the lock away too early.

• A "token" is handed to child processes: Build.pm calling raku-pm install X is common, and such children must not wait on the lock their own parent holds (self-deadlock). So on acquiring the lock we mint a random token (written to a .token sidecar next to the lockfile, plus the RAKUPM_LOCK_TOKEN env var); a child is let through only if both the on-disk token matches and the lock is currently held. The token dies with the lock — once the parent releases, a late-arriving child queues like everyone else, and a raku-pm started by hand in another terminal has no token at all. (Before 0.44.0 this was "broadcast RAKUPM_NO_LOCK=1 while holding" — i.e. open the door for everyone: any process inheriting that variable got a free pass, including test suites run under the lock, whose assertions then held trivially.)

• Escape hatch: --no-lock or RAKUPM_NO_LOCK=1 skips the lock entirely — concurrent writes corrupt the store and lockfile, so only use it when you're sure nothing else is running. This is an explicit user bypass, distinct from the child-process token above: it validates nothing, it just takes effect.

One more from the same family: the ecosystem index cache (~10MB) used to be written with $cache.spurt($text), so two refreshes at once could interleave into a half JSON file — and the victim is the next command, long after the evidence is gone. It now writes a temp file and renames it into place (atomic within a filesystem), so readers only ever see the old file or the new one.

Test execution: serial (as of 0.80.0; 0.38.0–0.79.x ran concurrently)

When installing, the t/ tests run serially, file by file: each test file in its own raku subprocess (process-isolated, but not sandboxed), one after another. The zef ecosystem itself runs tests serially, so this is the form closest to the ecosystem default.

$ raku-pm install Foo
  · Running tests: Foo 0.1.0 (3 files, serial)
    ✓ 01-basic.rakutest
    ✗ 02-race.rakutest
    | Failed test 'concurrent push' at …/02-race.rakutest line 12
    ↻ Re-verifying 1 failed case serially once (flaky upstream / transient environment issues)
    ✓ 02-race.rakutest (passed re-verify — the first-run failure was a false negative, ignored; the first-run log is kept at 02-race.rakutest.log)

Why we went back from concurrent to serial (0.80.0): running tests concurrently is fine in itself; the problem is that keeping it from harming installs required two extra safety nets — ① a serial precomp warm-up (removing the rakudo precomp-cache race caused by several subprocesses first-compiling the same lib/*.rakumod at once, whose symptom is a bogus ===SORRY!===); ② a serial re-verify of failures. That complexity does not pay off for a single-machine teaching tool, and self-upgrade had already forced serial on its own for this reason, so 0.80.0 removed the whole concurrency mechanism (and the --test-jobs flag with it), keeping only the "serial re-verify of failures" — it guards against upstream flakiness, unrelated to concurrency.

Key points (locked-in defaults as of 0.38.3):

• ① Serial execution: file by file, process-isolated. No parallelism knob any more.

• ② Log location defaults to the install target: full output goes to <target>/log/<dist-name>/<version>/<test-file>.log (<target> defaults to ~/.raku-pm); reinstall overwrites, no retention (naturally isolated per dist/version/file). Set RAKUPM_TEST_LOG_DIR=<dir> to relocate the log root to a global spot (e.g. ~/.raku-pm) for cross-target inspection.

• ③ Only the first failure line prints to the terminal — with one exception: nested compile errors, where the innermost cause line(s) after the ===SORRY!=== wrappers are printed (up to 3 lines; see RakuPM::Client::Tester.!failure-lines). Full output stays in the log file above, keeping the terminal clean for later inspection.

• ④ Timeout defaults to 300s: --test-timeout=N or RAKUPM_TEST_TIMEOUT caps each test file; on timeout the subprocess is killed and the file counts as failed (still goes through the serial re-verify below).

• ⑤ Live progress (terminal only): while testing, a single line is redrawn in place on stderr — test <file-i>/<file-n> <basename> <cases-done>/<cases-total> cases pct%. Tests run serially (one file at a time), so progress advances by the current file's cases (the numbers keep refreshing; with many cases in one file you can still see it move); before the plan appears, only the done count is shown. Compiling and running each file can go minutes without other output, so silence looks like a hang. Non-TTY (pipe / redirect / test capture) stays silent, leaving stdout and the log files untouched; RAKUPM_TEST_PROGRESS=0 forces it off, =1 forces it on.

• Failures are auto re-verified serially once: upstream flaky cases (e.g. stress.rakutest) fail intermittently; a re-verify pass means it's a false negative and won't abort the install on a one-off probability. The re-verify log goes to <name>.flake.log, without overwriting the first-run failure log.

• The known-platform-bug soft-pass (Log::Async's t/12-context.rakutest on Windows, etc.) and missing-compiler-tool hint are unchanged and still apply.

Test phase vs. install success: the causal chain (0.79.1 / revised 0.80.0)

When a user follows a failure hint to log/ and finds the directory emptied by the rollback (which is what happened before 0.78.1), this causal chain is exactly what they need.

• Where the boundary is: as of 0.80.0 the test phase is serial too; resolve / download / build / store / stage / promote are fully serial (Client.!install-chain-inner is a plain for @order -> $dist, bottom-up). The only cross-process concurrency is "two raku-pm runs at once", serialized by FileLock.

• What the serial re-verify fixes: upstream flaky cases (Concurrent::Stack 1.3's stress case was measured failing 3 runs out of 10), giving the user one retry. The re-verify log goes to <name>.flake.log, without overwriting the first-run log.

• What it cannot fix (residual risks, recorded honestly):

  • re-verify also fails → judged a real failure → whole chain rolls back (dependencies already built in this run are discarded too);

  • an upstream flaky case that fails both times still fails (re-verify only gives one retry);

  • a single slow test file: serially, slow cases add up wall-clock time → may hit --test-timeout.

• Knobs: --test-timeout=S / --allow-test-failure / --no-test; env var RAKUPM_TEST_TIMEOUT. Full user-facing description is in MANUAL.en.md §15.

Cache cleanup: raku-pm clean

The store only ever grows — every self-upgrade leaves a snapshot, upgrades leave old versions behind, uninstalled packages leave their store entries, and downloaded/unpacked tarballs pile up in cache/dist/. Measured on a machine that had been in use for a while: store 193M (a single package, RakuPM, held 30 versions), cache/dist 110M — while only the newest copy was actually installed.

raku-pm clean                 # dry run: list what could be reclaimed (deletes nothing)
raku-pm clean --yes           # actually delete
raku-pm clean --all --yes     # also reclaim index cache and git clone cache
raku-pm clean --older-than=30 # only entries untouched for 30 days
raku-pm clean --name=RakuPM   # only look at one distribution

Four categories are reclaimed by default:

--all adds two more (rebuilding them costs a download/clone, so they are kept by default): the cache/index-*.json index caches and the git-cache/ clones.

The safety boundary is the only thing that matters here: only versions that are referenced nowhere are deleted. Three sources are protected —

1. versions recorded in the installed.json ledger (all entries of versions);

2. versions actually installed in our own site/ (CUR::Installation) — the ledger may have gaps, disk wins;

3. versions pinned by raku-pm.lock — a --locked install would reuse them.

Nothing inside site/ is touched (it is a separate copy from the store); the worst outcome of deleting a store entry is a re-download on the next install. One more guardrail: only paths inside $RAKUPM_TARGET are ever deleted.

Deleting files is irreversible, so the default is a dry run. Deletions run item by item; one busy entry (common on Windows) does not abort the round — failures are summarized at the end.

Starting over: raku-pm flush

clean is protective reclamation (every rule says what must NOT be deleted). To nuke and reinstall, use flush: it wipes everything raku-pm installed in the current prefix — including its own entry points.

raku-pm flush          # dry run: lists what would go (deletes nothing by default)
raku-pm flush --yes    # actually delete

Removed: inside the prefix, store/ site/ git-cache/ cache/ log/ generations/ stage/ bin/ installed.json raku-pm.lock; plus the self-installed entry points outside the prefix (raku-pm / .bat / .ps1 under <rakudo>/share/perl6/site/bin).

Two guardrails (for a delete-everything command, the guardrails matter more than the feature):

1. The directory must look like a raku-pm prefix (store/, site/ or installed.json present) and must not be the home directory itself nor a filesystem root — otherwise it refuses to run. This is what catches a mistyped --target / RAKUPM_TARGET.

2. Only recognised entry names are deleted; everything else in the prefix is left alone (and listed for you). So a prefix sharing a directory with other files cannot cause collateral damage.

It never touches other tools: a zef-installed RakuPM does not live in this prefix (clean it with zef uninstall RakuPM), and <name>.exe.zef-old (the only backup of zef's entry point) is only reported, never deleted.

⚠ The entry points go too, so after a flush there is no raku-pm command left. To bring it back:

zef install <path-to-raku-pm-repo>                 # zef installs it and writes the entry
raku <repo>/bin/raku-pm.raku install .             # or use the repo script; it writes the entry itself

How the three related commands divide up: clean = what can be safely reclaimed; self-remove = remove raku-pm itself only; flush = wipe everything, start over. They are deliberately kept separate rather than merged — so nobody fires off flush thinking it is an upgraded clean.

Mapping onto zef

Core design ideas

1. A distribution is a metadata file

Every package is a directory with a META6.json at its root:

{
  "name": "HTTP-Client",
  "version": "1.2.0",
  "auth": "demo:raku",
  "provides": ["HTTP::Client"],
  "depends": { "JSON": ">= 1.0.0" }
}

The key insight: provides (which modules it offers) and name (the distribution's name) are two different concepts. One distribution HTTP-Client may provide several modules, and module names use ::, not -.

Malformed external data: sort it into three classes by "can it be repaired losslessly?" (the line was drawn explicitly in 0.88.1)

Both META6.json and ecosystem index rows are external data (written by the publisher / generated by a remote index), so malformed shapes are normal. The rule is neither "be lenient about everything" nor "reject everything" — ask whether the malformation can be repaired without losing information:

Two rules: ① the index path and the META path must share one normalisation implementation — measured, the array form of auth was handled only on the META path, so on the index path Text::Sift4 (cpan index) died with a raw Type check failed for return value; ② error messages must give an actionable fix, but must not echo a "corrected literal" (Rat normalises 1.0 to 1, so copying it would change the version to the wrong value).

2. Repositories are a pluggable role

RakuPM::Repository is a role; implement these and you have a new package source:

method find-providers(Str $module --> List)     # who can provide this module
method all-distributions(--> List)              # every package
method available-versions(Str $name --> List)   # every version of a package
method fetch-distribution(Str $name, Str $version)  # materialise the metadata
method source-dir-for(Str $name, Str $version)  # source dir (local repos only)

That's a simplified take on zef's "plug in a new service backend by editing config.json".

Note source-dir-for: the client must not guess a repository's internal layout. Directory names and distribution names can differ (directory JSON2/ may hold distribution JSON 2.0.0), so locating sources has to stay inside the repository.

3. Resolution is recursion + topological sort

• Recursive resolution: DFS from the target module over all dependencies.

• Conflict detection: when the same module is required with different constraints, check compatibility.

• Cycle handling: mutually dependent distributions really do exist in the ecosystem data, so a cycle is no longer fatal. We note it, warn, and append whatever the topological sort can't order at the end (never silently dropping packages).

• Topological sort (Kahn's algorithm): dependencies are installed before their dependents.

• Constraint semantics: >=/>/<=/< are ranges; = and == are exact matches; comma-joined constraints are AND; * or empty means any.

• How to write them in a request string (same for install / search / installed / store): Foo:ver<>=1.0> (≥), Foo:ver<>1.0> (>), Foo:ver<<=1.0> (≤), Foo:ver<<1.0> (<), Foo:ver<>=1.0, <2.0> (range). >= also has the equivalent suffix form Foo:ver<1.0+> (at-least). Note the trailing > is the phaser's closing bracket, not part of the constraint itself.

4. Installing a scripting language = handing sources to Raku's repo machinery

For Raku/Python/Perl, "installing" is fundamentally about putting sources on the module search path. But the version-aware step can't be skipped: raku-pm takes a version out of store/, hands it to CompUnit::Repository::Installation, which writes it into site/; installed.json records which versions were installed (see "Multiple versions and use Foo:ver<1.2.3>").

5. Content digests (borrowing nix's content addressing)

Every package admitted to the store gets an MD5 digest over META.json + META6.json + every file under lib/:

• content changes → digest changes → tampering is detectable;

• verify checks in bulk, and install verifies first, refusing to proceed otherwise.

Same lineage as nix's /nix/store/<hash>-name, simplified to verification rather than addressing.