NAME

LLM::Agent::Test::Kit - scripted backends, scripted tools, and settled-run helpers

SYNOPSIS

use lib 'lib';
use LLM::Agent::Test::Kit;

my $backend = scripted-backend(steps => [
    # round 1: the model asks for a tool
    { chunks => ['Let me look. '], tool-calls => [tool-call('c1', 'fs_read', '{"path":"a"}')] },
    # round 2: it answers, and the provider says what it was billed
    { chunks => ['The file says hello.'], usage => { prompt => 812, completion => 9 } },
]);

my $provider = ScriptedProvider.new(
    tools   => [tool-decl('fs_read', 'Read a file', { path => { type => 'string' } })],
    results => { fs_read => 'hello' },
);

my $run = $loop.run([msg('user', 'what is in a?')]);
run-settled($run, 'the scripted run');
my @events = drain-events($run);

is $backend.recorded-calls.elems, 2, 'two round trips';
is $provider.recorded-calls.elems, 1, 'one tool call';

DESCRIPTION

Everything the agent tests need in order to drive a loop without a network, a model, or a clock they do not control.

It is deliberately not LLM::Chat::Backend::Mock: that mock exists to prove LLM::Chat's own behaviour and widening it to script mid-stream failures, stalls and per-attempt usage would be changing one dist's test surface to suit another's. This kit is free to be as strict and as purpose-built as the agent tests need.

This is test support, not library surface. It ships inside the dist's own namespace (rather than under t/lib) deliberately, so that zef install's test phase precompiles it as part of lib/ staging before any test file runs. A t/lib CompUnit repo starts cold on a fresh zef install, and parallel test files pulling the fixture's dependency chain into that cold store can race and fail the install outright; moving test support into lib/ under a ::Test namespace is the fleet-wide fix for that failure class.

ScriptedBackend

A real LLM::Chat::Backend subclass driven by a list of steps: one step is consumed per call, in order, by both chat-completion-stream and chat-completion. Every step key is optional.

cached is a base-Response field, set through the same _set-usage call as prompt/completion/total — it does not select the OpenRouter subclass on its own. The last three usage keys are the provider-specific ones, and a step that scripts any of them gets an LLM::Chat::Backend::Response::OpenRouter (or its ::Stream) instead of a plain Response — the real subclass, set through the real _set-or-usage, because what the loop does with them is probe with .?cost. A step that scripts none of them gets exactly the Response every other test here was written against, so both sides of that probe are covered by ordinary use.

model is what model-of($backend) answers, and it is the key a LLM::Agent::RequestBudget profile and a LLM::Agent::TokenCount::Usage calibration are both stored under — so a test that wants two backends with different profiles has to give them different models. It defaults to scripted-model, which means two default backends share a profile, which is occasionally the thing being tested and occasionally a surprise.

gate is how a test holds a call open for exactly as long as it needs to, with no clock involved: a child agent whose only step is gated is still running when the test looks at it, however loaded the machine is, and finishes the moment the test keeps the Promise.

my $hold = Promise.new;
my $open = $hold.vow;
my $backend = scripted-backend(steps => [ { content => 'done', gate => $hold }, ]);
# ... the call is in flight; look at whatever has to be looked at ...
$open.keep(True);

stall is what an inactivity timeout is tested against: the response never reaches a terminal state on its own, last-activity-at stops advancing, and the loop is expected to notice, call cancel (which the backend records) and move on. It is streaming-only; a blocking call on a stalled step throws rather than hanging the suite forever.

One step, one trailing comma

steps = [{ chunks => ['hi'] }]> is not a one-step script: Raku's single-argument rule flattens a lone Hash in a list into its Pairs, so that reads as three steps of one key each and the failure surfaces somewhere else entirely. Write steps = [{ chunks => ['hi'] },]> — the trailing comma is the whole fix. The same applies to a step's tool-calls and to a provider's tools.

The kit checks for it rather than letting it happen: a list of Pairs where hashes were expected dies immediately, with that advice.

Scripting the failures a provider reports through the finish reason

finish-reason and usage are applied on the failure path as well as the success one, because some failures are not transport failures at all:

# The provider took the prompt, cut the answer off, and said so. This is
# what a real streaming backend produces for `finish_reason: 'length'` —
# fragments, then a quit whose Response still carries the reason and the
# tokens it billed.
{ chunks => ['I will start by ', 'reading the'],
  'finish-reason' => 'length',
  error => 'Hit max tokens', 'error-class' => 'response',
  usage => { prompt => 198_000, completion => 590 } }

The order the kit applies them in is the order LLM::Chat's own backends use — calls, then reason, then usage, then the error info, then the quit — so a consumer reading the settled Response sees a truncated completion rather than an anonymous stream failure. It works with fail-after-chunks too, which is the more faithful shape when the point of the test is that the fragments really were emitted before the cut.

tool-calls is scriptable on the failure path for the round a provider that is honest about a cut still produces: fragments, a partial argument list, finish_reason: 'length', quit.

# A truthful length that had already assembled a call. The loop is
# expected to read the LENGTH, not the call — nothing is dispatched.
{ chunks => ['Spawning: '],
  tool-calls => [tool-call('c1', 'task', '{"prompt":"do the'),],
  'finish-reason' => 'length',
  error => 'Hit max tokens', 'error-class' => 'response' }

Unlike the success path, a failing step's tool-calls do not imply a finish reason: the quit is the terminal there, and what the provider said about it is whatever the step scripted.

Running out of steps is an error, not an empty response: the backend answers with an error-class of 'response' saying which call it was. A test whose loop went round more times than it scripted should say so rather than quietly pass.

Everything is recorded: .recorded-calls (messages, tools, the step used, and whether it was streaming), and .cancelled-responses (the Responses cancel was called on, in order).

Each recorded call also carries settings: what the request's own settings said about its shape — json_schema (a detached copy, or undefined), json_schema_name and max_tokens, each undefined when the backend reports no settings (a subclass whose settings returns Nil, scripting a backend that declares no cap). A loop with tool-routing => 'schema' sends every routing request through a clone of the backend carrying the round's schema and cap, and the clone shares the script, the record and the lock with the backend the test built, so the test reads the whole story off the one object it holds:

my $backend = scripted-backend(steps => [
    { content => '{"tool":"read_note","arguments":{"id":"a"}}' },
    { content => '{"tool":"none"}' },
]);
# ...run a routed loop over it...
my %first = $backend.recorded-calls[0];
%first<tools>;                   # ()  — no tools on a routed request
%first<settings><max_tokens>;    # 512 — the routing cap
%first<settings><json_schema>;   # { type => 'object', anyOf => [...] }

Response ids (scripted-stream-N, scripted-blocking-N) number the calls in that shared record, so they stay unique across clones.

ScriptedProvider

The duck-typed tool provider — tools-for-llm and execute-tool-calls, the same pair MCP::Client, MCP::Client::Registry and MCP::Client::Policy present, so a scripted provider can be dropped under a real Policy in an integration test.

Like the real ones, execute-tool-calls never throws: a malformed call, an unknown tool and a responder that died all come back as is_error results, one per call, in the caller's order.

block-on is a Promise the batch waits on before returning — a blocked tool round (a permission prompt nobody has answered yet) without needing a real one.

Two more knobs are per call rather than per batch, which is what makes a per-operation loop testable at all: with one call dispatched at a time, "gate the first of three" is a thing a test needs to be able to say.

Both are hashes looked up by call id first, then tool name, so a test can slow down one specific call ( latency => { c2 => 0.4 } ) or every call of a tool ( latency => { fs_read => 0.4 } ). block-call is awaited inside the per-call answer, so a batch of three with only the first gated returns nothing until the gate opens — which is exactly what a sequential dispatcher does anyway.

latency is how a tool deadline is tested without a clock seam: a deadline of 0.15s against a call that takes 0.5s fires, deterministically, in well under a second.

grants is deliberately absent from ScriptedProvider and present on ScriptedProviderWithGrants, because the loop decides whether to synchronise grants with .can('grants'). A single class with an always-present method could not test both sides of that branch.

test-session / reopen-session / close-test-sessions

LLM::Agent::Session holds an open append handle for its whole life (JSONL::Writer in handle mode with :flush — see its Pod), and .close is the only thing that lets go of it. A test that creates one and never closes it is therefore holding a file open when its END block unlinks the temp directory, and on Windows that unlink fails with "resource busy or locked": POSIX hides the bug by allowing the unlink of an open file, so it only ever showed up on one platform's CI.

So sessions in tests go through here:

my $session = test-session(path => $path, meta => {});    # .create
my $again   = reopen-session(path => $path);              # .load

END {
    close-test-sessions();          # BEFORE the unlink, always
    if $tmp.e { .unlink for $tmp.dir; $tmp.rmdir }
}

Both track the session they hand back, and close-test-sessions closes every one of them — on the success path, the failure path and the cancel path alike, because an END block runs whatever happened. Session.close is idempotent and never throws, so a test that closes its own session (several here do, because reopening one is the thing being tested) is unaffected; the sweep just finds it already shut. It answers how many it closed, and clears the register, so it is safe to call more than once.

Nothing here swallows the unlink error: the point is that the handles really are closed by then.

run-settled / drained-settled / drain-events

run-settled polls is-done rather than awaiting the result Promise, and drain-events taps the event Supply exactly once. Both follow the lesson recorded in LLM-Chat's ChatTestKit: a scripted run can finish before the test gets to it, and a fresh tap on an already-finished stream never receives the done it is waiting for. So: poll for completion, tap once, and give up loudly after TIMEOUT seconds instead of hanging the suite.

drained-settled is the same shape for $run.drained: it waits for the run to stop producing, which on a cancelled run is later than run-settled — a detached tool batch keeps a run undrained long after its result was kept. Use it when the assertion afterwards is about what did not happen (no stray event, no cancelled response), because "nothing else arrived" is only worth asserting once nothing else can.