NAME

LLM::Chat::Backend::OpenAICommon - Generic OpenAI-compatible chat backend

SYNOPSIS

use LLM::Chat::Backend::OpenAICommon;
use LLM::Chat::Backend::Settings;

# Any OpenAI-compatible endpoint (Together, Groq, Fireworks, vLLM,
# llama.cpp's OAI server, ...). For OpenRouter specifically, prefer
# the L<LLM::Chat::Backend::OpenRouter> subclass — it adds the
# attribution headers, body extras, and cost/generation-id lifts.
my $backend = LLM::Chat::Backend::OpenAICommon.new(
    api_url  => 'https://api.together.xyz/v1',
    api_key  => %*ENV<TOGETHER_API_KEY>,
    model    => 'meta-llama/Llama-3.3-70B-Instruct-Turbo',
    settings => LLM::Chat::Backend::Settings.new(:max_tokens(4096)),
);

my $resp = $backend.chat-completion(@messages);
react {
    whenever $resp.supply -> $tok { print $tok }
    whenever $resp.supply.done    { say "done — {$resp.prompt-tokens} in / {$resp.completion-tokens} out" }
}

DESCRIPTION

OpenAI-compatible HTTP client. Uses Cro::HTTP::Client against /chat/completions and /completions endpoints, lifts the OAI-spec usage block onto the returned Response, and classifies failures into the categorical error-class shape the LLM::Data::Inference fallback policy expects.

EXTENSION

Provider-specific subclasses extend this class to add wire fields, headers, or Response shapes that aren't part of the OAI spec. To keep that path clean, the following internal methods are underscore-prefixed (rather than !-private) so subclasses can override them:

• _get-api-settings — body params for every request.

• _get-api-headers — HTTP headers for every request.

• wire-message-hashes — pure chat-message serialization, including the opt-in system-message compatibility mode and /MESSAGE NORMALIZATION.

• normalized-messages — the Message list the text paths hand their template, normalized the same way.

• _finalize-request-body — last look at the assembled body, after messages / prompt / tools / stream are attached and before it goes on the wire.

• _encode-request-body — turns the finished body into the exact bytes posted. See /REQUEST BODY SERIALIZATION.

• _lift-usage — extract usage / model from a response body or stream chunk into the Response.

• _reconcile-finish-reason — decide which finish reason a completion really ended with, given the provider's. See /Correcting a provider's finish reason.

• make-response — factory for non-streaming Response objects (used by chat-completion / text-completion).

• make-stream-response — factory for streaming Response objects.

• _on-blocking-complete — fires after a non-streaming completion's body has been parsed and lifted, before $response.done. Subclasses use it to attach post-call metadata (e.g. OpenRouter's /generation cost lookup).

• _on-stream-complete — same hook for the streaming path; fires after [DONE], before $response.done.

• probe-server-limits — ask the server for its context size and slot count. See /REASONING, SLOTS AND SERVER LIMITS.

LLM::Chat::Backend::OpenRouter overrides the settings, headers, finalization, lift, factory and completion hooks; it inherits wire-message-hashes and _encode-request-body unchanged. LLM::Chat::Backend::KoboldCpp overrides _reconcile-finish-reason and probe-server-limits.

_blob-text, _body-text, _decode-json-body and _stream-decoder are also underscore-prefixed, but as shared internals rather than override points: they exist so subclasses making their own HTTP calls (such as OpenRouter's /generation lookup) read response bodies the same way the completion paths do. See their declarations for why LLM::Chat decodes response bytes itself instead of using Cro's body-text / await .body, and why a streamed body needs an incremental decoder rather than a .decode per chunk.

_server-root (the api_url without its /v1), _probe-json and _probe-json-array (one bounded, never-throwing GET of a JSON object or array there) are shared internals of the same kind, for a subclass's own server probe.

!classify-exception stays private — it's pure Raku/Cro mapping with no provider-specific behaviour.

REASONING, SLOTS AND SERVER LIMITS

Three per-request settings (LLM::Chat::Backend::Settings) go on the wire only when they are set, so a server that knows none of them sees the body it always saw:

• reasoning_effort — the top-level OpenAI field. llama.cpp's server gives it to the chat template ('none' turns thinking off); a Mistral Small 4 template takes 'none' and 'high'.

• reasoning_budget_tokens — llama.cpp's per-request thinking budget (it overrides the server's --reasoning-budget; the server forces the end-of-thinking tag once the budget is spent).

• id_slot — the llama.cpp slot the request runs on.

my $backend = LLM::Chat::Backend::OpenAICommon.new(
    api_url  => 'http://127.0.0.1:8080/v1',
    settings => LLM::Chat::Backend::Settings.new(
        max_tokens              => 1024 + 2048,  # the answer's room plus the thinking's
        reasoning_effort        => 'high',
        reasoning_budget_tokens => 2048,
        id_slot                 => 1,
    ),
);
my $resp = $backend.chat-completion(@messages);
# ... once it is done:
$resp.msg;              # the answer only
$resp.reasoning-text;   # the thinking, never sent back by this class
$resp.completion-tokens;  # includes the thinking tokens

The thinking arrives as reasoning_content (reasoning on OpenRouter), on the final message or in stream deltas, and is kept on the Response's reasoning-text; it never enters the content, and nothing here puts it into a later request. It is generated like any other token, so it counts against max_tokens: raise max_tokens by the budget you allow. A provider that breaks the count out (usage.completion_tokens_details.reasoning_tokens) has it lifted onto reasoning-tokens.

A llama.cpp server keeps each slot's last prompt in KV cache and reuses the longest prefix the next request on that slot shares with it. Requests that share a long prefix (the same rules in the same place) are cache hits when they run on the same slot; left to itself the server picks a slot by prompt similarity or idleness, and the hits come and go. id_slot pins them. Pinned requests on one slot run one after the other.

probe-server-limits asks the server what it runs with: llama.cpp answers GET /props (default_generation_settings.n_ctx per slot, and total_slots); LLM::Chat::Backend::KoboldCpp asks GET /api/extra/true_max_context_length. The result always has source, then context_length (and slots) or a note saying why nothing was learnt. It never throws and never takes longer than server-probe-timeout (3 seconds by default).

Request-body finalization

_get-api-settings can only describe how to generate — model, samplers, response format — because it runs before the payload is attached. Anything that has to look at the conversation itself needs _finalize-request-body, which every completion path calls with the finished body just before the POST:

class MyBackend is LLM::Chat::Backend::OpenAICommon {
    # Some gateways bill a "priority" tier per request and want the
    # flag next to the payload rather than in the sampler block.
    method _finalize-request-body(%settings --> Hash) {
        my %out = %settings;
        %out<priority> = 'fast' if (%out<messages> // []).elems > 20;
        %out;
    }
}

Three rules make an override safe:

• Match the signature exactly. method _finalize-request-body(%settings --> Hash), character for character. Raku treats a divergent signature as a new method rather than an override, and the parent's no-op keeps answering the call with no error anywhere.

• Return the body. The caller replaces its body with your return value, so an override that mutates in place and returns something else (the last statement of an if, say) throws the request away.

• Build, don't reach back. %settings<messages> holds hashes minted by LLM::Chat::Conversation::Message.to-hash for this one request. Rewriting them by constructing new hashes is fine; treating them as a handle on the caller's conversation is not.

Both chat paths and both text paths call it, so an override that only makes sense for one shape must say so itself — check for %settings<messages>:exists (chat) or %settings<prompt>:exists (text) and return the body unchanged otherwise. LLM::Chat::Backend::OpenRouter does exactly that for its cache_control breakpoints.

REQUEST BODY SERIALIZATION

Every request body — chat and text, blocking and streaming, on this class, LLM::Chat::Backend::OpenRouter and LLM::Chat::Backend::KoboldCpp — is serialized by LLM::Chat itself and posted as bytes with Content-Type: application/json. The body is no longer handed to Cro as a Hash, because a Hash has no member order: the same request used to go out with its members in a different order on every call, and that included the JSON Schema of a structured-output request.

The order is not cosmetic there. llama.cpp and KoboldCpp compile the schema into a sampling grammar that makes the model write an object's fields in the order the schema lists them — so the model wrote its fields in a random order per request, and an application could not ask for, say, the evidence before the conclusion it supports.

The bytes are now fixed by one rule (full statement and edge cases in LLM::Chat::WireJSON):

• Arrays keep their order.

• Every object's members are sorted by code point (for ASCII names: digits, uppercase, _, lowercase).

• A schema's properties map follows the schema's required array — required names first, in array order, then the optional properties, sorted. This holds at every depth: nested objects, items, $defs, anyOf / oneOf / allOf branches, tool parameters, response_format's schema, KoboldCpp's json_schema.

• The conversation is data. Everything under messages is emitted with sorted members at every depth; nothing in it is ever read as a schema. The values of const, default, enum and examples inside a schema are data in the same way.

So the order of required is the order the model writes the fields:

my $settings = LLM::Chat::Backend::Settings.new(
    json_schema => %(
        type       => 'object',
        properties => %(
            change   => %( type => 'string' ),
            evidence => %( type => 'string', description => 'verbatim quote' ),
            note     => %( type => 'string' ),
        ),
        # evidence before the change it justifies; note is optional
        required   => [ 'evidence', 'change' ],
    ),
);

# response_format on the wire (one line; wrapped here):
#   {"json_schema":{"name":"response","schema":{"properties":{
#    "evidence":{"description":"verbatim quote","type":"string"},
#    "change":{"type":"string"},"note":{"type":"string"}},
#    "required":["evidence","change"],"type":"object"},"strict":false},
#    "type":"json_schema"}

Tool definitions get the same treatment, so a tool's parameters reach the model — and a grammar built from them — in required order too.

Beyond the order, what changed is only framing: the body is compact JSON as before, it parses to the same data as before, and the same Hash now produces the same bytes on every call and in every process — which is also what lets a provider's prompt cache match a repeated request. With LLM_CHAT_DEBUG set, the chat paths log the body as those exact bytes (compact, one line) rather than as a pretty-printed re-rendering, so a logged request can be replayed as sent.

A subclass that needs different bytes overrides _encode-request-body:

class PrettyWireBackend is LLM::Chat::Backend::OpenAICommon {
    # A debugging proxy that insists on sorted, pretty JSON.
    method _encode-request-body(%body --> Blob:D) {
        to-json(%body, :pretty, :sorted-keys).encode('utf-8');
    }
}

MESSAGE NORMALIZATION

Some chat templates accept only one conversation shape: an optional system message, then user and assistant turns strictly alternating. Mistral's refuse anything else outright —

After the optional system message, conversation roles must
    alternate user and assistant roles except for tool calls and
    results.

— and a server that renders the model's own template (llama.cpp, or KoboldCpp launched with --jinja) passes that refusal straight back. Applications build conversations that break the rule for good reasons: input split into several user messages, an instruction appended as a second user message, a repair packet after a bad reply, a system message injected mid-conversation. Turn on message-normalization and the backend reshapes the conversation on the way out:

my $backend = LLM::Chat::Backend::KoboldCpp.new(
    api_url               => 'http://localhost:5001/v1',
    settings              => LLM::Chat::Backend::Settings.new,
    message-normalization => 'alternate-roles',
);

my @messages =
    LLM::Chat::Conversation::Message.new(role => 'system',    content => 'You are terse.'),
    LLM::Chat::Conversation::Message.new(role => 'user',      content => 'Part 1 of the document'),
    LLM::Chat::Conversation::Message.new(role => 'user',      content => 'Part 2 of the document'),
    LLM::Chat::Conversation::Message.new(role => 'assistant', content => 'Summary: ...'),
    LLM::Chat::Conversation::Message.new(role => 'system',    content => 'Reply in JSON.'),
    LLM::Chat::Conversation::Message.new(role => 'user',      content => 'Again, please.');

$backend.wire-message-hashes(@messages);
# [
#   { role => 'system',    content => "You are terse." },
#   { role => 'user',      content => "Part 1 of the document\n\nPart 2 of the document" },
#   { role => 'assistant', content => "Summary: ..." },
#   { role => 'user',      content => "[System message]\nReply in JSON.\n\nAgain, please." },
# ]

The default, 'none', sends every message as given.

The rules

Only plain messages take part: user, assistant and system messages made of a role and text and nothing else. A tool result, an assistant turn carrying tool_calls, or a message with any other key is passed through exactly as it is, and nothing is ever merged into or across it.

• Adjacent plain messages of the same role merge into one, their contents joined with a blank line ("\n\n") in their original order. Two user messages become one user message; two assistant messages without tool calls become one assistant message; a run of system messages at the start becomes one system message.

• The leading system message stays where it is. Everything before the first non-system message is the leading block, and it is never rewritten beyond merging it into one message — it is the prefix every request shares, and rewriting it would change every request.

• A system message after the first turn becomes user text. Its content goes out as "[System message]\n" followed by its text (the marker line is folded-system-marker; set it to '' for none), in the place the system message stood. That text then merges like any user message: into the user message right after it (as a prefix), otherwise into the user message right before it (as a suffix), otherwise it stands as a user turn of its own — between two assistant turns, or after a tool result. It never moves past another message: folding it into a later user message across an assistant reply would reorder the conversation, and would make an earlier request's messages change when a later one appends to them.

• Tool calls stay whole. An assistant turn with tool_calls and the tool results answering it are left as they are, in place. A user message right after a tool result merges with user messages after it, never with one before the tool call.

• A user message right after tool results is acknowledged. Once the merges and folds above are done, a user message that directly follows a tool result gets an assistant turn in front of it, with the fixed text tool-result-acknowledgement ('Understood.'). See /Tool results are acknowledged.

• Nothing is reordered, and no text is dropped. The one thing removed is an empty system message, which carries nothing and would fold into a bare marker line; the one thing added is the acknowledgement above. An empty user or assistant message is kept: merged into a same-role neighbour it adds nothing (no stray blank line), and standing alone it is still a turn, which keeps the roles alternating. Whitespace is text and is kept verbatim.

Tool results are acknowledged

A template that checks alternation skips tool turns first — Mistral's drops every tool message and every assistant turn carrying tool_calls before it looks — so a user message straight after tool results reads as the user speaking twice:

user        What's the weather in Paris?
    assistant   (tool_calls: get_weather)      skipped
    tool        18°C, cloudy                   skipped
    user        And in Rome?                   a second user turn: refused

That conversation comes about whenever the user speaks before the model has answered the tool results: an application that stops a tool loop to take new input, or one that appends a user message to a transcript ending in tool results. The model has read the results; it has just not said anything since. Normalization says it for the model: one assistant turn, whose content is tool-result-acknowledgement, goes between the last tool result and the user message.

my $backend = LLM::Chat::Backend::KoboldCpp.new(
    api_url               => 'http://localhost:5001/v1',
    settings              => LLM::Chat::Backend::Settings.new,
    message-normalization => 'alternate-roles',
);

my %call = id => 'call-1', type => 'function',
    function => %( name => 'get_weather', arguments => '{"city":"Paris"}' );

my @messages =
    LLM::Chat::Conversation::Message.new(role => 'user', content => "What's the weather in Paris?"),
    LLM::Chat::Conversation::Message.new(role => 'assistant', tool-calls => [%call,]),
    LLM::Chat::Conversation::Message.new(role => 'tool', content => '18°C, cloudy', tool-call-id => 'call-1'),
    LLM::Chat::Conversation::Message.new(role => 'user', content => 'And in Rome?');

$backend.wire-message-hashes(@messages).map(*<role>);
# (user assistant tool assistant user)
#                      ^^^^^^^^^ { role => 'assistant', content => 'Understood.' }

Exactly:

• A user message directly after a tool result, once merging and folding are done, gets one acknowledgement in front of it. That user message may be the caller's own, several merged into one, or a mid-conversation system message folded into user text (so tool results followed by a system message then an assistant reply become tool, acknowledgement, folded user turn, assistant). A user message that carries more than a role and text is still the user's turn and is acknowledged the same way. Empty system messages are dropped before this is decided, and with system-message-mode('leading') the chat paths have already lifted every system message to the front.

• A run of tool results gets one acknowledgement, after the last of them.

• Nothing is inserted anywhere else. Tool results followed by an assistant turn — the model's answer, or another tool call — need nothing, and a conversation that ends in tool results (the usual request of a tool loop, asking the model to answer them) goes out as it is.

• The acknowledgement is the string, verbatim, as a plain assistant message. It is inserted after every merge has been decided, so it is never merged with anything, and the tool results, the tool-call turn and the user message are left exactly as they would be without it.

• Set tool-result-acknowledgement to other words, or to '' to insert nothing — the template will then refuse such a conversation. An undefined value dies when the backend is built.

Which tool results were acknowledged

A Response carries nothing about the request it was made from, so the acknowledgements are not reported on it. They are a pure function of the messages, though, and the acknowledgements are the only difference between the list normalized with them and the list normalized without them, each one standing where the list without them has the user message it precedes. To log them:

my @sent = $backend.normalized-messages(@messages);
my @bare = $backend.clone(tool-result-acknowledgement => '')
                   .normalized-messages(@messages);

my @positions = gather {
    my $b = 0;
    for @sent.kv -> $p, $message {
        if $b < @bare.elems && $message.role eq @bare[$b].role { $b++ }
        else { take $p }
    }
}
# @positions indexes @sent, the messages a text path renders;
# @positions.elems is how many acknowledgements there are.

The same comparison between two wire-message-hashes calls, reading .<role> , finds them in a chat body.

Deterministic, and friendly to prompt caches

Normalization is a pure function of the message list: the same messages always produce the same messages, and so the same bytes (see /REQUEST BODY SERIALIZATION). It is also prefix-stable: appending messages to a conversation never changes any normalized message except the last one, and that one only grows — its new content starts with its old content. So consecutive requests of a growing conversation repeat each other's bytes up to the previous request's last message, which is what a provider's prompt cache (or KoboldCpp's context reuse) needs to match. The last message does grow when the next request adds a message of its role — a second user message, a trailing system instruction — so a request that ends on a user message and is followed by another user message shares its prefix only up to that message.

Acknowledgements keep this. One is decided by a tool result and the message after it, and neither changes once both exist. A conversation that ends in tool results normalizes to a list ending in its last tool result; append a user message (or a system message, which folds into one) and the next request repeats that list unchanged, the tool result included, then carries two new messages: the acknowledgement and the user message. An acknowledgement is never the last message, so it never grows, and appending an assistant message after tool results inserts none.

What it does not fix

• A conversation whose first turn after the system message is an assistant message. Normalization never invents a user message.

• Assistant text on both sides of a tool call: a plain assistant message written just before a tool-call turn, then the assistant's answer after the tool results. A template that skips the tool-call turn sees two assistant turns in a row. Joining them would mean merging across the tool call, which is exactly what is not done.

Where it applies

Both chat paths (chat-completion, chat-completion-stream) on this class and every subclass normalize the messages they send, inside wire-message-hashes. With system-message-mode('leading') as well, the system messages are lifted to the front first, so there is nothing mid-conversation left to fold, and normalization then merges what remains.

Both text paths (text-completion, text-completion-stream) render their prompt from a template on this side of the wire, and a template — an LLM::Chat::Template::Jinja2 running the model's own chat template, say — enforces the same rule locally. They hand the template normalized-messages(@messages): the same normalization, as LLM::Chat::Conversation::Message objects. The text paths always hand the template a fresh Array, normalized or not, because several built-in templates pop the last message when rendering a continuation.

The caller's messages are never modified. LLM::Chat::TokenCounter counts the conversation as given; with normalization on, the prompt actually sent differs by the merge delimiters and marker lines.

STREAM TERMINATION CONTRACT

A streamed generation can end in exactly three ways, and both chat-completion-stream and text-completion-stream route every one of them through the same private !settle-stream, so the Response a consumer inspects means the same thing whichever method produced it and whichever provider was on the other end.

The finish chunk is recorded, not acted on

A finish_reason is stamped on the Response (.finish-reason and .provider-finish-reason) the moment it arrives, and the chunk carrying it is the last one whose content, reasoning and tool-call deltas count. The stream is not settled there. It keeps reading until [DONE] or the body closing, taking only usage and model from what follows — because that is where the usage is: an OpenAI-style stream with stream_options.include_usage sends

data: {"choices":[{"delta":{},"finish_reason":"length"}]}
    data: {"choices":[],"usage":{"prompt_tokens":900,"completion_tokens":64,...}}
    data: [DONE]

and a stream settled at the first line never read the second. Failed streams — the ones whose token spend a caller most wants to see — used to arrive with no usage at all, and so did every text stream.

A server that sends its finish chunk and then neither [DONE] nor a close is given stream-finish-grace seconds (default 10), then settled on the reason it gave, and its connection is dropped. Without that bound such a server would hold the Response open for the body's own timeout, which is unbounded by default.

Which way the stream settles is decided from the finish reason, after _reconcile-finish-reason has had its say (see /Correcting a provider's finish reason):

1. The provider says it is finished

Either the [DONE] sentinel arrives with no finish reason sent, or a finish_reason of stop / tool_calls arrived and then [DONE] or a close. This is the success path: _on-stream-complete fires first, then the supply completes.

.is-done      True
    .is-success   True
    .finish-reason  'stop' / 'tool_calls' / undefined ([DONE] with no
                    reason chunk — some providers send none)
    .error-class    undefined

2. The provider names a terminal failure

A finish_reason of length or content_filter, or one this library does not recognise, then [DONE], a close, or the grace running out. [DONE] does not rescue it — it only means the server stopped talking.

.is-done      True
    .is-success   False
    .finish-reason  'length' / 'content_filter' / whatever arrived
    .error-class    'response'
    .err            "Hit max tokens" / "Blocked by content filter" /
                    "Unknown finish reason: ..."
    .completion-tokens  (and the other usage fields) when the provider
                    sent a usage frame after the finish chunk

Note that length is a failure here. A generation stopped by the completion budget is a truncated one, and the caller decides whether a truncated answer is worth keeping — it never arrives labelled as a complete one. The partial text is still on the Response (.latest / .msg).

3. The body just stops

The connection closes with neither [DONE] nor a finish reason: a dropped connection, a killed upstream worker, a proxy that gave up mid-stream. There is no way to tell a reply that ended early from one that ended, so this fails.

.is-done      True
    .is-success   False
    .finish-reason  undefined
    .error-class    'response'
    .err            'Stream closed without [DONE] or a finish reason'

The important half of this case is that the Response settles at all. It used to be left unterminated, and every consumer waiting on .is-done waited out its own timeout instead — minutes, per occurrence, for a stream that had been over the whole time.

Cancelling between the finish chunk and the end

A cancel that lands after the finish chunk but before the stream settles is an ordinary cancel: the Response is cancelled at once, the connection is dropped, and no finish reason is reconciled for it. .finish-reason still names the reason that had arrived.

Correcting a provider's finish reason

Some providers put the wrong reason on the wire. KoboldCpp reports a stop on a custom stop sequence as length. _reconcile-finish-reason is the hook for that: it sees every finish reason once, on all four completion paths — at the settle point on a stream, so the usage frame has been read by then — and returns the reason the Response should act on. The default returns the provider's reason unchanged.

class MyGateway is LLM::Chat::Backend::OpenAICommon {
    # This gateway says 'eos' where OpenAI says 'stop'.
    method _reconcile-finish-reason(
        LLM::Chat::Backend::Response $response,
        Str:D $reason,
        %request,
        --> Str:D
    ) {
        $reason eq 'eos' ?? 'stop' !! $reason;
    }
}

The Response then carries both: .finish-reason is the decided reason, .provider-finish-reason the one on the wire, and .finish-reason-evidence whatever the hook recorded about how it decided. A hook that throws keeps the provider's reason.

Dropped frames

A data: line that does not parse as JSON is skipped rather than fatal — providers interleave error fragments and truncated frames into otherwise healthy streams — but each one bumps Response.dropped-frames, and that counter changes how the stream is allowed to end:

• Prose only. The stream terminates normally, per the three cases above, and .dropped-frames is left for the consumer to read. A hole in prose costs a few words and is visible to whoever reads the reply.

• Tool calls assembled. The stream fails with error-class response, whatever else was going on — including a perfectly well-formed [DONE]. Tool-call arguments are streamed as JSON fragments that are concatenated blind, so a hole in them produces either a parse failure or, far worse, valid JSON that says something the model never asked for. That is corruption no downstream consumer can detect, so it never reaches is-success.

CANCELLATION

There is a fourth way a generation can end, and it is not the provider's decision: the consumer changes its mind. cancel exists for that, and on an OpenAI-compatible endpoint it has to do something slightly counter-intuitive — close the connection — because the protocol has no abort message. A client that stops wanting a completion hangs up, and llama.cpp's server, OpenRouter and the gateways between them all stop generating when it does.

my $resp = $backend.chat-completion-stream(@messages);
# ... user hits escape ...
$backend.cancel($resp);      # or $resp.cancel — same thing

$resp.is-done             # True, immediately and synchronously
$resp.is-cancelled        # True
$resp.err                 # 'Cancelled by user'
$resp.error-class         # undefined — a cancel is not a failure
$resp.latest              # whatever had arrived, kept

What that buys is the difference between a cancel and the appearance of one. Quitting only the local supply leaves this class's react loop draining the body to the end: the consumer sees nothing more, and the model on the other side generates — and bills for — the entire reply anyway. On a slow model with a consumer that cancels freely (a chat UI where every re-roll supersedes the last request), that accumulates zombie generations nobody will ever read.

Three properties are worth relying on:

• The local close is synchronous and unconditional. cancel quits the supply before it touches the network, so a hung or already-dead endpoint cannot delay or prevent the consumer's cancel. Transport teardown is best-effort and swallows its own errors.

• A cancel during prefill is not lost. Before response headers arrive there is nothing to close — Cro's client exposes no handle on a request in flight. The Response records that a cancel was asked for and aborts the moment the transport exists, so the abort lands on the first response byte rather than never. This is the common case, not the corner one: prefill is exactly when a user gets bored.

• A cancel after the generation finished is a no-op. The aborter is dropped when the Response reaches any terminal state, so a late or duplicate cancel quits an already-quit supply and changes nothing — a completed Response keeps is-success.

A cancelled Response is deliberately not classified. Cro reports the torn-down body as X::Cro::HTTP::Client::Timeout(phase = 'body')>, which is indistinguishable from a genuinely stalled body, and filing it as error-class 'timeout' would tell a fallback policy to retry the request the user just abandoned. !record-exception checks Response.cancel-requested and leaves the terminal state that cancel already set.

The same mechanism covers deadlines. Cro's body and total request timeouts cancel the response object themselves, so a request that runs past request-timeout drops its connection instead of leaving the server generating into a socket nobody reads.

Subclasses

Neither LLM::Chat::Backend::OpenRouter nor LLM::Chat::Backend::KoboldCpp overrides the streaming methods — they extend settings, headers, Response shapes and the completion hooks — so this contract is theirs too. A subclass that did override a stream loop would have to settle the Response itself; the straightforward way is to keep the react block's structure and settle in the same three places this class does (the [DONE] arm, the stream-finish-grace timer, and once on the way out of the react block), recording a finish reason where it arrives rather than acting on it there.