NAME

LLM::Chat::Backend::KoboldCpp - KoboldCpp backend (OpenAI-compatible endpoints)

SYNOPSIS

use LLM::Chat::Backend::KoboldCpp;
use LLM::Chat::Backend::Settings;

my $backend = LLM::Chat::Backend::KoboldCpp.new(
    api_url              => 'http://localhost:5001/v1',
    settings             => LLM::Chat::Backend::Settings.new(
        max_tokens => 1024,
        stop       => ["\nUser:"],
    ),
    request-stream-usage => True,   # token counts on streams
);

my $resp = $backend.chat-completion-stream(@messages);
until $resp.is-done { sleep 0.01 }

if $resp.is-success {
    say $resp.latest;
    # A reply that ended on "\nUser:" finishes as 'stop', even though
    # KoboldCpp put 'length' on the wire:
    say $resp.finish-reason;            # stop
    say $resp.provider-finish-reason;   # length
}

DESCRIPTION

Subclass of LLM::Chat::Backend::OpenAICommon for KoboldCpp's OpenAI-compatible /v1/chat/completions and /v1/completions endpoints. It inherits the request, streaming, error and cancellation behaviour of that class, and adds:

• The full sampler surface. min_p, typical_p, DRY, XTC, rep_pen, max_context_length and friends go on every request, and json_schema goes out as KoboldCpp's native, grammar-enforced field.

• Unbounded prefill. The response-headers timeout is lifted (when left at its default), because a long local context can spend minutes in prompt processing before KoboldCpp answers.

• An abort that reaches prefill. cancel also sends POST /api/extra/abort. See cancel.

• Truthful stop reasons. A length finish is checked against KoboldCpp's own record of why the generation stopped. See below.

• Thinking under KoboldCpp's names. reasoning_effort goes out as is and reasoning_budget_tokens as thinking_budget_tokens (KoboldCpp applies a budget only when the effort is not one of its named levels, which set a budget of their own). id_slot is never sent: KoboldCpp has no slots.

• Its own context size. probe-server-limits asks GET /api/extra/true_max_context_length.

TRUTHFUL STOP REASONS

KoboldCpp's OpenAI-compatible endpoints know three reasons a generation stops, and put two finish reasons on the wire. From koboldcpp.py (1.121), in the streaming and the blocking completion handlers alike:

currfinishreason = "error" if sr==-2 else ("length" if (sr!=1) else "stop")

and the codes, from expose.h:

enum stop_reason
{
    ERROR_ENCOUNTERED=-2,
    INVALID=-1,
    OUT_OF_TOKENS=0,
    EOS_TOKEN_HIT=1,
    CUSTOM_STOPPER=2,
};

So a generation that ended on one of the request's stop strings (CUSTOM_STOPPER, 2) is reported exactly like one that ran out of budget (OUT_OF_TOKENS, 0): finish_reason: "length". The streaming paths treat length as a failure — a truncated reply must never pass for a complete one — so complete replies were being thrown away. It is common after a tool result, where KoboldCpp's emulated tool calling often stops on a stop string with an output of "" or "\n".

When verify-length-finish is on (the default) and a completion finishes with length, the backend asks GET /api/extra/perf — at the server root, not under /v1 — which reports, among much else:

stop_reason       the stop_reason code of the last generation
    last_token_count  how many tokens the last generation produced

The finish reason becomes stop only when all of this holds:

• stop_reason is 2 (CUSTOM_STOPPER).

• The count matches. /api/extra/perf describes the server's most recent generation, and nothing ties it to this one — another client's request could have finished in between. So last_token_count must equal the completion's own usage.completion_tokens. KoboldCpp fills both from the same count on the blocking path; on a stream its usage frame counts the tokens it streamed, which is the same count in every case read from its source — and a stream where they did differ would simply stay length, the safe direction.

• Without usage, the count is well under the budget. KoboldCpp sends no usage on a stream unless asked (request-stream-usage), and never on the streams of its emulated tool calling. With no usage to match, last_token_count must be at most 90% of the request's max_tokens. That cannot prove the perf record is this generation's — turn on request-stream-usage where you can — but it rules out a generation that used up its budget, which is the one being mistaken for a stop.

Anything else leaves length: another stop_reason, a count that does not match, a perf request that fails, answers something other than a JSON object with integer fields, or takes longer than verify-length-finish-timeout seconds (default 2). The check runs once per length completion, after the usage has been read, and never for stop or tool_calls.

The Response records what happened either way:

$resp.finish-reason;            # 'stop' — what the Response acted on
$resp.provider-finish-reason;   # 'length' — what KoboldCpp sent
$resp.finish-reason-evidence;
# {
#   source            => 'koboldcpp:/api/extra/perf',
#   verdict           => 'stop',
#   note              => 'custom stop sequence (stop_reason 2); last_token_count matches usage completion_tokens',
#   stop_reason       => 2,
#   last_token_count  => 3,
#   completion_tokens => 3,
# }

On a stream, stop completes the Response successfully, with whatever text arrived (possibly none). On the blocking paths, which do not fail on length in the first place, only finish-reason changes.

Limits. KoboldCpp's continuous-batching requests do not update the counters /api/extra/perf reads, so with batching on, the record can be an older generation's; the count match is what guards against that, and without usage it is weaker. Turn verify-length-finish off to take length at its word.

ATTRIBUTES

• $.api_url — the OpenAI-compatible base, e.g. http://localhost:5001/v1. KoboldCpp's own endpoints (/api/extra/abort, /api/extra/perf) are addressed at the same URL with the /v1 removed.

• $.api_key — sent as Authorization: Bearer ... (KoboldCpp's --password), on the perf and abort requests too.

• $.model — optional model name.

• $.verify-length-finish — Bool, default True. See /TRUTHFUL STOP REASONS.

• $.verify-length-finish-timeout — seconds, default 2; must be positive and finite.

Everything LLM::Chat::Backend::OpenAICommon accepts — request-stream-usage, message-normalization, system-message-mode, stream-finish-grace, request-timeout — applies here too. KoboldCpp launched with --jinja renders the model's own chat template, which for Mistral-family models demands strictly alternating roles; see message-normalization.