NAME

LLM::Agent::ToolRouting - a tool call as a grammar-constrained choice: the action schema, the parser, the result message and the capped backend

SYNOPSIS

use LLM::Agent::ToolRouting;

my @tools = $provider.tools-for-llm;   # the ordinary catalogue

# One JSON schema, one alternative per tool plus "no tool calls needed".
my %schema = route-schema(@tools);
route-choices(@tools);                 # ('memory_search', 'memory_read', 'none')

# What the model answered, checked against what it was offered.
my %action = parse-route('{"tool":"memory_search","arguments":{"query":"latch"}}',
    route-tool-names(@tools));
# { ok => True, tool => 'memory_search', arguments => { query => 'latch' },
#   text => '{"tool":"memory_search","arguments":{"query":"latch"}}' }

parse-route('{"tool":"none"}', route-tool-names(@tools));
# { ok => True, tool => 'none', text => '{"tool":"none"}' }

parse-route('Sure! Let me search.', route-tool-names(@tools));
# { ok => False, error => "the reply is not a JSON object" }

# The user message that answers a routed call.
routed-result-text('memory_search', routed-call-id(1), '{"records":[]}');
# "Tool result for memory_search (call route-1):\n{\"records\":[]}"

# The request backend: a clone carrying the schema and a low completion cap.
my $request = routed-backend($backend, %schema, routing-cap($backend, 512));

DESCRIPTION

LLM::Agent::Loop's tool-routing => 'schema' mode asks the model for its next action as a grammar-constrained JSON choice rather than through a backend's tool calling. This module is the part of that mode a host can use on its own: to build the same schema the loop will send, to record it, or to check a reply the way the loop checks it.

Why a choice, and not a tool call

Native tool calling has one branch nothing constrains: the reply that calls no tool. A backend that emulates tools (KoboldCpp polls "is a tool essential?" and, when the answer is no, hands the model a completely unconstrained turn) lets that branch run for thousands of tokens — echoing the prompt back until it happens to stop — and a real tool-calling template still leaves it free text. A retrieval phase that only wanted "search, read, or I am done" gets an essay it then has to throw away.

So every reply is one of a closed set of actions, and the set includes the answer "no tool calls needed":

{"tool": "memory_search", "arguments": {"query": "the latch"}}
{"tool": "memory_read",   "arguments": {"ids": ["M1", "M3"]}}
{"tool": "none"}

The schema

route-schema turns a catalogue in tools-for-llm shape ( { type => 'function', function => { name, description?, parameters? } } ) into one JSON schema with an anyOf alternative per tool, in catalogue order, and the none alternative last:

{
  "type": "object",
  "anyOf": [
    { "type": "object",
      "description": "Search memories.",
      "properties": { "tool": { "const": "memory_search" },
                      "arguments": { ...that tool's parameters schema... } },
      "required": ["tool", "arguments"],
      "additionalProperties": false },
    { "type": "object",
      "properties": { "tool": { "const": "none" } },
      "required": ["tool"],
      "additionalProperties": false }
  ]
}

Three details are deliberate:

• anyOf, not oneOf. The alternatives are mutually exclusive by their tool constant, so the two mean the same thing here; grammar compilers (llama.cpp, KoboldCpp) treat them identically, and anyOf is the one every structured-output implementation accepts.

• tool is required first. Grammar-compiling servers write properties in the order of required (see LLM::Chat::Backend::Settings), so the model names its action before it writes a single argument.

• The parameters schema is the tool's own, copied verbatim (and detached). A provider that wants a narrower grammar than its native declaration — an enum of the identifiers it has actually handed out, say — narrows its catalogue; see LLM::Agent::Loop's "Schema tool routing" for why the loop re-reads the catalogue for every routing request.

The catalogue is validated, and a bad one dies with the index and the reason: a declaration that is not type: function, a missing or blank name, a name used twice, a tool named none (it would be indistinguishable from the handoff), or parameters that are not an object schema. A tool with no parameters takes an empty object.

Parsing a reply

parse-route($text, @names) is strict, because a reply that needs repairing is a reply the grammar did not constrain. The whole trimmed text must be one JSON object, and:

• tool is a non-empty string;

• none carries no other key;

• any other tool is one of @names and carries exactly tool and arguments, where arguments is an object with no non-finite number in it (1e999 parses as infinity, and nothing downstream can render it).

No fence stripping, no "the JSON inside the prose", no key-case folding. It answers { ok => False, error => ... } rather than throwing; the error text quotes at most 64 characters of what the model wrote.

The result message

A routed call is answered by a user message, never a tool message: the action was an ordinary assistant message, so the conversation alternates user and assistant and needs no tool template at all. routed-result-text is the one rendering of it, and the header names the call:

Tool result for memory_search (call route-2):
{"records":[{"id":"M1","title":"The latch"}]}

Tool error for memory_read (call route-3):
memory_read ids take the codes memory_search lists

routed-result-header($tool, $call-id, :is-error) is that first line on its own, and split-routed-result($text) takes a result message apart again — (header, content), or an empty list for any other text — for a consumer that bounds or inspects the tool's content without the line the loop put above it:

my ($header, $content) = split-routed-result($message.content);
die 'tool content too large' if $content.defined && $content.encode.bytes > 32768;

routed-call-id($round) is the id the loop gives the call made in a round (route-1, route-2, ...): one action per round, so it is unique within a run and a pure function of it, which keeps a replayed run's bytes identical to the original's.

The request backend

routed-backend($backend, %schema, $max-tokens) clones the backend and its settings, sets json_schema (under the name tool_route) and max_tokens, and leaves the original untouched — so a structured phase built from the same backend keeps its own schema and its own completion budget. routing-cap($backend, $max) is the cap a routing request gets: $max, or the backend's own max_tokens when that is lower, because a routing cap only ever lowers a budget. can-route($backend) is the duck-typed test for a backend this works on: settings (with json_schema, json_schema_name, max_tokens and clone) and clone, which every LLM::Chat backend has.

SEE ALSO

LLM::Agent::Loop (the mode that uses all of this), LLM::Chat::Backend::Settings (json_schema and property order).