Overview
LLMSwitcher is a specialized version of ServiceSwitcher designed specifically for managing multiple LLM services. Beyond basic service switching, it provides convenient methods for running ad-hoc inferences and registering function handlers across all LLMs simultaneously.
This is particularly useful when you need to switch between different LLM providers based on task complexity, cost optimization, or specific model capabilities while maintaining a consistent function calling interface.
Constructor
The strategy class to use for switching logic. Pass the class itself, not an
instance. Defaults to
ServiceSwitcherStrategyManual.Properties
Methods
run_inference()
Run a one-shot inference with the currently active LLM, outside of the normal pipeline flow.Additional arguments forwarded to the active LLM’s
run_inference method.
Common options include max_tokens (maximum tokens to generate),
system_instruction (override the system prompt for this inference), and
response_schema (optional JSON schema the reply must follow).str | None - The LLM’s response as a string, or None if no LLM is active.
register_function()
You don’t need
register_function when your tool’s handler is bundled on its
FunctionSchema: listing it in LLMContext(tools=[...]) or adding it via
LLMSetToolsFrame is enough.The name of the function to handle. Use None for a catch-all handler that
processes all function calls.
Whether to cancel this function call when a user interruption occurs. When
False, the call is treated as asynchronous: the LLM continues the
conversation immediately without waiting for the result, and the result is
injected later via a developer message. Defaults to None, which falls back
to the handler’s @tool_options value, then to True.Optional per-tool timeout in seconds. Overrides the global
function_call_timeout_secs for this specific function.Whether the LLM may cancel this call while it runs, through the
cancel_<name> tool advertised alongside it. Pair it with
cancel_on_interruption=False. Defaults to None, which falls back to the
handler’s @tool_options value, then to False.register_direct_function()
Register a direct function handler with all LLMs in the switcher, regardless of which is currently active.Whether to cancel this function call when a user interruption occurs. When
False, the call is treated as asynchronous: the LLM continues the
conversation immediately without waiting for the result, and the result is
injected later via a developer message. Defaults to None, which falls back
to the handler’s @tool_options value, then to True.Optional per-tool timeout in seconds. Overrides the global
function_call_timeout_secs for this specific function.