feat: add tool calling support to m serve

[markstur]

Misc PR

Type of PR

• Bug Fix
• New Feature
• Documentation
• Other

Description

• Link to Issue: Fixes m serve OpenAI API tool calling round-trip #825

Successfully added tool calling support to m serve CLI with proper type annotations. Here's what was implemented:

Changes Made

1. Updated Models (cli/serve/models.py)

• Added ToolCallFunction model for function details in tool calls
• Added ChatCompletionMessageToolCall model for tool call structure
• Extended ChatCompletionMessage to include optional tool_calls field
• Updated Choice.finish_reason to support "tool_calls" value

2. Modified Server Logic (cli/serve/app.py)

• Added json and Literal imports for proper typing
• Imported new tool-related models
• Updated _build_model_options() to pass through tools (mapped to ModelOption.TOOLS) and tool_choice parameters
• Enhanced make_chat_endpoint() to:
  • Extract tool calls from ModelOutputThunk.tool_calls with proper type checking (isinstance(dict))
  • Generate unique IDs for each tool call in format call_<24-char-hex>
  • Serialize tool arguments to JSON
  • Set finish_reason with proper Literal type annotation
  • Return tool calls in OpenAI-compatible format

3. Comprehensive Tests (test/cli/test_serve_tool_calling.py)

• 8 new tests covering:
  • Single and multiple tool calls
  • Tool call formatting and serialization
  • Complex nested arguments
  • Tool parameters passed to model_options
  • Backward compatibility (requests without tools)
  • Usage info alongside tool calls

4. Updated Existing Test (test/cli/test_serve.py)

• Renamed test_tool_params_excluded_from_model_options to test_tool_params_passed_to_model_options
• Updated assertions to verify tools and tool_choice are now passed through

5. Example Code

• docs/examples/m_serve/m_serve_example_tool_calling.py: Complete server example with GetWeatherTool and GetStockPriceTool implementations
• docs/examples/m_serve/client_tool_calling.py: Client demonstrating how to call the tool-enabled server with various scenarios

Key Features

✅ OpenAI-Compatible: Follows OpenAI's tool calling API format
✅ Type-Safe: Proper Literal type annotations for finish_reason
✅ Robust Type Checking: Uses isinstance(dict) to avoid Mock object issues
✅ Automatic Tool Call Detection: Extracts tool calls from ModelOutputThunk
✅ Proper Finish Reasons: Returns "tool_calls" when tools are invoked, "stop" otherwise
✅ Unique Tool Call IDs: Generates unique IDs in format call_<24-char-hex>
✅ JSON Serialization: Properly serializes tool arguments to JSON strings
✅ Backward Compatible: Works with existing code that doesn't use tools
✅ Fully Tested: All 43 serve tests pass, including 8 new tool-specific tests
✅ Type Checked: Passes mypy type checking

Usage

Start server with tool support:

uv run m serve docs/examples/m_serve/m_serve_example_tool_calling.py

Call with tools from client:

response = requests.post(
    "http://localhost:8080/v1/chat/completions",
    json={
        "model": "gpt-3.5-turbo",
        "messages": [{"role": "user", "content": "What's the weather in Paris?"}],
        "tools": [...],  # Tool definitions
        "tool_choice": "auto"
    }
)

The implementation properly handles tool calls from Mellea's ModelOutputThunk and formats them according to OpenAI's API specification with full type safety.

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code as added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

[github-actions]

The PR description has been updated. Please fill out the template for your PR to be reviewed.

[planetf1]

@markstur Do you want review comments yet or still WIP?

[markstur]

@markstur Do you want review comments yet or still WIP?

Comments would be great! It is draft because I need to do more review/test myself on the generated code. I don't want to waste your time but comments early would be very welcome.

[psschwei]

Code Review: feat: add tool calling support to m serve

Good feature PR — the core plumbing is correct and the OpenAI-compatible response format looks right. A couple of bugs to fix before merge, plus some improvements.

Summary

The implementation correctly wires tool calling through the serve endpoint: tools maps to ModelOption.TOOLS, tool_choice passes through as-is, and the response extracts tool calls from ModelOutputThunk into the OpenAI format. The Pydantic models mirror the OpenAI types well, and tests cover the main paths.

Two bugs need fixing (see inline comments):

1. Empty tool_calls dict produces incorrect finish_reason: "tool_calls" with an empty array
2. Client example's multi-turn loop duplicates the assistant message for each tool call

Other improvements (see inline comments):

• Unused loop variable tool_name
• eval() in example code with # noqa suppressing the security lint for copy-pasters
• Missing test for the empty dict edge case
• hasattr check is always true for ModelOutputThunk — defensive but masks upstream bugs

What's working well

• Pydantic models (ToolCallFunction, ChatCompletionMessageToolCall) closely match OpenAI types
• _build_model_options change is clean — tools removed from exclusion set, mapped to ModelOption.TOOLS
• 8 well-structured tests covering single/multiple tool calls, finish reasons, model_options passthrough, complex args, usage info, and backward compat
• Existing test updated consistently from "excluded" to "passed"

[planetf1]

Two additional items not yet covered in existing review comments.

[planetf1]

Two additional items not covered in existing review comments.

[markstur]

Code Review: feat: add tool calling support to m serve

Good feature PR — the core plumbing is correct and the OpenAI-compatible response format looks right. A couple of bugs to fix before merge, plus some improvements.

Summary

The implementation correctly wires tool calling through the serve endpoint: tools maps to ModelOption.TOOLS, tool_choice passes through as-is, and the response extracts tool calls from ModelOutputThunk into the OpenAI format. The Pydantic models mirror the OpenAI types well, and tests cover the main paths.

Two bugs need fixing (see inline comments):

1. Empty tool_calls dict produces incorrect finish_reason: "tool_calls" with an empty array
2. Client example's multi-turn loop duplicates the assistant message for each tool call

Other improvements (see inline comments):

• Unused loop variable tool_name
• eval() in example code with # noqa suppressing the security lint for copy-pasters
• Missing test for the empty dict edge case
• hasattr check is always true for ModelOutputThunk — defensive but masks upstream bugs

What's working well

• Pydantic models (ToolCallFunction, ChatCompletionMessageToolCall) closely match OpenAI types
• _build_model_options change is clean — tools removed from exclusion set, mapped to ModelOption.TOOLS
• 8 well-structured tests covering single/multiple tool calls, finish reasons, model_options passthrough, complex args, usage info, and backward compat
• Existing test updated consistently from "excluded" to "passed"

Fixed all these.
The eval one goes away with the removal of calc (replaced by stock "look-up")

[psschwei]

LGTM
Since @planetf1 had also reviewed, will leave final approval to him

dismissing the requested change which was resolved
@psschwei approved but waiting for @planetf1 to approve

[planetf1]

Most issues fine - I just have one concern -- the finish reason I don't think is what you intended for streaming?

[planetf1]

delta.tool_calls is missing the required index field — breaks compatibility with real OpenAI SDK clients.

The OpenAI streaming spec requires each item in delta.tool_calls to carry index: int. Clients including the openai Python SDK, LangChain, and LiteLLM key their delta-reassembly state machine on this field — without it they silently drop tool calls, coalesce them incorrectly, or raise a TypeError depending on version.

The root cause is that ChatCompletionChunkDelta.tool_calls (models.py:173) reuses ChatCompletionMessageToolCall — a complete (non-streaming) model that has no index field. The streaming delta shape needs a distinct model where index is required and id, type, and function are optional (since args can arrive across multiple chunks).

The bundled client_streaming_tool_calling.py masks this because it reads delta.tool_calls verbatim rather than going through SDK delta reassembly — so the example passes but any spec-compliant consumer breaks.

[planetf1]

build_tool_calls is called here unconditionally, then discarded on the streaming branch.

When request.stream is True (line 212), execution returns via StreamingResponse before tool_calls is ever used — and streaming.py:108 calls build_tool_calls(output) again, generating entirely different call_<uuid> IDs. The IDs minted here are never sent to any client.

This is harmless today, but any logging or tracing that captures tool_calls_list from this path will record IDs that never appear in the actual response, making debugging painful. Consider moving this block into the non-streaming branch (after the if request.stream: guard), or computing once and passing the result into stream_chat_completion_chunks.

[planetf1]

The streaming tool-calling path in streaming.py (lines 107–145) has no test coverage here.

All 8 tests in this class call make_chat_endpoint directly with non-streaming requests. The new tool-call chunk emission, finish_reason logic, and build_tool_calls call in streaming.py are entirely untested — which is also how the missing index field (see other comment) shipped without being caught.

A streaming test using FastAPI's TestClient would catch chunk-shape regressions. For example:

@pytest.mark.asyncio
async def test_tool_calls_streaming(self, mock_module, sample_tool_request):
    sample_tool_request.stream = True
    # ... assert chunks contain delta.tool_calls with index field

[planetf1]

This "RootModel" key is non-standard and will cause any real OpenAI-compatible client to fail.

The standard OpenAI tool format sends parameters as a bare JSON Schema object:

"parameters": {
    "type": "object",
    "properties": { ... },
    "required": [...]
}

This server requires the non-standard {"parameters": {"RootModel": {...}}} envelope because FunctionParameters in cli/serve/models.py defines RootModel as a literal field name rather than accepting the schema directly. Any client using the OpenAI SDK, LangChain, or the standard tool format will get a Pydantic validation error.

This is a pre-existing issue in models.py, but this PR introduces the first public examples that document and normalise the non-standard shape. Worth opening a tracked issue to fix FunctionParameters to accept a bare dict[str, Any] before these examples become the established pattern.

[planetf1]

json.dumps with no fallback encoder will raise TypeError if model_tool_call.args contains a non-JSON-serialisable value — datetime, Decimal, Path, numpy scalar, etc.

Tool args come from model output and backend parsing, so there is no contract guaranteeing they are pure-JSON. This TypeError bubbles up through the generic except Exception handler in app.py and returns an opaque 500 to the client.

A simple guard:

args_json = json.dumps(model_tool_call.args, default=str)

Or catch TypeError here and raise a more descriptive error so the 500 message at least identifies the offending tool.