ONNX Function Calling Support

[Roger Barreto (rogerbarreto)]

I might be missing something but I didn't find any support for function calling on the APIs.

This capability arriving strong for SLM's (including Llama3+) there's a very big ask from community to start supporting function calling on SLMs.

As we seen this happening and arriving for Model Engines like Ollama would be very nice to have an abstraction that allow us to consume models with this capability with the Generator.

[Nat Kershaw (MSFT) (natke)]

Hi Roger Barreto (@rogerbarreto), we are working on this - via the constrained decoding feature. Do you have specific use-cases you can share to help us validate the feature? I'll move this into a discussion

[Roger Barreto (rogerbarreto)]

I have plenty of use cases, but most of them are related to the current OpenAI contracts on tools behavior, if we can have some examples (if possible following similar abstractions) from the ONNX GenAI API perspective that would be really helpful.

• Request.Tools[] (advertisement - Available Fuctions)
• Request.Tool Choice (Auto/Required/None) (How the model will behave about the functions advertised)
• Response.Message.Assistant + ToolCalls[] (Response from the model when it wants to call one or many functions)
• Request.Message.Tool + FunctionResult (giving back to the model the function response)

[madroid (madroidmaq)]

When I tried to construct a Function Calling Prompt myself, I encountered some issues, see #994.

[Ying Xiong (Taka152)]

Hi, we just create #1038 about constrained decoding that can limit generation with json schema and regex, and should support function call too. If you are interested and want to try, it may needs to install rust toolchain and build from source for now.

[Aaron LeMasters (lilhoser)]

Hello, just wondering what the plan is for the above PR - I see it is blocked on Rust dependency resolution.

[dantot32]

Hello everyone, if can help in the meanwhile I made this simply workaround. The full repo is here.

    // for example model should return syimply: Functools[{"name": "get_current_time", "arguments": {}}]
    var toolCalls = ToolCallHelper.TryExtractCalls(response);

    if (toolCalls.Any())
    {

#if DEBUG
        Console.WriteLine($"[Tool Call] {response}");
#endif

        // 1. add assistant message with tool calls
        history.Add(new Message
        {
            Role = "assistant",
            Content = JsonSerializer.Serialize(toolCalls.ToArray())
        });

        // 2. execute each call
        List<ToolCallOutput> outputs = new();
        foreach (var toolCall in toolCalls)
        {
            try
            {
                var toolCallOutput = ToolCallHelper.TryExecuteTool(toolCall);
                outputs.Add(toolCallOutput);

#if DEBUG
                Console.WriteLine($"[Tool Call Output: {toolCall.Name}] {toolCallOutput.Output}");
#endif
            }
            catch (Exception ex)
            {
                Console.WriteLine($"[ERROR] Function execution failed: {ex.Message}");
            }
        }

        // 3. return final response
        history.Add(new Message
        {
            Role = "assistant",
            Content = JsonSerializer.Serialize(outputs.ToArray())
        });

        messagesJson = JsonSerializer.Serialize(history.ToArray());
        chatTemplate = tokenizer.ApplyChatTemplate("", messagesJson, "", true);
        sequences = tokenizer.Encode(chatTemplate);
        generator.AppendTokenSequences(sequences);

        while (!generator.IsDone())
        {
            generator.GenerateNextToken();
            var lastTokenId = generator.GetSequence(0)[^1];
            var token = tokenizerStream.Decode(lastTokenId);
            Console.Write(token);
        }
   }

[kunal-vaishnavi]

All examples have been updated in this PR. Please see the READMEs in each example for more information on tool calling.