Skip to content

Refactor: Split Chat Format Init/Parse into Separate Files #18215

@aldehir

Description

@aldehir

Background Description

The common/chat.cpp and common/chat-parser.cpp files continue to grow as we add support for models. To help with maintainability, I recommend we use a registry to identify which init/parse functions to use for a given chat template. We can then move implementations into separate source files.

Possible Refactor Approaches

Change this:

static common_chat_params common_chat_templates_apply_jinja(
    const struct common_chat_templates        * tmpls,
    const struct common_chat_templates_inputs & inputs)
{
    // ...

    // DeepSeek V3.1: detect based on specific patterns in the template
    if (src.find("message['prefix'] is defined and message['prefix'] and thinking") != std::string::npos &&
        params.json_schema.is_null()) {
        return common_chat_params_init_deepseek_v3_1(tmpl, params);
    }

    // DeepSeek R1: use handler in all cases except json schema (thinking / tools).
    if (src.find("<|tool▁calls▁begin|>") != std::string::npos && params.json_schema.is_null()) {
        return common_chat_params_init_deepseek_r1(tmpl, params);
    }

    // Command R7B: : use handler in all cases except json schema (thinking / tools).
    if (src.find("<|END_THINKING|><|START_ACTION|>") != std::string::npos && params.json_schema.is_null()) {
        return common_chat_params_init_command_r7b(tmpl, params);
    }

    // ... 
}

static common_chat_params common_chat_params_init_deepseek_v3_1(const common_chat_template & tmpl, const struct templates_params & inputs) {
    // ...
}

static void common_chat_parse_deepseek_v3_1(common_chat_msg_parser & builder) {
    // ...
}

Into something like this:

common/chat-formats/
  deepseek.cpp
  command_r7b.cpp
static const auto _ = common_chat_format_register([](auto & handler) {
    handler.format = COMMON_CHAT_FORMAT_DEEPSEEK_V3_1;
    handler.priority = 50;

    // Define markers for efficient search by an Aho-Corasick Automaton
    handler.markers = {
        "message['prefix'] is defined and message['prefix'] and thinking",
    };

    handler.detect = [](const std::string & src, const templates_params & params) {
        // Any additional detection logic not covered by `markers`. This should only run if all defined markers are found.
        return params.json_schema.is_null();
    };

    handler.init = [](const common_chat_template & tmpl, const templates_params & inputs) {
        // ... deepseek v3.1 init ...
        return data;
    };

    handler.parse = [](const std::string & input, bool is_partial, const common_chat_syntax & syntax) {
        return parse_with_builder([](common_chat_msg_parser & builder) {
            // ... common_chat_parse_deepseek_v3_1  implementation moves here ...
        });
    };
});

Or through inheritance.

Benefits

  • Reduce compilation times of incremental builds.
  • Efficiently scan markers for all formats using an Aho-Corasick automaton, instead of multiple find() calls.
  • Simplifies code review

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions