[GGUF] Support GGUF format for tokenizers and detokenizers #2263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

andrei-kochin merged 19 commits into openvinotoolkit:master from rkazants:support_gguf_tokenizers

May 29, 2025

Collaborator

rkazants commented May 23, 2025 •

edited

Loading

Details: Support GGUF format for tokenizers and detokenizers.

It requires merge of openvinotoolkit/openvino_tokenizers#493


          [GGUF] Support GGUF format for tokenizers and detokenizers

ec62341

Signed-off-by: Kazantsev, Roman <[email protected]>

rkazants requested review from pavel-esir, Wovchena, as-suvorov and popovaan

May 23, 2025 11:48

rkazants added this to the 2025.2 milestone

rkazants requested a review from andrei-kochin

May 23, 2025 11:48

github-actions bot added category: tokenizers category: GGUF labels

rkazants mentioned this pull request

[Good First Issue]: Create a GGUF reader #1665

Closed

rkazants linked an issue

that may be closed by this pull request

[Good First Issue]: Create a GGUF reader #1665

Closed

Wovchena requested a review from sammysun0711

May 23, 2025 13:11

sammysun0711 reviewed

View reviewed changes

src/cpp/src/gguf_utils/gguf_tokenizer.cpp Outdated Show resolved Hide resolved

src/cpp/src/gguf_utils/gguf_tokenizer.cpp Show resolved Hide resolved

sammysun0711 reviewed

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

src/cpp/src/gguf_utils/gguf_tokenizer.cpp Show resolved Hide resolved

rkazants mentioned this pull request

[GGUF] Serialize Generated OV Model for Faster LLMPipeline Init #2218

Merged

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved


          Update src/cpp/src/tokenizer/tokenizer.cpp

077d856

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants commented

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Outdated Show resolved Hide resolved

rkazants added 2 commits

May 27, 2025 00:52


          Fix inference for smollm and qwen in gguf

01de621

Signed-off-by: Kazantsev, Roman <[email protected]>


          Update tokenizers

5a83a67

Signed-off-by: Kazantsev, Roman <[email protected]>

sammysun0711 reviewed

View reviewed changes

src/cpp/src/gguf_utils/gguf_tokenizer.cpp Show resolved Hide resolved

src/cpp/src/gguf_utils/gguf_tokenizer.cpp Show resolved Hide resolved


          Fix test for Smollm

41b930b

Signed-off-by: Kazantsev, Roman <[email protected]>

as-suvorov approved these changes

View reviewed changes

src/cpp/src/tokenizer/tokenizer.cpp Show resolved Hide resolved

rkazants added do_not_merge pr_needs_tests labels

rkazants added 3 commits

May 28, 2025 22:55


          Merge remote-tracking branch 'upstream/master' into support_gguf_toke…

2d64cf0

…nizers


          Patch qwen2.5 chat template

925c63d

Signed-off-by: Kazantsev, Roman <[email protected]>


          Test GGUF tokenizer and detokenizer support

54b1ad2

Signed-off-by: Kazantsev, Roman <[email protected]>

github-actions bot added category: LLM no-match-files labels

rkazants removed category: LLM pr_needs_tests no-match-files labels

rkazants commented

View reviewed changes

tests/python_tests/requirements.txt Outdated Show resolved Hide resolved


          Update tests/python_tests/requirements.txt

8b3cbc9

github-actions bot added category: LLM no-match-files labels


          Test full pipeline for GGUF model

7da67b1

Signed-off-by: Kazantsev, Roman <[email protected]>

github-actions bot removed the no-match-files label


          Add qwen2.5 gguf model for testing

c1089c5

Signed-off-by: Kazantsev, Roman <[email protected]>

github-actions bot added the no-match-files label

rkazants requested a review from sammysun0711

May 29, 2025 15:15

rkazants removed the do_not_merge label

sammysun0711 commented May 29, 2025

LGTM, approved, thanks for great work!

sammysun0711 approved these changes

View reviewed changes

andrei-kochin added this pull request to the merge queue

andrei-kochin removed this pull request from the merge queue due to a manual request

andrei-kochin merged commit 8988da2 into openvinotoolkit:master

66 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

as-suvorov as-suvorov approved these changes

pavel-esir Awaiting requested review from pavel-esir

Wovchena Awaiting requested review from Wovchena

popovaan Awaiting requested review from popovaan

andrei-kochin Awaiting requested review from andrei-kochin

+1 more reviewer

sammysun0711 sammysun0711 approved these changes

Labels

category: GGUF category: LLM category: tokenizers no-match-files