[GGUF] Serialize Generated OV Model for Faster LLMPipeline Init

[sammysun0711]

Details:
This PR aim to cache generated OV model from GGUF model in disk for faster subsequent LLMPipeline initialization w/ OpenVINO model cache.

• Serialize generated OV model from GGUF model w/ GGUF Reader with properties ov::genai::enable_save_ov_model (default value is false)
• User can check if OV model exists in same folder of GGUF model, load OV model directly instead of creating GGUF model w/ GGUF Reader.
• If GGUF model updated, user need to take the responsibility for cache invalidation and re-generate OV model with GGUF Reader.
• Use OPENVINO_LOG_LEVEL environment variable to control the verbose of GGUF related debug information, details please refer to DEBUG_LOG.md

Expected behavior:

• Set environment variable: export OPENVINO_LOG_LEVEL=3

• First run w/ GGUF model:

  • build/samples/cpp/text_generation/greedy_causal_lm gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf "Who are you?"

  [GGUF Reader]: Loading and unpacking model from: gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf
  [GGUF Reader]: Loading and unpacking model done. Time: 196ms
  [GGUF Reader]: Start generating OpenVINO model...
  [GGUF Reader]: Save generated OpenVINO model to: gguf_models/openvino_model.xml done. Time: 466 ms
  [GGUF Reader]: Model generation done. Time: 757ms
  I am Qwen, a large language model created by Alibaba Cloud. I am a language model designed to assist users in generating human-like text, such as writing articles, stories, and even writing books. I am trained on a vast corpus of text data, including books, articles, and other written works. I am also trained on a large corpus of human language data, including written and spoken language. I am designed to provide information and insights to users, and to assist them in their tasks and

• 2nd run w/ OV model:

  • build/samples/cpp/text_generation/greedy_causal_lm gguf_models "Who are you?"

  I am Qwen, a large language model created by Alibaba Cloud. I am a language model designed to assist users in generating human-like text, such as writing articles, stories, and even writing books. I am trained on a vast corpus of text data, including books, articles, and other written works. I am also trained on a large corpus of human language data, including written and spoken language. I am designed to provide information and insights to users, and to assist them in their tasks and

[Copilot]

Pull Request Overview

This PR enhances the model initialization performance by caching generated OpenVINO models on disk and reusing them for subsequent runs. Key changes include:

• In src/cpp/src/utils.cpp, adding a check for an existing cached OpenVINO model based on the GGUF model location.
• In src/cpp/src/gguf_utils/gguf_modeling.cpp, introducing a new function to serialize and save the generated OpenVINO model, and invoking it during model creation.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/cpp/src/utils.cpp	Added logic to reuse a cached OpenVINO model if it exists.
src/cpp/src/gguf_utils/gguf_modeling.cpp	Added serialization function and integrated it into the model creation flow.

[as-suvorov]

@sammysun0711
Proposal looks like implicit model caching. OpenVINO model is being implicitly serialized on disk. And if gguf model has changed it seems outdated serialized OpenVINO model will be loaded.
Can we reuse OpenVINO cache_dir property for such scenario?

Form the logs I see gguf model load time is 245ms. Could you please also provide loading time for OpenVINO serialized model?

[sammysun0711]

Hi @as-suvorov, thanks for your suggestion.

Can we reuse OpenVINO cache_dir property for such scenario?

Do you means if cache_dir property set in property, then we can enable save generated OV model in cache_dir for re-use? I think it is a good proposal that user can control from application whether to serialize OV model in disk explicitly.

Could you please also provide loading time for OpenVINO serialized model?

Sure, I will add loading time for OpenVINO serialized model

[as-suvorov]

@sammysun0711 yes, correct. I guess also need to check if and how OpenVINO invalidates cached model and implement same approach for gguf format.

[sammysun0711]

• Add support to save OV model based on ov::cache_dir properties explicit
• Add time measurement for loading OV model.

1st Run w/o model cache:

Loading and unpacking model from: gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf
Loading and unpacking model done. Time: 202ms
Start generating OpenVINO model...
Model generation done. Time: 292ms

2nd Run w/ model cache + serialize OV model:

Loading and unpacking model from: gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf
Loading and unpacking model done. Time: 189ms
Start generating OpenVINO model...
Save generated OpenVINO model to: model_cache/qwen2.5-0.5b-instruct-q4_0_openvino_model.xml done. Time: 379 ms
Model generation done. Time: 647ms

3rd Run w/ model cache using generated OV model

Found generated OpenVINO model: model_cache/qwen2.5-0.5b-instruct-q4_0_openvino_model.xml, skip creating from GGUF model.
Loading OpenVINO model done. Time: 71ms

check if and how OpenVINO invalidates cached model and implement same approach for gguf format

Current naive method use GGUF file name to check if new OV model need to be regenerated, OV invalidate outdated model cache via hash calculate by compute_hash.hpp, but it seems compute_hash.hpp belong to dev_api, which is not accessible unless openvino.genai static link to openvino. @as-suvorov, may I know if you have an suggestions?

[sammysun0711]

I added initial hash function in test branch, while found that hash function overhead is higher than model construction method.

Loading and unpacking model from: gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf
Loading and unpacking model done. Time: 185ms
Compute GGUF hash with SHA1 done. Time: 856ms
Start generating OpenVINO model...
Save generated OpenVINO model to: model_cache/qwen2.5-0.5b-instruct-q4_0_openvino_model.xml done. Time: 567 ms
Model generation done. Time: 1702ms

Although can try to optimize hash method with parallelism, the hash function will be called for gguf load, which is not idea if we would like to skip gguf model load & unpack process if OV model existed in model cache.

I would like to suggest to serialize OV model in cache_dir when user set cache_dir property explicitly, and add verbose that existed cache OV model will skip GGUF->OV conversion w/o cache invalidate. User may need to clean up model cache and re-generate OV model if gguf model updated.

We can add hash function check after compute_hash.hpp expose as public API for re-use.

[as-suvorov]

I added initial hash function in test branch, while found that hash function overhead is higher than model construction method.

Loading and unpacking model from: gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf
Loading and unpacking model done. Time: 185ms
Compute GGUF hash with SHA1 done. Time: 856ms
Start generating OpenVINO model...
Save generated OpenVINO model to: model_cache/qwen2.5-0.5b-instruct-q4_0_openvino_model.xml done. Time: 567 ms
Model generation done. Time: 1702ms

Although can try to optimize hash method with parallelism, the hash function will be called for gguf load, which is not idea if we would like to skip gguf model load & unpack process if OV model existed in model cache.

I would like to suggest to serialize OV model in cache_dir when user set cache_dir property explicitly, and add verbose that existed cache OV model will skip GGUF->OV conversion w/o cache invalidate. User may need to clean up model cache and re-generate OV model if gguf model updated.

We can add hash function check after compute_hash.hpp expose as public API for re-use.

@Wovchena What do you think?

[rkazants]

need review again

[sammysun0711]

need review again

Refactor test and all tests passed, please kindly review again, thanks!