Skip to content

Conversation

atone
Copy link
Contributor

@atone atone commented Mar 18, 2025

This PR adds support for Tele-FLM Model.

Tele-FLM (aka FLM-2) is a 52B open-sourced multilingual large language model that features a stable, efficient pre-training paradigm and enhanced factual judgement capabilities. Built upon the decoder-only transformer architecture, it has been trained on approximately 2T tokens. Tele-FLM demonstrates superior performances at its scale, and sometimes surpass larger models. In addition to sharing the model weights, we provide the core designs, engineering practices, and training details, anticipating their benefits for both academic and industrial communities.

The model collection can be found at Hugging Face (https://huggingface.co/collections/CofeAI/tele-flm-flm-2-669e4dbd2dbf53ccd2454304).

Veried that vllm serve CofeAI/FLM-2-52B-Instruct-2407 --trust-remote-code --chat-template /examples/tool_chat_template_teleflm.jinja -tp 2 works.

Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added the documentation Improvements or additions to documentation label Mar 18, 2025
@atone atone force-pushed the flm-0.7.3 branch 2 times, most recently from 5b956eb to 5f7f267 Compare March 18, 2025 10:39
@atone atone changed the title teleflm supported [Model] Add Tele-FLM Model Support Mar 18, 2025
@atone atone changed the title [Model] Add Tele-FLM Model Support [Model] Support Tele-FLM Model Mar 18, 2025
@atone atone changed the title [Model] Support Tele-FLM Model [Model] Support FLM-2 Model Mar 18, 2025
@atone atone changed the title [Model] Support FLM-2 Model [Model] Support Tele-FLM Model Mar 18, 2025
@jeejeelee jeejeelee requested a review from DarkLight1337 March 18, 2025 11:49
@atone atone marked this pull request as ready for review March 18, 2025 12:14
Comment on lines 283 to 286
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems that here is the only difference with Llama, can you refactor the model implementation to reduce duplicate code just like glm.py and telechat2.py?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is really necessary to port this config? I think custom config from hf dynamic module can work in most of cases.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: The supported_lora_modules var has been deprecated, plz remove it

@jeejeelee
Copy link
Collaborator

Hi, remember to update the documentation, see: https://github.com/vllm-project/vllm/blob/main/docs/source/models/supported_models.md

Jason3900 and others added 14 commits March 20, 2025 05:33
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Signed-off-by: jiangxin <[email protected]>
@Isotr0py
Copy link
Member

Don't forget updating tests/models/registry.py, otherwise test_initialization will fail! :)

Signed-off-by: jiangxin <[email protected]>
Signed-off-by: jiangxin <[email protected]>
@Jason3900 Jason3900 requested a review from ywang96 as a code owner March 20, 2025 09:31
Signed-off-by: jiangxin <[email protected]>
@atone atone requested a review from Isotr0py March 20, 2025 11:33
@horizon94
Copy link
Contributor

All the above suggestions have been adopted, and corresponding adjustments have been made:

  1. The code now inherits from the llama class to minimize redundancy.
  2. Updated tests/models/registry.py.
  3. Updated the documentation.

The CI failure occurred in the Entrypoints Test, but this error appears unrelated to our code modifications.
Any suggestion? :)

@Isotr0py Isotr0py added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 21, 2025
@DarkLight1337
Copy link
Member

Can you merge from main to fix the CI failure?

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) March 22, 2025 05:44
@DarkLight1337
Copy link
Member

CI failures look unrelated, merging. Thanks for your effort!

@vllm-bot vllm-bot merged commit 2f4bd35 into vllm-project:main Mar 22, 2025
28 of 36 checks passed
erictang000 pushed a commit to erictang000/vllm that referenced this pull request Mar 25, 2025
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Co-authored-by: Jason Fang <[email protected]>
Co-authored-by: jiangxin <[email protected]>
lulmer pushed a commit to lulmer/vllm that referenced this pull request Apr 7, 2025
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Co-authored-by: Jason Fang <[email protected]>
Co-authored-by: jiangxin <[email protected]>
Signed-off-by: Louis Ulmer <[email protected]>
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Apr 29, 2025
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Co-authored-by: Jason Fang <[email protected]>
Co-authored-by: jiangxin <[email protected]>
shreyankg pushed a commit to shreyankg/vllm that referenced this pull request May 3, 2025
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Co-authored-by: Jason Fang <[email protected]>
Co-authored-by: jiangxin <[email protected]>
RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025
Signed-off-by: Naitong Yu <[email protected]>
Signed-off-by: jiangxin <[email protected]>
Co-authored-by: Jason Fang <[email protected]>
Co-authored-by: jiangxin <[email protected]>
Signed-off-by: Mu Huai <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants