[CI] Optimize entrypoints API server tests #23896

csahithi · 2025-08-29T03:49:37Z

Purpose

Reduce CI time for entrypoint tests by creating shared server for grouped tests
Removed v0 references in entrypoint tests
Replaced large models with smaller ones - hmellor/tiny-random-LlamaForCausalLM, microsoft/DialoGPT-small

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

robertgshaw2-redhat · 2025-08-29T03:56:31Z

wow! great job!

tests/entrypoints/openai/embedding_tests/test_encoder_decoder.py

tests/entrypoints/openai/embedding_tests/test_optional_middleware.py

tests/entrypoints/openai/individual_tests/test_metrics.py

robertgshaw2-redhat · 2025-08-29T04:05:44Z

tests/entrypoints/openai/test_chat_with_tool_reasoning.py

+        "--reasoning-parser", "deepseek_r1", 
+        "--enable-auto-tool-choice", 
+        "--tool-call-parser", "hermes",
+        "--disable-log-stats",


you can remove --disable-log-stats and --disable-log-requests

tests/entrypoints/openai/multimodal_tests/conftest.py

mergify · 2025-08-29T16:04:41Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @csahithi.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

njhill

Thanks @csahithi this is great!!

Replaced large models with smaller ones - hmellor/tiny-random-LlamaForCausalLM, microsoft/DialoGPT-small

~~Is the reason for the latter that the former doesn't have a chat template?~~
~~If so we can just ask @hmellor to add the llama 3.2 chat template and replace them all with that.~~

Oh sorry I see that it does already have a chat template. Then I'm curious what's the reason for using microsoft/DialoGPT-small too?

I know you have ideas for possible further streamlining but in the interests of incremental improvement could we get this merged first?

Could you fix the merge conflicts and we can see what the new CI timings are like after that too.

hmellor · 2025-08-30T15:31:36Z

If anything needs changing about hmellor/tiny-random-LlamaForCausalLM to make it more useful for our tests do let me know, vLLM testing is what I made it for and it's easy to update!

njhill · 2025-09-05T20:06:16Z

CI failures look related

mergify · 2025-09-07T16:39:48Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @csahithi.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Sahithi Chigurupati <[email protected]>

hmellor · 2025-09-08T09:56:35Z

tests/entrypoints/openai/basic_tests/test_basic.py

-
-    with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
-        yield remote_server
+MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM"


Could this also be made a fixture so that we don't have to keep anything in sync with conftest?

hmellor · 2025-09-08T09:57:03Z

tests/entrypoints/openai/basic_tests/test_chat_echo.py

-
-    with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
-        yield remote_server
+MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM"


Could this also be made a fixture so that we don't have to keep anything in sync with conftest?

hmellor · 2025-09-08T09:57:14Z

tests/entrypoints/openai/basic_tests/test_chat_logit_bias_validation.py

-from ...utils import RemoteOpenAIServer
-
-MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct"
+MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM"


Could this also be made a fixture so that we don't have to keep anything in sync with conftest?

hmellor · 2025-09-08T09:57:39Z

tests/entrypoints/openai/basic_tests/test_return_token_ids.py

-    ]
-    with RemoteOpenAIServer(MODEL_NAME, args) as remote_server:
-        yield remote_server
+MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM"


Could this also be made a fixture so that we don't have to keep anything in sync with conftest?

hmellor

To make paths shorter should we rename the following directories:

basic_tests -> basic
correctness (unchanged)
embedding_tests -> embedding
individual_tests -> individual
lora_tests -> lora
multimodal_tests -> multimodal

hmellor · 2025-09-08T10:02:52Z

tests/entrypoints/openai/embedding_tests/test_optional_middleware.py


 # Use a small embeddings model for faster startup and smaller memory footprint.
 # Since we are not testing any chat functionality,
 # using a chat capable model is overkill.
-MODEL_NAME = "intfloat/multilingual-e5-small"
+MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM"


Could this also be made a fixture so that we don't have to keep anything in sync with conftest?

hmellor · 2025-09-08T10:06:22Z

tests/entrypoints/openai/multimodal_tests/test_video.py

The video_server fixture is not used here, is that intentional?

hmellor · 2025-09-08T10:07:16Z

tests/entrypoints/openai/multimodal_tests/test_vision.py

The vision_server fixture is not used here, is that intentional?

hmellor · 2025-09-08T10:07:20Z

tests/entrypoints/openai/multimodal_tests/test_vision_embedding.py

The vision_server fixture is not used here, is that intentional?

mergify · 2025-09-08T13:56:43Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @csahithi.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

njhill · 2025-09-08T15:45:07Z

@csahithi is out this week, we'll see if someone else can take this over

njhill · 2025-09-11T16:55:19Z

@debroy-rh has offered to work on this

mergify bot added ci/build tool-calling labels Aug 29, 2025

github-project-automation bot added this to Tool Calling Aug 29, 2025

robertgshaw2-redhat reviewed Aug 29, 2025

View reviewed changes

tests/entrypoints/openai/embedding_tests/test_encoder_decoder.py Outdated Show resolved Hide resolved

robertgshaw2-redhat reviewed Aug 29, 2025

View reviewed changes

tests/entrypoints/openai/embedding_tests/test_optional_middleware.py Outdated Show resolved Hide resolved

robertgshaw2-redhat reviewed Aug 29, 2025

View reviewed changes

tests/entrypoints/openai/individual_tests/test_metrics.py Show resolved Hide resolved

robertgshaw2-redhat reviewed Aug 29, 2025

View reviewed changes

tests/entrypoints/openai/multimodal_tests/conftest.py Show resolved Hide resolved

csahithi force-pushed the entrypoint-tests-optimize branch from 68a2e19 to 1b41751 Compare August 29, 2025 04:23

csahithi mentioned this pull request Aug 29, 2025

[CI]: Entrypoints tests cleanup #23667

Open

mergify bot added the needs-rebase label Aug 29, 2025

njhill self-requested a review August 29, 2025 16:04

csahithi marked this pull request as ready for review August 29, 2025 16:07

csahithi requested review from DarkLight1337, simon-mo and aarnphm as code owners August 29, 2025 16:07

njhill changed the title ~~Optimize entrypoints API server tests~~ [CI] Optimize entrypoints API server tests Aug 29, 2025

njhill reviewed Aug 29, 2025

View reviewed changes

njhill mentioned this pull request Sep 2, 2025

[CI] Replace large models with tiny alternatives in tests #24057

Open

csahithi force-pushed the entrypoint-tests-optimize branch from 4732592 to e799966 Compare September 2, 2025 23:25

mergify bot removed the needs-rebase label Sep 2, 2025

csahithi force-pushed the entrypoint-tests-optimize branch 2 times, most recently from aac1623 to fba4775 Compare September 5, 2025 13:33

njhill mentioned this pull request Sep 5, 2025

[CI] Add timeouts to tests #24260

Merged

mergify bot added the needs-rebase label Sep 7, 2025

csahithi added 2 commits September 7, 2025 18:28

Optimize entrypoints API server tests

901c744

Signed-off-by: Sahithi Chigurupati <[email protected]>

Remove unnecessary flags and tests

4b9c4fe

Signed-off-by: Sahithi Chigurupati <[email protected]>

csahithi force-pushed the entrypoint-tests-optimize branch from fba4775 to bedeffd Compare September 8, 2025 01:29

mergify bot removed the needs-rebase label Sep 8, 2025

Fix basic tests

d43d48b

Signed-off-by: Sahithi Chigurupati <[email protected]>

csahithi force-pushed the entrypoint-tests-optimize branch from bedeffd to d43d48b Compare September 8, 2025 01:40

hmellor reviewed Sep 8, 2025

View reviewed changes

mergify bot added the needs-rebase label Sep 8, 2025

Uh oh!

[CI] Optimize entrypoints API server tests #23896

Are you sure you want to change the base?

[CI] Optimize entrypoints API server tests #23896

Uh oh!

Conversation

csahithi commented Aug 29, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

robertgshaw2-redhat commented Aug 29, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mergify bot commented Aug 29, 2025

Uh oh!

njhill left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hmellor commented Aug 30, 2025

Uh oh!

njhill commented Sep 5, 2025

Uh oh!

mergify bot commented Sep 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hmellor left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Sep 8, 2025

Uh oh!

njhill commented Sep 8, 2025

Uh oh!

njhill commented Sep 11, 2025

Uh oh!

Uh oh!

csahithi commented Aug 29, 2025 •

edited by github-actions bot

Loading

njhill left a comment •

edited

Loading