[Misc] Replace flaky image urls in pixtral test #22574

Isotr0py · 2025-08-09T16:34:08Z

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

The connection to picsum.photos on CI is too flaky, I uploaded them to HF to avoid annoying failed downloading:

RuntimeError: Error downloading the image from https://picsum.photos/id/27/500/500: 525 Server Error: <none> for url: https://picsum.photos/id/27/500/500.

Test Plan

Test Result

(Optional) Documentation Update

Signed-off-by: Isotr0py <[email protected]>

github-actions · 2025-08-09T16:34:16Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Code Review

This pull request aims to improve the reliability of the pixtral test by replacing image URLs from picsum.photos with more stable ones hosted on Hugging Face. While this is a positive change for CI stability, it introduces a potential issue with the existing test fixtures. The test_chat test relies on fixture files with pre-computed logprobs, which are sensitive to the exact pixel data of the input images. Any change in the images, including re-encoding, will likely cause these tests to fail. The fixture files may need to be regenerated and included in this PR.

gemini-code-assist · 2025-08-09T16:35:04Z

tests/models/multimodal/generation/test_pixtral.py

+    "https://huggingface.co/datasets/Isotr0py/mistral-test-images/resolve/main/237-400x300.jpg",
+    "https://huggingface.co/datasets/Isotr0py/mistral-test-images/resolve/main/231-200x300.jpg",
+    "https://huggingface.co/datasets/Isotr0py/mistral-test-images/resolve/main/27-500x500.jpg",
+    "https://huggingface.co/datasets/Isotr0py/mistral-test-images/resolve/main/17-150x600.jpg",


Replacing the flaky picsum.photos URLs is a good step for test reliability. However, this change will likely break the test_chat test, which validates model outputs against fixture files with pre-computed logprobs (pixtral_chat.json and mistral_small_3_chat.json).

Unless the new images are byte-for-byte identical to the originals, the model's output will differ, causing the logprob comparison to fail. The corresponding fixture files must be regenerated using the new images and included in this pull request to ensure the tests pass.

Have verified that test_chat test can still pass locally after replacing image urls.

DarkLight1337

I think we need to regenerate the expected results which are stored under FIXTURE_LOGPROBS_CHAT

Isotr0py · 2025-08-09T16:44:23Z

Hmmm, they are still same images, so I think the results are still same unless these images lost quality during transferring. Anyway, let me double check it locally tomorrow.

DarkLight1337

Let's also defer the _create_engine_inputs calls. I just realized the tests in this file are not actually being run in CI, so it makes no sense for the CI to be broken because of it

Signed-off-by: Isotr0py <[email protected]>

Isotr0py · 2025-08-10T06:42:11Z

tests/models/multimodal/generation/test_pixtral.py

                         name_1="output")


-@large_gpu_test(min_gb=48)


There is no need to load the complete model for this test, because we only compare the Placeholder. We can use a quite samll dummy model to run this test on CI.

Isotr0py · 2025-08-10T06:42:59Z

tests/models/multimodal/generation/test_pixtral.py

+    "https://huggingface.co/datasets/Isotr0py/mistral-test-images/resolve/main/237-400x300.jpg",
+    "https://huggingface.co/datasets/Isotr0py/mistral-test-images/resolve/main/231-200x300.jpg",
+    "https://huggingface.co/datasets/Isotr0py/mistral-test-images/resolve/main/27-500x500.jpg",
+    "https://huggingface.co/datasets/Isotr0py/mistral-test-images/resolve/main/17-150x600.jpg",


Have verified that test_chat test can still pass locally after replacing image urls.

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Paul Pak <[email protected]>

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Diego-Castan <[email protected]>

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Isotr0py <[email protected]>

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Xiao Yu <[email protected]>

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Isotr0py <[email protected]>

replace flaky images in mistral test

76a8b9f

Signed-off-by: Isotr0py <[email protected]>

Isotr0py requested review from DarkLight1337 and ywang96 as code owners August 9, 2025 16:34

Isotr0py changed the title ~~[Misc] Replace flaky images in pixtral test~~ [Misc] Replace flaky image urls in pixtral test Aug 9, 2025

mergify bot added the multi-modality Related to multi-modality (#4194) label Aug 9, 2025

gemini-code-assist bot reviewed Aug 9, 2025

View reviewed changes

DarkLight1337 reviewed Aug 9, 2025

View reviewed changes

reduce test vram usage

4368a33

Signed-off-by: Isotr0py <[email protected]>

Isotr0py commented Aug 10, 2025

View reviewed changes

DarkLight1337 approved these changes Aug 10, 2025

View reviewed changes

DarkLight1337 enabled auto-merge (squash) August 10, 2025 06:47

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 10, 2025

vllm-bot merged commit 049c245 into vllm-project:main Aug 10, 2025
21 of 29 checks passed

Isotr0py deleted the flaky-images branch August 10, 2025 16:26

yiliu30 pushed a commit to yiliu30/vllm-fork that referenced this pull request Aug 19, 2025

[Misc] Replace flaky image urls in pixtral test (vllm-project#22574)

a8c79ba

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Isotr0py <[email protected]>

epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025

[Misc] Replace flaky image urls in pixtral test (vllm-project#22574)

3aa60ac

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Isotr0py <[email protected]>

zhewenl pushed a commit to zhewenl/vllm that referenced this pull request Aug 28, 2025

[Misc] Replace flaky image urls in pixtral test (vllm-project#22574)

21dc3b2

Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Isotr0py <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Misc] Replace flaky image urls in pixtral test #22574

[Misc] Replace flaky image urls in pixtral test #22574

Uh oh!

Isotr0py commented Aug 9, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Aug 9, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Aug 9, 2025

Uh oh!

Isotr0py Aug 10, 2025

Uh oh!

DarkLight1337 left a comment

Uh oh!

Isotr0py commented Aug 9, 2025

Uh oh!

DarkLight1337 left a comment

Uh oh!

Isotr0py Aug 10, 2025

Uh oh!

Isotr0py Aug 10, 2025

Uh oh!

Uh oh!

Uh oh!

		name_1="output")


		@large_gpu_test(min_gb=48)

Uh oh!

[Misc] Replace flaky image urls in pixtral test #22574

[Misc] Replace flaky image urls in pixtral test #22574

Uh oh!

Conversation

Isotr0py commented Aug 9, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Essential Elements of an Effective PR Description Checklist

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

github-actions bot commented Aug 9, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Aug 9, 2025

Choose a reason for hiding this comment

Uh oh!

Isotr0py Aug 10, 2025

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Isotr0py commented Aug 9, 2025

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Isotr0py Aug 10, 2025

Choose a reason for hiding this comment

Uh oh!

Isotr0py Aug 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Isotr0py commented Aug 9, 2025 •

edited by github-actions bot

Loading