[CI Failure]: Flaky test_structured_output tests

[22quinn]

Name of failing test

v1/entrypoints/llm/test_struct_output_generate.py::test_structured_output[mistralai/Ministral-8B-Instruct-2410-outlines-auto/mistral-None]

Basic information

• Flaky test
• Can reproduce locally
• Caused by external libraries (e.g. bug in transformers)

🧪 Describe the failing test

Output 44.44.44.44 not matching IP regex

📝 History of failing test

Nightly: https://buildkite.com/vllm/ci/builds/29742/steps/canvas?jid=019926a3-d31d-4c97-a348-1682c23ea20a
PR: https://buildkite.com/vllm/ci/builds/29732/steps/canvas?jid=0199253a-e2e3-4614-83ff-26c31f81fe8d

CC List.

@aarnphm @ywang96 @DarkLight1337

[njhill]

Thanks @22quinn for this.

However I'm worried about just disabling tests that start to fail, it should only be a last resort. This test was failing pretty consistently for me (are you sure it's flaky and not just broken?) but only started to do so sometime in the last week. We should find the root cause / pinpoint the commit where it started to happen.

[chaunceyjiang]

@njhill @22quinn I'm already investigating the root cause of this issue today. please assign it to me.

[22quinn]

are you sure it's flaky and not just broken?

I think it was flaky. They were able to pass a few times in my local environment

[njhill]

This one seems to be failing too:

[2025-09-08T16:48:39Z] FAILED v1/entrypoints/llm/test_struct_output_generate.py::test_structured_output[Qwen/Qwen2.5-1.5B-Instruct-lm-format-enforcer-auto-None] - json.decoder.JSONDecodeError: Invalid control character at: line 7 column 67 (char 80)

#24316

[aarnphm]

@njhill do you happen to know about the GPU we are using on CI?

[njhill]

@aarnphm sorry I don't know...

[chaunceyjiang]

It’s strange, I just can’t reproduce it locally no matter what.

[njhill]

I am guessing it's precision related. We should run this test using higher precision or last resort could add targeted retry using pytest.mark.flaky