[CI/Build] Disable flaky test_structured_output tests #24404
Conversation
Signed-off-by: 22quinn <[email protected]>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request disables several flaky tests in test_structured_output to improve CI stability, which is a reasonable temporary measure. The associated tracking issue is correctly referenced. However, I've identified a latent bug in one of the commented-out test configurations that should be fixed to prevent future errors.
| #FIXME: This tests are flaky on CI thus disabled. Tracking in Issue #24402 | ||
| # ("mistralai/Ministral-8B-Instruct-2410", "outlines", "auto", None), | ||
| # ("mistralai/Ministral-8B-Instruct-2410", "outlines", "mistral", None), | ||
| #("Qwen/Qwen2.5-1.5B-Instruct", "guidance", "auto"), |
There was a problem hiding this comment.
This commented-out test case tuple is malformed. It has 3 elements, but the pytest.mark.parametrize for test_structured_output expects 4 elements (model_name, guided_decoding_backend, tokenizer_mode, speculative_config). If this test is uncommented in the future, it will cause a ValueError during test collection. Please add None as the fourth element to represent the speculative_config.
| #("Qwen/Qwen2.5-1.5B-Instruct", "guidance", "auto"), | |
| #("Qwen/Qwen2.5-1.5B-Instruct", "guidance", "auto", None), |
22quinn
added
ready
…4404) Signed-off-by: 22quinn <[email protected]> Signed-off-by: JasonZhu1313 <[email protected]>
…4404) Signed-off-by: 22quinn <[email protected]>
…4404) Signed-off-by: 22quinn <[email protected]>
…4404) Signed-off-by: 22quinn <[email protected]> Signed-off-by: LopezCastroRoberto <[email protected]>
…4404) Signed-off-by: 22quinn <[email protected]>
Purpose
Mitigate #24402
Test Plan
CI
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.