[generate] PromptLookupCandidateGenerator won't generate forbidden tokens
#40726
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| "qwen2_5omni", # the file is named `qwen2_5_omni`, but the model class is `Qwen2_5Omni`, | ||
| # All models below: shouldn't suggest audio tokens. Can be fixed by passing `suppress_ids` to candidate generator: @joaa @raushan | ||
| "voxtral", | ||
| "qwen2audio", |
|
@zucchini-nlp great comment. We can indeed make it work with ANY logits processor that blocks tokens, not just The logic is slightly more complex, but it is added in the latest commit. In a nutshell, we simulate using the logits processor with fake input logits, using the same processors as we use for the main model, and any selected token with Regarding long-term maintenance: if we're keeping assisted generation (we are for now), we're also keeping this one. This is the candidate-based generation strategy with the fewest requirements. |
gante
changed the title
PromptLookupCandidateGenerator accepts bad_words_idsPromptLookupCandidateGenerator won't generate forbidden tokens
| max_matching_ngram_size (`int`): | ||
| The maximum ngram size to be considered for matching in the prompt | ||
| num_output_tokens (`int`): | ||
| eos_token_id (`torch.Tensor`, *optional*): |
There was a problem hiding this comment.
(docstring args were out of order, and some were missing)
| if self.logits_processor is not None: | ||
| sequence_with_candidate = input_ids | ||
| for candidate_idx, new_candidate_token in enumerate(chosen_ids): | ||
| fake_input_logits = torch.ones((bsz, self.vocab_size), device=input_ids.device) |
There was a problem hiding this comment.
prob we can create a fake logit once, since it's always same. Can be helpful with models like gemma with huuge vocab size
There was a problem hiding this comment.
I thought so at first, but then considered that in-place ops (in custom logits processors) might behave poorly. Double-checked it now, it seems resilient to in-place ops -> changed it.
|
[For maintainers] Suggested jobs to run (before merge) run-slow: idefics2, idefics3, qwen2_5_vl, smolvlm |
What does this PR do?
See title.
In the process, deflakes a lot of tests :) (It was generating forbidden tokens in tests, like image tokens in VLMs -- now we can specify which sequences are forbidden)
py.test tests/models/voxtral/test_modeling_voxtral.py -k test_prompt_lookup_decoding_matches_greedy_search --flake-finder --flake-runs 1000now runs with without problems.