Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #7119
Fixes #7080
Background:
When sft distilling data from deepseek-R1, we want to add two new tokens and to vocab
but after add two configs: new_special_tokens: and and resize_vocab: true
then do sft, we still can't get and in vllm reference
Cause:
1).I debugged the code and found the config new_special_tokens will call tokenizer.add_special_tokens which added new token to additional_special_tokens
2)vllm will default skip special tokens thus we can't get and in vllm reference
Fix:
In this backgroud, we should just add new tokens to vocab, not make them to be special tokens(special tokens like <|im_end|> always has special function), so just add a config: new_normal_tokens which call tokenizer.add_tokens, after that new tokens will be displayed in reference
Before submitting