Hi team,

I found an unexpected behavior when using HuggingFace Transformers' tokenizer with an empty string as input. When I run the following code:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B", trust_remote_code=True)
input_ids = tokenizer("", return_tensors="pt")['input_ids']
print(input_ids, input_ids.dtype)
# output:
# tensor([], size=(1, 0)) torch.float32

Potential impact:
For example, if this float32 tensor is concatenated with other torch.long tensors using torch.cat, the result will be promoted to float32, causing all token IDs to become floats. This can break downstream code that expects integer token IDs.

Could you please take a look? Thank you!

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B", trust_remote_code=True)

input_ids = tokenizer("", return_tensors="pt")['input_ids']
print(input_ids, input_ids.dtype, sep="\n")

# output:
# tensor([], size=(1, 0))
# torch.float32

Expected behavior

The tokenizer should always return torch.long tensors for input_ids, regardless of the input content.