Skip to content

when encoding text embeds, force the move to accelerator on round one #1431

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 24, 2025

Conversation

bghira
Copy link
Owner

@bghira bghira commented Apr 24, 2025

Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)
Traceback (most recent call last):
  File "/mnt/Linux/Code/SimpleTuner/train.py", line 50, in <module>
    trainer.init_data_backend()
  File "/mnt/Linux/Code/SimpleTuner/helpers/training/trainer.py", line 500, in init_data_backend
    raise e
  File "/mnt/Linux/Code/SimpleTuner/helpers/training/trainer.py", line 473, in init_data_backend
    configure_multi_databackend(
  File "/mnt/Linux/Code/SimpleTuner/helpers/data_backend/factory.py", line 1224, in configure_multi_databackend
    init_backend["text_embed_cache"].compute_embeddings_for_prompts(
  File "/mnt/Linux/Code/SimpleTuner/helpers/caching/text_embeds.py", line 294, in compute_embeddings_for_prompts
    output = self.compute_prompt_embeddings_with_model(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/helpers/caching/text_embeds.py", line 409, in compute_prompt_embeddings_with_model
    text_encoder_output = self.model.encode_text_batch(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/helpers/models/common.py", line 1123, in encode_text_batch
    encoded_text = self._encode_prompts(text_batch, is_negative_prompt)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/helpers/models/hidream/model.py", line 255, in _encode_prompts
    ]._encode_prompt(
      ^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/helpers/models/hidream/pipeline.py", line 752, in _encode_prompt
    t5_prompt_embeds = self._get_t5_prompt_embeds(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/helpers/models/hidream/pipeline.py", line 244, in _get_t5_prompt_embeds
    prompt_embeds = self.text_encoder_3(
                    ^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/venv/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py", line 2100, in forward
    encoder_outputs = self.encoder(
                      ^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/venv/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py", line 1009, in forward
    inputs_embeds = self.embed_tokens(input_ids)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/venv/lib/python3.12/site-packages/torch/nn/modules/sparse.py", line 190, in forward
    return F.embedding(
           ^^^^^^^^^^^^
  File "/mnt/Linux/Code/SimpleTuner/venv/lib/python3.12/site-packages/torch/nn/functional.py", line 2551, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)

(venv) rmbp@linux:/mnt/Linux/Code/SimpleTuner$ 

resolves issue reported on Discord.

@bghira bghira added bug Something isn't working regression This bug has regressed behaviour that previously worked. pending This issue has a fix that is awaiting test results. labels Apr 24, 2025
@bghira bghira self-assigned this Apr 24, 2025
@bghira bghira merged commit 5c2f906 into main Apr 24, 2025
1 check failed
@bghira bghira deleted the bugfix/text-enc-device-mismatch branch April 24, 2025 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pending This issue has a fix that is awaiting test results. regression This bug has regressed behaviour that previously worked.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant