weight is not tied within `init_empty_weights`

[Ghost-LZW]

System Info

- `Accelerate` version: 1.8.1
- Platform: macOS-14.3.1-arm64-arm-64bit
- `accelerate` bash location: /opt/homebrew/anaconda3/bin/accelerate
- Python version: 3.11.3
- Numpy version: 1.24.3
- PyTorch version: 2.4.1
- PyTorch accelerator: N/A
- System RAM: 32.00 GB

Information

• The official example scripts
• My own modified scripts

Tasks

• One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
• My own task or dataset (give details below)

Reproduction

from accelerate import init_empty_weights
import torch

a = torch.nn.ModuleList([torch.nn.Embedding(12, 12), torch.nn.Embedding(12, 12)])
# tie embedding
a[0].weight = a[1].weight
assert a[0].weight is a[1].weight  # success

with init_empty_weights():
    a = torch.nn.ModuleList([torch.nn.Embedding(12, 12), torch.nn.Embedding(12, 12)])
    # tie embedding
    a[0].weight = a[1].weight
    assert a[0].weight is a[1].weight  # error
"""
Traceback (most recent call last):
  File "test_empty_init.py", line 16, in <module>
    assert a[0].weight is a[1].weight
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
"""

Expected behavior

tie weights success with init_empty_weights.

This issue might come from the register_empty_parameter move the parameter to the meta device without checking whether the parameters are tied.

https://github.com/huggingface/accelerate/blob/main/src/accelerate/big_modeling.py#L129-L135

[Ghost-LZW]

There is an existing issue https://discuss.huggingface.co/t/error-the-model-weights-are-not-tied-please-use-the-tie-weights-method-before-using-the-infer-auto-device-function-even-after-adding-model-tie-weights/46325

[SunMarc]

We indeed lose tied weights link when initializing with init_empty_weights(). This is why users have to call model.tie_weights() afterwards. There were a PR that take care of that, if you want to have a look or finish it ! #2737