'BertEncoder' object has no attribute 'gradient_checkpointing'

[venkatesh-kulkarni]

Who can help

@LysandreJik

Information

The model I am using is Bert.

I get an error when I call the function test().

The function definition of 'test' is as follows:

def test():
    bert.eval()
    bert_outputs = []

    with torch.no_grad():
        for unw, data in enumerate(test_loader, 0):
            ids = data['ids'].to(device, dtype = torch.long)
            mask = data['mask'].to(device, dtype = torch.long)
            token_type_ids = data['token_type_ids'].to(device, dtype = torch.long)
            targets = data['targets'].to(device, dtype = torch.float)
            outputs = bert(ids, mask, token_type_ids)

            bert_outputs.extend(torch.sigmoid(outputs).cpu().detach().numpy().tolist())

    return bert_outputs

The call log is as follows:

AttributeError                            Traceback (most recent call last)
<ipython-input-51-833f1f639ea7> in <module>()
     18   test_loader = DataLoader(test_dataset, **bert_test_params)
     19 
---> 20   test_outputs = test()
     21 
     22   test_outputs = np.array(test_outputs)

<ipython-input-50-050b63b5247c> in test()
      9             token_type_ids = data['token_type_ids'].to(device, dtype = torch.long)
     10             targets = data['targets'].to(device, dtype = torch.float)
---> 11             outputs = bert(ids, mask, token_type_ids)
     12 
     13             bert_outputs.extend(torch.sigmoid(outputs).cpu().detach().numpy().tolist())

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

<ipython-input-45-1b7273ac2d08> in forward(self, ids, mask, token_type_ids, return_dict)
      7 
      8     def forward(self, ids, mask, token_type_ids, return_dict = False):
----> 9         unw, out_1 = self.layer1(ids, attention_mask = mask, token_type_ids = token_type_ids)[0], self.layer1(ids, attention_mask = mask, token_type_ids = token_type_ids)[1]
     10         out_2 = self.layer2(out_1)
     11         out_final = self.layer3(out_2)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
   1003             output_attentions=output_attentions,
   1004             output_hidden_states=output_hidden_states,
-> 1005             return_dict=return_dict,
   1006         )
   1007         sequence_output = encoder_outputs[0]

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    557             past_key_value = past_key_values[i] if past_key_values is not None else None
    558 
--> 559             if self.gradient_checkpointing and self.training:
    560 
    561                 if use_cache:

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
   1129                 return modules[name]
   1130         raise AttributeError("'{}' object has no attribute '{}'".format(
-> 1131             type(self).__name__, name))
   1132 
   1133     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'BertEncoder' object has no attribute 'gradient_checkpointing'

I am trying to perform sentiment analysis on tweets. The same code worked well a few weeks back without giving any errors. However, now I get the error mentioned above. I tried searching online for possible fixes but could not find any for this specific problem.

[NielsRogge]

Are you using BertEncoder rather than BertModel?

[venkatesh-kulkarni]

No, I am using BertModel.
This is the definition of BERT Class.

class BERT(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = transformers.BertModel.from_pretrained('bert-base-uncased')
        self.layer2 = torch.nn.Dropout(0.3)
        self.layer3 = torch.nn.Linear(768, 11)

    def forward(self, ids, mask, token_type_ids, return_dict = False):
        unw, out_1 = self.layer1(ids, attention_mask = mask, token_type_ids = token_type_ids)[0], self.layer1(ids, attention_mask = mask, token_type_ids = token_type_ids)[1]
        out_2 = self.layer2(out_1)
        out_final = self.layer3(out_2)
        return out_final

model = BERT()
model.to(device)

[NielsRogge]

Probably related to this recent PR: #13657. Which version of Transformers are you on?

[venkatesh-kulkarni]

Thanks for sharing the PR.
I am using 4.11.3 version of transformers.

[venkatesh-kulkarni]

@NielsRogge I am actually using a pre-trained bert model for a sentiment classification task. I was not able to completely follow the discussion on the PR that you have mentioned. Do the changes mean that I have re-train my model with a new set of parameters (with regards to gradient_checkpointing)? Also, is there a simpler workaround where I can use the same pre-trained models? If yes, what is it?

Thanks for your help!

[NielsRogge]

I'll let @sgugger handle this.

[sgugger]

I am not able to reproduce the bug, @venkatesh-kulkarni you will need to give us a sample of code that fully reproduces it if you want us to help you.

[venkatesh-kulkarni]

So, here is the code that I am using. I am basically using a pre-trained bert base model fine-tuned on the SenWave dataset for multi-label sentiment classification.

MAX_LEN = 200 
TRAIN_BATCH_SIZE = 1
VALID_BATCH_SIZE = 1
EPOCHS = 4
LEARNING_RATE = 1e-05 
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
device = 'cuda' if torch.cuda.is_available() else 'cpu'

class CustomDataset(Dataset):

    def __init__(self, dataframe, tokenizer, max_len):
        self.tokenizer = tokenizer
        self.dataframe = dataframe
        self.tweet = dataframe['Tweet']
        self.targets = self.dataframe.list
        self.max_len = max_len

    def __len__(self):
        return len(self.tweet)
    
    def __getitem__(self, index):
        tweet = str(self.tweet[index])
        tweet = " ".join(tweet.split())

        inputs = self.tokenizer.encode_plus(
            tweet,
            None,
            add_special_tokens = True,
            max_length = self.max_len,
            pad_to_max_length = True,
            return_token_type_ids = True
        )
        ids = inputs['input_ids']
        mask = inputs['attention_mask']
        token_type_ids = inputs['token_type_ids']

        return {
            'ids' : torch.tensor(ids, dtype = torch.long),
            'mask' : torch.tensor(mask, dtype = torch.long),
            'token_type_ids' : torch.tensor(token_type_ids, dtype = torch.long),
            'targets' : torch.tensor(self.targets[index], dtype = torch.float)
        }

class BERT(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = transformers.BertModel.from_pretrained('bert-base-uncased')
        self.layer2 = torch.nn.Dropout(0.3)
        self.layer3 = torch.nn.Linear(768, 11)

    def forward(self, ids, mask, token_type_ids, return_dict = False):
        unw, out_1 = self.layer1(ids, attention_mask = mask, token_type_ids = token_type_ids)[0], self.layer1(ids, attention_mask = mask, token_type_ids = token_type_ids)[1]
        out_2 = self.layer2(out_1)
        out_final = self.layer3(out_2)
        return out_final

model = BERT()
model.to(device)

bert = torch.load("path to pre-trained bert")
bert

def test():
    bert.eval()
    bert_outputs = []

    with torch.no_grad():
        for unw, data in enumerate(test_loader, 0):
            ids = data['ids'].to(device, dtype = torch.long)
            mask = data['mask'].to(device, dtype = torch.long)
            token_type_ids = data['token_type_ids'].to(device, dtype = torch.long)
            targets = data['targets'].to(device, dtype = torch.float)
            outputs = bert(ids, mask, token_type_ids)

            bert_outputs.extend(torch.sigmoid(outputs).cpu().detach().numpy().tolist())

    return bert_outputs

 new_df = pd.DataFrame()

  verses_df = pd.read_csv('path to csv file containing tweets')

  new_df['Tweet'] = verses_df['verse']
  values = [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]] * len(verses_df)
  new_df['list'] = values

  test_dataset = CustomDataset(new_df, tokenizer, MAX_LEN)

  bert_test_params = {'batch_size': 1,
                    'shuffle': False,
                    'num_workers': 0
                    }   

  test_loader = DataLoader(test_dataset, **bert_test_params)

  test_outputs = test()

  test_outputs = np.array(test_outputs)

  for i in range(test_outputs.shape[0]):
      for j in range(test_outputs.shape[1]):
          if test_outputs[i][j] >= 0.5: test_outputs[i][j] = 1
          else: test_outputs[i][j] = 0

Whenever I call test(), I get the error 'BertEncoder' object has no attribute 'gradient_checkpointing'.
I hope these segments of code are enough to help you reproduce the error @sgugger .

[sgugger]

I have no access to your personal files and

bert = torch.load("path to pre-trained bert")

cannot be run on my side.

In general you should never save PyTorch models like this, just their weights. The problem that is probably happening is probably some conflict between the file you saved with a previous version of Transformers and the new one you have one your environment.

You will need to revert your Transformers install to the same version you used to save your model to be able to load it again. Then I suggest you use the PyTorch recommended way of just saving the weights of your model, then reload those weights after initializing a BERT model. This way you will stay compatible across different versions of the library.

[venkatesh-kulkarni]

Updating the tranformers version to 4.10.1 fixed it. Thanks a lot for your help @sgugger

[Jefffish09]

Updating the tranformers version to 4.10.1 fixed it. Thanks a lot for your help @sgugger

I met same error on 4.11.3, when I downgrade to 4.10.3, it became normal.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.