Faster label smoothing

CHANGELOG.md

@@ -29,6 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Ensured `ensure_model_can_train_save_and_load` is consistently random.
 - Fixed weight tying logic in `T5` transformer module. Previously input/output embeddings were always tied. Now this is optional,
   and the default behavior is taken from the `config.tie_word_embeddings` value when instantiating `from_pretrained_module()`.
+- Implemented slightly faster label smoothing
 - Fixed the docs for `PytorchTransformerWrapper`
 - Fixed recovering training jobs with models that expect `get_metrics()` to not be called until they have seen at least one batch.
 

allennlp/nn/util.py

@@ -822,10 +822,9 @@ def sequence_cross_entropy_with_logits(
         num_classes = logits.size(-1)
         smoothing_value = label_smoothing / num_classes
         # Fill all the correct indices with 1 - smoothing value.
-        one_hot_targets = torch.zeros_like(log_probs_flat).scatter_(
-            -1, targets_flat, 1.0 - label_smoothing
+        smoothed_targets = torch.full_like(log_probs_flat, smoothing_value).scatter_(
+            -1, targets_flat, 1.0 - label_smoothing + smoothing_value
         )
-        smoothed_targets = one_hot_targets + smoothing_value
         negative_log_likelihood_flat = -log_probs_flat * smoothed_targets
         negative_log_likelihood_flat = negative_log_likelihood_flat.sum(-1, keepdim=True)
     else: