only show validation progress bar from main process (#4476)

* only show validation progress bar from main process

* fix

Author: epwalsh

CHANGELOG.md

@@ -34,6 +34,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fixed a bug that erronously increased last label's false positive count in calculating fbeta metrics.
 - `Tqdm` output now looks much better when the output is being piped or redirected.
 - Small improvements to how the API documentation is rendered.
+- Only show validation progress bar from main process in distributed training.
 
 ### Added
 

allennlp/training/trainer.py

@@ -707,7 +707,14 @@ def _validation_loss(self, epoch: int) -> Tuple[float, float, int]:
             )
 
         regularization_penalty = self.model.get_regularization_penalty()
-        val_generator_tqdm = Tqdm.tqdm(validation_data_loader)
+
+        # Having multiple tqdm bars in case of distributed training will be a mess. Hence only the master's
+        # progress is shown
+        if self._master:
+            val_generator_tqdm = Tqdm.tqdm(validation_data_loader)
+        else:
+            val_generator_tqdm = validation_data_loader
+
         batches_this_epoch = 0
         val_loss = 0
         if regularization_penalty is not None:
@@ -760,7 +767,8 @@ def _validation_loss(self, epoch: int) -> Tuple[float, float, int]:
             )
 
             description = training_util.description_from_metrics(val_metrics)
-            val_generator_tqdm.set_description(description, refresh=False)
+            if self._master:
+                val_generator_tqdm.set_description(description, refresh=False)
 
             for callback in self._batch_callbacks:
                 callback(