optionally skip logging when rank >0

[wyli]

Is your feature request related to a problem? Please describe.
it's useful to remove the logging handlers when rank > 0, example code:

import logging
import torch.distributed as dist
from monai.apps.auto3dseg.auto_runner import logger


def mute_logger(_logger):
    if _logger.hasHandlers():
        for h in list(_logger.handlers):
            _logger.removeHandler(h)
    _logger.addHandler(logging.NullHandler())


logger.warning("logging start")  # example of logging (all ranks)
if dist.is_available() and dist.is_initialized() and dist.get_rank() > 0:
    mute_logger(logger)
logger.warning("logging end")  # example of logging (only rank 0)

would be great to make it a generic utility in monai:

• adds the capability of 'unmute'
• adds the support of modifying any monai logger -- logging.getLogger("name")
• when starting a logger, provides an option of rank0 only?

[mingxin-zheng]

Hi @wyli @myron @Nic-Ma , I was thinking of an implementation of this feature, and wondering if it would be good to use the filterer to meet the needs in the ticket

import logging
import torch
import torch.distributed as dist

class RankFilter(logging.Filter):
    def __init__(self, func=None):
        """default filter is to log only if rank == 0, but users can write their lamda function for filtering"""
        super().__init__()
        self.func = func if func else lambda rank: rank == 0
        if dist.is_available() and dist.is_initialized():
            self.rank = dist.get_rank()
        else:
            raise ValueError("torch.distributed is not available")

    def filter(self, record):
        if self.func(self.rank):
            return True
        else:
            return False

logger = logging.getLogger("auto3dseg")

if torch.cuda.device_count() > 1:
    dist.init_process_group(backend="nccl", init_method="env://")
    world_size = dist.get_world_size()
    rank_filter = RankFilter()
    logger.addFilter(rank_filter)
else:
    world_size = 1


logger.warning("warning")  # will only log when rank==0 with the default filter

If this looks good to you, I will move on and write a PR.

[mingxin-zheng]

The idea is the same as the original post. Instead of removing all the log handlers, we can add a filter to stop the flow as stated in:

https://docs.python.org/3.10/howto/logging.html#logging-flow

[wyli]

thank, that's better than the method of changing handlers.. please feel free to create a PR, we can discuss the details and merge.

[myron]

apologies for late comments, after merge or #6243

but it seems we don't need this class RankFilter. All we need is to allow a user to enable or disable logging for a given process (in distributed). I don't think the torch.distributed logic should be inside of our logger methods.
It's better for a user to manually check distributed, get rank, and then enable or disable logging manually.

After reading about python logging, it seems this functionality already exists , by doing either:

1. logger.propagate = False
   https://docs.python.org/3/library/logging.html#logging.Logger

or

2. logging.disable(level=...)
   https://docs.python.org/3/library/logging.html#logging.disable
   or

3. simple callable filter function (not class)
   https://docs.python.org/3/library/logging.html#logging.Filter.filter
   e.g. logger.addFilter(lambda x: False)

I think this is perfectly sufficient, and we don't need to maintain an extra class for RankFilter() .
@mingxin-zheng @wyli what do you think?

[mingxin-zheng]

Hi @myron , I did not fully get the points. Maybe a snippet of using one of the methods you mentioned can help me understand the use case better. What confuses me is that how the users are supposed to change the logging method outside of the train.py when they are using Auto3DSeg. Thanks!

[wyli]

@myron we also used it in the dictconfig based approach for setting up the logging https://gist.github.com/wyli/a63895305b86163cb1929bb687f7488b

[myron]

Hi @myron , I did not fully get the points. Maybe a snippet of using one of the methods you mentioned can help me understand the use case better. What confuses me is that how the users are supposed to change the logging method outside of the train.py when they are using Auto3DSeg. Thanks!

Sorry for the delay, but I just tried a very simple approach and it works fine, e.g. a pseudocode in multi-gpu DDP:

logger = ....
if distributed and global_rank!=0:
     logger.addFilter(lambda x: False)

this disables logging on all processes, except the rank 0. and this way the logger/filter itself does not need to call any DDP methods to filter anything.

this is also more flexible, if e.g. we want to keep logging for all local ranks=0 on multinode (not just global rank 0), we can easily achieve that.

Since we already have RankFilter, and it does the same thing, I guess it doesn't matter what to use. but it seems safer not to call DDP method from the filter.