Errors when importing a library that also uses Loguru

[kace]

Hi Delgan,

I've run into issues integrating with another lib that recently moved their logging to Loguru from the stdlib. For reference I'm referring to the napalm dependency ciscoconfparse. I've linked their commits below for tag v1.5.48 for you reference.

Anyways, the ciscoconfparse dev's seem to configure the Loguru logger quite a bit - they remove all of the handlers and add their own. This results in errors on my side. First they are BlockIOError's then if I let it go a a bit longer also include OSError's - see below. It looks like my logging configuration is removed by the logger.remove() call in mpenning/ciscoconfparse@72dcc02 on line 138 after I import napalm and ciscoconfparse by extension.

I know that you probably can't help me too much since it's another lib causing me problems, but I was curious if you could offer me some advice on how to handle conflicting Loguru logger config given that there is only one global logger? I unsuccessfully tried isolating a copy of the logger as described in https://loguru.readthedocs.io/en/stable/resources/recipes.html#creating-independent-loggers-with-separate-set-of-handlers.

I've also created mpenning/ciscoconfparse#211 to see if the lib dev's have any ideas.

As always thanks for all your effort on Loguru!

--- Logging error in Loguru Handler #2 ---
Traceback (most recent call last):
  File "/home/<user>/.local/share/virtualenvs/test_deps-m_cu5jxE/lib/python3.8/site-packages/loguru/_handler.py", line 270, in _queued_writer
    message = queue.get()
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 356, in get
    res = self._reader.recv_bytes()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
BlockingIOError: [Errno 11] Resource temporarily unavailable
--- End of logging error ---

--- Logging error in Loguru Handler #2 ---
Record was: None
    self._send(header + buf)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 368, in _send
Traceback (most recent call last):
  File "/home/ksiman/.local/share/virtualenvs/test_deps-m_cu5jxE/lib/python3.8/site-packages/loguru/_handler.py", line 270, in _queued_writer
    message = queue.get()
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 356, in get
    res = self._reader.recv_bytes()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
    n = write(self._handle, buf)
BlockingIOError: [Errno 11] Resource temporarily unavailable
OSError: [Errno 9] Bad file descriptor
--- End of logging error ---

Relevant ciscoconfparse commits

• mpenning/ciscoconfparse@72dcc02
• mpenning/ciscoconfparse@0612d33

[mpenning]

Speaking only for ciscoconfparse, this problem has been sufficiently addressed by the following commit:

• mpenning/ciscoconfparse@dd0bd46

See ciscoconfparse issue #211 for usage details.

I suspect we can close this loguru ticket, but I'll let @kace confirm...

[kace]

Thanks @mpenning!

I still would like to follow up on this topic with regards to Loguru though. It would be great if I didn't have to ask library developers who also use Loguru for a method to disable their logger.

[tlambert03]

I'm also curious to hear what the "official" loguru stance is on this. From #553, it looks like the basic sentiment is that:

Packages shouldn't touch the logger handlers, it's the user responsibility to configure them.

which, correct me if I'm wrong, leads me to believe that packages & libraries that wish use loguru are sort of at the mercy of the rest of the python ecosystem – and whatever other libraries may be imported at runtime?

Is that a correct interpretation of the state of multi-package loguru usage? Or are there any "tricks" that a library could use to protect themselves from a "rogue" library (who might not have read these posts and didn't realize they shouldn't touch the handlers) that calls logger.remove() ... perhaps because they stumbled on #138, tried the suggestion in #138 (comment) ... realized it worked for their use case and stopped there.

thanks for your thoughts @Delgan!

[Delgan]

I apologize for overlooking this ticket for so long. I'll try to clarify for anyone who comes across this issue.

Regarding the concerns raised by @kace and @tlambert03, it's indeed critical for library authors to follow the recommended best practices and not add or remove any handlers during import. At the very least, authors should provide an API to add and/or remove their custom handler.

Unfortunately, I don't think it's possible to easily circumvent such misuse of Loguru, given the design of the unique logger. Modifying the logging configuration should be considered a bug in the third-party library doing so.

I'm actually curious about why library authors would want to add any handler. If anyone knows, explanations could help me understand the use cases. From my point of view, using the logging methods from the logger should suffice, configuring the handlers only makes sense in end-user code. The user will see the logs through the default handler (sys.stderr) if logger.disable() isn't called anyway.

This is consistent with the standard logging library, which strongly discourages library authors from adding handlers to loggers.

I know that you probably can't help me too much since it's another lib causing me problems, but I was curious if you could offer me some advice on how to handle conflicting Loguru logger config given that there is only one global logger? I unsuccessfully tried isolating a copy of the logger as described in https://loguru.readthedocs.io/en/stable/resources/recipes.html#creating-independent-loggers-with-separate-set-of-handlers.

I don't know the exact problem you faced, but I'm planning to create a logger.new() method to replace the recipe you linked.
This method will allow the creation of a different and independent logger instance. However, I'm not sure it will be of much use in solving this issue. Library authors could use it to completely dissociate their logs from the end-user application, but they need to be aware of the best practices in the first place.

The only workaround I can think of is to first import the misbehaving library, then re-configure the logger as expected.

I'm also curious to hear what the "official" loguru stance is on this. From #553, it looks like the basic sentiment is that:

Packages shouldn't touch the logger handlers, it's the user responsibility to configure them.

which, correct me if I'm wrong, leads me to believe that packages & libraries that wish use loguru are sort of at the mercy of the rest of the python ecosystem – and whatever other libraries may be imported at runtime?

Is that a correct interpretation of the state of multi-package loguru usage? Or are there any "tricks" that a library could use to protect themselves from a "rogue" library (who might not have read these posts and didn't realize they shouldn't touch the handlers) that calls logger.remove() ... perhaps because they stumbled on #138, tried the suggestion in #138 (comment) ... realized it worked for their use case and stopped there.

Well, Loguru does indeed expect users to follow the guidelines, especially library authors. A library that behaves incorrectly will not integrate properly with others.

If you are the author of a library, there's not much you can do about it. If your users depend on your library but then import a faulty library, they'll encounter problems using Loguru, but technically, that's not your concern.

If you're the author of a library that depends on another library that's at fault, it's more complicated. Perhaps it's possible to create a mock of the logger for the time of the import so that the real logger isn't touched.

If you are the author of an application that depends on a faulty library, you can try importing it before reconfiguring your logger.

In any case, it's best to report the problem to the library author so that it can be fixed at the root. I can't think of a better solution.