Skip to content

ImportError and ModuleNotFoundError While Trying to Generate HTML Documentation #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
navneethc opened this issue Dec 13, 2019 · 18 comments
Labels
invalid This is not a (valid) bug report

Comments

@navneethc
Copy link

navneethc commented Dec 13, 2019

Environment:
Windows 10
Anaconda Environment
Python 3.7.5
pdoc3 0.7.2 (installed with pip from PyPI, inside the conda env.)

I currently have a package of Python scripts with some sub-packages and modules within it. When I try to generate HTML documentation, it quickly terminates with an error.

Expected Behavior

HTML documentation is generated.

Actual Behavior

(acd) C:\Users\<username>\Documents\ACD\code>pdoc --html lcms_nmr_script
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.configuration'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.dblookup'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
Traceback (most recent call last):
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 184, in import_module
    module = importlib.import_module(module_path)
  File "c:\programdata\anaconda3\envs\acd\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\<username>\Documents\ACD\code\lcms_nmr_script\dblookup\cms.py", line 12, in <module>
    from configuration.config import get_config
ModuleNotFoundError: No module named 'configuration'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\programdata\anaconda3\envs\acd\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\envs\acd\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\envs\acd\Scripts\pdoc.exe\__main__.py", line 7, in <module>
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\cli.py", line 449, in main
    for module in args.modules]
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\cli.py", line 449, in <listcomp>
    for module in args.modules]
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 595, in __init__
    context=self._context)
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 593, in __init__
    self.doc[root] = m = Module(import_module(fullname),
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 186, in import_module
    raise ImportError('Error importing {!r}: {}'.format(module, e))
ImportError: Error importing 'lcms_nmr_script.dblookup.cms': No module named 'configuration'

Steps to Reproduce

  1. Open a terminal and activate the environment
  2. Change to directory containing the package
  3. pdoc --html <package directory>

Additional info

  • I can confirm that all those modules that are allegedly missing are, in fact, available. This code has been tested independently and works as intended otherwise.
  • pdoc version: 0.7.2
  • Please let me know if you need any other information.
@kernc
Copy link
Member

kernc commented Dec 15, 2019

I can confirm that all those modules that are allegedly missing are, in fact, available.

There is configuration.config module in current working directory where pdoc is run from?

@kernc kernc added the bug Something isn't working label Dec 15, 2019
@navneethc
Copy link
Author

Yes, .\lcms_nmr_script\configuration\config.py exists.

@kernc
Copy link
Member

kernc commented Dec 15, 2019

Hmmm, lcms_nmr_script.dblookup.cms looks for it in the current directory. It should be ./configuration/config.py. Alternatively, lcms_nmr_script/__init__.py should maybe execute sys.path.append(os.path.dirname(__file__))?

@navneethc
Copy link
Author

navneethc commented Dec 16, 2019

The lcms_nmr_script/__init__.py now looks like this:

import os
import sys

sys.path.append(os.path.dirname(__file__))

However the error changes to not being able to find a section of an INI file. But again, the INI file exists and so does the section. (Also, get_config() takes a list of strings as the first argument.)

Just to make sure, I tested the config info retrieval from a separate test file in the same directory as cms.py, and it worked correctly.

(acd) C:\Users\<username>\Documents\ACD\code>pdoc --html lcms_nmr_script
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.configuration'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.dblookup'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
Traceback (most recent call last):
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 184, in import_module
    module = importlib.import_module(module_path)
  File "c:\programdata\anaconda3\envs\acd\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\<username>\Documents\ACD\code\lcms_nmr_script\dblookup\cms.py", line 15, in <module>
    cms_db_params = get_config(['cms_oracledb'])
  File "C:\Users\<username>\Documents\ACD\code\lcms_nmr_script\configuration\config.py", line 35, in get_config
    raise Exception('Section {} not found in the {} file'.format(section, filename))
Exception: Section cms_oracledb not found in the configuration/config.ini file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\programdata\anaconda3\envs\acd\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\envs\acd\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\envs\acd\Scripts\pdoc.exe\__main__.py", line 7, in <module>
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\cli.py", line 449, in main
    for module in args.modules]
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\cli.py", line 449, in <listcomp>
    for module in args.modules]
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 595, in __init__
    context=self._context)
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 593, in __init__
    self.doc[root] = m = Module(import_module(fullname),
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 186, in import_module
    raise ImportError('Error importing {!r}: {}'.format(module, e))
ImportError: Error importing 'lcms_nmr_script.dblookup.cms': Section cms_oracledb not found in the configuration/config.ini file

@navneethc
Copy link
Author

navneethc commented Dec 16, 2019

UPDATE: I tried running pdoc inside lcms_nmr_script, going through each package and module within it, and it seems to work fine, except for a few UserWarnings like the ones in the previous post. However it would be better if one could run it for the entire package and access the documentation (sub-)sections from a common sidebar.

@kernc
Copy link
Member

kernc commented Dec 16, 2019

I think the issues you're facing might stem from the imports you're using. Can you make sure you use either absolute imports (all import lines starting with import lcms_nmr_script...) or relative imports (from . import ...), but not implicit relative imports (e.g. import configuration).

@navneethc
Copy link
Author

Doesn't seem to change the result, I'm afraid.

All the imports of custom modules are now of the form

from lcms_nmr_script.<package/module> import <module/class/function>

(You can see the change in the UserWarnings.)

The error occurs both with and without the change to sys.path.

(acd) C:\Users\<username>\Documents\ACD\code>pdoc --html lcms_nmr_script
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.configuration'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.dblookup'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
Traceback (most recent call last):
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 184, in import_module
    module = importlib.import_module(module_path)
  File "c:\programdata\anaconda3\envs\acd\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\<username>\Documents\ACD\code\lcms_nmr_script\dblookup\cms.py", line 15, in <module>
    cms_db_params = get_config(['cms_oracledb'])
  File "C:\Users\<username>\Documents\ACD\code\lcms_nmr_script\configuration\config.py", line 35, in get_config
    raise Exception('Section {} not found in the {} file'.format(section, filename))
Exception: Section cms_oracledb not found in the configuration/config.ini file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\programdata\anaconda3\envs\acd\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\envs\acd\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\envs\acd\Scripts\pdoc.exe\__main__.py", line 7, in <module>
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\cli.py", line 449, in main
    for module in args.modules]
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\cli.py", line 449, in <listcomp>
    for module in args.modules]
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 595, in __init__
    context=self._context)
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 593, in __init__
    self.doc[root] = m = Module(import_module(fullname),
  File "c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py", line 186, in import_module
    raise ImportError('Error importing {!r}: {}'.format(module, e))
ImportError: Error importing 'lcms_nmr_script.dblookup.cms': Section cms_oracledb not found in the configuration/config.ini file

@navneethc
Copy link
Author

Another update: Traced the issue to the path supplied for the INI file (it wasn't the path from the root package directory). Once that was corrected, all the documentation was generated in one go. The documentation looks great on first glance, however a bunch of UserWarnings were thrown.

Thanks for your help and patience. I have pasted the warnings for your reference. I will leave it up to you to decide whether or not this is a bug and if this issue can be closed. :)

(acd) C:\Users\<username>\Documents\ACD\code>pdoc --html lcms_nmr_script
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.configuration'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.dblookup'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.extract_load'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
<unknown>:260: DeprecationWarning: invalid escape sequence \d
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.logger'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.transform'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
<unknown>:125: DeprecationWarning: invalid escape sequence \d
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.transform.uk_us_lcms'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))
<unknown>:265: DeprecationWarning: invalid escape sequence \s
c:\programdata\anaconda3\envs\acd\lib\site-packages\pdoc\__init__.py:227: UserWarning: Couldn't get/parse source of '<Module 'lcms_nmr_script.transform.uk_us_nmr'>'
  warn("Couldn't get/parse source of '{!r}'".format(doc_obj))

@kernc
Copy link
Member

kernc commented Dec 17, 2019

If lcms_nmr_script.configuration is a namespace package (i.e. a directory without __init__.py, with just config.py and possibly other files), then the warning for it is likely expected (no file provides the source code for lcms_nmr_script.configuration itself).

The same goes for other listed modules. They are either implicit and unavailable (as above), compiled to binaries, or otherwise unreadable by inspect.getsource().

pdoc/pdoc/__init__.py

Lines 224 to 227 in 86eac27

try:
tree = ast.parse(inspect.getsource(doc_obj.obj))
except (OSError, TypeError, SyntaxError):
warn("Couldn't get/parse source of '{!r}'".format(doc_obj))

Glad you have it working. 🥂

@kernc kernc closed this as completed Dec 17, 2019
@kernc kernc added invalid This is not a (valid) bug report and removed bug Something isn't working labels Dec 17, 2019
@mfrigerio17
Copy link

mfrigerio17 commented Jul 21, 2020

Hi, sorry about commenting on a past, closed issue, but I have exaclty the same as the original problem of @navneethc

@kernc, is pdoc supposed to work with namespace packages (besides the warnings)?

@mfrigerio17
Copy link

BTW, although I have the same problem that @navneethc originally had (module not found even though it is there and it is perfectly importable e.g. from within the python shell), I get a different warning upon invocation of pdoc:

/usr/lib/python3/dist-packages/apport/report.py:13: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import fnmatch, glob, traceback, errno, sys, atexit, locale, imp
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pdoc/__init__.py", line 200, in import_module
    module = importlib.import_module(module_path)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

python 3.8.2, pdoc 0.8.4

@kernc
Copy link
Member

kernc commented Jul 21, 2020

Have you tried following the recommended workaround? The issue is most likely related to process' current working directory and stated import paths.

Traceback (most recent call last):

@mfrigerio17 Can you post the full stack trace? Yours seem to be missing the crucial final lines ...

The DeprecationWarning about imp module is mostly irrelevant.

@mfrigerio17
Copy link

Thanks for the swift reply. I always use only absolute imports, so I guess that is not my problem (?)

Full stack trace

/usr/lib/python3/dist-packages/apport/report.py:13: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import fnmatch, glob, traceback, errno, sys, atexit, locale, imp
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pdoc/__init__.py", line 200, in import_module
    module = importlib.import_module(module_path)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<my home>/projects/rmt/src/rmt/kinematics.py", line 4, in <module>
    import kgprim.motions as motions
ModuleNotFoundError: No module named 'kgprim'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pdoc", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/pdoc/cli.py", line 459, in main
    modules = [pdoc.Module(module, docfilter=docfilter,
  File "/usr/local/lib/python3.8/dist-packages/pdoc/cli.py", line 459, in <listcomp>
    modules = [pdoc.Module(module, docfilter=docfilter,
  File "/usr/local/lib/python3.8/dist-packages/pdoc/__init__.py", line 665, in __init__
    m = Module(import_module(fullname),
  File "/usr/local/lib/python3.8/dist-packages/pdoc/__init__.py", line 202, in import_module
    raise ImportError('Error importing {!r}: {}: {}'
ImportError: Error importing 'rmt.kinematics': ModuleNotFoundError: No module named 'kgprim'

kgprim is an actual package, importable when using the shell.

Maybe I should mention I am working in a python virtual environment, and my kgprim package was installed in this environment in development mode (with pip install -e . from the root). However, as I said, importing from the shell works fine so it seems that the package is properly installed.

@kernc
Copy link
Member

kernc commented Jul 21, 2020

... Importing from the shell, with activated venv, from arbitrary directory?

( cd /tmp; python -c 'import kgprim.motions as motions'; )

We do play around with import paths, but not in a way that shouldn't work ... I think. 🤔

pdoc/pdoc/__init__.py

Lines 183 to 191 in 04960e4

path = '_pdoc_dummy_nonexistent'
module_name = inspect.getmodulename(module)
if isdir(module):
path, module = split(abspath(module))
elif isfile(module) and module_name:
path, module = dirname(abspath(module)), module_name
try:
sys.path.insert(0, path)
yield module

Note, isdir() and isfile() branches hit only if module is a literal filesystem path. This is really only relevant for the top referenced module.

@kernc
Copy link
Member

kernc commented Jul 21, 2020

Is pdoc run from the same venv (which pdoc)?

File "/usr/local/bin/pdoc", line 8, in <module>

This indicates it might not be the case.

Maybe related: #44 (comment).

@mfrigerio17
Copy link

... Importing from the shell, with activated venv, from arbitrary directory?

Yes, indeed.

Is pdoc run from the same venv (which pdoc)?

mhmm... indeed "/usr/local/bin/pdoc" is strange. which pdoc gives me the right path (in the virtual env) but somehow the system-wide pdoc is executed. (indeed related to #44 , it seems)

However I restarted my terminal and reactived the virtualenv, and it now works!! Thanks a lot!!

So, for other users who may run into the same issue as me:

  • install pdoc inside the virtual environment (pip install pdoc3 from a shell with the activated environment)
  • check the right binary is picked: which pdoc
  • if you still get the error, check whether the error reports that the system wide pdoc is used instead
  • if that is the case, try by opening a new shell and reactivate the environment, and run pdoc again

Thanks again

@kernc
Copy link
Member

kernc commented Jul 21, 2020

#44 should have been fixed though (PR #45). It would help having a clear, reproducible mwe. 😅

@mfrigerio17
Copy link

I think I managed to reproduce the issue. It appears only when installing in development mode.

Steps:

  • create package A and B, B depends on A (i.e. has an (absolute) import of A)
  • make and activate the virtual environment:
    cd /tmp && mkdir pyenv && cd pyenv && python3 -m venv . && source bin/activate
  • install A in development mode; from the root of A:
    pip install -e .
  • go to the root of B, and run pdoc, e.g.:
    pdoc --html -o /tmp/docs B/

It should fail saying that A is not a module. Solution: install pdoc in the virtual env and reactivate the environment.

Note that with a regular install (not development mode) I had no issue at all, and it wasn't even necessary to install pdoc in the virtual environment. The system-wide one worked fine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
invalid This is not a (valid) bug report
Development

No branches or pull requests

3 participants