Cannot resolve google.cloud imports when types-protobuf are installed

[nullie]

Describe the bug
When types-protobuf stub is installed, google.cloud imports cannot be resolved. I don't think it's a stub bug, because mypy seems to handle this right.

To Reproduce
In a fresh venv:

pip install types-protobuf
pip install google-cloud-firestore
cat > test.py
from google.cloud import firestore
^D
npx pyright --verbose

No configuration file found.
No pyproject.toml file found.
stubPath /home/nullie/work/pyright-bug/typings is not a valid directory.
Assuming Python platform Linux
Search paths for /home/nullie/work/pyright-bug
  /home/nullie/work/pyright-bug/node_modules/pyright/dist/typeshed-fallback/stdlib
  /home/nullie/work/pyright-bug
  /home/nullie/work/pyright-bug/typings
  /home/nullie/work/pyright-bug/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /usr/lib/python3.8
  /usr/lib/python3.8/lib-dynload
  /home/nullie/work/pyright-bug/venv/lib/python3.8/site-packages
Searching for source files
Auto-excluding /home/nullie/work/pyright-bug/venv
Found 1 source file
Could not import 'google.cloud' in file '/home/nullie/work/pyright-bug/test.py'
  Looking for typeshed stdlib path
  Attempting to resolve using root path '/home/nullie/work/pyright-bug/node_modules/pyright/dist/typeshed-fallback/stdlib'
  Typeshed path not found
  Looking in stubPath '/home/nullie/work/pyright-bug/typings'
  Attempting to resolve using root path '/home/nullie/work/pyright-bug/typings'
  Attempting to resolve using root path '/home/nullie/work/pyright-bug/typings'
  Looking in root directory of execution environment '/home/nullie/work/pyright-bug'
  Attempting to resolve using root path '/home/nullie/work/pyright-bug'
  Attempting to resolve using root path '/home/nullie/work/pyright-bug'
  Looking in python search path '/usr/lib/python3.8'
  Attempting to resolve using root path '/usr/lib/python3.8'
  Attempting to resolve using root path '/usr/lib/python3.8'
  Looking in python search path '/usr/lib/python3.8/lib-dynload'
  Attempting to resolve using root path '/usr/lib/python3.8/lib-dynload'
  Attempting to resolve using root path '/usr/lib/python3.8/lib-dynload'
  Looking in python search path '/home/nullie/work/pyright-bug/venv/lib/python3.8/site-packages'
  Attempting to resolve using root path '/home/nullie/work/pyright-bug/venv/lib/python3.8/site-packages'
  Resolved import with file '/home/nullie/work/pyright-bug/venv/lib/python3.8/site-packages/google-stubs/__init__.pyi'
  Looking for typeshed path
  Looking for typeshed stubs path
  Attempting to resolve using root path '/home/nullie/work/pyright-bug/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf'
  Resolved import with file '/home/nullie/work/pyright-bug/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf/google/__init__.pyi'
  Typeshed path not found
/home/nullie/work/pyright-bug/test.py
  /home/nullie/work/pyright-bug/test.py:1:6 - error: Import "google.cloud" could not be resolved (reportMissingImports)
1 error, 0 warnings, 0 infos
Completed in 0.619sec

Expected behavior
No errors

Screenshots or Code
from google.cloud import firestore

VS Code extension or command-line
Running pyright 1.1.157 with npx

Additional context
Add any other context about the problem here.

[erictraut]

I'm receiving a similar error message from mypy:

test13.py:1: error: Skipping analyzing "google.cloud": found module but no type hints or library stubs
test13.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

The problem is that the types-protobuf library is installing a directory called google-stubs. PEP 561 indicates that when a stub package is present, it should be used for type information even if the corresponding implementation library is also present. PEP 561 also provides a way for a stub library to be marked as "partial" using a py.typed file that contains the word partial in it. The types-protobuf library contains no such py.typed file. If I add one, it works as expected. So this appears to be a bug in the types-protobuf stub library. Please report the bug to the library maintainers.

[nullie]

Okay, thanks for the info

[nullie]

The one thing that confuses me is that currently typeshed has no py.typed files

[erictraut]

The typeshed stubs are not "stub-only packages" as defined by PEP 561 — i.e. they're not installed directly in the site-packages directory of your Python environment. Plus, they are not partial. For those reasons, they don't need a py.typed file.

[romanofski]

I wanted to fix python/typeshed#5800 and came across a problem I think is more suited to be commented here rather than on the typeshed issue.

I actually tried out the attempt with the partial py.typed file using the reproducer from here. I don't think this will help us unfortunately if we're dealing with more google packages. It does fix the immediate issue with the two packages (google.cloud.firestore and google.protobuf), however if we add a third google namespace package we're in trouble again:

The partial is in the root of the installed types-protobuf package in my site-packages

goldie :: lib/python3.9/site-packages % tree -L 1 google-stubs                                                                      1 ↵
google-stubs
├── cloud
├── __init__.pyi
├── METADATA.toml
├── protobuf
└── py.typed <-- "partial"

2 directories, 3 files

Using the given reproducer:

$ cat test.py                                                                                                 
from google.cloud import firestore
from google.protobuf import __version__

# I hacked up the __version__ in google-stubs/cloud/__init__.pyi and changed it to str to avoid the checker inferring the type 
# (which is correctly `bytes`) to see whether it's actually picked up.
reveal_type(__version__)

and run it against pyright:

% foo/node_modules/.bin/pyright --version                                                                     
pyright 1.1.160
% foo/node_modules/.bin/pyright test.py                                                                       
No configuration file found.
No pyproject.toml file found.
stubPath /tmp/typings is not a valid directory.
Assuming Python platform Linux
Searching for source files
Found 1 source file
/tmp/test.py
  /tmp/test.py:4:13 - info: Type of "__version__" is "str"
0 errors, 0 warnings, 1 info 
Completed in 0.563sec

No trouble. I install another google.cloud package generated from using the stub_uploader in typeshed-internals (see python/typeshed#5800 (comment)):

(python3) goldie :: /tmp % pip install tmp4_qz07i1/                                                                                    
Processing ./tmp4_qz07i1
Requirement already satisfied: types-six in ./python3/lib/python3.9/site-packages (from types-google-cloud-ndb==1.9.0.1) (0.1.7)
Using legacy 'setup.py install' for types-google-cloud-ndb, since package 'wheel' is not installed.
Installing collected packages: types-google-cloud-ndb
[...]

And rerun pyright against the reproducer again I have no trouble. However, if I run it against my reproducer from
python/typeshed#5821 (comment) I get:

 % foo/node_modules/.bin/pyright --verbose foo.py                                                           1 ↵
No configuration file found.
No pyproject.toml file found.
stubPath /tmp/typings is not a valid directory.
Assuming Python platform Linux
Search paths for /tmp
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib
  /tmp
  /tmp/typings
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /usr/lib64/python3.9
  /usr/lib64/python3.9/lib-dynload
  /tmp/python3/lib/python3.9/site-packages
Searching for source files
Found 1 source file
Could not import 'google.cloud.ndb' in file '/tmp/foo.py'
  Looking for typeshed stdlib path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib'
  Typeshed path not found
  Looking in stubPath '/tmp/typings'
  Attempting to resolve using root path '/tmp/typings'
  Attempting to resolve using root path '/tmp/typings'
  Looking in root directory of execution environment '/tmp'
  Attempting to resolve using root path '/tmp'
  Attempting to resolve using root path '/tmp'
  Looking in python search path '/usr/lib64/python3.9'
  Attempting to resolve using root path '/usr/lib64/python3.9'
  Attempting to resolve using root path '/usr/lib64/python3.9'
  Looking in python search path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
  Looking in python search path '/tmp/python3/lib/python3.9/site-packages'
  Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages'
  Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages'
  Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google/__init__.pyi'
  Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google/cloud/__init__.pyi'
  Looking for typeshed path
  Looking for typeshed stubs path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf'
  Resolved import with file '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf/google/__init__.pyi'
  Typeshed path not found
/tmp/foo.py
  /tmp/foo.py:1:6 - error: Import "google.cloud.ndb" could not be resolved (reportMissingImports)
  /tmp/foo.py:8:13 - info: Type of "foo.name" is "Unknown"
  /tmp/foo.py:9:13 - info: Type of "foo.key" is "Unknown"
1 error, 0 warnings, 2 infos 
Completed in 0.559sec

it doesn't pick up the types because the protobuf stubs are installed.

Now you might say, hang on a tick here. The partial py.typed needs to go into the sub-directory of google-stubs/protobuf and google-stubs/cloud/ndb respectively. Then I get:

oldie :: lib/python3.9/site-packages % tree -P py.typed google-stubs                                                                  
google-stubs
├── cloud
│   └── ndb
│       └── py.typed
└── protobuf
    ├── compiler
    ├── internal
    ├── py.typed
    └── util

% foo/node_modules/.bin/pyright --verbose test.py                                                          1 ↵
No configuration file found.
No pyproject.toml file found.
stubPath /tmp/typings is not a valid directory.
Assuming Python platform Linux
Search paths for /tmp
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib
  /tmp
  /tmp/typings
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /usr/lib64/python3.9
  /usr/lib64/python3.9/lib-dynload
  /tmp/python3/lib/python3.9/site-packages
Searching for source files
Found 1 source file
/tmp/test.py
  /tmp/test.py:1:26 - error: "firestore" is unknown import symbol (reportGeneralTypeIssues)
  /tmp/test.py:4:13 - info: Type of "__version__" is "str"
1 error, 0 warnings, 1 info 
Completed in 0.567sec

The __version__ is correctly typed again, but now I have problems importing google.cloud.firestore.

Shouldn't partial be only used for packages which have unfinished stub definitions?

I removed my virtualenv a few times because I ended up confused with the reproducer and the results. I think, uninstalling the 'typeshed-protobuf' package leaves the google-stubs directory behind, confusing the typecheckers. So perhaps I went all the wrong way into the rabbit hole and this is not an issue. If that's the case, please ignore.

[erictraut]

I'm a bit confused about all of the locations where you have type stub files associated with the google namespace. Based on the logs and the other information you've provided above, I think you have google type stubs in three locations:

1. In your site-packages directory under the google-stubs subdirectory
2. In your site-packages directory under the google subdirectory
3. In the typeshed stubs that are packaged with pyright

That is the search order that pyright uses when attempting to resolve an import, consistent with PEP 561.

Looking at the pyright logs above, it doesn't appear to find any google-stubs subdirectory in site-packages. Are you sure that a google-stubs directory existed at the time you ran pyright? The log includes the line Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages' twice in a row. The first time it was looking for google-stubs, and the second it was looking for google. (I just updated the pyright logging messages to make that more clear.)

It doesn't find a google-stubs subdirectory, but it does find stubs in the google directory based on this log message:

Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google/init.pyi'

It's finding an __init__.pyi directly under google, which means that google is not a namespace package. Per PEP 420:

Namespace packages cannot contain an init.py

So I see a couple of issues here. First, I'm not sure why pyright is not finding a google-stubs subdirectory. Second, I don't think there should be a google/__init__.pyi file if google is a namespace package.

[romanofski]

I'm a bit confused about all of the locations where you have type stub files associated with the google namespace. Based on the logs and the other information you've provided above, I think you have google type stubs in three locations:

1. In your site-packages directory under the `google-stubs` subdirectory

2. In your site-packages directory under the `google` subdirectory

3. In the typeshed stubs that are packaged with pyright

That is the search order that pyright uses when attempting to resolve an import, consistent with PEP 561.

Looking at the pyright logs above, it doesn't appear to find any google-stubs subdirectory in site-packages. Are you sure that a google-stubs directory existed at the time you ran pyright?

Yes I'm pretty sure. I went back today, removed the virtualenv and recreated it. Then installed google-cloud-firestore and then types-protobuf. Comparing to my initial starting point from my comment above, I do realize that I did make a mistake. The first code area already includes the types-google-cloud-ndb package installed. However, installing types-protobuf brings the __init__.pyi in the root google-stubs folder:

site-packages % tree -L 1 google-stubs                                                                  
google-stubs
├── __init__.pyi
├── METADATA.toml
└── protobuf

1 directory, 2 files

The uploader would generate a similar package for any other google namespace package in typeshed at the moment. I think that has to be addressed in typeshed (see python/typeshed#5800)

The log includes the line Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages' twice in a row. The first time it was looking for google-stubs, and the second it was looking for google. (I just updated the pyright logging messages to make that more clear.)

It doesn't find a google-stubs subdirectory, but it does find stubs in the google directory based on this log message:

Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google/init.pyi'

It's finding an __init__.pyi directly under google, which means that google is not a namespace package. Per PEP 420:

Namespace packages cannot contain an init.py

So I see a couple of issues here. First, I'm not sure why pyright is not finding a google-stubs subdirectory. Second, I don't think there should be a google/__init__.pyi file if google is a namespace package.

I think there must be a problem with the traversal? Because the google directory does not contain the __init__.pyi, google-stubs does:

site-packages % ls google                                                                             
api  api_core  auth  cloud  gapic  logging  longrunning  oauth2  protobuf  rpc  type
site-packages % ls google-stubs                                                                       
__init__.pyi  METADATA.toml  protobuf  py.typed

I actually did remove the google-stubs/__init__.py file just to see whether that makes a difference. I've entirely removed the py.typed file too and there is no difference between it being absent to it being placed into google-stubs/protobuf/ subdirectory:

/tmp % foo/node_modules/.bin/pyright test.py --verbose                                                          1 ↵
No configuration file found.
No pyproject.toml file found.
stubPath /tmp/typings is not a valid directory.
Assuming Python platform Linux
Search paths for /tmp
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib
  /tmp
  /tmp/typings
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /usr/lib64/python3.9
  /usr/lib64/python3.9/lib-dynload
  /tmp/python3/lib/python3.9/site-packages
Searching for source files
Found 1 source file
Could not import 'google.cloud' in file '/tmp/test.py'
  Looking for typeshed stdlib path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib'
  Typeshed path not found
  Looking in stubPath '/tmp/typings'
  Attempting to resolve using root path '/tmp/typings'
  Attempting to resolve using root path '/tmp/typings'
  Looking in root directory of execution environment '/tmp'
  Attempting to resolve using root path '/tmp'
  Attempting to resolve using root path '/tmp'
  Looking in python search path '/usr/lib64/python3.9'
  Attempting to resolve using root path '/usr/lib64/python3.9'
  Attempting to resolve using root path '/usr/lib64/python3.9'
  Looking in python search path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
  Looking in python search path '/tmp/python3/lib/python3.9/site-packages'
  Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages'
  Looking for typeshed path
  Looking for typeshed stubs path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf'
  Resolved import with file '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf/google/__init__.pyi'
  Typeshed path not found
/tmp/test.py
  /tmp/test.py:1:6 - error: Import "google.cloud" could not be resolved (reportMissingImports)
  /tmp/test.py:6:13 - info: Type of "__version__" is "bytes"
1 error, 0 warnings, 1 info 
Completed in 0.56sec

[erictraut]

I published pyright 1.1.161, which improves the logging for import failures. Could you please install the updated version of pyright and paste the logged output? Thanks!

[romanofski]

Ok, here we go. I removed the old virtualenv, and recreated a new python3 venv. Installed google-cloud-firestore as well as types-protobuf:

$ foo/node_modules/.bin/pyright --version                                                                  1 ↵
pyright 1.1.161
$ cat test.py
from google.cloud import firestore
from google.protobuf import __version__

reveal_type(__version__)

$ foo/node_modules/.bin/pyright --verbose test.py                                                             
No configuration file found.
No pyproject.toml file found.
stubPath /tmp/typings is not a valid directory.
Assuming Python platform Linux
Search paths for /tmp
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib
  /tmp
  /tmp/typings
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /usr/lib64/python3.9
  /usr/lib64/python3.9/lib-dynload
  /tmp/python3/lib/python3.9/site-packages
Searching for source files
Found 1 source file
Could not import 'google.cloud' in file '/tmp/test.py'
  Looking for typeshed stdlib path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib'
  Typeshed path not found
  Looking in stubPath '/tmp/typings'
  Attempting to resolve stub package using root path '/tmp/typings'
  Attempting to resolve using root path '/tmp/typings'
  Looking in root directory of execution environment '/tmp'
  Attempting to resolve stub package using root path '/tmp'
  Attempting to resolve using root path '/tmp'
  Looking in python search path '/usr/lib64/python3.9'
  Attempting to resolve stub package using root path '/usr/lib64/python3.9'
  Attempting to resolve using root path '/usr/lib64/python3.9'
  Looking in python search path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve stub package using root path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
  Looking in python search path '/tmp/python3/lib/python3.9/site-packages'
  Attempting to resolve stub package using root path '/tmp/python3/lib/python3.9/site-packages'
  Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google-stubs/__init__.pyi'
  Looking for typeshed path
  Looking for typeshed stubs path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf'
  Resolved import with file '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf/google/__init__.pyi'
  Typeshed path not found
/tmp/test.py
  /tmp/test.py:1:6 - error: Import "google.cloud" could not be resolved (reportMissingImports)
  /tmp/test.py:4:13 - info: Type of "__version__" is "bytes"
1 error, 0 warnings, 1 info 
Completed in 0.574sec

I thought it might be useful to have a diff too:

-/tmp % foo/node_modules/.bin/pyright test.py --verbose                                                          1 ↵
+$ foo/node_modules/.bin/pyright --verbose test.py
 No configuration file found.
 No pyproject.toml file found.
 stubPath /tmp/typings is not a valid directory.
@@ -18,19 +18,20 @@
   Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib'
   Typeshed path not found
   Looking in stubPath '/tmp/typings'
-  Attempting to resolve using root path '/tmp/typings'
+  Attempting to resolve stub package using root path '/tmp/typings'
   Attempting to resolve using root path '/tmp/typings'
   Looking in root directory of execution environment '/tmp'
-  Attempting to resolve using root path '/tmp'
+  Attempting to resolve stub package using root path '/tmp'
   Attempting to resolve using root path '/tmp'
   Looking in python search path '/usr/lib64/python3.9'
-  Attempting to resolve using root path '/usr/lib64/python3.9'
+  Attempting to resolve stub package using root path '/usr/lib64/python3.9'
   Attempting to resolve using root path '/usr/lib64/python3.9'
   Looking in python search path '/usr/lib64/python3.9/lib-dynload'
-  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
+  Attempting to resolve stub package using root path '/usr/lib64/python3.9/lib-dynload'
   Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
   Looking in python search path '/tmp/python3/lib/python3.9/site-packages'
-  Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages'
+  Attempting to resolve stub package using root path '/tmp/python3/lib/python3.9/site-packages'
+  Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google-stubs/__init__.pyi'
   Looking for typeshed path
   Looking for typeshed stubs path
   Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf'
@@ -38,6 +39,6 @@
   Typeshed path not found
 /tmp/test.py
   /tmp/test.py:1:6 - error: Import "google.cloud" could not be resolved (reportMissingImports)
-  /tmp/test.py:6:13 - info: Type of "__version__" is "bytes"
+  /tmp/test.py:4:13 - info: Type of "__version__" is "bytes"
 1 error, 0 warnings, 1 info 
-Completed in 0.56sec
+Completed in 0.574sec