GCP provider

[quasiben]

Attempt at resolving #130

cc @jacobtomlinson @mrocklin

Currently, this code is non-functional but scaffolding is in place to get things up and running

PR should be in a runnable state now

[jacobtomlinson]

I took this for a spin but am stuck with credentials errors. It's likely something I'm doing, but I'm going to need some docs/guidance.

I've tried:

• Reauthenticating with gcloud auth login
• Upgrading my SDK version pip install google-api-python-client --upgrade

See my comments for details on what I'm seeing.

[jacobtomlinson]

This doesn't match the ImportError which says pip install "dask-cloudprovider[gcp]".

[jacobtomlinson]

I'm initially having credentials issues and am seeing the following exception. We should probably catch this and raise something friendlier.

    @pytest.mark.asyncio
    async def test_init():
>       cluster = GCPCluster(asynchronous=True)

dask_cloudprovider/providers/gcp/tests/test_gcp.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dask_cloudprovider/providers/gcp/instances.py:284: in __init__
    self.compute = googleapiclient.discovery.build("compute", "v1")
../../../miniconda3/envs/daskdev/lib/python3.8/site-packages/googleapiclient/_helpers.py:134: in positional_wrapper
    return wrapped(*args, **kwargs)
../../../miniconda3/envs/daskdev/lib/python3.8/site-packages/googleapiclient/discovery.py:276: in build
    return build_from_document(
../../../miniconda3/envs/daskdev/lib/python3.8/site-packages/googleapiclient/_helpers.py:134: in positional_wrapper
    return wrapped(*args, **kwargs)
../../../miniconda3/envs/daskdev/lib/python3.8/site-packages/googleapiclient/discovery.py:516: in build_from_document
    credentials = _auth.default_credentials(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

scopes = None, quota_project_id = None

    def default_credentials(scopes=None, quota_project_id=None):
        """Returns Application Default Credentials."""
        if HAS_GOOGLE_AUTH:
>           credentials, _ = google.auth.default(scopes=scopes, quota_project_id=quota_project_id)
E           TypeError: default() got an unexpected keyword argument 'quota_project_id'

../../../miniconda3/envs/daskdev/lib/python3.8/site-packages/googleapiclient/_auth.py:54: TypeError

[jacobtomlinson]

I have configured credentials as described here but the test is still skipping. See the traceback I raised before.

I'm guessing something else is required to configure the credentials here, or the try/except is being greedy and catching some other exception.

[quasiben]

@eric-czech I finished some touches. If you have time/interest in the coming weeks I think this is safe to test

[jacobtomlinson]

Thanks for the fixes here @quasiben. I have now managed to run the tests locally.

Once #129 is merged this should be good to go.

[eric-czech]

Hey @quasiben, I wasn't quite sure what basic configuration + usage would look like but I started with this:

from dask_cloudprovider.providers.gcp.instances import GCPCluster
cluster = GCPCluster(
    name='dask-gcp-test-1', 
    zone='us-east1-c', 
    machine_type='n1-standard-8', 
    docker_image="daskdev/dask:latest",
    worker_command='dask-worker',
    ngpus=0,
    projectid=PROJECT_ID,
)
~/miniconda3/envs/dask-cp/lib/python3.8/site-packages/distributed/deploy/spec.py in _start(self)
    299         if isinstance(cls, str):
    300             cls = import_term(cls)
--> 301         self.scheduler = cls(**self.scheduler_spec.get("options", {}))
    302 
    303         self.status = Status.starting

TypeError: 'NoneType' object is not callable

Full Trace

--------------------------------------------------------------------------- TypeError Traceback (most recent call last) in ----> 1 cluster = GCPCluster( 2 name='dask-gcp-test-1', zone='us-east1-c', machine_type='n1-standard-8', 3 projectid='uk-biobank-279813', 4 docker_image="daskdev/dask:latest", 5 worker_command='dask-worker',

~/repos/dask-cloudprovider/dask_cloudprovider/providers/gcp/instances.py in init(self, name, zone, machine_type, projectid, source_image, docker_image, ngpus, gpu_type, worker_command, worker_extra_args, auto_shutdown, **kwargs)
296 **kwargs,
297 ):
--> 298 super().init(**kwargs)
299 try:
300 self.compute = googleapiclient.discovery.build("compute", "v1")

~/repos/dask-cloudprovider/dask_cloudprovider/providers/generic/vmcluster.py in init(self, n_workers, **kwargs)
92 self.bootstrap = None
93 self.auto_shutdown = True
---> 94 super().init(**kwargs)
95
96 async def _start(self,):

~/miniconda3/envs/dask-cp/lib/python3.8/site-packages/distributed/deploy/spec.py in init(self, workers, scheduler, worker, asynchronous, loop, security, silence_logs, name)
274 if not self.asynchronous:
275 self._loop_runner.start()
--> 276 self.sync(self._start)
277 self.sync(self._correct_state)
278

~/miniconda3/envs/dask-cp/lib/python3.8/site-packages/distributed/deploy/cluster.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
181 return future
182 else:
--> 183 return sync(self.loop, func, *args, **kwargs)
184
185 def _log(self, log):

~/miniconda3/envs/dask-cp/lib/python3.8/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
338 if error[0]:
339 typ, exc, tb = error[0]
--> 340 raise exc.with_traceback(tb)
341 else:
342 return result[0]

~/miniconda3/envs/dask-cp/lib/python3.8/site-packages/distributed/utils.py in f()
322 if callback_timeout is not None:
323 future = asyncio.wait_for(future, callback_timeout)
--> 324 result[0] = yield future
325 except Exception as exc:
326 error[0] = sys.exc_info()

~/miniconda3/envs/dask-cp/lib/python3.8/site-packages/tornado/gen.py in run(self)
733
734 try:
--> 735 value = future.result()
736 except Exception:
737 exc_info = sys.exc_info()

~/repos/dask-cloudprovider/dask_cloudprovider/providers/generic/vmcluster.py in _start(self)
115 "Hang tight! ",
116 ):
--> 117 await super()._start()
118
119 def render_cloud_init(self, *args, **kwargs):

~/miniconda3/envs/dask-cp/lib/python3.8/site-packages/distributed/deploy/spec.py in _start(self)
299 if isinstance(cls, str):
300 cls = import_term(cls)
--> 301 self.scheduler = cls(**self.scheduler_spec.get("options", {}))
302
303 self.status = Status.starting

TypeError: 'NoneType' object is not callable

Am I missing something obvious?

[quasiben]

@eric-czech things should be a far better state now with sync and async support thanks to @jacobtomlinson.

[jacobtomlinson]

I've just pulled master and pushed a commit to shuffle the imports to the new layout.

[quasiben]

Thanks @jacobtomlinson . I'll fix the imports in tests and push. After that would you be willing to give this another review ?

[quasiben]

Thank you @jacobtomlinson !

[mrocklin]

Ooh, I'm happy to see this in.

…

On Fri, Nov 6, 2020 at 5:50 AM Benjamin Zaitlen ***@***.***> wrote: Thank you @jacobtomlinson <https://github.com/jacobtomlinson> ! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#131 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTBT5UBUZFNQBBJQ5XLSOP5JRANCNFSM4RAPZLCQ> .