docs: streamline 'timeout' / 'retry' docs in docstrings (#461)

* Add 'requests' intersphinx refs
* Add narrative docs for timeouts and retries
* Include API docs for the 'retry' module, as well, to unbreak links.
* Replace boilerplate docstring entries for 'timeout' / 'retry' with links to new narrative docs.
* Add docstrings for default conditional policies.

Closes #455.

Author: tseaver

docs/conf.py

@@ -364,6 +364,7 @@
     "grpc": ("https://grpc.github.io/grpc/python/", None),
     "proto-plus": ("https://proto-plus-python.readthedocs.io/en/latest/", None),
     "protobuf": ("https://googleapis.dev/python/protobuf/latest/", None),
+    "requests": ("https://docs.python-requests.org/en/master/", None),
 }
 
 

docs/index.rst

@@ -21,6 +21,7 @@ API Reference
   constants
   hmac_key
   notification
+  retry_timeout
 
 Changelog
 ---------

docs/retry_timeout.rst

@@ -0,0 +1,152 @@
+Configuring Timeouts and Retries
+================================
+
+When using object methods which invoke Google Cloud Storage API methods,
+you have several options for how the library handles timeouts and
+how it retries transient errors.
+
+
+.. _configuring_timeouts:
+
+Configuring Timeouts
+--------------------
+
+For a number of reasons, methods which invoke API methods may take
+longer than expected or desired.  By default, such methods all time out
+after a default interval, 60.0 seconds.  Rather than blocking your application
+code for that interval, you may choose to configure explicit timeouts
+in your code, using one of three forms:
+
+- You can pass a single integer or float which functions as the timeout for the
+  entire request. E.g.:
+
+.. code-block:: python
+
+   bucket = client.get_bucket(BUCKET_NAME, timeout=300.0)  # five minutes
+
+- You can also be passed as a two-tuple, ``(connect_timeout, read_timeout)``,
+  where the ``connect_timeout`` sets the maximum time required to establish
+  the connection to the server, and the ``read_timeout`` sets the maximum
+  time to wait for a completed response.  E.g.:
+
+.. code-block:: python
+
+   bucket = client.get_bucket(BUCKET_NAME, timeout=(3, 10))
+
+
+- You can also pass ``None`` as the timeout value:  in this case, the library
+  will block indefinitely for a response.  E.g.:
+
+.. code-block:: python
+
+   bucket = client.get_bucket(BUCKET_NAME, timeout=None)
+
+.. note::
+   Depending on the retry strategy, a request may be
+   repeated several times using the same timeout each time.
+
+See also:
+
+  :ref:`Timeouts in requests <requests:timeouts>`
+
+
+.. _configuring_retries:
+
+Configuring Retries
+--------------------
+
+.. note::
+
+   For more background on retries, see also the
+   `GCS Retry Strategies Document <https://cloud.google.com/storage/docs/retry-strategy#python>`_ 
+
+Methods which invoke API methods may fail for a number of reasons, some of
+which represent "transient" conditions, and thus can be retried
+automatically.  The library tries to provide a sensible default retry policy
+for each method, base on its semantics:
+
+- For API requests which are always idempotent, the library uses its
+  :data:`~google.cloud.storage.retry.DEFAULT_RETRY` policy, which
+  retries any API request which returns a "transient" error.
+
+- For API requests which are idempotent only if the blob has
+  the same "generation", the library uses its
+  :data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_GENERATION_SPECIFIED`
+  policy, which retries API requests which returns a "transient" error,
+  but only if the original request includes an ``ifGenerationMatch`` header.
+
+- For API requests which are idempotent only if the bucket or blob has
+  the same "metageneration", the library uses its
+  :data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED`
+  policy, which retries API requests which returns a "transient" error,
+  but only if the original request includes an ``ifMetagenerationMatch`` header.
+
+- For API requests which are idempotent only if the bucket or blob has
+  the same "etag", the library uses its
+  :data:`~google.cloud.storage.retry.DEFAULT_RETRY_IF_ETAG_IN_JSON`
+  policy, which retries API requests which returns a "transient" error,
+  but only if the original request includes an ``ETAG`` in its payload.
+
+- For those API requests which are never idempotent, the library passes
+  ``retry=None`` by default, suppressing any retries.
+
+Rather than using one of the default policies, you may choose to configure an
+explicit policy in your code.
+
+- You can pass ``None`` as a retry policy to disable retries.  E.g.:
+
+.. code-block:: python
+
+   bucket = client.get_bucket(BUCKET_NAME, retry=None)
+
+- You can pass an instance of :class:`google.api_core.retry.Retry` to enable
+  retries;  the passed object will define retriable response codes and errors,
+  as well as configuring backoff and retry interval options.  E.g.:
+
+.. code-block:: python
+
+   from google.api_core import exceptions
+   from google.api_core.retry import Retry
+
+   _MY_RETRIABLE_TYPES = [
+      exceptions.TooManyRequests,  # 429
+      exceptions.InternalServerError,  # 500
+      exceptions.BadGateway,  # 502
+      exceptions.ServiceUnavailable,  # 503
+   ]
+
+   def is_retryable(exc):
+       return isinstance(exc, _MY_RETRIABLE_TYPES)
+
+   my_retry_policy = Retry(predicate=is_retryable)
+   bucket = client.get_bucket(BUCKET_NAME, retry=my_retry_policy)
+
+- You can pass an instance of
+  :class:`google.cloud.storage.retry.ConditionalRetryPolicy`, which wraps a
+  :class:`~google.cloud.storage.retry.RetryPolicy`, activating it only if
+  certain conditions are met. This class exists to provide safe defaults
+  for RPC calls that are not technically safe to retry normally (due to
+  potential data duplication or other side-effects) but become safe to retry
+  if a condition such as if_metageneration_match is set.  E.g.:
+
+.. code-block:: python
+
+   from google.api_core.retry import Retry
+   from google.cloud.storage.retry import ConditionalRetryPolicy
+   from google.cloud.storage.retry import is_etag_in_json
+
+   def is_retryable(exc):
+       ... # as above
+
+   my_retry_policy = Retry(predicate=is_retryable)
+   my_cond_policy = ConditionalRetryPolicy(
+       my_retry_policy, conditional_predicate=is_etag_in_json)
+   bucket = client.get_bucket(BUCKET_NAME, retry=my_cond_policy)
+
+
+Retry Module API
+----------------
+
+.. automodule:: google.cloud.storage.retry
+  :members:
+  :show-inheritance:

google/cloud/storage/_helpers.py

@@ -162,11 +162,9 @@ def reload(
                            properties to return.
 
         :type timeout: float or tuple
-        :param timeout: (Optional) The amount of time, in seconds, to wait
-            for the server response.
-
-            Can also be passed as a tuple (connect_timeout, read_timeout).
-            See :meth:`requests.Session.request` documentation for details.
+        :param timeout:
+            (Optional) The amount of time, in seconds, to wait
+            for the server response.  See: :ref:`configuring_timeouts`
 
         :type if_generation_match: long
         :param if_generation_match: (Optional) Make the operation conditional on whether
@@ -190,18 +188,8 @@ def reload(
                                             blob's current metageneration does not match the given value.
 
         :type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
-        :param retry: (Optional) How to retry the RPC. A None value will disable retries.
-            A google.api_core.retry.Retry value will enable retries, and the object will
-            define retriable response codes and errors and configure backoff and timeout options.
-
-            A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a Retry object and
-            activates it only if certain conditions are met. This class exists to provide safe defaults
-            for RPC calls that are not technically safe to retry normally (due to potential data
-            duplication or other side-effects) but become safe to retry if a condition such as
-            if_metageneration_match is set.
-
-            See the retry.py source code and docstrings in this package (google.cloud.storage.retry) for
-            information on retry types and how to configure them.
+        :param retry:
+            (Optional) How to retry the RPC. See: :ref:`configuring_retries`
         """
         client = self._require_client(client)
         query_params = self._query_params
@@ -275,11 +263,9 @@ def patch(
                        ``client`` stored on the current object.
 
         :type timeout: float or tuple
-        :param timeout: (Optional) The amount of time, in seconds, to wait
-            for the server response.
-
-            Can also be passed as a tuple (connect_timeout, read_timeout).
-            See :meth:`requests.Session.request` documentation for details.
+        :param timeout:
+            (Optional) The amount of time, in seconds, to wait
+            for the server response.  See: :ref:`configuring_timeouts`
 
         :type if_generation_match: long
         :param if_generation_match: (Optional) Make the operation conditional on whether
@@ -303,18 +289,8 @@ def patch(
                                             blob's current metageneration does not match the given value.
 
         :type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
-        :param retry: (Optional) How to retry the RPC. A None value will disable retries.
-            A google.api_core.retry.Retry value will enable retries, and the object will
-            define retriable response codes and errors and configure backoff and timeout options.
-
-            A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a Retry object and
-            activates it only if certain conditions are met. This class exists to provide safe defaults
-            for RPC calls that are not technically safe to retry normally (due to potential data
-            duplication or other side-effects) but become safe to retry if a condition such as
-            if_metageneration_match is set.
-
-            See the retry.py source code and docstrings in this package (google.cloud.storage.retry) for
-            information on retry types and how to configure them.
+        :param retry:
+            (Optional) How to retry the RPC. See: :ref:`configuring_retries`
         """
         client = self._require_client(client)
         query_params = self._query_params
@@ -363,11 +339,9 @@ def update(
                        ``client`` stored on the current object.
 
         :type timeout: float or tuple
-        :param timeout: (Optional) The amount of time, in seconds, to wait
-            for the server response.
-
-            Can also be passed as a tuple (connect_timeout, read_timeout).
-            See :meth:`requests.Session.request` documentation for details.
+        :param timeout:
+            (Optional) The amount of time, in seconds, to wait
+            for the server response.  See: :ref:`configuring_timeouts`
 
         :type if_generation_match: long
         :param if_generation_match: (Optional) Make the operation conditional on whether
@@ -391,18 +365,8 @@ def update(
                                             blob's current metageneration does not match the given value.
 
         :type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
-        :param retry: (Optional) How to retry the RPC. A None value will disable retries.
-            A google.api_core.retry.Retry value will enable retries, and the object will
-            define retriable response codes and errors and configure backoff and timeout options.
-
-            A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a Retry object and
-            activates it only if certain conditions are met. This class exists to provide safe defaults
-            for RPC calls that are not technically safe to retry normally (due to potential data
-            duplication or other side-effects) but become safe to retry if a condition such as
-            if_metageneration_match is set.
-
-            See the retry.py source code and docstrings in this package (google.cloud.storage.retry) for
-            information on retry types and how to configure them.
+        :param retry:
+            (Optional) How to retry the RPC. See: :ref:`configuring_retries`
         """
         client = self._require_client(client)
 

google/cloud/storage/acl.py

@@ -219,11 +219,9 @@ def _ensure_loaded(self, timeout=_DEFAULT_TIMEOUT):
         """Load if not already loaded.
 
         :type timeout: float or tuple
-        :param timeout: (Optional) The amount of time, in seconds, to wait
-            for the server response.
-
-            Can also be passed as a tuple (connect_timeout, read_timeout).
-            See :meth:`requests.Session.request` documentation for details.
+        :param timeout:
+            (Optional) The amount of time, in seconds, to wait
+            for the server response.  See: :ref:`configuring_timeouts`
         """
         if not self.loaded:
             self.reload(timeout=timeout)
@@ -442,20 +440,13 @@ def reload(self, client=None, timeout=_DEFAULT_TIMEOUT, retry=DEFAULT_RETRY):
         :param client: (Optional) The client to use.  If not passed, falls back
                        to the ``client`` stored on the ACL's parent.
         :type timeout: float or tuple
-        :param timeout: (Optional) The amount of time, in seconds, to wait
-            for the server response.
-
-            Can also be passed as a tuple (connect_timeout, read_timeout).
-            See :meth:`requests.Session.request` documentation for details.
+        :param timeout:
+            (Optional) The amount of time, in seconds, to wait
+            for the server response.  See: :ref:`configuring_timeouts`
 
         :type retry: :class:`~google.api_core.retry.Retry`
-        :param retry: (Optional) How to retry the RPC.
-
-            A None value will disable retries.
-
-            A google.api_core.retry.Retry value will enable retries,
-            and the object will define retriable response codes and errors
-            and configure backoff and timeout options.
+        :param retry:
+            (Optional) How to retry the RPC. See: :ref:`configuring_retries`
         """
         path = self.reload_path
         client = self._require_client(client)
@@ -489,21 +480,15 @@ def _save(self, acl, predefined, client, timeout=_DEFAULT_TIMEOUT):
                       ``NoneType``
         :param client: (Optional) The client to use.  If not passed, falls back
                        to the ``client`` stored on the ACL's parent.
-        :type timeout: float or tuple
-        :param timeout: (Optional) The amount of time, in seconds, to wait
-            for the server response.
 
-            Can also be passed as a tuple (connect_timeout, read_timeout).
-            See :meth:`requests.Session.request` documentation for details.
+        :type timeout: float or tuple
+        :param timeout:
+            (Optional) The amount of time, in seconds, to wait
+            for the server response.  See: :ref:`configuring_timeouts`
 
         :type retry: :class:`~google.api_core.retry.Retry`
-        :param retry: (Optional) How to retry the RPC.
-
-            A None value will disable retries.
-
-            A google.api_core.retry.Retry value will enable retries,
-            and the object will define retriable response codes and errors
-            and configure backoff and timeout options.
+        :param retry:
+            (Optional) How to retry the RPC. See: :ref:`configuring_retries`
         """
         client = self._require_client(client)
         query_params = {"projection": "full"}
@@ -545,12 +530,11 @@ def save(self, acl=None, client=None, timeout=_DEFAULT_TIMEOUT):
                       ``NoneType``
         :param client: (Optional) The client to use.  If not passed, falls back
                        to the ``client`` stored on the ACL's parent.
-        :type timeout: float or tuple
-        :param timeout: (Optional) The amount of time, in seconds, to wait
-            for the server response.
 
-            Can also be passed as a tuple (connect_timeout, read_timeout).
-            See :meth:`requests.Session.request` documentation for details.
+        :type timeout: float or tuple
+        :param timeout:
+            (Optional) The amount of time, in seconds, to wait
+            for the server response.  See: :ref:`configuring_timeouts`
         """
         if acl is None:
             acl = self
@@ -577,12 +561,11 @@ def save_predefined(self, predefined, client=None, timeout=_DEFAULT_TIMEOUT):
                       ``NoneType``
         :param client: (Optional) The client to use.  If not passed, falls back
                        to the ``client`` stored on the ACL's parent.
-        :type timeout: float or tuple
-        :param timeout: (Optional) The amount of time, in seconds, to wait
-            for the server response.
 
-            Can also be passed as a tuple (connect_timeout, read_timeout).
-            See :meth:`requests.Session.request` documentation for details.
+        :type timeout: float or tuple
+        :param timeout:
+            (Optional) The amount of time, in seconds, to wait
+            for the server response.  See: :ref:`configuring_timeouts`
         """
         predefined = self.validate_predefined(predefined)
         self._save(None, predefined, client, timeout=timeout)
@@ -601,12 +584,11 @@ def clear(self, client=None, timeout=_DEFAULT_TIMEOUT):
                       ``NoneType``
         :param client: (Optional) The client to use.  If not passed, falls back
                        to the ``client`` stored on the ACL's parent.
-        :type timeout: float or tuple
-        :param timeout: (Optional) The amount of time, in seconds, to wait
-            for the server response.
 
-            Can also be passed as a tuple (connect_timeout, read_timeout).
-            See :meth:`requests.Session.request` documentation for details.
+        :type timeout: float or tuple
+        :param timeout:
+            (Optional) The amount of time, in seconds, to wait
+            for the server response.  See: :ref:`configuring_timeouts`
         """
         self.save([], client=client, timeout=timeout)