Skip to content

Commit 524c5f1

Browse files
arithmetic1728beccasaurusgguusskurtisvgdpebot
authored
chore: move samples from python-docs-sample to here (#29)
* Data Labeling Beta samples [(#2096)](#2096) * add files * upate create_annotation_spec_set and test * add requirements.txt * update create_instruction and test * update import data and test * add label image and test * add label_text test * add label_video_test * add manage dataset and tests * flake * fix * add README * Adds updates including compute [(#2436)](#2436) * Adds updates including compute * Python 2 compat pytest * Fixing weird \r\n issue from GH merge * Put asset tests back in * Re-add pod operator test * Hack parameter for k8s pod operator * Update datalabeling samples to hit test endpoint. [(#2641)](#2641) * Auto-update dependencies. [(#2005)](#2005) * Auto-update dependencies. * Revert update of appengine/flexible/datastore. * revert update of appengine/flexible/scipy * revert update of bigquery/bqml * revert update of bigquery/cloud-client * revert update of bigquery/datalab-migration * revert update of bigtable/quickstart * revert update of compute/api * revert update of container_registry/container_analysis * revert update of dataflow/run_template * revert update of datastore/cloud-ndb * revert update of dialogflow/cloud-client * revert update of dlp * revert update of functions/imagemagick * revert update of functions/ocr/app * revert update of healthcare/api-client/fhir * revert update of iam/api-client * revert update of iot/api-client/gcs_file_to_device * revert update of iot/api-client/mqtt_example * revert update of language/automl * revert update of run/image-processing * revert update of vision/automl * revert update testing/requirements.txt * revert update of vision/cloud-client/detect * revert update of vision/cloud-client/product_search * revert update of jobs/v2/api_client * revert update of jobs/v3/api_client * revert update of opencensus * revert update of translate/cloud-client * revert update to speech/cloud-client Co-authored-by: Kurtis Van Gent <[email protected]> Co-authored-by: Doug Mahugh <[email protected]> * Update datalabeling to match lint. [(#2642)](#2642) * datalabeling: ensure all tests use test endpoint [(#2918)](#2918) * datalabeling: ensure all tests use test endpoint * requires an input csv for text input, slight print statement cleanup Co-authored-by: Leah E. Cole <[email protected]> * chore(deps): update dependency google-cloud-datalabeling to v0.4.0 [(#3081)](#3081) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-datalabeling](https://togithub.com/googleapis/python-datalabeling) | minor | `==0.3.0` -> `==0.4.0` | --- ### Release Notes <details> <summary>googleapis/python-datalabeling</summary> ### [`v0.4.0`](https://togithub.com/googleapis/python-datalabeling/blob/master/CHANGELOG.md#&#8203;040-httpswwwgithubcomgoogleapispython-datalabelingcomparev030v040-2020-01-31) [Compare Source](https://togithub.com/googleapis/python-datalabeling/compare/v0.3.0...v0.4.0) ##### Features - **datalabeling:** undeprecate resource name helper methods (via synth) ([#&#8203;10039](https://www.github.com/googleapis/python-datalabeling/issues/10039)) ([88f8090](https://www.github.com/googleapis/python-datalabeling/commit/88f809008ee6a709c02c78b1d93af779fab19adb)) ##### Bug Fixes - **datalabeling:** deprecate resource name helper methods (via synth) ([#&#8203;9832](https://www.github.com/googleapis/python-datalabeling/issues/9832)) ([e5f9021](https://www.github.com/googleapis/python-datalabeling/commit/e5f902154ebe7fcb139aa405babfe9993fd51319)) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples). * Simplify noxfile setup. [(#2806)](#2806) * chore(deps): update dependency requests to v2.23.0 * Simplify noxfile and add version control. * Configure appengine/standard to only test Python 2.7. * Update Kokokro configs to match noxfile. * Add requirements-test to each folder. * Remove Py2 versions from everything execept appengine/standard. * Remove conftest.py. * Remove appengine/standard/conftest.py * Remove 'no-sucess-flaky-report' from pytest.ini. * Add GAE SDK back to appengine/standard tests. * Fix typo. * Roll pytest to python 2 version. * Add a bunch of testing requirements. * Remove typo. * Add appengine lib directory back in. * Add some additional requirements. * Fix issue with flake8 args. * Even more requirements. * Readd appengine conftest.py. * Add a few more requirements. * Even more Appengine requirements. * Add webtest for appengine/standard/mailgun. * Add some additional requirements. * Add workaround for issue with mailjet-rest. * Add responses for appengine/standard/mailjet. Co-authored-by: Renovate Bot <[email protected]> * testing: mark some tests as flaky [(#3288)](#3288) fixes #3138 * [datalabeling] testing: wrap rpcs with backoff [(#3443)](#3443) * wrap all the rpcs with backoff * add a shared testing lib * remove flaky * [datalabeling] fix: clean up old datasets before the test [(#3707)](#3707) fixes #3710 fixes #3711 * [datalabeling] testing: retry upon ServerError [(#3762)](#3762) fixes #3760 * Replace GCLOUD_PROJECT with GOOGLE_CLOUD_PROJECT. [(#4022)](#4022) * chore(deps): update dependency pytest to v5.4.3 [(#4279)](#4279) * chore(deps): update dependency pytest to v5.4.3 * specify pytest for python 2 in appengine Co-authored-by: Leah Cole <[email protected]> * Update dependency pytest to v6 [(#4390)](#4390) * chore: update templates * chore: fix docs error * chore: skip unavailable samples * chore: use staging endpoint for labeling tests Co-authored-by: Rebecca Taylor <[email protected]> Co-authored-by: Gus Class <[email protected]> Co-authored-by: Kurtis Van Gent <[email protected]> Co-authored-by: DPEBot <[email protected]> Co-authored-by: Doug Mahugh <[email protected]> Co-authored-by: Noah Negrey <[email protected]> Co-authored-by: Leah E. Cole <[email protected]> Co-authored-by: WhiteSource Renovate <[email protected]> Co-authored-by: Takashi Matsuo <[email protected]> Co-authored-by: Leah Cole <[email protected]> Co-authored-by: Bu Sun Kim <[email protected]>
0 parents  commit 524c5f1

24 files changed

+1835
-0
lines changed

datalabeling/AUTHORING_GUIDE.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
See https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md

datalabeling/CONTRIBUTING.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
See https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/CONTRIBUTING.md

datalabeling/snippets/README.rst

+78
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
.. This file is automatically generated. Do not edit this file directly.
2+
3+
Google Cloud Data Labeling Service Python Samples
4+
===============================================================================
5+
6+
.. image:: https://gstatic.com/cloudssh/images/open-btn.png
7+
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=datalabeling/README.rst
8+
9+
10+
This directory contains samples for Google Cloud Data Labeling Service. `Google Cloud Data Labeling Service`_ allows developers to request having human labelers label a collection of data that you plan to use to train a custom machine learning model.
11+
12+
13+
14+
15+
.. _Google Cloud Data Labeling Service: https://cloud.google.com/data-labeling/docs/
16+
17+
Setup
18+
-------------------------------------------------------------------------------
19+
20+
21+
Authentication
22+
++++++++++++++
23+
24+
This sample requires you to have authentication setup. Refer to the
25+
`Authentication Getting Started Guide`_ for instructions on setting up
26+
credentials for applications.
27+
28+
.. _Authentication Getting Started Guide:
29+
https://cloud.google.com/docs/authentication/getting-started
30+
31+
Install Dependencies
32+
++++++++++++++++++++
33+
34+
#. Clone python-docs-samples and change directory to the sample directory you want to use.
35+
36+
.. code-block:: bash
37+
38+
$ git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
39+
40+
#. Install `pip`_ and `virtualenv`_ if you do not already have them. You may want to refer to the `Python Development Environment Setup Guide`_ for Google Cloud Platform for instructions.
41+
42+
.. _Python Development Environment Setup Guide:
43+
https://cloud.google.com/python/setup
44+
45+
#. Create a virtualenv. Samples are compatible with Python 2.7 and 3.4+.
46+
47+
.. code-block:: bash
48+
49+
$ virtualenv env
50+
$ source env/bin/activate
51+
52+
#. Install the dependencies needed to run the samples.
53+
54+
.. code-block:: bash
55+
56+
$ pip install -r requirements.txt
57+
58+
.. _pip: https://pip.pypa.io/
59+
.. _virtualenv: https://virtualenv.pypa.io/
60+
61+
62+
63+
The client library
64+
-------------------------------------------------------------------------------
65+
66+
This sample uses the `Google Cloud Client Library for Python`_.
67+
You can read the documentation for more details on API usage and use GitHub
68+
to `browse the source`_ and `report issues`_.
69+
70+
.. _Google Cloud Client Library for Python:
71+
https://googlecloudplatform.github.io/google-cloud-python/
72+
.. _browse the source:
73+
https://github.com/GoogleCloudPlatform/google-cloud-python
74+
.. _report issues:
75+
https://github.com/GoogleCloudPlatform/google-cloud-python/issues
76+
77+
78+
.. _Google Cloud SDK: https://cloud.google.com/sdk/

datalabeling/snippets/README.rst.in

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# This file is used to generate README.rst
2+
3+
product:
4+
name: Google Cloud Data Labeling Service
5+
short_name: Cloud Data Labeling
6+
url: https://cloud.google.com/data-labeling/docs/
7+
description: >
8+
`Google Cloud Data Labeling Service`_ allows developers to request having
9+
human labelers label a collection of data that you plan to use to train a
10+
custom machine learning model.
11+
12+
setup:
13+
- auth
14+
- install_deps
15+
16+
cloud_client_library: true
17+
18+
folder: datalabeling
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
#!/usr/bin/env python
2+
3+
# Copyright 2019 Google LLC
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
import argparse
18+
import os
19+
20+
from google.api_core.client_options import ClientOptions
21+
22+
23+
# [START datalabeling_create_annotation_spec_set_beta]
24+
def create_annotation_spec_set(project_id):
25+
"""Creates a data labeling annotation spec set for the given
26+
Google Cloud project.
27+
"""
28+
from google.cloud import datalabeling_v1beta1 as datalabeling
29+
client = datalabeling.DataLabelingServiceClient()
30+
# [END datalabeling_create_annotation_spec_set_beta]
31+
# If provided, use a provided test endpoint - this will prevent tests on
32+
# this snippet from triggering any action by a real human
33+
if 'DATALABELING_ENDPOINT' in os.environ:
34+
opts = ClientOptions(api_endpoint=os.getenv('DATALABELING_ENDPOINT'))
35+
client = datalabeling.DataLabelingServiceClient(client_options=opts)
36+
# [START datalabeling_create_annotation_spec_set_beta]
37+
38+
project_path = client.project_path(project_id)
39+
40+
annotation_spec_1 = datalabeling.types.AnnotationSpec(
41+
display_name='label_1',
42+
description='label_description_1'
43+
)
44+
45+
annotation_spec_2 = datalabeling.types.AnnotationSpec(
46+
display_name='label_2',
47+
description='label_description_2'
48+
)
49+
50+
annotation_spec_set = datalabeling.types.AnnotationSpecSet(
51+
display_name='YOUR_ANNOTATION_SPEC_SET_DISPLAY_NAME',
52+
description='YOUR_DESCRIPTION',
53+
annotation_specs=[annotation_spec_1, annotation_spec_2]
54+
)
55+
56+
response = client.create_annotation_spec_set(
57+
project_path, annotation_spec_set)
58+
59+
# The format of the resource name:
60+
# project_id/{project_id}/annotationSpecSets/{annotationSpecSets_id}
61+
print('The annotation_spec_set resource name: {}'.format(response.name))
62+
print('Display name: {}'.format(response.display_name))
63+
print('Description: {}'.format(response.description))
64+
print('Annotation specs:')
65+
for annotation_spec in response.annotation_specs:
66+
print('\tDisplay name: {}'.format(annotation_spec.display_name))
67+
print('\tDescription: {}\n'.format(annotation_spec.description))
68+
69+
return response
70+
# [END datalabeling_create_annotation_spec_set_beta]
71+
72+
73+
if __name__ == '__main__':
74+
parser = argparse.ArgumentParser(
75+
description=__doc__,
76+
formatter_class=argparse.RawDescriptionHelpFormatter
77+
)
78+
79+
parser.add_argument(
80+
'--project-id',
81+
help='Project ID. Required.',
82+
required=True
83+
)
84+
85+
args = parser.parse_args()
86+
87+
create_annotation_spec_set(args.project_id)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#!/usr/bin/env python
2+
3+
# Copyright 2019 Google, Inc
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
import os
18+
19+
import backoff
20+
from google.api_core.exceptions import ServerError
21+
import pytest
22+
23+
import create_annotation_spec_set
24+
import testing_lib
25+
26+
27+
PROJECT_ID = os.getenv('GOOGLE_CLOUD_PROJECT')
28+
29+
30+
@pytest.fixture(scope='module')
31+
def cleaner():
32+
resource_names = []
33+
34+
yield resource_names
35+
36+
for resource_name in resource_names:
37+
testing_lib.delete_annotation_spec_set(resource_name)
38+
39+
40+
def test_create_annotation_spec_set(cleaner, capsys):
41+
42+
@backoff.on_exception(
43+
backoff.expo, ServerError, max_time=testing_lib.RETRY_DEADLINE)
44+
def run_sample():
45+
return create_annotation_spec_set.create_annotation_spec_set(PROJECT_ID)
46+
47+
response = run_sample()
48+
49+
# For cleanup
50+
cleaner.append(response.name)
51+
52+
out, _ = capsys.readouterr()
53+
assert 'The annotation_spec_set resource name:' in out
+103
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
#!/usr/bin/env python
2+
3+
# Copyright 2019 Google LLC
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
import argparse
18+
import os
19+
20+
from google.api_core.client_options import ClientOptions
21+
22+
23+
# [START datalabeling_create_instruction_beta]
24+
def create_instruction(project_id, data_type, instruction_gcs_uri):
25+
""" Creates a data labeling PDF instruction for the given Google Cloud
26+
project. The PDF file should be uploaded to the project in
27+
Google Cloud Storage.
28+
"""
29+
from google.cloud import datalabeling_v1beta1 as datalabeling
30+
client = datalabeling.DataLabelingServiceClient()
31+
# [END datalabeling_create_instruction_beta]
32+
# If provided, use a provided test endpoint - this will prevent tests on
33+
# this snippet from triggering any action by a real human
34+
if 'DATALABELING_ENDPOINT' in os.environ:
35+
opts = ClientOptions(api_endpoint=os.getenv('DATALABELING_ENDPOINT'))
36+
client = datalabeling.DataLabelingServiceClient(client_options=opts)
37+
# [START datalabeling_create_instruction_beta]
38+
39+
project_path = client.project_path(project_id)
40+
41+
pdf_instruction = datalabeling.types.PdfInstruction(
42+
gcs_file_uri=instruction_gcs_uri)
43+
44+
instruction = datalabeling.types.Instruction(
45+
display_name='YOUR_INSTRUCTION_DISPLAY_NAME',
46+
description='YOUR_DESCRIPTION',
47+
data_type=data_type,
48+
pdf_instruction=pdf_instruction
49+
)
50+
51+
operation = client.create_instruction(project_path, instruction)
52+
53+
result = operation.result()
54+
55+
# The format of the resource name:
56+
# project_id/{project_id}/instruction/{instruction_id}
57+
print('The instruction resource name: {}'.format(result.name))
58+
print('Display name: {}'.format(result.display_name))
59+
print('Description: {}'.format(result.description))
60+
print('Create time:')
61+
print('\tseconds: {}'.format(result.create_time.seconds))
62+
print('\tnanos: {}'.format(result.create_time.nanos))
63+
print('Data type: {}'.format(
64+
datalabeling.enums.DataType(result.data_type).name))
65+
print('Pdf instruction:')
66+
print('\tGcs file uri: {}\n'.format(
67+
result.pdf_instruction.gcs_file_uri))
68+
69+
return result
70+
# [END datalabeling_create_instruction_beta]
71+
72+
73+
if __name__ == '__main__':
74+
parser = argparse.ArgumentParser(
75+
description=__doc__,
76+
formatter_class=argparse.RawDescriptionHelpFormatter
77+
)
78+
79+
parser.add_argument(
80+
'--project-id',
81+
help='Project ID. Required.',
82+
required=True
83+
)
84+
85+
parser.add_argument(
86+
'--data-type',
87+
help='Data type. Only support IMAGE, VIDEO, TEXT and AUDIO. Required.',
88+
required=True
89+
)
90+
91+
parser.add_argument(
92+
'--instruction-gcs-uri',
93+
help='The URI of Google Cloud Storage of the instruction. Required.',
94+
required=True
95+
)
96+
97+
args = parser.parse_args()
98+
99+
create_instruction(
100+
args.project_id,
101+
args.data_type,
102+
args.instruction_gcs_uri
103+
)

0 commit comments

Comments
 (0)