Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add compression option ZSTD. #1890

Merged
merged 13 commits into from
Apr 11, 2024
Merged

feat: Add compression option ZSTD. #1890

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a61a82e
Add ZSTD to compression types
EthanSteinberg Apr 4, 2024
df23c7d
feat: adds tests re Compression types
chalmerlowe Apr 8, 2024
b26a974
revise datatype from Enum to object
chalmerlowe Apr 8, 2024
b801aaf
adds license text and docstring
chalmerlowe Apr 8, 2024
f6da457
change object back to enum datatype
chalmerlowe Apr 10, 2024
b57dd66
updates compression object comparison
chalmerlowe Apr 10, 2024
867e5bc
updates Compression class
chalmerlowe Apr 10, 2024
3b0f4b0
jsonify and sort the input and output for testing
chalmerlowe Apr 11, 2024
1af9900
Merge branch 'main' into patch-1
chalmerlowe Apr 11, 2024
594a2a9
Update tests/unit/job/test_extract.py
tswast Apr 11, 2024
e685c2d
moved json import statement
chalmerlowe Apr 11, 2024
37e7f53
removed enums test and file
chalmerlowe Apr 11, 2024
9e1bb36
Merge branch 'main' into patch-1
chalmerlowe Apr 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion google/cloud/bigquery/enums.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ class AutoRowIDs(enum.Enum):
GENERATE_UUID = enum.auto()


class Compression(object):
class Compression(str, enum.Enum):
"""The compression type to use for exported files. The default value is
:attr:`NONE`.

Expand All @@ -39,6 +39,9 @@ class Compression(object):
SNAPPY = "SNAPPY"
"""Specifies SNAPPY format."""

ZSTD = "ZSTD"
"""Specifies ZSTD format."""

NONE = "NONE"
"""Specifies no compression."""

Expand Down
2 changes: 1 addition & 1 deletion tests/unit/job/test_extract.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ def test_to_api_repr(self):
from google.cloud.bigquery import job

config = self._make_one()
config.compression = job.Compression.SNAPPY
config.compression = job.Compression.SNAPPY.value
config.destination_format = job.DestinationFormat.AVRO
config.field_delimiter = "ignored for avro"
config.print_header = False
Expand Down
27 changes: 27 additions & 0 deletions tests/unit/test_enums.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


def test_compression_enums():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder how useful this test is? Seems an awful lot like a change-detector test to me. I'd be fine adding the constant without adding the test.

Alternatively, maybe there's a system test we could write to make sure this is synced with the bigquery discovery document? But even then, compression isn't a true enum. The allowed values are only listed in the documentation string from what I can tell.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tswast

I will remove this test.

In terms of attempting to ensure that this matches the discovery docs... the terms referenced by the docs are present in the description, as you note, so we would need some mechanism to extract them from the discovery doc, which feels somewhat fragile (ie extract all words that are ALL CAPS and deduplicate them). Thoughts?

"JobConfigurationExtract": {
      ...
      "properties": {
        "compression": {
          "description": "Optional. The compression type to use for exported files.
Possible values include DEFLATE, GZIP, NONE, SNAPPY, and ZSTD. The
default value is NONE. Not all compression formats are support for all
file formats. DEFLATE is only supported for Avro. ZSTD is only supported
for Parquet. Not applicable when extracting models.",
          "type": "string"
        },

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Without a structured representation of the allowed values, it's too fragile.

"""Ensures that the compression enum has all the allowed types of
compression algorithms present.
"""

from google.cloud.bigquery.enums import Compression

expected_comps_sorted = ["DEFLATE", "GZIP", "NONE", "SNAPPY", "ZSTD"]

result_comps_sorted = sorted(comp.value for comp in Compression)

assert result_comps_sorted == expected_comps_sorted