Skip to content

🐛 Normalization: Fix sync from HubSpot to MySQL fails with "Row size too large" on create table #10485

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 22, 2022
Merged

🐛 Normalization: Fix sync from HubSpot to MySQL fails with "Row size too large" on create table #10485

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
62bbc0a
Update mysql normalization to cast string as text.
htrueman Feb 20, 2022
919dfee
Update docs PR reference
htrueman Feb 20, 2022
72c3d92
Update mysql normalization to cast string as for is_timestamp_with_ti…
htrueman Feb 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/bases/base-normalization/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,5 @@ WORKDIR /airbyte
ENV AIRBYTE_ENTRYPOINT "/airbyte/entrypoint.sh"
ENTRYPOINT ["/airbyte/entrypoint.sh"]

LABEL io.airbyte.version=0.1.67
LABEL io.airbyte.version=0.1.68
LABEL io.airbyte.name=airbyte/normalization
5 changes: 5 additions & 0 deletions ...integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,8 @@ def cast_property_type(self, property_name: str, column_name: str, jinja_column:
return f"parseDateTime64BestEffortOrNull(trim(BOTH '\"' from {replace_operation})) as {column_name}"
# in all other cases
sql_type = jinja_call("type_timestamp_with_timezone()")
if self.destination_type == DestinationType.MYSQL:
sql_type = f"{sql_type}(1024)"
return f"cast({replace_operation} as {sql_type}) as {column_name}"
elif is_date(definition):
if self.destination_type.value == DestinationType.MYSQL.value:
Expand All @@ -538,6 +540,9 @@ def cast_property_type(self, property_name: str, column_name: str, jinja_column:
trimmed_column_name = f"trim(BOTH '\"' from {column_name})"
sql_type = f"'{sql_type}'"
return f"nullif(accurateCastOrNull({trimmed_column_name}, {sql_type}), 'null') as {column_name}"
elif self.destination_type == DestinationType.MYSQL:
# Cast to `text` datatype. See https://github.com/airbytehq/airbyte/issues/7994
sql_type = f"{sql_type}(1024)"
Comment on lines +543 to +545
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proper way of changing a datatype for a certain destination should be done in dbt macros, not directly in the python code:

see

airbyte/airbyte-integrations/bases/base-normalization/dbt-project-template/macros/cross_db_utils/datatypes.sql

Line 43 in c7b8db7

char

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be causing this error?

Failure Origin: normalization, Message: Normalization failed during the dbt run. This may indicate a problem with the data itself.

1292 (22007): Truncated incorrect CHAR(1024) value:

else:
print(f"WARN: Unknown type {definition['type']} for column {property_name} at {self.current_json_path()}")
return column_name
Expand Down
2 changes: 1 addition & 1 deletion ...te-workers/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
public class NormalizationRunnerFactory {

public static final String BASE_NORMALIZATION_IMAGE_NAME = "airbyte/normalization";
public static final String NORMALIZATION_VERSION = "0.1.67";
public static final String NORMALIZATION_VERSION = "0.1.68";

static final Map<String, ImmutablePair<String, DefaultNormalizationRunner.DestinationType>> NORMALIZATION_MAPPING =
ImmutableMap.<String, ImmutablePair<String, DefaultNormalizationRunner.DestinationType>>builder()
Expand Down
1 change: 1 addition & 0 deletions docs/understanding-airbyte/basic-normalization.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,6 +350,7 @@ Therefore, in order to "upgrade" to the desired normalization version, you need

| Airbyte Version | Normalization Version | Date | Pull Request | Subject |
|:----------------| :--- | :--- | :--- | :--- |
| 0.35.32-alpha | 0.1.68 | 2022-02-20 | [\#10485](https://github.com/airbytehq/airbyte/pull/10485) | Fix row size too large for table with numerous `string` fields |
| | 0.1.66 | 2022-02-04 | [\#9341](https://github.com/airbytehq/airbyte/pull/9341) | Fix normalization for bigquery datasetId and tables |
| 0.35.13-alpha | 0.1.65 | 2021-01-28 | [\#9846](https://github.com/airbytehq/airbyte/pull/9846) | Tweak dbt multi-thread parameter down |
| 0.35.12-alpha | 0.1.64 | 2021-01-28 | [\#9793](https://github.com/airbytehq/airbyte/pull/9793) | Support PEM format for ssh-tunnel keys |
Expand Down