You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
BigQuery Destination : Fix GCS processing of Facebook data (#9415)
* Fix GCS Avro file processing with invalid "-" character
* Extend test data to cover the case
* incr ver
* s3 ver upd
* add dependency
* add dependency
Copy file name to clipboardExpand all lines: airbyte-config/init/src/main/resources/config/STANDARD_DESTINATION_DEFINITION/079d5540-f236-4294-ba7c-ade8fd918496.json
Copy file name to clipboardExpand all lines: airbyte-config/init/src/main/resources/config/STANDARD_DESTINATION_DEFINITION/22f6c74f-5699-40ff-833c-4a879ea40133.json
Copy file name to clipboardExpand all lines: airbyte-config/init/src/main/resources/config/STANDARD_DESTINATION_DEFINITION/4816b78f-1489-44c1-9060-4b19d5fa9362.json
Copy file name to clipboardExpand all lines: airbyte-integrations/connectors/destination-bigquery-denormalized/src/test-integration/java/io/airbyte/integrations/destination/bigquery/util/BigQueryDenormalizedTestDataUtils.java
+4-4
Original file line number
Diff line number
Diff line change
@@ -29,7 +29,7 @@ public static JsonNode getSchema() {
29
29
+ " \"string\"\n"
30
30
+ " ]\n"
31
31
+ " },\n"
32
-
+ " \"permissions\": {\n"
32
+
+ " \"permission-list\": {\n"
33
33
+ " \"type\": [\n"
34
34
+ " \"array\"\n"
35
35
+ " ],\n"
@@ -135,7 +135,7 @@ public static JsonNode getSchemaWithInvalidArrayType() {
135
135
+ " \"string\"\n"
136
136
+ " ]\n"
137
137
+ " },\n"
138
-
+ " \"permissions\": {\n"
138
+
+ " \"permission-list\": {\n"
139
139
+ " \"type\": [\n"
140
140
+ " \"array\"\n"
141
141
+ " ],\n"
@@ -167,7 +167,7 @@ public static JsonNode getData() {
Copy file name to clipboardExpand all lines: airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/AvroNameTransformer.java
Copy file name to clipboardExpand all lines: airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonToAvroSchemaConverter.java
+1-1
Original file line number
Diff line number
Diff line change
@@ -175,7 +175,7 @@ public Schema getAvroSchema(final JsonNode jsonSchema,
175
175
// Omit the namespace for root level fields, because it is directly assigned in the builder above.
Copy file name to clipboardExpand all lines: docs/integrations/destinations/bigquery.md
+15-13
Original file line number
Diff line number
Diff line change
@@ -153,6 +153,7 @@ Therefore, Airbyte BigQuery destination will convert any invalid characters into
153
153
154
154
| Version | Date | Pull Request | Subject |
155
155
|:--------| :--- | :--- | :--- |
156
+
| 0.6.3 | 2022-01-12 | [\#9415](https://github.com/airbytehq/airbyte/pull/9415) | BigQuery Destination : Fix GCS processing of Facebook data |
156
157
| 0.6.2 | 2022-01-10 | [\#9121](https://github.com/airbytehq/airbyte/pull/9121) | Fixed check method for GCS mode to verify if all roles assigned to user |
157
158
| 0.6.1 | 2021-12-22 | [\#9039](https://github.com/airbytehq/airbyte/pull/9039) | Added part_size configuration to UI for GCS staging |
158
159
| 0.6.0 | 2021-12-17 | [\#8788](https://github.com/airbytehq/airbyte/issues/8788) | BigQuery/BiqQuery denorm Destinations : Add possibility to use different types of GCS files |
@@ -169,21 +170,22 @@ Therefore, Airbyte BigQuery destination will convert any invalid characters into
169
170
170
171
### bigquery-denormalized
171
172
172
-
| Version | Date | Pull Request | Subject |
173
-
|:--------| :--- | :--- | :--- |
174
-
| 0.2.2 | 2021-12-22 | [\#9039](https://github.com/airbytehq/airbyte/pull/9039) | Added part_size configuration to UI for GCS staging |
175
-
| 0.2.1 | 2021-12-21 | [\#8574](https://github.com/airbytehq/airbyte/pull/8574) | Added namespace to Avro and Parquet record types |
176
-
| 0.2.0 | 2021-12-17 | [\#8788](https://github.com/airbytehq/airbyte/pull/8788) | BigQuery/BiqQuery denorm Destinations : Add possibility to use different types of GCS files |
| 0.1.4 | 2021-09-04 | [\#5813](https://github.com/airbytehq/airbyte/pull/5813) | fix Stackoverflow error when receive a schema from source where "Array" type doesn't contain a required "items" element |
185
-
| 0.1.3 | 2021-08-07 | [\#5261](https://github.com/airbytehq/airbyte/pull/5261) | 🐛 Destination BigQuery\(Denormalized\): Fix processing arrays of records |
| 0.1.4 | 2021-09-04 | [\#5813](https://github.com/airbytehq/airbyte/pull/5813) | fix Stackoverflow error when receive a schema from source where "Array" type doesn't contain a required "items" element |
187
+
| 0.1.3 | 2021-08-07 | [\#5261](https://github.com/airbytehq/airbyte/pull/5261) | 🐛 Destination BigQuery\(Denormalized\): Fix processing arrays of records |
Copy file name to clipboardExpand all lines: docs/integrations/destinations/s3.md
+23-22
Original file line number
Diff line number
Diff line change
@@ -222,26 +222,27 @@ Under the hood, an Airbyte data stream in Json schema is first converted to an A
222
222
## CHANGELOG
223
223
224
224
| Version | Date | Pull Request | Subject |
225
-
| :--- | :--- | :--- | :--- |
226
-
| 0.2.3 | 2022-01-11 |[\#9367](https://github.com/airbytehq/airbyte/pull/9367)| Avro & Parquet: support array field with unknown item type; default any improperly typed field to string. |
227
-
| 0.2.2 | 2021-12-21 |[\#8574](https://github.com/airbytehq/airbyte/pull/8574)| Added namespace to Avro and Parquet record types |
228
-
| 0.2.1 | 2021-12-20 |[\#8974](https://github.com/airbytehq/airbyte/pull/8974)| Release a new version to ensure there is no excessive logging. |
229
-
| 0.2.0 | 2021-12-15 |[\#8607](https://github.com/airbytehq/airbyte/pull/8607)| Change the output filename for CSV files - it's now `bucketPath/namespace/streamName/timestamp_epochMillis_randomUuid.csv`|
230
-
| 0.1.16 | 2021-12-10 |[\#8562](https://github.com/airbytehq/airbyte/pull/8562)| Swap dependencies with destination-jdbc. |
231
-
| 0.1.15 | 2021-12-03 |[\#8501](https://github.com/airbytehq/airbyte/pull/8501)| Remove excessive logging for Avro and Parquet invalid date strings. |
232
-
| 0.1.14 | 2021-11-09 |[\#7732](https://github.com/airbytehq/airbyte/pull/7732)| Support timestamp in Avro and Parquet |
233
-
| 0.1.13 | 2021-11-03 |[\#7288](https://github.com/airbytehq/airbyte/issues/7288)| Support Json `additionalProperties`. |
234
-
| 0.1.12 | 2021-09-13 |[\#5720](https://github.com/airbytehq/airbyte/issues/5720)| Added configurable block size for stream. Each stream is limited to 10,000 by S3 |
235
-
| 0.1.11 | 2021-09-10 |[\#5729](https://github.com/airbytehq/airbyte/pull/5729)| For field names that start with a digit, a `_` will be appended at the beginning for the`Parquet` and `Avro` formats. |
| 0.1.3 | 2021-06-13 |[\#4038](https://github.com/airbytehq/airbyte/pull/4038)| Added support for alternative S3. |
244
-
| 0.1.2 | 2021-06-10 |[\#4029](https://github.com/airbytehq/airbyte/pull/4029)| Fixed `_airbyte_emitted_at` field to be a UTC instead of local timestamp for consistency. |
245
-
| 0.1.1 | 2021-06-09 |[\#3973](https://github.com/airbytehq/airbyte/pull/3973)| Added `AIRBYTE_ENTRYPOINT` in base Docker image for Kubernetes support. |
| 0.2.4 | 2022-01-12 |[\#9415](https://github.com/airbytehq/airbyte/pull/9415)| BigQuery Destination : Fix GCS processing of Facebook data |
227
+
| 0.2.3 | 2022-01-11 |[\#9367](https://github.com/airbytehq/airbyte/pull/9367)| Avro & Parquet: support array field with unknown item type; default any improperly typed field to string. |
228
+
| 0.2.2 | 2021-12-21 |[\#8574](https://github.com/airbytehq/airbyte/pull/8574)| Added namespace to Avro and Parquet record types |
229
+
| 0.2.1 | 2021-12-20 |[\#8974](https://github.com/airbytehq/airbyte/pull/8974)| Release a new version to ensure there is no excessive logging. |
230
+
| 0.2.0 | 2021-12-15 |[\#8607](https://github.com/airbytehq/airbyte/pull/8607)| Change the output filename for CSV files - it's now `bucketPath/namespace/streamName/timestamp_epochMillis_randomUuid.csv`|
231
+
| 0.1.16 | 2021-12-10 |[\#8562](https://github.com/airbytehq/airbyte/pull/8562)| Swap dependencies with destination-jdbc. |
232
+
| 0.1.15 | 2021-12-03 |[\#8501](https://github.com/airbytehq/airbyte/pull/8501)| Remove excessive logging for Avro and Parquet invalid date strings. |
233
+
| 0.1.14 | 2021-11-09 |[\#7732](https://github.com/airbytehq/airbyte/pull/7732)| Support timestamp in Avro and Parquet |
234
+
| 0.1.13 | 2021-11-03 |[\#7288](https://github.com/airbytehq/airbyte/issues/7288)| Support Json `additionalProperties`. |
235
+
| 0.1.12 | 2021-09-13 |[\#5720](https://github.com/airbytehq/airbyte/issues/5720)| Added configurable block size for stream. Each stream is limited to 10,000 by S3 |
236
+
| 0.1.11 | 2021-09-10 |[\#5729](https://github.com/airbytehq/airbyte/pull/5729)| For field names that start with a digit, a `_` will be appended at the beginning for the`Parquet` and `Avro` formats. |
| 0.1.3 | 2021-06-13 |[\#4038](https://github.com/airbytehq/airbyte/pull/4038)| Added support for alternative S3. |
245
+
| 0.1.2 | 2021-06-10 |[\#4029](https://github.com/airbytehq/airbyte/pull/4029)| Fixed `_airbyte_emitted_at` field to be a UTC instead of local timestamp for consistency. |
246
+
| 0.1.1 | 2021-06-09 |[\#3973](https://github.com/airbytehq/airbyte/pull/3973)| Added `AIRBYTE_ENTRYPOINT` in base Docker image for Kubernetes support. |
0 commit comments