Skip to content

Commit 1807c8f

Browse files
sergio-roperooctavia-squidington-iii
sergio-ropero
and
octavia-squidington-iii
authored
feat: Filter out system views out of system namespaces (#22427)
* This changes allows to filter out system views created out of system namespaces * Add extra view * Fix issue * Bump Postgres source version * bump version * Bump alloydb * Bump versioning * auto-bump connector version * Set default implementation and remove from unneeded places --------- Co-authored-by: Octavia Squidington III <[email protected]>
1 parent 3a3365a commit 1807c8f

File tree

11 files changed

+42
-22
lines changed

11 files changed

+42
-22
lines changed

‎airbyte-config/init/src/main/resources/seed/source_definitions.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@
4545
- name: AlloyDB for PostgreSQL
4646
sourceDefinitionId: 1fa90628-2b9e-11ed-a261-0242ac120002
4747
dockerRepository: airbyte/source-alloydb
48-
dockerImageTag: 1.0.43
48+
dockerImageTag: 1.0.44
4949
documentationUrl: https://docs.airbyte.com/integrations/sources/alloydb
5050
icon: alloydb.svg
5151
sourceType: database
@@ -1376,7 +1376,7 @@
13761376
- name: Postgres
13771377
sourceDefinitionId: decd338e-5647-4c0b-adf4-da0e75f5a750
13781378
dockerRepository: airbyte/source-postgres
1379-
dockerImageTag: 1.0.43
1379+
dockerImageTag: 1.0.44
13801380
documentationUrl: https://docs.airbyte.com/integrations/sources/postgres
13811381
icon: postgresql.svg
13821382
sourceType: database

‎airbyte-config/init/src/main/resources/seed/source_specs.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -370,7 +370,7 @@
370370
supportsNormalization: false
371371
supportsDBT: false
372372
supported_destination_sync_modes: []
373-
- dockerImage: "airbyte/source-alloydb:1.0.43"
373+
- dockerImage: "airbyte/source-alloydb:1.0.44"
374374
spec:
375375
documentationUrl: "https://docs.airbyte.com/integrations/sources/postgres"
376376
connectionSpecification:
@@ -11617,7 +11617,7 @@
1161711617
supportsNormalization: false
1161811618
supportsDBT: false
1161911619
supported_destination_sync_modes: []
11620-
- dockerImage: "airbyte/source-postgres:1.0.43"
11620+
- dockerImage: "airbyte/source-postgres:1.0.44"
1162111621
spec:
1162211622
documentationUrl: "https://docs.airbyte.com/integrations/sources/postgres"
1162311623
connectionSpecification:

‎airbyte-integrations/connectors/source-alloydb-strict-encrypt/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,5 +16,5 @@ ENV APPLICATION source-alloydb-strict-encrypt
1616

1717
COPY --from=build /airbyte /airbyte
1818

19-
LABEL io.airbyte.version=1.0.43
19+
LABEL io.airbyte.version=1.0.44
2020
LABEL io.airbyte.name=airbyte/source-alloydb-strict-encrypt

‎airbyte-integrations/connectors/source-alloydb/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,5 +16,5 @@ ENV APPLICATION source-alloydb
1616

1717
COPY --from=build /airbyte /airbyte
1818

19-
LABEL io.airbyte.version=1.0.43
19+
LABEL io.airbyte.version=1.0.44
2020
LABEL io.airbyte.name=airbyte/source-alloydb

‎airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,5 +16,5 @@ ENV APPLICATION source-postgres-strict-encrypt
1616

1717
COPY --from=build /airbyte /airbyte
1818

19-
LABEL io.airbyte.version=1.0.43
19+
LABEL io.airbyte.version=1.0.44
2020
LABEL io.airbyte.name=airbyte/source-postgres-strict-encrypt

‎airbyte-integrations/connectors/source-postgres/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,5 +16,5 @@ ENV APPLICATION source-postgres
1616

1717
COPY --from=build /airbyte /airbyte
1818

19-
LABEL io.airbyte.version=1.0.43
19+
LABEL io.airbyte.version=1.0.44
2020
LABEL io.airbyte.name=airbyte/source-postgres

‎airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,11 @@ public Set<String> getExcludedInternalNameSpaces() {
202202
return Set.of("information_schema", "pg_catalog", "pg_internal", "catalog_history");
203203
}
204204

205+
@Override
206+
protected Set<String> getExcludedViews() {
207+
return Set.of("pg_stat_statements", "pg_stat_statements_info");
208+
}
209+
205210
@Override
206211
public AirbyteCatalog discover(final JsonNode config) throws Exception {
207212
final AirbyteCatalog catalog = super.discover(config);

‎airbyte-integrations/connectors/source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresStressTest.java

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,11 @@ public Set<String> getExcludedInternalNameSpaces() {
123123
return Set.of("information_schema", "pg_catalog", "pg_internal", "catalog_history");
124124
}
125125

126+
@Override
127+
protected Set<String> getExcludedViews() {
128+
return Set.of("pg_stat_statements", "pg_stat_statements_info");
129+
}
130+
126131
public static void main(final String[] args) throws Exception {
127132
final Source source = new PostgresTestSource();
128133
LOGGER.info("starting source: {}", PostgresTestSource.class);

‎airbyte-integrations/connectors/source-relational-db/src/main/java/io/airbyte/integrations/source/relationaldb/AbstractDbSource.java

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -304,10 +304,11 @@ private List<TableInfo<CommonField<DataType>>> discoverWithoutSystemTables(
304304
final Database database)
305305
throws Exception {
306306
final Set<String> systemNameSpaces = getExcludedInternalNameSpaces();
307+
final Set<String> systemViews = getExcludedViews();
307308
final List<TableInfo<CommonField<DataType>>> discoveredTables = discoverInternal(database);
308309
return (systemNameSpaces == null || systemNameSpaces.isEmpty() ? discoveredTables
309310
: discoveredTables.stream()
310-
.filter(table -> !systemNameSpaces.contains(table.getNameSpace())).collect(
311+
.filter(table -> !systemNameSpaces.contains(table.getNameSpace()) && !systemViews.contains(table.getName())).collect(
311312
Collectors.toList()));
312313
}
313314

@@ -667,12 +668,19 @@ protected abstract List<CheckedConsumer<Database, Exception>> getCheckOperations
667668
protected abstract JsonSchemaType getAirbyteType(DataType columnType);
668669

669670
/**
670-
* Get list of system namespaces(schemas) in order to exclude them from the discover result list.
671+
* Get list of system namespaces(schemas) in order to exclude them from the `discover` result list.
671672
*
672673
* @return set of system namespaces(schemas) to be excluded
673674
*/
674675
protected abstract Set<String> getExcludedInternalNameSpaces();
675676

677+
/**
678+
* Get list of system views in order to exclude them from the `discover` result list.
679+
*
680+
* @return set of views to be excluded
681+
*/
682+
protected Set<String> getExcludedViews() { return Collections.emptySet(); };
683+
676684
/**
677685
* Discover all available tables in the source database.
678686
*

‎docs/integrations/sources/alloydb.md

Lines changed: 13 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -325,15 +325,16 @@ According to Postgres [documentation](https://www.postgresql.org/docs/14/datatyp
325325

326326
## Changelog
327327

328-
| Version | Date | Pull Request | Subject |
329-
|:--------|:-----------|:---------------------------------------------------------|:-------------------------------------------------|
330-
| 1.0.43 | 2022-02-06 | [21634](https://github.com/airbytehq/airbyte/pull/21634) | Improve Standard sync performance by caching objects.|
331-
| 1.0.36 | 2023-01-24 | [21825](https://github.com/airbytehq/airbyte/pull/21825) | Put back the original change that will cause an incremental sync to error if table contains a NULL value in cursor column.|
332-
| 1.0.35 | 2022-12-14 | [20436](https://github.com/airbytehq/airbyte/pull/20346) | Consolidate date/time values mapping for JDBC sources |
333-
| 1.0.34 | 2022-12-13 | [20378](https://github.com/airbytehq/airbyte/pull/20378) | Improve descriptions |
334-
| 1.0.17 | 2022-10-31 | [18538](https://github.com/airbytehq/airbyte/pull/18538) | Encode database name |
335-
| 1.0.16 | 2022-10-25 | [18256](https://github.com/airbytehq/airbyte/pull/18256) | Disable allow and prefer ssl modes in CDC mode |
336-
| | 2022-10-13 | [15535](https://github.com/airbytehq/airbyte/pull/16238) | Update incremental query to avoid data missing when new data is inserted at the same time as a sync starts under non-CDC incremental mode |
337-
| 1.0.15 | 2022-10-11 | [17782](https://github.com/airbytehq/airbyte/pull/17782) | Align with Postgres source v.1.0.15 |
338-
| 1.0.0 | 2022-09-15 | [16776](https://github.com/airbytehq/airbyte/pull/16776) | Align with strict-encrypt version |
339-
| 0.1.0 | 2022-09-05 | [16323](https://github.com/airbytehq/airbyte/pull/16323) | Initial commit. Based on source-postgres v.1.0.7 |
328+
| Version | Date | Pull Request | Subject |
329+
|:--------|:-----------|:---------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
330+
| 1.0.44 | 2022-02-06 | [22221](https://github.com/airbytehq/airbyte/pull/22221) | Exclude new set of system tables when using `pg_stat_statements` extension. |
331+
| 1.0.43 | 2022-02-06 | [21634](https://github.com/airbytehq/airbyte/pull/21634) | Improve Standard sync performance by caching objects. |
332+
| 1.0.36 | 2023-01-24 | [21825](https://github.com/airbytehq/airbyte/pull/21825) | Put back the original change that will cause an incremental sync to error if table contains a NULL value in cursor column. |
333+
| 1.0.35 | 2022-12-14 | [20436](https://github.com/airbytehq/airbyte/pull/20346) | Consolidate date/time values mapping for JDBC sources |
334+
| 1.0.34 | 2022-12-13 | [20378](https://github.com/airbytehq/airbyte/pull/20378) | Improve descriptions |
335+
| 1.0.17 | 2022-10-31 | [18538](https://github.com/airbytehq/airbyte/pull/18538) | Encode database name |
336+
| 1.0.16 | 2022-10-25 | [18256](https://github.com/airbytehq/airbyte/pull/18256) | Disable allow and prefer ssl modes in CDC mode |
337+
| | 2022-10-13 | [15535](https://github.com/airbytehq/airbyte/pull/16238) | Update incremental query to avoid data missing when new data is inserted at the same time as a sync starts under non-CDC incremental mode |
338+
| 1.0.15 | 2022-10-11 | [17782](https://github.com/airbytehq/airbyte/pull/17782) | Align with Postgres source v.1.0.15 |
339+
| 1.0.0 | 2022-09-15 | [16776](https://github.com/airbytehq/airbyte/pull/16776) | Align with strict-encrypt version |
340+
| 0.1.0 | 2022-09-05 | [16323](https://github.com/airbytehq/airbyte/pull/16323) | Initial commit. Based on source-postgres v.1.0.7 |

‎docs/integrations/sources/postgres.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -411,6 +411,7 @@ The root causes is that the WALs needed for the incremental sync has been remove
411411

412412
| Version | Date | Pull Request | Subject |
413413
|:--------|:-----------|:---------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
414+
| 1.0.44 | 2022-02-06 | [22221](https://github.com/airbytehq/airbyte/pull/22221) | Exclude new set of system tables when using `pg_stat_statements` extension. |
414415
| 1.0.43 | 2022-02-06 | [21634](https://github.com/airbytehq/airbyte/pull/21634) | Improve Standard sync performance by caching objects. |
415416
| 1.0.42 | 2022-01-23 | [21523](https://github.com/airbytehq/airbyte/pull/21523) | Check for null in cursor values before replacing. |
416417
| 1.0.41 | 2022-01-25 | [20939](https://github.com/airbytehq/airbyte/pull/20939) | Adjust batch selection memory limits databases. |

0 commit comments

Comments
 (0)