🎉 Postgres Source : Allow streams not in CDC publication to be synced in Full-refresh mode #24622

akashkulk · 2023-03-27T23:19:25Z

Changes source-postgres to discover streams that are not in the CDC publication and allows to sync those streams in full-refresh mode

…blication

akashkulk · 2023-03-27T23:20:57Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4537531825
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4537531825
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
=================== 68 passed, 5 skipped in 80.01s (0:01:20) ===================

github-actions · 2023-03-27T23:23:52Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (3)

Connector	Version	Changelog	Publish
`source-alloydb`	`2.0.17`	✅	✅
`source-alloydb-strict-encrypt`	`2.0.17`	🔵 (ignored)	🔵 (ignored)
`source-postgres-strict-encrypt`	`2.0.17`	🔵 (ignored)	🔵 (ignored)

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

subodh1810 · 2023-03-29T09:04:54Z

...rs/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java

@@ -113,6 +112,8 @@ public class PostgresSource extends AbstractJdbcSource<PostgresType> implements
  public static final String MODE = "mode";

  private List<String> schemas;
+
+  private Set<AirbyteStreamNameNamespacePair> publicizedTablesInCdc;


I dont like the idea of having a global variable and hoping/assuming it will get initialised cause a chain of methods will be executed before we use it. Any change in the execution steps will lead to a potential bug

Unfortunately, we already have that with schemas()

I think the real issue is that we have a chain of methods where you can't predict what order they'd run in. Ideally, it should be clear what the ordering and so eventually we should gut a lot of the abstract methods deeper in the call stack.

For now, I've moved the initialization of this global variable in createDatabase(). In general, this is part of the initialization phase and is the first thing that runs in every protocol method. So, any code that refers to this in check/discover/read methods should have this variable initialized

subodh1810

Honestly I still dont like the way we are doing this but I understand that there is no better way! I think the dependency on the database object for all these operations is a big pain. We need to sort it out

akashkulk · 2023-04-05T21:18:29Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4622949399
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4622949399
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
=================== 68 passed, 5 skipped in 82.69s (0:01:22) ===================

akashkulk · 2023-04-05T21:18:37Z

/test connector=connectors/source-postgres-strict-encrypt

🕑 connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/4622950646
✅ connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/4622950646
No Python unittests run

Build Passed

Test summary info:

All Passed

akashkulk · 2023-04-05T21:47:33Z

/publish connector=connectors/source-postgres run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/4623151204

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

akashkulk · 2023-04-05T21:47:57Z

/publish connector=connectors/source-postgres-strict-encrypt run-tests=false

akashkulk · 2023-04-05T21:48:12Z

/publish connector=connectors/source-alloydb run-tests=false

🕑 Publishing the following connectors:
connectors/source-alloydb
https://github.com/airbytehq/airbyte/actions/runs/4623155230

Connector	Did it publish?	Were definitions generated?
connectors/source-alloydb	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

akashkulk · 2023-04-05T21:48:27Z

/publish connector=connectors/source-alloydb-strict-encrypt run-tests=false

🕑 Publishing the following connectors:
connectors/source-alloydb-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4623156543

Connector	Did it publish?	Were definitions generated?
connectors/source-alloydb-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

akashkulk · 2023-04-05T22:21:07Z

/publish connector=connectors/source-postgres-strict-encrypt run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4623399072

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Postgres should not restrict discovered tables to those present in pu…

0c43b71

…blication

octavia-squidington-iii added area/connectors Connector related issues connectors/source/postgres labels Mar 27, 2023

Merge branch 'master' into postgres-discover-publication

e6ae8cd

akashkulk requested a review from subodh1810 March 27, 2023 23:21

akashkulk marked this pull request as ready for review March 28, 2023 00:05

akashkulk requested a review from a team as a code owner March 28, 2023 00:05

Merge branch 'master' into postgres-discover-publication

51c7278

subodh1810 requested changes Mar 29, 2023

View reviewed changes

Small update

2fb144a

akashkulk requested a review from subodh1810 March 30, 2023 05:43

akashkulk added 2 commits March 29, 2023 22:43

Merge branch 'master' into postgres-discover-publication

418122a

Merge branch 'master' into postgres-discover-publication

0e303c8

subodh1810 approved these changes Apr 4, 2023

View reviewed changes

akashkulk and others added 3 commits April 4, 2023 08:39

Merge branch 'master' into postgres-discover-publication

32a0cdb

Merge branch 'master' into postgres-discover-publication

daf67de

Update Dockerfiles + documentation

ae63302

octavia-squidington-iii added area/documentation Improvements or additions to documentation connectors/source/postgres-strict-encrypt connectors/source/alloydb connectors/source/alloydb-strict-encrypt labels Apr 5, 2023

akashkulk and others added 2 commits April 5, 2023 21:08

Automated Change

7843062

Merge branch 'master' into postgres-discover-publication

3da3b47

auto-bump connector version

5a20c53

akashkulk and others added 3 commits April 5, 2023 15:41

manually update alloydb source definitions

42147fd

generate source_specs

5be3ce5

Merge branch 'master' into postgres-discover-publication

13e1b7c

akashkulk enabled auto-merge (squash) April 5, 2023 22:46

akashkulk merged commit e96f9e0 into master Apr 5, 2023

akashkulk deleted the postgres-discover-publication branch April 5, 2023 23:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Postgres Source : Allow streams not in CDC publication to be synced in Full-refresh mode #24622

🎉 Postgres Source : Allow streams not in CDC publication to be synced in Full-refresh mode #24622

akashkulk commented Mar 27, 2023

akashkulk commented Mar 27, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Mar 27, 2023 •

edited

Loading

✅ Sources (3)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

subodh1810 Mar 29, 2023

akashkulk Mar 30, 2023

subodh1810 left a comment

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 5, 2023

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading

🎉 Postgres Source : Allow streams not in CDC publication to be synced in Full-refresh mode #24622

🎉 Postgres Source : Allow streams not in CDC publication to be synced in Full-refresh mode #24622

Conversation

akashkulk commented Mar 27, 2023

akashkulk commented Mar 27, 2023 • edited by github-actions bot Loading

Build Passed

github-actions bot commented Mar 27, 2023 • edited Loading

Affected Connector Report

✅ Sources (3)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

subodh1810 Mar 29, 2023

Choose a reason for hiding this comment

akashkulk Mar 30, 2023

Choose a reason for hiding this comment

subodh1810 left a comment

Choose a reason for hiding this comment

akashkulk commented Apr 5, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Apr 5, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Apr 5, 2023 • edited by github-actions bot Loading

akashkulk commented Apr 5, 2023

akashkulk commented Apr 5, 2023 • edited by github-actions bot Loading

akashkulk commented Apr 5, 2023 • edited by github-actions bot Loading

akashkulk commented Apr 5, 2023 • edited by github-actions bot Loading

akashkulk commented Mar 27, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Mar 27, 2023 •

edited

Loading

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 5, 2023 •

edited by github-actions bot

Loading