Rollout ctid cdc #28708

rodireich · 2023-07-25T21:59:50Z

What

Used to pre release CDC + ctid

github-actions · 2023-07-25T22:00:11Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

octavia-squidington-iii · 2023-07-25T22:12:03Z

source-alloydb test report (commit `fb8bed2924`) - ❌

⏲️ Total pipeline duration: 07mn56s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

octavia-squidington-iii · 2023-07-25T22:27:50Z

source-postgres test report (commit `fb8bed2924`) - ❌

⏲️ Total pipeline duration: 15mn37s

Step	Result
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres:integrationTest	❌
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

github-actions · 2023-07-25T22:30:01Z

Coverage report for source-postgres

File	Coverage [86.59%]	🍏
PostgresSource.java	86.59%	🍏

Total Project Coverage	69.45%	🍏

octavia-squidington-iii · 2023-07-25T22:33:08Z

source-alloydb-strict-encrypt test report (commit `fb8bed2924`) - ❌

⏲️ Total pipeline duration: 05mn08s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb-strict-encrypt test

octavia-squidington-iii · 2023-07-25T22:39:53Z

source-postgres-strict-encrypt test report (commit `fb8bed2924`) - ❌

⏲️ Total pipeline duration: 06mn35s

Step	Result
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

octavia-squidington-iii · 2023-07-25T22:49:23Z

source-alloydb test report (commit `2d3025307c`) - ❌

⏲️ Total pipeline duration: 07mn36s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

octavia-squidington-iii · 2023-07-25T22:59:14Z

source-postgres-strict-encrypt test report (commit `2d3025307c`) - ❌

⏲️ Total pipeline duration: 09mn38s

Step	Result
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

octavia-squidington-iii · 2023-07-25T23:17:11Z

source-postgres test report (commit `2d3025307c`) - ❌

⏲️ Total pipeline duration: 17mn44s

Step	Result
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres:integrationTest	❌
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

octavia-squidington-iii · 2023-07-25T23:24:46Z

source-alloydb-strict-encrypt test report (commit `2d3025307c`) - ❌

⏲️ Total pipeline duration: 07mn22s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb-strict-encrypt test

octavia-squidington-iii · 2023-07-26T22:15:07Z

source-alloydb-strict-encrypt test report (commit `092abe8636`) - ❌

⏲️ Total pipeline duration: 09mn02s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb-strict-encrypt test

octavia-squidington-iii · 2023-07-26T22:25:59Z

source-postgres-strict-encrypt test report (commit `092abe8636`) - ❌

⏲️ Total pipeline duration: 10mn39s

Step	Result
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

octavia-squidington-iii · 2023-07-26T22:46:23Z

source-postgres test report (commit `092abe8636`) - ✅

⏲️ Total pipeline duration: 20mn09s

Step	Result
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres:integrationTest	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

octavia-squidington-iii · 2023-07-26T22:55:19Z

source-alloydb test report (commit `092abe8636`) - ❌

⏲️ Total pipeline duration: 08mn42s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

octavia-squidington-iii · 2023-07-26T23:06:36Z

source-alloydb-strict-encrypt test report (commit `6249385f74`) - ❌

⏲️ Total pipeline duration: 08mn50s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb-strict-encrypt test

octavia-squidington-iii · 2023-07-26T23:27:14Z

source-postgres test report (commit `6249385f74`) - ✅

⏲️ Total pipeline duration: 20mn23s

Step	Result
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres:integrationTest	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

octavia-squidington-iii · 2023-07-26T23:36:19Z

source-alloydb test report (commit `6249385f74`) - ❌

⏲️ Total pipeline duration: 08mn50s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

octavia-squidington-iii · 2023-07-26T23:50:53Z

source-postgres-strict-encrypt test report (commit `6249385f74`) - ❌

⏲️ Total pipeline duration: 14mn19s

Step	Result
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

octavia-squidington-iii · 2023-08-03T19:07:31Z

source-alloydb-strict-encrypt test report (commit `74d9740d61`) - ❌

⏲️ Total pipeline duration: 18.04s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb-strict-encrypt test

octavia-squidington-iii · 2023-08-03T19:07:43Z

source-postgres-strict-encrypt test report (commit `74d9740d61`) - ❌

⏲️ Total pipeline duration: 3.51s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

octavia-squidington-iii · 2023-08-03T19:07:55Z

source-alloydb test report (commit `74d9740d61`) - ❌

⏲️ Total pipeline duration: 3.37s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

octavia-squidington-iii · 2023-08-03T19:08:11Z

source-postgres test report (commit `74d9740d61`) - ❌

⏲️ Total pipeline duration: 7.20s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

nguyenaiden

+1. Yay for full release!

octavia-squidington-iii · 2023-08-03T19:14:20Z

source-postgres-strict-encrypt test report (commit `2730f18679`) - ❌

⏲️ Total pipeline duration: 3.45s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

octavia-squidington-iii · 2023-08-03T19:14:32Z

source-alloydb test report (commit `2730f18679`) - ❌

⏲️ Total pipeline duration: 3.28s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

octavia-squidington-iii · 2023-08-03T19:14:44Z

source-alloydb-strict-encrypt test report (commit `2730f18679`) - ❌

⏲️ Total pipeline duration: 3.32s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb-strict-encrypt test

octavia-squidington-iii · 2023-08-03T19:15:00Z

source-postgres test report (commit `2730f18679`) - ❌

⏲️ Total pipeline duration: 7.24s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

akashkulk · 2023-08-03T19:28:10Z

...byte/integrations/io/airbyte/integration_tests/sources/XminPostgresSourceAcceptanceTest.java

@@ -81,22 +81,6 @@ protected void setupEnvironment(final TestDestinationEnv environment) throws Exc
    }
  }

-  private JsonNode getConfig(final String username, final String password, final List<String> schemas) {


Why are we getting rid of the override here?

This PR is a copy of the original CDC + ctid PR (that's the one I merged by mistake and had to roll back)

octavia-squidington-iii · 2023-08-03T20:09:38Z

source-postgres test report (commit `d5de6f44b2`) - ❌

⏲️ Total pipeline duration: 22.40s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

octavia-squidington-iii · 2023-08-03T20:09:50Z

source-alloydb-strict-encrypt test report (commit `d5de6f44b2`) - ❌

⏲️ Total pipeline duration: 3.31s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb-strict-encrypt test

octavia-squidington-iii · 2023-08-03T20:10:02Z

source-postgres-strict-encrypt test report (commit `d5de6f44b2`) - ❌

⏲️ Total pipeline duration: 3.46s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

octavia-squidington-iii · 2023-08-03T20:10:14Z

source-alloydb test report (commit `d5de6f44b2`) - ❌

⏲️ Total pipeline duration: 3.52s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

rodireich · 2023-08-03T20:35:27Z

/approve-and-merge reason="Tested locally. Pipeline testings are down"

octavia-approvington · 2023-08-03T20:36:01Z

A crack team of mammals has made a decision.

* source-postgres: enable ctid+cdc implementation * 100% ctid rollout for cdc * remove CtidFeatureFlags * fix CdcPostgresSourceAcceptanceTest * Bump versions and release notes * Fix compilation error due to previous merge --------- Co-authored-by: subodh <[email protected]>

* Add everything for BQ but migrate, refactor interface after practical work * Make new default methods, refactor to single implemented method * MigrationInterface and BQ impl created * Trying to integrate with standard inserts * remove unnecessary NameAndNamespacePair class * Shimmed in * Java Docs * Initial Testing Setup * Tests! * Move Migrator into TyperDeduper * Functional Migration * Add Integration Test * Pr updates * bump version * bump version * version bump * Update to airbyte-ci-internal (#29026) * 🐛 Source Github, Instagram, Zendesk-support, Zendesk-talk: fix CAT tests fail on `spec` (#28910) * connectors-ci: better modified connectors detection logic (#28855) * connectors-ci: report path should always start with `airbyte-ci/` (#29030) * make report path always start with airbyte-ci * revert report path in orchestrator * add more test cases * bump version * Updated docs (#29019) * CDK: Embedded reader utils (#28873) * relax pydantic dep * Automated Commit - Format and Process Resources Changes * wip * wrap up base integration * add init file * introduce CDK runner and improve error message * make state param optional * update protocol models * review comments * always run incremental if possible * fix --------- Co-authored-by: flash1293 <[email protected]> * 🤖 Bump minor version of Airbyte CDK * 🚨🚨 Low code CDK: Decouple SimpleRetriever and HttpStream (#28657) * fix tests * format * review comments * Automated Commit - Formatting Changes * review comments * review comments * review comments * log all messages * log all message * review comments * review comments * Automated Commit - Formatting Changes * add comment --------- Co-authored-by: flash1293 <[email protected]> * 🤖 Bump minor version of Airbyte CDK * 🐛 Source Github, Instagram, Zendesk Support / Talk - revert `spec` changes and improve (#29031) * Source oauth0: new streams and fix incremental (#29001) * Add new streams Organizations,OrganizationMembers,OrganizationMemberRoles * relax schema definition to allow additional fields * Bump image tag version * revert some changes to the old schemas * Format python so gradle can pass * update incremental * remove unused print * fix unit test --------- Co-authored-by: Vasilis Gavriilidis <[email protected]> * 🐛 Source Mongo: Fix failing acceptance tests (#28816) * Fix failing acceptance tests * Fix failing strict acceptance tests * Source-Greenhouse: Fix unit tests for new CDK version (#28969) Fix unit tests * Add CSV options to the CSV parser (#28491) * remove invalid legacy option * remove unused option * the tests pass but this is quite messy * very slight clean up * Add skip options to csv format * fix some of the typing issues * fixme comment * remove extra log message * fix typing issues * skip before header * skip after header * format * add another test * Automated Commit - Formatting Changes * auto generate column names * delete dead code * update title and description * true and false values * Update the tests * Add comment * missing test * rename * update expected spec * move to method * Update comment * fix typo * remove unused import * Add a comment * None records do not pass the WaitForDiscoverPolicy * format * remove second branch to ensure we always go through the same processing * Raise an exception if the record is None * reset * Update tests * handle unquoted newlines * Automated Commit - Formatting Changes * Update test case so the quoting is explicit * Update comment * Automated Commit - Formatting Changes * Fail validation if skipping rows before header and header is autogenerated * always fail if a record cannot be parsed * format * set write line_no in error message * remove none check * Automated Commit - Formatting Changes * enable autogenerate test * remove duplicate test * missing unit tests * Update * remove branching * remove unused none check * Update tests * remove branching * format * extract to function * comment * missing type * type annotation * use set * Document that the strings are case-sensitive * public -> private * add unit test * newline --------- Co-authored-by: girarda <[email protected]> * Dagster: Add sentry logging (#28822) * Add sentry * add sentry decorator * Add traces * Use sentry trace * Improve duplicate logging * Add comments * DNC * Fix up issues * Move to scopes * Remove breadcrumb * Update lock * ✨Source Shortio: Migrate Python CDK to Low-code CDK (#28950) * Migrate Shortio to Low-Code * Update abnormal state * Format * Update Docs * Fix metadata.yaml * Add pagination * Add incremental sync * add incremental parameters * update metadata * rollback update version * release date --------- Co-authored-by: marcosmarxm <[email protected]> * Update to new verbiage (#29051) * [skip ci] Metadata: Remove leading underscore (#29024) * DNC * Add test models * Add model test * Remove underscore from metadata files * Regenerate models * Add test to check for key transformation * Allow additional fields on metadata * Delete transform * Proof of concept parallel source stream reading implementation for MySQL (#26580) * Proof of concept parallel source stream reading implementation for MySQL * Automated Change * Add read method that supports concurrent execution to Source interface * Remove parallel iterator * Ensure that executor service is stopped * Automated Commit - Format and Process Resources Changes * Expose method to fix compilation issue * Use concurrent map to avoid access issues * Automated Commit - Format and Process Resources Changes * Ensure concurrent streams finish before closing source * Fix compile issue * Formatting * Exclude concurrent stream threads from orphan thread watcher * Automated Commit - Format and Process Resources Changes * Refactor orphaned thread logic to account for concurrent execution * PR feedback * Implement readStreams in wrapper source * Automated Commit - Format and Process Resources Changes * Add readStream override * Automated Commit - Format and Process Resources Changes * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * Debug logging * Reduce logging level * Replace synchronized calls to System.out.println when concurrent * Close consumer * Flush before close * Automated Commit - Format and Process Resources Changes * Remove charset * Use ASCII and flush periodically for parallel streams * Test performance harness patch * Automated Commit - Format and Process Resources Changes * Cleanup * Logging to identify concurrent read enabled * Mark parameter as final --------- Co-authored-by: jdpgrailsdev <[email protected]> Co-authored-by: octavia-squidington-iii <[email protected]> Co-authored-by: Rodi Reich Zilberman <[email protected]> Co-authored-by: rodireich <[email protected]> * connectors-ci: disable dependency scanning (#29033) * updates (#29059) * Metadata: skip breaking change validation on prerelease (#29017) * skip breaking change validation * Move ValidatorOpts higher in call * Add prerelease test * Fix test * ✨ Source MongoDB Internal POC: Generate Test Data (#29049) * Add script to generate test data * Fix prose * Update credentials example * PR feedback * Bump Airbyte version from 0.50.12 to 0.50.13 * Bump versions for mssql strict-encrypt (#28964) * Bump versions for mssql strict-encrypt * Fix failing test * Fix failing test * 🎨 Improve replication method selection UX (#28882) * update replication method in MySQL source * bump version * update expected specs * update registries * bump strict encrypt version * make password always_show * change url * update registries * 🐛 Avoid writing records to log (#29047) * Avoid writing records to log * Update version * Rollout ctid cdc (#28708) * source-postgres: enable ctid+cdc implementation * 100% ctid rollout for cdc * remove CtidFeatureFlags * fix CdcPostgresSourceAcceptanceTest * Bump versions and release notes * Fix compilation error due to previous merge --------- Co-authored-by: subodh <[email protected]> * connectors-ci: fix `unhashable type 'set'` (#29064) * Add Slack Alert lifecycle to Dagster for Metadata publish (#28759) * DNC * Add slack lifecycle logging * Update to use slack * Update slack to use resource and bot * Improve markdown * Improve log * Add sensor logging * Extend sensor time * merge conflict * PR Refactoring * Make the tests work * remove unnecessary classes, pr feedback * more merging * Update airbyte-integrations/bases/base-typing-deduping-test/src/main/java/io/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java Co-authored-by: Edward Gao <[email protected]> * snowflake updates --------- Co-authored-by: Ben Church <[email protected]> Co-authored-by: Baz <[email protected]> Co-authored-by: Augustin <[email protected]> Co-authored-by: Serhii Lazebnyi <[email protected]> Co-authored-by: Joe Reuter <[email protected]> Co-authored-by: flash1293 <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Vasilis Gavriilidis <[email protected]> Co-authored-by: Jonathan Pearlin <[email protected]> Co-authored-by: Alexandre Girard <[email protected]> Co-authored-by: girarda <[email protected]> Co-authored-by: btkcodedev <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Natalie Kwong <[email protected]> Co-authored-by: jdpgrailsdev <[email protected]> Co-authored-by: octavia-squidington-iii <[email protected]> Co-authored-by: Rodi Reich Zilberman <[email protected]> Co-authored-by: rodireich <[email protected]> Co-authored-by: Alexandre Cuoci <[email protected]> Co-authored-by: terencecho <[email protected]> Co-authored-by: Lake Mossman <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> Co-authored-by: subodh <[email protected]> Co-authored-by: Edward Gao <[email protected]>

* Add everything for BQ but migrate, refactor interface after practical work * Make new default methods, refactor to single implemented method * MigrationInterface and BQ impl created * Trying to integrate with standard inserts * remove unnecessary NameAndNamespacePair class * Shimmed in * Java Docs * Initial Testing Setup * Tests! * Move Migrator into TyperDeduper * Functional Migration * Add Integration Test * Pr updates * bump version * bump version * version bump * Update to airbyte-ci-internal (airbytehq#29026) * 🐛 Source Github, Instagram, Zendesk-support, Zendesk-talk: fix CAT tests fail on `spec` (airbytehq#28910) * connectors-ci: better modified connectors detection logic (airbytehq#28855) * connectors-ci: report path should always start with `airbyte-ci/` (airbytehq#29030) * make report path always start with airbyte-ci * revert report path in orchestrator * add more test cases * bump version * Updated docs (airbytehq#29019) * CDK: Embedded reader utils (airbytehq#28873) * relax pydantic dep * Automated Commit - Format and Process Resources Changes * wip * wrap up base integration * add init file * introduce CDK runner and improve error message * make state param optional * update protocol models * review comments * always run incremental if possible * fix --------- Co-authored-by: flash1293 <[email protected]> * 🤖 Bump minor version of Airbyte CDK * 🚨🚨 Low code CDK: Decouple SimpleRetriever and HttpStream (airbytehq#28657) * fix tests * format * review comments * Automated Commit - Formatting Changes * review comments * review comments * review comments * log all messages * log all message * review comments * review comments * Automated Commit - Formatting Changes * add comment --------- Co-authored-by: flash1293 <[email protected]> * 🤖 Bump minor version of Airbyte CDK * 🐛 Source Github, Instagram, Zendesk Support / Talk - revert `spec` changes and improve (airbytehq#29031) * Source oauth0: new streams and fix incremental (airbytehq#29001) * Add new streams Organizations,OrganizationMembers,OrganizationMemberRoles * relax schema definition to allow additional fields * Bump image tag version * revert some changes to the old schemas * Format python so gradle can pass * update incremental * remove unused print * fix unit test --------- Co-authored-by: Vasilis Gavriilidis <[email protected]> * 🐛 Source Mongo: Fix failing acceptance tests (airbytehq#28816) * Fix failing acceptance tests * Fix failing strict acceptance tests * Source-Greenhouse: Fix unit tests for new CDK version (airbytehq#28969) Fix unit tests * Add CSV options to the CSV parser (airbytehq#28491) * remove invalid legacy option * remove unused option * the tests pass but this is quite messy * very slight clean up * Add skip options to csv format * fix some of the typing issues * fixme comment * remove extra log message * fix typing issues * skip before header * skip after header * format * add another test * Automated Commit - Formatting Changes * auto generate column names * delete dead code * update title and description * true and false values * Update the tests * Add comment * missing test * rename * update expected spec * move to method * Update comment * fix typo * remove unused import * Add a comment * None records do not pass the WaitForDiscoverPolicy * format * remove second branch to ensure we always go through the same processing * Raise an exception if the record is None * reset * Update tests * handle unquoted newlines * Automated Commit - Formatting Changes * Update test case so the quoting is explicit * Update comment * Automated Commit - Formatting Changes * Fail validation if skipping rows before header and header is autogenerated * always fail if a record cannot be parsed * format * set write line_no in error message * remove none check * Automated Commit - Formatting Changes * enable autogenerate test * remove duplicate test * missing unit tests * Update * remove branching * remove unused none check * Update tests * remove branching * format * extract to function * comment * missing type * type annotation * use set * Document that the strings are case-sensitive * public -> private * add unit test * newline --------- Co-authored-by: girarda <[email protected]> * Dagster: Add sentry logging (airbytehq#28822) * Add sentry * add sentry decorator * Add traces * Use sentry trace * Improve duplicate logging * Add comments * DNC * Fix up issues * Move to scopes * Remove breadcrumb * Update lock * ✨Source Shortio: Migrate Python CDK to Low-code CDK (airbytehq#28950) * Migrate Shortio to Low-Code * Update abnormal state * Format * Update Docs * Fix metadata.yaml * Add pagination * Add incremental sync * add incremental parameters * update metadata * rollback update version * release date --------- Co-authored-by: marcosmarxm <[email protected]> * Update to new verbiage (airbytehq#29051) * [skip ci] Metadata: Remove leading underscore (airbytehq#29024) * DNC * Add test models * Add model test * Remove underscore from metadata files * Regenerate models * Add test to check for key transformation * Allow additional fields on metadata * Delete transform * Proof of concept parallel source stream reading implementation for MySQL (airbytehq#26580) * Proof of concept parallel source stream reading implementation for MySQL * Automated Change * Add read method that supports concurrent execution to Source interface * Remove parallel iterator * Ensure that executor service is stopped * Automated Commit - Format and Process Resources Changes * Expose method to fix compilation issue * Use concurrent map to avoid access issues * Automated Commit - Format and Process Resources Changes * Ensure concurrent streams finish before closing source * Fix compile issue * Formatting * Exclude concurrent stream threads from orphan thread watcher * Automated Commit - Format and Process Resources Changes * Refactor orphaned thread logic to account for concurrent execution * PR feedback * Implement readStreams in wrapper source * Automated Commit - Format and Process Resources Changes * Add readStream override * Automated Commit - Format and Process Resources Changes * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * Debug logging * Reduce logging level * Replace synchronized calls to System.out.println when concurrent * Close consumer * Flush before close * Automated Commit - Format and Process Resources Changes * Remove charset * Use ASCII and flush periodically for parallel streams * Test performance harness patch * Automated Commit - Format and Process Resources Changes * Cleanup * Logging to identify concurrent read enabled * Mark parameter as final --------- Co-authored-by: jdpgrailsdev <[email protected]> Co-authored-by: octavia-squidington-iii <[email protected]> Co-authored-by: Rodi Reich Zilberman <[email protected]> Co-authored-by: rodireich <[email protected]> * connectors-ci: disable dependency scanning (airbytehq#29033) * updates (airbytehq#29059) * Metadata: skip breaking change validation on prerelease (airbytehq#29017) * skip breaking change validation * Move ValidatorOpts higher in call * Add prerelease test * Fix test * ✨ Source MongoDB Internal POC: Generate Test Data (airbytehq#29049) * Add script to generate test data * Fix prose * Update credentials example * PR feedback * Bump Airbyte version from 0.50.12 to 0.50.13 * Bump versions for mssql strict-encrypt (airbytehq#28964) * Bump versions for mssql strict-encrypt * Fix failing test * Fix failing test * 🎨 Improve replication method selection UX (airbytehq#28882) * update replication method in MySQL source * bump version * update expected specs * update registries * bump strict encrypt version * make password always_show * change url * update registries * 🐛 Avoid writing records to log (airbytehq#29047) * Avoid writing records to log * Update version * Rollout ctid cdc (airbytehq#28708) * source-postgres: enable ctid+cdc implementation * 100% ctid rollout for cdc * remove CtidFeatureFlags * fix CdcPostgresSourceAcceptanceTest * Bump versions and release notes * Fix compilation error due to previous merge --------- Co-authored-by: subodh <[email protected]> * connectors-ci: fix `unhashable type 'set'` (airbytehq#29064) * Add Slack Alert lifecycle to Dagster for Metadata publish (airbytehq#28759) * DNC * Add slack lifecycle logging * Update to use slack * Update slack to use resource and bot * Improve markdown * Improve log * Add sensor logging * Extend sensor time * merge conflict * PR Refactoring * Make the tests work * remove unnecessary classes, pr feedback * more merging * Update airbyte-integrations/bases/base-typing-deduping-test/src/main/java/io/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java Co-authored-by: Edward Gao <[email protected]> * snowflake updates --------- Co-authored-by: Ben Church <[email protected]> Co-authored-by: Baz <[email protected]> Co-authored-by: Augustin <[email protected]> Co-authored-by: Serhii Lazebnyi <[email protected]> Co-authored-by: Joe Reuter <[email protected]> Co-authored-by: flash1293 <[email protected]> Co-authored-by: Marcos Marx <[email protected]> Co-authored-by: Vasilis Gavriilidis <[email protected]> Co-authored-by: Jonathan Pearlin <[email protected]> Co-authored-by: Alexandre Girard <[email protected]> Co-authored-by: girarda <[email protected]> Co-authored-by: btkcodedev <[email protected]> Co-authored-by: marcosmarxm <[email protected]> Co-authored-by: Natalie Kwong <[email protected]> Co-authored-by: jdpgrailsdev <[email protected]> Co-authored-by: octavia-squidington-iii <[email protected]> Co-authored-by: Rodi Reich Zilberman <[email protected]> Co-authored-by: rodireich <[email protected]> Co-authored-by: Alexandre Cuoci <[email protected]> Co-authored-by: terencecho <[email protected]> Co-authored-by: Lake Mossman <[email protected]> Co-authored-by: Benoit Moriceau <[email protected]> Co-authored-by: subodh <[email protected]> Co-authored-by: Edward Gao <[email protected]>

subodh1810 added 5 commits July 7, 2023 17:33

source-postgres: enable ctid+cdc implementation

3bed26f

Merge branch 'master' into enable-ctid-cdc

de1e066

Merge branch 'master' into enable-ctid-cdc

b616b6e

Merge branch 'master' into enable-ctid-cdc

968c454

100% ctid rollout for cdc

9831147

rodireich requested a review from a team as a code owner July 25, 2023 21:59

octavia-squidington-iii added area/connectors Connector related issues connectors/source/postgres labels Jul 25, 2023

rodireich added 2 commits July 25, 2023 15:02

Merge branch 'master' into rollout-ctid-cdc

fb8bed2

remove CtidFeatureFlags

2d30253

rodireich added 2 commits July 26, 2023 15:01

fix CdcPostgresSourceAcceptanceTest

092abe8

Merge branch 'master' into rollout-ctid-cdc

6249385

Bump versions and release notes

74d9740

octavia-squidington-iii added area/documentation Improvements or additions to documentation connectors/source/postgres-strict-encrypt connectors/source/alloydb connectors/source/alloydb-strict-encrypt labels Aug 3, 2023

nguyenaiden approved these changes Aug 3, 2023

View reviewed changes

Merge branch 'master' into rollout-ctid-cdc

2730f18

akashkulk reviewed Aug 3, 2023

View reviewed changes

akashkulk approved these changes Aug 3, 2023

View reviewed changes

Fix compilation error due to previous merge

d5de6f4

octavia-approvington approved these changes Aug 3, 2023

View reviewed changes

octavia-approvington merged commit 2d2cddd into master Aug 3, 2023

octavia-approvington deleted the rollout-ctid-cdc branch August 3, 2023 20:36

Rollout ctid cdc #28708

Rollout ctid cdc #28708

Uh oh!

Conversation

rodireich commented Jul 25, 2023

What

Uh oh!

github-actions bot commented Jul 25, 2023

Before Merging a Connector Pull Request

Uh oh!

octavia-squidington-iii commented Jul 25, 2023

source-alloydb test report (commit fb8bed2924) - ❌

Uh oh!

octavia-squidington-iii commented Jul 25, 2023

source-postgres test report (commit fb8bed2924) - ❌

Uh oh!

github-actions bot commented Jul 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report for source-postgres

Uh oh!

octavia-squidington-iii commented Jul 25, 2023

source-alloydb-strict-encrypt test report (commit fb8bed2924) - ❌

Uh oh!

octavia-squidington-iii commented Jul 25, 2023

source-postgres-strict-encrypt test report (commit fb8bed2924) - ❌

Uh oh!

octavia-squidington-iii commented Jul 25, 2023

source-alloydb test report (commit 2d3025307c) - ❌

Uh oh!

octavia-squidington-iii commented Jul 25, 2023

source-postgres-strict-encrypt test report (commit 2d3025307c) - ❌

Uh oh!

octavia-squidington-iii commented Jul 25, 2023

source-postgres test report (commit 2d3025307c) - ❌

Uh oh!

octavia-squidington-iii commented Jul 25, 2023

source-alloydb-strict-encrypt test report (commit 2d3025307c) - ❌

Uh oh!

octavia-squidington-iii commented Jul 26, 2023

source-alloydb-strict-encrypt test report (commit 092abe8636) - ❌

Uh oh!

octavia-squidington-iii commented Jul 26, 2023

source-postgres-strict-encrypt test report (commit 092abe8636) - ❌

Uh oh!

octavia-squidington-iii commented Jul 26, 2023

source-postgres test report (commit 092abe8636) - ✅

Uh oh!

octavia-squidington-iii commented Jul 26, 2023

source-alloydb test report (commit 092abe8636) - ❌

Uh oh!

octavia-squidington-iii commented Jul 26, 2023

source-alloydb-strict-encrypt test report (commit 6249385f74) - ❌

Uh oh!

octavia-squidington-iii commented Jul 26, 2023

source-postgres test report (commit 6249385f74) - ✅

Uh oh!

octavia-squidington-iii commented Jul 26, 2023

source-alloydb test report (commit 6249385f74) - ❌

Uh oh!

octavia-squidington-iii commented Jul 26, 2023

source-postgres-strict-encrypt test report (commit 6249385f74) - ❌

Uh oh!

octavia-squidington-iii commented Aug 3, 2023

source-alloydb-strict-encrypt test report (commit 74d9740d61) - ❌

Uh oh!

octavia-squidington-iii commented Aug 3, 2023

source-postgres-strict-encrypt test report (commit 74d9740d61) - ❌

Uh oh!

octavia-squidington-iii commented Aug 3, 2023

source-alloydb test report (commit 74d9740d61) - ❌

Uh oh!

octavia-squidington-iii commented Aug 3, 2023

source-postgres test report (commit 74d9740d61) - ❌

Uh oh!

nguyenaiden left a comment

Choose a reason for hiding this comment

Uh oh!

octavia-squidington-iii commented Aug 3, 2023

source-postgres-strict-encrypt test report (commit 2730f18679) - ❌

Uh oh!

source-alloydb test report (commit `fb8bed2924`) - ❌

source-postgres test report (commit `fb8bed2924`) - ❌

github-actions bot commented Jul 25, 2023 •

edited

Loading

source-alloydb-strict-encrypt test report (commit `fb8bed2924`) - ❌

source-postgres-strict-encrypt test report (commit `fb8bed2924`) - ❌

source-alloydb test report (commit `2d3025307c`) - ❌

source-postgres-strict-encrypt test report (commit `2d3025307c`) - ❌

source-postgres test report (commit `2d3025307c`) - ❌

source-alloydb-strict-encrypt test report (commit `2d3025307c`) - ❌

source-alloydb-strict-encrypt test report (commit `092abe8636`) - ❌

source-postgres-strict-encrypt test report (commit `092abe8636`) - ❌

source-postgres test report (commit `092abe8636`) - ✅

source-alloydb test report (commit `092abe8636`) - ❌

source-alloydb-strict-encrypt test report (commit `6249385f74`) - ❌

source-postgres test report (commit `6249385f74`) - ✅

source-alloydb test report (commit `6249385f74`) - ❌

source-postgres-strict-encrypt test report (commit `6249385f74`) - ❌

source-alloydb-strict-encrypt test report (commit `74d9740d61`) - ❌

source-postgres-strict-encrypt test report (commit `74d9740d61`) - ❌

source-alloydb test report (commit `74d9740d61`) - ❌

source-postgres test report (commit `74d9740d61`) - ❌

source-postgres-strict-encrypt test report (commit `2730f18679`) - ❌

source-alloydb test report (commit `2730f18679`) - ❌

source-alloydb-strict-encrypt test report (commit `2730f18679`) - ❌

source-postgres test report (commit `2730f18679`) - ❌

source-postgres test report (commit `d5de6f44b2`) - ❌

source-alloydb-strict-encrypt test report (commit `d5de6f44b2`) - ❌

source-postgres-strict-encrypt test report (commit `d5de6f44b2`) - ❌

source-alloydb test report (commit `d5de6f44b2`) - ❌