🎉 MySQL Source : Make initial cdc timeout configurable for MySQL source #17170

akashkulk · 2022-09-26T20:14:22Z

What

Debezium sometimes takes more than 5 minutes to startup, which causes it to shut down for MySQL connector. This is essentially a port of the existing fix for Postgres, which makes the debezium timeout configurable.

Postgres PRs:

This PR

Makes the CDC wait time configurable
Does not change default 5 minutes
Speeds up tests to use shorter time

Resolve https://github.com/airbytehq/alpha-beta-issues/issues/136
Resolve #16373

🚨 User Impact 🚨

User can now configure a CDC wait time in the UI for MySQL source connector

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…' into akashkulk/mysql_debezium_timeout

github-actions · 2022-09-26T20:15:13Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-alloydb-strict-encrypt
source-postgres
source-bigquery
source-postgres-strict-encrypt
source-mssql-strict-encrypt
source-redshift
source-scaffold-java-jdbc
source-db2
source-db2-strict-encrypt
source-tidb
source-mssql
source-cockroachdb-strict-encrypt
source-mysql-strict-encrypt
source-clickhouse
source-snowflake
source-oracle-strict-encrypt
source-alloydb
source-cockroachdb
source-oracle
source-clickhouse-strict-encrypt
source-mysql

…' into akashkulk/mysql_debezium_timeout

github-actions · 2022-09-26T20:16:43Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

github-actions · 2022-09-26T20:21:48Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

…ezium_timeout

…' into akashkulk/mysql_debezium_timeout

github-actions · 2022-09-26T20:49:26Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql
source-mysql-strict-encrypt

github-actions · 2022-09-27T19:37:59Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql
source-mysql-strict-encrypt

…' into akashkulk/mysql_debezium_timeout

github-actions · 2022-09-27T20:48:12Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

akashkulk · 2022-09-27T20:49:42Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3138868705
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3138868705
No Python unittests run

Build Passed

Test summary info:

All Passed

akashkulk · 2022-09-27T20:50:11Z

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3138871709
❌ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3138871709
🐛 https://gradle.com/s/kzije5vgx5d64

Build Failed

Test summary info:

Could not find result summary

akashkulk · 2022-09-27T21:40:38Z

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3139163906
✅ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3139163906
No Python unittests run

Build Passed

Test summary info:

All Passed

github-actions · 2022-09-27T21:41:17Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

github-actions · 2022-09-27T22:32:46Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql
source-mysql-strict-encrypt

akashkulk · 2022-09-30T21:10:59Z

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3161704819
✅ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3161704819
No Python unittests run

Build Passed

Test summary info:

All Passed

github-actions · 2022-09-30T23:12:15Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

subodh1810

Looks good. Just pull in the latest changes from master cause I made a 1.0.1 release

airbyte-integrations/connectors/source-mysql-strict-encrypt/Dockerfile

subodh1810 · 2022-10-03T15:41:04Z

docs/integrations/sources/mysql.md

@@ -226,6 +241,7 @@ WHERE actor_definition_id ='435bb9a5-7887-4809-aa58-28c27df0d7ad' AND (configura

 | Version | Date       | Pull Request                                               | Subject                                                                                                                                       |
 |:--------|:-----------|:-----------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------|
+| 1.0.1  | 2022-09-30 | [17170](https://github.com/airbytehq/airbyte/pull/17170)   | Make initial CDC waiting time configurable                                                                                                            |


This would needed to be changed as well

Bumped to 1.0.2

github-actions · 2022-10-03T16:05:07Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

github-actions · 2022-10-03T16:11:29Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

akashkulk · 2022-10-03T16:11:32Z

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3175690131
❌ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3175690131
🐛 https://gradle.com/s/kmnth57drf6wk

Build Failed

Test summary info:

Could not find result summary

akashkulk · 2022-10-03T17:23:35Z

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3176124835
✅ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3176124835
No Python unittests run

Build Passed

Test summary info:

All Passed

github-actions · 2022-10-03T17:23:45Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

github-actions · 2022-10-03T17:24:53Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql
source-mysql-strict-encrypt

github-actions · 2022-10-03T20:06:11Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql
source-mysql-strict-encrypt

akashkulk · 2022-10-03T20:07:10Z

/publish connector=connectors/source-mysql run-tests=false

🕑 Publishing the following connectors:
connectors/source-mysql
https://github.com/airbytehq/airbyte/actions/runs/3177128020

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

akashkulk · 2022-10-03T20:07:27Z

/publish connector=connectors/source-mysql-strict-encrypt run-tests=false

🕑 Publishing the following connectors:
connectors/source-mysql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3177129547

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2022-10-03T20:29:39Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql
source-mysql-strict-encrypt

…ce (airbytehq#17170) - Implement logic + update spec - Fix tests - Bump dockerfile Co-authored-by: Octavia Squidington III <[email protected]>

akashkulk added 3 commits September 26, 2022 13:09

Update mysql spec.json file

436363c

Update mysql spec.json file

21ccf5b

Merge remote-tracking branch 'origin/akashkulk/mysql_debezium_timeout…

791b005

…' into akashkulk/mysql_debezium_timeout

github-actions bot added the area/connectors Connector related issues label Sep 26, 2022

octavia-squidington-iii added connectors/source/mysql connectors/source/jdbc labels Sep 26, 2022

akashkulk added 2 commits September 26, 2022 13:15

Merge remote-tracking branch 'origin/akashkulk/mysql_debezium_timeout…

8d54421

…' into akashkulk/mysql_debezium_timeout

Merge remote-tracking branch 'origin/akashkulk/mysql_debezium_timeout…

e99aae7

…' into akashkulk/mysql_debezium_timeout

Merge branch 'master' into akashkulk/mysql_debezium_timeout

aaeae0f

akashkulk added 3 commits September 26, 2022 13:31

Merge remote-tracking branch 'origin/master' into akashkulk/mysql_deb…

af921bf

…ezium_timeout

Update documentation

1b2052a

Merge remote-tracking branch 'origin/akashkulk/mysql_debezium_timeout…

69579b8

…' into akashkulk/mysql_debezium_timeout

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 26, 2022

Merge branch 'master' into akashkulk/mysql_debezium_timeout

d34b027

akashkulk added 2 commits September 27, 2022 13:34

Add configurable CDC timeout for MySQL

4e42577

Merge remote-tracking branch 'origin/akashkulk/mysql_debezium_timeout…

6e01b90

…' into akashkulk/mysql_debezium_timeout

Update expected_spec.json to fix source-mysql-strict-encrypt tests

07bc947

octavia-squidington-iii added the connectors/source/mysql-strict-encrypt label Sep 27, 2022

Attempt to speed up source-mysql tests

19ee470

- Bump up docker version for source-mysql-strict-encrypt

1d96e4d

subodh1810 approved these changes Oct 3, 2022

View reviewed changes

Merge branch 'master' into akashkulk/mysql_debezium_timeout

cc58926

- Bump up docker versions

4024073

- Fix failing test

3a7759c

Merge branch 'master' into akashkulk/mysql_debezium_timeout

dbf988f

Merge branch 'master' into akashkulk/mysql_debezium_timeout

343df78

akashkulk removed the request for review from rodireich October 3, 2022 20:06

octavia-squidington-iii and others added 2 commits October 3, 2022 20:16

auto-bump connector version [ci skip]

5e50261

Merge branch 'master' into akashkulk/mysql_debezium_timeout

6995056

akashkulk temporarily deployed to more-secrets October 3, 2022 20:31 Inactive

akashkulk merged commit bb6dff5 into master Oct 3, 2022

akashkulk deleted the akashkulk/mysql_debezium_timeout branch October 3, 2022 20:49

octavia-squidington-iii mentioned this pull request Oct 5, 2022

Bump Airbyte version from 0.40.10 to 0.40.11 #17610

Merged

🎉 MySQL Source : Make initial cdc timeout configurable for MySQL source #17170

🎉 MySQL Source : Make initial cdc timeout configurable for MySQL source #17170

Uh oh!

Conversation

akashkulk commented Sep 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

Uh oh!

github-actions bot commented Sep 26, 2022

Uh oh!

github-actions bot commented Sep 26, 2022

Uh oh!

github-actions bot commented Sep 26, 2022

Uh oh!

github-actions bot commented Sep 26, 2022

Uh oh!

github-actions bot commented Sep 27, 2022

Uh oh!

github-actions bot commented Sep 27, 2022

Uh oh!

akashkulk commented Sep 27, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

akashkulk commented Sep 27, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

akashkulk commented Sep 27, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

github-actions bot commented Sep 27, 2022

Uh oh!

github-actions bot commented Sep 27, 2022

Uh oh!

akashkulk commented Sep 30, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

github-actions bot commented Sep 30, 2022

Uh oh!

subodh1810 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

subodh1810 Oct 3, 2022

Choose a reason for hiding this comment

Uh oh!

akashkulk Oct 3, 2022

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 3, 2022

Uh oh!

github-actions bot commented Oct 3, 2022

Uh oh!

akashkulk commented Oct 3, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

Uh oh!

akashkulk commented Oct 3, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

github-actions bot commented Oct 3, 2022

Uh oh!

github-actions bot commented Oct 3, 2022

Uh oh!

github-actions bot commented Oct 3, 2022

Uh oh!

akashkulk commented Oct 3, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akashkulk commented Sep 26, 2022 •

edited

Loading

akashkulk commented Sep 27, 2022 •

edited by github-actions bot

Loading

akashkulk commented Sep 27, 2022 •

edited by github-actions bot

Loading

akashkulk commented Sep 27, 2022 •

edited by github-actions bot

Loading

akashkulk commented Sep 30, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 3, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 3, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 3, 2022 •

edited by github-actions bot

Loading

akashkulk commented Oct 3, 2022 •

edited by github-actions bot

Loading