upgrade debezium version to 1.9.2 for mysql source #14542

subodh1810 · 2022-07-08T17:00:15Z

subodh1810 · 2022-07-09T10:41:18Z

...s/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSourceOperations.java

@@ -107,7 +107,7 @@ public void setJsonField(final ResultSet resultSet, final int colIndex, final Ob
   */
  @Override
  protected void putBoolean(final ObjectNode node, final String columnName, final ResultSet resultSet, final int index) throws SQLException {
-    node.put(columnName, resultSet.getInt(index) == 1);
+    node.put(columnName, resultSet.getInt(index) > 0);


This is not related to debezium version upgrade, I found a bug while testing in Full refresh mode so went ahead and fixed it

subodh1810 · 2022-07-09T10:42:48Z

...-v1-9-2/src/main/java/io/airbyte/integrations/debezium/internals/MySQLDateTimeConverter.java

@@ -40,24 +39,9 @@ public void configure(final Properties props) {}
  public void converterFor(final RelationalColumn field, final ConverterRegistration<SchemaBuilder> registration) {
    if (Arrays.stream(DATE_TYPES).anyMatch(s -> s.equalsIgnoreCase(field.typeName()))) {
      registerDate(field, registration);
-    } else if (Arrays.stream(TEXT_TYPES).anyMatch(s -> s.equalsIgnoreCase(field.typeName()))) {


The new debezium version handles string values as expected so our custom logic is not required anymore

subodh1810 · 2022-07-09T10:44:00Z

...-2/src/main/java/io/airbyte/integrations/debezium/internals/FilteredFileDatabaseHistory.java

- * {@link #storeRecord(HistoryRecord)} method but {@link FilteredFileDatabaseHistory} is a final
- * class and can not be inherited
- */
-public class FilteredFileDatabaseHistory extends AbstractDatabaseHistory {


The new debezium version filters out the schema history records only for the database that its syncing so our custom logic is not required anymore

subodh1810 · 2022-07-09T10:46:30Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2640817796
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2640817796
No Python unittests run

Build Passed

Test summary info:

All Passed

subodh1810 · 2022-07-09T18:14:39Z

.../connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java


      final Optional<CdcState> cdcState = Optional.ofNullable(stateManager.getCdcStateManager().getCdcState());
      final MySqlCdcSavedInfoFetcher fetcher = new MySqlCdcSavedInfoFetcher(cdcState.orElse(null));
-      cdcState.ifPresent(cdc -> checkBinlog(cdc.getState(), database));


This is no longer required cause now if the last saved binlog position in the source database is purged, we would handle it gracefully and sync data from beginning. I have added a test case as well for this scenario.

subodh1810 · 2022-07-09T18:16:12Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2641969287
❌ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2641969287
🐛 https://gradle.com/s/bmbg3lbaq7zha

Build Failed

Test summary info:

Could not find result summary

subodh1810 · 2022-07-10T09:39:49Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2644179215
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2644179215
No Python unittests run

Build Passed

Test summary info:

All Passed

grishick · 2022-07-13T01:40:41Z

I cannot tell from the code if/how this change will break jobs that were created with the current version of MySQL Connector. Could you test that manually?

subodh1810 · 2022-07-13T09:45:27Z

@grishick I tested this manually by setting up a mysql connector on version 0.5.15 (current version) in CDC mode and then ran few syncs and then switched the connector to use my local dev version which contained the changes from the PR and it worked fine so this is not going to cause a breaking change.

I incorrectly assumed that it would have caused a breaking change cause of failing tests when I had upgraded the debezium version and realised that the FileDatabaseHistory is no longer required with the latest debezium version.

edgao

nice! just a couple nitpicks

edgao · 2022-07-13T18:48:39Z

...rs/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlCdcStateHandler.java

@@ -51,4 +51,9 @@ public AirbyteMessage saveState(final Map<String, String> offset, final String d
    return new AirbyteMessage().withType(Type.STATE).withState(stateMessage);
  }

+  @Override
+  public AirbyteMessage saveStateAfterCompletionOfSnapshotOfNewStreams() {
+    throw new RuntimeException("Snapshot of individual tables currently not supported in MySQL");


doublechecking: this only runs when someone adds new streams to an existing catalog, which isn't currently supported in the UI anyway?

@edgao correct plus the stream state type for mysql is LEGACY so even when the feature would be enabled in the UI, it would not be triggered for mysql

edgao · 2022-07-13T18:52:31Z

...tors/source-mysql/src/test/java/io/airbyte/integrations/source/mysql/CdcMysqlSourceTest.java

+        "test_schema", "table_with_tiny_int", "id", "bool_col", "tiny_int_one_col",
+        "tiny_int_two_col", "id"));
+
+    executeQuery(String


nitpick: are these two queries identical? if yes, probably better to loop over data rather than copy/pasting them?

edgao · 2022-07-13T18:54:32Z

...tors/source-mysql/src/test/java/io/airbyte/integrations/source/mysql/CdcMysqlSourceTest.java

+    return expectedCatalog;
+  }
+
+  //TODO : Enable this test once we fix handling of DATETIME values


which issue is this waiting for? #14628 ?

edgao · 2022-07-13T18:55:23Z

...tors/source-mysql/src/test/java/io/airbyte/integrations/source/mysql/CdcMysqlSourceTest.java

+  }
+
+  @Test
+  protected void syncShouldHandlePurgedLogsGracefully() throws Exception {


can this live in the base cdc source test? (and sources could explicitly opt out if it's not supported)

I plan to make this a generic test in another PR

subodh1810 · 2022-07-20T08:58:01Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2703506127
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2703506127
No Python unittests run

Build Passed

Test summary info:

All Passed

* upgrade debezium version to 1.9.2 for mysql source * new debezium version doen't require special string handling * fix log message * add test tor date time data type * gracefully handle the case when logs are purged * fix test * address review comment

upgrade debezium version to 1.9.2 for mysql source

360b39d

github-actions bot added the area/connectors Connector related issues label Jul 8, 2022

octavia-squidington-iii added the connectors/source/mysql label Jul 8, 2022

subodh1810 added 2 commits July 8, 2022 22:49

Merge branch 'master' into upgrade-debezium-version-mysql-1-9-2

aa44d02

new debezium version doen't require special string handling

3d9109d

subodh1810 self-assigned this Jul 9, 2022

subodh1810 commented Jul 9, 2022

View reviewed changes

subodh1810 marked this pull request as ready for review July 9, 2022 10:44

subodh1810 requested a review from a team as a code owner July 9, 2022 10:44

subodh1810 linked an issue Jul 9, 2022 that may be closed by this pull request

Upgrade debezium version for MySQL source #14425

Closed

subodh1810 added 2 commits July 9, 2022 16:17

fix log message

3c7e5e0

add test tor date time data type

a54dc8d

subodh1810 requested a review from edgao July 9, 2022 14:56

gracefully handle the case when logs are purged

9fdad7c

subodh1810 commented Jul 9, 2022

View reviewed changes

subodh1810 requested a review from grishick July 9, 2022 18:15

fix test

628ce8a

Merge branch 'master' into upgrade-debezium-version-mysql-1-9-2

ae2c9b7

edgao approved these changes Jul 13, 2022

View reviewed changes

grishick approved these changes Jul 14, 2022

View reviewed changes

subodh1810 added 2 commits July 20, 2022 14:11

Merge branch 'master' into upgrade-debezium-version-mysql-1-9-2

e2981f0

address review comment

c193ba0

subodh1810 merged commit 83b0f81 into master Jul 20, 2022

subodh1810 deleted the upgrade-debezium-version-mysql-1-9-2 branch July 20, 2022 10:45

octavia-squidington-iii mentioned this pull request Jul 23, 2022

Bump Airbyte version from 0.39.37-alpha to 0.39.38-alpha #14976

Merged

subodh1810 mentioned this pull request Jul 25, 2022

Fix multiple log bindings #14801

Merged

marcosmarxm mentioned this pull request Jul 25, 2022

Mysql to Bigquery Incremental Deduped Sync Failure #15011

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upgrade debezium version to 1.9.2 for mysql source #14542

upgrade debezium version to 1.9.2 for mysql source #14542

subodh1810 commented Jul 8, 2022 •

edited

Loading

subodh1810 Jul 9, 2022

subodh1810 Jul 9, 2022 •

edited

Loading

subodh1810 Jul 9, 2022 •

edited

Loading

subodh1810 commented Jul 9, 2022 •

edited by github-actions bot

Loading

subodh1810 Jul 9, 2022 •

edited

Loading

subodh1810 commented Jul 9, 2022 •

edited by github-actions bot

Loading

subodh1810 commented Jul 10, 2022 •

edited by github-actions bot

Loading

grishick commented Jul 13, 2022

subodh1810 commented Jul 13, 2022

edgao left a comment

edgao Jul 13, 2022

subodh1810 Jul 19, 2022

edgao Jul 13, 2022

subodh1810 Jul 20, 2022

edgao Jul 13, 2022

subodh1810 Jul 20, 2022

edgao Jul 13, 2022

subodh1810 Jul 20, 2022

subodh1810 commented Jul 20, 2022 •

edited by github-actions bot

Loading

upgrade debezium version to 1.9.2 for mysql source #14542

upgrade debezium version to 1.9.2 for mysql source #14542

Conversation

subodh1810 commented Jul 8, 2022 • edited Loading

Choose a reason for hiding this comment

subodh1810 Jul 9, 2022 • edited Loading

Choose a reason for hiding this comment

subodh1810 Jul 9, 2022 • edited Loading

Choose a reason for hiding this comment

subodh1810 commented Jul 9, 2022 • edited by github-actions bot Loading

Build Passed

subodh1810 Jul 9, 2022 • edited Loading

Choose a reason for hiding this comment

subodh1810 commented Jul 9, 2022 • edited by github-actions bot Loading

Build Failed

subodh1810 commented Jul 10, 2022 • edited by github-actions bot Loading

Build Passed

grishick commented Jul 13, 2022

subodh1810 commented Jul 13, 2022

edgao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

subodh1810 commented Jul 20, 2022 • edited by github-actions bot Loading

Build Passed

subodh1810 commented Jul 8, 2022 •

edited

Loading

subodh1810 Jul 9, 2022 •

edited

Loading

subodh1810 Jul 9, 2022 •

edited

Loading

subodh1810 commented Jul 9, 2022 •

edited by github-actions bot

Loading

subodh1810 Jul 9, 2022 •

edited

Loading

subodh1810 commented Jul 9, 2022 •

edited by github-actions bot

Loading

subodh1810 commented Jul 10, 2022 •

edited by github-actions bot

Loading

subodh1810 commented Jul 20, 2022 •

edited by github-actions bot

Loading