Skip to content

Add debezium engine logs #37754

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 2, 2024
Merged

Add debezium engine logs #37754

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
18e5839
Add debezium engine logs
rodireich May 1, 2024
007a384
Add debezium engine logs
rodireich May 1, 2024
c480af4
Add debezium engine logs
rodireich May 1, 2024
7913556
bump metadata
rodireich May 1, 2024
ddd67ac
Merge branch 'master' into 4849-add-notify
rodireich May 2, 2024
c6c8bbf
reference published cdk
rodireich May 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions airbyte-cdk/java/airbyte-cdk/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ corresponds to that version.

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :--------------------------------------------------------- |:---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0.31.4 | 2024-04-30 | [\#37754](https://github.com/airbytehq/airbyte/pull/37754) | Add DebeziumEngine notification log |
| 0.31.3 | 2024-04-30 | [\#37726](https://github.com/airbytehq/airbyte/pull/37726) | Remove debezium retries |
| 0.31.2 | 2024-04-30 | [\#37507](https://github.com/airbytehq/airbyte/pull/37507) | Better error messages when switching between global/per-stream modes. |
| 0.31.0 | 2024-04-26 | [\#37584](https://github.com/airbytehq/airbyte/pull/37584) | Update S3 destination deps to exclude zookeeper and hadoop-yarn-common |
Expand Down
2 changes: 1 addition & 1 deletion airbyte-cdk/java/airbyte-cdk/core/src/main/resources/version.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
version=0.31.3
version=0.31.4
25 changes: 22 additions & 3 deletions ...src/main/kotlin/io/airbyte/cdk/integrations/debezium/internals/DebeziumRecordPublisher.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ class DebeziumRecordPublisher(private val debeziumPropertiesManager: DebeziumPro
.using(
debeziumPropertiesManager.getDebeziumProperties(
offsetManager,
schemaHistoryManager
)
schemaHistoryManager,
),
)
.using(OffsetCommitPolicy.AlwaysCommitOffsetPolicy())
.notifying { e: ChangeEvent<String?, String?> ->
Expand All @@ -59,7 +59,7 @@ class DebeziumRecordPublisher(private val debeziumPropertiesManager: DebeziumPro
.using { success: Boolean, message: String?, error: Throwable? ->
LOGGER.info(
"Debezium engine shutdown. Engine terminated successfully : {}",
success
success,
)
LOGGER.info(message)
if (!success) {
Expand All @@ -74,6 +74,25 @@ class DebeziumRecordPublisher(private val debeziumPropertiesManager: DebeziumPro
}
engineLatch.countDown()
}
.using (
object : DebeziumEngine.ConnectorCallback {
override fun connectorStarted() {
LOGGER.info("DebeziumEngine notify: connector started")
}

override fun connectorStopped() {
LOGGER.info("DebeziumEngine notify: connector stopped")
}

override fun taskStarted() {
LOGGER.info("DebeziumEngine notify: task started")
}

override fun taskStopped() {
LOGGER.info("DebeziumEngine notify: task stopped")
}
},
)
.build()

// Run the engine asynchronously ...
Expand Down
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-postgres/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ java {
}

airbyteJavaConnector {
cdkVersionRequired = '0.31.3'
cdkVersionRequired = '0.31.4'
features = ['db-sources', 'datastore-postgres']
useLocalCdk = false
useLocalCdk = true
}

application {
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-postgres/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ data:
connectorSubtype: database
connectorType: source
definitionId: decd338e-5647-4c0b-adf4-da0e75f5a750
dockerImageTag: 3.3.29
dockerImageTag: 3.3.31
dockerRepository: airbyte/source-postgres
documentationUrl: https://docs.airbyte.com/integrations/sources/postgres
githubIssueLabel: source-postgres
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,7 @@ According to Postgres [documentation](https://www.postgresql.org/docs/14/datatyp

| Version | Date | Pull Request | Subject |
|---------|------------|----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 3.3.31 | 2024-04-30 | [37754](https://github.com/airbytehq/airbyte/pull/37754) | Add CDC logs |
| 3.3.30 | 2024-04-30 | [37726](https://github.com/airbytehq/airbyte/pull/37726) | Remove debezium retries |
| 3.3.29 | 2024-04-23 | [37509](https://github.com/airbytehq/airbyte/pull/37509) | remove excessive logs |
| 3.3.28 | 2024-04-23 | [37509](https://github.com/airbytehq/airbyte/pull/37509) | Better error messages on switching between sync modes. |
Expand Down
Loading