Cap each log field (message, stacktrace) to max 32K characters #30806

rodireich · 2023-09-27T21:10:04Z

What

A thread safety issue causes log lines to be interleaved with other AirbyteMessages when:

logging from another thread
Log line is over 8K characters

How

Cap max log line length to undo 8k by limiting both message size and logged stack trace size to under 16,000 characters

…oid overall size going over 8K

vercel · 2023-09-27T21:10:09Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 11, 2023 7:18pm

airbyte-integrations/bases/base-java/src/main/resources/log4j2.xml

…cate-log

rodireich · 2023-09-28T23:46:10Z

airbyte-integrations/bases/base-java/javabase.sh

@@ -15,6 +15,9 @@ if [[ $IS_CAPTURE_HEAP_DUMP_ON_ERROR = true ]]; then
      echo "APPLICATION=$APPLICATION"
  fi
 fi
+#30781 - Allocate 32KB for log4j appender buffer to ensure that each line is logged in a single println
+JAVA_OPTS=$JAVA_OPTS" -Dlog4j.encoder.byteBufferSize=32768"


This control the size of buffer defined in log4j's ConsoleAppender
(See here line 47)

This is the base shell command for all java connector.
We could put this JVM options separately on each connector's build.gradle but it's likely to be forgotten.
@davinchia @evantahler @prateekmukhedkar

airbyte-oss-build-runner · 2023-09-29T18:44:28Z

source-alloydb test report (commit `2c6028b9db`) - ❌

⏲️ Total pipeline duration: 04mn01s

Step	Result
Build connector tar	✅
Build source-alloydb docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	❌
Validate metadata for source-alloydb	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

airbyte-oss-build-runner · 2023-10-11T16:45:54Z

source-postgres-strict-encrypt test report (commit `40d3f99ca5`) - ❌

⏲️ Total pipeline duration: 10mn33s

Step	Result
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate metadata for source-postgres-strict-encrypt	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

airbyte-oss-build-runner · 2023-10-11T16:56:23Z

source-pokeapi test report (commit `884985241e`) - ✅

⏲️ Total pipeline duration: 01mn13s

Step	Result
Build source-pokeapi docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-pokeapi	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-pokeapi test

airbyte-oss-build-runner · 2023-10-11T17:23:34Z

source-mysql test report (commit `884985241e`) - ✅

⏲️ Total pipeline duration: 28mn57s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-mysql	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

airbyte-oss-build-runner · 2023-10-11T17:32:17Z

source-mysql-strict-encrypt test report (commit `884985241e`) - ✅

⏲️ Total pipeline duration: 08mn31s

Step	Result
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-mysql-strict-encrypt	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

airbyte-oss-build-runner · 2023-10-11T17:38:49Z

source-postgres-strict-encrypt test report (commit `884985241e`) - ❌

⏲️ Total pipeline duration: 06mn20s

Step	Result
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	❌
Acceptance tests	✅
Validate metadata for source-postgres-strict-encrypt	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

airbyte-oss-build-runner · 2023-10-11T18:02:44Z

source-pokeapi test report (commit `dfaadee40b`) - ✅

⏲️ Total pipeline duration: 02mn00s

Step	Result
Build source-pokeapi docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-pokeapi	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-pokeapi test

airbyte-oss-build-runner · 2023-10-11T18:27:56Z

source-postgres test report (commit `dfaadee40b`) - ✅

⏲️ Total pipeline duration: 28mn01s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-oss-build-runner · 2023-10-11T18:35:54Z

source-postgres-strict-encrypt test report (commit `dfaadee40b`) - ✅

⏲️ Total pipeline duration: 07mn46s

Step	Result
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-postgres-strict-encrypt	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

airbyte-oss-build-runner · 2023-10-11T18:39:52Z

source-mysql-strict-encrypt test report (commit `dfaadee40b`) - ✅

⏲️ Total pipeline duration: 03mn45s

Step	Result
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-mysql-strict-encrypt	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

airbyte-oss-build-runner · 2023-10-11T18:43:46Z

source-mysql test report (commit `dfaadee40b`) - ✅

⏲️ Total pipeline duration: 03mn40s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-mysql	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

airbyte-oss-build-runner · 2023-10-11T19:14:41Z

source-mysql-strict-encrypt test report (commit `b74634a7a4`) - ✅

⏲️ Total pipeline duration: 12mn05s

Step	Result
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-mysql-strict-encrypt	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

airbyte-oss-build-runner · 2023-10-11T19:32:01Z

source-postgres-strict-encrypt test report (commit `50d8f7664d`) - ✅

⏲️ Total pipeline duration: 11mn11s

Step	Result
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-postgres-strict-encrypt	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

airbyte-oss-build-runner · 2023-10-11T19:58:02Z

source-postgres test report (commit `50d8f7664d`) - ✅

⏲️ Total pipeline duration: 25mn47s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-postgres	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-oss-build-runner · 2023-10-11T20:25:35Z

source-mysql test report (commit `50d8f7664d`) - ✅

⏲️ Total pipeline duration: 27mn21s

Step	Result
Build connector tar	✅
Build source-mysql docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-mysql	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

airbyte-oss-build-runner · 2023-10-11T20:34:26Z

source-mysql-strict-encrypt test report (commit `50d8f7664d`) - ✅

⏲️ Total pipeline duration: 08mn39s

Step	Result
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate metadata for source-mysql-strict-encrypt	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

…tehq#30806) Co-authored-by: rodireich <[email protected]>

Cap each log field (message, stacktrace) to max 3750 characters to av…

59ea251

…oid overall size going over 8K

rodireich requested review from davinchia and a team September 27, 2023 21:10

rodireich requested a review from a team as a code owner September 27, 2023 21:10

rodireich linked an issue Sep 27, 2023 that may be closed by this pull request

🐛 Large log message in java connectors can lead to loss of records #30781

Closed

Automated Commit - Formatting Changes

e29f19c

tryangul reviewed Sep 27, 2023

View reviewed changes

airbyte-integrations/bases/base-java/src/main/resources/log4j2.xml Outdated Show resolved Hide resolved

rodireich added 2 commits September 27, 2023 14:45

add comment

37c95ae

sanity

bae81f8

prateekmukhedkar approved these changes Sep 27, 2023

View reviewed changes

rodireich added 2 commits September 28, 2023 16:24

Increate log appender buffer size to 32KB

63f2404

sanity

a007233

rodireich requested a review from evantahler September 28, 2023 23:33

Merge branch 'master' of github.com:airbytehq/airbyte into 30781-trun…

29f22c0

…cate-log

octavia-squidington-iii added the CDK Connector Development Kit label Sep 28, 2023

evantahler approved these changes Sep 28, 2023

View reviewed changes

vercel bot deployed to Preview September 28, 2023 23:36 View deployment

rodireich commented Sep 28, 2023

View reviewed changes

Merge branch 'master' into 30781-truncate-log

6d891d3

vercel bot deployed to Preview September 29, 2023 17:08 View deployment

bump versions and update changelogs

2c6028b

rodireich changed the title ~~Cap each log field (message, stacktrace) to max 3750 characters~~ Cap each log field (message, stacktrace) to max 32K characters Oct 6, 2023

rebase from master

40d3f99

vercel bot deployed to Preview October 11, 2023 16:35 View deployment

Automated Commit - Formatting Changes

8849852

octavia-squidington-iii added the connectors/source/pokeapi label Oct 11, 2023

update build gardle to fix failing tests

dfaadee

Merge branch 'master' into 30781-truncate-log

b74634a

vercel bot deployed to Preview October 11, 2023 19:02 View deployment

update test

50d8f76

rodireich merged commit 3d9578f into master Oct 11, 2023

rodireich deleted the 30781-truncate-log branch October 11, 2023 20:34

mrhallak mentioned this pull request Oct 18, 2023

Cannot view bigger logs or download them #31178

Closed

ariesgun pushed a commit to ariesgun/airbyte that referenced this pull request Oct 23, 2023

Cap each log field (message, stacktrace) to max 32K characters (airby…

6f3df54

…tehq#30806) Co-authored-by: rodireich <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cap each log field (message, stacktrace) to max 32K characters #30806

Cap each log field (message, stacktrace) to max 32K characters #30806

rodireich commented Sep 27, 2023 •

edited

Loading

vercel bot commented Sep 27, 2023 •

edited

Loading

rodireich Sep 28, 2023

airbyte-oss-build-runner commented Sep 29, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

airbyte-oss-build-runner commented Oct 11, 2023

Cap each log field (message, stacktrace) to max 32K characters #30806

Cap each log field (message, stacktrace) to max 32K characters #30806

Conversation

rodireich commented Sep 27, 2023 • edited Loading

What

How

vercel bot commented Sep 27, 2023 • edited Loading

rodireich Sep 28, 2023

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Sep 29, 2023

source-alloydb test report (commit 2c6028b9db) - ❌

airbyte-oss-build-runner commented Oct 11, 2023

source-postgres-strict-encrypt test report (commit 40d3f99ca5) - ❌

airbyte-oss-build-runner commented Oct 11, 2023

source-pokeapi test report (commit 884985241e) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-mysql test report (commit 884985241e) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-mysql-strict-encrypt test report (commit 884985241e) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-postgres-strict-encrypt test report (commit 884985241e) - ❌

airbyte-oss-build-runner commented Oct 11, 2023

source-pokeapi test report (commit dfaadee40b) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-postgres test report (commit dfaadee40b) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-postgres-strict-encrypt test report (commit dfaadee40b) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-mysql-strict-encrypt test report (commit dfaadee40b) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-mysql test report (commit dfaadee40b) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-mysql-strict-encrypt test report (commit b74634a7a4) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-postgres-strict-encrypt test report (commit 50d8f7664d) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-postgres test report (commit 50d8f7664d) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-mysql test report (commit 50d8f7664d) - ✅

airbyte-oss-build-runner commented Oct 11, 2023

source-mysql-strict-encrypt test report (commit 50d8f7664d) - ✅

rodireich commented Sep 27, 2023 •

edited

Loading

vercel bot commented Sep 27, 2023 •

edited

Loading

source-alloydb test report (commit `2c6028b9db`) - ❌

source-postgres-strict-encrypt test report (commit `40d3f99ca5`) - ❌

source-pokeapi test report (commit `884985241e`) - ✅

source-mysql test report (commit `884985241e`) - ✅

source-mysql-strict-encrypt test report (commit `884985241e`) - ✅

source-postgres-strict-encrypt test report (commit `884985241e`) - ❌

source-pokeapi test report (commit `dfaadee40b`) - ✅

source-postgres test report (commit `dfaadee40b`) - ✅

source-postgres-strict-encrypt test report (commit `dfaadee40b`) - ✅

source-mysql-strict-encrypt test report (commit `dfaadee40b`) - ✅

source-mysql test report (commit `dfaadee40b`) - ✅

source-mysql-strict-encrypt test report (commit `b74634a7a4`) - ✅

source-postgres-strict-encrypt test report (commit `50d8f7664d`) - ✅

source-postgres test report (commit `50d8f7664d`) - ✅

source-mysql test report (commit `50d8f7664d`) - ✅

source-mysql-strict-encrypt test report (commit `50d8f7664d`) - ✅