Better formatting/logging for pre-sync data #25459

akashkulk · 2023-04-24T23:59:20Z

Better logging includes

Human readable formatting for table estimates + number of rows
Include cursor information with indexes to quickly verify if cursor field is indexed

akashkulk · 2023-04-25T00:00:24Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4792174590
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4792174590
🐛 https://gradle.com/s/kfloqve2odlpe

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs1] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs1]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
============== 4 failed, 64 passed, 5 skipped in 68.05s (0:01:08) ==============

github-actions · 2023-04-25T00:03:20Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (22)

Connector	Version	Changelog	Publish
`source-alloydb`	`2.0.23`	✅	✅
`source-alloydb-strict-encrypt`	`2.0.23`	🔵 (ignored)	🔵 (ignored)
`source-bigquery`	`0.2.3`	✅	✅
`source-clickhouse`	`0.1.17`	✅	✅
`source-clickhouse-strict-encrypt`	`0.1.17`	🔵 (ignored)	🔵 (ignored)
`source-cockroachdb`	`0.1.22`	✅	✅
`source-cockroachdb-strict-encrypt`	`0.1.22`	🔵 (ignored)	🔵 (ignored)
`source-db2`	`0.1.19`	✅	✅
`source-db2-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`source-mssql`	`1.0.14`	✅	✅
`source-mssql-strict-encrypt`	`1.0.14`	🔵 (ignored)	🔵 (ignored)
`source-mysql`	`2.0.18`	✅	✅
`source-mysql-strict-encrypt`	`2.0.18`	🔵 (ignored)	🔵 (ignored)
`source-oracle`	`0.3.24`	✅	✅
`source-oracle-strict-encrypt`	`0.3.24`	🔵 (ignored)	🔵 (ignored)
`source-postgres`	`2.0.25`	✅	✅
`source-postgres-strict-encrypt`	`2.0.25`	🔵 (ignored)	🔵 (ignored)
`source-redshift`	`0.3.16`	✅	✅
`source-scaffold-java-jdbc`	`0.1.0`	🔵 (ignored)	🔵 (ignored)
`source-snowflake`	`0.1.34`	✅	✅
`source-teradata`	`0.1.0`	✅	✅
`source-tidb`	`0.2.4`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

akashkulk · 2023-04-25T00:08:46Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/4792237247
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/4792237247
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
================= 22 passed, 6 skipped, 30 warnings in 18.17s ==================

akashkulk · 2023-04-25T02:26:01Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4793030419
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4793030419
🐛 https://gradle.com/s/zyozhliggzoao

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_core.py::TestBasicRead::test_read[inputs1] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs1]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
============== 4 failed, 64 passed, 5 skipped in 68.38s (0:01:08) ==============

akashkulk · 2023-04-25T16:46:55Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4800068601
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4800068601
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
=================== 68 passed, 5 skipped in 81.90s (0:01:21) ===================

akashkulk · 2023-04-25T17:28:52Z

/publish connector=connectors/source-postgres run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/4800427748

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-postgres	2.0.25	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

akashkulk · 2023-04-25T17:29:34Z

/publish connector=connectors/source-postgres-strict-encrypt run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4800432181

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	2.0.25	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

CLAassistant · 2023-04-25T17:41:05Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ akashkulk
❌ airbyteio
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

akashkulk · 2023-04-25T18:54:43Z

/publish connector=connectors/source-postgres run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/4801139082

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-postgres	2.0.25	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

akashkulk · 2023-04-25T18:55:05Z

/publish connector=connectors/source-postgres-strict-encrypt run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4801141802

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	2.0.25	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

akashkulk · 2023-04-25T20:08:06Z

Need to fix failing checks. Have deleted published images for now, so will need to go back and take a look

akashkulk · 2023-04-25T20:22:05Z

/publish connector=connectors/source-postgres run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/4801846096

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-postgres	2.0.25	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

akashkulk · 2023-04-25T20:22:14Z

/publish connector=connectors/source-postgres-strict-encrypt run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4801847692

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	2.0.25	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

akashkulk · 2023-04-25T20:59:26Z

/publish connector=connectors/source-postgres run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/4802125999

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-postgres	2.0.25	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

akashkulk · 2023-04-25T20:59:36Z

/publish connector=connectors/source-postgres-strict-encrypt run-tests=false

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4802126898

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	2.0.25	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

This reverts commit 0d9b7b4.

* Revert "Better formatting/logging for pre-sync data (#25459)" This reverts commit 0d9b7b4. * Fix dockerfile + docs * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <[email protected]>

* Log cursor alongside indexes * Better formatting for table estimates, indexes * Doc & dockerfile bump * Fix formatting * auto-bump connector version * Automated Commit - Formatting Changes * Fixing connector base issues * Fix failure * Fix changelog --------- Co-authored-by: Octavia Squidington III <[email protected]> Co-authored-by: airbyteio <[email protected]>

* Revert "Better formatting/logging for pre-sync data (airbytehq#25459)" This reverts commit 0d9b7b4. * Fix dockerfile + docs * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <[email protected]>

akashkulk added 2 commits April 20, 2023 13:36

Log cursor alongside indexes

66d3e7f

Better formatting for table estimates, indexes

0b7d903

octavia-squidington-iii added the area/connectors Connector related issues label Apr 24, 2023

Merge branch 'master' into moar_logging

9a44fad

octavia-squidington-iii added connectors/source/postgres connectors/source/jdbc labels Apr 24, 2023

akashkulk assigned rodireich Apr 25, 2023

akashkulk marked this pull request as ready for review April 25, 2023 00:00

akashkulk requested a review from a team as a code owner April 25, 2023 00:00

rodireich approved these changes Apr 25, 2023

View reviewed changes

Doc & dockerfile bump

a18fc88

octavia-squidington-iii added area/documentation Improvements or additions to documentation connectors/source/postgres-strict-encrypt labels Apr 25, 2023

Merge branch 'master' into moar_logging

45d6eb0

akashkulk and others added 2 commits April 25, 2023 09:46

Fix formatting

153c3f9

Merge branch 'master' into moar_logging

2995423

octavia-squidington-iii and others added 2 commits April 25, 2023 17:38

auto-bump connector version

cb85047

Automated Commit - Formatting Changes

292970b

Merge branch 'master' into moar_logging

eae85db

akashkulk enabled auto-merge (squash) April 25, 2023 18:35

Fixing connector base issues

906a900

akashkulk disabled auto-merge April 25, 2023 18:52

Merge branch 'master' into moar_logging

e356e21

akashkulk and others added 2 commits April 25, 2023 13:19

Fix failure

9cfb0ec

Merge branch 'master' into moar_logging

7eaaea3

akashkulk and others added 2 commits April 25, 2023 13:37

Fix changelog

7a83974

Merge branch 'master' into moar_logging

1e10492

akashkulk merged commit 0d9b7b4 into master Apr 25, 2023

akashkulk deleted the moar_logging branch April 25, 2023 21:12

akashkulk added a commit that referenced this pull request Apr 26, 2023

Revert "Better formatting/logging for pre-sync data (#25459)"

44ca0ca

This reverts commit 0d9b7b4.

akashkulk mentioned this pull request Apr 26, 2023

Revert "Better formatting/logging for pre-sync data" #25560

Merged

akashkulk mentioned this pull request Apr 26, 2023

https://github.com/airbytehq/airbyte/pull/25459 #25583

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Better formatting/logging for pre-sync data #25459

Better formatting/logging for pre-sync data #25459

akashkulk commented Apr 24, 2023

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Apr 25, 2023 •

edited

Loading

✅ Sources (22)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

CLAassistant commented Apr 25, 2023

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

Better formatting/logging for pre-sync data #25459

Better formatting/logging for pre-sync data #25459

Conversation

akashkulk commented Apr 24, 2023

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

Build Failed

github-actions bot commented Apr 25, 2023 • edited Loading

Affected Connector Report

✅ Sources (22)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

Build Failed

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

CLAassistant commented Apr 25, 2023

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

akashkulk commented Apr 25, 2023

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

akashkulk commented Apr 25, 2023 • edited by github-actions bot Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Apr 25, 2023 •

edited

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading

akashkulk commented Apr 25, 2023 •

edited by github-actions bot

Loading