Postgres Source Strict Encrypt: Add tests with SSL and SSH #19388

VitaliiMaltsev · 2022-11-14T17:42:37Z

What

Added integration tests with secure and insecure connections

🚨 User Impact 🚨

none

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

VitaliiMaltsev · 2022-11-14T17:43:13Z

/test connector=connectors/source-postgres-strict-encrypt

🕑 connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3463834273
✅ connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3463834273
No Python unittests run

Build Passed

Test summary info:

All Passed

VitaliiMaltsev · 2022-11-14T17:44:07Z

/test connector=connectors/source-alloydb-strict-encrypt

🕑 connectors/source-alloydb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3463840221
✅ connectors/source-alloydb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3463840221
No Python unittests run

Build Passed

Test summary info:

All Passed

VitaliiMaltsev · 2022-11-14T17:44:15Z

/test connector=connectors/source-alloydb

🕑 connectors/source-alloydb https://github.com/airbytehq/airbyte/actions/runs/3463840691
✅ connectors/source-alloydb https://github.com/airbytehq/airbyte/actions/runs/3463840691
No Python unittests run

Build Passed

Test summary info:

All Passed

github-actions · 2022-11-14T17:44:24Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector version
Add changelog
Publish the new version

⚠ Sources (3)

Connector	Version	Changelog	Publish
`source-alloydb`	`1.0.17`	✅	✅
`source-alloydb-strict-encrypt`	`1.0.17`	✅	⚠ (not in seed)
`source-postgres-strict-encrypt`	`1.0.23`	✅	⚠ (not in seed)

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

VitaliiMaltsev · 2022-11-15T08:34:29Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3468745384
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3468745384
🐛 https://gradle.com/s/wvxausri2orzk

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_enum_usage[inputs0] - TypeError: list ind...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
============ 1 failed, 13 passed, 5 skipped, 21 warnings in 26.54s =============

rodireich · 2022-11-15T18:13:49Z

...tgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSourceStrictEncrypt.java

+        if (Set.of(SSL_MODE_DISABLE, SSL_MODE_ALLOW, SSL_MODE_PREFER).contains(config.get(SSL_MODE).get(MODE).asText())) {
+          // Fail in case SSL mode is disable, allow or prefer
+          return new AirbyteConnectionStatus()
+              .withStatus(AirbyteConnectionStatus.Status.FAILED)


nit: in source-postgres-strict-encrypt it's slightly different

.withStatus(Status.FAILED) .withMessage( "Unsecured connection not allowed. If no SSH Tunnel set up, please use one of the following SSL modes: require, verify-ca, verify-full"); }```

nit: in source-postgres-strict-encrypt it's slightly different

.withStatus(Status.FAILED) .withMessage( "Unsecured connection not allowed. If no SSH Tunnel set up, please use one of the following SSL modes: require, verify-ca, verify-full"); }```

now up-to-date with source-postgres-strict-encrypt

VitaliiMaltsev · 2022-11-16T16:04:50Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3481017806
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3481017806
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1563    349    78%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 30.74s ==================

* Postgres Source Strict Encrypt: Add tests with SSL and SSH * updated No tunnel test * fixed spec test * fixed spec test * refactoring

Postgres Source Strict Encrypt: Add tests with SSL and SSH

be8312a

octavia-squidington-iv added area/connectors Connector related issues connectors/source/postgres labels Nov 14, 2022

VitaliiMaltsev temporarily deployed to more-secrets November 14, 2022 17:44 Inactive

updated No tunnel test

b9ace20

VitaliiMaltsev temporarily deployed to more-secrets November 14, 2022 17:48 Inactive

fixed spec test

9bfa0d1

VitaliiMaltsev temporarily deployed to more-secrets November 14, 2022 19:03 Inactive

fixed spec test

8fa8dd2

VitaliiMaltsev temporarily deployed to more-secrets November 15, 2022 08:36 Inactive

VitaliiMaltsev marked this pull request as ready for review November 15, 2022 13:27

VitaliiMaltsev requested a review from a team as a code owner November 15, 2022 13:27

VitaliiMaltsev requested review from rodireich and bleonard November 15, 2022 13:27

rodireich reviewed Nov 15, 2022

View reviewed changes

rodireich approved these changes Nov 15, 2022

View reviewed changes

VitaliiMaltsev added 2 commits November 16, 2022 12:58

Merge branch 'master' into vmaltsev/19209-postgres-source-ssh-ssl-tests

40814af

refactoring

47a0428

VitaliiMaltsev temporarily deployed to more-secrets November 16, 2022 11:05 Inactive

Merge branch 'master' into vmaltsev/19209-postgres-source-ssh-ssl-tests

93b635c

VitaliiMaltsev temporarily deployed to more-secrets November 16, 2022 16:05 Inactive

VitaliiMaltsev merged commit 7d050cf into master Nov 16, 2022

VitaliiMaltsev deleted the vmaltsev/19209-postgres-source-ssh-ssl-tests branch November 16, 2022 17:50

github-actions bot mentioned this pull request Nov 17, 2022

Bump helm chart version reference to 0.40.44 #19546

Merged

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

VitaliiMaltsev mentioned this pull request Nov 22, 2022

🐛 Postgres Source Strict Encrypt: Allow connections with sslmodes 'allow' and 'prefer' if SSH tunnel established #19551

Merged

37 tasks

akashkulk pushed a commit that referenced this pull request Dec 2, 2022

Postgres Source Strict Encrypt: Add tests with SSL and SSH (#19388)

e786be2

* Postgres Source Strict Encrypt: Add tests with SSL and SSH * updated No tunnel test * fixed spec test * fixed spec test * refactoring

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Postgres Source Strict Encrypt: Add tests with SSL and SSH #19388

Postgres Source Strict Encrypt: Add tests with SSL and SSH #19388

VitaliiMaltsev commented Nov 14, 2022 •

edited

Loading

VitaliiMaltsev commented Nov 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Nov 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Nov 14, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Nov 14, 2022 •

edited

Loading

⚠ Sources (3)

✅ Destinations (0)

Actionable Items

VitaliiMaltsev commented Nov 15, 2022 •

edited by github-actions bot

Loading

rodireich Nov 15, 2022

VitaliiMaltsev Nov 16, 2022

VitaliiMaltsev commented Nov 16, 2022 •

edited by github-actions bot

Loading

Postgres Source Strict Encrypt: Add tests with SSL and SSH #19388

Postgres Source Strict Encrypt: Add tests with SSL and SSH #19388

Conversation

VitaliiMaltsev commented Nov 14, 2022 • edited Loading

What

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

VitaliiMaltsev commented Nov 14, 2022 • edited by github-actions bot Loading

Build Passed

VitaliiMaltsev commented Nov 14, 2022 • edited by github-actions bot Loading

Build Passed

VitaliiMaltsev commented Nov 14, 2022 • edited by github-actions bot Loading

Build Passed

github-actions bot commented Nov 14, 2022 • edited Loading

Affected Connector Report

⚠ Sources (3)

✅ Destinations (0)

Actionable Items

VitaliiMaltsev commented Nov 15, 2022 • edited by github-actions bot Loading

Build Failed

rodireich Nov 15, 2022

Choose a reason for hiding this comment

VitaliiMaltsev Nov 16, 2022

Choose a reason for hiding this comment

VitaliiMaltsev commented Nov 16, 2022 • edited by github-actions bot Loading

Build Passed

VitaliiMaltsev commented Nov 14, 2022 •

edited

Loading

VitaliiMaltsev commented Nov 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Nov 14, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Nov 14, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Nov 14, 2022 •

edited

Loading

VitaliiMaltsev commented Nov 15, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Nov 16, 2022 •

edited by github-actions bot

Loading