Skip to content

Postgres Source: use native Postgres timeout if it's not set by the user #19291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Nov 16, 2022
Merged

Postgres Source: use native Postgres timeout if it's not set by the user #19291

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
6817961
Postgres Source: use native Postgres timeout if it's not set by the user
VitaliiMaltsev Nov 10, 2022
01625a7
refactoring
VitaliiMaltsev Nov 11, 2022
0002cc1
Merge branch 'master' into vmaltsev/3063-postgres-source-timeouts
VitaliiMaltsev Nov 11, 2022
a016852
updated connection timeout logic and added tests for different dataso…
VitaliiMaltsev Nov 15, 2022
21b1146
Merge remote-tracking branch 'origin/vmaltsev/3063-postgres-source-ti…
VitaliiMaltsev Nov 15, 2022
7df7e83
fixed pmd
VitaliiMaltsev Nov 16, 2022
3b6e70b
refactoring
VitaliiMaltsev Nov 16, 2022
e07e9e3
Merge branch 'master' into vmaltsev/3063-postgres-source-timeouts
VitaliiMaltsev Nov 16, 2022
559d275
bump version
VitaliiMaltsev Nov 16, 2022
bebdfc8
auto-bump connector version
octavia-squidington-iii Nov 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 18 additions & 12 deletions airbyte-db/db-lib/src/main/java/io/airbyte/db/factory/DataSourceFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,15 @@

package io.airbyte.db.factory;

import static org.postgresql.PGProperty.CONNECT_TIMEOUT;

import com.google.common.base.Preconditions;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import java.io.Closeable;
import java.time.Duration;
import java.util.Map;
import java.util.Objects;
import javax.sql.DataSource;

/**
Expand Down Expand Up @@ -61,7 +64,7 @@ public static DataSource create(final String username,
.withJdbcUrl(jdbcConnectionString)
.withPassword(password)
.withUsername(username)
.withConnectionTimeoutMs(DataSourceBuilder.getConnectionTimeoutMs(connectionProperties))
.withConnectionTimeoutMs(DataSourceBuilder.getConnectionTimeoutMs(connectionProperties, driverClassName))
.build();
}

Expand Down Expand Up @@ -179,7 +182,6 @@ private static class DataSourceBuilder {
private String password;
private int port = 5432;
private String username;
private static final String CONNECT_TIMEOUT_KEY = "connectTimeout";
private static final Duration CONNECT_TIMEOUT_DEFAULT = Duration.ofSeconds(60);

private DataSourceBuilder() {}
Expand All @@ -196,19 +198,23 @@ private DataSourceBuilder() {}
*
* @param connectionProperties custom jdbc_url_parameters containing information on connection
* properties
* @param driverClassName name of the JDBC driver
* @return DataSourceBuilder class used to create dynamic fields for DataSource
*/
private static long getConnectionTimeoutMs(final Map<String, String> connectionProperties) {
final Duration connectionTimeout;
// TODO: the usage of CONNECT_TIMEOUT_KEY is Postgres specific, may need to extend for other
// databases
connectionTimeout =
connectionProperties.containsKey(CONNECT_TIMEOUT_KEY) ? Duration.ofSeconds(Long.parseLong(connectionProperties.get(CONNECT_TIMEOUT_KEY)))
: CONNECT_TIMEOUT_DEFAULT;
if (connectionTimeout.getSeconds() == 0) {
return connectionTimeout.toMillis();
private static long getConnectionTimeoutMs(final Map<String, String> connectionProperties, String driverClassName) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test for this new functionality in DataSourceFactoryTest.java?

Copy link
Contributor

@ryankfu ryankfu Nov 15, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears these changes were caught within the build failures

To add onto what Akash said, since this is a postgres specific feature I would add a test primarily for postgres and keep a generic one to account for non-postgres behavior.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears these changes were caught within the build failures

To add onto what Akash said, since this is a postgres specific feature I would add a test primarily for postgres and keep a generic one to account for non-postgres behavior.

@akashkulk @ryankfu added tests for Postgres and MySQL to see the difference

// TODO: the usage of CONNECT_TIMEOUT is Postgres specific, may need to extend for other databases

if (driverClassName.equals(DatabaseDriver.POSTGRESQL.getDriverClassName())) {
final String pgPropertyConnectTimeout = CONNECT_TIMEOUT.getName();
// If the PGProperty.CONNECT_TIMEOUT was set by the user, then take its value, if not take the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: would remove the comment on #L206 and move this comment to above the if block for the Postgres logic. Otherwise changes look good

// default
return (connectionProperties.containsKey(pgPropertyConnectTimeout)
&& (Long.parseLong(connectionProperties.get(pgPropertyConnectTimeout)) > 0))
? Duration.ofSeconds(Long.parseLong(connectionProperties.get(pgPropertyConnectTimeout))).toMillis()
: Duration.ofSeconds(Long.parseLong(Objects.requireNonNull(CONNECT_TIMEOUT.getDefaultValue()))).toMillis();

} else {
return (connectionTimeout.compareTo(CONNECT_TIMEOUT_DEFAULT) > 0 ? connectionTimeout : CONNECT_TIMEOUT_DEFAULT).toMillis();
return CONNECT_TIMEOUT_DEFAULT.toMillis();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm understanding this correctly this changes the default behavior for all other connectors where the user can pass in a connection_timeout property. In the javadoc description, we allow for users to set the connection_timeout to 0 which means timeout duration is infinity

This seems to change that behavior for all connectors that isn't postgres. Also as a note the reason why default of 60 seconds was chosen in the past was due to integration tests failing on the default of 30 seconds (also in the javadoc description). If that has changed, then updating the javadoc comment to match makes sense

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm understanding this correctly this changes the default behavior for all other connectors where the user can pass in a connection_timeout property. In the javadoc description, we allow for users to set the connection_timeout to 0 which means timeout duration is infinity

This seems to change that behavior for all connectors that isn't postgres. Also as a note the reason why default of 60 seconds was chosen in the past was due to integration tests failing on the default of 30 seconds (also in the javadoc description). If that has changed, then updating the javadoc comment to match makes sense

@ryankfu good point
changed logic on
if (driverClassName.equals(DatabaseDriver.POSTGRESQL.getDriverClassName())) { new behaviour ; } else { old behaviour ; }

}
}

Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/postgres.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -400,6 +400,7 @@ The root causes is that the WALs needed for the incremental sync has been remove

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1.0.24 | 2022-11-07 | [19291](https://github.com/airbytehq/airbyte/pull/19291) | Use native Postgres connection timeout |
| 1.0.23 | 2022-11-07 | [19025](https://github.com/airbytehq/airbyte/pull/19025) | Stop enforce SSL if ssl mode is disabled |
| 1.0.22 | 2022-10-31 | [18538](https://github.com/airbytehq/airbyte/pull/18538) | Encode database name |
| 1.0.21 | 2022-10-25 | [18256](https://github.com/airbytehq/airbyte/pull/18256) | Disable allow and prefer ssl modes in CDC mode |
Expand Down