Skip to content

🎉 Source redshift: implement privileges check #9744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 34 commits into from
Mar 1, 2022
Merged

🎉 Source redshift: implement privileges check #9744

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
a98a8e6
fix: postgres priviledge check for redshift
ethanve Jan 6, 2022
ee65b0d
add missing imports
alafanechere Jan 24, 2022
f17818c
Merge branch 'master' into augustin/fix/redshift-privileges
alafanechere Jan 24, 2022
fcecd98
Merge branch 'master' into augustin/fix/redshift-privileges
alafanechere Jan 26, 2022
0396618
bump version
alafanechere Jan 26, 2022
33ada8f
format
alafanechere Jan 26, 2022
3ba2418
Update docs/integrations/sources/redshift.md
alafanechere Jan 27, 2022
7ecc9e4
remove not used fields from query results
alafanechere Jan 31, 2022
3d996e4
improve changelog
alafanechere Jan 31, 2022
72690a7
create unreadable table
alafanechere Jan 31, 2022
fb2b6a6
Merge branch 'master' into augustin/fix/redshift-privileges
alafanechere Jan 31, 2022
5eaf663
update documentation url in spec.json
alafanechere Jan 31, 2022
b079fdc
fix tests
alafanechere Jan 31, 2022
4ce02a6
fix tests
alafanechere Jan 31, 2022
c945988
bump version to 0.3.8 in dockerfile
alafanechere Feb 4, 2022
b9b9227
Fix method override
tuliren Feb 4, 2022
1c9ed2d
Limit schemas in jdbc test
tuliren Feb 4, 2022
81591a2
Revert jdbc acceptance test change
tuliren Feb 5, 2022
ceba782
Fix sql statement
tuliren Feb 5, 2022
81e73de
Fix empty stream name and wrong username
tuliren Feb 5, 2022
1137d6a
Close the connection that queries for privileges in redshift
tuliren Feb 5, 2022
0c3e8a7
Add more comment
tuliren Feb 5, 2022
f214661
fix SQL query to check privileges
alafanechere Feb 21, 2022
7f2eec5
Merge branch 'master' into augustin/fix/redshift-privileges
alafanechere Feb 21, 2022
a7782f4
bump to 0.3.9
alafanechere Feb 21, 2022
d698334
fix SQL query to check privileges
alafanechere Feb 21, 2022
d940fc0
fix SQL query to check privileges
alafanechere Feb 21, 2022
5285228
use specific test user
alafanechere Feb 22, 2022
87d343e
Merge branch 'master' into augustin/fix/redshift-privileges
alafanechere Feb 22, 2022
cddf939
Merge branch 'master' into augustin/fix/redshift-privileges
alafanechere Feb 23, 2022
e29b817
make changes suggested by liren
alafanechere Feb 28, 2022
f7aa4f2
update source_specs.yaml
alafanechere Mar 1, 2022
b47cb35
Merge branch 'master' into augustin/fix/redshift-privileges
alafanechere Mar 1, 2022
740c8bc
format
alafanechere Mar 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...ain/resources/config/STANDARD_SOURCE_DEFINITION/e87ffa8e-a3b5-f69c-9076-6011339de1f6.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"sourceDefinitionId": "e87ffa8e-a3b5-f69c-9076-6011339de1f6",
"name": "Redshift",
"dockerRepository": "airbyte/source-redshift",
"dockerImageTag": "0.3.7",
"dockerImageTag": "0.3.8",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, we no longer depend on these individual connector definitions. You don't need to update them any more. These files are still in our codebase for backward compatibility.

Actually I will check if we can remove all of them, since we already had that pseudo major version bump, and backward compatibility may not be needed.

"documentationUrl": "https://docs.airbyte.io/integrations/sources/redshift",
"icon": "redshift.svg"
}
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/source_definitions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -606,7 +606,7 @@
- name: Redshift
sourceDefinitionId: e87ffa8e-a3b5-f69c-9076-6011339de1f6
dockerRepository: airbyte/source-redshift
dockerImageTag: 0.3.7
dockerImageTag: 0.3.8
documentationUrl: https://docs.airbyte.io/integrations/sources/redshift
icon: redshift.svg
sourceType: database
Expand Down
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/source_specs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6330,7 +6330,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-redshift:0.3.7"
- dockerImage: "airbyte/source-redshift:0.3.8"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/destinations/redshift"
connectionSpecification:
Expand Down
24 changes: 24 additions & 0 deletions ...source-redshift/src/main/java/io/airbyte/integrations/source/redshift/RedshiftSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

package io.airbyte.integrations.source.redshift;

import static java.util.stream.Collectors.toSet;

import com.fasterxml.jackson.databind.JsonNode;
import com.google.common.collect.ImmutableMap;
import io.airbyte.commons.json.Jsons;
Expand All @@ -12,9 +14,12 @@
import io.airbyte.integrations.base.IntegrationRunner;
import io.airbyte.integrations.base.Source;
import io.airbyte.integrations.source.jdbc.AbstractJdbcSource;
import io.airbyte.integrations.source.jdbc.dto.JdbcPrivilegeDto;
import io.airbyte.integrations.source.relationaldb.TableInfo;
import io.airbyte.protocol.models.CommonField;
import java.sql.JDBCType;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
Expand Down Expand Up @@ -93,6 +98,25 @@ public Set<String> getExcludedInternalNameSpaces() {
return Set.of("information_schema", "pg_catalog", "pg_internal", "catalog_history");
}

@Override
public Set<JdbcPrivilegeDto> getPrivilegesTableForCurrentUser(final JdbcDatabase database, final String schema) throws SQLException {
return database.query(connection -> {
final PreparedStatement ps = connection.prepareStatement(
"SELECT DISTINCT table_catalog, table_schema, table_name, privilege_type\n"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The values in column table_catalog and privilege_type are not used. So I think they can be removed from the SELECT clause.

+ "FROM information_schema.table_privileges\n"
+ "WHERE grantee = ? AND privilege_type = 'SELECT'");
ps.setString(1, database.getDatabaseConfig().get("username").asText());
return ps;
}, sourceOperations::rowToJson)
.collect(toSet())
.stream()
.map(e -> JdbcPrivilegeDto.builder()
.schemaName(e.get("table_schema").asText())
.tableName(e.get("table_name").asText())
.build())
.collect(toSet());
}

public static void main(final String[] args) throws Exception {
final Source source = new RedshiftSource();
LOGGER.info("starting source: {}", RedshiftSource.class);
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/redshift.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ All Redshift connections are encrypted using SSL

| Version | Date | Pull Request | Subject |
| :------ | :-------- | :----- | :------ |
| 0.3.8 | 2022-01-26 | [9744](https://github.com/airbytehq/airbyte/pull/9721) | Implement getPrivilegesTableForCurrentUser |
| 0.3.7 | 2022-01-26 | [9721](https://github.com/airbytehq/airbyte/pull/9721) | Added schema selection |
| 0.3.6 | 2022-01-20 | [8617](https://github.com/airbytehq/airbyte/pull/8617) | Update connector fields title/description |
| 0.3.5 | 2021-12-24 | [8958](https://github.com/airbytehq/airbyte/pull/8958) | Add support for JdbcType.ARRAY |
Expand Down