[Enhancement] Support analyze iceberg table with partition transform #39907
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Youngwb <[email protected]>
![starrocks-cr[bot]](https://avatars.githubusercontent.com/in/404334?s=60&v=4){kind=small}
fe/fe-core/src/main/java/com/starrocks/connector/iceberg/IcebergPartitionUtils.java
Show resolved
Hide resolved
| return null; | ||
| } | ||
|
|
||
|
|
||
| public long nextPartitionId() { | ||
| return partitionIdGen.getAndIncrement(); | ||
| } |
There was a problem hiding this comment.
The most risky bug in this code is:
Potential null pointer access in the call to getNativeTable().spec() if getNativeTable() returns null.
You can modify the code like this:
public PartitionField getPartitionField(String partitionColumnName) {
NativeTable nativeTable = getNativeTable();
if (nativeTable == null) {
return null; // or throw an exception depending on expected behavior
}
List<PartitionField> allPartitionFields = nativeTable.spec().fields();
Schema schema = nativeTable.schema();
for (PartitionField field : allPartitionFields) {
if (getPartitionSourceName(schema, field).equalsIgnoreCase(partitionColumnName)) {
return field;
}
}
return null;
}
fe/fe-core/src/main/java/com/starrocks/statistic/ExternalFullStatisticsCollectJob.java
Show resolved
Hide resolved
Signed-off-by: Youngwb <[email protected]>
Signed-off-by: Youngwb <[email protected]>
Signed-off-by: Youngwb <[email protected]>
|
[FE Incremental Coverage Report]✅ pass : 45 / 53 (84.91%) file detail
|
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
stephen-shelby
approved these changes
Jan 30, 2024
HangyuanLiu
approved these changes
Jan 30, 2024
packy92
approved these changes
Jan 30, 2024
|
@Mergifyio backport branch-3.2 |
✅ Backports have been created
|
mergify bot
pushed a commit
that referenced
this pull request
Jan 30, 2024
…39907) Signed-off-by: Youngwb <[email protected]> (cherry picked from commit 1dc523c)
Merged
18 tasks
wanpengfei-git
pushed a commit
that referenced
this pull request
Jan 30, 2024
…(backport #39907) (#40347) Co-authored-by: Youngwb <[email protected]>
liubotao
pushed a commit
to liubotao/starrocks
that referenced
this pull request
Jan 31, 2024
…tarRocks#39907) Signed-off-by: Youngwb <[email protected]> Signed-off-by: liubotao <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
Iceberg table with partition transform would not work normally
What I'm doing:
iceberg has partition transform like year/month/day/hour, so we need to map partition column to range with year/month/day/hour, eg. ts is partition column and type is date with year transform,
origin predicate is ts = '2022',
but ts with year partition transform needs predicate
ts >= '2022-01-01' and ts < '2023-01-01';
Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: