[Enhancement]Enhance datetime format #39986

leoyy0316 · 2024-01-25T07:35:32Z

Why I'm doing:

when iceberg table partition filed is timestamp and partitions value second is 0 , starrocks will throw [date literal [2024-01-24T10:10] is invalid] exception.Because starrocks DATETIME_FORMATTERS not support format [%Y-%m-%e %H:%i].

What I'm doing:

Enhance datetime format

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

starrocks-cr · 2024-01-25T07:38:50Z

fe/fe-core/src/main/java/com/starrocks/common/util/DateUtils.java

            int withMs = str.contains(".") ? 1 : 0;
            int withSplitT = str.contains("T") ? 1 : 0;
-            DateTimeFormatter formatter = DATETIME_FORMATTERS[isTwoDigit][withMs][withSplitT];
+            DateTimeFormatter formatter = DATETIME_FORMATTERS[isTwoDigit][withMs][withSplitT][withSec];
            return parseStringWithDefaultHSM(str, formatter);
        } else {
            // date


The most risky bug in this code is:
Missing DateTimeFormatter initializations for when milliseconds are not included (withMs is 0) but seconds are included (withSec is 1).

You can modify the code like this:

@@ -91,19 +91,23 @@ public class DateUtils { ... + // Initialization for cases when withMs is 0 and withSec is 1 + DATETIME_FORMATTERS[0][0][0][1] = unixDatetimeStrictFormatter("%Y-%m-%e %H:%i:%s", false); + DATETIME_FORMATTERS[0][0][1][1] = unixDatetimeStrictFormatter("%Y-%m-%eT%H:%i:%s", false); + DATETIME_FORMATTERS[1][0][0][1] = unixDatetimeStrictFormatter("%y-%m-%e %H:%i:%s", false); + DATETIME_FORMATTERS[1][0][1][1] = unixDatetimeStrictFormatter("%y-%m-%eT%H:%i:%s", false);

The final initializations should look like this:

DATETIME_FORMATTERS[0][0][0][0] = unixDatetimeStrictFormatter("%Y-%m-%e %H:%i", false); DATETIME_FORMATTERS[0][0][0][1] = unixDatetimeStrictFormatter("%Y-%m-%e %H:%i:%s", false); // ... continued original initialization for other options... DATETIME_FORMATTERS[0][1][1][1] = unixDatetimeStrictFormatter("%Y-%m-%eT%H:%i:%s.%f", false); // New initializations for missing formatters DATETIME_FORMATTERS[0][0][1][1] = unixDatetimeStrictFormatter("%Y-%m-%eT%H:%i:%s", false); DATETIME_FORMATTERS[1][0][0][1] = unixDatetimeStrictFormatter("%y-%m-%e %H:%i:%s", false); DATETIME_FORMATTERS[1][0][1][1] = unixDatetimeStrictFormatter("%y-%m-%eT%H:%i:%s", false);

Note that each new formatter I've added assumes %s must be formatted strictly (which you will want to verify).

stephen-shelby · 2024-01-26T16:26:45Z

it's best to add sql test about iceberg

packy92 · 2024-01-29T05:47:43Z

fe/fe-core/src/main/java/com/starrocks/common/util/DateUtils.java

+        DATETIME_FORMATTERS[1][0][1][0] = unixDatetimeStrictFormatter("%y-%m-%eT%H:%i", false);
+        DATETIME_FORMATTERS[1][0][1][1] = unixDatetimeStrictFormatter("%y-%m-%eT%H:%i:%s", false);
+        DATETIME_FORMATTERS[1][1][0][1] = unixDatetimeStrictFormatter("%y-%m-%e %H:%i:%s.%f", false);
+        DATETIME_FORMATTERS[1][1][1][1] = unixDatetimeStrictFormatter("%y-%m-%eT%H:%i:%s.%f", false);
    }

    public static String formatDateTimeUnix(LocalDateTime dateTime) {


Could you add some unit tests for this method?

Hi @packy92 i add ut for DateUtils.parseStrictDateTime(s) method. This PR not affect formatDateTimeUnix method.

Signed-off-by: leoyy0316 <[email protected]>

sonarqubecloud · 2024-01-30T06:31:51Z

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

4 New issues
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-01-30T07:42:59Z

[FE Incremental Coverage Report]

✅ pass : 15 / 15 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/common/util/DateUtils.java	15	15	100.00%	[]

github-actions · 2024-01-30T07:43:28Z

[BE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

stephen-shelby · 2024-01-31T15:35:28Z

@Mergifyio backport branch-3.2

mergify · 2024-01-31T15:35:43Z

backport branch-3.2

✅ Backports have been created

#40473 [Enhancement]Enhance datetime format (backport #39986) has been created for branch branch-3.2

stephen-shelby · 2024-01-31T15:35:49Z

@Mergifyio backport branch-3.1

stephen-shelby · 2024-01-31T15:36:00Z

https://github.com/Mergifyio backport branch-3.0

stephen-shelby · 2024-01-31T15:36:14Z

@Mergifyio backport branch-2.5

Signed-off-by: leoyy0316 <[email protected]> (cherry picked from commit 20e682c)

mergify · 2024-01-31T15:36:32Z

backport branch-3.1

✅ Backports have been created

#40474 [Enhancement]Enhance datetime format (backport #39986) has been created for branch branch-3.1

mergify · 2024-01-31T15:36:33Z

backport branch-3.0

✅ Backports have been created

#40475 [Enhancement]Enhance datetime format (backport #39986) has been created for branch branch-3.0

mergify · 2024-01-31T15:36:35Z

backport branch-2.5

✅ Backports have been created

#40476 [Enhancement]Enhance datetime format (backport #39986) has been created for branch branch-2.5

Signed-off-by: leoyy0316 <[email protected]> (cherry picked from commit 20e682c)

Co-authored-by: leoyy0316 <[email protected]>

Signed-off-by: leoyy0316 <[email protected]>

Co-authored-by: leoyy0316 <[email protected]>

mergify bot assigned leoyy0316 Jan 25, 2024

starrocks-cr bot reviewed Jan 25, 2024

View reviewed changes

packy92 reviewed Jan 29, 2024

View reviewed changes

leoyy0316 added 3 commits January 30, 2024 13:59

Enhance datetime format

8ed0b0f

Signed-off-by: leoyy0316 <[email protected]>

Enhance datetime format

2c765d6

Signed-off-by: leoyy0316 <[email protected]>

Enhance datetime format

030b24d

Signed-off-by: leoyy0316 <[email protected]>

leoyy0316 requested a review from packy92 January 30, 2024 07:44

packy92 approved these changes Jan 30, 2024

View reviewed changes

stephen-shelby approved these changes Jan 30, 2024

View reviewed changes

stephen-shelby merged commit 20e682c into StarRocks:main Jan 30, 2024

github-actions bot added 2.5 3.2 3.1 3.0 labels Jan 30, 2024

leoyy0316 deleted the leo.cheng branch January 30, 2024 10:44

mergify bot pushed a commit that referenced this pull request Jan 31, 2024

[Enhancement]Enhance datetime format (#39986)

d48d003

Signed-off-by: leoyy0316 <[email protected]> (cherry picked from commit 20e682c)

mergify bot mentioned this pull request Jan 31, 2024

[Enhancement]Enhance datetime format (backport #39986) #40473

Merged

18 tasks

mergify bot pushed a commit that referenced this pull request Jan 31, 2024

[Enhancement]Enhance datetime format (#39986)

5a65c3d

Signed-off-by: leoyy0316 <[email protected]> (cherry picked from commit 20e682c)

mergify bot pushed a commit that referenced this pull request Jan 31, 2024

[Enhancement]Enhance datetime format (#39986)

390058f

Signed-off-by: leoyy0316 <[email protected]> (cherry picked from commit 20e682c)

mergify bot pushed a commit that referenced this pull request Jan 31, 2024

[Enhancement]Enhance datetime format (#39986)

06c372b

Signed-off-by: leoyy0316 <[email protected]> (cherry picked from commit 20e682c)

This was referenced Jan 31, 2024

[Enhancement]Enhance datetime format (backport #39986) #40474

Merged

[Enhancement]Enhance datetime format (backport #39986) #40475

Closed

[Enhancement]Enhance datetime format (backport #39986) #40476

Closed

wanpengfei-git pushed a commit that referenced this pull request Jan 31, 2024

[Enhancement]Enhance datetime format (backport #39986) (#40473)

c52fadb

Co-authored-by: leoyy0316 <[email protected]>

github-actions bot added 3.2-merged and removed 2.5 labels Jan 31, 2024

murphyatwork pushed a commit that referenced this pull request Feb 4, 2024

[Enhancement]Enhance datetime format (backport #39986) (#40503)

3726b13

Signed-off-by: leoyy0316 <[email protected]>

murphyatwork pushed a commit that referenced this pull request Feb 4, 2024

[Enhancement]Enhance datetime format (backport #39986) (#40505)

c812fe0

Signed-off-by: leoyy0316 <[email protected]>

github-actions bot added 3.0-merged 2.5-merged labels Feb 4, 2024

wanpengfei-git pushed a commit that referenced this pull request Feb 5, 2024

[Enhancement]Enhance datetime format (backport #39986) (#40474)

bed776c

Co-authored-by: leoyy0316 <[email protected]>

github-actions bot added the 3.1-merged label Feb 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Enhancement]Enhance datetime format #39986

[Enhancement]Enhance datetime format #39986

Uh oh!

leoyy0316 commented Jan 25, 2024 •

edited by packy92

Loading

Uh oh!

starrocks-cr bot Jan 25, 2024

Uh oh!

stephen-shelby commented Jan 26, 2024

Uh oh!

packy92 Jan 29, 2024

Uh oh!

leoyy0316 Jan 29, 2024

Uh oh!

sonarqubecloud bot commented Jan 30, 2024

Uh oh!

github-actions bot commented Jan 30, 2024

Uh oh!

github-actions bot commented Jan 30, 2024

Uh oh!

stephen-shelby commented Jan 31, 2024

Uh oh!

mergify bot commented Jan 31, 2024 •

edited

Loading

Uh oh!

stephen-shelby commented Jan 31, 2024

Uh oh!

stephen-shelby commented Jan 31, 2024

Uh oh!

stephen-shelby commented Jan 31, 2024

Uh oh!

mergify bot commented Jan 31, 2024 •

edited

Loading

Uh oh!

mergify bot commented Jan 31, 2024 •

edited

Loading

Uh oh!

mergify bot commented Jan 31, 2024 •

edited

Loading

Uh oh!

Uh oh!

[Enhancement]Enhance datetime format #39986

[Enhancement]Enhance datetime format #39986

Uh oh!

Conversation

leoyy0316 commented Jan 25, 2024 • edited by packy92 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

starrocks-cr bot Jan 25, 2024

Choose a reason for hiding this comment

Uh oh!

stephen-shelby commented Jan 26, 2024

Uh oh!

packy92 Jan 29, 2024

Choose a reason for hiding this comment

Uh oh!

leoyy0316 Jan 29, 2024

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Jan 30, 2024

Quality Gate passed

Uh oh!

github-actions bot commented Jan 30, 2024

[FE Incremental Coverage Report]

file detail

Uh oh!

github-actions bot commented Jan 30, 2024

[BE Incremental Coverage Report]

Uh oh!

stephen-shelby commented Jan 31, 2024

Uh oh!

mergify bot commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

stephen-shelby commented Jan 31, 2024

Uh oh!

stephen-shelby commented Jan 31, 2024

Uh oh!

stephen-shelby commented Jan 31, 2024

Uh oh!

mergify bot commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

mergify bot commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

mergify bot commented Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Uh oh!

leoyy0316 commented Jan 25, 2024 •

edited by packy92

Loading

mergify bot commented Jan 31, 2024 •

edited

Loading

mergify bot commented Jan 31, 2024 •

edited

Loading

mergify bot commented Jan 31, 2024 •

edited

Loading

mergify bot commented Jan 31, 2024 •

edited

Loading