Skip to content

Add TimeMilli and TimeMicro fields and conversions for the record API #7544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

njaremko
Copy link
Contributor

@njaremko njaremko commented May 23, 2025

Which issue does this PR close?

Rationale for this change

I recently wrote a PR for #7510 that was wrong. So this is rectifying that.

What changes are included in this PR?

I had used TimestampMillis for a time value that does not have a date, which is wrong.

I've added two new Fields for the missing concept of "time without date", and fixed the conversion to use them, and added the required trait implementations

Are there any user-facing changes?

There are new Field variants, so if people are matching Field, this'll cause a compile error when they upgrade

@github-actions github-actions bot added the parquet Changes to the parquet crate label May 23, 2025
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @njaremko -- this makes sense to me

Sorry for missing this in the first round of review

@@ -602,6 +602,12 @@ pub enum Field {
/// Date without a time of day, stores the number of days from the
/// Unix epoch, 1 January 1970.
Date(i32),

/// The total number of milliseconds since midnight.
TimeMillis(i32),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding these new variants means this is an API change and thus must wait for our next breaking release (in July)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fine, I expected as much.

What's the process here, do we just let this sit around till July? There doesn't seem to be a branch for stable (55) or the next major release (56)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we'll just leave the PR around until main opens for 56. That will probably be June sometime.

We could also potentially make a new branch to merge all the queued changes as well, but we haven't done that historically

@alamb alamb added next-major-release the PR has API changes and it waiting on the next major version api-change Changes to the arrow API labels May 26, 2025
@alamb alamb changed the title Implement proper time fields and conversions for the record API Add TimeMilli and TimeMicro fields and conversions for the record API May 26, 2025
@njaremko
Copy link
Contributor Author

Thanks @njaremko -- this makes sense to me

Sorry for missing this in the first round of review

No worries

Copy link
Contributor

@etseidl etseidl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Since we're breaking the API anyway, is there any interest in adding logical types that were introduced more recently? Nanosecond time and timestamp for instance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api-change Changes to the arrow API next-major-release the PR has API changes and it waiting on the next major version parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Introduce Time fields in record API, and resolve bug from #7510
3 participants