Skip to content

Feat: add dtntz date macro variant #4838

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 1, 2025
Merged

Conversation

georgesittas
Copy link
Contributor

Addresses the issue discussed in this Slack thread: https://tobiko-data.slack.com/archives/C044BRE5W4S/p1749739413928599.

The _dt date macro variables always inject a time zone in the resulting datetime object, even if it's UTC. This is problematic in certain cases, e.g., when these values are compared to a zone-unaware timestamps. For instance, Redshift isn't clever enough when comparing zone-aware and unaware timestamps, leading to full scans instead of pruning.

The only alternative that exists today to work around this issue is to have string versions of these date values and cast them, which isn't a great experience. This PR covers this feature gap.

@georgesittas georgesittas requested a review from a team June 27, 2025 12:29
Copy link
Collaborator

@erindru erindru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want to document this on the Redshift engine adapter page for people to consider if they're having trouble with _dt

@georgesittas georgesittas force-pushed the jo/add_dtntz_macro_variable branch from 354b6b7 to c086ce1 Compare July 1, 2025 08:03
@georgesittas georgesittas force-pushed the jo/add_dtntz_macro_variable branch from c086ce1 to 7333fe4 Compare July 1, 2025 08:04
@georgesittas georgesittas enabled auto-merge (squash) July 1, 2025 08:14
@georgesittas georgesittas merged commit 0060cab into main Jul 1, 2025
26 of 27 checks passed
@georgesittas georgesittas deleted the jo/add_dtntz_macro_variable branch July 1, 2025 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants