Skip to content

Copy source before modifying it in GroupBy #972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

earangol-stripe
Copy link
Contributor

@earangol-stripe earangol-stripe commented Apr 16, 2025

Summary

Copies the sources passed to a GB before applying modifications (normalizing and sanitizing).

There is special handling for join sources because the reference to the Join must be maintained to set the object name correctly from GC referrers.

Why / Goal

To prevent unexpected modifications to sources when shared across Joins/GBs.

Sources may be reused across Joins and GBs. The sanitize code here modifies the source in-place, which could cause unintended behaviors for GBs that reuse the source and don't expect those modifications. One specific example:

  • group_by1 uses source1. One of its aggregations has input_col1 as the input column. If that column is not present in source1, _sanitize_columns will add it to the query selects.
  • group_by2 also uses source1. It does not have input_col1 in any of its aggregations. If group_by1 was loaded before group_by2, its source would include that column unexpectedly.

Test Plan

  • Added Unit Tests
  • Covered by existing CI
  • Integration tested

Checklist

  • Documentation update

Reviewers

@earangol-stripe earangol-stripe changed the title Copy source before modifying it in GroupBy. Copy source before modifying it in GroupBy Apr 16, 2025
@earangol-stripe earangol-stripe marked this pull request as ready for review April 16, 2025 15:26
Copy link
Collaborator

@pengyu-hou pengyu-hou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Could you also add a unit test?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants