Skip to content

fix: exclude manually added columns from copy #598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 10, 2025

Conversation

Askir
Copy link
Contributor

@Askir Askir commented Apr 1, 2025

No description provided.

@Askir Askir temporarily deployed to internal-contributors April 1, 2025 11:02 — with GitHub Actions Inactive
@Askir Askir force-pushed the jascha/allow-additional-columns branch from 5ce31c8 to 4bae8b1 Compare April 1, 2025 11:03
@Askir Askir temporarily deployed to internal-contributors April 1, 2025 11:03 — with GitHub Actions Inactive
@Askir Askir force-pushed the jascha/allow-additional-columns branch from 4bae8b1 to 7784037 Compare April 1, 2025 15:03
@Askir Askir temporarily deployed to internal-contributors April 1, 2025 15:03 — with GitHub Actions Inactive
@Askir Askir changed the title fix: exclude nullable columns from copy fix: exclude manually added columns from copy Apr 1, 2025
Base automatically changed from mat/repackage to main April 2, 2025 16:04
@Askir Askir force-pushed the jascha/allow-additional-columns branch from 7784037 to f247916 Compare April 3, 2025 09:28
@Askir Askir temporarily deployed to internal-contributors April 3, 2025 09:28 — with GitHub Actions Inactive
@Askir Askir force-pushed the jascha/allow-additional-columns branch from f247916 to 350f583 Compare April 3, 2025 10:01
@Askir Askir temporarily deployed to internal-contributors April 3, 2025 10:01 — with GitHub Actions Inactive
Copy link
Collaborator

@cevian cevian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should change this to look up types by column names, not just rely on the table prefix by attnum. That would be much more robust

@Askir Askir marked this pull request as ready for review April 3, 2025 14:56
@Askir Askir requested a review from a team as a code owner April 3, 2025 14:56
@jgpruitt
Copy link
Collaborator

jgpruitt commented Apr 3, 2025

I think we should change this to look up types by column names, not just rely on the table prefix by attnum. That would be much more robust

Agreed. But if we do this, we must construct the copy command to allow for the arbitrary ordering of columns too. Maybe it almost handles that already. I'm not sure how you could "inject" a new column into the middle without some major finnagling.

@Askir Askir temporarily deployed to internal-contributors April 9, 2025 19:26 — with GitHub Actions Inactive
@Askir Askir force-pushed the jascha/allow-additional-columns branch from b2f5e1d to 712a710 Compare April 9, 2025 19:33
@Askir Askir temporarily deployed to internal-contributors April 9, 2025 19:34 — with GitHub Actions Inactive
@Askir Askir force-pushed the jascha/allow-additional-columns branch from 712a710 to fa13340 Compare April 9, 2025 19:42
@Askir Askir temporarily deployed to internal-contributors April 9, 2025 19:42 — with GitHub Actions Inactive
@Askir Askir requested a review from cevian April 9, 2025 20:22
@Askir
Copy link
Contributor Author

Askir commented Apr 9, 2025

@cevian I updated the query. I don't really know how to write a test for this though, since you cant just add a column at any point in postgres, it's always added at the end.

self.vectorizer.target_schema,
self.vectorizer.target_table,
target_columns,
),
)
self.copy_types = [row[0] for row in await cursor.fetchall()]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is no guarantee right now of the order of the results. The query should return the column name, type and then we can finangle things into the right order

@Askir Askir temporarily deployed to internal-contributors April 10, 2025 20:46 — with GitHub Actions Inactive
@Askir Askir force-pushed the jascha/allow-additional-columns branch from 9366b7a to 312ebc4 Compare April 10, 2025 20:48
@Askir Askir temporarily deployed to internal-contributors April 10, 2025 20:48 — with GitHub Actions Inactive
@Askir Askir merged commit cc62dfe into main Apr 10, 2025
8 checks passed
@Askir Askir deleted the jascha/allow-additional-columns branch April 10, 2025 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants