Skip to content

perf: New neo4j csv publisher to improve performance using batched params #1957

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Aug 16, 2022
Merged

perf: New neo4j csv publisher to improve performance using batched params #1957

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
98fbe60
New publisher using managed transactions, unwind statements, configur…
kristenarmes Aug 11, 2022
1bf9938
Add logic for preserving adhoc ui data, and move write transactions l…
kristenarmes Aug 11, 2022
02051bf
Fixing flake8 error (not sure how it got through, not from my change)
kristenarmes Aug 11, 2022
92d04f8
Pulling in a few of the latest changes from the original publisher an…
kristenarmes Aug 12, 2022
aed878c
Fix tests
kristenarmes Aug 12, 2022
2635bdc
Addressing PR feedback and change index creation to use a managed tra…
kristenarmes Aug 12, 2022
d74943e
Refactor props body function and separate out constants
kristenarmes Aug 15, 2022
7ea393b
Addressing PR feedback plus refactoring
kristenarmes Aug 15, 2022
8875336
Addressing PR feedback, more refactoring
kristenarmes Aug 16, 2022
ce017d1
Minor fixes and bump version
kristenarmes Aug 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions databuilder/databuilder/extractor/hive_table_last_updated_extractor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,8 +113,8 @@ class HiveTableLastUpdatedExtractor(Extractor):
AND NOT EXISTS (SELECT * FROM PARTITION_KEYS WHERE PARTITION_KEYS.TBL_ID = TBLS.TBL_ID)
"""

DEFAULT_POSTGRES_ADDTIONAL_WHERE_CLAUSE = """ NOT EXISTS (SELECT * FROM "PARTITIONS" p WHERE p."TBL_ID" = t."TBL_ID")
AND NOT EXISTS (SELECT * FROM "PARTITION_KEYS" pk WHERE pk."TBL_ID" = t."TBL_ID")
DEFAULT_POSTGRES_ADDTIONAL_WHERE_CLAUSE = """ NOT EXISTS (SELECT * FROM "PARTITIONS" p
WHERE p."TBL_ID" = t."TBL_ID") AND NOT EXISTS (SELECT * FROM "PARTITION_KEYS" pk WHERE pk."TBL_ID" = t."TBL_ID")
"""

DATABASE = 'hive'
Expand Down
Loading