Skip to content

feat: added additional fields config to publisher #1898

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Jun 23, 2022
Merged

feat: added additional fields config to publisher #1898

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
382d7fb
feat: added optional published_by publishing metadata to nodes
allisonsuarez Jun 10, 2022
900c147
bumped to 6.9.0
allisonsuarez Jun 10, 2022
1bf7fba
set default value for publish by to None
allisonsuarez Jun 10, 2022
104f9ec
added quote around published by value
allisonsuarez Jun 22, 2022
9c4acd8
Merge branch 'main' of github.com:amundsen-io/amundsen into asm-publi…
allisonsuarez Jun 22, 2022
cdf60e4
made publisher accept a dict of fields through config
allisonsuarez Jun 22, 2022
0bfb5d5
autopep
allisonsuarez Jun 22, 2022
01543cb
lint
allisonsuarez Jun 22, 2022
7dff867
r in wrong place
allisonsuarez Jun 23, 2022
3e24a81
morel inting
allisonsuarez Jun 23, 2022
d4a0b24
rerun checks
allisonsuarez Jun 23, 2022
ccc8f92
flake
allisonsuarez Jun 23, 2022
9639d43
isort
allisonsuarez Jun 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions databuilder/databuilder/publisher/neo4j_csv_publisher.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,9 @@
# This will be used to provide unique tag to the node and relationship
JOB_PUBLISH_TAG = 'job_publish_tag'

# any additional fields that should be added to nodes and rels through config
ADDITIONAL_FIELDS = 'additional_fields'

# Neo4j property name for published tag
PUBLISHED_TAG_PROPERTY_NAME = 'published_tag'

Expand Down Expand Up @@ -103,6 +106,7 @@
NEO4J_MAX_CONN_LIFE_TIME_SEC: 50,
NEO4J_ENCRYPTED: True,
NEO4J_VALIDATE_SSL: False,
ADDITIONAL_FIELDS: {},
ADD_PUBLISHER_METADATA: True,
RELATION_PREPROCESSOR: NoopRelationPreprocessor()})

Expand Down Expand Up @@ -156,6 +160,7 @@ def init(self, conf: ConfigTree) -> None:
self.deadlock_node_labels = set(conf.get_list(NEO4J_DEADLOCK_NODE_LABELS, default=[]))
self.labels: Set[str] = set()
self.publish_tag: str = conf.get_string(JOB_PUBLISH_TAG)
self.additional_fields: Dict = conf.get(ADDITIONAL_FIELDS)
self.add_publisher_metadata: bool = conf.get_bool(ADD_PUBLISHER_METADATA)
if self.add_publisher_metadata and not self.publish_tag:
raise Exception(f'{JOB_PUBLISH_TAG} should not be empty')
Expand Down Expand Up @@ -414,6 +419,11 @@ def _create_props_body(self,
props.append(f"{identifier}.{PUBLISHED_TAG_PROPERTY_NAME} = '{self.publish_tag}'")
props.append(f"{identifier}.{LAST_UPDATED_EPOCH_MS} = timestamp()")

# add additional metatada fields from config
for k, v in self.additional_fields.items():
val = v if isinstance(v, int) or isinstance(v, float) else f"'{v}'"
props.append(f"{identifier}.{k}= {val}")

return ', '.join(props)

def _execute_statement(self,
Expand Down
2 changes: 1 addition & 1 deletion databuilder/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

from setuptools import find_packages, setup

__version__ = '6.8.0'
__version__ = '6.9.0'

requirements_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'requirements.txt')
with open(requirements_path) as requirements_file:
Expand Down