Skip to content

feat: Add configurable prop types to neo4j csv publisher #1993

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

kristenarmes
Copy link
Contributor

Signed-off-by: Kristen Armes [email protected]

Summary of Changes

Added publisher config NEO4J_PROP_TYPES_TO_CONFIGURE to Neo4jCsvUnwindPublisher. If used, this should be set to a dict using node or relation property names as keys mapped to the function name used to configure a specific type for that property. The values of the properties should be in the correct format that the function accepts.

For example, a config of {'start_time': 'datetime', 'publish_tag': 'date'} where the property values are in the format <date>T<time> and <date> would apply datetime(n.start_time) and date(n.publish_tag) in the prop merge statement to create the props as DateTime and Date types instead of strings.

Tests

N/A

Documentation

N/A

CheckList

Make sure you have checked all steps below to ensure a timely review.

  • PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • PR includes a summary of changes.
  • PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does

@boring-cyborg boring-cyborg bot added the area:databuilder From databuilder folder label Sep 19, 2022
@kristenarmes kristenarmes marked this pull request as ready for review September 19, 2022 18:31
Copy link
Contributor

@allisonsuarez allisonsuarez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the logic here makes perfect sense to me, thanks Kristen!

@@ -317,23 +319,38 @@ def _create_props_body(self,
template = Template("""
{% for k in record_keys %}
{% if preserve_empty_props %}
{{ identifier }}.{{ k }} = row.{{ k }}
{% if k in prop_types_to_configure %}
{{ identifier }}.{{ k }} = {{ prop_types_to_configure[k] }}(row.{{ k }})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you were to convert something to another type like string you would have to do toString instead of just string right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's correct, whichever cypher functions are available would need to be used as the associated value in the dict (toString being one option)

@kristenarmes kristenarmes merged commit 261134a into amundsen-io:main Sep 19, 2022
@kristenarmes kristenarmes deleted the add-neo4j-configurable-prop-types branch September 19, 2022 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:databuilder From databuilder folder
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants