feat: add get_lineage support for neptune backend

[Owen-CH-Leung]

Summary of Changes

Currently, using AWS Neptune as the backend, the get_lienage function is not implemented at the NeptuneGremlinProxy, AbstractGremlinProxy and BaseProxy, which means that the lineage graph & the frontend tab displaying lineage does not work properly due to the missing implementation. For details, you can refer to this chat :

https://amundsenworkspace.slack.com/archives/C01A87A5EUU/p1653483057967299?thread_ts=1653374819.089879&cid=C01A87A5EUU

Hence, this PR attempts to create such functionality so that lineage works fine using Neptune as backend.

Tests

I tested the code using my company's aws environment (which uses AWS neptune & opensearch as the backend) and it worked fine.

Documentation

N/A

CheckList

Make sure you have checked all steps below to ensure a timely review.

• PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
• PR includes a summary of changes.
• PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

[boring-cyborg]

Congratulations on your first Pull Request and welcome to Amundsen community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/amundsen-io/amundsen/blob/main/CONTRIBUTING.md)

[feng-tao]

could you add a test for the feature? and thanks for the contribution

[feng-tao]

the CI fails

[Owen-CH-Leung]

the CI fails

Just pushed another commit which fixed the failed CI on my local environment. It should pass this time

[Owen-CH-Leung]

could you add a test for the feature? and thanks for the contribution

Sure. Let me take a look at the existing test cases and see how I can modify them

[Owen-CH-Leung]

could you add a test for the feature? and thanks for the contribution

@feng-tao Would you mind sharing with me how I can contribute to adding more test cases for the gremlin proxy that I've modified ? I've looked at this file which is for neo4j proxy but I couldn't find a similar one for neptune / gremlin proxy. The closest file I could find is this but it didn't look similar to that of neo4j...

Thanks a lot for your help.

[feng-tao]

@Owen-LCH I think we could add a new test file, but we could do it in a different pr. The ci still fails with mypy error:

============ 268 passed, 42 skipped, 8 warnings in 89.47s (0:01:29) ============
python3 -m flake8
mypy --ignore-missing-imports --follow-imports=skip --strict-optional --warn-no-return .
metadata_service/proxy/gremlin_proxy.py:1816: error: Need type annotation for 'downstream_tables' (hint: "downstream_tables: List[<type>] = ...")
metadata_service/proxy/gremlin_proxy.py:1817: error: Need type annotation for 'upstream_tables' (hint: "upstream_tables: List[<type>] = ...")
Found 2 errors in 1 file (checked 96 source files)
setup.cfg: [mypy]: follow_imports: invalid choice 'True' (choose from 'normal', 'silent', 'skip', 'error')
make: *** [mypy] Error 1
Makefile:20: recipe for target 'mypy' failed
Error: Process completed with exit code 2.

[Owen-CH-Leung]

python3 -m flake8

@feng-tao Sure. Let's do it in a seperate PR. I just pushed a new commit which fixes the mypy error & the import sequence error that is returned from isort ./ --check --diff. Running make test should pass now

[Owen-CH-Leung]

@feng-tao Can I seek your help to start the unit test again ? Thanks

[feng-tao]

just restart the ci

[Owen-LCH]

Thanks. I saw the CI tests are all passed. We can merge it to main now =)

[boring-cyborg]

Awesome work, congrats on your first merged pull request!