Skip to content

Implement SFTPSensorAsync #654

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Sep 28, 2022
Merged

Implement SFTPSensorAsync #654

merged 24 commits into from
Sep 28, 2022

Conversation

tseruga
Copy link
Contributor

@tseruga tseruga commented Sep 20, 2022

Addresses #629

Implements a basic, functional asynchronous SFTPSensor using asyncssh.

First time contributor here, so open to feedback on style and technical decisions around how these operators are generally formatted/created.

closes #629

@tseruga
Copy link
Contributor Author

tseruga commented Sep 20, 2022

Will address failures

@bharanidharan14
Copy link
Contributor

Hi @tseruga. Thanks for contributing to this project and creating this PR. We will definitely take a look at this PR and share the feedback.

@tseruga
Copy link
Contributor Author

tseruga commented Sep 21, 2022

Unsure of how to address the remaining build failures:
build_docs is complaining about my usage of * in my doc-strings. I annotated the line so the static checker locally doesn't complain about it, but I'm not sure if there's another annotation I should use for build-docs to accept it. I can also remove it if it's just easier to link to the fnmatch docs instead of having the info in the doc-string itself.

mypy is complaining about type hints due to class inheritance. The parent SFTP sensor class expects Optional[SFTPHook] but I'm overriding the hook with the new custom async hook SFTPHookAsync, which is a child of SFTPHook.

There also numerous errors from mypy which aren't due to my changes. Not sure what to do about those

@bharanidharan14
Copy link
Contributor

For the build docs failure, you need to add the .rst file for sftp sensor in docs/providers (follow dbt or http sensors)
and you need to make an entry in docs/providers/index.rst as well

@bharanidharan14
Copy link
Contributor

@tseruga For docs failure,
You need to add sftp in docs/providers/index.rst and you need to create folder sftp as I mentioned in the earlier comment

For mypy failure, which is not related to your changes , that is covered in one of the PR once it is merged it will cover

@tseruga
Copy link
Contributor Author

tseruga commented Sep 23, 2022

Im clearly missing something obvious with sphinx docs. I've added an entry to the providers/index.rst but it still seems to be building the docs incorrectly for sftp. I've looked for an inconsistencies compared to other providers, but can't find it. I'm sure it's something small

@pankajastro
Copy link
Contributor

pankajastro commented Sep 23, 2022

Im clearly missing something obvious with sphinx docs. I've added an entry to the providers/index.rst but it still seems to be building the docs incorrectly for sftp. I've looked for an inconsistencies compared to other providers, but can't find it. I'm sure it's something small

Docs build should be working now (c1db648). mypy will be fixed by #659

@pankajastro
Copy link
Contributor

pankajastro commented Sep 23, 2022

Any thoughts on how we will run this in our integration tests? I think we may take this in separate PR, I have created an issue to track it #662 @phanikumv @rajaths010494 @bharanidharan14

@pankajastro
Copy link
Contributor

pankajastro commented Sep 27, 2022

Please update/rebase your branch.

@codecov
Copy link

codecov bot commented Sep 27, 2022

Codecov Report

Base: 98.36% // Head: 98.38% // Increases project coverage by +0.02% 🎉

Coverage data is based on head (69de543) compared to base (d0055cb).
Patch coverage: 99.13% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #654      +/-   ##
==========================================
+ Coverage   98.36%   98.38%   +0.02%     
==========================================
  Files          81       84       +3     
  Lines        4223     4339     +116     
==========================================
+ Hits         4154     4269     +115     
- Misses         69       70       +1     
Impacted Files Coverage Δ
astronomer/providers/sftp/hooks/sftp.py 98.33% <98.33%> (ø)
astronomer/providers/sftp/sensors/sftp.py 100.00% <100.00%> (ø)
astronomer/providers/sftp/triggers/sftp.py 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@tseruga
Copy link
Contributor Author

tseruga commented Sep 27, 2022

Ill look through the missed lines and add some tests

@tseruga
Copy link
Contributor Author

tseruga commented Sep 27, 2022

Surprised 97.4% coverage still fails the check 😆 I can cover the missing lines

@pankajastro
Copy link
Contributor

Thanks @tseruga for addressing the comments. Looking great 🚀

@bharanidharan14 bharanidharan14 self-requested a review September 28, 2022 06:54
@pankajastro pankajastro merged commit deb315a into astronomer:main Sep 28, 2022
@pankajastro
Copy link
Contributor

This is awesome, thanks @tseruga 👏 for creating the PR. Appreciate it.

@rajaths010494
Copy link
Contributor

Awesome work @tseruga , congrats on your first merged pull request!

@tseruga
Copy link
Contributor Author

tseruga commented Sep 28, 2022

Thanks @pankajastro @rajaths010494 @bharanidharan14 !

Hopefully will be contributing more in the future!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SFTP Async Sensor
4 participants