Skip to content

Add support to write_to_textfile for custom tmpdir #1115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

aadityadhruv
Copy link

While the try/except block does prevent most of the temp files from persisting, if there is a non catchable exception, those temp files continue to pollute the directory. Optionally set the temp directory would let us write to something like /tmp, so the target directory isn't polluted.

This does mean that the target and source filesystems could be different, hence I've used shutil here. I didn't want to modify the original behavior, specifically for the Windows bit, so I've kept it as is. shutil will use os.rename if the filesystems are the same, so we don't lose that atomic behavior in that case.

@aadityadhruv aadityadhruv force-pushed the master branch 2 times, most recently from aad9be9 to 4f72683 Compare June 26, 2025 05:22
While the try/except block does prevent most of the temp files from
persisting, if there is a non catchable exception, those temp files
continue to pollute the directory. Optionally set the temp
directory would let us write to something like /tmp, so the target
directory isn't polluted

Signed-off-by: Aaditya Dhruv <[email protected]>
@aadityadhruv
Copy link
Author

Tagging @csmarchbanks as per CONTRIBUTING.md guidelines.

Copy link
Member

@csmarchbanks csmarchbanks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

I think I would prefer to just not support different filesystems. The atomic operation is important for the expected usage and I would prefer to fail than work around that.

Would you also add a bit to the help text describing what the tmpdir is there for and that it needs to be on the same filesystem?

@aadityadhruv
Copy link
Author

Thanks for the reply. The reason I've ended up adding support for the different filesystems is because /tmp, which is a pretty common location for temp files, is pretty much always on a different filesystem on Linux (tmpfs). If we don't allow different filesystem support, most people wouldn't be able to use /tmp.

@csmarchbanks
Copy link
Member

That makes sense, but would also be quite the footgun. It would be pretty hard to debug failed or incorrect scrapes in that scenario unless they know about the atomic issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants