File-like objects are closed after uploading

[JordonPhillips]

Source boto/boto3#929

This is because we wrap them with the InterruptReader which closes them on exit.

[kyleknap]

Yeah looks like it will only happen for the non-multipart upload case because the provided file object is used. Will probably have to figure out a way to safeguard against accidentally closing the file.

[ramurti]

Any workaround to prevent the file being closed? I'm using this function with unnamed temporary files, so I can't simply reopen the file. I'd like to do something else with this file (for example, rotate it to local disk) in case upload_fileobj fails, but it closes the file.... (sigh)

[micahfay]

The workaround by leonsmith here worked for me- matthewwithanm/django-imagekit#391

[ofassley]

Same issue here. I'm also using Imagekit.

self = <ImageCacheFile: CACHE/avatars/a0cbd7767bf54e649f4952e8c3407418/9dd51a8dd80342ae7576d1c376b51e43.jpg>

    def _generate(self):
        # Generate the file
        content = generate(self.generator)

        actual_name = self.storage.save(self.name, content)

        # We're going to reuse the generated file, so we need to reset the pointer.
>       content.seek(0)
E       ValueError: I/O operation on closed file.

actual_name = 'CACHE/avatars/a0cbd7767bf54e649f4952e8c3407418/9dd51a8dd80342ae7576d1c376b51e43.jpg'
content    = <File: None>
self       = <ImageCacheFile: CACHE/avatars/a0cbd7767bf54e649f4952e8c3407418/9dd51a8dd80342ae7576d1c376b51e43.jpg>

../../../.virtualenvs/myapp/lib/python3.5/site-packages/imagekit/cachefiles/__init__.py:102: ValueError

[mgrdcm]

I can't tell - is this a bug? I'll be happy to take a stab at a PR if it is, and if nobody else wants to handle it. I'm also suffering from the django-imagekit issue.

[micahfay]

I've stopped using django-storages and moved to this for django-s3 stuff- lighter weight, and seems to be better supported. Also, it doesn't require that workaround :) https://github.com/etianen/django-s3-storage switching was easy even with existing data.

[joshma]

@kyleknap when you get a chance, could you chime in on whether calling self._fileobj.close() in InterruptReader is necessary? We're running into the same issue, and it's particularly difficult because we're accepting a few kinds of file-like objects so we can't just re-open it later.

[freemanlevinetang]

Ran into this issue recently while attempting to generate images, upload them to s3 for backup, and then attach them to an email.
Workaround was to pull the file back down from s3 after uploading, which seems pretty inefficient.

[leogzyl]

I'm using the following workaround:

from io import BufferedReader

class NonCloseableBufferedReader(BufferedReader):
    def close(self):
        self.flush()

buffer = NonCloseableBufferedReader(myfile)
client.upload_fileobj(Fileobj=buffer, Bucket=bucket, Key=name)
buffer.detach()
assert not myfile.closed

[zachliu]

@mgrdcm I think it is a bug. The control should be relinquished to the end user instead of closing it prematurely.

[zachliu]

@leogzyl A near perfect workaround except mypy complains if I pass in tempfile.TemporaryFile() object:

error: Argument 1 to "NonCloseableBufferedReader" has incompatible type "IO[Any]"; expected "RawIOBase"

Any solutions other than # type: ignore?

[ncvc]

@zachliu I added an init method with a cast to work around the mypy issue:

class NonCloseableBufferedReader(BufferedReader):
    def __init__(self, raw: IO[bytes], *args: Any, **kwargs: Any):
        super().__init__(cast(RawIOBase, raw), *args, **kwargs)

    def close(self) -> None:
        self.flush()

NonCloseableBufferedReader(myfile)  # No mypy errors

[al-muammar]

3 years passed... Can it be either fixed or declared to be not a bug, so other libraries should build workarounds for it?