Minimising disk usage via TorrentDataOpener

[jpillora]

Hey Matt, I'm wanting to stream individual torrent files either a cloud storage or directly to the user. This will result in varying degrees of outgoing bandwidth. The restriction is I want to use as little disk as possible. For example, say I have a 1.5GB ubuntu distribution that I want to stream to a user. Could this be done with say 50MB of disk?

1. I know I can grab a Reader from the torrent and just stream out the file, though wouldn't this consume the disk as the download progresses? Can I delete the section that's already been streamed? I imagine trimming off the front of a file would produce invalid offsets...
2. My next idea to save disk was to implement my own TorrentDataOpener to store the torrent's "working files" in the cloud. I'm even sure if this is possible because it writing a single piece, like 32KB, in the middle of a file might be difficult...
3. As I was writing this, I came up with a 3rd way. Using pure memory implementation of TorrentDataOpener which I can tightly control. For example, keep up-to 100MB of the torrent (how ever many pieces) and then free pieces as this goes over. I saw torrent/mmap though this seems to memory map over file/s.

I'm guessing 3 would be easiest, though what would you suggest?

[anacrolix]

Yes, this is supported. I have just this use case myself, and the client can redownload pieces that go missing. I use a LRU cache, and recently fixed a bug when the cache is smaller than a file being downloaded.

Storing data in S3 for example, will also work. However you may want to optimize reads from the storage to reuse open streams or add your own caching layer due to the high connect and open latency on S3. Writes to storage do not lock the client, but will reduce the throughout from peers if they're slow.

1. Should be fine. Has been done. Trimming is handled correctly.
2. See above. Also writes in middle of cloud files depends on your API.
3. Easiest solution.

[jpillora]

Awesome, thanks for the info! Will start with 3 and see how I go from there.