gateway: range requests issue with big >5GiB files

[lidel]

Many CDNs (incl. Cloudflare) won't support range requests of big files (>1-4GiB) if origin HTTP server does not return explicit support for range requests indicated by Accept-Ranges: bytes in GET responses.

This should apply to deserialized, deterministic response types.

TODO

• boxo/gateway fix (GET and investigate HEAD)
• include in Kubo 0.34
• explicit gateway-conformance tests: Add tests for HTTP Range requests gateway-conformance#154

[lidel]

Seems that something else is at play. I've checked multiple times and both GET and HEAD for 20GiB file include accept-ranges: bytes in response already.

Pausing until we get some clarifications from Cloudflare.

[lidel]

Closing as the bug turned out to be on the Cloudflare side.

Cloudflare, by default only supports caching of range requests for files <=5GiB.
Range requests for bigger files are ignored and full file is returned as HTTP 200 instead.

Enterprise clients are able to opt-in into increased "cache keys + origin range requests" feature, which at the time of me writing it, is NOT documented anywhere.

So if anyone is reading this in future:

• If you are Cloudflare client, reach out to their support, and reference this issue
• If you write HTTP Client, make sure to check response code before processing response body