Gateway support for /ipfs/{cid}?format=car|raw|...

[lidel]

@mikeal @olizilla @autonome @warpfork @aschmahmann – this is a quick memo with my initial synthesis of the ?format= idea. Bit thin on details, but want to get early feedback/temperature check before I start elaborating on this in gateway specs in the following weeks.

This is a meta issue for streamlining various feature requests and needs under a single opt-in query parameter that enables gateway users to fetch a specific representation of a specific content path.

Support for each format can be discussed/added via a separate issue/PR – this issue is just for tracking the bigger picture around unified format paramerer.

Note: if you need CARs from an ipfs gateway today, POST to /v0/api/dag/export?arg=<cid>, see: https://docs.ipfs.io/reference/http/api/#api-v0-dag-export

MVP formats

Ability to fetch every CID as full DAG in CAR or a single Block

This is the key feature to enable Verifiable Gateway Responses and "HTTP-based transport for IPFS" (mobile browsers, IoT) without introducing even more dependency on /api/v0, and giving us flexibility for adding new features in the future.

• ?format=car – implemented in feat(gateway): Block and CAR response formats #8758
  • Returns binary stream with CAR archive for entire DAG behind the content path
  • Supersedes /api/v0/dag/export, but with better UX:
    • works on DNSLink websites that do not expose /api/v0
    • content-disposition defaults to {filename|cid}.car
• ?format=block ?format=raw – implemented in feat(gateway): Block and CAR response formats #8758
  • Returns binary array with the root block identified by CID
  • Supersedes /api/v0/block/get, but with better UX:
    • works on DNSLink websites that do not expose /api/v0
    • content-disposition defaults to {cid}.bin

CBOR / JSON

Moved to #8823

Future ideas / lower priorities

• ?format=tar|zip
  • add tarfile/zipfile downloads to HTTP gateway #7746
  • incorrect content-type for tarfile from /api/v0/get #7959
• ?format=share-img
  • Gateway: dynamic image for social sharing #8811
• ?format=unixfs-stats
  • similar to share-img but in dag-json format, could provide information about Size and Type (dir/file) and be leveraged for efficient pagination of huge directories (Gateway directory listings should be paginated #8455, Get some info about UnixFS objects on public IPFS HTTP API #8528)

Behaviors

• ?format missing
  • if codec is dag-pb or raw return file/directory
    (current gateway behavior)
  • else (codec without default behavior), return error suggesting passing ?format=car|block|..

[mikeal]

I like this querystring approach a lot. You can easily imagine extending this with new parameters for partial dag queries and the like.

One thing I’d like to specify is how an implementation exposes what formats it does and does not support. Then clients can implement fallback logic in order to be more robust and servers aren’t required to implement every feature ever.

[olizilla]

It looks promising! Does it matter that it'd be mixing codecs ?format=dag-json and containers ?format=car. How do we deal with verisoning, would CARv2 be ?format=carv2? It's not called out in the issue, but is the plan to also honour an Accept header if provided.

else (codec without default behavior), return error suggesting passing ?format=car|block|..

can a dag-json be returned as json without needing to specify the format? I know this has come up before, but I can't recall why that would be bad.

and can has mime/types ipld/specs#368

[ribasushi]

but I can't recall why that would be bad.

@olizilla relevant 🧵
#8037 (comment)

[lidel]

• On default behavior
  • car and block would work for every CID, other things like json or cbor will be available per-codec-basis
  • When requested CID is not available in specified format (or does not have a default one), a human-readable error message suggesting appending explicit ?format=dag-json|dag-dbor|block|car is returned.
  • To facilitate things like automated fallback or format discovery via HEAD request, the list of supported formats could be included as HTTP header such as Link from RFC6249: Metalink/HTTP: Mirrors and Hashes  in-web-browsers#179:

    Link: <ipfs://bafy?format=block>; rel=describedby; type="application/octet-stream"
    Link: <ipfs://bafy?format=car>; rel=describedby; type="application/octet-stream"    
    Link: <ipfs://bafy?format=dag-json>; rel=describedby; type="application/json"
    Link: <ipfs://bafy?format=dag-cbor>; rel=describedby; type="application/cbor"
    

  • The concern about default behavior was about binary dag-cbor (thread linked above by @ribasushi).
    • For dag-cbor the default state will be error, as there is no obvious default response, and we may want to render some GUI on gateways in the future.
    • Returning application/json for dag-json by default (without explicit ?format) is ok.
• On mixing codecs and containers: my reasoning for a single ?format= is that both codecs and containers translate to distinct response formats. HTTP client does not care about IPFS-specific taxonomy, it requests specific thing in specific format (Accept or ?format= ) and gets it.
  • Sidenote: I used explicit dag-cbor and dag-json just to highlight that CIDs will be traversable thanks to IPLD conventions, but we may shorten this to json and cbor to improve UX.
• On CAR versions: AFAIK we have built-in versioning in CARs: CARv2 will includes version in the header in a backward-compatible way, so CARv1 parser will return "unsupported version" for CARv2. Due to this I see no need for versioning here.