Gateway support for dag-json and dag-cbor responses

[lidel]

TODO

Ability to request dag-json or dag-cbor response with:

• ?format=dag-json, Accept: application/vnd.ipld.dag-json
• ?format=dag-cbor, Accept: application/vnd.ipld.dag-cbor
• TDB: should requesting Accept: application/json / Accept: application/cbor will return IPLD Data Model representation (as DAG-JSON/CBOR) of any IPLD-compatible CID (could be dag-pb, or whatever) with Content-Type: application/json
  • This is probably the killer feature for making IPLD interoperable with Web stack (JS, JSON everywhere), and allowing for creating custom UI in HTML/JS/CSS for data that ships with data itself.

When we have this, we should also:

• reuse or supersede feat: serve CBOR encoded DAG nodes from the gateway #8037
• open IPIP with new response types
• register new media types with IANA
  • Done;
    • https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-json
    • https://www.iana.org/assignments/media-types/application/vnd.ipld.dag-cbor
• ensure Size column in generated HTML Dir Index has the same value as Tsize in dag-json – descoped, tracked in: gateway/dir-index-html: switch dir listing sizes to Tsize #9058

Why

See ipfs/in-web-browsers#182

In short, to unlock the power of IPLD by making IPFS data that follows IPLD Data Model available as dag-json and dag-cbor HTTP gateway response formats .

It is already possible on the CLI:

$ ipfs dag get --output-codec=dag-json  $DIR_CID
{"Data": ...

Gateway support for this was previously attempted in #8037 – we need to refactor/rewrite that PR to follow conventions introduced in #8234 and #8758

Ecosystem impact around UnixFS

DAG-PB has a Logical Format which makes it possible to represent dag-pb directory as dag-json document.

This means ?format=dag-json will provide a way for supporting JSON responses for directory listing, which was also requested by our users.

Q&A

• How should non-unixfs CIDs be rendered by gateway when no explicit format is requested by the client?
  • TBD, initial idea is to return 400 Bad Request error with HTML body that links to ?format=dag-json|dag-cbor|raw|car
    • key reason is to avoid implicit defaults, and train users and tools to always request specific response type where it matters
• Given that dag-json and dag-cbor are subsets of JSON and CBOR, should we return responses as application/vnd.ipld.dag-json or appliation/json ?
  • TBD. A good case for returning application/json response is to improve interop with existing tools that speak JSON

[Gozala]

initial idea is to return 400 Bad Request error with HTML body that links to ?format=dag-json|dag-cbor|raw|car
key reason is to avoid implicit defaults, and train users and tools to always request specific response type where it matters

I think this is a bad idea. It makes sense if user is a developer who you can train, but if user is someone just clicking you won’t train them, you’d just make their experience bad.

I would argue that:

1. Gateway should still render something useful e.g ipld explorer like view or at least some links that users can click. Alternatively it could just redirect to URL with forma query param attached
2. Status code still could be an error code if desired, yet body should probably provide useful content never the less. That way programs still going to interpret it as error while end users will still get to see a page they can interact with.

I think it is generally worth thinking who’s the gateway audience and who we want it to be. I think for devs we do have HTTP API, gateway should be for everyone and consequently less errors would mean better experience with IPFS