Make ipfs.dag.export built-in feature of HTTP gateways

[lidel]

tldr this is attempt to improve how do we want to represent non-unixfs DAGs on the HTTP gateway under /ipfs/ and /ipns/ paths by doing CAR export when data is not a raw/unixfs

Today: only unixfs+raw

Right now the /ipfs/ path on gateway only supports dag-pb+raw, everything else fails with unrecognized object type error:

Gateway exposes /api/v0/get so one can read blobs with other DAGs that way, but people rarely use it.

Future: export anything

I believe the HTTP gateways should return every DAG type.

Here is initial idea: we recently added support for CAR import/export to go-ipfs (ipfs/kubo#6870) – what if we return non-unixfs/raw DAGs as .car .ipfs.dag files?

• CAR format makes it easy for people to import/export DAGs
• gateway would not need to know how to render specific codec to be useful for data distribution
• gateway could return CAR with proper cache control header, namely the immutable hint for everything under /ipfs/
• we would set proper content-disposition header to ensure {cid}.ipfs.dag filename and trigger download, so the browser does not try to render the blob
• we could support ?download=dag everywhere, so even unixfs DAG could be fetched as CAR from any gateway
  • update: ?format=car is more flexible, as it allows for format=tar or json for dag-cbor (feat: serve CBOR encoded DAG nodes from the gateway kubo#8037)

cc @mikeal @aschmahmann @autonome @Gozala @achingbrain – is this a good idea? any concerns? would PR be accepted?

Future: import DAGs

If we have export.. could we also add import? This could be safely enabled on localhost, and people could experiment with this on public gateways (there, it could be guarded by reverse proxy or some bearer token):

• we could improve the concept of a writable gateway to support DAG import via HTTP PUT /ipfs/{cid}
• IPNS publishing could be as easy as HTTP PUT /ipns/{libp2p-key}

References

• https://docs.ipfs.io/reference/cli/#ipfs-dag-export / https://docs.ipfs.io/reference/cli/#ipfs-dag-import
• DAG Import/Export : DAG Import/Export kubo#6870
  • .car support pieces left out of 0.5
• import/export a DAG from/to a CAR file: feat: import/export a DAG from/to a CAR file js-ipfs#2745
• ?download=true support was recently added in https://github.com/ipfs/go-ipfs/pull/7677
• DAR format (streamable + optional index for random access) https://github.com/anorth/go-dar
• Example of Content-Location in response to specific Accept in request: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Location#examples

[Gozala]

I like general idea of allowing to get / put data without dedicated http client, that this would enable.

My only (weak) concern would be that this could lead to some unexpected behavior if hitting any non unixfs path would start .car download. It might be better to make this more explicit e.g. via ?download=car query parameter, or dare I say Accept header ?

Another argument for explicitness would be that if in the future we add support for another "file" like codec that would not require a breaking change.

[lidel]

@Gozala good point. I think for non-unixfs DAGs we would return an error by default informing that there is no available preview for CID, but the original DAG can be downloaded if ?download=car or Accept are passed.

Q: any thoughts regarding Content-Type returned with CAR payload?
We could return application/octet-stream to indicate binary payload, will work fine provisionally, but if we want to support Accept we need something specific to CAR that we could add to https://www.iana.org/assignments/media-types/media-types.xhtml at some point in the future.

@ribasushi

• does CAR format have mime|media|content-type? If not, any thoughts how it should look like?
• are there any "magic bytes" that can be used by content-type sniffers to identify CAR archive based on its header?
  adding CAR content-type detection to libraries like file-type (js) and mimetype (go) would improve developer experience

[Gozala]

Q: any thoughts regarding Content-Type returned with CAR payload?
We could return application/octet-stream to indicate binary payload, will work fine provisionally, but if we want to support Accept we need something specific to CAR that we could add to https://www.iana.org/assignments/media-types/media-types.xhtml at some point in the future.

I think using custom mime types over application/octet-stream is more useful as it provides some info beyond it just being binary data. From browser standpoint primary diff between octet-stream and unknown mime types handling was that later is been sniffed falling back to former, while former is not.

[rvagg]

Comment I made on this elsewhere:

The varint at the front of the CAR format messes this up, so the first byte or two don't conform to a tight pattern.

We did come up with some ideas to get us out of this annoying mess and fix the header in place, but that's a dream for CARv2. For now we're stuck with this design where even the version field comes late in a DAG-CBOR block after a dynamically sized roots field.

This byte pattern is likely to catch most CAR files:

0x??a265726f6f7473

Assuming ?? will fit a varint describing the length of the header, which may break down for a very large CID or multiple CIDs (not supported by go-car though, but supported by JS). That string is basically this: "single byte varint + a CBOR encoded { roots: ...". Going beyond that makes more assumptions that start to break down.

A common case will be:

0x3aa265726f6f7473 as a prefix, which will be the case of a single CID with a 256 bit hash and a low multihash number like sha2-256.

but Filecoin uses blake2b-256 which has a higher multihash number so it makes its CIDs longer .. So they’re going to start with:`

0x3ca265726f6f7473 simply because Filecoin CIDs are 2 bytes longer… yay.

So, no “magic number” sadly, we’re lumped with this design for now. But you could approximate if you were keen.

[achingbrain]

Part of me would like to see the ipld explorer on the gateway, then a download link/querystring param for either file data or a car file.

That'd give a preview of any DAG type and discourage people from using it as a cheap CDN.

[lidel]

Somehow related: CAR export at gateways would provide means for Verifiable HTTP Gateway Responses (#128) without the need for exposing DAG metadata as custom HTTP headers.

Exporting full DAG may not always be the best performance-wise: in the future we may want to be are able to control the depth of CAR export to facilitate parallel streaming from multiple gateways and/or seeking within media files.

@rvagg are there prior issues/plans about controlling the depth of CAR export?

[ec1oud]

I would just like to have dag get -f cbor working ( ipfs/kubo#4313 )

dag export exports only car format, and dag get exports only json. Either way, the ipfs daemon (or the ipfs embedded implementation) is spending CPU time to convert from native cbor to something else, but cbor itself would be fine as an export format, I think.