Use standard IPLD codec names across the CLI/HTTP API

[aschmahmann]

Update 11/26 (@schomatis):

For review: #8568.

Working in draft branch with working notes and annotated code.

First attempt at adding the multicodec option for block put in ipfs/interface-go-ipfs-core#80.

Need to confirm this issue's description is in sync with conclusions from thread here and other multicodec issues.

---

While the recent #7976 related changes were a big step forwards in terms of both using the standardized codec names from https://github.com/multiformats/multicodec/blob/master/table.csv and making better use of the codecs they've left a couple of our APIs as confusing.

ipfs dag put and ipfs dag get both use codec names correctly as pulled from the table above. However, ipfs block put and ipfs cid codecs both use the older set of names defined in https://github.com/ipfs/go-cid/blob/5640b0169f6284562b1b8636b10e32dd85ea0a8a/cid.go#L85-L112.

We should switch both of those commands to use the correct names.

Some recommendations for particulars about the changes:

• ipfs cid codecs should contain a --supported flag that lists which codecs are known to go-ipfs per Enumerate available encodings for dag get #8171 (review). This has some tradeoffs if different commands support different subsets of codecs, but for now this seems reasonable.
• ipfs block put --format should probably be kept and deprecated alongside a new ipfs block put --store-codec and if both flags are used the user should receive an error. This would allow us to not break existing users, but still move forward.
  • Unlike with ipfs dag put where there was a backwards compatible changes anyway it seems useful to not break users here since we shouldn't have to.

[lidel]

👍 + when this is done we should also do a quick grep over content at https://docs.ipfs.io and https://proto.school

[schomatis]

Looking into this.

[schomatis]

@aschmahmann Looking at the block put command: we don't set the CID prefix directly as dag put does but go through the github.com/ipfs/interface-go-ipfs-core abstraction (which I'm not sure what's its purpose nor see much documentation about).

Should the added --store-codec just bypass all this and construct the CID prefix locally as dag put does? (As you said this would go in parallel with the current --format option which would still be using interface-go-ipfs-core; the two being mutually exclusive.)

[aschmahmann]

@schomatis I suspect there are some changes to be made both in go-ipfs and in interface-go-ipfs-core. Generally interface-go-ipfs-core is meant as an abstraction that can be used without coupling too tightly to the underlying IPFS implementation (e.g. https://github.com/ipfs/go-ipfs-http-client uses it as well).

Some things that are probably needed here:

• To support ipfs cid codecs we need a list of all the IPLD codecs. This list can either be manually generated (as it is in go-cid), or a likely better approach is to modify https://github.com/multiformats/go-multicodec to allow retrieving codes by type (e.g. the IPLD type). WDYT @mvdan ?
  • This could then be leveraged inside of go-cid to generate the list of CIDs on init by depending on go-multicodec
• To support ipfs cid codecs --supported we can for now leverage the ipld-prime global codec registry https://github.com/ipld/go-ipld-prime/blob/b9a89e847312334af141788b303fd78b4f076fe3/multicodec/defaultRegistry.go#L23
• Special support around handling the old name types is needed (particular attention to be paid to the names v0, protobuf, cbor). Supporting these names in interface-go-ipfs-core by having two options there (format and store-codec) is ok, although it's also fair game to make interface-go-ipfs-core and go-ipfs-http-client make a breaking change to only take the new format and push the legacy handling to the commands in go-ipfs itself since changing types in Go is an easy compile time error, whereas changing bash scripts or HTTP requests only give run time errors.

[mvdan]

go-multicodec currently does not have a way to enumerate all known codecs, but we could certainly look into that. I've raised multiformats/go-multicodec#58

It's worth noting that the ipld tag in the multicodec table is broad. It includes CID versions as well as IPLD codecs - some of which are supported by go-ipld-prime out of the box, and some which require external implementations.

[aschmahmann]

It's worth noting that the ipld tag in the multicodec table is broad. It includes CID versions as well as IPLD codecs

Thanks for the flag here. I wonder if it's worth adding any more differentiating information into the type? Probably not a big deal since we can just manually ignore the CID codes for now, but using ipld-codec doesn't seem too wacky.

some of which are supported by go-ipld-prime out of the box, and some which require external implementations.

Yep, this is fine. I'm thinking there's some value in figuring out what's a valid IPLD codec separate from what's available. This is what go-cid currently does https://github.com/ipfs/go-cid/blob/28f4a5eab6c28351e4c0c3769a258dd779531ac7/cid.go#L85. Separately we'll want to list what codecs are available, but those can be pulled from the go-ipld-prime registry (whether global or a local one).

[mvdan]

Thanks for the flag here. I wonder if it's worth adding any more differentiating information into the type? Probably not a big deal since we can just manually ignore the CID codes for now, but using ipld-codec doesn't seem too wacky.

I assume you mean altering the multicodec table in some form. I don't think we should teach go-multicodec about extra information that isn't on the table, because then we'd start getting into the Go implementation having out-of-spec information that other languages might not have or might differ on.