How to best provide multiple CIDs after merkleization

[2color]

Assuming you merkelize some files with Helia and UnixFS as follows:

const helia = await createHelia()
const fs = unixfs(helia)

// Glob source will recursively add all files and directories in the path
const source = globSource(path, '**/*')

const toProvide = []
let last
// add all files and directories in the path and wrap it all in a directory
for await (const entry of fs.addAll(source, { wrapWithDirectory: true })) {
  last = entry.cid
  if (entry.unixfs?.type !== 'raw') {
    toProvide.push(entry.cid)
  }
}

And now you have the toProvide array, what's the best way to announce all of those cids to the kad-dht concurrently?

A naive implementation would look as follows:

const promises = []
for (const cid of toProvide) {
  promises.push(helia.routing.provide(cid, {
    signal: AbortSignal.timeout(120_000)
  }))
}
await Promise.allSettled(promises)

However, as the number of CIDs grows this can cause network spikes. Moreover, each of the helia.routing.provide(cid) calls instantiates a concurrent queue with Alpha concurrency parameter (α). The spec isn't clear whether that concurrency parameter is global or per operation.

So I guess the question here is: should one use a concurrent queue like @libp2p/utils/queue when calling multiple provides?

[achingbrain]

Using a queue would give you better concurrency control, yes.

The spec isn't clear whether that concurrency parameter is global or per operation.

It's defined in the Kademlia paper and is per query.

what's the best way to announce all of those cids

Announcing lots of CIDs is expensive and has a knock-on effect on the reprovide queue - do you need to provide all the intermediate files and directories or will just the root do?

[2color]

Announcing lots of CIDs is expensive and has a knock-on effect on the reprovide queue - do you need to provide all the intermediate files and directories or will just the root do?

No doubt, which is why I narrowed it to UnixFS entries, skipping raw blocks. Obviously this is application/use-case specific, so it's hard to make generalisations.

Kubo announces all CIDs by default, but that's also the cause of problems with reprovides, especially if you consider that if you're dealing with larger files (compared to the block size), most provides will be for raw blocks.

[achingbrain]

Yeah, I don't think providing every CID for every block is a great idea, better to let the user decide what to provide and when though obviously this shifts a lot of the mental load onto the user.