feat: opt-in http retrieval client #10772

hsanjuan · 2025-04-01T20:31:30Z

This introduces the http-retrieval capability as an experimental feature.

It can be enabled in the configuration HTTPRetrieval.Enabled = true.

Documentation and changelog to be added later.

Part of Native HTTP across the IPFS Stack, to enable Filecoin direct retrieval — IPFS/2025 ipshipyard/roadmaps#9

hsanjuan · 2025-04-01T20:32:38Z

Is this going in the right direction?
What do we expect in terms of testing, if anything here?

lidel

Thank you @hsanjuan, yes, this looks right (small asks in the inline comments)

In terms of testing, yes, we need something, but it should be enough if we have a basic end-to-end regression test in test/cli that attempts to ipfs cat a CID that is provided only over HTTP.

Some pointers: it should be possible to mock (override) cid.contact responses by setting IPFS_HTTP_ROUTERS in env used by test to a local mock HTTP server that handles /routing/v1/providers/cid and returns static JSON with a single provider. The provider would have multiaddr pointing at the same mock server, handling /ipfs/cid path and returning the block bytes.

config/routing.go

cmd/ipfs/kubo/daemon.go

This introduces the http-retrieval capability as an experimental feature. It can be enabled in the configuration `Experimental.HTTPRetrieval.Enabled = true`. Documentation and changelog to be added later.

allows self-signed certificates in tests

makes is easier to track which specific rainbow build/release is responsible for requests. we need this, because Kubo and Rainbow may use the same boxo, but initialize client with slightly different config Ref. ipfs/kubo#10772

this spawns two http services on localhost: 1. HTTP router that returns HTTP provider when /routing/v1/providers/cid i queried 2. HTTP provider that returns a block when /ipfs/cid is queried 3. Configures Kubo to use (1) instead of cid.contact this seems to work (running test with DEBUG=true shows (1) was queried for the test CID and returned multiaddr of (2), but Kubo never requested test CID block from (2) – needs investigation

lidel · 2025-04-25T19:57:13Z

Pushed config improvements, and end-to-end test (a6db7af) suggested in #10772 (review).

E2E test in test/cli/http_retrieval_client_test.go spawns two HTTP services on localhost:

Mock HTTP provider (trustless block gateway) that returns a block when /ipfs/cid is queried
Mock delegated HTTP router that returns HTTP provider (1) when /routing/v1/providers/cid queried
Configures Kubo to use (1) instead of cid.contact

Version from a6db7af seems to partially work (running DEBUG=true go test -v test/cli/http_retrieval_client_test.go shows (2) was queried for the test CID and returned /tls/http multiaddr of (1), but Kubo never sent HTTP GET request for /ipfs/cid at (1).

Needs further investigation.

makes is easier to track which specific rainbow build/release is responsible for requests. we need this, because Kubo and Rainbow may use the same boxo, but initialize client with slightly different config Ref. ipfs/kubo#10772

we artificallly crippled every delegated routing endpoint because of cid.contact being limited to one endpoint

make it easy to override the hardcoded implicit HTTP routeur URL without having to set the entire custom Router.Routers and Router.Methods (http_retrieval_client_test.go still needs to be fixed in future commit)

lidel · 2025-04-30T23:36:45Z

Resolved the merge conflicts.

I have been busy with other required work for version 0.35 (#10760) and haven't had enough time to finish debugging why the E2E tests are failing, but did quick ad-hoc test and the E2E seems to fail with non-localhost HTTP/2 URL as well.

Writing down quick checkpoint so its easier to resume next week (or someone else can look, if spare time):

Running ipfs daemon with GOLOG_LOG_LEVEL="error,httpnet=debug,bitswap=debug,routing/http/client=debug,routing/delegated=debug" and custom mock IPFS_HTTP_ROUTERS="http://127.0.0.1:42045" and then calling ipfs cat bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq produces:

2025-05-01T01:33:54.685+0200	DEBUG	routing/delegated	routing/composer.go:69	composer: calling findProvidersAsync: bafkreibmom2y6fpldr5a5z3snkbdpivqjn7qjrekje63yifzbmb36tyl64
2025-05-01T01:33:54.705+0200	DEBUG	routing/delegated	routing/composer.go:69	composer: calling findProvidersAsync: bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq
2025-05-01T01:33:54.708+0200	DEBUG	routing/delegated	routing/composer.go:74	composer: calling findPeer: 12D3KooWCjfPiojcCUmv78Wd1NJzi4Mraj1moxigp7AfQVQvGLwH
2025-05-01T01:33:54.708+0200	DEBUG	routing/delegated	routing/composer.go:77	composer: calling findPeer error: 12D3KooWCjfPiojcCUmv78Wd1NJzi4Mraj1moxigp7AfQVQvGLwH{: []}Get "http://127.0.0.1:42045/routing/v1/peers/bafzaajaiaejcak26ug272wdirfytcn2py74wjbnf22mr5m5i2tsu2ebez6heebea?filter-protocols=transport-bitswap,transport-ipfs-gateway-http,unknown": context canceled

which is odd – delegated routing request seems to work, but no action is taken on the result (?) and ipfs cat hangs / timeouts – needs further investigation.

docs/config.md

this fixes two regressions: (1) introduced in #10717 where we only used bitswapLib2p query manager (this is why E2E did not act on http provider) (2) introduced in #10765 where it was not possible to set binary peerID in IgnoreProviders (we changed to []string)

replaces Bitswap.Enabled with Bitswap.Libp2pEnabled adds tests that confirm it is possible to disable libp2p bitswap fully and only keep http in client mode also, removes the need for passing empty blockstore in client-only mode

lidel · 2025-05-06T01:22:35Z

Turns out we had providerQueryManager not wired correctly after resolving conflicts with #10717.

Refactored in fac1314.

It is now possible to fully disable bitswap over libp2p (only have HTTP retrieval client) and remove /ipfs/bitswap* support from ipfs id – included E2E test for that as well.

Will take a look with fresh eyes tomorrow again, but likely good enough for RC1.

lidel

Lgtm.

Added docs and release notes.
renamed Bitswap.Enabled to Bitswap.Libp2pEnabled
E2E tests for HTTPRouting.Enabled and Bitswap.*

Merging, we will test this on staging and include in 0.35.0-rc1 (#10760)

hsanjuan self-assigned this Apr 1, 2025

hsanjuan requested a review from a team as a code owner April 1, 2025 20:31

lidel changed the title ~~Feat: http retrieval as experimental feature~~ feat: http retrieval as experimental feature Apr 3, 2025

lidel reviewed Apr 3, 2025

View reviewed changes

config/routing.go Outdated Show resolved Hide resolved

cmd/ipfs/kubo/daemon.go Outdated Show resolved Hide resolved

hsanjuan force-pushed the http-retr branch from b41994b to e0af7e7 Compare April 8, 2025 12:30

lidel mentioned this pull request Apr 8, 2025

Release 0.35 #10760

Closed

45 tasks

Feat: http retrieval as experimental feature

42d6057

This introduces the http-retrieval capability as an experimental feature. It can be enabled in the configuration `Experimental.HTTPRetrieval.Enabled = true`. Documentation and changelog to be added later.

hsanjuan force-pushed the http-retr branch from 04f8508 to 42d6057 Compare April 8, 2025 14:21

lidel mentioned this pull request Apr 23, 2025

Native HTTP across the IPFS Stack, to enable Filecoin direct retrieval — IPFS/2025 ipshipyard/roadmaps#9

Closed

lidel added 2 commits April 23, 2025 22:27

Merge 'origin/master' into http-retr

a379c66

refactor: HTTPRetrieval.Enabled as Flag

07b47fa

lidel force-pushed the http-retr branch from fcf5153 to 07b47fa Compare April 23, 2025 21:08

lidel added 4 commits April 24, 2025 00:42

docs(config): HTTPRetrieval section

e5f3dac

docs: changelog

5eeb443

refactor: reusable MockHTTPContentRouter

d910989

feat: HTTPRetrieval.TLSInsecureSkipVerify

0d1e1b3

allows self-signed certificates in tests

lidel mentioned this pull request Apr 25, 2025

fix: set User-Agent for HTTP retrieval client ipfs/rainbow#259

Merged

lidel added 2 commits April 25, 2025 19:59

feat(config): HTTPRetrieval.MaxBlockSize

0785f33

lidel force-pushed the http-retr branch from c0a0699 to a6db7af Compare April 25, 2025 19:55

lidel changed the title ~~feat: http retrieval as experimental feature~~ feat: opt-in http retrieval client Apr 25, 2025

lidel added 4 commits April 29, 2025 23:03

fix: /routing/v1/peers for non-cid.contact

08ecf77

we artificallly crippled every delegated routing endpoint because of cid.contact being limited to one endpoint

feat: Routing.DelegatedRouters

e96a15d

make it easy to override the hardcoded implicit HTTP routeur URL without having to set the entire custom Router.Routers and Router.Methods (http_retrieval_client_test.go still needs to be fixed in future commit)

test: flag remaining work

9df0e80

Merge 'origin/master' into http-retr

15cf0fa

lidel added the status/blocked Unable to be worked further until needs are met label Apr 30, 2025

2color reviewed May 5, 2025

View reviewed changes

docs/config.md Outdated Show resolved Hide resolved

lidel added 2 commits May 5, 2025 17:36

docs: review feedback

6e83aba

lidel mentioned this pull request May 5, 2025

Support WithIgnoreProviders() in provider query manager #10765

Merged

refactor: Bitswap.Libp2pEnabled

fac1314

replaces Bitswap.Enabled with Bitswap.Libp2pEnabled adds tests that confirm it is possible to disable libp2p bitswap fully and only keep http in client mode also, removes the need for passing empty blockstore in client-only mode

lidel force-pushed the http-retr branch from ba3d5bb to fac1314 Compare May 6, 2025 01:06

docs: changelog

895010a

lidel approved these changes May 6, 2025

View reviewed changes

lidel merged commit b5d7369 into master May 6, 2025
16 checks passed

lidel deleted the http-retr branch May 6, 2025 17:06

BrewTestBot mentioned this pull request May 21, 2025

ipfs 0.35.0 Homebrew/homebrew-core#224309

Merged

lidel mentioned this pull request Jun 12, 2025

feat: HTTP retrieval enabled by default #10836

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat: opt-in http retrieval client #10772

feat: opt-in http retrieval client #10772

Uh oh!

hsanjuan commented Apr 1, 2025 •

edited by lidel

Loading

Uh oh!

hsanjuan commented Apr 1, 2025

Uh oh!

lidel left a comment

Uh oh!

Uh oh!

Uh oh!

lidel commented Apr 25, 2025 •

edited

Loading

Uh oh!

lidel commented Apr 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

lidel commented May 6, 2025 •

edited

Loading

Uh oh!

lidel left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat: opt-in http retrieval client #10772

feat: opt-in http retrieval client #10772

Uh oh!

Conversation

hsanjuan commented Apr 1, 2025 • edited by lidel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hsanjuan commented Apr 1, 2025

Uh oh!

lidel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lidel commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lidel commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

lidel commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lidel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hsanjuan commented Apr 1, 2025 •

edited by lidel

Loading

lidel commented Apr 25, 2025 •

edited

Loading

lidel commented Apr 30, 2025 •

edited

Loading

lidel commented May 6, 2025 •

edited

Loading

lidel left a comment •

edited

Loading