feat(iroh): publish and resolve user-defined data in discovery #3176

Frando · 2025-02-10T12:56:04Z

Description

Based on #3175

Adds support for publishing and resolving a string of user-defined data through Iroh's discovery services.

This gives applications a way to add additional information to the records published and resolved through discovery, in addition to the addressing information (relay URL and direct addresses). Iroh itself does not parse or use this user-defined data string in any way.

All existing discovery services are updated to support the user data. All DNS-based discovery services encode the user-defined data in a TXT record, in the same way as the DNS/pkarr services already encode the relay URL and direct addresses. Therefore, the length of the user-data=<user-data> string is limited to 255 bytes, which is the max length for a single character string in DNS messages. Thus, the length of the actual user-defined data string is limited to 245 bytes (255 minux user-data=). This limit is enforced when constructing a UserData (which is just a wrapper around a String otherwise).

The local swarm discovery uses swarm-discovery under the hood, for which I added support for TXT records in rkuhn/swarm-discovery#12. This was released as [email protected].

Breaking Changes

Notes & open questions

Note that currently the DiscoveryItem which contains the user-defined data is only accessible when directly calling the methods on the Discovery trait. For discoveries initiated by the endpoint, both active from Endpoint::connect and passive from the magicsock's subscription to Discovery::subscribe, the DiscoveryItems are currently not exposed in any way, thus there's no way for an app to access the user data of nodes discovered by the endpoint. However, with #3181, there will be a way to watch for all DiscoveryItems found by the endpoint.

Change checklist

Self-review.
Documentation updates following the style guide, if relevant.
Tests if relevant.
All breaking changes documented.

github-actions · 2025-02-10T13:14:14Z

Netsim report & logs for this PR have been generated and is available at: LOGS
This report will remain available for 3 days.

Last updated for commit: 2507c94

github-actions · 2025-02-10T14:01:20Z

Documentation for this PR has been generated and is available at: https://n0-computer.github.io/iroh/pr/3176/docs/iroh/

Last updated: 2025-02-21T15:33:17Z

ramfox

This is looking good to me! some nits around comments and potential API changes.

ramfox · 2025-02-12T22:27:17Z

iroh-relay/src/dns/node_info.rs

+    }
+}
+
+// User defined data that can be published and resolved through node discovery.


Suggested change

// User defined data that can be published and resolved through node discovery.

/// User-defined data that can be published and resolved through node discovery.

iroh/src/discovery/static_provider.rs

iroh-relay/src/dns/node_info.rs

ramfox

looks good! @rklaehn might be a good second set of eyes for opinions on this and the next PR #3181.

## Description Adds a `NodeData` struct that contains the information about a node that can be published in discovery services. This struct is both used in `iroh_relay::dns::NodeInfo` and in `iroh::discovery::Discovery`. Also, `iroh::discovery::DiscoveryItem` has no public fields anymore and uses `NodeInfo` internally to again reduce duplication. With these changes, the fields published about a node in discovery now are only defined in a single place. This PR also serves these prupsoes: * We want to add some "user defined data" to discovery (see #3176). This would add a third argument to `Discovery::publish`; with this PR instead we can just add it as an optional field to `NodeData`. * It makes it possible to potentially add data to discovery after 1.0. The fields of `NodeData` are private, so we can add fields, and setter/getter methods, without this being a semver-breaking change. A few other places are changed for consistency: `StaticProvider` now works with `NodeInfo` instead of `NodeAddr` (`NodeInfo` can be easily converted into `NodeAddr`). `DnsProvider::lookup_node_by_id` etc. now return `NodeInfo` instead of `NodeAddr`. A few method names are adjusted for consistency. `NodeInfo` is constructed in a builder-style fashion instead of passing all fields to `new`. ## Breaking Changes * `iroh::discovery::Discovery::publish` now takes `data: &NodeData` as its single argument. `iroh::discovery::NodeData` is a re-export of `iroh_relay::dns::node_info::NodeData`, and contains relay URL and direct addresses. See docs for `NodeData` for details. * `iroh::discovery::DiscoveryItem` no longer has any public fields. There are now getters for the contained data, and constructors for createing a `DiscoveryItem` from a `NodeInfo`. * `iroh_relay::dns::node_info::NodeInfo` is changed. * `NodeInfo::new` now has a single `NodeId` argument. Use `NodeInfo::with_direct_addresses` and `NodeInfo::with_relay_url` to set addresses and relay URL. Alternatively, use `NodeInfo::from_parts` and pass a `NodeData` struct. * `NodeInfo` now has two public fields `node_id: NodeId` and `data: NodeData`, and setter and getter methods for the relay URL and direct addresses. * ` iroh::discovery::pkarr::PkarrPublisher::update_addr_info` now takes a `NodeData` argument ## Notes & open questions  ## Change checklist - [x] Self-review. - [x] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [x] Tests if relevant. - [x] All breaking changes documented.

dignifiedquire · 2025-02-21T13:33:44Z

iroh/src/discovery/local_swarm_discovery.rs

@@ -63,6 +63,8 @@ const N0_LOCAL_SWARM: &str = "iroh.local.swarm";
 /// Used in the [`crate::endpoint::Source::Discovery`] enum variant as the `name`.
 pub const NAME: &str = "local.swarm.discovery";

+const USER_DATA_ATTRIBUTE: &str = "user-data";


comment for what this is used?

dignifiedquire · 2025-02-21T13:34:37Z

iroh/src/discovery/local_swarm_discovery.rs

+    // Get the user-defined data from the resolved peer info. We expect an attribute with a value
+    // that parses as `UserData`. Otherwise, omit.
+    let user_data = if let Some(Some(user_data)) = peer.txt_attribute(USER_DATA_ATTRIBUTE) {
+        user_data.parse().ok()


I think we should print at least a debug message of the errror here, so that it isn't fully silent if there is invalid data floating around, and you are trying to debug why it can't get resolved

Frando force-pushed the feat/discovery-user-data branch from 5e49912 to 9432910 Compare February 10, 2025 13:01

Frando force-pushed the feat/discovery-user-data branch 5 times, most recently from 32a29f4 to 14b1f3b Compare February 11, 2025 11:18

Frando mentioned this pull request Feb 11, 2025

refactor!: streamline discovery and node info types #3175

Merged

4 tasks

Frando force-pushed the feat/discovery-user-data branch from 14b1f3b to 0324657 Compare February 11, 2025 13:04

Frando changed the base branch from feat/discovery-data to main February 11, 2025 13:10

Frando changed the base branch from main to feat/discovery-data February 11, 2025 13:10

Frando force-pushed the feat/discovery-user-data branch 4 times, most recently from a2cd248 to 644e762 Compare February 12, 2025 13:43

Frando changed the title ~~feat(iroh): support publishing and resolving a string of user-defined data in discovery~~ feat(iroh): publish and resolve user-defined data in discovery Feb 12, 2025

ramfox reviewed Feb 12, 2025

View reviewed changes

Frando force-pushed the feat/discovery-user-data branch 7 times, most recently from 9970bb3 to 66f9691 Compare February 13, 2025 23:17

Frando force-pushed the feat/discovery-data branch from 761ffe7 to 181a4be Compare February 13, 2025 23:18

Frando force-pushed the feat/discovery-user-data branch from 66f9691 to 1514e11 Compare February 14, 2025 10:37

Frando marked this pull request as ready for review February 14, 2025 10:40

Frando force-pushed the feat/discovery-user-data branch from 1514e11 to c1627cb Compare February 14, 2025 10:45

ramfox approved these changes Feb 17, 2025

View reviewed changes

feat: add user data to discovery

51319ec

Frando force-pushed the feat/discovery-user-data branch from c1627cb to 51319ec Compare February 18, 2025 22:13

Frando changed the base branch from feat/discovery-data to main February 18, 2025 22:40

docs: improve!

0603872

dignifiedquire reviewed Feb 21, 2025

View reviewed changes

dignifiedquire approved these changes Feb 21, 2025

View reviewed changes

address review

1fdf075

Frando enabled auto-merge February 21, 2025 13:50

Frando added this pull request to the merge queue Feb 21, 2025

github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch Feb 21, 2025

dignifiedquire added this to the v0.33.0 milestone Feb 21, 2025

matheus23 added 3 commits February 21, 2025 16:28

Merge branch 'main' into feat/discovery-user-data

d2984f7

Fix double import & format

952c099

Add missing import

6b21d23

matheus23 added this pull request to the merge queue Feb 21, 2025

Merged via the queue into main with commit ac78cf2 Feb 21, 2025
26 checks passed

matheus23 deleted the feat/discovery-user-data branch February 24, 2025 10:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(iroh): publish and resolve user-defined data in discovery #3176

feat(iroh): publish and resolve user-defined data in discovery #3176

Uh oh!

Frando commented Feb 10, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Feb 10, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Feb 10, 2025 •

edited

Loading

Uh oh!

ramfox left a comment

Uh oh!

ramfox Feb 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ramfox left a comment •

edited

Loading

Uh oh!

dignifiedquire Feb 21, 2025

Uh oh!

dignifiedquire Feb 21, 2025

Uh oh!

Uh oh!

Uh oh!

	// User defined data that can be published and resolved through node discovery.
	/// User-defined data that can be published and resolved through node discovery.

feat(iroh): publish and resolve user-defined data in discovery #3176

feat(iroh): publish and resolve user-defined data in discovery #3176

Uh oh!

Conversation

Frando commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Breaking Changes

Notes & open questions

Change checklist

Uh oh!

github-actions bot commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ramfox left a comment

Choose a reason for hiding this comment

Uh oh!

ramfox Feb 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ramfox left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dignifiedquire Feb 21, 2025

Choose a reason for hiding this comment

Uh oh!

dignifiedquire Feb 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Frando commented Feb 10, 2025 •

edited

Loading

github-actions bot commented Feb 10, 2025 •

edited

Loading

github-actions bot commented Feb 10, 2025 •

edited

Loading

ramfox left a comment •

edited

Loading