Skip to content

Implement PeerDAS #14129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 183 commits into
base: develop
Choose a base branch
from
Draft

Implement PeerDAS #14129

wants to merge 183 commits into from

Conversation

nalepae
Copy link
Contributor

@nalepae nalepae commented Jun 21, 2024

Reading:

Peerdas from scratch

Remaining tasks must have:

Features:

  • Backfill: Backfill data columns when checkpoint syncing and when validator custody increases (in progress @kasey).
  • Builder: Implement builder with cells proofs (in progress @terencechain).
  • Initial sync: Implement robust initial sync.
  • Validator custody: Implement the "validator custody up" only and update accordingly min_available_slot.

Misc:

  • Peers research: Avoid searching peers in parallel. (@nalepae - PR under review)
  • Grandine (not PeerDAS related): See why Prysm cannot peer with Grandine nodes.
  • Metrics: Add PeerDAS specific metrics (In progress @KatyaRyazantseva)
  • Peer downscoring: Uncomment / modify peer downscoring when a peer requests too many data columns.
  • Peer downscoring: Downscore properly peers when responding invalid data columns sidecars by RPC.
  • Spectests: Implement all.
  • GetBlobsV2: Try as soon as we have a block or a column, and retry periodically (every 200 ms?) until we have what we want, by any way possible. And also call it as soon as we receive the first data column sidecar or block.
  • Metadata: Check why in devnets we have Metadata is nil, defaulting to the ENR value.
  • Attestations subnets: (Not peerdas related): Check why we are subscribed to so many attestations subnets. (It may be normal.)

Remaining tasks nice to have have:

  • KZG library: Replace the cKZG library by the goKZG one. (In progress @nalepae @kevaundray)
  • Per root/range data columns request: Implement a rescue mode where trying, for full nodes, if no way to retrieve needed columns, to retrieve any of 64 columns then reconstruct the rest.
  • prysmctl: Implement byRoot and byRange data columns requests.
  • Chaos testing: Indroduce a nasty peer flag that holds AND/OR (both are important) alterate data columns on RPC responses. (In progress 0xPC02)
  • ENR (not peerDAS related) Find why sometimes ENR records are not correctly updated via discovery.
  • Trusted setup: Add a unit test to pull the upstream JSON and compare it against ours to ensure there’s no discrepancy.
  • Blob sidecars beacon API: If 64 <= cgc < 128 and requiring not all the blobs, avoid reconstructing all of them (while respecting the VerifiedRoDataColumnSidecar paradigm.
  • KZG: Avoid the power switch.
  • Verification: Implement tests in a different package.
  • Blocks: Implement tests in a different package.

Remaining open questions:

  • Subnet unsubscription (not peerDAS related): Handle the same way dynamic (fork epoch) and non dynamic (fork epoch + 1) unsubscriptions.
  • Peer ban: What happens if a peer announces less custody groups via Metadata/ENR than the minimal requirement? Ban?

nalepae and others added 11 commits November 27, 2024 10:11
* Add Support For Discovery Of Column Subnets

* Lint for SubnetsPerNode

* Manu's Review

* Change to a better name
* Add Data Column Subscriber

* Add Data Column Vaidator

* Wire all Handlers In

* Fix Build

* Fix Test

* Fix IP in Test

* Fix IP in Test
* Add RPC Handler

* Add Column Requests

* Update beacon-chain/db/filesystem/blob.go

Co-authored-by: Manu NALEPA <[email protected]>

* Update beacon-chain/p2p/rpc_topic_mappings.go

Co-authored-by: Manu NALEPA <[email protected]>

* Manu's Review

* Manu's Review

* Interface Fixes

* mock manager

---------

Co-authored-by: Manu NALEPA <[email protected]>
* Bump `c-kzg-4844` lib to the `das` branch.

* Implement `MerkleProofKZGCommitments`.

* Implement `das-core.md`.

* Use `peerdas.CustodyColumnSubnets` and `peerdas.CustodyColumns`.

* `CustodyColumnSubnets`: Include `i` in the for loop.

* Remove `computeSubscribedColumnSubnet`.

* Remove `peerdas.CustodyColumns` out of the for loop.
* Remove capital letter from error messages.

* `[4]byte` => `[fieldparams.VersionLength]byte`.

* Prometheus: Remove extra `committee`.

They are probably due to a bad copy/paste.

Note: The name of the probe itself is remaining,
to ensure backward compatibility.

* Implement Proposer RPC for data columns.

* Fix TestProposer_ProposeBlock_OK test.

* Remove default peerDAS activation.

* `validateDataColumn`: Workaround to return a `VerifiedRODataColumn`
* Add new DA check

* Exit early in the event no commitments exist.

* Gazelle

* Fix Mock Broadcaster

* Fix Test Setup

* Update beacon-chain/blockchain/process_block.go

Co-authored-by: Manu NALEPA <[email protected]>

* Manu's Review

* Fix Build

---------

Co-authored-by: Manu NALEPA <[email protected]>
* Update `consensus_spec_version` to `v1.5.0-alpha.1`.

* `CustodyColumns`: Fix and implement spec tests.

* Make deepsource happy.

* `^uint64(0)` => `math.MaxUint64`.

* Fix `TestLoadConfigFile` test.
nalepae added 4 commits May 28, 2025 21:05
* `ExchangeCapabilities`: Transform `O(n**2)` into `O(2n)` and fix logging.

* Find peers with subnets and logs: Refactor

* Validator custody: Do not wait being subscribed to advertise correct `cgc`. (temp hack)
nalepae and others added 25 commits June 6, 2025 12:46
* Add reconstruction metrics

* Fix time

* Fix format

* Fix format

* Update cells count function

* fix cells count

* Update reconstruction counter

* Fix peerDAS reconstruction counter metric

* Replace dataColumnSidecars with dataColumnSideCars
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked Blocked by research or external factors peer-das
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants