Add PeerDAS metrics

[KatyaRyazantseva]

This PR introduces a subset of the recommended metrics for monitoring PeerDAS - data column, kzg and custody metrics. The proposed changes attempt to standardize the naming convention for these metrics across client implementations. By adopting these metrics, clients can provide better PeerDAS monitoring, compatibility and simplify the process of building summary dashboards for PeerDAS performance tracking and analysis.

The list is open for discussion. Each client has the opportunity to contribute to it by suggesting additions or disputing existing metrics.

Issue #11

Data column, kzg, custody metrics

Name	Usage
beacon_data_column_sidecar_processing_requests_total	Number of data column sidecars submitted for processing (counter)
beacon_data_column_sidecar_processing_successes_total	Number of data column sidecars verified for gossip (counter)
beacon_data_column_sidecar_gossip_verification_seconds	Full runtime of data column sidecars gossip verification (histogram)
beacon_data_availability_reconstructed_columns_total	Total count of reconstructed columns (counter)
beacon_data_availability_reconstruction_time_seconds	Time taken to reconstruct columns (histogram)
beacon_data_column_sidecar_computation_seconds	Time taken to compute data column sidecar, including cells and inclusion proof (histogram)
beacon_data_column_sidecar_inclusion_proof_verification_seconds	Time taken to verify data column sidecar inclusion proof (histogram)
beacon_kzg_verification_data_column_batch_seconds	Runtime of batched data column kzg verification (histogram)
beacon_custody_groups	Total number of custody groups within a node (gauge)
beacon_custody_groups_backfilled	Total number of custody groups backfilled by a node (gauge)
beacon_engine_getBlobsV2_requests_total	Total number of engine_getBlobsV2 requests sent (counter)
beacon_engine_getBlobsV2_responses_total	Total number of engine_getBlobsV2 successful responses received (counter)
beacon_engine_getBlobsV2_request_duration_seconds	Duration of engine_getBlobsV2 requests (histogram)

[KatyaRyazantseva]

Clients' Status Overview

Metric	Lighthouse	Teku	Prysm	Lodestar	Nimbus	Grandine
Raised issue	#6018	#65	#14129
beacon_data_column_sidecar_processing_requests_total	✅	✅	✅	✅	✅	✅
beacon_data_column_sidecar_processing_successes_total	✅	✅	✅	✅	✅	✅
beacon_data_column_sidecar_gossip_verification_seconds	✅	✅	✅	✅	✅	✅
beacon_data_availability_reconstructed_columns_total	✅	✅	📝	□	□	📝
beacon_data_availability_reconstruction_time_seconds	✅	✅	📝	□	□	📝
beacon_data_column_sidecar_computation_seconds	✅	✅	✅	✅	✅	📝
beacon_data_column_sidecar_inclusion_proof_verification_seconds	✅	✅	✅	✅	✅	📝
beacon_kzg_verification_data_column_batch_seconds	✅	✅	✅	✅	✅	📝
beacon_custody_groups	□	✅	□	□	□	□
beacon_custody_groups_backfilled	□	✅	□	□	□	□
beacon_engine_getBlobsV2_requests_total	📝	✅	□	✅	📝	□
beacon_engine_getBlobsV2_responses_total	📝	✅	□	✅	📝	□
beacon_engine_getBlobsV2_request_duration_seconds	📝	✅	□	✅	📝	□

✅ - implemented
📝 - in progress, requiring adjustments
□ - not implemented

[jimmygchen]

@KatyaRyazantseva in the last call you mentioned that there was proposals to make this milliseconds for more granularity.

Lighthouse currently record these metrics in seconds with floating-point precision, which already provides a high degree of granularity, depending on how the histogram buckets are set.

The units for histograms in all prometheus client libraries are standardised to seconds:

A histogram SHOULD have the following methods:
Some way to time code for users in seconds. In Python this is the time() decorator/context manager. In Java this is startTimer/observeDuration. Units other than seconds MUST NOT be offered (if a user wants something else, they can do it by hand). This should follow the same pattern as Gauge/Summary.

https://prometheus.io/docs/instrumenting/writing_clientlibs/#histogram

[jimmygchen]

As discussed on Discord, we'll leave it up to clients to choose between seconds vs milliseconds.

[jimmygchen]

As discussed on the call, this metric can be removed.

[jimmygchen]

The name and description may be a bit confusing here, as "processing" could mean different things in different clients. The description makes sense in Lighthouse because we have a task scheduling component called BeaconProcessor, and all the tasks are "submitted" to this scheduler for processing - so you see this terminology used quite frequently in our metrics.

It's more of a convenient metric rather than a must-have in Lighthouse, as we can get the same data with count(beacon_data_column_sidecar_gossip_verification_seconds). IMO we should probably minimise the number of standardise metrics, so we don't force all clients to implement metrics that aren't necessary for them. It comes with maintenance cost (once introduced, renaming / removing would be a breaking change) as well as extra prometheus storage cost.

[jimmygchen]

It may still make sense to have this metric though.

For a few of our metrics we discard the timer if the operations fails, so it may not reflect the actual attemtpt count.

Is it also worth mentioning whether we should discard a timer metric if the operation fails?

[jimmygchen]

ah right, I see this implemented in Teku already, if it make sense feel free to ignore the above comment!

[jimmygchen]

Is it worth suggesting a historgram bucket? e.g. [0.1, 0.15, 0.25, 0.35, 0.5, 0.7, 1.0, 2.5, 5.0, 10.0]

[jimmygchen]

We've also implemented a blob_count label, so that we can individual timings for different blob counts.

[jimmygchen]

Our current buckets for this histogram in case it helps:
0.002, 0.004, 0.006, 0.008, 0.01, 0.012, 0.015, 0.02, 0.03, 0.05, 0.07

[jimmygchen]

A few more reconstruction metrics that could be useful:

• kzg_data_column_reconstruction_attempts: Count of times data column reconstruction has been attempted
• kzg_data_column_reconstruction_failures: Count of times data column reconstruction has failed

Lighthouse source for this here

[jimmygchen]

Is there another metric for tracking the custody count?

[KatyaRyazantseva]

No, only this one

[nalepae]

I think the comment count of columns in custody within the data availability boundary is a bit misleading.

This is a counter, so by definition it always increases (or restart at 0).
This counter could be used, but summing all the increases during the retention window, to effectively know the count of columns we custody.

But saying that the counter itself ``count of columns in custody within the data availability boundary` is a bit false IMO.

[KatyaRyazantseva]

@KatyaRyazantseva

Do you think we could have a few more metrics: Network & Storage Metrics

I would prefer to raise a new issue on that, share it for discussion with the clients and then open a new PR.
For example, this one beacon_data_column_sidecar_processing_failures_total already exists on the dashboard as a difference between beacon_data_column_sidecar_processing_requests_total and beacon_data_column_sidecar_processing_successes_total.

[barnabasbusa]

removed my comment and moved it to an issue: #17