Lighthouse CL < > Prysm VC incompatibility

[barnabasbusa]

Describe the bug

We are in the process to test cross beacon <> validator client compatibility, and found a bug when testing nimbus CL with prysm VC.

Prysm reports:

time="2024-07-04 11:57:36" level=info msg="Waiting for deposit to be observed by beacon node" prefix=client pubkey=0xb9e03b94bb69 status="UNKNOWN_STATUS

Lighthouse reports:

nothing out of the ordinary

Snooper between cl <> vc reports:

time="2024-07-04T14:12:48Z" level=info msg="REQUEST #11: POST /eth/v1/beacon/states/head/validators" length=6487 type=json
{
  "ids": [
    "0xaaf6c1251e73fb600624937760fef218aace5b253bf068ed45398aeb29d821e4d2899343ddcbbe37cb3f6cf500dff26c",
   ...
"0x8c0d15baa72bfcd317e9b9402ca9bb6e7ae1db35ffce7faccae0bd19b3c8e5de7d5524aef0377770b3a90626627a9304",
    "0xaf61f263addfb41c46d66e60ecfb598a5942f648f58718b6b4e4c92019fdb12328efbff98703134bcf28e9c1fab4bb60"
  ],
  "statuses": []
}

time="2024-07-04T14:12:48Z" level=info msg="RESPONSE #11: POST /eth/v1/beacon/states/head/validators" length=58 status=200 type=json
{
  "data": [],
  "execution_optimistic": false,
  "finalized": false
}

time="2024-07-04T14:12:48Z" level=info msg="REQUEST #12: GET /eth/v1/node/version" length=0 type=json
time="2024-07-04T14:12:48Z" level=info msg="RESPONSE #12: GET /eth/v1/node/version" length=62 status=200 type=json
{
  "data": {
    "version": "Lighthouse/v5.2.1-9e12c21/aarch64-linux"
  }
}

It seems like prysm is stuck on requesting genesis state, and stops there.

Expected behavior

I would expect all client combinations would work.

Steps to reproduce

config.yaml:

participants:
  - cl_type: lighthouse
    vc_type: prysm
additional_services:
  - dora
snooper_enabled: true
global_log_level: debug

kurtosis run github.com/kurtosis-tech/ethereum-package --args-file config.yaml

Additional context

Current BN <> VC Compatibility list tracker

Has this worked before in a previous version?

No response

🔬 Minimal Reproduction

No response

Error

No response

Platform(s)

No response

What version of Prysm are you running? (Which release)

No response

Anything else relevant (validator index / public key)?

No response

cc: @pk910

[nflaig]

This is likely related to Prysm VC sending an empty statuses array in POST /eth/v1/beacon/states/head/validators

[barnabasbusa]

Looks like using the latest versions, its looking even worse:

time="2025-03-26 11:43:30" level=error msg="error getting validator duties" error="failed to get duties for next epoch `1`: failed to get attester duties for epoch `1`: HTTP request unsuccessful (500: UNHANDLED_ERROR: UnableToReadSlot)" prefix=client
time="2025-03-26 11:43:30" level=error msg="Failed to update assignments" error="failed to get duties for next epoch `1`: failed to get attester duties for epoch `1`: HTTP request unsuccessful (500: UNHANDLED_ERROR: UnableToReadSlot)" prefix=client
time="2025-03-26 11:43:30" level=info msg="Starting health check routine for beacon node apis" prefix=client
time="2025-03-26 11:43:30" level=info msg="Event stream reconnecting..." prefix=client
time="2025-03-26 11:43:30" level=info msg="Starting event stream" prefix=client topics=[head]
time="2025-03-26 11:43:30" level=info msg="Listening to Beacon API events" topics=head
time="2025-03-26 11:43:30" level=fatal msg="Failed to update proposer settings" error="HTTP request unsuccessful (500: UNHANDLED_ERROR: UnableToReadSlot)" prefix=client

config to reproduce:

participants:
  - el_type: geth
    el_image: ethereum/client-go:latest
    cl_type: lighthouse
    cl_image: sigp/lighthouse:latest-unstable
    vc_type: prysm
    vc_image: gcr.io/prysmaticlabs/prysm/validator:latest
    count: 1
network_params:
  genesis_delay: 120
  min_validator_withdrawability_delay: 1
  shard_committee_period: 1
  num_validator_keys_per_node: 250
  churn_limit_quotient: 16
additional_services:
  - assertoor
snooper_enabled: false
disable_peer_scoring: true
assertoor_params:
  image: "ethpandaops/assertoor:master"
  run_stability_check: false
  run_block_proposal_check: false
  tests:
    - https://raw.githubusercontent.com/ethpandaops/assertoor-test/5a9c3c0278e6cd977ec2ef56f4091fdce7e8b7f7/assertoor-tests/block-proposal-check.yaml

[james-prysm]

waiting on sigp/lighthouse#7729 to fix this

[barnabasbusa]

Can confirm prysm vc works now with ethpandaops/lighthouse:pawanjay176-duties-pre-genesis-552421c