downloader: segmentation fault on restart when downloading beacon headers #26764
Description
System information
Geth version: Geth/v1.10.26-stable-e5eb32ac/linux-amd64/go1.18.8
CL client & version: [email protected]
OS & Version: Linux (Docker@kubernetes)
Expected behaviour
- Geth receives a SIGTERM and gracefully stops
- Geth is started again
- Starts syncing normally
Actual behaviour
- Geth receives a SIGTERM and gracefully stops
- Geth is started again
- Keeps crashing with the following stack trace (even after restarting Geth or Teku):
2023-02-22T07:22:14+00:00 INFO [02-22|07:22:14.046] Syncing beacon headers downloaded=2 left=0 eta=0s
2023-02-22T07:22:14+00:00 panic: runtime error: invalid memory address or nil pointer dereference
2023-02-22T07:22:14+00:00 [signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xa9857b]
2023-02-22T07:22:14+00:00
2023-02-22T07:22:14+00:00 goroutine 4999 [running]:
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc00017c780, 0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader/downloader.go:491 +0x49b
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0x0?, {0x0, 0x0}, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader/downloader.go:439 +0x505
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader.(*beaconBackfiller).resume.func1()
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:108 +0x96
2023-02-22T07:22:14+00:00 created by github.com/ethereum/go-ethereum/eth/downloader.(*beaconBackfiller).resume
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:98 +0x132
Steps to reproduce the behaviour
Kept happening on restart. However, I'm unable to reproduce in other running Geth/Teku.
The only way to fix was to sync Geth from scratch. It's like DB got corrupted.
Backtrace
Note: [redacted] added to hide sensitive information.
Geth logs:
2023-02-22T07:19:38+00:00 INFO [02-22|07:19:38.496] Imported new potential chain segment blocks=1 txs=144 mgas=18.713 elapsed=1.654s mgasps=11.307 number=16,682,464 hash=9ddbf1..5502aa dirty=89.77MiB
2023-02-22T07:19:38+00:00 INFO [02-22|07:19:38.517] Chain head was updated number=16,682,464 hash=9ddbf1..5502aa root=d699a2..408e8e elapsed=4.340253ms
2023-02-22T07:19:38+00:00 INFO [02-22|07:19:38.519] Unindexed transactions blocks=1 txs=305 tail=14,332,465 elapsed=2.336ms
2023-02-22T07:19:39+00:00 WARN [02-22|07:19:39.603] Checkpoint challenge timed out, dropping id=9dd7044669d7923a conn=dyndial addr=2.126.138.241:30303 type=Geth/v1.11.1-unstabl...
2023-02-22T07:19:48+00:00 INFO [02-22|07:19:48.232] Imported new potential chain segment blocks=1 txs=107 mgas=9.151 elapsed=116.998ms mgasps=78.212 number=16,682,465 hash=fba52f..dabb36 dirty=91.11MiB
2023-02-22T07:19:48+00:00 INFO [02-22|07:19:48.245] Chain head was updated number=16,682,465 hash=fba52f..dabb36 root=8a3296..edc8aa elapsed=4.303542ms
2023-02-22T07:19:48+00:00 INFO [02-22|07:19:48.254] Unindexed transactions blocks=1 txs=355 tail=14,332,466 elapsed=8.281ms
2023-02-22T07:20:00+00:00 INFO [02-22|07:20:00.104] Got interrupt, shutting down...
2023-02-22T07:20:00+00:00 WARN [02-22|07:20:00.105] HTTP server graceful shutdown timed out
2023-02-22T07:20:00+00:00 INFO [02-22|07:20:00.105] HTTP server stopped endpoint=[::]:8545
2023-02-22T07:20:00+00:00 WARN [02-22|07:20:00.106] HTTP server graceful shutdown timed out
2023-02-22T07:20:00+00:00 INFO [02-22|07:20:00.106] HTTP server stopped endpoint=[::]:8546
2023-02-22T07:20:00+00:00 INFO [02-22|07:20:00.148] Imported new potential chain segment blocks=1 txs=107 mgas=12.345 elapsed=182.103ms mgasps=67.792 number=16,682,466 hash=f36b1c..3b9533 dirty=92.34MiB
2023-02-22T07:20:00+00:00 WARN [02-22|07:20:00.174] HTTP server graceful shutdown timed out
2023-02-22T07:20:00+00:00 INFO [02-22|07:20:00.174] HTTP server stopped endpoint=[::]:8550
2023-02-22T07:20:00+00:00 INFO [02-22|07:20:00.174] IPC endpoint closed url=/data/geth.ipc
2023-02-22T07:20:00+00:00 INFO [02-22|07:20:00.501] Ethereum protocol stopped
2023-02-22T07:20:00+00:00 INFO [02-22|07:20:00.501] Transaction pool stopped
2023-02-22T07:20:00+00:00 INFO [02-22|07:20:00.552] Writing cached state to disk block=16,682,465 hash=fba52f..dabb36 root=8a3296..edc8aa
2023-02-22T07:20:01+00:00 INFO [02-22|07:20:01.115] Persisted trie from memory database nodes=113,101 size=37.46MiB time=562.673164ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=88185 livesize=36.05MiB
2023-02-22T07:20:01+00:00 INFO [02-22|07:20:01.115] Writing cached state to disk block=16,682,464 hash=9ddbf1..5502aa root=d699a2..408e8e
2023-02-22T07:20:01+00:00 INFO [02-22|07:20:01.127] Persisted trie from memory database nodes=1633 size=691.41KiB time=12.112434ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=86552 livesize=35.38MiB
2023-02-22T07:20:01+00:00 INFO [02-22|07:20:01.128] Writing cached state to disk block=16,682,338 hash=50f020..e67af4 root=201040..6e430d
2023-02-22T07:20:01+00:00 INFO [02-22|07:20:01.128] Persisted trie from memory database nodes=0 size=0.00B time="4.1µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=86552 livesize=35.38MiB
2023-02-22T07:20:01+00:00 INFO [02-22|07:20:01.128] Writing snapshot state to disk root=bcd0dc..756907
2023-02-22T07:20:01+00:00 INFO [02-22|07:20:01.128] Persisted trie from memory database nodes=0 size=0.00B time="1.8µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=86552 livesize=35.38MiB
2023-02-22T07:20:01+00:00 INFO [02-22|07:20:01.320] Writing clean trie cache to disk path=/data/geth/triecache threads=8
2023-02-22T07:20:01+00:00 INFO [02-22|07:20:01.529] Persisted the clean trie cache path=/data/geth/triecache elapsed=209.577ms
2023-02-22T07:20:01+00:00 INFO [02-22|07:20:01.529] Blockchain stopped
2023-02-22T07:20:25+00:00 INFO [02-22|07:20:25.331] Starting Geth on Ethereum mainnet...
2023-02-22T07:20:25+00:00 INFO [02-22|07:20:25.331] Bumping default cache on mainnet provided=1024 updated=4096
2023-02-22T07:20:25+00:00 INFO [02-22|07:20:25.333] Maximum peer count ETH=50 LES=0 total=50
2023-02-22T07:20:25+00:00 INFO [02-22|07:20:25.334] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
2023-02-22T07:20:25+00:00 INFO [02-22|07:20:25.338] Set global gas cap cap=50,000,000
2023-02-22T07:20:25+00:00 INFO [02-22|07:20:25.348] Allocated trie memory caches clean=614.00MiB dirty=1024.00MiB
2023-02-22T07:20:25+00:00 INFO [02-22|07:20:25.348] Allocated cache and file handles database=/data/geth/chaindata cache=2.00GiB handles=524,288
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.279] Opened ancient database database=/data/geth/chaindata/ancient/chain readonly=false
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429]
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] ---------------------------------------------------------------------------------------------------------------------------------------------------------
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] Chain ID: 1 (mainnet)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429]
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] Pre-Merge hard forks:
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Homestead: 1150000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - DAO Fork: 1920000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Tangerine Whistle (EIP 150): 2463000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Spurious Dragon/1 (EIP 155): 2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Spurious Dragon/2 (EIP 158): 2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Byzantium: 4370000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Constantinople: 7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Petersburg: 7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Istanbul: 9069000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Muir Glacier: 9200000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Berlin: 12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - London: 12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Arrow Glacier: 13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429] - Gray Glacier: 15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.429]
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.430] Merge configured:
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.430] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.430] - Network known to be merged: true
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.430] - Total terminal difficulty: 58750000000000000000000
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.430] - Merge netsplit block: <nil>
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.430] ---------------------------------------------------------------------------------------------------------------------------------------------------------
2023-02-22T07:20:28+00:00 INFO [02-22|07:20:28.430]
2023-02-22T07:20:29+00:00 INFO [02-22|07:20:29.048] Disk storage enabled for ethash caches dir=/data/geth/ethash count=3
2023-02-22T07:20:29+00:00 INFO [02-22|07:20:29.048] Disk storage enabled for ethash DAGs dir=/.ethash count=2
2023-02-22T07:20:29+00:00 INFO [02-22|07:20:29.055] Initialising Ethereum protocol network=1 dbversion=8
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.450] Loaded most recent local header number=16,682,465 hash=fba52f..dabb36 td=58,750,003,716,598,352,816,469 age=43s
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.450] Loaded most recent local full block number=16,682,465 hash=fba52f..dabb36 td=58,750,003,716,598,352,816,469 age=43s
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.450] Loaded most recent local fast block number=16,682,465 hash=fba52f..dabb36 td=58,750,003,716,598,352,816,469 age=43s
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.450] Loaded most recent local finalized block number=16,682,396 hash=5b271d..3e1faa td=58,750,003,716,598,352,816,469 age=14m31s
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.453] Loaded last fast-sync pivot marker number=16,631,568
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.588] Loaded local transaction journal transactions=0 dropped=0
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.589] Regenerated local transaction journal transactions=0 accounts=0
2023-02-22T07:20:30+00:00 WARN [02-22|07:20:30.589] Switch sync mode from snap sync to full sync
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.589] Chain post-merge, sync via beacon client
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.589] Gasprice oracle is ignoring threshold set threshold=2
2023-02-22T07:20:30+00:00 WARN [02-22|07:20:30.590] Engine API enabled protocol=eth
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.591] Starting peer-to-peer node instance=Geth/v1.10.26-stable-e5eb32ac/linux-amd64/go1.18.8
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.626] Mapped network port proto=tcp extport=30410 intport=30410 interface=ExtIP(52.224.131.167)
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.626] Mapped network port proto=udp extport=30410 intport=30410 interface=ExtIP(52.224.131.167)
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.641] New local node record seq=1,676,371,331,486 id=cb43262cbb1052d0 ip=52.224.131.167 udp=30410 tcp=30410
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.641] Started P2P networking self=enode://5cf70def6b4823ec89a3b6911a276e7697a3229e9540c7164f8cf8960f6371e14944ce0c41984826f39cb69fac40f046971e2afc107d528b7d5d6b931d6cb194@52.224.131.167:30410
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.642] IPC endpoint opened url=/data/geth.ipc
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.643] Loaded JWT secret file path=[redacted] crc32=[redacted]
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.643] HTTP server started endpoint=[::]:8545 auth=false prefix= cors=* vhosts=*
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.643] WebSocket enabled url=ws://[::]:8546
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.643] WebSocket enabled url=ws://[::]:8550
2023-02-22T07:20:30+00:00 INFO [02-22|07:20:30.643] HTTP server started endpoint=[::]:8550 auth=true prefix= cors=localhost vhosts=*
2023-02-22T07:20:37+00:00 WARN [02-22|07:20:37.535] State not available, ignoring new payload
2023-02-22T07:20:37+00:00 INFO [02-22|07:20:37.650] Forkchoice requested sync to new head number=16,682,467 hash=017a52..de9237
2023-02-22T07:20:37+00:00 INFO [02-22|07:20:37.827] Forkchoice requested sync to new head number=16,682,468 hash=b21cce..e26ed0
2023-02-22T07:20:37+00:00 INFO [02-22|07:20:37.994] Forkchoice requested sync to new head number=16,682,469 hash=7398fd..c77eb6
2023-02-22T07:20:41+00:00 INFO [02-22|07:20:41.005] Looking for peers peercount=0 tried=20 static=6
2023-02-22T07:20:47+00:00 INFO [02-22|07:20:47.649] Forkchoice requested sync to new head number=16,682,470 hash=6546de..554bee
2023-02-22T07:20:51+00:00 INFO [02-22|07:20:51.008] Looking for peers peercount=0 tried=26 static=6
2023-02-22T07:20:57+00:00 WARN [02-22|07:20:57.443] Snapshot extension registration failed peer=2e150a78 err="peer connected on snap without compatible eth support"
2023-02-22T07:21:00+00:00 INFO [02-22|07:21:00.496] Forkchoice requested sync to new head number=16,682,471 hash=ec886b..5f0590
2023-02-22T07:21:01+00:00 INFO [02-22|07:21:01.020] Looking for peers peercount=0 tried=29 static=6
2023-02-22T07:21:11+00:00 INFO [02-22|07:21:11.054] Looking for peers peercount=1 tried=41 static=6
2023-02-22T07:21:12+00:00 INFO [02-22|07:21:12.152] Forkchoice requested sync to new head number=16,682,472 hash=9e7213..c7cfed
2023-02-22T07:21:22+00:00 INFO [02-22|07:21:22.011] Looking for peers peercount=0 tried=36 static=6
2023-02-22T07:21:32+00:00 INFO [02-22|07:21:32.192] Looking for peers peercount=1 tried=37 static=6
2023-02-22T07:21:33+00:00 INFO [02-22|07:21:33.523] Forkchoice requested sync to new head number=16,682,473 hash=e8f38f..57927f
2023-02-22T07:21:37+00:00 WARN [02-22|07:21:37.495] Snapshot extension registration failed peer=1d4fc8e3 err="peer connected on snap without compatible eth support"
2023-02-22T07:21:42+00:00 INFO [02-22|07:21:42.259] Looking for peers peercount=1 tried=35 static=6
2023-02-22T07:21:47+00:00 INFO [02-22|07:21:47.291] Syncing beacon headers downloaded=1 left=0 eta=0s
2023-02-22T07:21:47+00:00 INFO [02-22|07:21:47.291] Block synchronisation started
2023-02-22T07:21:47+00:00 WARN [02-22|07:21:47.297] Retrieved pivot header from local number=16,682,409 hash=4f7e76..718af2 latest=16,682,473 oldest=16,682,466
2023-02-22T07:21:47+00:00 INFO [02-22|07:21:47.388] Downloader queue stats receiptTasks=0 blockTasks=0 itemSize=5.20KiB throttle=8192
2023-02-22T07:21:48+00:00 INFO [02-22|07:21:48.608] Imported new chain segment blocks=1 txs=107 mgas=12.345 elapsed=1.219s mgasps=10.126 number=16,682,466 hash=f36b1c..3b9533 age=1m49s dirty=1.23MiB
2023-02-22T07:21:48+00:00 INFO [02-22|07:21:48.617] Unindexed transactions blocks=1 txs=104 tail=14,332,467 elapsed=8.485ms
2023-02-22T07:21:52+00:00 INFO [02-22|07:21:52.338] Looking for peers peercount=1 tried=33 static=6
2023-02-22T07:21:52+00:00 WARN [02-22|07:21:52.648] Ignoring payload with missing parent number=16,682,474 hash=fb6d0a..aeba9f parent=e8f38f..57927f
2023-02-22T07:21:52+00:00 INFO [02-22|07:21:52.838] Forkchoice requested sync to new head number=16,682,474 hash=fb6d0a..aeba9f
2023-02-22T07:21:52+00:00 WARN [02-22|07:21:52.839] Beacon chain forked ancestor=16,682,473 hash=df95b1..88ff60 want=e8f38f..57927f
2023-02-22T07:22:01+00:00 INFO [02-22|07:22:01.991] Imported new chain segment blocks=5 txs=701 mgas=80.755 elapsed=13.383s mgasps=6.034 number=16,682,471 hash=ec886b..5f0590 age=1m2s dirty=10.90MiB
2023-02-22T07:22:02+00:00 INFO [02-22|07:22:02.662] Looking for peers peercount=1 tried=34 static=6
2023-02-22T07:22:12+00:00 INFO [02-22|07:22:12.794] Looking for peers peercount=2 tried=33 static=6
2023-02-22T07:22:13+00:00 INFO [02-22|07:22:13.636] Imported new chain segment blocks=2 txs=380 mgas=42.841 elapsed=11.644s mgasps=3.679 number=16,682,473 hash=df95b1..88ff60 dirty=16.18MiB
2023-02-22T07:22:13+00:00 INFO [02-22|07:22:13.643] Unindexed transactions blocks=7 txs=1373 tail=14,332,474 elapsed=7.335ms
2023-02-22T07:22:13+00:00 INFO [02-22|07:22:13.648] Forkchoice requested sync to new head number=16,682,475 hash=f77938..b521a7
2023-02-22T07:22:13+00:00 INFO [02-22|07:22:13.819] Forkchoice requested sync to new head number=16,682,476 hash=1ffc0b..109fd8
2023-02-22T07:22:14+00:00 INFO [02-22|07:22:14.046] Syncing beacon headers downloaded=2 left=0 eta=0s
2023-02-22T07:22:14+00:00 panic: runtime error: invalid memory address or nil pointer dereference
2023-02-22T07:22:14+00:00 [signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xa9857b]
2023-02-22T07:22:14+00:00
2023-02-22T07:22:14+00:00 goroutine 4999 [running]:
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc00017c780, 0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader/downloader.go:491 +0x49b
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0x0?, {0x0, 0x0}, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader/downloader.go:439 +0x505
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader.(*beaconBackfiller).resume.func1()
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:108 +0x96
2023-02-22T07:22:14+00:00 created by github.com/ethereum/go-ethereum/eth/downloader.(*beaconBackfiller).resume
2023-02-22T07:22:14+00:00 github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:98 +0x132
Teku logs in the same timeframe:
2023-02-22T07:19:39+00:00 2023-02-22 07:19:39.134 INFO - Slot Event *** Slot: 5852196, Block: 708176be799d16e97165318b4a69172c6747d54b6c9481e08c902ffc001853a6, Justified: 182880, Finalized: 182879, Peers: 64
2023-02-22T07:19:51+00:00 2023-02-22 07:19:51.063 INFO - Slot Event *** Slot: 5852197, Block: 5fade4f9513845fc142829188c2b344fe5de0c2fe203b1b37531daf2b11206c6, Justified: 182880, Finalized: 182879, Peers: 64
2023-02-22T07:20:00+00:00 2023-02-22 07:20:00.170 ERROR - Execution Client request failed. Make sure the Execution Client is online and in sync.
2023-02-22T07:20:00+00:00 java.net.ConnectException: Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at tech.pegasys.teku.ethereum.executionclient.auth.JwtAuthHttpInterceptor.intercept(JwtAuthHttpInterceptor.java:41) ~[teku-ethereum-executionclient-22.12.0.jar:22.12.0]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at org.web3j.protocol.http.HttpService.performIO(HttpService.java:165) ~[core-4.9.4.jar:?]
2023-02-22T07:20:00+00:00 at org.web3j.protocol.Service.send(Service.java:48) ~[core-4.9.4.jar:?]
2023-02-22T07:20:00+00:00 at org.web3j.protocol.Service.lambda$sendAsync$0(Service.java:60) ~[core-4.9.4.jar:?]
2023-02-22T07:20:00+00:00 at org.web3j.utils.Async.lambda$run$1(Async.java:38) ~[core-4.9.4.jar:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 Suppressed: java.io.IOException: unexpected end of stream on http://[redacted]:8550/...
2023-02-22T07:20:00+00:00 at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:202) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at tech.pegasys.teku.ethereum.executionclient.auth.JwtAuthHttpInterceptor.intercept(JwtAuthHttpInterceptor.java:41) ~[teku-ethereum-executionclient-22.12.0.jar:22.12.0]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at org.web3j.protocol.http.HttpService.performIO(HttpService.java:165) ~[core-4.9.4.jar:?]
2023-02-22T07:20:00+00:00 at org.web3j.protocol.Service.send(Service.java:48) ~[core-4.9.4.jar:?]
2023-02-22T07:20:00+00:00 at org.web3j.protocol.Service.lambda$sendAsync$0(Service.java:60) ~[core-4.9.4.jar:?]
2023-02-22T07:20:00+00:00 at org.web3j.utils.Async.lambda$run$1(Async.java:38) ~[core-4.9.4.jar:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 Caused by: java.io.EOFException: \n not found: limit=0 content=…
2023-02-22T07:20:00+00:00 at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:332) ~[okio-jvm-2.8.0.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:178) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 ... 23 more
2023-02-22T07:20:00+00:00 Caused by: java.net.ConnectException: Connection refused
2023-02-22T07:20:00+00:00 at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
2023-02-22T07:20:00+00:00 at sun.nio.ch.Net.pollConnectNow(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at sun.nio.ch.NioSocketImpl.timedFinishConnect(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at sun.nio.ch.NioSocketImpl.connect(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.net.Socket.connect(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:00+00:00 ... 25 more
2023-02-22T07:20:00+00:00 2023-02-22 07:20:00.171 ERROR - Failed to update fork choice
2023-02-22T07:20:00+00:00 java.util.concurrent.CompletionException: tech.pegasys.teku.ethereum.executionlayer.InvalidRemoteResponseException: Invalid remote response from the execution client: Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:148) ~[teku-infrastructure-async-22.12.0.jar:22.12.0]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:148) ~[teku-infrastructure-async-22.12.0.jar:22.12.0]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.whenComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at tech.pegasys.teku.infrastructure.async.SafeFuture.whenComplete(SafeFuture.java:603) ~[teku-infrastructure-async-22.12.0.jar:22.12.0]
2023-02-22T07:20:00+00:00 at tech.pegasys.teku.infrastructure.async.SafeFuture.whenComplete(SafeFuture.java:33) ~[teku-infrastructure-async-22.12.0.jar:22.12.0]
2023-02-22T07:20:00+00:00 at tech.pegasys.teku.infrastructure.async.SafeFuture.propagateResult(SafeFuture.java:143) ~[teku-infrastructure-async-22.12.0.jar:22.12.0]
2023-02-22T07:20:00+00:00 at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$exceptionallyCompose$33(SafeFuture.java:422) ~[teku-infrastructure-async-22.12.0.jar:22.12.0]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:146) ~[teku-infrastructure-async-22.12.0.jar:22.12.0]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at org.web3j.utils.Async.lambda$run$1(Async.java:40) ~[core-4.9.4.jar:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-02-22T07:20:00+00:00 Caused by: tech.pegasys.teku.ethereum.executionlayer.InvalidRemoteResponseException: Invalid remote response from the execution client: Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:00+00:00 at tech.pegasys.teku.ethereum.executionlayer.ResponseUnwrapper.unwrapResponseOrThrow(ResponseUnwrapper.java:34) ~[teku-ethereum-executionlayer-22.12.0.jar:22.12.0]
2023-02-22T07:20:00+00:00 at tech.pegasys.teku.ethereum.executionlayer.ResponseUnwrapper.unwrapExecutionClientResponseOrThrow(ResponseUnwrapper.java:21) ~[teku-ethereum-executionlayer-22.12.0.jar:22.12.0]
2023-02-22T07:20:00+00:00 ... 30 more
2023-02-22T07:20:03+00:00 2023-02-22 07:20:03.077 INFO - Slot Event *** Slot: 5852198, Block: 8ea080075f151c7a2a9bbea94879710a053c39de6f46bce45f28e143777e6ac8, Justified: 182880, Finalized: 182879, Peers: 64
2023-02-22T07:20:04+00:00 2023-02-22 07:20:04.896 ERROR - Failed to update eth1 chain head - Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:07+00:00 2023-02-22 07:20:07.904 ERROR - Failed to update eth1 chain head - Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:10+00:00 2023-02-22 07:20:10.912 ERROR - Failed to update eth1 chain head - Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:15+00:00 2023-02-22 07:20:15.072 INFO - Slot Event *** Slot: 5852199, Block: ... empty, Justified: 182880, Finalized: 182879, Peers: 64
2023-02-22T07:20:15+00:00 2023-02-22 07:20:15.296 ERROR - Unable to import block: Execution Client returned an error: tech.pegasys.teku.ethereum.executionlayer.InvalidRemoteResponseException: Invalid remote response from the execution client: Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:15+00:00 2023-02-22 07:20:15.296 WARN - Late Block Import *** Block: aa84fb1d0bb9abd1cbefb15f20d2be82f87d97639985bf24de228eaf9cb81a7b (5852199) proposer 452459 arrival 1259ms, pre-state_retrieved +6ms, processed +107ms, execution_payload_result_received +2924ms, begin_importing +0ms, completed +0ms
2023-02-22T07:20:17+00:00 2023-02-22 07:20:17.296 INFO - Retrying execution of block aa84fb1d0bb9abd1cbefb15f20d2be82f87d97639985bf24de228eaf9cb81a7b (5852199)
2023-02-22T07:20:18+00:00 2023-02-22 07:20:18.368 ERROR - Unable to import block: Execution Client returned an error: tech.pegasys.teku.ethereum.executionlayer.InvalidRemoteResponseException: Invalid remote response from the execution client: Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:20+00:00 2023-02-22 07:20:20.913 ERROR - Failed to update eth1 chain head
2023-02-22T07:20:20+00:00 java.io.InterruptedIOException: timeout
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:398) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.RealCall.callDone(RealCall.kt:360) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.RealCall.noMoreExchanges$okhttp(RealCall.kt:325) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:209) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at org.web3j.protocol.http.HttpService.performIO(HttpService.java:165) ~[core-4.9.4.jar:?]
2023-02-22T07:20:20+00:00 at org.web3j.protocol.Service.send(Service.java:48) ~[core-4.9.4.jar:?]
2023-02-22T07:20:20+00:00 at org.web3j.protocol.Service.lambda$sendAsync$0(Service.java:60) ~[core-4.9.4.jar:?]
2023-02-22T07:20:20+00:00 at org.web3j.utils.Async.lambda$run$1(Async.java:38) ~[core-4.9.4.jar:?]
2023-02-22T07:20:20+00:00 at java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source) ~[?:?]
2023-02-22T07:20:20+00:00 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
2023-02-22T07:20:20+00:00 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
2023-02-22T07:20:20+00:00 at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-02-22T07:20:20+00:00 Caused by: java.net.SocketException: Socket closed
2023-02-22T07:20:20+00:00 at sun.nio.ch.NioSocketImpl.endConnect(Unknown Source) ~[?:?]
2023-02-22T07:20:20+00:00 at sun.nio.ch.NioSocketImpl.connect(Unknown Source) ~[?:?]
2023-02-22T07:20:20+00:00 at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?]
2023-02-22T07:20:20+00:00 at java.net.Socket.connect(Unknown Source) ~[?:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at tech.pegasys.teku.ethereum.executionclient.auth.JwtAuthHttpInterceptor.intercept(JwtAuthHttpInterceptor.java:41) ~[teku-ethereum-executionclient-22.12.0.jar:22.12.0]
2023-02-22T07:20:20+00:00 at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.9.3.jar:?]
2023-02-22T07:20:20+00:00 ... 9 more
2023-02-22T07:20:21+00:00 2023-02-22 07:20:21.727 WARN - Endpoint http://[redacted]:8550 is INVALID | Connection refused
2023-02-22T07:20:22+00:00 2023-02-22 07:20:22.368 INFO - Retrying execution of block aa84fb1d0bb9abd1cbefb15f20d2be82f87d97639985bf24de228eaf9cb81a7b (5852199)
2023-02-22T07:20:23+00:00 2023-02-22 07:20:23.935 ERROR - Failed to update eth1 chain head - Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:26+00:00 2023-02-22 07:20:26.273 ERROR - Unable to import block: Execution Client returned an error: tech.pegasys.teku.ethereum.executionlayer.InvalidRemoteResponseException: Invalid remote response from the execution client: Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:26+00:00 2023-02-22 07:20:26.945 ERROR - Failed to update eth1 chain head - Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:27+00:00 2023-02-22 07:20:27.040 INFO - Slot Event *** Slot: 5852200, Block: ... empty, Justified: 182880, Finalized: 182879, Peers: 64
2023-02-22T07:20:28+00:00 2023-02-22 07:20:28.947 ERROR - Failed to update eth1 chain head - Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:29+00:00 2023-02-22 07:20:29.426 ERROR - An error occurred while querying remote transition configuration
2023-02-22T07:20:29+00:00 java.util.concurrent.CompletionException: tech.pegasys.teku.ethereum.executionlayer.InvalidRemoteResponseException: Invalid remote response from the execution client: Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:148) ~[teku-infrastructure-async-22.12.0.jar:22.12.0]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:146) ~[teku-infrastructure-async-22.12.0.jar:22.12.0]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at org.web3j.utils.Async.lambda$run$1(Async.java:40) ~[core-4.9.4.jar:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-02-22T07:20:29+00:00 Caused by: tech.pegasys.teku.ethereum.executionlayer.InvalidRemoteResponseException: Invalid remote response from the execution client: Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:29+00:00 at tech.pegasys.teku.ethereum.executionlayer.ResponseUnwrapper.unwrapResponseOrThrow(ResponseUnwrapper.java:34) ~[teku-ethereum-executionlayer-22.12.0.jar:22.12.0]
2023-02-22T07:20:29+00:00 at tech.pegasys.teku.ethereum.executionlayer.ResponseUnwrapper.unwrapExecutionClientResponseOrThrow(ResponseUnwrapper.java:21) ~[teku-ethereum-executionlayer-22.12.0.jar:22.12.0]
2023-02-22T07:20:29+00:00 ... 18 more
2023-02-22T07:20:29+00:00 2023-02-22 07:20:29.474 ERROR - Unable to import block: Execution Client returned an error: tech.pegasys.teku.ethereum.executionlayer.InvalidRemoteResponseException: Invalid remote response from the execution client: Failed to connect to [redacted]/[redacted]:8550
2023-02-22T07:20:37+00:00 2023-02-22 07:20:37.474 INFO - Retrying execution of block aa84fb1d0bb9abd1cbefb15f20d2be82f87d97639985bf24de228eaf9cb81a7b (5852199)
2023-02-22T07:20:37+00:00 2023-02-22 07:20:37.544 INFO - Execution Client is responding to requests again after a previous failure
2023-02-22T07:20:37+00:00 2023-02-22 07:20:37.657 WARN - Unable to execute the current chain head block payload because the Execution Client is syncing. Activating optimistic sync of the beacon chain node
2023-02-22T07:20:38+00:00 2023-02-22 07:20:38.085 INFO - Syncing *** Target slot: 5852201, Head slot: 5852201, Waiting for execution layer sync, Connected peers: 64
2023-02-22T07:20:47+00:00 2023-02-22 07:20:47.049 INFO - Syncing *** Target slot: 5852202, Head slot: 5852201, Waiting for execution layer sync, Connected peers: 62
2023-02-22T07:20:59+00:00 2023-02-22 07:20:59.057 INFO - Syncing *** Target slot: 5852203, Head slot: 5852202, Waiting for execution layer sync, Connected peers: 62
2023-02-22T07:21:11+00:00 2023-02-22 07:21:11.058 INFO - Syncing *** Target slot: 5852204, Head slot: 5852203, Waiting for execution layer sync, Connected peers: 65
2023-02-22T07:21:23+00:00 2023-02-22 07:21:23.059 INFO - Syncing *** Target slot: 5852205, Head slot: 5852204, Waiting for execution layer sync, Connected peers: 65
2023-02-22T07:21:33+00:00 2023-02-22 07:21:33.532 WARN - Late Block Import *** Block: fe54086cfc0d216d7ceb9298630ce43eb0a782aef8cf1f95c275f82333222cf7 (5852205) proposer 25461 arrival 10400ms, pre-state_retrieved +6ms, processed +118ms, execution_payload_result_received +0ms, begin_importing +0ms, transaction_prepared +0ms, transaction_committed +0ms, completed +8ms
2023-02-22T07:21:35+00:00 2023-02-22 07:21:35.059 INFO - Syncing *** Target slot: 5852206, Head slot: 5852205, Waiting for execution layer sync, Connected peers: 65
2023-02-22T07:21:47+00:00 2023-02-22 07:21:47.060 INFO - Syncing *** Target slot: 5852207, Head slot: 5852205, Waiting for execution layer sync, Connected peers: 65
2023-02-22T07:21:52+00:00 2023-02-22 07:21:52.847 WARN - Late Block Import *** Block: 5e1a00d64a4fed146da7629fa7aa8c3d2b6246eb19e10163af8f333ae3b410d4 (5852207) proposer 90275 arrival 5632ms, pre-state_retrieved +7ms, processed +198ms, execution_payload_result_received +0ms, begin_importing +1ms, transaction_prepared +0ms, transaction_committed +0ms, completed +9ms
2023-02-22T07:21:59+00:00 2023-02-22 07:21:59.053 INFO - Syncing *** Target slot: 5852208, Head slot: 5852207, Waiting for execution layer sync, Connected peers: 65
2023-02-22T07:22:09+00:00 2023-02-22 07:22:09.792 WARN - Execution Client request timed out. Make sure the Execution Client is online and in sync.
2023-02-22T07:22:09+00:00 2023-02-22 07:22:09.792 ERROR - Unable to import block: Execution Client returned an error: tech.pegasys.teku.ethereum.executionlayer.InvalidRemoteResponseException: Invalid remote response from the execution client: TimeoutException
2023-02-22T07:22:09+00:00 2023-02-22 07:22:09.793 WARN - Late Block Import *** Block: c99b9697f222a9efc6b0352d077fcd0ca19224141c9d295b5ebae3133ee40d3b (5852208) proposer 501437 arrival 2782ms, pre-state_retrieved +6ms, processed +141ms, execution_payload_result_received +7863ms, begin_importing +0ms, completed +0ms
2023-02-22T07:22:11+00:00 2023-02-22 07:22:11.036 INFO - Syncing *** Target slot: 5852209, Head slot: 5852207, Waiting for execution layer sync, Connected peers: 65
2023-02-22T07:22:11+00:00 2023-02-22 07:22:11.793 INFO - Retrying execution of block c99b9697f222a9efc6b0352d077fcd0ca19224141c9d295b5ebae3133ee40d3b (5852208)
2023-02-22T07:22:13+00:00 2023-02-22 07:22:13.647 INFO - Execution Client is responding to requests again after a previous failure