`panic` on geth/downloader/beaconsync.go:220

[holiman]

version Geth/v1.11.0-unstable-722bb210-20221117/linux-amd64/go1.19.3

Panic on the disco

 Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.937] Snapshot is not continuous with chain snaproot=9ac2ed..16d504 chainroot=e459a1..23facd
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:29.937] Loaded local transaction journal transactions=0 dropped=0
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:29.937] Regenerated local transaction journal transactions=0 accounts=0
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.937] Switch sync mode from snap sync to full sync
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:29.937] Chain post-merge, sync via beacon client
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:29.938] Gasprice oracle is ignoring threshold set threshold=2
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Old unclean shutdowns found count=1593
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Unclean shutdown detected booted=2022-12-04T13:04:18+0000 age=2m11s
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Unclean shutdown detected booted=2022-12-04T13:04:30+0000 age=1m59s
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Unclean shutdown detected booted=2022-12-04T13:04:42+0000 age=1m47s
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Unclean shutdown detected booted=2022-12-04T13:04:53+0000 age=1m36s
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Unclean shutdown detected booted=2022-12-04T13:05:06+0000 age=1m23s
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Unclean shutdown detected booted=2022-12-04T13:05:17+0000 age=1m12s
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Unclean shutdown detected booted=2022-12-04T13:05:29+0000 age=1m
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Unclean shutdown detected booted=2022-12-04T13:05:42+0000 age=47s
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Unclean shutdown detected booted=2022-12-04T13:05:53+0000 age=36s
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Unclean shutdown detected booted=2022-12-04T13:06:05+0000 age=24s
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.938] Unclean shutdown detected booted=2022-12-04T13:06:17+0000 age=12s
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:29.938] Allocated cache and file handles database=/root/.ethereum/geth/les.server cache=16.00MiB handles=16
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:29.947] Configured checkpoint oracle address=0x9a9070028361F7AAbeB3f2F2Dc07F82C4a98A02a signers=5 threshold=2
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:29.948] Loaded latest checkpoint section=490 head=96c7bd..ab961d chtroot=532f7c..e64252 bloomroot=07d2a8..68e346
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:29.948] Engine API enabled protocol=eth
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:29.948] Starting peer-to-peer node instance=Geth/v1.11.0-unstable-722bb210-20221117/linux-amd64/go1.19.3
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:29.964] Mapped network port proto=tcp extport=30303 intport=30303 interface=ExtIP(191.234.162.198)
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:29.964] Mapped network port proto=udp extport=30303 intport=30303 interface=ExtIP(191.234.162.198)
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:30.062] New local node record seq=3041 id=32f7e89d30e83b68 ip=191.234.162.198 udp=30303 tcp=30303
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:30.203] Started P2P networking self=enode://103858bdb88756c71f15e9b5e09b56dc1be52f0a5021d46301dbbfb7e130029cc9d0d6f73f693bc29b665770fff7da4d34f3c6379fe12721b5d7a0bcb5ca1fc1@191.234.162.198:30303
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:30.205] IPC endpoint opened url=/root/.ethereum/geth.ipc
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:30.205] Loaded JWT secret file path=/execution-auth.jwt crc32=0xba247cd0
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:30.205] WebSocket enabled url=ws://[::]:8551
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:30.205] HTTP server started endpoint=[::]:8551 auth=true prefix= cors=localhost vhosts=*
Dec 04 14:06:30 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:30.206] Stats daemon started
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth WARN [12-04|13:06:36.807] Ignoring payload with missing parent number=16,111,632 hash=e9c1fd..f8ab50 parent=fe62f5..69bae3
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:36.924] Forkchoice requested sync to new head number=16,111,632 hash=e9c1fd..f8ab50
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth INFO [12-04|13:06:36.926] Block synchronisation started
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth panic: runtime error: invalid memory address or nil pointer dereference
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth [signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xad3ac8]
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth goroutine 6345 [running]:
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findBeaconAncestor(0xc000122c00)
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:220 +0xe8
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc000122c00, 0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth github.com/ethereum/go-ethereum/eth/downloader/downloader.go:527 +0x911
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0x3?, {0x0, 0x0}, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth github.com/ethereum/go-ethereum/eth/downloader/downloader.go:439 +0x512
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth github.com/ethereum/go-ethereum/eth/downloader.(*beaconBackfiller).resume.func1()
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:108 +0x96
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth created by github.com/ethereum/go-ethereum/eth/downloader.(*beaconBackfiller).resume
Dec 04 14:06:37 bootnode-azure-brazilsouth-001 geth github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:98 +0x136

[holiman]

I've stopped geth now, to avoid the panic-cycle. cc @skylenet

[holiman]

Duplicate of #25787

[holiman]

Old unclean shutdowns found count=1593 👀

[holiman]

This is where it crashes:

	var linked bool
	switch d.getMode() {
	case FullSync:
		linked = d.blockchain.HasBlock(beaconTail.ParentHash, beaconTail.Number.Uint64()-1) //<--- THIS LINE
	case SnapSync:
		linked = d.blockchain.HasFastBlock(beaconTail.ParentHash, beaconTail.Number.Uint64()-1)

Which means that our beaconTail is nil. It comes from here:

	// Retrieve the skeleton bounds and ensure they are linked to the local chain
	beaconHead, beaconTail, err := d.skeleton.Bounds()

func (s *skeleton) Bounds() (head *types.Header, tail *types.Header, err error) {
	// Read the current sync progress from disk and figure out the current head.
	// Although there's a lot of error handling here, these are mostly as sanity
	// checks to avoid crashing if a programming error happens. These should not
	// happen in live code.
	status := rawdb.ReadSkeletonSyncStatus(s.db)
	if len(status) == 0 {
		return nil, nil, errors.New("beacon sync not yet started")
	}
	progress := new(skeletonProgress)
	if err := json.Unmarshal(status, progress); err != nil {
		return nil, nil, err
	}
	head = rawdb.ReadSkeletonHeader(s.db, progress.Subchains[0].Head)
	tail = rawdb.ReadSkeletonHeader(s.db, progress.Subchains[0].Tail)

	return head, tail, nil

We should check this on the live node (example from my laptop):

$ geth db metadata 2>&1 | grep Skeleton
| SkeletonSyncStatus      | {"Subchains":[{"Head":15816882,"Tail":15576754,"Next":"0xcd98dd5656dfcad37e6123d56e9a17f6eade505c902e8e6ac33679ea497656ee"}]} |

[holiman]

$ docker run -it -v "/geth/.ethereum":"/root/.ethereum" ethereum/client-go --datadir=/root/.ethereum db metadata 
INFO [12-05|07:11:18.086] Maximum peer count                       ETH=50 LES=0 total=50
INFO [12-05|07:11:18.086] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [12-05|07:11:18.088] Set global gas cap                       cap=50,000,000
INFO [12-05|07:11:18.088] Allocated cache and file handles         database=/root/.ethereum/geth/chaindata cache=512.00MiB handles=524,288 readonly=true
INFO [12-05|07:11:19.123] Found legacy ancient chain path          location=/root/.ethereum/geth/chaindata/ancient
INFO [12-05|07:11:19.185] Opened ancient database                  database=/root/.ethereum/geth/chaindata/ancient readonly=true
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------+
|          FIELD          |                                                             VALUE                                                             |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------+
| databaseVersion         | 8 (0x8)                                                                                                                       |
| headBlockHash           | 0x41996543523a59a4d4c14ba991f2604c9320adc1a889eadac5ec4d28a5e370cd                                                            |
| headFastBlockHash       | 0x0946fa1055b221632c128ec795aeb0b8326762a636da45dc1084f8aa9646f44b                                                            |
| headHeaderHash          | 0x0946fa1055b221632c128ec795aeb0b8326762a636da45dc1084f8aa9646f44b                                                            |
| headBlock.Hash          | 0x41996543523a59a4d4c14ba991f2604c9320adc1a889eadac5ec4d28a5e370cd                                                            |
| headBlock.Root          | 0xe459a1d4eb3ba327b9dc8b4eabedd86f5c0486d6192cfccec25dc9a56b23facd                                                            |
| headBlock.Number        | 16109599 (0xf5d01f)                                                                                                           |
| SkeletonSyncStatus      | {"Subchains":[{"Head":16111652,"Tail":16111325,"Next":"0x5cea4e7a3419cb3ac7fa51ca087576b952b7d6409c2850cbb0c3878433ecd1d0"}]} |
| headHeader.Hash         | 0x0946fa1055b221632c128ec795aeb0b8326762a636da45dc1084f8aa9646f44b                                                            |
| headHeader.Root         | 0x238e234bb5a02b0b57ae7053e9b5f6066c5d835a7908ce01552d94b4fae9878d                                                            |
| headHeader.Number       | 16111325 (0xf5d6dd)                                                                                                           |
| frozen                  | 15606389 items                                                                                                                |
| lastPivotNumber         | 16045661 (0xf4d65d)                                                                                                           |
| len(snapshotSyncStatus) | 288 bytes                                                                                                                     |
| snapshotGenerator       | Done: true, Accounts: 0,                                                                                                      |
|                         | Slots: 0, Storage: 0, Marker:                                                                                                 |
| snapshotDisabled        | false                                                                                                                         |
| snapshotJournal         | 0 bytes                                                                                                                       |
| snapshotRecoveryNumber  | 16111137 (0xf5d621)                                                                                                           |
| snapshotRoot            | 0x9ac2ed8d99598b6b5cecf55b324494172602d95bc141fad26490d9135116d504                                                            |
| txIndexTail             | 13761326 (0xd1fb2e)                                                                                                           |
| fastTxLookupLimit       | 2350000 (0x23dbb0)                                                                                                            |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------+

[holiman]

The Head-key (via skeletonHeaderKey(16111652) does resolve:

key 0x530000000000f5d824: 0xf90215a0270de9f6a4bd113c40262329c55849625b2ea6960566b2b9f41773759014ecf5a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d4934794f2f5c73fa04406b1995e397b55c24ab1f3ea726ca0686d6bd0ee2d072b8efd315501a1632d07bec632894005faa2eb1378d561389ea063c9ad77383d23220114e7c53595231b5fd1338cfc526c8bd4b038acdbc755a0a0848afccd27b31968124d872c7999f5d5d74a5746476703ef1969413dc69579c0b90100fc6b08e20b82d289c6122867e8107b6551b60861891c15ca3e85a230f066e1246f062760c30e898c5734402b1e3dc5ec06c5931c0e5c6dc22e72154018a4304974c0280a1da4489cfae98e2a815050ea8560a802464536f29e1a100c8b8e1302522c040013030e65055399902850181e405641d0787cac110609055b4c1f4ca0529700bd46a20315117c8408c920200dd942c41b430054dacb8131c0409c1a2fafe711076c0c68307692c1a600103dc55e5692335886b40336dc0caf41a55d454fb58c0a0fc3188d01351d4126b69d1251aa13c41d0a0b166420bd0b16106407c892a80d8c6a19a9820f942c62028c3013538028199a035ec8129c43d846200b8083f5d8248401c9c38083ad93ac84638c9c4b94506f776572656420627920626c6f58726f757465a07063bd2b16805627c72b586156d30998d8a393ec3f4c3d5858b485715a877f31880000000000000000850348b10f67

But the Tail key (skeletonHeaderKey(16111325))does not resolve to a header:

INFO [12-05|07:42:34.585] Get operation failed                     key=0x530000000000f5d6dd error="leveldb: not found"
leveldb: not found

[holiman]

This is the very first of the panics, which did not happen in conjuction with a reboot (slightly cleaned) :

INFO [12-04|12:04:00.172] Forkchoice requested sync to new head number=16,111,322 hash=d0c18d..8b964d
INFO [12-04|12:04:01.100] Imported new chain segment number=16,111,306 hash=95fa2a..39901e blocks=5 txs=704 mgas=70.796 elapsed=13.613s mgasps=5.200 age=3m14s dirty=993.03MiB
INFO [12-04|12:04:03.184] Imported new chain segment number=16,111,307 hash=af366d..bd916b blocks=1 txs=267 mgas=22.671 elapsed=2.084s mgasps=10.876 age=3m4s dirty=994.42MiB
INFO [12-04|12:04:03.185] Downloader queue stats receiptTasks=0 blockTasks=0 itemSize=79.14KiB throttle=3313
INFO [12-04|12:04:24.338] Forkchoice requested sync to new head number=16,111,324 hash=5cea4e..ecd1d0
ERROR[12-04|12:04:26.120] Error in block freeze operation err="block receipts missing, can't freeze block 15606390"
INFO [12-04|12:04:27.393] Imported new chain segment number=16,111,312 hash=c6ebac..c1c3ab blocks=5 txs=948 mgas=69.596 elapsed=14.120s mgasps=4.929 age=2m28s dirty=995.57MiB ignored=1
INFO [12-04|12:04:35.997] Forkchoice requested sync to new head number=16,111,325 hash=7d3760..38cac6
INFO [12-04|12:04:47.802] Imported new chain segment number=16,111,321 hash=272ca0..25cf33 blocks=9 txs=1217 mgas=142.953 elapsed=20.409s mgasps=7.004 age=1m dirty=997.49MiB
INFO [12-04|12:04:48.731] Imported new chain segment number=16,111,323 hash=88ec6d..20dd8f blocks=2 txs=240 mgas=18.027 elapsed=928.874ms mgasps=19.408 dirty=996.39MiB
INFO [12-04|12:04:54.932] Imported new chain segment number=16,111,325 hash=0946fa..46f44b blocks=2 txs=253 mgas=7.600 elapsed=130.270ms mgasps=58.340 dirty=995.09MiB ignored=1
WARN [12-04|12:05:02.230] Ignoring payload with missing parent number=16,111,326 hash=3a0143..2678a4 parent=7d3760..38cac6
WARN [12-04|12:05:13.531] Ignoring payload with missing parent number=16,111,327 hash=e97873..822c73 parent=3a0143..2678a4
INFO [12-04|12:05:15.114] Forkchoice requested sync to new head number=16,111,327 hash=e97873..822c73
WARN [12-04|12:05:15.114] Beacon chain gapped head=16,111,325 newHead=16,111,327
INFO [12-04|12:05:15.256] Syncing beacon headers downloaded=90 left=0 eta=0s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xad6fdb]
goroutine 1555316 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc000041500, 0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
github.com/ethereum/go-ethereum/eth/downloader/downloader.go:491 +0x49b
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0xc168424d80?, {0x0, 0x0}, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
github.com/ethereum/go-ethereum/eth/downloader/downloader.go:439 +0x512
github.com/ethereum/go-ethereum/eth/downloader.(*beaconBackfiller).resume.func1()
github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:108 +0x96
created by github.com/ethereum/go-ethereum/eth/downloader.(*beaconBackfiller).resume
github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:98 +0x136

[holiman]

INFO [12-04|12:04:54.932] Imported new chain segment number=16,111,325 hash=0946fa..46f44b blocks=2 txs=253 mgas=7.600 elapsed=130.270ms mgasps=58.340 dirty=995.09MiB ignored=1
^ That number is the tail

[holiman]

Chain progression: we have 325:0946fa, but we do not have 325:7d3760. We later detect
the gap, and try to sync to 327, which is a descendant of (the missing) 325:7d3760.

325: `0946fa`     `7d3760`   // Imported number=16,111,325 hash=`0946fa`
326:              `3a0143`   // Ignoring payload with missing parent number
327:              `e97873`   // Ignoring payload with missing parent number

Could it be that our tail is 325@0946fa, but later on when we detect the that 325@0946fa is
reorged, we remove it but do not update the tail-marker?

{
  "Subchains": [
    {
      "Head": 16111652,
      "Tail": 16111325,
      "Next": "0x5cea4e7a3419cb3ac7fa51ca087576b952b7d6409c2850cbb0c3878433ecd1d0"
    }
  ]
}

cc @karalabe

[HamidullahMuslih]

Any update on this please, my Goerli node is having the same issue.

Dec 09 16:09:48 ip-10-0-29-185 geth[478486]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xa9bbe8]
Dec 09 16:09:48 ip-10-0-29-185 geth[478486]: goroutine 98175 [running]:
Dec 09 16:09:48 ip-10-0-29-185 geth[478486]: github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findBeaconAncestor(0xc000170780)
Dec 09 16:09:48 ip-10-0-29-185 geth[478486]:         github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:220 +0xe8
Dec 09 16:09:48 ip-10-0-29-185 geth[478486]: github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc000170780, 0x0, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
Dec 09 16:09:48 ip-10-0-29-185 geth[478486]:         github.com/ethereum/go-ethereum/eth/downloader/downloader.go:527 +0x911
Dec 09 16:09:48 ip-10-0-29-185 geth[478486]: github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0xc06712c500?, {0x0, 0x0}, {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
Dec 09 16:09:48 ip-10-0-29-185 geth[478486]:         github.com/ethereum/go-ethereum/eth/downloader/downloader.go:439 +0x505
Dec 09 16:09:48 ip-10-0-29-185 geth[478486]: github.com/ethereum/go-ethereum/eth/downloader.(*beaconBackfiller).resume.func1()
Dec 09 16:09:48 ip-10-0-29-185 geth[478486]:         github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:108 +0x96
Dec 09 16:09:48 ip-10-0-29-185 geth[478486]: created by github.com/ethereum/go-ethereum/eth/downloader.(*beaconBackfiller).resume
Dec 09 16:09:48 ip-10-0-29-185 geth[478486]:         github.com/ethereum/go-ethereum/eth/downloader/beaconsync.go:98 +0x132
Dec 09 16:09:48 ip-10-0-29-185 systemd[1]: geth.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

[HamidullahMuslih]

db metadata

 geth --datadir=/mnt/data/.goerli/  db metadata
INFO [12-09|17:27:07.076] Maximum peer count                       ETH=50 LES=0 total=50
INFO [12-09|17:27:07.076] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [12-09|17:27:07.078] Set global gas cap                       cap=50,000,000
INFO [12-09|17:27:07.078] Allocated cache and file handles         database=/mnt/data/.goerli/geth/chaindata cache=512.00MiB handles=524,288 readonly=true
INFO [12-09|17:27:10.309] Opened ancient database                  database=/mnt/data/.goerli/geth/chaindata/ancient/chain readonly=true
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------+
|          FIELD          |                                                            VALUE                                                            |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| databaseVersion         | 8 (0x8)                                                                                                                     |
| headBlockHash           | 0xf2e9595bb87d8b03d82c54950d61afc97690aa183d376284b3c6a991ffb1c536                                                          |
| headFastBlockHash       | 0xf2e9595bb87d8b03d82c54950d61afc97690aa183d376284b3c6a991ffb1c536                                                          |
| headHeaderHash          | 0xf2e9595bb87d8b03d82c54950d61afc97690aa183d376284b3c6a991ffb1c536                                                          |
| headBlock.Hash          | 0xf2e9595bb87d8b03d82c54950d61afc97690aa183d376284b3c6a991ffb1c536                                                          |
| headBlock.Root          | 0x7674ddee35e611c22206badf3b7414138e8862cb074c2dc44bb73d4aba6754f4                                                          |
| headBlock.Number        | 8102428 (0x7ba21c)                                                                                                          |
| SkeletonSyncStatus      | {"Subchains":[{"Head":8104794,"Tail":8102428,"Next":"0xa4374b7006c70b5951e5f9fccbfbc41a3df3eb198bf8a888de15ac86fcf729c2"}]} |
| headHeader.Hash         | 0xf2e9595bb87d8b03d82c54950d61afc97690aa183d376284b3c6a991ffb1c536                                                          |
| headHeader.Root         | 0x7674ddee35e611c22206badf3b7414138e8862cb074c2dc44bb73d4aba6754f4                                                          |
| headHeader.Number       | 8102428 (0x7ba21c)                                                                                                          |
| frozen                  | 8012429 items                                                                                                               |
| lastPivotNumber         | <nil>                                                                                                                       |
| len(snapshotSyncStatus) | 0 bytes                                                                                                                     |
| snapshotGenerator       | Done: false, Accounts: 88972, Slots:                                                                                        |
|                         | 321822, Storage: 31162808, Marker: at                                                                                       |
|                         | 0x014e71ecc21b7124b3c4eeeb25dc8370a2eb508a53431c7c277ad4e84aaaa310                                                          |
| snapshotDisabled        | false                                                                                                                       |
| snapshotJournal         | 34 bytes                                                                                                                    |
| snapshotRecoveryNumber  | <nil>                                                                                                                       |
| snapshotRoot            | 0x7674ddee35e611c22206badf3b7414138e8862cb074c2dc44bb73d4aba6754f4                                                          |
| txIndexTail             | 0 (0x0)                                                                                                                     |
| fastTxLookupLimit       | <nil>                                                                                                                       |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------+```