Full node sync error - Masternodes lists are different in checkpoint header and snapshot number=65106900 masternodes_from_checkpoint_header

[lucksin3]

Attached are the logs from the node, the node has stopped sync'n with below error and it restarts the node every few seconds:

INFO [09-11|10:48:53] Persisted trie from memory database nodes=0 size=0.00B time=5.564µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [09-11|10:48:53] Persisted trie from memory database nodes=0 size=0.00B time=2.853µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [09-11|10:48:53] Imported new chain segment blocks=2 txs=5 mgas=12.878 elapsed=215.684ms mgasps=59.707 number=65106749 hash=083e15…7fdeb7 cache=61.75kB
INFO [09-11|10:48:54] Imported new chain segment blocks=3 txs=110 mgas=18.801 elapsed=234.442ms mgasps=80.195 number=65106752 hash=31247c…c194b4 cache=304.17kB
INFO [09-11|10:48:54] Imported new chain segment blocks=11 txs=24 mgas=69.273 elapsed=681.495ms mgasps=101.649 number=65106763 hash=42cafc…5c7abd cache=416.37kB
INFO [09-11|10:48:56] Imported new chain segment blocks=34 txs=496 mgas=213.822 elapsed=2.099s mgasps=101.859 number=65106797 hash=4454ea…726eb5 cache=1.37mB
WARN [09-11|10:48:58] Full stats report failed err="ping timed out"
WARN [09-11|10:48:58] Failed to decode stats server message err="read tcp 172.26.0.2:58158->66.94.120.128:3000: use of closed network connection"
INFO [09-11|10:49:03] Persisted trie from memory database nodes=0 size=0.00B time=757.763µs gcnodes=11 gcsize=4.64kB gctime=59.891µs livenodes=7013 livesize=2.51mB
INFO [09-11|10:49:03] Persisted trie from memory database nodes=0 size=0.00B time=5.432µs gcnodes=0 gcsize=0.00B gctime=1.942µs livenodes=1 livesize=0.00B
INFO [09-11|10:49:03] Persisted trie from memory database nodes=0 size=0.00B time=3.783µs gcnodes=0 gcsize=0.00B gctime=1.678µs livenodes=1 livesize=0.00B
INFO [09-11|10:49:05] Imported new chain segment blocks=99 txs=1047 mgas=627.424 elapsed=6.122s mgasps=102.482 number=65106896 hash=611a6c…bdfbd0 cache=3.17mB
INFO [09-11|10:49:05] Imported new chain segment blocks=3 txs=6 mgas=18.803 elapsed=168.030ms mgasps=111.901 number=65106899 hash=11833a…a27901 cache=3.19mB
ERROR[09-11|10:49:05] Masternodes lists are different in checkpoint header and snapshot number=65106900 masternodes_from_checkpoint_header="[[1 68 40 230 15 170 135 79 130 102 208 100 140 82 168 53 6 221 158 198] [1 223 95 73 134 151 110 140 241 89 44 202 123 121 38 100 81 11 96 134] [3 217 225 122 227 255 44 103 18 228 78 37 176 154 197 238 145 246 201 255] [4 127 254 31 199 246 208 183 22 140 76 204 49 34 33 8 150 41 244 112] [6 85 81 240 220 172 111 0 202 225 17 146 212 98 219 112 155 227 117 140] [7 14 237 10 44 119 101 13 212 21 153 146 11 90 0 137 220 96 191 230] [11 10 251 65 193 77 217 33 169 164 81 28 34 109 194 146 84 186 13 143] [24 211 21 17 184 239 191 66 21 196 140 2 108 86 173 70 156 233 3 116] [26 124 61 116 129 173 199 224 103 242 247 63 53 244 65 67 215 191 2 212] [26 255 23 28 170 140 30 169 59 172 76 39 103 111 53 109 5 68 18 180] [29 37 167 7 105 147 207 205 103 98 55 218 234 144 216 237 201 156 127 70] [29 57
node-logs-invalid-masternode-AWS.txt

[gzliudan]

Which golang version are you used to build XDC ?

[gzliudan]

I synced from snapshot:

grep Imported xinfin-snapshot-fix-issue-268.debug-20230920-015836.log | sed -ne '1p; $p'

INFO [09-20|01:59:01] Imported new chain segment               blocks=2 txs=103 mgas=5.974 elapsed=230.553ms mgasps=25.911 number=64524978 hash=432349…493448 cache=0.00B
INFO [09-20|22:16:02] Imported new chain segment               blocks=92  txs=823  mgas=639.273 elapsed=8.051s    mgasps=79.400  number=65296174 hash=702e1d…4de34d cache=0.00B

This log file started from block number 64524978, passed the block number 65106900 which you mentioned. I use golang v1.14.15 to build binary programs. I guess this issue was caused by golang version. But I found your golang version is v1.14.15 from your log file:

INFO [09-11|10:40:47] Starting peer-to-peer node               instance=XDC/v1.4.6-stable/linux-amd64/go1.14.15

[gzliudan]

My sync test caught up with the latest bocks now. I guess your golang version is mismatch.

[gzliudan]

Your node passed block number 65106900 now:

[gzliudan]

According to my test results, we can use golang v1.14.15-v1.18.10 for sync purpose. Golang versions v1.19-v1.21 are not supported now.

[lucksin3]

issues fixed node sync'g fine post the fix