History
Showing
with
312 additions
and 13 deletions.
- +122 −0 2025-07-02-Testnet-Rollback-and-Restart.md
- +119 −0 2025-07-03-Devnet-rollback-and-restart.md
- +3 −1 Feature-Gate-Tracker-Schedule.md
- +1 −1 _Sidebar.md
- +67 −11 feature-gate-tracker-schedule.json
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| ## Edit | ||
| As of 2025-07-03 16:00 UTC testnet is back online. The instructions below are no longer relevant. Nodes that haven't yet joined the cluster will need to update their shred version, download a snapshot, and start normally: | ||
|
|
||
| --expected-shred-version 9065 \ | ||
|
|
||
| *** | ||
| This testnet restart is NOT urgent. Follow these instructions when you have time, but don’t skip sleep or disrupt other plans for this. | ||
|
|
||
| ## Summary | ||
| |Attribute|Value| | ||
| |---------|-----| | ||
| |Validator version|Agave: v2.3.2 </br> Frankendancer: v0.603.20216| | ||
| |Snapshot slot|343175552| | ||
| |Restart slot|343175553| | ||
| |Shred version|9065| | ||
| |Expected bank hash|4oMrSXsLTiCc1X7S27kxSfGVraTCZoZ7YTy2skEB9bPk| | ||
|
|
||
|
|
||
| ## Step 1. Stop validator process if you haven’t already | ||
|
|
||
| ## Step 2: Install Latest Version | ||
| This is necessary in order to create the correct snapshot in step 3. | ||
|
|
||
| Agave: `agave-install init v2.3.2` | ||
|
|
||
| Frankendancer: Install `v0.603.20216` | ||
|
|
||
| ## Step 3. Create snapshot | ||
| This command creates a snapshot but removes 4 activated feature gate account. | ||
|
|
||
| agave-ledger-tool --ledger <ledger-path> create-snapshot \ | ||
| --incremental \ | ||
| --snapshot-archive-path <snapshot-path> \ | ||
| --hard-fork 343175552 \ | ||
| --deactivate-feature-gate \ | ||
| 2aQJYqER2aKyb3cZw22v4SL2xMX7vwXBRWfvS4pTrtED \ | ||
| LTsNAP8h1voEVVToMNBNqoiNQex4aqfUrbFhRH3mSQ2 \ | ||
| 5KLGJSASDVxKPjLCDWNtnABLpZjsQSrYZ8HKwcEdAMC8 \ | ||
| 6oMCUgfY6BzZ6jwB681J6ju5Bh6CjVXbd7NeWYqiXBSu \ | ||
| --enable-capitalization-change \ | ||
| -- 343175552 <snapshot-path> | ||
|
|
||
|
|
||
| The output should include this at (or near) the end: | ||
| ``` | ||
| Successfully created snapshot for slot 343175553, hash 4oMrSXsLTiCc1X7S27kxSfGVraTCZoZ7YTy2skEB9bPk, base slot: <BASE_SLOT>>: /home/sol/ledger-snapshots/incremental-snapshot-<BASE_SLOT>-343175553-<SNAPSHOT_HASH>.tar.zst | ||
| Capitalization change: -3814080 lamports | ||
| Shred version: 9065 | ||
| ``` | ||
|
|
||
| The capitalization change is expected because we deactivated a feature gate. | ||
|
|
||
| Note that each operator's snapshot file name may contain different base slot number and hash, but | ||
| * the bank hash should be 4oMrSXsLTiCc1X7S27kxSfGVraTCZoZ7YTy2skEB9bPk | ||
| * the second slot number should be 343175553 | ||
| * the shred version should be 9065 | ||
|
|
||
| Once you have created a snapshot move all the other snapshots to a backup directory, so your snapshot directory contains one full snapshot and one incremental snapshot. Note that the <BASE_SLOT> in these two filenames should match. | ||
|
|
||
| snapshot-<BASE_SLOT>-<BASE_SNAPSHOT_HASH>.tar.zst | ||
| incremental-snapshot-<BASE_SLOT>-343175553-<SNAPSHOT_HASH>.tar.zst | ||
|
|
||
| If you fail to create a snapshot see the appendix for possible fixes. | ||
|
|
||
| ## Step 5: Update startup config and start your validator | ||
| ### Agave | ||
| Add these arguments to your validator startup script: | ||
|
|
||
| --wait-for-supermajority 343175553 \ | ||
| --expected-shred-version 9065 \ | ||
| --expected-bank-hash 4oMrSXsLTiCc1X7S27kxSfGVraTCZoZ7YTy2skEB9bPk \ | ||
|
|
||
|
|
||
| As it starts, the validator will load the snapshot for slot `343175553` and wait for 80% of the stake to come online before producing/validating new blocks. | ||
|
|
||
| To confirm your restarted validator is correctly waiting for 80% stake, look for this periodic log message to confirm it is waiting: | ||
|
|
||
| INFO solana_core::validator] Waiting for 80% of activated stake at slot 343175553 to be in gossip... | ||
|
|
||
| And if you have RPC enabled, ask it for the current slot: | ||
|
|
||
| solana --url http://127.0.0.1:8899 slot | ||
|
|
||
| Any number other than `343175553` means you did not complete the steps correctly. | ||
|
|
||
| Once started you should see log entries for “active stake” visible in gossip and “waiting for 80% of stake” to be visible. You can track these to see how the stake progresses. | ||
|
|
||
|
|
||
| *** | ||
|
|
||
| ## Appendix (use this only if step 3 failed) | ||
|
|
||
| If you get an error like this: | ||
|
|
||
| Error: Slot 343175552 is not available | ||
|
|
||
| Or this: | ||
|
|
||
| Unable to process blockstore from starting slot <slot> to 343175552; the ending slot is less than the starting slot. The starting slot will be the latest snapshot slot, or genesis if the --no-snapshot flag is specified or if no snapshots are found. | ||
|
|
||
| Your snapshots directory contains a snapshot that is for a slot `>343175552`. If you also have a snapshot for slot `<=343175552` then move snapshots for slots `>343175552` to a backup directory and run the `agave-ledger-tool` command again. If you do not have a snapshot for slot `<=343175552` then you will need to download a snapshot | ||
|
|
||
| If you successfully created a snapshot, resume the instructions above starting at Step 4. If you are unable to create a snapshot, follow the instructions below on downloading a snapshot. | ||
|
|
||
| If you couldn’t produce your snapshot locally follow these appendix steps | ||
|
|
||
| ### Step 1: Download a snapshot from a known validator | ||
|
|
||
| If you are unable to generate a snapshot locally for slot `343175553` you will need to download one from a known validator. Add these lines to your startup script. | ||
|
|
||
| --known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \ | ||
| --expected-shred-version 9065 \ | ||
|
|
||
| Remove the flag `--no-snapshot-fetch` in your startup script if it is present. | ||
|
|
||
| ### Step 2: After download, restart | ||
|
|
||
| Verify that you have a new snapshot in your snapshot directory. If the snapshot is done downloading, stop your validator process. | ||
|
|
||
| Add the flag `--no-snapshot-fetch` to your startup script | ||
|
|
||
| Resume the instructions above starting at Step 4. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| ## Edit | ||
| As of 2025-07-03 16:10 UTC devnet is back online. The instructions below are no longer relevant. Nodes that haven't yet joined the cluster will need to update their shred version, download a snapshot, and start normally: | ||
|
|
||
| --expected-shred-version 29062 \ | ||
|
|
||
| *** | ||
| This devnet restart is NOT urgent. Follow these instructions when you have time, but don’t skip sleep or disrupt other plans for this. | ||
|
|
||
| ## Summary | ||
| |Attribute|Value| | ||
| |---------|-----| | ||
| |Validator version|Agave: v2.3.2 | | ||
| |Snapshot slot|391843993| | ||
| |Restart slot|391843994| | ||
| |Shred version|29062| | ||
| |Expected bank hash|TvMCp3TJJTQmpYyXh5E36wLkH9DqpmKZqXy882S8PnK| | ||
|
|
||
|
|
||
| ## Step 1. Stop validator process | ||
|
|
||
| ## Step 2: Install the Latest Version | ||
|
|
||
| This is necessary in order to create the correct snapshot in step 3. | ||
|
|
||
| Agave: `agave-install init v2.3.2` | ||
|
|
||
| ## Step 3. Create snapshot | ||
| This command creates a snapshot but removes 2 activated feature gate account. | ||
|
|
||
| agave-ledger-tool --ledger <ledger-path> create-snapshot \ | ||
| --incremental \ | ||
| --snapshot-archive-path <snapshot-path> \ | ||
| --hard-fork <snapshot-slot> \ | ||
| --deactivate-feature-gate \ | ||
| 2aQJYqER2aKyb3cZw22v4SL2xMX7vwXBRWfvS4pTrtED \ | ||
| LTsNAP8h1voEVVToMNBNqoiNQex4aqfUrbFhRH3mSQ2 \ | ||
| --enable-capitalization-change \ | ||
| -- <snapshot-slot> <snapshot-path> | ||
|
|
||
|
|
||
| The output should include this at (or near) the end: | ||
| ``` | ||
| Successfully created snapshot for slot 391843994, hash TvMCp3TJJTQmpYyXh5E36wLkH9DqpmKZqXy882S8PnK, base slot: <BASE_SLOT>>: /home/sol/ledger-snapshots/incremental-snapshot-<BASE_SLOT>-391843994-<SNAPSHOT_HASH>.tar.zst | ||
| Capitalization change: -1907040 lamports | ||
| Shred version: 29062 | ||
| ``` | ||
|
|
||
| The capitalization change is expected because we deactivated a feature gate. | ||
|
|
||
| Note that each operator's snapshot file name may contain different base slot number and hash, but | ||
| * the bank hash should be TvMCp3TJJTQmpYyXh5E36wLkH9DqpmKZqXy882S8PnK | ||
| * the second slot number should be 391843994 | ||
| * the shred version should be 29062 | ||
|
|
||
| Once you have created a snapshot move all the other snapshots to a backup directory, so your snapshot directory contains one full snapshot and one incremental snapshot. Note that the <BASE_SLOT> in these two filenames should match. | ||
|
|
||
| snapshot-<BASE_SLOT>-<BASE_SNAPSHOT_HASH>.tar.zst | ||
| incremental-snapshot-<BASE_SLOT>-391843994-<SNAPSHOT_HASH>.tar.zst | ||
|
|
||
| If you fail to create a snapshot see the appendix for possible fixes. | ||
|
|
||
| ## Step 4: Update startup config and start your validator | ||
| ### Agave | ||
| Add these arguments to your validator startup script: | ||
|
|
||
| --wait-for-supermajority 391843994 \ | ||
| --expected-shred-version 29062 \ | ||
| --expected-bank-hash TvMCp3TJJTQmpYyXh5E36wLkH9DqpmKZqXy882S8PnK \ | ||
|
|
||
|
|
||
| As it starts, the validator will load the snapshot for slot `391843994` and wait for 80% of the stake to come online before producing/validating new blocks. | ||
|
|
||
| To confirm your restarted validator is correctly waiting for 80% stake, look for this periodic log message to confirm it is waiting: | ||
|
|
||
| INFO solana_core::validator] Waiting for 80% of activated stake at slot 391843994 to be in gossip... | ||
|
|
||
| And if you have RPC enabled, ask it for the current slot: | ||
|
|
||
| solana --url http://127.0.0.1:8899 slot | ||
|
|
||
| Any number other than `391843994` means you did not complete the steps correctly. | ||
|
|
||
| Once started you should see log entries for “active stake” visible in gossip and “waiting for 80% of stake” to be visible. You can track these to see how the stake progresses. | ||
|
|
||
|
|
||
| *** | ||
|
|
||
| ## Appendix (use this only if step 3 failed) | ||
|
|
||
| If you get an error like this: | ||
|
|
||
| Error: Slot <SNAPSHOT_SLOT> is not available | ||
|
|
||
| Or this: | ||
|
|
||
| Unable to process blockstore from starting slot <slot> to 391843993; the ending slot is less than the starting slot. The starting slot will be the latest snapshot slot, or genesis if the --no-snapshot flag is specified or if no snapshots are found. | ||
|
|
||
| Your snapshots directory contains a snapshot that is for a slot `>391843993`. If you also have a snapshot for slot `<=391843993` then move snapshots for slots `>391843993` to a backup directory and run the `agave-ledger-tool` command again. If you do not have a snapshot for slot `<=391843993` then you will need to download a snapshot | ||
|
|
||
| If you successfully created a snapshot, resume the instructions above starting at Step 4. If you are unable to create a snapshot, follow the instructions below on downloading a snapshot. | ||
|
|
||
| If you couldn’t produce your snapshot locally follow these appendix steps | ||
|
|
||
| ### Step 1: Download a snapshot from a known validator | ||
|
|
||
| If you are unable to generate a snapshot locally for slot `391843994` you will need to download one from a known validator. Add these lines to your startup script. | ||
|
|
||
| --known-validator dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 \ | ||
| --expected-shred-version 29062 \ | ||
|
|
||
| Remove the flag `--no-snapshot-fetch` in your startup script if it is present. | ||
|
|
||
| ### Step 2: After download, restart | ||
|
|
||
| Verify that you have a new snapshot in your snapshot directory. If the snapshot is done downloading, stop your validator process. | ||
|
|
||
| Add the flag `--no-snapshot-fetch` to your startup script | ||
|
|
||
| Resume the instructions above starting at Step 4. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters