From 6e45206128e593d10a09d6356b0e08935766bfae Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Thu, 15 Aug 2024 15:11:37 +0200 Subject: [PATCH 01/17] First draft --- proposals/0167-loader-v4.md | 296 ++++++++++++++++++++++++++++++++++++ 1 file changed, 296 insertions(+) create mode 100644 proposals/0167-loader-v4.md diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md new file mode 100644 index 000000000..0ce6523d1 --- /dev/null +++ b/proposals/0167-loader-v4.md @@ -0,0 +1,296 @@ +--- +simd: '0167' +title: Loader-v4 +authors: + - Alexander Meißner +category: Standard +type: Core +status: Draft +created: 2024-08-15 +feature: TBD +--- + +## Summary + +A new upgradeable loader which only requires a single account per program. + +## Motivation + +Loader-v3, which is currently the only deployable loader, requires two accounts +per program. This was a workaround to circumvent the finality of the +`is_executable` flag, which is removed in SIMD-0162. Consequentially, this +setup of the program account being a proxy account, containing the address of +the actual programdata account, is no longer necessary and should be removed. + +Another issue with loader-v3 is that the executable file stored in the +programdata account is misaligned relative to the beginning of the account. + +Additionally, there currently is no complete specification of the loaders +program management instructions. This proposal would thus fill that gap once +loader-v4 goes into production. + +See impact for further motivation. + +## Alternatives Considered + +A delay-visibility-free redeployment could be achieved by keeping the swap +program around until the end of the slot. This would however mean that two +accounts per program must be loaded until the dapp developer reclaims the +second one. That would defy the purpose of this proposal which is to get rid +of the proxy account. + +## New Terminology + +None. + +## Detailed Design + +`LoaderV411111111111111111111111111111111111` program management and execution +must be enabled with the associated feature gate, which simultaneously disables +new deployments on loader-v3 (`BPFLoaderUpgradeab1e11111111111111111111111`), +throwing `InvalidIstructionData` if `DeployWithMaxDataLen` is called. + +### Owned Program Accounts + +Accounts of programs owned by loader-v4 must have the following layout: + +- Header (which is 48 bytes long): + - `u64` Slot in which the program was last deployed, retracted or + initialized. + - `[u8; 32]` Authority address which can send program management + instructions. + - `u64` status enum: + - Enum variant `0u64`: Retracted, program is in maintenance + - Enum variant `1u64`: Deployed, program is ready to be executed + - Enum variant `2u64`: Finalized, same as `Deployed`, but can not be + modified anymore +- Body: + - `[u8]` The programs executable file + +Verification the program account checks in the following order that: + +- the owner of the program account is loader-v4, +otherwise throw `InvalidAccountOwner` +- the program account is at least as long enough for the header, +otherwise throw `AccountDataTooSmall` +- the program account is writable, otherwise throw `InvalidArgument` +- the provided authority (instruction account at index 1) signed, +otherwise throw `MissingRequiredSignature` +- the authority stored in the program account is the one provided, +otherwise throw `IncorrectAuthority` +- the status stored in the program account is not finalized, +otherwise throw `Immutable` + +### Execution / Invocation + +Invoking programs owned by loader-v4 checks in the following order that: + +- the owner of the program account is loader-v4, +otherwise throw `UnsupportedProgramId` +- the program account is at least as long enough for the header, +otherwise throw `AccountDataTooSmall` +- the status stored in the program account is not retracted, +otherwise throw `UnsupportedProgramId` +- the program account was not deployed recently (delay-visibility), +otherwise throw `UnsupportedProgramId` +- the executable file stored in the program account passes executable +verification, otherwise throw `UnsupportedProgramId` + +Otherwise the execution semantics stay the same as in loader-v2 and v3. +Delay-visibility also stays the same as in loader-v3: + +- Re/deployed programs act as closed until the end of the slot, +only then becoming available for execution +- The feature set that the executable file is verified against is not +necessarily the current one, but the one of the epoch of the next slot + +### Program Management Instructions + +#### Write + +- Instruction accounts: + - `[writable]` The program account to write to. + - `[signer]` The authority of the program. +- Instruction data: + - Enum variant `0u32` + - `u32` Offset at which to write the given bytes + - `[u8]` Chunk of the programs executable file +- Behavior: + - Check there are at least two instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Verify the program account + - Check the status stored in the program account is retracted, + otherwise throw `InvalidArgument` + - Check that the end offset (sum of offset and length of the chunk) does + not exceed the maximum (program account length minus the header size), + otherwise throw `AccountDataTooSmall` + - Copy the chunk into the program account at the offset shifted by the + header size + +#### Truncate + +- Instruction accounts: + - `[(signer), writable]` The program account to change the size of. + - `[signer]` The authority of the program. + - `[writable]` Optional, the recipient account. +- Instruction data: + - Enum variant `1u32` + - `u32` The new size after the operation. +- Behavior: + - Check there are at least two instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - If this is an initialization (program account length is too short to + contain the header and the requested new size is greater 0): + - the owner of the program account is loader-v4, + otherwise throw `InvalidAccountOwner` + - the program account is writable, otherwise throw `InvalidArgument` + - the program account (instruction account at index 0) signed, + otherwise throw `MissingRequiredSignature` + - the provided authority (instruction account at index 1) signed, + otherwise throw `MissingRequiredSignature` + - If this is not an initialization: + - Verify the program account + - Check that the status stored in the program account is retracted, + otherwise throw `InvalidArgument` + - Check that there are enough funds in the program account for rent + exemption, otherwise throw `InsufficientFunds` + - If there are more than enough funds: + - Check there are at least three instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Check that the recipient account (instruction account at index 2) is + writable, otherwise throw `InvalidArgument` + - Transfer the surplus from the program account to the recipient account + - If the requested new size is zero: + - Delete the entire program account, including the header + - If the requested new size is greater than zero: + - Set the length of the program account to the requested new size plus + the header size + - In case that this is an initialization, also initialize the header: + - Set the `is_executable` flag to `true` + - Set the slot to zero, **not** the current slot + - Set the authority address (from the instruction account at index 1) + - Set the status to retracted + +#### Deploy + +- Instruction accounts: + - `[writable]` The program account to deploy. + - `[signer]` The authority of the program. + - `[writable]` Optional, an undeployed source program account to take data + and lamports from. +- Instruction data: + - Enum variant `2u32` +- Behavior: + - Check there are at least two instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Verify the program account + - Check that the deployment cooldown expired, + otherwise throw `InvalidArgument` + - Check that the status stored in the program account is retracted + otherwise throw `InvalidArgument` + - In case a source program was provided (instruction account at index 2): + - Verify the source program account + - Check that the status stored in the source program account is retracted, + otherwise throw `InvalidArgument` + - Check that the executable file stored in the source program account + passes executable verification + - Copy the entire source program account into the program account + - Set the length of the source program account to zero + - Transfer all funds of the source program account to the program + account + - In case no source program was provided: + - Check that the executable file stored in the program account passes + executable verification + - Change the slot in the program account to the current slot + - Change the status stored in the program account to deployed + +#### Retract + +- Instruction accounts: + - `[writable]` The program account to retract. + - `[signer]` The authority of the program. +- Instruction data: + - Enum variant `3u32` +- Behavior: + - Check there are at least two instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Verify the program account + - Check that the deployment cooldown expired, + otherwise throw `InvalidArgument` + - Check that the status stored in the program account is deployed, + otherwise throw `InvalidArgument` + - Note: The slot is **not** set to the current slot to allow a + retract-modify-redeploy-sequence within the same slot + - Change the status stored in the program account to retracted + +#### TransferAuthority + +- Instruction accounts: + - `[writable]` The program account to change the authority of. + - `[signer]` The current authority of the program. + - `[signer]` Optional, the new authority of the program. +- Instruction data: + - Enum variant `4u32` +- Behavior: + - Check there are at least two instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Verify the program account + - In case a new authority was provided (instruction account at index 2): + - Check that it signed as well, + otherwise throw `MissingRequiredSignature` + - Check that the authority stored in the program account is different + from the one provided, otherwise throw `InvalidArgument` + - Copy the new authority address into the program account + - In case no new authority was provided: + - Check that the status stored in the program account is deployed, + otherwise throw `InvalidArgument` + - Change the status stored in the program account to finalized + +## Impact + +This proposal: + +- covers all the use cases loader-v3 had but in a cleaner way and comes with +a specification. +- makes deployment slightly cheaper for dapp developers as they would no longer +have to burn funds for the rent exception of the proxy account. +- provides an alternative redeployment path which does not require a big +deposit of funds for rent exception during the upload. +- enables dapp developers to withdrawl the surplus of funds required for rent +exception when shortening the length of program accounts or closing them. +- shortens the workflow of temporarily closing a program to a single +instruction, instead of having to build and redeploy an empty program. +- properly alignes the executable file relative to the beginning of the +account. In loader-v3 it is misaligned. +- once all loader-v3 programs are migrated: + - allows transaction account loading to be simplifed, because every program + would load exactly one account, no need to load the proxy account to get to + the actual program data (which is not listed in the transaction accounts). + - allows the removal of the write lock demotion exception if loader-v3 is + present in a transaction. + - corrects the miscounting of the proxy account size towards the total + transaction account loading limit. + +## Security Considerations + +None. + +## Backwards Compatibility + +This proposal does not break any existing programs. However, dapp developers +might want to profit from the new program mangement instructions without +influencing their users work flows. To do so they would need a way to turn the +program accounts of loader-v3 to program accounts of loader-v4, changing the +account owner but keeping the program address. A potential issue is that the +programdata header of loader-v3 is only 45 bytes long while loader-v4 takes 48 +bytes. An automatic mechanism in the program runtime (triggered by feature +activation) could then perform the following steps per program: + +- loader-v3 clears the program proxy account (setting its size to zero) +- loader-v3 transfers all funds from the programdata to the proxy account +- loader-v3 gifts the program proxy account to loader-v4 +- loader-v4 initializes it via `Truncate` +- loader-v4 copies the data from the programdata account via `Write` +- loader-v4 deploys it via `Deploy` +- Optinally, loader-v4 finalizes it without a next version forwarding +- loader-v3 closes the programdata account (setting its size to zero) From a8d8a6396659fc7784f7ef333543dff219708f85 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Mon, 26 Aug 2024 12:23:49 +0200 Subject: [PATCH 02/17] Adds explicit finalization and next version forwarding. --- proposals/0167-loader-v4.md | 51 ++++++++++++++++++++++++++++--------- 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 0ce6523d1..0ddf03deb 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -58,7 +58,8 @@ Accounts of programs owned by loader-v4 must have the following layout: - `u64` Slot in which the program was last deployed, retracted or initialized. - `[u8; 32]` Authority address which can send program management - instructions. + instructions. Or if the status is finalized, then the address of the next + version of the program. - `u64` status enum: - Enum variant `0u64`: Retracted, program is in maintenance - Enum variant `1u64`: Deployed, program is ready to be executed @@ -228,23 +229,46 @@ necessarily the current one, but the one of the epoch of the next slot - Instruction accounts: - `[writable]` The program account to change the authority of. - `[signer]` The current authority of the program. - - `[signer]` Optional, the new authority of the program. + - `[signer]` The new authority of the program. - Instruction data: - Enum variant `4u32` - Behavior: - - Check there are at least two instruction accounts, + - Check there are at least three instruction accounts, otherwise throw `NotEnoughAccountKeys` - Verify the program account - - In case a new authority was provided (instruction account at index 2): - - Check that it signed as well, - otherwise throw `MissingRequiredSignature` - - Check that the authority stored in the program account is different - from the one provided, otherwise throw `InvalidArgument` - - Copy the new authority address into the program account - - In case no new authority was provided: - - Check that the status stored in the program account is deployed, + - Check that the new authority (instruction account at index 2) + signed as well, otherwise throw `MissingRequiredSignature` + - Check that the authority stored in the program account is different + from the one provided, otherwise throw `InvalidArgument` + - Copy the new authority address into the program account + +#### Finalize + +- Instruction accounts: + - `[writable]` The program account to change the authority of. + - `[signer]` The current authority of the program. + - `[]` Optional, the reserved address for the next version of the program. +- Instruction data: + - Enum variant `5u32` +- Behavior: + - Check there are at least three instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Verify the program account + - Check that the status stored in the program account is deployed, otherwise throw `InvalidArgument` - - Change the status stored in the program account to finalized + - for the program account of the next version + (instruction account at index 2) check that: + - the owner of the program account is loader-v4, + otherwise throw `InvalidAccountOwner` + - the program account is at least as long enough for the header, + otherwise throw `AccountDataTooSmall` + - the authority stored in the program account is the one provided, + otherwise throw `IncorrectAuthority` + - the status stored in the program account is not finalized, + otherwise throw `Immutable` + - Copy the address of the next version into the next version field stored in + the previous versions program account + - Change the status stored in the program account to finalized ## Impact @@ -252,6 +276,9 @@ This proposal: - covers all the use cases loader-v3 had but in a cleaner way and comes with a specification. +- allows finalized programs to mark which other program supersedes them which +can then be offered as an option in forntends. This provides a more secure +alternative to redeployment / upgrading of programs at the same address. - makes deployment slightly cheaper for dapp developers as they would no longer have to burn funds for the rent exception of the proxy account. - provides an alternative redeployment path which does not require a big From 2c848cb1cac62b4deec6a873598084dd91468965 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Tue, 1 Oct 2024 14:30:08 +0200 Subject: [PATCH 03/17] Clearifies delay visibility and deployment cooldown. --- proposals/0167-loader-v4.md | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 0ddf03deb..3afb03976 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -92,19 +92,11 @@ otherwise throw `UnsupportedProgramId` otherwise throw `AccountDataTooSmall` - the status stored in the program account is not retracted, otherwise throw `UnsupportedProgramId` -- the program account was not deployed recently (delay-visibility), -otherwise throw `UnsupportedProgramId` +- the program account was not deployed within the current slot +(delay visibility), otherwise throw `UnsupportedProgramId` - the executable file stored in the program account passes executable verification, otherwise throw `UnsupportedProgramId` -Otherwise the execution semantics stay the same as in loader-v2 and v3. -Delay-visibility also stays the same as in loader-v3: - -- Re/deployed programs act as closed until the end of the slot, -only then becoming available for execution -- The feature set that the executable file is verified against is not -necessarily the current one, but the one of the epoch of the next slot - ### Program Management Instructions #### Write @@ -185,8 +177,8 @@ necessarily the current one, but the one of the epoch of the next slot - Check there are at least two instruction accounts, otherwise throw `NotEnoughAccountKeys` - Verify the program account - - Check that the deployment cooldown expired, - otherwise throw `InvalidArgument` + - Check that the slot stored in the program account is not the current + (deployment cooldown), otherwise throw `InvalidArgument` - Check that the status stored in the program account is retracted otherwise throw `InvalidArgument` - In case a source program was provided (instruction account at index 2): @@ -195,6 +187,8 @@ necessarily the current one, but the one of the epoch of the next slot otherwise throw `InvalidArgument` - Check that the executable file stored in the source program account passes executable verification + - The feature set that the executable file is verified against is not +necessarily the current one, but the one of the epoch of the next slot - Copy the entire source program account into the program account - Set the length of the source program account to zero - Transfer all funds of the source program account to the program @@ -216,8 +210,8 @@ necessarily the current one, but the one of the epoch of the next slot - Check there are at least two instruction accounts, otherwise throw `NotEnoughAccountKeys` - Verify the program account - - Check that the deployment cooldown expired, - otherwise throw `InvalidArgument` + - Check that the slot stored in the program account is not the current + (deployment cooldown), otherwise throw `InvalidArgument` - Check that the status stored in the program account is deployed, otherwise throw `InvalidArgument` - Note: The slot is **not** set to the current slot to allow a From c8448d62e1ae01a92e73c95f8fc4c9f770d3a199 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Tue, 19 Nov 2024 16:04:17 +0100 Subject: [PATCH 04/17] Further clarifications after implementing integration tests. --- proposals/0167-loader-v4.md | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 3afb03976..d57de874f 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -5,7 +5,7 @@ authors: - Alexander Meißner category: Standard type: Core -status: Draft +status: Review created: 2024-08-15 feature: TBD --- @@ -45,9 +45,13 @@ None. ## Detailed Design -`LoaderV411111111111111111111111111111111111` program management and execution -must be enabled with the associated feature gate, which simultaneously disables -new deployments on loader-v3 (`BPFLoaderUpgradeab1e11111111111111111111111`), +The associated feature gate must: + +- add loader-v4 to the write lock demotion exceptions +- enable loader-v4 `LoaderV411111111111111111111111111111111111` program +management and execution +- simultaneously disable new deployments on loader-v3 +(`BPFLoaderUpgradeab1e11111111111111111111111`), throwing `InvalidIstructionData` if `DeployWithMaxDataLen` is called. ### Owned Program Accounts @@ -86,19 +90,20 @@ otherwise throw `Immutable` Invoking programs owned by loader-v4 checks in the following order that: -- the owner of the program account is loader-v4, -otherwise throw `UnsupportedProgramId` -- the program account is at least as long enough for the header, -otherwise throw `AccountDataTooSmall` -- the status stored in the program account is not retracted, -otherwise throw `UnsupportedProgramId` -- the program account was not deployed within the current slot -(delay visibility), otherwise throw `UnsupportedProgramId` +- the owner of the program account is loader-v4 +- the program account is at least as long enough for the header +- the status stored in the program account is not retracted +- the program account was not deployed within the current slot (delay +visibility) - the executable file stored in the program account passes executable -verification, otherwise throw `UnsupportedProgramId` +verification + +failing any of the above checks must throw `UnsupportedProgramId`. ### Program Management Instructions +All program management instructions must cost 2000 CUs. + #### Write - Instruction accounts: @@ -187,8 +192,10 @@ verification, otherwise throw `UnsupportedProgramId` otherwise throw `InvalidArgument` - Check that the executable file stored in the source program account passes executable verification - - The feature set that the executable file is verified against is not -necessarily the current one, but the one of the epoch of the next slot + - The feature set that the executable file is verified against is not + necessarily the current one, but the one of the epoch of the next slot + - Also, during deployment certain deprecated syscalls are disabled, + this stays the same as in the older loaders - Copy the entire source program account into the program account - Set the length of the source program account to zero - Transfer all funds of the source program account to the program From 5f362c33323fd9a4a92ef6a6bfefb1a15f2e9d60 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Wed, 29 Jan 2025 14:57:11 +0100 Subject: [PATCH 05/17] Changes the instruction `Truncate` to `SetProgramLength`. --- proposals/0167-loader-v4.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index d57de874f..1559a2762 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -125,7 +125,7 @@ All program management instructions must cost 2000 CUs. - Copy the chunk into the program account at the offset shifted by the header size -#### Truncate +#### SetProgramLength - Instruction accounts: - `[(signer), writable]` The program account to change the size of. @@ -142,8 +142,6 @@ All program management instructions must cost 2000 CUs. - the owner of the program account is loader-v4, otherwise throw `InvalidAccountOwner` - the program account is writable, otherwise throw `InvalidArgument` - - the program account (instruction account at index 0) signed, - otherwise throw `MissingRequiredSignature` - the provided authority (instruction account at index 1) signed, otherwise throw `MissingRequiredSignature` - If this is not an initialization: @@ -157,7 +155,9 @@ All program management instructions must cost 2000 CUs. otherwise throw `NotEnoughAccountKeys` - Check that the recipient account (instruction account at index 2) is writable, otherwise throw `InvalidArgument` - - Transfer the surplus from the program account to the recipient account + - If a recipient account was provided that is not the program account: + - Transfer the surplus from the program account to the recipient account + - otherwise, if the requested new size is zero throw `InvalidArgument` - If the requested new size is zero: - Delete the entire program account, including the header - If the requested new size is greater than zero: From ad40283598236023fd9cfcf3a38591a347efcebb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Wed, 29 Jan 2025 15:50:42 +0100 Subject: [PATCH 06/17] Adds the `Copy` instruction. --- proposals/0167-loader-v4.md | 45 ++++++++++++++++++++++++++++++++----- 1 file changed, 40 insertions(+), 5 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 1559a2762..5317fd085 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -125,6 +125,41 @@ All program management instructions must cost 2000 CUs. - Copy the chunk into the program account at the offset shifted by the header size +#### Copy + +- Instruction accounts: + - `[writable]` The program account to copy to. + - `[signer]` The authority of the program. + - `[]` The program(data) account to copy from. +- Instruction data: + - Enum variant `1u32` + - `u32` Byte offset at which to write + - `u32` Byte offset at which to read + - `u32` Length of the chunk to copy in bytes +- Behavior: + - Check there are at least three instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Check that program account and source account do not alias, + otherwise throw `AccountBorrowFailed` + - Verify the program account + - Check the status stored in the program account is retracted, + otherwise throw `InvalidArgument` + - Check that the source account is owned by loader v1, v2, v3 or v4, + otherwise throw `InvalidArgument` + - and look-up the source header size: + - loader-v1: 0 bytes + - loader-v2: 0 bytes + - loader-v3: 45 bytes + - loader-v4: 48 bytes + - Check that the source end offset (sum of source offset and length) does + not exceed the maximum (source account length minus the source header size), + otherwise throw `AccountDataTooSmall` + - Check that the destination end offset (sum of destination offset and + length) does not exceed the maximum (program account length minus the loader-v4 + header size), otherwise throw `AccountDataTooSmall` + - Copy the chunk between the program accounts at the offsets, each shifted by + the header size of their loader (account owner) respectively + #### SetProgramLength - Instruction accounts: @@ -132,7 +167,7 @@ All program management instructions must cost 2000 CUs. - `[signer]` The authority of the program. - `[writable]` Optional, the recipient account. - Instruction data: - - Enum variant `1u32` + - Enum variant `2u32` - `u32` The new size after the operation. - Behavior: - Check there are at least two instruction accounts, @@ -177,7 +212,7 @@ All program management instructions must cost 2000 CUs. - `[writable]` Optional, an undeployed source program account to take data and lamports from. - Instruction data: - - Enum variant `2u32` + - Enum variant `3u32` - Behavior: - Check there are at least two instruction accounts, otherwise throw `NotEnoughAccountKeys` @@ -212,7 +247,7 @@ All program management instructions must cost 2000 CUs. - `[writable]` The program account to retract. - `[signer]` The authority of the program. - Instruction data: - - Enum variant `3u32` + - Enum variant `4u32` - Behavior: - Check there are at least two instruction accounts, otherwise throw `NotEnoughAccountKeys` @@ -232,7 +267,7 @@ All program management instructions must cost 2000 CUs. - `[signer]` The current authority of the program. - `[signer]` The new authority of the program. - Instruction data: - - Enum variant `4u32` + - Enum variant `5u32` - Behavior: - Check there are at least three instruction accounts, otherwise throw `NotEnoughAccountKeys` @@ -250,7 +285,7 @@ All program management instructions must cost 2000 CUs. - `[signer]` The current authority of the program. - `[]` Optional, the reserved address for the next version of the program. - Instruction data: - - Enum variant `5u32` + - Enum variant `6u32` - Behavior: - Check there are at least three instruction accounts, otherwise throw `NotEnoughAccountKeys` From 2571f83d7e68804973372c4167d67fbd5c0e8683 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Wed, 29 Jan 2025 16:06:04 +0100 Subject: [PATCH 07/17] Adds loader-v3 instruction `Migrate`. --- proposals/0167-loader-v4.md | 52 +++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 5317fd085..daa8e5bdc 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -306,6 +306,58 @@ All program management instructions must cost 2000 CUs. the previous versions program account - Change the status stored in the program account to finalized +### Loader-v3 Instruction: Migrate + +- Instruction accounts: + - `[writable]` The program data account. + - `[writable]` The program account. + - `[signer]` The migration authority. +- Instruction data: + - Enum variant `8u32` +- Behavior: + - Check that there are at least three instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Check that the program data account is writable, + otherwise throw `InvalidArgument` + - Check that the program data was last modified before the current slot + if the program data has the state `ProgramData`, + otherwise throw `InvalidArgument` + - Check that the provided authority is either: + - the migration authority + (pubkey is `3Scf35jMNk2xXBD6areNjgMtXgp5ZspDhms8vdcbzC42`) + - or the upgrade authority stored in the program data account + - or the program signer if the program is finalized, closed or + uninitialized + otherwise throw `IncorrectAuthority` + - Check that the provided authority is a signer, + otherwise throw `MissingRequiredSignature` + - Check that the program account is writable, + otherwise throw `InvalidArgument` + - Check that the program account is owned by loader-v3, + otherwise throw `IncorrectProgramId` + - Check that the program account has the state `Program`, + otherwise throw `InvalidAccountData` + - Check that the program account points to the program data account, + otherwise throw `InvalidArgument` + - Clear the program account (setting its size to zero) + - Transfer all funds from the program data account to the program account + - If the program data account was closed / empty or uninitialized: + - Assign ownership of the program account to the system program + - otherwise, if the program data account contains actual program data: + - Assign ownership of the program account to loader-v4 + - CPI loader-v4 `SetProgramLength` the program account to the size of the + program data account minus the loader-v3 header size and use the migration + authority. + - CPI loader-v4 `Copy` the program data account into the program account + - CPI loader-v4 `Deploy` the program account + - If the program data account was finalized (upgrade authority is `None`): + - CPI loader-v4 `Finalize` without a next version forwarding + - otherwise, if the program data account was not finalized and the + migration authority (as opposed to the upgrade authority) was provided: + - CPI loader-v4 `TransferAuthority` to the upgrade authority + - Clear the program data account (setting its size to zero) + - Assign ownership of the program data account to the system program + ## Impact This proposal: From 87b57e8c13c3d4df20c478c39982a13eb0588ab6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Wed, 29 Jan 2025 14:58:37 +0100 Subject: [PATCH 08/17] Expands on how loader v3 to v4 migration works. --- proposals/0167-loader-v4.md | 113 ++++++++++++++++++++---------------- 1 file changed, 62 insertions(+), 51 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index daa8e5bdc..0de868ae2 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -7,7 +7,9 @@ category: Standard type: Core status: Review created: 2024-08-15 -feature: TBD +feature: + - 8Cb77yHjPWe9wuWUfXeh6iszFGCDGNCoFk3tprViYHNm + - EmhbpdVtZ2hWRGFWBDjn2i3SJD8Z36z4mpMcZJEnebnP --- ## Summary @@ -45,13 +47,15 @@ None. ## Detailed Design -The associated feature gate must: +The feature gate `8Cb77yHjPWe9wuWUfXeh6iszFGCDGNCoFk3tprViYHNm` must: -- add loader-v4 to the write lock demotion exceptions - enable loader-v4 `LoaderV411111111111111111111111111111111111` program -management and execution -- simultaneously disable new deployments on loader-v3 -(`BPFLoaderUpgradeab1e11111111111111111111111`), +management and execution. +- enable the loader-v3 `BPFLoaderUpgradeab1e11111111111111111111111` +instruction `UpgradeableLoaderInstruction::Migrate`. + +An additional feature gate `EmhbpdVtZ2hWRGFWBDjn2i3SJD8Z36z4mpMcZJEnebnP` +must disable new deployments on loader-v3, throwing `InvalidIstructionData` if `DeployWithMaxDataLen` is called. ### Owned Program Accounts @@ -111,7 +115,7 @@ All program management instructions must cost 2000 CUs. - `[signer]` The authority of the program. - Instruction data: - Enum variant `0u32` - - `u32` Offset at which to write the given bytes + - `u32` Byte offset at which to write the given bytes - `[u8]` Chunk of the programs executable file - Behavior: - Check there are at least two instruction accounts, @@ -221,7 +225,8 @@ All program management instructions must cost 2000 CUs. (deployment cooldown), otherwise throw `InvalidArgument` - Check that the status stored in the program account is retracted otherwise throw `InvalidArgument` - - In case a source program was provided (instruction account at index 2): + - In case a source program was provided (instruction account at index 2) + which is not the program account: - Verify the source program account - Check that the status stored in the source program account is retracted, otherwise throw `InvalidArgument` @@ -235,7 +240,7 @@ All program management instructions must cost 2000 CUs. - Set the length of the source program account to zero - Transfer all funds of the source program account to the program account - - In case no source program was provided: + - otherwise, if no source program was provided: - Check that the executable file stored in the program account passes executable verification - Change the slot in the program account to the current slot @@ -360,31 +365,53 @@ All program management instructions must cost 2000 CUs. ## Impact -This proposal: - -- covers all the use cases loader-v3 had but in a cleaner way and comes with -a specification. -- allows finalized programs to mark which other program supersedes them which -can then be offered as an option in forntends. This provides a more secure -alternative to redeployment / upgrading of programs at the same address. -- makes deployment slightly cheaper for dapp developers as they would no longer -have to burn funds for the rent exception of the proxy account. -- provides an alternative redeployment path which does not require a big -deposit of funds for rent exception during the upload. -- enables dapp developers to withdrawl the surplus of funds required for rent -exception when shortening the length of program accounts or closing them. -- shortens the workflow of temporarily closing a program to a single -instruction, instead of having to build and redeploy an empty program. -- properly alignes the executable file relative to the beginning of the -account. In loader-v3 it is misaligned. -- once all loader-v3 programs are migrated: - - allows transaction account loading to be simplifed, because every program - would load exactly one account, no need to load the proxy account to get to - the actual program data (which is not listed in the transaction accounts). - - allows the removal of the write lock demotion exception if loader-v3 is - present in a transaction. - - corrects the miscounting of the proxy account size towards the total - transaction account loading limit. +- This proposal covers all the use cases loader-v3 had but in a cleaner way and +comes with a specification. +- loader-v3 had a separate account type for buffers and extra commands for +these buffer accounts, in loader-v4 program accounts can act as buffers, there +is no more distinction. +- loader-v3 deployments always needed a buffer, in loader-v4 it is optional, +one can upload a redeployment into the program account directly. +- loader-v3 had two accounts per program, loader-v4 goes back to having only +one, thus needs less funds to reach rent exemption. +- loader-v3 closing of programs did finalize them, in loader-v4 all the funds +can be retireved and the program address repurposed. +- loader-v3 programs could only grow, loader-v4 can shrink programs and also +retireve the surplus of funds no longer required for rent exception. +- loader-v3 programs were always "live" after the first deployment, with +loader-v4 one can temporarily put a program into maintenance mode without a +redeployment. +- loader-v3 always required the entire program to be uploaded for a +redeployment, loader-v4 supports partial uploads for patching chunks of the +program. +- loader-v3 ELFs were misaligned, loader-v4 properly alignes the executable +file relative to the beginning of the account. +- loader-v4 allows finalized programs to mark which other program supersedes +them which can then be offered as an option in forntends. This provides a +more secure alternative to redeployment / upgrading of programs at the same +address. +- An option to migrate programs from loader-v3 to loader-v4 without changing +their program address will be available via a new loader-v3 instruction. This +will count as a redeployment and thus render the program unavailable for the +rest of the slot (delay visibility). + +Once new programs can not be deployed on loader-v3 anymore, the list of all +loader-v3 programs becomes fixed and can be extracted from a snapshot. Using +the added loader-v3 migration instruction and the global migration authority, +the core protocol developers will then migrate all loader-v3 programs to +loader-v4 programs, which once completed: + +- allows transaction account loading to be simplifed, because every program +would load exactly one account, no need to load the proxy account to get to +the actual program data (which is not listed in the transaction accounts). +- allows the removal of the write lock demotion exception if loader-v3 is +present in a transaction. +- corrects the miscounting of the program data account size towards the total +transaction account loading limit. +- allows dApp devs to resuscitate closed loader-v3 programs if they still +control the program authority. This allows redeployment at the same address +or completely closing the program account in order to retrieve the locked +funds. ## Security Considerations @@ -392,20 +419,4 @@ None. ## Backwards Compatibility -This proposal does not break any existing programs. However, dapp developers -might want to profit from the new program mangement instructions without -influencing their users work flows. To do so they would need a way to turn the -program accounts of loader-v3 to program accounts of loader-v4, changing the -account owner but keeping the program address. A potential issue is that the -programdata header of loader-v3 is only 45 bytes long while loader-v4 takes 48 -bytes. An automatic mechanism in the program runtime (triggered by feature -activation) could then perform the following steps per program: - -- loader-v3 clears the program proxy account (setting its size to zero) -- loader-v3 transfers all funds from the programdata to the proxy account -- loader-v3 gifts the program proxy account to loader-v4 -- loader-v4 initializes it via `Truncate` -- loader-v4 copies the data from the programdata account via `Write` -- loader-v4 deploys it via `Deploy` -- Optinally, loader-v4 finalizes it without a next version forwarding -- loader-v3 closes the programdata account (setting its size to zero) +None. From 15c56366fb17ef658958a56971dd4776aea9e3a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Mon, 10 Feb 2025 20:13:59 +0100 Subject: [PATCH 09/17] Review feedback --- proposals/0167-loader-v4.md | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 0de868ae2..2c7657f2c 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -22,7 +22,7 @@ Loader-v3, which is currently the only deployable loader, requires two accounts per program. This was a workaround to circumvent the finality of the `is_executable` flag, which is removed in SIMD-0162. Consequentially, this setup of the program account being a proxy account, containing the address of -the actual programdata account, is no longer necessary and should be removed. +the actual program data account, is no longer necessary and should be removed. Another issue with loader-v3 is that the executable file stored in the programdata account is misaligned relative to the beginning of the account. @@ -214,7 +214,7 @@ All program management instructions must cost 2000 CUs. - `[writable]` The program account to deploy. - `[signer]` The authority of the program. - `[writable]` Optional, an undeployed source program account to take data - and lamports from. + and funds from. - Instruction data: - Enum variant `3u32` - Behavior: @@ -223,6 +223,8 @@ All program management instructions must cost 2000 CUs. - Verify the program account - Check that the slot stored in the program account is not the current (deployment cooldown), otherwise throw `InvalidArgument` + - Note: The cooldown enforces that each pair of an address and a slot can + uniquely identify a deployment of a program, which simplifies caching logic. - Check that the status stored in the program account is retracted otherwise throw `InvalidArgument` - In case a source program was provided (instruction account at index 2) @@ -238,8 +240,12 @@ All program management instructions must cost 2000 CUs. this stays the same as in the older loaders - Copy the entire source program account into the program account - Set the length of the source program account to zero - - Transfer all funds of the source program account to the program - account + - Swap the funds of the source program account and the program account. + Note: This ensures correct amount for rent exemption (as it was calculated + for the source program account) remains with the program account and the + rest is deposited into the source program account to be retrieved. It works + with programs growing, staying the same size or shrinking. + - Assign ownership of the source program account to the system program - otherwise, if no source program was provided: - Check that the executable file stored in the program account passes executable verification @@ -262,7 +268,8 @@ All program management instructions must cost 2000 CUs. - Check that the status stored in the program account is deployed, otherwise throw `InvalidArgument` - Note: The slot is **not** set to the current slot to allow a - retract-modify-redeploy-sequence within the same slot + retract-modify-redeploy-sequence within the same slot or even within the + same transaction. - Change the status stored in the program account to retracted #### TransferAuthority @@ -375,21 +382,22 @@ one can upload a redeployment into the program account directly. - loader-v3 had two accounts per program, loader-v4 goes back to having only one, thus needs less funds to reach rent exemption. - loader-v3 closing of programs did finalize them, in loader-v4 all the funds -can be retireved and the program address repurposed. +can be retrieved and the program address repurposed. - loader-v3 programs could only grow, loader-v4 can shrink programs and also -retireve the surplus of funds no longer required for rent exception. +retrieve the surplus of funds no longer required for rent exception. - loader-v3 programs were always "live" after the first deployment, with loader-v4 one can temporarily put a program into maintenance mode without a redeployment. - loader-v3 always required the entire program to be uploaded for a redeployment, loader-v4 supports partial uploads for patching chunks of the program. -- loader-v3 ELFs were misaligned, loader-v4 properly alignes the executable +- loader-v3 ELFs were misaligned, loader-v4 properly aligns the executable file relative to the beginning of the account. - loader-v4 allows finalized programs to mark which other program supersedes -them which can then be offered as an option in forntends. This provides a +them which can then be offered as an option in frontends. This provides a more secure alternative to redeployment / upgrading of programs at the same -address. +address. The keypair for the next version linked during finalization should be +generated beforehand. - An option to migrate programs from loader-v3 to loader-v4 without changing their program address will be available via a new loader-v3 instruction. This will count as a redeployment and thus render the program unavailable for the @@ -401,7 +409,7 @@ the added loader-v3 migration instruction and the global migration authority, the core protocol developers will then migrate all loader-v3 programs to loader-v4 programs, which once completed: -- allows transaction account loading to be simplifed, because every program +- allows transaction account loading to be simplified, because every program would load exactly one account, no need to load the proxy account to get to the actual program data (which is not listed in the transaction accounts). - allows the removal of the write lock demotion exception if loader-v3 is From 3b5dd11f9a85a02f99d62900e212e3daf9262195 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Thu, 24 Apr 2025 16:38:05 +0200 Subject: [PATCH 10/17] Removes the deploy from source instruction and rekeys the feature. --- proposals/0167-loader-v4.md | 36 ++++-------------------------------- 1 file changed, 4 insertions(+), 32 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 2c7657f2c..abf530394 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -7,9 +7,7 @@ category: Standard type: Core status: Review created: 2024-08-15 -feature: - - 8Cb77yHjPWe9wuWUfXeh6iszFGCDGNCoFk3tprViYHNm - - EmhbpdVtZ2hWRGFWBDjn2i3SJD8Z36z4mpMcZJEnebnP +feature: G8yMNsNUd4p3VB22ycrPEB1qRgepCFeFpAqD2Lr66s36 --- ## Summary @@ -47,17 +45,13 @@ None. ## Detailed Design -The feature gate `8Cb77yHjPWe9wuWUfXeh6iszFGCDGNCoFk3tprViYHNm` must: +The feature gate must: - enable loader-v4 `LoaderV411111111111111111111111111111111111` program management and execution. - enable the loader-v3 `BPFLoaderUpgradeab1e11111111111111111111111` instruction `UpgradeableLoaderInstruction::Migrate`. -An additional feature gate `EmhbpdVtZ2hWRGFWBDjn2i3SJD8Z36z4mpMcZJEnebnP` -must disable new deployments on loader-v3, -throwing `InvalidIstructionData` if `DeployWithMaxDataLen` is called. - ### Owned Program Accounts Accounts of programs owned by loader-v4 must have the following layout: @@ -213,8 +207,6 @@ All program management instructions must cost 2000 CUs. - Instruction accounts: - `[writable]` The program account to deploy. - `[signer]` The authority of the program. - - `[writable]` Optional, an undeployed source program account to take data - and funds from. - Instruction data: - Enum variant `3u32` - Behavior: @@ -227,28 +219,8 @@ All program management instructions must cost 2000 CUs. uniquely identify a deployment of a program, which simplifies caching logic. - Check that the status stored in the program account is retracted otherwise throw `InvalidArgument` - - In case a source program was provided (instruction account at index 2) - which is not the program account: - - Verify the source program account - - Check that the status stored in the source program account is retracted, - otherwise throw `InvalidArgument` - - Check that the executable file stored in the source program account - passes executable verification - - The feature set that the executable file is verified against is not - necessarily the current one, but the one of the epoch of the next slot - - Also, during deployment certain deprecated syscalls are disabled, - this stays the same as in the older loaders - - Copy the entire source program account into the program account - - Set the length of the source program account to zero - - Swap the funds of the source program account and the program account. - Note: This ensures correct amount for rent exemption (as it was calculated - for the source program account) remains with the program account and the - rest is deposited into the source program account to be retrieved. It works - with programs growing, staying the same size or shrinking. - - Assign ownership of the source program account to the system program - - otherwise, if no source program was provided: - - Check that the executable file stored in the program account passes - executable verification + - Check that the executable file stored in the program account passes + executable verification - Change the slot in the program account to the current slot - Change the status stored in the program account to deployed From a97c758ff8a6eab95fae7725476d1c8f4d9d34de Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Tue, 6 May 2025 16:05:25 +0200 Subject: [PATCH 11/17] Migrates empty / closed loader-v3 programs to loader-v4 as well, allows uninitialized program accounts to be closed and rekeys the feature. --- proposals/0167-loader-v4.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index abf530394..833ea9f94 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -7,7 +7,7 @@ category: Standard type: Core status: Review created: 2024-08-15 -feature: G8yMNsNUd4p3VB22ycrPEB1qRgepCFeFpAqD2Lr66s36 +feature: 2aQJYqER2aKyb3cZw22v4SL2xMX7vwXBRWfvS4pTrtED --- ## Summary @@ -171,7 +171,7 @@ All program management instructions must cost 2000 CUs. - Check there are at least two instruction accounts, otherwise throw `NotEnoughAccountKeys` - If this is an initialization (program account length is too short to - contain the header and the requested new size is greater 0): + contain the header): - the owner of the program account is loader-v4, otherwise throw `InvalidAccountOwner` - the program account is writable, otherwise throw `InvalidArgument` @@ -325,10 +325,8 @@ All program management instructions must cost 2000 CUs. otherwise throw `InvalidArgument` - Clear the program account (setting its size to zero) - Transfer all funds from the program data account to the program account - - If the program data account was closed / empty or uninitialized: - - Assign ownership of the program account to the system program - - otherwise, if the program data account contains actual program data: - - Assign ownership of the program account to loader-v4 + - Assign ownership of the program account to loader-v4 + - If the program data account was not closed / empty or uninitialized: - CPI loader-v4 `SetProgramLength` the program account to the size of the program data account minus the loader-v3 header size and use the migration authority. From aee9c74cdcef7037bea1155b73608d97c85dfbfe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Tue, 17 Jun 2025 12:50:21 +0200 Subject: [PATCH 12/17] Removes signer flag from parameters of SetProgramLength. --- proposals/0167-loader-v4.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 833ea9f94..1e13042c6 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -161,7 +161,7 @@ All program management instructions must cost 2000 CUs. #### SetProgramLength - Instruction accounts: - - `[(signer), writable]` The program account to change the size of. + - `[writable]` The program account to change the size of. - `[signer]` The authority of the program. - `[writable]` Optional, the recipient account. - Instruction data: From 03b920daf7572667f5fe3549cfd56a2cc2d7bef1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Fri, 20 Jun 2025 11:23:17 +0200 Subject: [PATCH 13/17] Describes the removal of explicit buffer accounts and the purpose of retract in the "Motivation" section. --- proposals/0167-loader-v4.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 1e13042c6..85a4b3344 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -17,13 +17,25 @@ A new upgradeable loader which only requires a single account per program. ## Motivation Loader-v3, which is currently the only deployable loader, requires two accounts -per program. This was a workaround to circumvent the finality of the -`is_executable` flag, which is removed in SIMD-0162. Consequentially, this -setup of the program account being a proxy account, containing the address of -the actual program data account, is no longer necessary and should be removed. +per program. And has three account types: Program, program-data and buffer. +This was a workaround to circumvent the finality of the `is_executable` flag, +which will be ignored by the program runtime from SIMD-0162 onwards. +Consequentially, this setup of the program account being a proxy account, +containing the address of the actual program data account, is no longer +necessary and should be removed. Likewise the distinction of executable program +accounts and non-executable buffer accounts is also no longer necessary and +should be removed as well. + +In loader-v3 every instruction which modified the program data had to re-verify +the ELF in the end. Instead we are now aiming for a more modular workflow which +explicates these steps in a Retract-Modify-Redeploy sequence of instructions +that allows multiple modifications to share one verification of the ELF in the +end. This sequence can be a single transaction or split across multiple +transactions. Another issue with loader-v3 is that the executable file stored in the programdata account is misaligned relative to the beginning of the account. +This currently requires a copy in the ELF loader to re-align the program. Additionally, there currently is no complete specification of the loaders program management instructions. This proposal would thus fill that gap once From f359def60fdd5a51d928cf71626553224157e48d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Tue, 1 Jul 2025 09:44:47 +0200 Subject: [PATCH 14/17] Factors out loader-v3 to loader-v4 migration into its own SIMD. --- proposals/0167-loader-v4.md | 75 ++----------------------------------- 1 file changed, 3 insertions(+), 72 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 85a4b3344..12283f921 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -62,7 +62,7 @@ The feature gate must: - enable loader-v4 `LoaderV411111111111111111111111111111111111` program management and execution. - enable the loader-v3 `BPFLoaderUpgradeab1e11111111111111111111111` -instruction `UpgradeableLoaderInstruction::Migrate`. +instruction `UpgradeableLoaderInstruction::Migrate` (see SIMD-0315). ### Owned Program Accounts @@ -302,56 +302,6 @@ All program management instructions must cost 2000 CUs. the previous versions program account - Change the status stored in the program account to finalized -### Loader-v3 Instruction: Migrate - -- Instruction accounts: - - `[writable]` The program data account. - - `[writable]` The program account. - - `[signer]` The migration authority. -- Instruction data: - - Enum variant `8u32` -- Behavior: - - Check that there are at least three instruction accounts, - otherwise throw `NotEnoughAccountKeys` - - Check that the program data account is writable, - otherwise throw `InvalidArgument` - - Check that the program data was last modified before the current slot - if the program data has the state `ProgramData`, - otherwise throw `InvalidArgument` - - Check that the provided authority is either: - - the migration authority - (pubkey is `3Scf35jMNk2xXBD6areNjgMtXgp5ZspDhms8vdcbzC42`) - - or the upgrade authority stored in the program data account - - or the program signer if the program is finalized, closed or - uninitialized - otherwise throw `IncorrectAuthority` - - Check that the provided authority is a signer, - otherwise throw `MissingRequiredSignature` - - Check that the program account is writable, - otherwise throw `InvalidArgument` - - Check that the program account is owned by loader-v3, - otherwise throw `IncorrectProgramId` - - Check that the program account has the state `Program`, - otherwise throw `InvalidAccountData` - - Check that the program account points to the program data account, - otherwise throw `InvalidArgument` - - Clear the program account (setting its size to zero) - - Transfer all funds from the program data account to the program account - - Assign ownership of the program account to loader-v4 - - If the program data account was not closed / empty or uninitialized: - - CPI loader-v4 `SetProgramLength` the program account to the size of the - program data account minus the loader-v3 header size and use the migration - authority. - - CPI loader-v4 `Copy` the program data account into the program account - - CPI loader-v4 `Deploy` the program account - - If the program data account was finalized (upgrade authority is `None`): - - CPI loader-v4 `Finalize` without a next version forwarding - - otherwise, if the program data account was not finalized and the - migration authority (as opposed to the upgrade authority) was provided: - - CPI loader-v4 `TransferAuthority` to the upgrade authority - - Clear the program data account (setting its size to zero) - - Assign ownership of the program data account to the system program - ## Impact - This proposal covers all the use cases loader-v3 had but in a cleaner way and @@ -381,27 +331,8 @@ more secure alternative to redeployment / upgrading of programs at the same address. The keypair for the next version linked during finalization should be generated beforehand. - An option to migrate programs from loader-v3 to loader-v4 without changing -their program address will be available via a new loader-v3 instruction. This -will count as a redeployment and thus render the program unavailable for the -rest of the slot (delay visibility). - -Once new programs can not be deployed on loader-v3 anymore, the list of all -loader-v3 programs becomes fixed and can be extracted from a snapshot. Using -the added loader-v3 migration instruction and the global migration authority, -the core protocol developers will then migrate all loader-v3 programs to -loader-v4 programs, which once completed: - -- allows transaction account loading to be simplified, because every program -would load exactly one account, no need to load the proxy account to get to -the actual program data (which is not listed in the transaction accounts). -- allows the removal of the write lock demotion exception if loader-v3 is -present in a transaction. -- corrects the miscounting of the program data account size towards the total -transaction account loading limit. -- allows dApp devs to resuscitate closed loader-v3 programs if they still -control the program authority. This allows redeployment at the same address -or completely closing the program account in order to retrieve the locked -funds. +their program address will be available via a new loader-v3 instruction. (see +SIMD-0315) ## Security Considerations From 0e582c9c93b3b12a9781ba97da6cce85ab661069 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Fri, 4 Jul 2025 14:35:41 +0200 Subject: [PATCH 15/17] Removes the word "delete". --- proposals/0167-loader-v4.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 12283f921..df5f3d8e5 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -204,7 +204,7 @@ All program management instructions must cost 2000 CUs. - Transfer the surplus from the program account to the recipient account - otherwise, if the requested new size is zero throw `InvalidArgument` - If the requested new size is zero: - - Delete the entire program account, including the header + - Set the length of the program account to 0 (removing the header too) - If the requested new size is greater than zero: - Set the length of the program account to the requested new size plus the header size From 0f177e23f6ae1ddbd3edad75442e10644fff34c2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Mon, 7 Jul 2025 16:53:16 +0200 Subject: [PATCH 16/17] Adds "Adding a loader-v3 state" to "Alternatives Considered". --- proposals/0167-loader-v4.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index df5f3d8e5..565294cba 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -45,12 +45,30 @@ See impact for further motivation. ## Alternatives Considered +### Hot swap redeployment + A delay-visibility-free redeployment could be achieved by keeping the swap program around until the end of the slot. This would however mean that two accounts per program must be loaded until the dapp developer reclaims the second one. That would defy the purpose of this proposal which is to get rid of the proxy account. +### Adding a loader-v3 status + +Instead of adding a loader-v3 status a new loader was added for the following +reasons: + +- There is a lot of special casing throughout the validator for loader-v3, +which does not apply to loader-v4. +- The existing instruction interface of loader-v3 (e.g. the distinction +of program account and programdata account) are incompatible with loader-v4. +- Adding a new loader-v3 status would create new unforseen interactions with +the existing status enum and instructions. +- Modifying the existing loader-v3 implementation risks breaking it, while +a independent new loader does not. +- Loader-v3 is undocumented and unspecified. Starting fresh allows to have +a complete implementation, specification and documentation. + ## New Terminology None. From aa7c34f4daa7a5550e3fedc88984f8d55d858ac0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexander=20Mei=C3=9Fner?= Date: Wed, 9 Jul 2025 14:59:00 +0200 Subject: [PATCH 17/17] Suggestions from jstarry, buffalojoec and t-nelson --- proposals/0167-loader-v4.md | 308 +++++++++++++++++++++++------------- 1 file changed, 195 insertions(+), 113 deletions(-) diff --git a/proposals/0167-loader-v4.md b/proposals/0167-loader-v4.md index 565294cba..70b167879 100644 --- a/proposals/0167-loader-v4.md +++ b/proposals/0167-loader-v4.md @@ -7,7 +7,7 @@ category: Standard type: Core status: Review created: 2024-08-15 -feature: 2aQJYqER2aKyb3cZw22v4SL2xMX7vwXBRWfvS4pTrtED +feature: TBD --- ## Summary @@ -22,9 +22,7 @@ This was a workaround to circumvent the finality of the `is_executable` flag, which will be ignored by the program runtime from SIMD-0162 onwards. Consequentially, this setup of the program account being a proxy account, containing the address of the actual program data account, is no longer -necessary and should be removed. Likewise the distinction of executable program -accounts and non-executable buffer accounts is also no longer necessary and -should be removed as well. +necessary and should be removed. In loader-v3 every instruction which modified the program data had to re-verify the ELF in the end. Instead we are now aiming for a more modular workflow which @@ -36,6 +34,8 @@ transactions. Another issue with loader-v3 is that the executable file stored in the programdata account is misaligned relative to the beginning of the account. This currently requires a copy in the ELF loader to re-align the program. +To avoid any other alignment issues all states of the accounts owned by a +loader should have the same layout and only be differentiated by a status enum. Additionally, there currently is no complete specification of the loaders program management instructions. This proposal would thus fill that gap once @@ -69,34 +69,55 @@ a independent new loader does not. - Loader-v3 is undocumented and unspecified. Starting fresh allows to have a complete implementation, specification and documentation. +### Transfering closed accounts to the system program + +While it would be more intuitive to give the program account back to the system +program immediately in the close instruction, this would also allow a program +to reopen itself under a different loader in the same transaction, while it is +still running. In order to prevent this the program account is only cleared and +the rent collection mechanism will then implicitly give the account back to the +system program at the very end of the transaction. + +### Removing the redeployment cooldown + +The cooldown enforces that each pair of an address and a slot can uniquely +identify a version of a program, which simplifies caching logic. Thus this +one slot cooldown will be kept in loader-v4. + +### Inferring the program length from the ELF header + +There are two issues with that: First, a ELF header does not know how long +its file is, it has to be parsed and calculated from offsets of various +sections, which increases the validator implementation complexity. Second, +this would require the ELF header to always be uploaded first and always in a +single chunk, wich increases the complexity of the uploading logic. Instead we +choose the user having to announce the length of the program to be uploaded +explicitly before the actual upload starts. + ## New Terminology -None. +The _current slot_ is as in the Clock sysvar. ## Detailed Design -The feature gate must: - -- enable loader-v4 `LoaderV411111111111111111111111111111111111` program -management and execution. -- enable the loader-v3 `BPFLoaderUpgradeab1e11111111111111111111111` -instruction `UpgradeableLoaderInstruction::Migrate` (see SIMD-0315). +The feature gate must enable loader-v4 program management and execution. ### Owned Program Accounts Accounts of programs owned by loader-v4 must have the following layout: - Header (which is 48 bytes long): + - `u64` status enum: + - Enum variant `0u64`: `Invalid`, account was zero-filled externally + - Enum variant `1u64`: `NeverBeenDeployed`, used as write buffer + - Enum variant `2u64`: `Retracted`, program is in maintenance + - Enum variant `3u64`: `Deployed`, program is ready to be executed + - Enum variant `4u64`: `Finalized`, same as `Deployed`, but can not be + modified anymore - `u64` Slot in which the program was last deployed, retracted or initialized. - `[u8; 32]` Authority address which can send program management - instructions. Or if the status is finalized, then the address of the next - version of the program. - - `u64` status enum: - - Enum variant `0u64`: Retracted, program is in maintenance - - Enum variant `1u64`: Deployed, program is ready to be executed - - Enum variant `2u64`: Finalized, same as `Deployed`, but can not be - modified anymore + instructions. - Body: - `[u8]` The programs executable file @@ -111,7 +132,9 @@ otherwise throw `AccountDataTooSmall` otherwise throw `MissingRequiredSignature` - the authority stored in the program account is the one provided, otherwise throw `IncorrectAuthority` -- the status stored in the program account is not finalized, +- the status stored in the program account is not `Invalid`, +otherwise throw `InvalidArgument` +- the status stored in the program account is not `Finalized`, otherwise throw `Immutable` ### Execution / Invocation @@ -120,7 +143,7 @@ Invoking programs owned by loader-v4 checks in the following order that: - the owner of the program account is loader-v4 - the program account is at least as long enough for the header -- the status stored in the program account is not retracted +- the status stored in the program account is `Deployed` or `Finalized` - the program account was not deployed within the current slot (delay visibility) - the executable file stored in the program account passes executable @@ -130,8 +153,6 @@ failing any of the above checks must throw `UnsupportedProgramId`. ### Program Management Instructions -All program management instructions must cost 2000 CUs. - #### Write - Instruction accounts: @@ -142,10 +163,12 @@ All program management instructions must cost 2000 CUs. - `u32` Byte offset at which to write the given bytes - `[u8]` Chunk of the programs executable file - Behavior: + - Charge 32 + chunk_length_in_bytes / cpi_bytes_per_unit CUs - Check there are at least two instruction accounts, otherwise throw `NotEnoughAccountKeys` - Verify the program account - - Check the status stored in the program account is retracted, + - Check the status stored in the program account is `NeverBeenDeployed` or + `Retracted`, otherwise throw `InvalidArgument` - Check that the end offset (sum of offset and length of the chunk) does not exceed the maximum (program account length minus the header size), @@ -158,26 +181,28 @@ All program management instructions must cost 2000 CUs. - Instruction accounts: - `[writable]` The program account to copy to. - `[signer]` The authority of the program. - - `[]` The program(data) account to copy from. + - `[]` The account to copy from. - Instruction data: - Enum variant `1u32` - `u32` Byte offset at which to write - `u32` Byte offset at which to read - `u32` Length of the chunk to copy in bytes - Behavior: + - Charge 32 + chunk_length_in_bytes / cpi_bytes_per_unit CUs - Check there are at least three instruction accounts, otherwise throw `NotEnoughAccountKeys` - Check that program account and source account do not alias, otherwise throw `AccountBorrowFailed` - Verify the program account - - Check the status stored in the program account is retracted, + - Check the status stored in the program account is `NeverBeenDeployed` or + `Retracted`, otherwise throw `InvalidArgument` - - Check that the source account is owned by loader v1, v2, v3 or v4, + - Check that the source account is owned by loader v2, v3 or v4, otherwise throw `InvalidArgument` - and look-up the source header size: - - loader-v1: 0 bytes - loader-v2: 0 bytes - - loader-v3: 45 bytes + - loader-v3 buffer: 37 bytes + - loader-v3 programdata: 45 bytes - loader-v4: 48 bytes - Check that the source end offset (sum of source offset and length) does not exceed the maximum (source account length minus the source header size), @@ -193,44 +218,61 @@ All program management instructions must cost 2000 CUs. - Instruction accounts: - `[writable]` The program account to change the size of. - `[signer]` The authority of the program. - - `[writable]` Optional, the recipient account. - Instruction data: - Enum variant `2u32` - `u32` The new size after the operation. - Behavior: + - Charge 32 + new_size_in_bytes / cpi_bytes_per_unit CUs - Check there are at least two instruction accounts, otherwise throw `NotEnoughAccountKeys` - - If this is an initialization (program account length is too short to - contain the header): - - the owner of the program account is loader-v4, - otherwise throw `InvalidAccountOwner` - - the program account is writable, otherwise throw `InvalidArgument` - - the provided authority (instruction account at index 1) signed, - otherwise throw `MissingRequiredSignature` - - If this is not an initialization: - - Verify the program account - - Check that the status stored in the program account is retracted, - otherwise throw `InvalidArgument` + - Verify the program account + - Check that the status stored in the program account is + `NeverBeenDeployed` or `Retracted`, + otherwise throw `InvalidArgument` - Check that there are enough funds in the program account for rent - exemption, otherwise throw `InsufficientFunds` - - If there are more than enough funds: - - Check there are at least three instruction accounts, + exemption of the new length, + otherwise throw `InsufficientFunds` + - Set the length of the program account to the requested new size plus + the header size + +#### WithdrawExcessLamports + +- Instruction accounts: + - `[writable]` The program account to withdraw from. + - `[signer]` The authority of the program. + - `[]` The recipient account. +- Instruction data: + - Enum variant `3u32` +- Behavior: + - Charge 32 CUs + - Check there are at least three instruction accounts, + otherwise throw `NotEnoughAccountKeys` + - Check that program account and recipient account do not alias, + otherwise throw `AccountBorrowFailed` + - Verify the program account + - Transfer lamports which are not needed for rent exemption from the + program account to the recipient account + +#### WithdrawAllLamports + +- Instruction accounts: + - `[writable]` The program account to withdraw from. + - `[signer]` The authority of the program. + - `[]` The recipient account. +- Instruction data: + - Enum variant `4u32` +- Behavior: + - Charge 32 CUs + - Check there are at least three instruction accounts, otherwise throw `NotEnoughAccountKeys` - - Check that the recipient account (instruction account at index 2) is - writable, otherwise throw `InvalidArgument` - - If a recipient account was provided that is not the program account: - - Transfer the surplus from the program account to the recipient account - - otherwise, if the requested new size is zero throw `InvalidArgument` - - If the requested new size is zero: - - Set the length of the program account to 0 (removing the header too) - - If the requested new size is greater than zero: - - Set the length of the program account to the requested new size plus - the header size - - In case that this is an initialization, also initialize the header: - - Set the `is_executable` flag to `true` - - Set the slot to zero, **not** the current slot - - Set the authority address (from the instruction account at index 1) - - Set the status to retracted + - Check that program account and recipient account do not alias, + otherwise throw `AccountBorrowFailed` + - Verify the program account + - Check that the status stored in the program account is + `NeverBeenDeployed` or `Retracted`, + otherwise throw `InvalidArgument` + - Set the length of the program account to 0 (removing the header too) + - Transfer all lamports from the program account to the recipient account #### Deploy @@ -238,21 +280,23 @@ All program management instructions must cost 2000 CUs. - `[writable]` The program account to deploy. - `[signer]` The authority of the program. - Instruction data: - - Enum variant `3u32` + - Enum variant `5u32` - Behavior: + - Charge 32 CUs - Check there are at least two instruction accounts, otherwise throw `NotEnoughAccountKeys` - Verify the program account - Check that the slot stored in the program account is not the current (deployment cooldown), otherwise throw `InvalidArgument` - - Note: The cooldown enforces that each pair of an address and a slot can - uniquely identify a deployment of a program, which simplifies caching logic. - - Check that the status stored in the program account is retracted + - Check that the status stored in the program account is `NeverBeenDeployed` + or `Retracted` otherwise throw `InvalidArgument` + - Charge program_length_in_bytes / cpi_bytes_per_unit CUs - Check that the executable file stored in the program account passes executable verification - Change the slot in the program account to the current slot - - Change the status stored in the program account to deployed + - Change the status stored in the program account to `Deployed` + - Set the `is_executable` flag to `true` #### Retract @@ -260,98 +304,136 @@ All program management instructions must cost 2000 CUs. - `[writable]` The program account to retract. - `[signer]` The authority of the program. - Instruction data: - - Enum variant `4u32` + - Enum variant `6u32` - Behavior: + - Charge 32 CUs - Check there are at least two instruction accounts, otherwise throw `NotEnoughAccountKeys` - Verify the program account - Check that the slot stored in the program account is not the current (deployment cooldown), otherwise throw `InvalidArgument` - - Check that the status stored in the program account is deployed, + - Check that the status stored in the program account is `Deployed`, otherwise throw `InvalidArgument` - Note: The slot is **not** set to the current slot to allow a retract-modify-redeploy-sequence within the same slot or even within the same transaction. - - Change the status stored in the program account to retracted + - Change the status stored in the program account to `Retracted` + - Set the `is_executable` flag to `false` -#### TransferAuthority +#### SetAuthority - Instruction accounts: - `[writable]` The program account to change the authority of. - `[signer]` The current authority of the program. - - `[signer]` The new authority of the program. + - `[optional(signer)]` The new authority of the program. - Instruction data: - - Enum variant `5u32` + - Enum variant `7u32` - Behavior: + - Charge 32 CUs - Check there are at least three instruction accounts, otherwise throw `NotEnoughAccountKeys` - - Verify the program account + - If this is an initialization (program account length is too short to + contain the header): + - the owner of the program account is loader-v4, + otherwise throw `InvalidAccountOwner` + - the program account is writable, otherwise throw `InvalidArgument` + - Check that there are enough funds in the program account for rent + exemption of the header, + otherwise throw `InsufficientFunds` + - otherwise, if this is not an initialization: + - Verify the program account - Check that the new authority (instruction account at index 2) - signed as well, otherwise throw `MissingRequiredSignature` - - Check that the authority stored in the program account is different - from the one provided, otherwise throw `InvalidArgument` + is either the system program or has signed, + otherwise throw `MissingRequiredSignature` + - If this is an initialization: + - Set the length of the program account to the header size + - Set the slot to zero, **not** the current slot + - Set the status to `NeverBeenDeployed` + - otherwise, if it is not an initialization: + - Check that the authority stored in the program account is different + from the one provided, otherwise throw `InvalidArgument` - Copy the new authority address into the program account + - If the the new authority is the system program: + - Check that the status stored in the program account is `Deployed` or + that the status is `Retracted` and the program length is 0 (header only), + otherwise throw `InvalidArgument` + - Change the status stored in the program account to `Finalized` + +### Workflows + +#### Inital deployment + +- Assign account to loader-v4 +- SetAuthority to the new authority +- SetProgramLength to ELF size +- [Transaction boundary] +- Write chunks repeatedly +- [Transaction boundary] +- Deploy + +#### Redeployment + +- Assign buffer account to loader-v4 +- SetAuthority of the buffer to the new authority +- SetProgramLength of buffer to ELF size +- [Transaction boundary] +- Write chunks repeatedly +- [Transaction boundary] +- Retract program +- SetProgramLength of program to ELF size +- Copy from buffer to program +- Deploy program +- WithdrawLamports of buffer to program +- WithdrawLamports of program + +#### Close: Temporary + +- Retract + +#### Close: Recycle + +- Retract +- WithdrawLamports all + +#### Close: Permanent + +- Retract +- SetProgramLength to 0 +- WithdrawLamports leaving enough for rent expemtion of the header +- SetAuthority to the system program + +#### Transfer authority + +- SetAuthority to the new authority #### Finalize -- Instruction accounts: - - `[writable]` The program account to change the authority of. - - `[signer]` The current authority of the program. - - `[]` Optional, the reserved address for the next version of the program. -- Instruction data: - - Enum variant `6u32` -- Behavior: - - Check there are at least three instruction accounts, - otherwise throw `NotEnoughAccountKeys` - - Verify the program account - - Check that the status stored in the program account is deployed, - otherwise throw `InvalidArgument` - - for the program account of the next version - (instruction account at index 2) check that: - - the owner of the program account is loader-v4, - otherwise throw `InvalidAccountOwner` - - the program account is at least as long enough for the header, - otherwise throw `AccountDataTooSmall` - - the authority stored in the program account is the one provided, - otherwise throw `IncorrectAuthority` - - the status stored in the program account is not finalized, - otherwise throw `Immutable` - - Copy the address of the next version into the next version field stored in - the previous versions program account - - Change the status stored in the program account to finalized +- SetAuthority to the system program ## Impact - This proposal covers all the use cases loader-v3 had but in a cleaner way and comes with a specification. -- loader-v3 had a separate account type for buffers and extra commands for -these buffer accounts, in loader-v4 program accounts can act as buffers, there -is no more distinction. -- loader-v3 deployments always needed a buffer, in loader-v4 it is optional, -one can upload a redeployment into the program account directly. +- loader-v3 had a separate account layout for buffers and extra commands for +these buffer accounts, in loader-v4 they are only differentiated by status. - loader-v3 had two accounts per program, loader-v4 goes back to having only one, thus needs less funds to reach rent exemption. -- loader-v3 closing of programs did finalize them, in loader-v4 all the funds -can be retrieved and the program address repurposed. +- loader-v3 closing of programs did always finalize them, in loader-v4 there +is an option to retrieve all the funds the program address can be repurposed. - loader-v3 programs could only grow, loader-v4 can shrink programs and also retrieve the surplus of funds no longer required for rent exception. - loader-v3 programs were always "live" after the first deployment, with loader-v4 one can temporarily put a program into maintenance mode without a redeployment. -- loader-v3 always required the entire program to be uploaded for a -redeployment, loader-v4 supports partial uploads for patching chunks of the -program. - loader-v3 ELFs were misaligned, loader-v4 properly aligns the executable file relative to the beginning of the account. -- loader-v4 allows finalized programs to mark which other program supersedes -them which can then be offered as an option in frontends. This provides a -more secure alternative to redeployment / upgrading of programs at the same -address. The keypair for the next version linked during finalization should be -generated beforehand. - An option to migrate programs from loader-v3 to loader-v4 without changing their program address will be available via a new loader-v3 instruction. (see SIMD-0315) +Increases in size via `SetProgramLength` will remain limited to 10 KiB in CPI. +Thus, it is reccomended to call `SetProgramLength` as a top-level instruction. + ## Security Considerations None.