Skip to content

OCPBUGS-56008: default Azure to create VM user-assigned identities #9718

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 21, 2025
Merged

OCPBUGS-56008: default Azure to create VM user-assigned identities #9718

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
19a86cc
OCPBUGS-56008: default to create VM Identity
patrickdillon May 19, 2025
11c4d5d
OCPBUGS-56008: restore azure identity creation
patrickdillon May 19, 2025
c3d4af2
OCPBUGS-56008: azure cloud config identity
patrickdillon May 19, 2025
8e54931
vendor: Azure packages for identity creation
patrickdillon May 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 16 additions & 8 deletions data/data/install.openshift.io_installconfigs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,14 +249,16 @@ spec:
host.
type: boolean
identity:
default: None
default: UserAssigned
description: |-
Identity is the type of identity used for the virtual machine.
The type 'SystemAssigned' is an implicitly created identity.
The generated identity will be assigned a Subscription contributor role.
The type 'UserAssigned' is a standalone Azure resource provided by the user
and assigned to the VM
and assigned to the VM.
Identity can only be set for control-plane nodes.
The default identity is a user-assigned identity, generated by the installer.
This default is expected to change in future releases.
properties:
type:
description: Type specifies the type of identity to be
Expand Down Expand Up @@ -1571,14 +1573,16 @@ spec:
VM host.
type: boolean
identity:
default: None
default: UserAssigned
description: |-
Identity is the type of identity used for the virtual machine.
The type 'SystemAssigned' is an implicitly created identity.
The generated identity will be assigned a Subscription contributor role.
The type 'UserAssigned' is a standalone Azure resource provided by the user
and assigned to the VM
and assigned to the VM.
Identity can only be set for control-plane nodes.
The default identity is a user-assigned identity, generated by the installer.
This default is expected to change in future releases.
properties:
type:
description: Type specifies the type of identity to
Expand Down Expand Up @@ -2832,14 +2836,16 @@ spec:
host.
type: boolean
identity:
default: None
default: UserAssigned
description: |-
Identity is the type of identity used for the virtual machine.
The type 'SystemAssigned' is an implicitly created identity.
The generated identity will be assigned a Subscription contributor role.
The type 'UserAssigned' is a standalone Azure resource provided by the user
and assigned to the VM
and assigned to the VM.
Identity can only be set for control-plane nodes.
The default identity is a user-assigned identity, generated by the installer.
This default is expected to change in future releases.
properties:
type:
description: Type specifies the type of identity to be
Expand Down Expand Up @@ -4567,14 +4573,16 @@ spec:
host.
type: boolean
identity:
default: None
default: UserAssigned
description: |-
Identity is the type of identity used for the virtual machine.
The type 'SystemAssigned' is an implicitly created identity.
The generated identity will be assigned a Subscription contributor role.
The type 'UserAssigned' is a standalone Azure resource provided by the user
and assigned to the VM
and assigned to the VM.
Identity can only be set for control-plane nodes.
The default identity is a user-assigned identity, generated by the installer.
This default is expected to change in future releases.
properties:
type:
description: Type specifies the type of identity to be
Expand Down
2 changes: 2 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,12 @@ require (
github.com/Azure/azure-sdk-for-go v68.0.0+incompatible
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.14.0
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v3 v3.0.0-beta.2
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4 v4.2.1
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5 v5.7.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/dns/armdns v1.2.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/keyvault/armkeyvault v1.4.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi v1.2.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork v1.1.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v2 v2.2.1
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/privatedns/armprivatedns v1.2.0
Expand Down
4 changes: 4 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 h1:tfLQ34V6F7tVSwoTf/4lH
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0/go.mod h1:9kIvujWAA58nmPmWB1m23fyWic1kYZMxD9CxaWn4Qpg=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 h1:ywEEhmNahHBihViHepv3xPBn1663uRv2t2q/ESv9seY=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0/go.mod h1:iZDifYGJTIgIIkYRNWPENUnqx6bJ2xnSDFI2tjwZNuY=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v3 v3.0.0-beta.2 h1:qiir/pptnHqp6hV8QwV+IExYIf6cPsXBfUDUXQ27t2Y=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/authorization/armauthorization/v3 v3.0.0-beta.2/go.mod h1:jVRrRDLCOuif95HDYC23ADTMlvahB7tMdl519m9Iyjc=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4 v4.2.1 h1:UPeCRD+XY7QlaGQte2EVI2iOcWvUYA2XY8w5T/8v0NQ=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4 v4.2.1/go.mod h1:oGV6NlB0cvi1ZbYRR2UN44QHxWFyGk+iylgD0qaMXjA=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5 v5.7.0 h1:LkHbJbgF3YyvC53aqYGR+wWQDn2Rdp9AQdGndf9QvY4=
Expand All @@ -47,6 +49,8 @@ github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/keyvault/armkeyvault v1.4.
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/keyvault/armkeyvault v1.4.0/go.mod h1:StGsLbuJh06Bd8IBfnAlIFV3fLb+gkczONWf15hpX2E=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/managementgroups/armmanagementgroups v1.0.0 h1:pPvTJ1dY0sA35JOeFq6TsY2xj6Z85Yo23Pj4wCCvu4o=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/managementgroups/armmanagementgroups v1.0.0/go.mod h1:mLfWfj8v3jfWKsL9G4eoBoXVcsqcIUTapmdKy7uGOp0=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi v1.2.0 h1:z4YeiSXxnUI+PqB46Yj6MZA3nwb1CcJIkEMDrzUd8Cs=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/msi/armmsi v1.2.0/go.mod h1:rko9SzMxcMk0NJsNAxALEGaTYyy79bNRwxgJfrH0Spw=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork v1.1.0 h1:QM6sE5k2ZT/vI5BEe0r7mqjsUSnhVBFbOsVkEuaEfiA=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork v1.1.0/go.mod h1:243D9iHbcQXoFUtgHJwL7gl2zx1aDuDMjvBZVGr2uW0=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/v2 v2.2.1 h1:bWh0Z2rOEDfB/ywv/l0iHN1JgyazE6kW/aIA89+CEK0=
Expand Down
11 changes: 11 additions & 0 deletions pkg/asset/machines/azure/azuremachines.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,17 @@ func GenerateMachines(clusterID, resourceGroup, subscriptionID string, session *
for i, id := range mpool.Identity.UserAssignedIdentities {
userAssignedIdentities[i] = capz.UserAssignedIdentity{ProviderID: id.ProviderID()}
}

// If identity type is UserAssigned, but no identities are provided, the installer
// will create one. Populate the manifest with a reference to that identity.
if mpool.Identity.Type == capz.VMIdentityUserAssigned && len(userAssignedIdentities) == 0 {
userAssignedIdentities = []capz.UserAssignedIdentity{
{
ProviderID: fmt.Sprintf("/subscriptions/%s/resourcegroups/%s/providers/Microsoft.ManagedIdentity/userAssignedIdentities/%s-identity", subscriptionID, resourceGroup, clusterID),
},
}
}

storageAccountName := aztypes.GetStorageAccountName(clusterID)

defaultDiag := &capz.Diagnostics{
Expand Down
9 changes: 6 additions & 3 deletions pkg/asset/machines/azure/machines.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"sigs.k8s.io/cluster-api-provider-azure/api/v1beta1"
capz "sigs.k8s.io/cluster-api-provider-azure/api/v1beta1"

v1 "github.com/openshift/api/config/v1"
machinev1 "github.com/openshift/api/machine/v1"
Expand Down Expand Up @@ -216,6 +216,9 @@ func provider(platform *azure.Platform, mpool *azure.MachinePool, osImage string
managedIdentity := ""
if len(mpool.Identity.UserAssignedIdentities) > 0 {
managedIdentity = mpool.Identity.UserAssignedIdentities[0].ProviderID()
} else if mpool.Identity.Type == capz.VMIdentityUserAssigned {
// In this case, the installer will create the user-assigned identity.
managedIdentity = fmt.Sprintf("%s-identity", clusterID)
}

var diskEncryptionSet *machineapi.DiskEncryptionSetParameters
Expand Down Expand Up @@ -299,11 +302,11 @@ func getBootDiagnosticObject(diag *azure.BootDiagnostics, cloudName string, role
}
return nil
}
if diag.Type == v1beta1.DisabledDiagnosticsStorage {
if diag.Type == capz.DisabledDiagnosticsStorage {
return nil
}
bootDiagnostics := &machineapi.AzureDiagnostics{Boot: &machineapi.AzureBootDiagnostics{}}
if diag.Type == v1beta1.ManagedDiagnosticsStorage {
if diag.Type == capz.ManagedDiagnosticsStorage {
bootDiagnostics.Boot.StorageAccountType = machineapi.AzureManagedAzureDiagnosticsStorage
} else {
bootDiagnostics.Boot.StorageAccountType = machineapi.CustomerManagedAzureDiagnosticsStorage
Expand Down
2 changes: 1 addition & 1 deletion pkg/asset/machines/clusterapi.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ func (c *ClusterAPI) Generate(ctx context.Context, dependencies asset.Parents) e
}
c.FileList = append(c.FileList, bootstrapAWSMachine...)
case azuretypes.Name:
mpool := defaultAzureMachinePoolPlatform()
mpool := defaultAzureMachinePoolPlatform(installConfig.Config.Platform.Azure.CloudName)
mpool.InstanceType = azuredefaults.ControlPlaneInstanceType(
installConfig.Config.Platform.Azure.CloudName,
installConfig.Config.Platform.Azure.Region,
Expand Down
2 changes: 1 addition & 1 deletion pkg/asset/machines/master.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -339,7 +339,7 @@ func (m *Master) Generate(ctx context.Context, dependencies asset.Parents) error
}
openstack.ConfigMasters(machines, clusterID.InfraID)
case azuretypes.Name:
mpool := defaultAzureMachinePoolPlatform()
mpool := defaultAzureMachinePoolPlatform(installConfig.Config.Platform.Azure.CloudName)
mpool.InstanceType = azuredefaults.ControlPlaneInstanceType(
installConfig.Config.Platform.Azure.CloudName,
installConfig.Config.Platform.Azure.Region,
Expand Down
11 changes: 8 additions & 3 deletions pkg/asset/machines/worker.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,13 +114,18 @@ func defaultAWSMachinePoolPlatform(poolName string) awstypes.MachinePool {
}
}

func defaultAzureMachinePoolPlatform() azuretypes.MachinePool {
func defaultAzureMachinePoolPlatform(env azuretypes.CloudEnvironment) azuretypes.MachinePool {
idType := capz.VMIdentityUserAssigned
if env == azuretypes.StackCloud {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we special casing stackcloud? won't this then break stackcloud? 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Azure Stack has never used node-attached identities, so I think we're safe here. TBH, I'm not certain why we never attached them, but the current state of this PR would continue the existing behavior we already have.

idType = capz.VMIdentityNone
}

return azuretypes.MachinePool{
OSDisk: azuretypes.OSDisk{
DiskSizeGB: powerOfTwoRootVolumeSize,
DiskType: azuretypes.DefaultDiskType,
},
Identity: &azuretypes.VMIdentity{Type: capz.VMIdentityNone},
Identity: &azuretypes.VMIdentity{Type: idType},
}
}

Expand Down Expand Up @@ -486,7 +491,7 @@ func (w *Worker) Generate(ctx context.Context, dependencies asset.Parents) error
machineSets = append(machineSets, set)
}
case azuretypes.Name:
mpool := defaultAzureMachinePoolPlatform()
mpool := defaultAzureMachinePoolPlatform(installConfig.Config.Platform.Azure.CloudName)
mpool.InstanceType = azuredefaults.ComputeInstanceType(
installConfig.Config.Platform.Azure.CloudName,
installConfig.Config.Platform.Azure.Region,
Expand Down
11 changes: 11 additions & 0 deletions pkg/asset/manifests/azure/cloudproviderconfig.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ type CloudProviderConfig struct {
SubnetName string
ResourceManagerEndpoint string
ARO bool
UseManagedIdentity bool
}

// JSON generates the cloud provider json config for the azure platform.
Expand Down Expand Up @@ -57,6 +58,16 @@ func (params CloudProviderConfig) JSON() (string, error) {
ExcludeMasterFromStandardLB: &excludeMasterFromStandardLB,
}

if params.UseManagedIdentity {
config.UseManagedIdentityExtension = true
// The cloud provider needs the clientID which is only known after terraform has run.
// When left empty, the existing managed identity on the VM will be used.
// By leaving it empty, we don't have to create the identity before running the installer.
// We only need to know that there will be one assigned to the VM, and we control this.
// ref: https://github.com/kubernetes/kubernetes/blob/4b7c607ba47928a7be77fadef1550d6498397a4c/staging/src/k8s.io/legacy-cloud-providers/azure/auth/azure_auth.go#L69
config.UserAssignedIdentityID = ""
}

if params.CloudName == azure.StackCloud {
config.authConfig.ResourceManagerEndpoint = params.ResourceManagerEndpoint
config.UseInstanceMetadata = false
Expand Down
1 change: 1 addition & 0 deletions pkg/asset/manifests/cloudproviderconfig.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@ func (cpc *CloudProviderConfig) Generate(ctx context.Context, dependencies asset
VirtualNetworkName: vnet,
SubnetName: subnet,
ResourceManagerEndpoint: installConfig.Config.Azure.ARMEndpoint,
UseManagedIdentity: installConfig.Config.CreateAzureIdentity(),
}.JSON()
if err != nil {
return errors.Wrap(err, "could not create cloud provider config")
Expand Down
17 changes: 17 additions & 0 deletions pkg/infrastructure/azure/azure.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,23 @@ func (p *Provider) InfraReady(ctx context.Context, in clusterapi.InfraReadyInput
p.clientOptions = opts
p.computeClientOptions = computeClientOpts

if err = handleIdentity(ctx, identityInput{
installConfig: installConfig,
region: platform.Region,
resourceGroupName: p.ResourceGroupName,
subscriptionID: subscriptionID,
tokenCredential: tokenCredential,
infraID: in.InfraID,
clientOpts: p.clientOptions,
tags: p.Tags,
}); err != nil {
errMsg := "error creating user-assigned identity: please ensure your user credentials " +
"have the User Access Admin Role or if you are not utilizing an Azure Container Registry " +
"you can set installconfig.platform.azure.defaultMachinePlatform.identity: None to skip " +
"the creation of the identity: creation failed with: %w"
return fmt.Errorf(errMsg, err)
}

// Creating a dummy nsg for existing vnets installation to appease the ingress operator.
if in.InstallConfig.Config.Azure.VirtualNetwork != "" {
networkClientFactory, err := armnetwork.NewClientFactory(subscriptionID, tokenCredential, p.clientOptions)
Expand Down
Loading