-
Notifications
You must be signed in to change notification settings - Fork 335
feat: added regional secret support for secret-manager #3365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: added regional secret support for secret-manager #3365
Conversation
@@ -1,14 +1,19 @@ | |||
package com.google.cloud.spring.autoconfigure.secretmanager; | |||
|
|||
import static org.assertj.core.api.Assertions.assertThatCode; | |||
import static org.junit.jupiter.api.Assertions.assertEquals; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We, in my company, always try to avoid having AssertJ AND JUnit assertions mixed in one class.
At least in my team we prefere to only use AssertJ as it let you chain assertions on same object.
Not sure if there are coding guidlines in Google projects, but maybe you consider to only use one assertions library to keep the mental load low.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've replaced the JUnit Assertion with the AssertJ assertion.
/** | ||
* Defines the region of the secrets when Regional Stack is used. | ||
* | ||
* <p>When not specified, the secret manager will use the Global Stack. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it syntactically correct if there is no closing </p>
tag?
* <p>When not specified, the secret manager will use the Global Stack. | |
* <p>When not specified, the secret manager will use the Global Stack.</p> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've referred other files for the docstrings and there is no use of closing </p>
tag. Following are some of such files:
- https://github.com/GoogleCloudPlatform/spring-cloud-gcp/blob/main/spring-cloud-gcp-core/src/main/java/com/google/cloud/spring/core/DefaultGcpEnvironmentProvider.java#L26
- https://github.com/GoogleCloudPlatform/spring-cloud-gcp/blob/main/spring-cloud-gcp-kms/src/main/java/com/google/cloud/spring/kms/KmsOperations.java#L24
- https://github.com/GoogleCloudPlatform/spring-cloud-gcp/blob/main/spring-cloud-gcp-core/src/main/java/com/google/cloud/spring/core/Credentials.java#L33
Let me know your thoughts on this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, sorry. You are right. I wasn't aware that the closing tag is not mandatory.
ConfigData configData = loader.load(loaderContext, resource); | ||
SecretManagerPropertySource propertySource = | ||
(SecretManagerPropertySource) configData.getPropertySources().get(0); | ||
assertThat(Optional.ofNullable(location)).isEqualTo(propertySource.getLocation()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First, I'm not from Google, so my opinion is truly not more than my opinion. Google may see things different 😉
Now, in assertions you normally test assumtions/expectations on an object. Having this on mind the assertions should be more in this way:
assertThat(objectToTest)
.verificationOne(...)
.verificationTwo(...);
The result is the same, but now it is in a more logical order:
assertThat(propertySource)
.returns(Optional.ofNullable(location), SecretManagerPropertySource::getLocation);
You could add more verifications if suitable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I followed the existing convention used throughout the codebase, where assertions are written in the same order as the current pattern. However, I understand your perspective, and if the team prefers the alternative style you suggested, I’m happy to make the adjustment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do not have a strong preference on the assertion style.
However, in your current code, for better readability please revert actual and expected. isEqualTo()
takes the expected value.
assertThat(secretIdentifier.getProject()).isEqualTo("defaultProject"); | ||
assertThat(secretIdentifier.getLocation()).isEqualTo("us-central1"); | ||
assertThat(secretIdentifier.getSecret()).isEqualTo("the-reg-secret"); | ||
assertThat(secretIdentifier.getSecretVersion()).isEqualTo("latest"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In reference to my other comment, this is how AssertJ chaining performs several checks on the same object:
assertThat(secretIdentifier)
.returns("defaultProject", SecretVersionName::getProject)
.returns("us-central1", SecretVersionName::getLocation)
.returns("the-reg-secret", SecretVersionName::getSecret)
.returns("latest", SecretVersionName::getSecretVersion);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As mentioned in the previous comment, I opted to maintain consistency with other assertions used across the project.
* | ||
* <p>When not specified, the secret manager will use the Global Stack. | ||
*/ | ||
private Optional<String> location = Optional.empty(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I think your PR (or the feature you're bringing) is totally cool, I've taken a look at the code in its entirety today, not just the delta shown up here in 'Files changed' view.
May I ask questions about the way you've implemented the 'location'?
The first one is, why as Optional
? I like Optional
s, but in this specific case, I don't see a great deal of added value.
- Similarly to
projectId
, thelocation
can be unset. TheprojectId
in the existing code is implemented as a simpleString
, and thelocation
asOptional
. In terms of code style (not in the sense of arrangement), I would tend to follow the direction of the rest of the code (String
). - There are no mappings (
.map(...)
) of theOptional
type taking place. The only use case is the check forisPresent()
. This could also be implemented - similarly toprojectId
- with an IF condition.
The other is, if the Optional
type is still to be used, could the variable at least be renamed accordingly?
- In the spring-cloud-gcp library, it's common practice (although this is just a gut feeling and not backed up by evidence) to work with
String
s or special objects (SecretVersionName
,LocationName
, ...). So, if anOptional
is introduced, the variable name should somehow reflect this. Although I must admit that this is debatable. I know many voices that say technical implementation details shouldn't be included in variable names. - One could, of course, argue that every IDE displays the return type of
getLocation()
asOptional<String>
. However, if you look at the code without an IDE (as is the case here on GitHub), the return type is not immediately apparent and expectations are set. Providing a hint through variable names could at least be considered.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The use of Optional
here is to explicitly signal that location may or may not be present, which I believe makes the intent clearer than a plain String. While it's true we're not using .map()
or chaining operations now, Optional
forces consumers to consider the absence of location explicitly (rather than relying on null checks). This can help avoid future bugs or assumptions about the field always being set.
I understand your suggestion regarding variable naming, but I believe the intent is already clear from the type itself (Optional<String>
), and I prefer to keep the variable name focused on the domain rather than the technical implementation detail.
Let me know your thoughts, and I'm happy to make further adjustments if needed!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Optional
is not recommended in Spring Boot for ConfigurationProperties
. The are typically not used directly anyway. Source: spring-projects/spring-boot#21868
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Additionally, from source quoted above, behavior for Optional properties might not be expected
"if you do declare an Optional
property and it has no value, null
rather than an empty Optional
will be bound."
ref: spring-projects/spring-boot@7b3c0a9
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regarding the use of Optional
, same thoughts here.
As in SecretManagerPropertyUtils is already a handling regarding if location
is set or not:
https://github.com/abheda-crest/spring-cloud-gcp/blob/9f6bf3899938e195810b5f1cd2c585dd3be8e98d/spring-cloud-gcp-secretmanager/src/main/java/com/google/cloud/spring/secretmanager/SecretManagerPropertyUtils.java#L86-L93
Maybe you could ease things here if leaving location
as String
. Even in getProperty(String)
is no special handling necessary (is done in SecretManagerPropertyUtils)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The getProperty()
method will still require special handling to account for different cases when retrieving the property value. Specifically, I have added an additional check for location, which is used to form the version name based on the location value. This logic will still be necessary, even if we keep the location as a String.
@mieseprem I've addressed the review comments. Could you please review these and let me know if those looks good? |
Hej, I am pleased that my opinion is so valued. But the following also applies:
So, I had already made my opinion known and I think now someone from Google would either have to give their approval or make some comments |
@burkedavison Could you please review this PR? |
@@ -42,6 +42,7 @@ This can be overridden using the authentication properties. | |||
| `spring.cloud.gcp.secretmanager.credentials.location` | OAuth2 credentials for authenticating to the Google Cloud Secret Manager API. | No | By default, infers credentials from https://cloud.google.com/docs/authentication/production[Application Default Credentials]. | |||
| `spring.cloud.gcp.secretmanager.credentials.encoded-key` | Base64-encoded contents of OAuth2 account private key for authenticating to the Google Cloud Secret Manager API. | No | By default, infers credentials from https://cloud.google.com/docs/authentication/production[Application Default Credentials]. | |||
| `spring.cloud.gcp.secretmanager.project-id` | The default Google Cloud project used to access Secret Manager API for the template and property source. | No | By default, infers the project from https://cloud.google.com/docs/authentication/production[Application Default Credentials]. | |||
| spring.cloud.gcp.secretmanager.location | Defines the region of the Secret Manager where your secrets are stored, specifically when using a regional stack. This option is particularly useful for applications that need to access secrets from a specific geographical location. | No | By default, the global stack will be utilized. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: s/spring.cloud.gcp.secretmanager.location/spring.cloud.gcp.secretmanager.location
/
@@ -48,6 +52,7 @@ static SecretVersionName getSecretVersionName( | |||
if (usedPrefix.isEmpty() || isAttemptingFullStringMatch) { | |||
return null; | |||
} | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: remove this empty line
} | ||
|
||
static SecretVersionName getSecretVersionName( | ||
final String input, GcpProjectIdProvider projectIdProvider, String location) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think of using Optional<String> location
as input here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I have changed it to Optional<String>
static SecretVersionName getSecretVersionName( | ||
String projectId, String secretId, String version, String location) { | ||
if (location != null) { | ||
return SecretVersionName.newProjectLocationSecretSecretVersionBuilder().setLocation(location).setProject(projectId).setSecret(secretId).setSecretVersion(version).build(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To confirm, location needs to be String here, right? What do you think of having location to be Optional<String>
everywhere else until we reach this line?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, location
needs to be a String
here, but I have modified the code to use Optional<String>
even after this point for consistency. Let me know if you have any further suggestions!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the PR. Left a few comments. PTAL.
|
||
@Override | ||
public SecretManagerServiceClient getClient() { | ||
return clientCache.computeIfAbsent("", loc -> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you just re-use getClient(String)
with some modification to avoid code duplication?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated
public SecretManagerTemplate( | ||
SecretManagerServiceClient secretManagerServiceClient, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a breaking change. Consider just adding an additional constructor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added the additional constructor and also tested with sample application to check the backward compatibility.
@@ -60,22 +58,14 @@ public GcpSecretManagerAutoConfiguration( | |||
|
|||
@Bean | |||
@ConditionalOnMissingBean | |||
public SecretManagerServiceClient secretManagerClient() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Customers may already be using this one or defining a custom secretManagerClient
bean and expecting it to be used. Consider leaving this bean and passing it down into clientFactory
. I understand that it might not be used for apps that only use regional secrets, but it would be more backwards-compatible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated. Also checked sample application by using the SecretManagerServiceClient
with the SecretManagerTemplate
.
@ConditionalOnMissingBean | ||
public SecretManagerTemplate secretManagerTemplate( | ||
SecretManagerServiceClient client, SecretManagerServiceClientFactory clientFactory) { | ||
if (clientFactory != null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you sure Spring Boot will not throw an an exception if clientFactory
bean does not exist? I believe you have to use ObjectProvider
in the method signature.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I have changed it with ObjectProvider
|
||
defaultBootstrapContext.register(GcpSecretManagerProperties.class, | ||
BootstrapRegistry.InstanceSupplier.of(properties)); | ||
defaultBootstrapContext.register(CredentialsProvider.class, | ||
BootstrapRegistry.InstanceSupplier.of(credentialsProvider)); | ||
defaultBootstrapContext.register(SecretManagerServiceClient.class, | ||
BootstrapRegistry.InstanceSupplier.of(secretManagerServiceClient)); | ||
defaultBootstrapContext.register(SecretManagerServiceClientFactory.class, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we also have a test without this bean registered?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added the test cases
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jinseopkim0 @zhumin8 Please do a second pass.
@zhumin8 I have resolved the merge conflict. Can you please review the PR? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good in general, commented on some smaller points. Will do another pass.
...gle/cloud/spring/autoconfigure/secretmanager/GcpSecretManagerAutoConfigurationUnitTests.java
Outdated
Show resolved
Hide resolved
.../google/cloud/spring/autoconfigure/secretmanager/SecretManagerConfigDataLoaderUnitTests.java
Outdated
Show resolved
Hide resolved
...ing-cloud-gcp-secretmanager-sample/src/main/java/com/example/SecretManagerWebController.java
Outdated
Show resolved
Hide resolved
...er-sample/src/test/java/com/example/SecretManagerRegionalSampleTemplateIntegrationTests.java
Outdated
Show resolved
Hide resolved
/** | ||
* Default value for the latest version of the secret. | ||
*/ | ||
public static final String GLOBAL_LOCATION = "global"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
GLOBAL_LOCATION seems only used within this class, does it needs to be public? We try to restrict public access when possible to avoid future breaking changes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Used it in DefaultSecretManagerServiceClientFactory.java
and removed the duplicate constant in that file.
/** | ||
* Default value for the latest version of the secret. | ||
*/ | ||
public static final String GLOBAL_LOCATION = "global"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
GLOBAL_LOCATION seems only used within this class, does it needs to be public? We try to restrict public access when possible to avoid future breaking changes.
Also see this defined separately in SecretManagerTemplate below
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Used constant from the SecretManagerTemplate
...oogle/cloud/spring/autoconfigure/secretmanager/DefaultSecretManagerServiceClientFactory.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@abheda-crest Sorry for the inconvenience, but do you mind opening a new pr for this change once done with issue below? I had to revert this after merging in.
registerBean(context, GcpSecretManagerProperties.class, getSecretManagerProperties(context)); | ||
// Register the CredentialsProvider. | ||
registerBean(context, CredentialsProvider.class, getCredentialsProvider(context)); | ||
// Register the Secret Manager client factory. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@abheda-crest Sorry this was missed in earlier reviews.
Merging this PR caused IT tests failures across modules. I have reverted this change. More context here
I believe the root cause is here: This SecretManagerConfigDataLocationResolver is used regardless if secretmanager dependency is present. When not present, for example if you run the vision sample app, this code createSecretManagerServiceClientFactory()
calls DefaultSecretManagerServiceClientFactory that eventually gives "NoClassDefFoundError" for "com/google/cloud/spring/secretmanager/SecretManagerServiceClientFactory"
Note that secretmanager is optional dependency for autoconfig. You will need to guard this logic to only run when secretmanager is present.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have fixed the issue
@@ -62,6 +63,10 @@ public class SecretManagerWebController { | |||
@Value("${sm@application-secret:DEFAULT}") | |||
private String appSecretFromValue; | |||
|
|||
// Application secrets with regions can be accessed with this syntax. | |||
@Value("${sm@locations/us-central1/application-secret:DEFAULT}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I’ve updated the README.
Yes, global and regional secrets can coexist in the same project. The choice depends on the organization's data residency needs — global secrets use a single endpoint, while regional secrets are location-specific for compliance reasons.
I have raised another PR after fixing the issue: #3746 |
Issue: #3331
Description:
Added the support for regional secret creation/updation, deletion and fetch for secret manager service.
Note: Fixed the integration test
testUpdateSecrets
in the filesecretmanager/it/SecretManagerTemplateIntegrationTests.java
Performed the below mentioned manual unit tests to validate the working of the global and regional secret operations.
@Value
annotationMore information about regional secrets: https://cloud.google.com/secret-manager/regional-secrets/data-residency
note: this PR is later reverted by #3734
BEGIN_COMMIT_OVERRIDE
chore: a placeholder so this does not show up in release notes by release please.
END_COMMIT_OVERRIDE