Migration from v3 to V4

[jt-pt-dev]

Question

I am struggling a little with the migration from v3 to v4. I understand consulting is a paid for option and that is fair, but will a migration doc be released at some point?

My solution followed the Quickstart docs pretty closely for v3. Now with v4 I have changed ApiResources to ApiScopes (as I was getting an 'invalid_scope' error) but now I get 'Audience validation error' when the MVC app calls the Api. Database migrations were run based on RockSolidKnowledge/IdentityServer4.Migration.Scripts but only needed the new columns in DeviceCodes and PersistedGrants as Api, Clients and Identity resources are run in memory.

[StefanKoenigMUC]

would also be interested for a migration guide.

[brockallen]

but now I get 'Audience validation error'

https://identityserver4.readthedocs.io/en/latest/topics/resources.html#apis

[leastprivilege]

OK - I pinned this issue.

Please report all migration issues you run into. This can be then used to create a guide at some point. This is a community effort.

[jt-pt-dev]

Thank you Brock and Dominick for your replies. I have my solution working now. I will try and summarise my experience in case it helps others or in case I am "doing it wrong" and others can learn.

My system has multiple clients (currently ASP.NET Core web applications and in the future mobile apps). They are related apps in a safety management system for mining/construction/engineering scenarios. Each client basically has its own API. They are relatively simple and small in size. IdentityServer is used for authentication for each app and API and I am using ASPNET Identity. It very similar to the Quickstart examples.

I keep Identity resources, API resources and Clients in Config.cs and keep the Operational store in SQL Server. I do this as the resources and clients will change very infrequently and all apps and APIs and IdentityServer will live on the one server.

The relevant part of my v3 Config.cs was like the docs:

public IEnumerable<ApiResource> Apis =>
	new ApiResource[]
	{
		new ApiResource(configuration["ApiNames:MyActionsApi"], "MyActions API")
	};

The relevant part of my v3 Startup.cs was:

var builder = services.AddIdentityServer()
			.AddInMemoryIdentityResources(config.Ids)
			.AddInMemoryApiResources(config.Apis)
			.AddInMemoryClients(config.Clients)
			.AddOperationalStore(options =>
			{
				options.ConfigureDbContext = b => b.UseSqlServer(Configuration.GetConnectionString("DefaultConnection"));
				options.EnableTokenCleanup = true;
			})
			.AddAspNetIdentity<ApplicationUser>();

I upgraded the Nuget packages to v4 and ran the appropriate migrations scripts from https://github.com/RockSolidKnowledge/IdentityServer4.Migration.Scripts/tree/CreateScripts for my database.

The first problem I had was IdentityServer returned an 'invalid_scope' error. After reading the docs I changed my Config.cs Apis from ApiResource to ApiScope. This solved the 'invalid_scope' error but authentication at the API was failing with 'Audience validation error'. This occurs because "When using the scope-only model, no aud (audience) claim will be added to the token". (https://identityserver4.readthedocs.io/en/latest/topics/resources.html#apis)

I then changed to include BOTH ApiScope and ApiResource and this solved the problem. I used the same value for Scope name and Resource name. I don't know if this is the "right" solution but it works so far for my simple project.

The relevant part of my v4 Config.cs:

public IEnumerable<ApiScope> ApiScopes =>
	new ApiScope[]
	{
		new ApiScope(configuration["ApiNames:MyActionsApi"], "MyActions API")
	};

public IEnumerable<ApiResource> ApiResources =>
	new ApiResource[]
	{
		new ApiResource(configuration["ApiNames:MyActionsApi"], "MyActions API")
		{
			Scopes = { configuration["ApiNames:MyActionsApi"] }
		}
	};

The relevant part of my v4 Startup.cs:

var builder = services.AddIdentityServer()
			.AddInMemoryIdentityResources(config.Ids)
			.AddInMemoryApiScopes(config.ApiScopes)
			.AddInMemoryApiResources(config.ApiResources)
			.AddInMemoryClients(config.Clients)
			.AddOperationalStore(options =>
			{
				options.ConfigureDbContext = b => b.UseSqlServer(Configuration.GetConnectionString("DefaultConnection"));
				options.EnableTokenCleanup = true;
			})
			.AddAspNetIdentity<ApplicationUser>();

Hope that helps and I am not leading people in the wrong direction.

[bryantlikes]

I have a fairly simple API so I just switched out the ApiResource for ApiScope and it worked fine. The update blurb was helpful but it would be nice if it was more visible instead of hidden at the bottom of that page.

I also ran into an issue where the ClientId property was removed from the AuthorizationRequest model and replaced with Client (which has the missing ClientId property) so that was a fairly easy fix. I create a PR for the documentation which still has ClientId in it even though it isn't there.

Everything else seems to be working fine.

[zyofeng]

One of the issue I am encountering, is that version 4.0 is checking Content-Type header for "application/x-www-form-urlencoded"

I'm calling token endpoint from Axios in React Native 0.62 and there are some compatiblity issues with Flipper causing Content-Type header to be stripped out. I understand it's not specifically a bug but it would be nice if there is a way to disable HasApplicationFormContentType checking. Additionally I have legacy client apps deployed without using Content-Type header which prevents upgrade to version 4.

[hallidev]

Few issues I've run into that I'm unsure how to handle:

1:

Current broken code:

await _interaction.GrantConsentAsync(context, ConsentResponse.Denied);

ConsentResponse.Denied no longer exists, but I'm unsure how to construct a ConsentResponse that means Denied. The old implementation of Denied looked like this:

public static ConsentResponse Denied = new ConsentResponse();

Is an empty ConsentResponse still the equivalent of Denied?

2:

Current broken code:

await HttpContext.SignInAsync(user.Id, user.UserName, props);

I notice that there is now an overload that takes IdentityServerUser, but I'm unfamiliar with this class and how to get an instance of it. I see that there's a constructor that takes sub. Is it enough to just do var user = new IdentityServerUser(user.Id) and pass that to SignInAsync?

3:

Current broken code:

await _events.RaiseAsync(new ConsentGrantedEvent(User.GetSubjectId(), request.Client.ClientId, request.ScopesRequested, grantedConsent.ScopesValuesConsented, grantedConsent.RememberConsent));

There used to be a ScopesRequested property on AuthorizationRequest. I'm not sure what the intended replacement is.

4: Above, @jt-pt-dev links to some SQL scripts for updating the database, but are there Entity Framework migration files available? If so, how do we use them?

[brockallen]

ConsentResponse.Denied no longer exists, but I'm unsure how to construct a ConsentResponse that means Denied.

There's a new DenyConsent API.

[hallidev]

@brockallen Gotcha - I will take a look at that. Are there simple answers for my other questions? Was just hoping to get some guidance on the breaking changes.

[srinijss]

I am getting the error "Unable to resolve service for type - 'IdentityServer4.Services.IReplayCache' when i run my IdentityServer project after upgrading IdentityServer4 nuget package from 3.1.0 to 4.0.1. Please let me know if you need more details on this. Appreciate any help on this. Would also be interested for a migration guide. Thanks.

[mikegoatly]

@hallidev I can help with this question:

2:

Current broken code:

await HttpContext.SignInAsync(user.Id, user.UserName, props);

I notice that there is now an overload that takes IdentityServerUser, but I'm unfamiliar with this class and how to get an instance of it. I see that there's a constructor that takes sub. Is it enough to just do var user = new IdentityServerUser(user.Id) and pass that to SignInAsync?

If you look at the updated quickstart sample you can see an example of how to construct IdentityServerUser here.

I suspect you'll be able to answer some of your other questions by looking through the updated quickstart code as well, but I've not had to cross those particular bridges 😄

[Omotayad]

Hi,
I am currently facing this issue after migration, whenever I call the token endpoint. Am i missing a library or configuration?

An exception was thrown while activating IdentityServer4.Endpoints.TokenEndpoint -> IdentityServer4.Validation.ClientSecretValidator -> IdentityServer4.Validation.SecretValidator -> λ:IdentityServer4.Validation.ISecretValidator[] -> IdentityServer4.Validation.PrivateKeyJwtSecretValidator."

[leastprivilege]

https://www.identityserver.com/articles/migrating-your-identityserver4-v3-database-to-identityserver4-v4

[jinweijie]

Finished deploying the migration to production yesterday.

The overall migration experience was smooth, it took around 2 days to finish preparing the upgrade script, code changes, test and deployment. Thanks to the migration script: https://github.com/RockSolidKnowledge/IdentityServer4.Migration.Scripts

One gotcha was the PublicOrigin property was removed in v4, so we didn't find the issue util we deployed to dev environment (We use Nginx as reverse proxy). Then follow this instruction to replace the PublicOrigin with SetIdentityServerOrigin method in Configure method of Startup.cs: #4535 (Please be aware that place SetIdentityServerOrigin above UseIdentityServer method)

Thanks for all the resources and hope this also helps a bit.

[jinweijie]

Few issues I've run into that I'm unsure how to handle:

1:

Current broken code:

await _interaction.GrantConsentAsync(context, ConsentResponse.Denied);

ConsentResponse.Denied no longer exists, but I'm unsure how to construct a ConsentResponse that means Denied. The old implementation of Denied looked like this:

public static ConsentResponse Denied = new ConsentResponse();

Is an empty ConsentResponse still the equivalent of Denied?

2:

Current broken code:

await HttpContext.SignInAsync(user.Id, user.UserName, props);

I notice that there is now an overload that takes IdentityServerUser, but I'm unfamiliar with this class and how to get an instance of it. I see that there's a constructor that takes sub. Is it enough to just do var user = new IdentityServerUser(user.Id) and pass that to SignInAsync?

3:

Current broken code:

await _events.RaiseAsync(new ConsentGrantedEvent(User.GetSubjectId(), request.Client.ClientId, request.ScopesRequested, grantedConsent.ScopesValuesConsented, grantedConsent.RememberConsent));

There used to be a ScopesRequested property on AuthorizationRequest. I'm not sure what the intended replacement is.

4: Above, @jt-pt-dev links to some SQL scripts for updating the database, but are there Entity Framework migration files available? If so, how do we use them?

for Q1
maybe:

grantedConsent = new ConsentResponse { Error = AuthorizationError.AccessDenied };

for Q3:
maybe:

await _events.RaiseAsync(new ConsentDeniedEvent(User.GetSubjectId(), request.Client.ClientId, request.ValidatedResources.RawScopeValues));

[MCFHTAGENTS]

Hi, I am using the
PublicOrigin option in my call to AddIdentityServer. Can someone tell me where that has gone in version 4?

also in my

                Mock<IIdentityServerInteractionService> mockInterationService = new Mock<IIdentityServerInteractionService>();
                mockInterationService.Setup(a => a.GetAuthorizationContextAsync(It.IsAny<string>())).Returns(Task.FromResult(new AuthorizationRequest()
                {
                    ScopesRequested = dummyScope,
                }));

I get the ScopeRequested failing what should this call be replaced with?

Thank you

[ignite-404]

@MCFHTAGENTS

two ways to fix it

#4631

app.Use(async (context, next) =>
            {
                //升级到v4后这publicorigin选项不见了，改为在中间件中设置publicorigin
                var config = context.RequestServices.ResolveConfig_();
                var publicOrigin = config["public_origin"];
                if (publicOrigin?.Length > 0)
                {
                    context.SetIdentityServerOrigin(publicOrigin);
                    context.SetIdentityServerBasePath(context.Request.PathBase.Value.TrimEnd('/'));
                }
                await next.Invoke();
            });

[MCFHTAGENTS]

Thank you - has anyone any idea on my other point above?

I have also now moved forward to testing the new system and get the following error in my logs (and this all used to work on the old version of IdentityServer4):

2020-07-19 09:47:16.692 +01:00 [INF] Invoking IdentityServer endpoint: IdentityServer4.Endpoints.AuthorizeEndpoint for /connect/authorize
2020-07-19 09:47:16.693 +01:00 [DBG] Start authorize request
2020-07-19 09:47:16.695 +01:00 [DBG] No user present in authorize request
2020-07-19 09:47:16.697 +01:00 [DBG] Start authorize request protocol validation
2020-07-19 09:47:16.780 +01:00 [DBG] [redacted email address] found in database: true
2020-07-19 09:47:16.780 +01:00 [VRB] Calling into client configuration validator: IdentityServer4.Validation.DefaultClientConfigurationValidator
2020-07-19 09:47:16.783 +01:00 [DBG] client configuration validation for client [redacted email address] succeeded.
2020-07-19 09:47:16.791 +01:00 [DBG] Checking for PKCE parameters
2020-07-19 09:47:16.791 +01:00 [DBG] No PKCE used.
2020-07-19 09:47:16.828 +01:00 [DBG] Found ["openid"] identity scopes in database
2020-07-19 09:47:16.872 +01:00 [DBG] Found [] API resources in database
2020-07-19 09:47:16.887 +01:00 [DBG] Found ["[redacted scope name]"] scopes in database
2020-07-19 09:47:16.909 +01:00 [ERR] Scope [redacted scope name] not found in store.
2020-07-19 09:47:16.911 +01:00 [ERR] Request validation failed

I can see an entry with [redacted scope name] in the following tables in my database: ApiScopes, ClientScopes but NOT IdentityResources

I can see in the code this error arising from a failed call to:
resourcesFromStore.FindApiScope(requestedScope.ParsedName);

but it's there - any thoughts and ideas given the changes?

[MCFHTAGENTS]

OK - one update on the error above - I needed to set Enabled to true in the ApiScope definition

[ishchatul]

For anyone that has implemented their own ISecretsListValidator the params the Validate async method receives have been switched:
now - Task ValidateAsync(IEnumerable secrets, ParsedSecret parsedSecret);
used to be - Task ValidateAsync(ParsedSecret parsedSecret, IEnumerable secrets);

[kikaragyozov]

#4742

[IndexOverflow]

I'm having some issues in my ProfileService after updating to v4. When the ProfileDataRequestContext.Caller is for an access_token ResourceValidationResult contains the expected ApiResources. However, when the caller is for an identity_token the validation result is empty; no resources and the parsed scopes list is also empty. As far as I can tell the refactor regarding ApiResource and ApiScopes shouldn't affect IdentityResources (which again, works as expected after a minor refactor).

[sevenam]

IdentityServer4.EntityFramework.Entities.ApiSecret appears to have been renamed to IdentityServer4.EntityFramework.Entities.ApiResourceSecret

And IdentityClaim seems to have been renamed to IdentityResourceClaim

IIdentityServerInteractionService.GetAllUserConsentsAsync() seems to have been replaced by IIdentityServerInteractionService.GetAllUserGrantsAsync() and return type has changed from Consent to Grant.

[sevenam]

After adding migrations and updating the database schema we ended up with an invalid_scope error. This turned out to be due to a new field in the ApiScope table called Enabled which was set to 0 as default.

Changed the defaultvalue for this to true in the migration:

     migrationBuilder.AddColumn<bool>(
                name: "Enabled",
                table: "ApiScopes",
                nullable: false,
                defaultValue: true);

[sevenam]

Also had to change the AddAuthentication method in Startup.cs due to deprecated package: IdentityServer4.AccessValidation. Using the package Microsoft.AspNetCore.Authentication.JwtBearer instead.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
            .AddJwtBearer(options =>
            {
                // base-address of your identityserver
                options.Authority = "https://demo.identityserver.io";

                // if you are using API resources, you can specify the name here
                options.Audience = "resource1";

                // IdentityServer emits a typ header by default, recommended extra check
                options.TokenValidationParameters.ValidTypes = new[] { "at+jwt" };
            });
    }
}

After switching library we also had to add this to Startup.cs for our API as the "sub" claim was missing:
JwtSecurityTokenHandler.DefaultInboundClaimTypeMap.Clear();

Also note the comment on ValidateAudience:

If you are not using the audience claim, you can turn off the audience check via options.TokenValidationParameters.ValidateAudience = false;

See:https://identityserver4.readthedocs.io/en/latest/topics/apis.html

[daver77]

Hi, i'm just having an issue around API's when updating from v3 > v4

I have migrated my db to v4, my API was setup using AddIdentityServerAuthentication (is this still supported?)

(I have no ApiResourceClaims(ApiClaims)/ApiResourceProperties(ApiProperties)/ApiResourceSecrets(ApiSecrets) records)

services.AddAuthentication(IdentityServerAuthenticationDefaults.AuthenticationScheme)
                .AddIdentityServerAuthentication("SsoApiAuth", options =>
                {
                    options.Authority = Configuration.GetValue<string>("IdentityServer:Authority");
                    options.ApiName = Configuration.GetValue<string>("IdentityServer:ApiName");
                });

Now, testing with Postman, I can get the JWT token fine but I get a 401 accessing the API. Is there anything new that I need to add or anything I need to change in order for this to work?

[daver77]

Ah, I have found the issue, I need to populate ApiResourceScopes from ApiScopes

https://github.com/RockSolidKnowledge/IdentityServer4.Migration.Scripts/blob/master/MSSQL/ConfigurationDbContext.sql#L211

Does that mean ApiScopes is deprecated?

[joch0a]

Ah, I have found the issue, I need to populate ApiResourceScopes from ApiScopes

https://github.com/RockSolidKnowledge/IdentityServer4.Migration.Scripts/blob/master/MSSQL/ConfigurationDbContext.sql#L211

Does that mean ApiScopes is deprecated?

ApiScopes is not deprecated, the relationship between the ApiResource was moved to another table.

To populate the table I added this line on the Up() method of the migration.

migrationBuilder.Sql("INSERT INTO dbo.\"ApiResourceScopes\"(\"Scope\", \"ApiResourceId\") SELECT \"Name\", \"ApiResourceId\" FROM dbo.\"ApiScopes\"; ");
This works fine with postgres.

[daver77]

Thanks, there seems to be duplication in the db. I have my ApiScopes data, my ApiResources data, surely ApiResourceScopes should be a link table between those 2 eg Id, ApiScopeId (instead of Scope), ApiResourceId, rather than duplicating the Scope name in ApiResourceScopes.

Or am I misunderstanding it

[joch0a]

Yes, you are right, idk why the ApiResourceScopes table doesn't use the ApiScopeId instead of the Name. The same happens on the ClientScopes table IIRC

[jivpif]

Hi,

After migrating, we have the following issue related with JWKS:
The "kid" value is changed (found in /.well-known/openid-configuration/jwks). It has the "RS256" string appended to it.

Previous version: '"kid": "01D94A4A417A86B10366467C70980FCC9442E6D3"'
New version: '"kid": "01D94A4A417A86B10366467C70980FCC9442E6D3RS256"'

We will appreciate any guidance how to set it to it's previous value.
And do you think it's related to the IdentityServer upgrade?

Thanks

[leastprivilege]

I think AddSigningCredential always allows you to pass a raw SigningCredential that in turn allows you to control the kid yourself.

What kind of problem causes this?

[jivpif]

Seems some of the clients using our service keep this value hardcoded on their side. They are using it to validate the JWTs that we send to them and they fail as the "kid" in the JWT is not the same anymore.

I think I found something that might be causing this:
https://github.com/IdentityServer/IdentityServer4/blob/main/src/IdentityServer4/src/Configuration/DependencyInjection/BuilderExtensions/Crypto.cs#L87

We are calling the AddSigningCredential(X509Certificate2) without passing a signingAlgorithm. Its default value is "RS256" and it's appended to the key id as the code shows.

Probably will try with the AddSigningCredential(SigningCredentials) that you suggested, to skip this appending.

Thanks for the quick response

[leastprivilege]

keep this value hardcoded on their side

well - this is wrong. The kid can change at any point in time.

[bitbound]

Maybe I missed this somewhere in the conversation, but... Why was PublicOrigin removed? Will it be added back?

[bitbound]

Nevermind. Saw the comments in the linked issues. Worked for me.

[viveknaragude]

Yes, you are right, idk why the ApiResourceScopes table doesn't use the ApiScopeId instead of the Name. The same happens on the ClientScopes table IIRC

Any Update

[viveknaragude]

Thanks, there seems to be duplication in the db. I have my ApiScopes data, my ApiResources data, surely ApiResourceScopes should be a link table between those 2 eg Id, ApiScopeId (instead of Scope), ApiResourceId, rather than duplicating the Scope name in ApiResourceScopes.

Or am I misunderstanding it

Any Update

[sevenam]

@viveknaragude Due to the missing relation between ApiResourceScopes and ApiScopes, we ended up duplicating the scopes data into both of these tables after upgrading to v4 unfortunately...

[viveknaragude]

@viveknaragude Due to the missing relation between ApiResourceScopes and ApiScopes, we ended up duplicating the scopes data into both of these tables after upgrading to v4 unfortunately...

Thank you.

[MikaelElkiaer]

I had issues with claim mappings and audience, but none of the previous comments quite got me there.
I too swapped out IdentityServer4.AccessTokenValidation for IdentityModel.AspNetCore.AccessTokenValidation, but found myself digging into the deprecated package to find stuff I still needed.
I ended up with a configuration like this in order to support the correct sub and role claims, as well as ignoring audience:

services.AddAuthentication("token")
                .AddJwtBearer("token", o =>
                {
                    o.Authority = BoundConfiguration.AuthConfig.AuthBaseUrl;
                    o.Audience = "manager";

                    o.TokenValidationParameters.ValidateAudience = false;
                    o.TokenValidationParameters.ValidTypes = new[] { "at+jwt" };
                    o.SecurityTokenValidators.Clear();
                    o.SecurityTokenValidators.Add(new JwtSecurityTokenHandler { MapInboundClaims = false });
                    o.TokenValidationParameters.NameClaimType = ClaimConstants.NameIdentifier; // "sub"
                    o.TokenValidationParameters.RoleClaimType = ClaimConstants.Role; // "role"
                });

[kvijayasingh]

i tried the above steps , it worked on the identity server, but test application is using the pipelinefactory , modified the add scopes and resource but still failing on all the controller test here is pipeline factory code
PipelineFactory

public static TestServer CreateServer(Action options,
DelegatingHandler backChannelHandler, bool addCaching = false)
{
return new TestServer(new WebHostBuilder().UseUrls("http://localhost:5000")
.ConfigureServices(services =>
{
if (addCaching) services.AddDistributedMemoryCache();

            // Start Custom Code

            services.AddTransient<IClientsRepository, ClientsRepository>();
            services.AddTransient<IApiResourceRepository, ApiResourceRepository>();

            // Add AspNet Identity Database context
            services.AddDbContext<ApplicationDbContext>(dbContextOptionsBuilder =>
                dbContextOptionsBuilder.UseInMemoryDatabase("xxxx"));

            // Add AspNet Identity provider to Identity Server
            services.AddIdentity<ApplicationUser, IdentityRole>()
                .AddEntityFrameworkStores<ApplicationDbContext>()
                .AddDefaultTokenProviders();

            // Add InMemory Identity Server configuration
            services.AddIdentityServer()
                .AddDeveloperSigningCredential()
                .AddConfigurationStore(configurationStoreOptions =>
                {
                    configurationStoreOptions.ConfigureDbContext = builder =>
                        builder.UseInMemoryDatabase("xxxx");
                })
                .AddOperationalStore(operationalStoreOptions =>
                {
                    operationalStoreOptions.ConfigureDbContext = builder =>
                        builder.UseInMemoryDatabase("xxxx");
                })
                .AddInMemoryApiScopes(GetApiScopes())
                .AddInMemoryApiResources(GetApiResource());

please suggest do i miss anything