Json: error when trying to materialize json entity with nullable property that is null in the json string - should materialize the property as null instead

[r-Larch]

Maybe a regression of #29219

Materializing a JSON entity with null property results in an error even if the target model allows null.
Expected to materialize the property as null instead.

var error2 = await db.LawCategories
    .AsNoTracking()
    .Where(_ => _.Id == 2)
    .Select(_ => new ModelDto(_.Id, _.Meta.Release))
    .FirstOrDefaultAsync(token); // error: System.InvalidOperationException: Entity LawRelease is required but the JSON element containing it is null.
    
public record ModelDto(int Id, LawRelease? Release);

System.InvalidOperationException: Entity LawRelease is required but the JSON element containing it is null.
   at Microsoft.EntityFrameworkCore.Query.RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.MaterializeJsonEntity[TEntity](QueryContext queryContext, Object[] keyPropertyValues, JsonReaderData jsonReaderData, Boolean nullable, Func`4 shaper)
   at lambda_method697(Closure, QueryContext, DbDataReader, ResultContext, SingleQueryResultCoordinator)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable`1 asyncEnumerable, CancellationToken cancellationToken)

Entity Definition:

public class LawCategory {
    public int Id { get; set; }
    // ... props
    public LawCategoryMeta Meta { get; set; } = null!;

    // ... navigation
}

public class LawCategoryMeta {
    public LawRelease? Release { get; set; }
}

public class LawCategoryConfiguration : IEntityTypeConfiguration<LawCategory> {
    public void Configure(EntityTypeBuilder<LawCategory> builder) {
        builder.OwnsOne(_ => _.Meta, _ => {
            _.ToJson();
            _.OwnsOne(_ => _.Release);
        });
        // ... more
    }
}

Data

Id	Meta	Note
2	{"Release": null}	error in all cases
19	{"Release": {"Date": "2022-10-10", "Number": 149, "Source": "Lgs."}}	works in all cases

Example Code to show error

var works = await db.LawCategories
    .AsNoTracking()
    .Where(_ => _.Id == 19)
    .Select(_ => _.Meta.Release)
    .FirstOrDefaultAsync(token); // works

var error1 = await db.LawCategories
    .AsNoTracking()
    .Where(_ => _.Id == 2)
    .Select(_ => _.Meta.Release)
    .FirstOrDefaultAsync(token); // error: System.InvalidOperationException: Entity LawRelease is required but the JSON element containing it is null.
    
var error2 = await db.LawCategories
    .AsNoTracking()
    .Where(_ => _.Id == 2)
    .Select(_ => new ModelDto(_.Id, _.Meta.Release))
    .FirstOrDefaultAsync(token); // error: System.InvalidOperationException: Entity LawRelease is required but the JSON element containing it is null.
    
public record ModelDto(int Id, LawRelease? Release);

Provider and version information

EF Core version:
Microsoft.EntityFrameworkCore v8.0.7
Database provider: Npgsql.EntityFrameworkCore.PostgreSQL v8.0.4
Target framework: .NET 8.0
Operating system: Windows
IDE: Microsoft Visual Studio Community 2022 (64-bit) Version 17.10.4

[dashiell-zhang]

Entity OrganizationData is required but the JSON element containing it is null

I also encountered this problem today

public OrganizationData? Organization { get; set; }

{
  "User": {
    "Gender": 1,
    "IsInJob": false,
    "UserName": "xiaodong",
    "TravelLevel": "ygzj-1",
    "OrganizationCode": "XXB",
    "SettlementEntityCode": "xxB"
  },
  "Organization": null,
  "SettlementEntity": null
}

Normal operation：
var sss = db.TImportData.AsNoTracking().ToList();

---

There is a bug：
var sss1 = db.TImportData.AsNoTracking().Select(t => t.Data).ToList();

[dashiell-zhang]

@roji From what we have observed so far, if you query directly against the table, there is no problem, but if you query against a new dto, there will be a problem.

[maumar]

The problem is we were incorrectly computing the nullability of json reference - using navigation.ForeignKey.IsRequired rather than navigation.ForeignKey.IsRequiredDependent

[dashiell-zhang]

#35412
This problem seems to still exist

[maumar]

@dashiell-zhang this issue has been fixed in the upcoming EF10 - we didn't fix it in time for EF9.

[boukenka]

@maumar Any plan to backport it in EF9 ? It is a blocking point for me.

[maumar]

@boukenka at the moment we don't plan to port this to EF9, reason being that it's not a regression (scenario also fails on EF8) and relatively few people have hit the issue so far.

[dheardal]

We would also appreciate this being backported to EF9 instead of having to wait for EF10

[eaton-sam]

Also facing this problem. The workaround for my use case isn't great. Would also really appreciate a patch into 9 @maumar. Thanks

[c5racing]

A Backport to EFCore 9 would also be appreciated.