Allow mapping optional complex properties

[roji]

We currently always return a non-null instance for complex properties, including when all their columns are null. We may need to add an option to opt into optional complex types, where if all properties are null, null is returned instead.

See #9005 for the same thing with owned entity types.

• Model building and validation
• Relational and compiled model support
• Migrations support
• Change tracking support
• Add complex type discriminators that can be used as a marker if all other properties are optional
• Query support
• ExecuteUpdate support

[marchy]

Without this the use of complex types is extremely limited.

Since these are "identity-less" concepts by definition, they may often be un-set in the domain (ie: it's not a strong enough domain entity concept to warrant its own identifiable entity – high correlation to not always being set).

Please adopt nullability/optionality to both the JSON-backed and column-backed complex types in the future – until then back to good-old nullable string-column JSON mapping without any notion of complex types it is. 😔

[roji]

@marchy please note that you can already use owned entity modeling as a sort of complex type modeling; that supports both optionality and JSON columns. The new complex type modeling being introduced in 8.0 is still new and does not support everything, but we plan to improve that.

[marchy]

Thanks for the suggestion @roji, well familiar with the implications of Owned Types, as as you mention it has been around for years.

However they fail on the identity-less semantics of value objects (ie: multiple records cannot have the same Owned Entity values without conflicting) – the very reason complex types have been introduced.

Requiring value objects to be non-optional is highly arbitrary and extremely limiting.

🤞 Really looking forward to a big subsequent EF9 push on complex types to support optionality (this issue), type hierarchies and JSON mapping before they can become realistically feasible to adopt.

[marchy]

PS: One of the scenarios this may enable as well is complex type hierarchies – where each type in the hierarchy has to be modelled as optional, as each sub-class complex type would essentially be mutually exclusive to the other sub-classes (ie: optional, XOR-like semantics, as any instance can only be of one sub-class type or another, but never multiple)

[roji]

Requiring value objects to be non-optional is highly arbitrary and extremely limiting.

This wasn't a design decision or anything - we simply didn't have enough time to implement optional complex types for 8.0 (this is more complex than it looks).

One of the scenarios this may enable as well is #31250 (comment) – where each type in the hierarchy has to be modelled as optional [...]

Optionality and inheritance mapping are pretty much orthogonal, but we do plan to do both. Optional complex types will likely be prioritized much higher though.

[marchy]

Thank you @roji.

Having been on EF since v4 (ie: the first code-first version circa 2009)... I do appreciate that anything to do with ORM's is extremely difficult and takes time to cover the many scenarios that need to be considered. 😄

Appreciate the insight on the thinking in priority.

The optionals support could potentially enable hand-rolling associative enums however – since they are essentially the same as TPC and as long as you can null out all other sub-class complex objects except the the one the instance conforms to. In theory you could achieve this without any official support for inheritance – so long as the framework doesn't get in the way by detecting the abstract base class and preventing the mapping in some way (a scenario to consider).

This is the specific scenario riddled throughout our domain (and indeed many domains):

public abstract record Identity {
	public record Anonymous( string InstallationID, Platform Platform, Installation? Installation ) : Identity;
	public record Authenticated( long AccountID, Account? Account ) : Identity;
}

The moment you can model these as complex types, you can add all sorts of variances throughout the domain – for example when purchasing a ticket for something, you might have variances that have different fields based on the ticket type etc. Same thing when choosing a payment type, or pickup/delivery option, or login provider etc.

If we could map the above based on a manual discriminator column (IdentityType:string/enum) and two nullable complex types (AnonymousIdentity? and AuthenticatedIdentity?) that would already be a win.

Thinking this:

public partial class SomeEntity {
	// ... other state of the entity
	
	// NOTE: complex type hierarchy
	public Identity Identity {
		get => IdentityType switch {
			nameof(Identity.Anonymous) => AnonymousIdentity!,
			nameof(Identity.Authenticated) => AuthenticatedIdentity!,
			_ => throw new ArgumentOutOfRangeException( $"Unknown identity type '{IdentityType}' ),
		};
		init => {
			// NOTE: XOR-like logic to set all sub-class complex types in the TPC hierarchy to null except for the one the instance conforms to
			(string IdentityType, AnonymousIdentity? AnonymousIdentity, AuthenticatedIdentity? AuthenticatedIdentity) persistedValue = value switch {
				Identity.Anonymous anonymousIdentity => {
					IdentityType: nameof(Identity.Anonymous),
					AnonymousIdentity: anonymousIdentity,
					AuthenticatedIdentity: null,
				},
				Identity.Authenticated authenticatedIdentity => {
					IdentityType: nameof(Identity.Authenticated),
					AnonymousIdentity: null,
					AuthenticatedIdentity: authenticatedIdentity,
				}
			};
			(IdentityType, AnonymousIdentity, AuthenticatedIdentity) = persisted value
		}
	}
}


// HACK: Hide DAL fields away from the domain model
/*DAL*/partial class SomeEntity {
	internal string IdentityType { get; private set; }
	public AnonymousIdentity? AnonymousIdentity { get; private set; }
	public AuthenticatedIdentity? AuthenticatedIdentity { get; private set; }
}

Having EF automatically link the two parts together in TPC-style would definitely be the end-game (maybe EFC 10!) 🚀

End-game: (no partial class hackiness needed)

public class SomeEntity {
	// ... other state of the entity
	
	// complex type hierarchy
	// NOTE: this is still not itself optional – despite its constituent parts of each TPC-style sub-class getting mapped to optional complex types under the cover
	public Identity Identity { get; private set; }
}

[roji]

@marchy when we do get to complex type inheritance mapping (#31250), it will definitely not be composition-based as in your sample above, but rather inheritance-based (much like the current TPH for non-complex-types). You're free to implement what composition-based scheme you want (once we implement optional support), but that generally seems like quite a complicated way to go about things, pushing a lot of manual work on the query writer (e.g. needing to deal with the discriminator manually).

[alexmurari]

This is a must have for always valid domains.

E.g. a Phone value object can't be instantiated with null/empty Number value. The class protects its invariants. Nullability must be at the value object level (Phone?).

Here's an excelent article about when value objects should or should not be null: https://enterprisecraftsmanship.com/posts/nulls-in-value-objects/

[jscarle]

I completely agree with @alexmurari. Value Object validity dictates that its the Value Object itself that should be optional and nullable.

[aradalvand]

I second that complex types are currently basically unusable in a good 50% of scenarios precisely because of this limitation. Hopefully this makes it to v9; should be considered a high-priority item IMO.

[marchy]

Thanks @roji, that sounds ideal indeed – was just showing how the composition-based approach can let you model the associative enums scenario (ie: no shared state between different sub-classes) with just the support of optionals, rather than needing #31250 (where I did drop an example of how that simplify things even further).

Hope that helps prioritize.

[AndriySvyryd]

We should set NRT annotations correctly to avoid warnings when configuring nullable complex types:

modelBuilder.Entity<MyEntity>().ComplexProperty(e => e.Complex).Property(c => c!.Details);

[ManeeshTripathi14]

I am working on a project where we have already designed our domain entity based on DDD, where we have nullable complex property (value object) because those are NOT mandatory by business requirement. i was facing one issue with OwnsOne, in case of OwnsOne ef is not able to detect changes and entity state does not change to modified if we update the value of complex property( update means here replacing the old with new and NOT modifying the property of complex property individually ) so now due to this limitation i can't use complex property.
Please allow NULL for complex property. so that we can desin domain entity without considering infrastructure concerns.
thanks

[oleg-varlamov]

@ManeeshTripathi14
I absolutely agree with you. In all previous projects we used NHibernate, which works very well with DDD. But in new projects, due to good JSON support, we decided to switch to EFCore. We have been waiting for about a year for the appearance of new Complex Properties, since OwnedEntity is a very strange implementation that brought more problems than benefits. And in the end, when Complex Properties came out, we were so disappointed :( It is almost necessary for Complex Properties to support NULL values.

[cjblomqvist]

Preferably, do not limit this to require att least one non nullable property like it is for owned. Rather, add a "hidden" nullable column (or bool or whatever) if needed.

[davisnw]

all we're doing is requiring the user to make a clear decision on the behavior they want.

but we'll have a simple way of adding a required shadow property that would serve as a sentinel/discriminator.

If I've understood correctly, this would mean that you would need to add an additional discriminator column to the database table in some scenarios?

If that's the case, my general preference would be to instead (or additionally) have a mechanism to configure this at the entity level via an attribute on the complex property of the entity and / or an option in the mapping api without requiring a discriminator column.

We sometimes interact with databases that we don't readily have control over the schema.

[Timovzl]

Thank you kindly for explaining the motivations. I can definitely understand the complexity, and the ambiguity between null vs. set-of-null-values.

Based on my own experience and what I'm hearing in this thread, there is a big use case for having a null object when all the values are null - arguably greater than the one for having an empty object.

An optional setting to let us have null instead of an empty object would be ideal, of course, although there is that cost and complexity again. We'd be explicitly indicating that we're not planning to ever store an object containing all nulls, which takes out the ambiguity.

As an alternative, what kind of interceptor or other mechanism could we use as a workaround? That is, when materializing entities containing complex types, what would be the ideal moment to null out an empty object?

[roji]

Based on my own experience and what I'm hearing in this thread, there is a big use case for having a null object when all the values are null - arguably greater than the one for having an empty object.

To be clear, the discussion here is about allowing complex types with all-optional properties, and for EF to arbitrarily choose one of the following:

1. Dematerialize a non-null object with all properties set to null
2. Dematerialize null

This is one of those cases where there's simply no one correct solution that will satisfy everyone - no matter what we choose, some proportion of users will ask for the other. @Timovzl I'm not sure what you're basing your statement above, that a "null object" is somehow better/more useful.

@davisnw

In any case, as @AndriySvyryd wrote, all we're doing is requiring the user to make a clear decision on the behavior they want.

It's worth pointing out again that as @AndriySvyryd wrote, you always have the option of simply configuring the complex type as required. That means you can have all-optional properties, and we'll always materialize a non-null object for you. Since you have this option, asking us to support null materialization really is nothing more than a mere style preference in terms of your coding: you're not limited to working with any existing database schema (as suggested above), we're only telling you that you have to interact with that schema via non-null object instances, rather than null.

Another advantage of this planned design, is that if lots of users actually complain that they'd like configurability on a per-entity level, that's something we can always decide to implement later. I suspect very few will actually care and this won't be needed, but if I'm wrong (as @Timovzl suggests) we can always improve this in the future.

[alexmurari]

Suggestion for users who stumble upon the limitation of optional complex types with only optional properties

I fail to see a valid use case for a nullable complex type where all properties are nullable.

(Maybe you don't need a nullable complex type that contains only nullable properties)

What’s the difference between a materialized Address object with all properties set to null and a null Address object? Neither contains meaningful data! But the materialized version increases the risk of null reference errors.

In my view, complex types (value objects) where all properties are nullable are probably wrong because they violate invariants—an address in the real world can't exist without at least a street name, city, etc. The code should reflect this. You shouldn’t be able to new() an Address without the required minimum data (the contructor should raise an exception). This is precisely why the risk of null reference errors increases: as a developer, I would assume that a materialized Address object contains valid data, and not a null StreetName property, for example.

My understanding comes from this article (it answers this exact question - nullable value object vs value object with nullable properties): https://enterprisecraftsmanship.com/posts/nulls-in-value-objects/

[roji]

What’s the difference between a materialized Address object with all properties set to null and a null Address object? Neither contains meaningful data! Worse, the materialized version increases the risk of null reference errors.

That might be your opinion, but C# thinks differently - the two are completely different things, and depending on your specific application, that distinction can be very important. Importantly, EF already does preserve that distinction for regular entities with navigations, and will also be able to preserve that distinction when mapping complex to JSON (since JSON has different representations null and empty objects).

The problem is specifically when mapping a complex type to relational columns in the same table (table splitting). In this mapping strategy, the distinction between null and empty can't be preserved without some property to tell us whether the object exists or not.

In my view, complex types (value objects) where all properties are nullable are probably wrong because they violate invariants—an address in the real world can't exist without at least a street name, city, etc.

That is simply untrue. Where exactly null can appear depends on your domain model, and there's nothing saying that all properties can't be nullable.

You shouldn’t be able to new() an Address without the required minimum data (the contructor should raise an exception).

Are you saying that the C# compiler should allow the instantiation of a class with all its properties initialized to null?

as a developer, I would assume that a materialized Address object contains valid data, and not a null StreetName property, for example.

The point is that what's considered "valid data" depends entirely on your specific domain model/application.

Thanks for the article. But at the end of the day, EF's job here as an ORM is to preserve the .NET data given to it, and roundtrip it faithfully. Since .NET has the distinction between null and empty, EF needs to preserve it as well.

[alexmurari]

@roji You got my comment all wrong. I wasn’t suggesting a change in how EF handles these scenarios or redefining the C# compiler rules. I was talking about domain modeling. This was simply my opinion on code design (bases on DDD principles) when defining complex types, in case anyone encounters this limitation of optional complex types with only optional properties. These suggestions are for user code. 😄

[djdanne]

I agree with @alexmurari that this does present problems when implementing value objects based on DDD principles, which is what we intend to use complex types for (previously we used owned types). The main problem is that it breaks the nullability pattern of C#. For example, if we have a value object for a mail address, there is no way to see if the mail can be null or not unless you inspect the value object class. Using the required keyword won't help in most of our cases simply because the required keyword requires that the set:er can't be less visible than the containing type. This will almost always be false in DDD, since an entity shouldn't expose set:ers for the properties, they should only be modified by the entity itself. This would mean that we can't catch some nullability problems in compile time.

The loss off compile time validation and risk of misunderstandings regarding nullability is the problem for me, not the concept itself of having to set all the properties to null. If the required property had worked on protected properties it would have been an adequate solution. Now, if you miss to actively set all properties to null for a complex type C# will default to setting the complex type to null and we will have a runtime error.

Of course I have respect for that it might be to complicated to implement nullability or that you want to avoid locking down how you can use complex types (nullable object vs object with nullable properties). Maybe it can be solved by a new attribute like required to opt in to a specific behavior and just get that compile time validation?

[roji]

Everyone, as arguments are being repeated and no new information is being added, I'll try to summarize the above discussion.

I understand that the current plan might not provide everyone's preferred access style, for the specific case where the object contains only nullable properties; for that specific scenario, you will be able to map it as a required complex type and interact with it as a non-null object with null properties. You indeed won't be able dematerialize such an object as null.

Fundamentally, this is a result of a limitation of table splitting itself, as a mapping strategy (again, JSON complex type mapping will not have this limitation, since it can distinguish between null and empty objects). This mapping strategy simply does not support making the distinction of null vs. empty, and at least for now, we do not believe EF should arbitrarily pick one of these - making some proportion of our users happy and another angry - and also cause a loss of round-trippability.

To reiterate, our first and most important commitment here is to faithfully roundtrip .NET values to the database; since .NET has the null vs. empty distinction, we don't think it's appropriate to implement something that muddles that distinction by having an empty .NET object serialize to some representation which later is deserialized to null. That breaks roundtripping, and removes a distinction that may be important to various .NET applications using EF.

I understand that some DDD users - or rather users of some interpretations of DDD - may find this inconvenient in some cases; but support for these specific patterns is secondary to EF's goal of preserving roundtrippability, and the fundamental .NET distinction of null vs. empty.

As I wrote above, if many people find this problematic, we can always add a configuration knob to make entities materialize as null for this scenario; but I think we should wait to see that feedback first, after the feature is released.