Translate COALESCE as ISNULL

[ranma42]

COALESCEis a syntactic shortcut for the CASE expression. As such, the input values are evaluated multiple times.

ISNULL does not have this shortcoming.

Fixes #32519.

[ranma42]

At least a test for the type propagation is needed

[ranma42]

It looks like the type mapping computed in Coalesce does not match the actual expression type :(
At least the test was helpful in identifying this issue 😅
This problem could be related to #14719 or #15586

[raymens]

Another application for this is that COALESCE cannot be used for Indexed Views in SQL Server, however ISNULL is supported in case of aggregations.

This makes it difficult/impossible to make EF Core generated queries use the indexed views as I suppose the query processor thinks they're different.

For example:

CREATE VIEW [dbo].[vAggregation]
   WITH SCHEMABINDING
   AS
      SELECT a, SUM(ISNULL(b, 0.0)) AS bSum, COUNT_BIG(*) AS Count 
      FROM dbo.c
      GROUP BY a
GO

CREATE UNIQUE CLUSTERED INDEX IDX_vAggregation
   ON vAggregation(a)
GO

SELECT SUM(b) FROM c -- SLOW: not using the indexed view
SELECT SUM(COALESCE(b, 0)) FROM c -- SLOW: not using the indexed view
SELECT SUM(ISNULL(b, 0)) FROM c -- FAST: uses the indexed view

[cincuranet]

@ranma42, gentle nudge on this. We like the ISNULL idea and we'd like to see this done.

[ranma42]

@cincuranet I will rebase the PR and check the new test results ASAP (worst case: next weekend)

Last time I kind of got stuck because of some typing issues as mentioned in #34171 (comment)

I will try to point out explicitly the issues that must be solved (ideally as questions) to unblock this

[ranma42]

This test is designed to ensure that EFCore takes care of the difference in type propagation between ISNULL and COALESCE.

From the docs:

Data type determination of the resulting expression is different. ISNULL uses the data type of the first parameter, COALESCE follows the CASE expression rules and returns the data type of value with the highest precedence.

Specifically, in this case c.Region is an nvarchar(15), while "no region specified" cannot be represented as a value of that type (it's longer than 15).

The implementation I did in this PR would take care of this, under the assumption that the typeMapping for CONVERT can be used to determine the type (and/or whether any casting/conversion is needed), but unfortunately EFCore currently computes it as nvarchar(15) even though (I believe) its result is actually an nvarchar(18).

See https://dbfiddle.uk/ITGS6ZVJ

DECLARE @what sql_variant;
DECLARE @foo varchar(15) = NULL;

SELECT @what = ISNULL(@foo, 'no region selected');
SELECT
    @what,
    SQL_VARIANT_PROPERTY(@what, 'BaseType'),
    SQL_VARIANT_PROPERTY(@what, 'MaxLength');

SELECT @what = COALESCE(@foo, 'no region selected');
SELECT
    @what,
    SQL_VARIANT_PROPERTY(@what, 'BaseType'),
    SQL_VARIANT_PROPERTY(@what, 'MaxLength');

[ranma42]

What is the best way forward for the type issue?
Also, what should be the scope? Just the SqlServer provider or EFCore in general?

I see that the type inference already handles string concatenation in a special way, but it looks like nothing at all happens for other values/operations (CASE and COALESCE included), which is kind of suspicious (what should the typeMapping of c.IntColumn + c.DoubleColumn be?)

NB: I am not sure I actually grasp what typeMappings are supposed to represent (hence how they should be computed/used), so maybe I am simply misusing it in this PR 😇

[cincuranet]

What is the best way forward for the type issue?

Are you talking about the nvarchar(15) and nvarchar(18)?

Also, what should be the scope? Just the SqlServer provider or EFCore in general?

Likely just SQL Server, given the ISNULL is SQL Server specific.

(what should the typeMapping of c.IntColumn + c.DoubleColumn be?)

It should result in double, I believe.

[ranma42]

What is the best way forward for the type issue?

Are you talking about the nvarchar(15) and nvarchar(18)?

Yes. More in general, I am referring to the mismatch between the typeMapping computed by EFCore and the type of the SQL (sub-)expressions.
I delved a little more in type mappings and I am still missing some pieces of the puzzle 😅
IIUC they are used for few main purposes:

• during insert/update, to convert the C# values into the SQL literal representation
• during materialization, to convert the SQL values into corresponding C# values
• sometimes, in expressions (AFAICT only against constants/parameters, which for example makes this inconsistent when 2 columns are compared 😕 )

If this is correct, for use case 1 & 2 typeMappings could simply be attached to columns, with no complex machinery to propagate them across all expressions.
EDIT: this is obviously wrong: expressions can be part of projections, hence need to be materialized

I am unsure whether the expression/comparison behavior is intentional (it is not mentioned as a limitation of value conversion); I'll prepare an issue to showcase the behavior I am seeing and understand if I am simply misreading the docs or if the query conversion has some problems in this corner case.

In this case I am not looking for a representation of the C#/SQL conversion, but rather the (SQL) type of the COALESCE expression; I would like to use in order to preserve the same type when converting the expression to ISNULL, but AFAICT this is something that can't be done simply using typeMappings, as they don't really compute the type of the expression. As an example of a place that does this, see

efcore/src/EFCore.Relational/Query/SqlExpressionFactory.cs

Lines 252 to 265 in ac5bd7b

	case ExpressionType.Add:
	case ExpressionType.Subtract:
	case ExpressionType.Multiply:
	case ExpressionType.Divide:
	case ExpressionType.Modulo:
	case ExpressionType.And:
	case ExpressionType.Or:
	case ExpressionType.ExclusiveOr:
	{
	inferredTypeMapping = typeMapping ?? ExpressionExtensions.InferTypeMapping(left, right);
	resultType = inferredTypeMapping?.ClrType ?? (left.Type != typeof(object) ? left.Type : right.Type);
	resultTypeMapping = inferredTypeMapping;
	break;
	}

If a has a type mapping, the type mapping of a + b, a - b, ... is simply the type mapping of a, even if a is an int and b is a double.

Also, what should be the scope? Just the SqlServer provider or EFCore in general?

Likely just SQL Server, given the ISNULL is SQL Server specific.

👍 other providers might want to use a similar type propagation mechanism, but as long as it is only used in the SqlServer provider, we can look for a reasonable API/implementation in there; I'll move along this direction

(what should the typeMapping of c.IntColumn + c.DoubleColumn be?)

It should result in double, I believe.

This would be my expectation as well; I'll double check (actually, I'll add a test), but IIRC it currently results in int 😞

[ranma42]

If a has a type mapping, the type mapping of a + b, a - b, ... is simply the type mapping of a, even if a is an int and b is a double.

In most cases this is not actually a problem, as the compiler inserts casts to ensure that the type of a and b is matching, but apparently it does not take care of the ?? operator :\

[ranma42]

I fixed the conflicts and force-pushed the branch. I added the test for int/double coalescing, but I found out that it also fails regardless of ISNULL, so I opened #35516 to track it.

I also opened #35515 regarding another part of type mapping (value conversion).