Skip to content

Fix collection filtering API for IN/NOT IN comparisons that require type conversions#12190

Merged
greg0ire merged 1 commit intodoctrine:2.20.xfrom
mpdude:criteria-matching-custom-type-retry
Oct 16, 2025
Merged

Fix collection filtering API for IN/NOT IN comparisons that require type conversions#12190
greg0ire merged 1 commit intodoctrine:2.20.xfrom
mpdude:criteria-matching-custom-type-retry

Conversation

@mpdude
Copy link
Copy Markdown
Contributor

@mpdude mpdude commented Oct 7, 2025

This PR fixes the Criteria matching API for IN and NIN conditions with values that are arrays, by making sure that type information for the matched field is passed to the DBAL level correctly.

Passing the right parameter type to DBAL is important to make sure parameter conversions are applied before matching at the database level.

Memory-based collections (ArrayCollections or initialized collection fields) would perform matching on the objects in memory where no type conversion to the database representation is required, giving correct results.

But uninitialized collections that have their conditions evaluated at the database level need to convert parameter values to the database representation before performing the comparison.

One extra challenge is that the DBAL type system does currently not support array-valued parameters for custom types. Only a limited list of types is supported.

I discussed this with @morozov at the Doctrine Hackathon and came to the conclusion that it would be best to work around this limitation at the ORM level. Thus, this fix recognizes array-valued parameters and creates multiple placeholders (like ?, ?, ?) for them, flattening out the arrays in the parameter list and repeating the type information for each one of them.

Previous stalled attempt to fix this was in #11897.

morozov
morozov previously approved these changes Oct 9, 2025
Copy link
Copy Markdown
Member

@morozov morozov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from the DBAL perspective 👍

@greg0ire
Copy link
Copy Markdown
Member

The commits would need to be squashed either by you or the person who merges.

…ype conversions

This PR fixes the `Criteria` matching API for `IN` and `NIN` conditions with values that are arrays, by making sure that type information for the matched field is passed to the DBAL level correctly.

Passing the right parameter type to DBAL is important to make sure parameter conversions are applied before matching at the database level.

Memory-based collections (`ArrayCollection`s or initialized collection fields) would perform matching on the objects in memory where no type conversion to the database representation is required, giving correct results.

But uninitialized collections that have their conditions evaluated at the database level need to convert parameter values to the database representation before performing the comparison.

One extra challenge is that the DBAL type system does currently not support array-valued parameters for custom types. Only a [limited list of types](https://www.doctrine-project.org/projects/doctrine-dbal/en/4.2/reference/data-retrieval-and-manipulation.html#list-of-parameters-conversion) is supported.

I discussed this with @morozov at the Doctrine Hackathon and came to the conclusion that it would be best to work around this limitation at the ORM level. Thus, this fix recognizes array-valued parameters and creates multiple placeholders (like `?, ?, ?`) for them, flattening out the arrays in the parameter list and repeating the type information for each one of them.

Previous stalled attempt to fix this was in doctrine#11897.
@greg0ire greg0ire force-pushed the criteria-matching-custom-type-retry branch from 0d93196 to 7a59281 Compare October 13, 2025 21:12
@greg0ire greg0ire added the Bug label Oct 16, 2025
@greg0ire greg0ire added this to the 2.20.7 milestone Oct 16, 2025
@greg0ire greg0ire merged commit 9a55cf4 into doctrine:2.20.x Oct 16, 2025
91 checks passed
@greg0ire
Copy link
Copy Markdown
Member

Thanks @mpdude

@grossmannmartin
Copy link
Copy Markdown

Hi,
Have you considered following code $repository->findBy(['uuid' => []]?

In version 2.20.6 it generates the following SQL

SELECT ...  FROM table t0 WHERE t0.uuid IN (NULL) 

But 2.20.7 generates invalid SQL

SELECT ...  FROM table t0 WHERE t0.uuid IN () 

This may be very surprising after update to a patch version, especially when the value is passed as a variable from another part of the application.

@mpdude
Copy link
Copy Markdown
Contributor Author

mpdude commented Oct 28, 2025

Hm, seems this was not covered by the existing test suite ☹️.

IN (NULL) is probably always false, right?

So, should passing an empty array mean that we want a false condition (like 1=0), and passing a null value in the array means to use x IN (…) OR x IS NULL?

@grossmannmartin
Copy link
Copy Markdown

Yes, IN (NULL) is always false.
We may debate whether it makes sense in terms of SQL. But the harsh fact is that it is really easy to write something like

public function getByIds(array $ids): array
{
    return $this->getRepository()->findBy(['id' => $ids]);
}

I would expect this code to return no results (as it's not changing the existing behavior).

So, should passing an empty array mean that we want a false condition (like 1=0), and passing a null value in the array means to use x IN (…) OR x IS NULL?

Yes, that summarizes it very well 💚

@vuryss
Copy link
Copy Markdown

vuryss commented Oct 30, 2025

Hi, Have you considered following code $repository->findBy(['uuid' => []]?

In version 2.20.6 it generates the following SQL

SELECT ...  FROM table t0 WHERE t0.uuid IN (NULL) 

But 2.20.7 generates invalid SQL

SELECT ...  FROM table t0 WHERE t0.uuid IN () 

This may be very surprising after update to a patch version, especially when the value is passed as a variable from another part of the application.

This is definitely a breaking change. We have a few cases where we use that and a patch version should not introduce a breaking change. Changing from 3.5.2 to 3.5.3 breaks out projects badly because of this.

It means we have to go in every case where we use IN() and add a if ([] === $argument) {...} because if this.

@DiegoRBaquero
Copy link
Copy Markdown

This breaks a custom type on the convertToDatabaseValue method because instead of receiving an array ['my value'] it's now receiving plain string 'my value'.

This is when doing a findOneBy(['field' => ['my value']]);

Any pointers on how to fix this would be appreciated 🙏🏼 . For now I've reverted to 3.5.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants