Handle removed parameters by tree walker in Paginator by plfort · Pull Request #7328 · doctrine/orm

plfort · 2018-07-31T07:28:26Z

There is a bug introduced in #6820 (my bad)
Given this code :

$query = $em->createQuery('SELECT a.id, (SELECT b.id FROM EntityB b WHERE b.id = :paramB) AS foo FROM EntityA a ORDER BY a.id');

$query->setParameter('paramB', 123);
$paginator = new Paginator($query);
$paginator->setUseOutputWalkers(false);
$paginator->getIterator();

The resulting subquery in the Paginator will look like this :

SELECT DISTINCT a.id FROM EntityA a ORDER BY a.id

The query is OK but the parameter 'paramB' is still binded to the query, causing a "tooManyParameters" exception.

Edit: I don't really know how to test this behaviour ...

Ocramius · 2018-07-31T07:33:46Z

I didn't even know that we allowed subqueries in the SELECT clause - that seems already like a problem to me.

plfort · 2018-07-31T07:44:11Z

Why do you think it is a problem ?

Ocramius · 2018-07-31T07:45:46Z

Mostly because the hydration process doesn't consider it, nor we have tests for it AFAIK

ostrolucky · 2018-08-06T08:15:24Z

Subselects are supported by design, we even have Subselect node and Parser method. We also do have tests for it https://github.com/doctrine/doctrine2/blob/ce1e3250b4bea05dc0c7a4ed09a6fb05e21705d6/tests/Doctrine/Tests/ORM/Functional/QueryTest.php#L701 and it's pretty widely used.

Ocramius · 2018-08-06T08:16:36Z

Hmm, interesting - fairly sure that subselects were never allowed in the SELECT clause before, so I probably completely missed when this change happened :|

plfort · 2018-08-06T08:18:15Z

It is also documented on the DQL page : https://www.doctrine-project.org/projects/doctrine-orm/en/2.6/reference/dql-doctrine-query-language.html#clauses

ostrolucky

Test case should be added

ostrolucky · 2018-08-06T08:19:27Z

lib/Doctrine/ORM/Tools/Pagination/Paginator.php

+    }
+
+    /**
+     * Remove unused parameters from $query


Method should be named in way this comment is not needed. How about unbindUnusedQueryParams

plfort · 2018-08-06T08:32:41Z

You're right but I don't know how to test it ... I'd like to mock the Paginator and test that the method unbindUnusedQueryParams is called and test the method itself but how can I do it due to its private accessibility ?

ostrolucky · 2018-08-06T08:58:16Z

Yeah all pagination tests are currently functional ones. I will try to help you create unit test for this. But if my effort stalls, just create another functional one, similar like this one https://github.com/doctrine/doctrine2/blob/ce1e3250b4bea05dc0c7a4ed09a6fb05e21705d6/tests/Doctrine/Tests/ORM/Functional/PaginationTest.php#L567-L575

Btw test shouldn't just assert working of unbindUnusedQueryParams, but rationalize it. This means it should assert there is either no exception thrown when query is being executed, or count of paramMappings matches count of parameters, basically same as is done here https://github.com/doctrine/doctrine2/blob/ce1e3250b4bea05dc0c7a4ed09a6fb05e21705d6/lib/Doctrine/ORM/Query.php#L309-L315

plfort · 2018-08-06T10:08:23Z

@ostrolucky thank you for your help

ostrolucky · 2018-08-09T18:57:19Z

I've replaced your test with unit one, hopefully you don't mind. @Ocramius can you have a look?

addressed

Ocramius

Overall, patch seems fine. There is one left problem that needs to be verified, which is a query like following:

SELECT f FROM Foo f WHERE f.foo = :foo

with bound parameters:

[
    'foo' => 'foo',
    'bar' => 'bar',
];

This kind of parameter binding is invalid: would the paginator make it valid because of the automatically removed parameters?

Ocramius · 2018-08-10T08:52:28Z

tests/Doctrine/Tests/ORM/Tools/Pagination/PaginatorTest.php

+            FROM Doctrine\\Tests\\Models\\CMS\\CmsUser u
+            WHERE u.id = :paramInWhere"
+        );
+        $query->setParameters(compact('paramInWhere', 'paramInSubSelect'));


Let's please never ever ever ever use compact :-)

So @Majkl578 can you add it to forbidden functions in CS standard? I was careful in not introducing any new violations.

I think it is there: https://github.com/doctrine/coding-standard/blob/6432a4e17ef37199794559e86eddb05b519832c0/lib/Doctrine/ruleset.xml#L59

Ah it's because 2.6 is locked to version 1

https://github.com/doctrine/doctrine2/blob/f1143f591f1bfcda269141759a20d117e54217b3/composer.json#L30

Any particular reason for that?

Ocramius · 2018-08-10T08:53:32Z

tests/Doctrine/Tests/ORM/Tools/Pagination/PaginatorTest.php

+        $abstractHydrator->method('hydrateAll')->willReturn([$returnedIds]);
+        $connection->expects($this->exactly(3))->method('executeQuery');
+
+        for ($invocationIndex=0; $invocationIndex<2; $invocationIndex++) {


Since this is repeated only 2 times, repeating the expects() is OK

So what's the general rule when to duplicate code and when to put it to loop? 3 time repeat?

It's a loop done for 2 iterations - seems easier to read with explicit at(0) at(1), at(2)

Ocramius · 2018-08-10T08:53:55Z

tests/Doctrine/Tests/ORM/Tools/Pagination/PaginatorTest.php

+        $paginator = (new Paginator($query, true))->setUseOutputWalkers(false);
+
+        $abstractHydrator->method('hydrateAll')->willReturn([$returnedIds]);
+        $connection->expects($this->exactly(3))->method('executeQuery');


Note that exactly, any, at and such are static methods

Asserts as well

ostrolucky · 2018-08-10T11:29:42Z

This kind of parameter binding is invalid: would the paginator make it valid because of the automatically removed parameters?

Probably yes. Doesn't seem worth it to me to add extra logic to deal with such edge case though.

Ocramius · 2018-08-10T11:35:25Z

Well, I'd still expect a test that either expects an exception or just runs regardless of bound parameters :-)

This is just to prevent regressions btw.

ostrolucky · 2018-08-10T21:36:03Z

@plfort I have changed your patch to only strip extra params if there is LimitSubqueryWalker or CountWalker used, as only these strip out select statements. Is that ok?

I have also added tests which deal with edge case @Ocramius suggested. Dealing with that more correctly might be by moving such logic directly into Parser, instead of in Query#_doExecute()

addressed

plfort · 2018-08-11T08:48:44Z

Absolutely 👍

ostrolucky · 2018-08-24T14:02:14Z

bump @Ocramius

Ocramius · 2018-08-26T20:08:35Z

@ostrolucky can you merge this to master and to 2.6 please?

@seferov

v2.6.3 [![Build Status](https://travis-ci.org/doctrine/doctrine2.svg?branch=v2.6.3)](https://travis-ci.org/doctrine/doctrine2) This release provides fixes for many things, specially: - Regression in commit order calculation - BC-break in `EntityManager#find()` using optimistic lock outside of transaction - PHP 7.3 compatibility issues -------------------------------------------- - Total issues resolved: **8** - Total pull requests resolved: **26** - Total contributors: **26** Documentation ------------- - [7472: fix incorrect phpdoc typehint](doctrine#7472) thanks to @seferov - [7465: Fixes tiny typo in the 'Working with DateTime instances' documentation](doctrine#7465) thanks to @unguul - [7444: Fixed URLs of doctrine-mapping.xsd in docs](doctrine#7444) thanks to @Naitsirch - [7441: $hydrationMode throughout can be a string as well as int (for custom modes)](doctrine#7441) thanks to @asgrim - [7435: Fix a typo on Documentation](doctrine#7435) thanks to @oguzdumanoglu - [7434: Removed FAQ paragraph stating public variables are disallowed](doctrine#7434) thanks to @Naitsirch and @flaushi - [7423: Update association-mapping.rst](doctrine#7423) thanks to @ThomasLandauer - [7421: JIRA to Github issues on Limitations and Known Issues](doctrine#7421) thanks to @seferov - [7412: Some formatting improvements](doctrine#7412) thanks to @ThomasLandauer - [7411: Autoload error when following the Getting Started Guide](doctrine#7411) thanks to @ThomasLandauer - [7401: &doctrine#91;docs&doctrine#93; Fix docblock in `inheritance-mapping.rst`](doctrine#7401) thanks to @bobdenotter - [7397: Update getting-started.rst](doctrine#7397) thanks to @eibt - [7394: Class 'Doctrine\Common\Persistence\Mapping\Driver\AnnotationDriver' not found](doctrine#7394) thanks to @ekosynth - [7378: Typo fix](doctrine#7378) thanks to @BenMorel - [7377: Fix query andX doctype](doctrine#7377) thanks to @sserbin - [7374: Deprecation message in documentation for YAML](doctrine#7374) thanks to @SenseException and @iltar - [7360: Document getPartialReference() properly](doctrine#7360) thanks to @lcobucci Bug --- - [7471: Fix parameter value processing for objects with unloaded metadata](doctrine#7471) thanks to @alcaeus - [7367: Fix for BC break in 2.6.2 when calling EM::find() with LockMode::OPTIMISTIC outside of a TX](doctrine#7367) thanks to @timdev - [7328: Handle removed parameters by tree walker in Paginator](doctrine#7328) thanks to @plfort - [7325: Make code php 7.3 lint-compatible](doctrine#7325) thanks to @paxal - [7317: &doctrine#91;XML&doctrine#93; Fix default value of many-to-many order-by to ASC](doctrine#7317) thanks to @alexdenvir - [7260: Fix the handling of circular references in the commit order calculator](doctrine#7260) thanks to @stof - [6830: fix applying column options on foreign key columns](doctrine#6830) thanks to @Tobion Improvement ----------- - [7428: CI: Test against PHP 7.3](doctrine#7428) thanks to @Majkl578 - [7363: Fix compatibility with phan](doctrine#7363) thanks to @philippe-unitiz - [7345: Correct DOMDocument constructor in test](doctrine#7345) thanks to @guilliamxavier - [7307: Fix remaining usages of deprecated ClassLoader and Inflector from doctrine/common](doctrine#7307) thanks to @Majkl578 and @simonwelsh

ostrolucky previously requested changes Aug 6, 2018

View reviewed changes

ostrolucky added the Missing Tests label Aug 6, 2018

ostrolucky added Requires merge/review and removed Missing Tests labels Aug 9, 2018

Ocramius previously requested changes Aug 10, 2018

View reviewed changes

ostrolucky mentioned this pull request Aug 10, 2018

Update doctrine/coding-standard in 2.x branch #7350

Closed

ostrolucky added the Bug label Aug 14, 2018

Ocramius assigned ostrolucky Aug 26, 2018

Ocramius added this to the 2.6.3 milestone Aug 26, 2018

Ocramius previously approved these changes Aug 26, 2018

View reviewed changes

Handle removed parameters by tree walker in Paginator

1847d8d

ostrolucky dismissed Ocramius’s stale review via 1847d8d August 27, 2018 08:58

ostrolucky approved these changes Aug 27, 2018

View reviewed changes

ostrolucky merged commit 32efbd3 into doctrine:2.6 Aug 27, 2018

ostrolucky added Pick to master and removed Requires merge/review labels Aug 29, 2018

Majkl578 mentioned this pull request Sep 23, 2018

Cherry-pick stuff from 2.x into master #7405

Merged

Majkl578 removed the Pick to master label Oct 5, 2018

Conversation

plfort commented Jul 31, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ocramius commented Jul 31, 2018

Uh oh!

plfort commented Jul 31, 2018

Uh oh!

Ocramius commented Jul 31, 2018

Uh oh!

ostrolucky commented Aug 6, 2018

Uh oh!

Ocramius commented Aug 6, 2018

Uh oh!

plfort commented Aug 6, 2018

Uh oh!

ostrolucky left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plfort commented Aug 6, 2018

Uh oh!

ostrolucky commented Aug 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

plfort commented Aug 6, 2018

Uh oh!

ostrolucky commented Aug 9, 2018

Uh oh!

Ocramius left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ostrolucky Aug 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ostrolucky commented Aug 10, 2018

Uh oh!

Ocramius commented Aug 10, 2018

Uh oh!

ostrolucky commented Aug 10, 2018

Uh oh!

plfort commented Aug 11, 2018

Uh oh!

ostrolucky commented Aug 24, 2018

Uh oh!

Ocramius commented Aug 26, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

plfort commented Jul 31, 2018 •

edited

Loading

ostrolucky commented Aug 6, 2018 •

edited

Loading

ostrolucky Aug 10, 2018 •

edited

Loading