RankEval takes page size into consideration:

Currently RankEval only looks at the first 10 results of a query. Even if the number of results is specified, rank eval only requests 10 results and compares them with the expected results.

RankEval should consider the specified 'size' for calculations of the search quality.