Consider per-page settings when iterating paginated properties by EnricoMi · Pull Request #3377 · PyGithub/PyGithub

EnricoMi · 2025-09-16T19:21:16Z

The GitHub API has the "feature" of paginated properties. Some objects returned by the API have a property that allows for pagination. Fetching subsequent pages of that property means fetching the entire object (with all other properties) and the specified page of the paginated property. Iterating over the paginated property means fetching all other properties multiple times. Fortunately, the allowed size of each page (per_page is usually 300, in contrast to the "usual" per_page maximum of 100).

Objects with paginated properties:

Commit.files
Comparison.commits
EnterpriseConsumedLicenses.users

This PR makes iterating those paginated properties use the configured per_page setting.

It further allows to specify an individual per_page when either retrieving such objects, or fetching paginated properties.

Methods returning objects with paginated properties:

Repository.compare
Repository.get_commit
Enterprise.get_consumed_licenses

Accessing paginated properties with individual per_page value:

Commit.get_files
Comparison.get_commits
EnterpriseConsumedLicenses.get_users

Note: setting the Github(per_page=…) to a value other than the default of 30 will cause paginated properties to use that setting, unless a different value is specified when retrieving such objects or their paginated properties via methods listed above. For instance, Commit.files fetches paged with that number of files. Before, 300 files were fetched with each request.

github-actions · 2025-09-16T19:25:22Z

Test Results

8 files ± 0 8 suites ±0 4m 31s ⏱️ +41s
1 107 tests + 24 1 107 ✅ + 24 0 💤 ±0 0 ❌ ±0
14 048 runs +464 14 047 ✅ +464 1 💤 ±0 0 ❌ ±0

Results for commit 17cf134. ± Comparison against base commit 24305f6.

♻️ This comment has been updated with latest results.

github/GithubObject.py

github/Enterprise.py

github/GithubObject.py

iarspider · 2025-10-07T14:41:38Z

tests/GithubObject.py

+        # commits is a PaginatedList, which should respect per_page
+        commits = list(reversed(comparison.commits))
+        commit = commits[-1]
+        # files is a PaginatedList, which should respect per_page


This is not true. Even for explicitly paginated requests (page=1&per_page=N), the url attribute is returned without pagination information, thus overwriting url used in construction of object(s).

So maybe we need to change all _useAttributes methods to not overwrite _url if it's set?

iarspider · 2025-10-07T14:41:49Z

Before

PyGithub/github/PaginatedList.py

Line 217 in 9ca6b4b

# this paginated list contains a single page

, I suggest to add

            # firstUrl, taken from url property of returned object, doesn't contain per_page
            if self._reversed:
                if self.__lastUrl:
                    self.url = self.__lastUrl
            else:
                if self.__firstUrl:
                    self.url = self.__firstUrl

iarspider · 2025-10-09T16:17:39Z

Unit Tests are green, yay!

iarspider · 2025-11-12T09:59:05Z

@EnricoMi a kind reminder of this PR. We would really like to get this merged and not maintain patches.

EnricoMi · 2025-11-18T10:22:09Z

@iarspider I have to go over this once more as this is a significant change. But it should be done properly. I hope the notion of paginated properties and when to provide a per_page does affect what is properly documented and understandable.

…class

…dProperty

EnricoMi · 2026-01-12T10:58:43Z

@iarspider this is finally complete, please have a look and test with your use-case.

https://pygithub--3377.org.readthedocs.build/en/3377/utilities.html#classes-with-paginated-properties

iarspider · 2026-01-14T19:31:17Z

@iarspider this is finally complete, please have a look and test with your use-case.

https://pygithub--3377.org.readthedocs.build/en/3377/utilities.html#classes-with-paginated-properties

Thanks! I'll try to check this ASAP.

EnricoMi · 2026-02-15T10:37:41Z

@iarspider have you had a chance to take a look / test on your side?

EnricoMi · 2026-03-22T11:12:10Z

@iarspider thanks for raising this issue!

iarspider · 2026-03-22T11:22:08Z

@EnricoMi thanks! Sorry, I couldn't test it - I am no longer working on that project.

This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [PyGithub](https://redirect.github.com/pygithub/pygithub) | `==2.8.1` → `==2.9.0` | ![age](https://developer.mend.io/api/mc/badges/age/pypi/pygithub/2.9.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/pygithub/2.8.1/2.9.0?slim=true) | | [pygithub](https://redirect.github.com/pygithub/pygithub) | `==2.8.1` → `==2.9.0` | ![age](https://developer.mend.io/api/mc/badges/age/pypi/pygithub/2.9.0?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/pygithub/2.8.1/2.9.0?slim=true) | --- ### Release Notes <details> <summary>pygithub/pygithub (PyGithub)</summary> ### [`v2.9.0`](https://redirect.github.com/PyGithub/PyGithub/releases/tag/v2.9.0) [Compare Source](https://redirect.github.com/pygithub/pygithub/compare/v2.8.1...v2.9.0) ##### Notable changes ##### Lazy PyGithub objects The notion of lazy objects has been added to some PyGithub classes in version 2.6.0. This release now makes all `CompletableGithubObject`s optionally lazy (if useful). See [PyGithub/PyGithub#3403](https://redirect.github.com/PyGithub/PyGithub/pull/3403) for a complete list. In lazy mode, getting a PyGithub object does not send a request to the GitHub API. Only accessing methods and properties sends the necessary requests to the GitHub API: ```python # Use lazy mode g = Github(auth=auth, lazy=True) # these method calls do not send requests to the GitHub API user = g.get_user("PyGithub") # get the user repo = user.get_repo("PyGithub") # get the user's repo pull = repo.get_pull(3403) # get a known pull request issue = pull.as_issue() # turn the pull request into an issue # these method and property calls send requests to Github API issue.create_reaction("rocket") # create a reaction created = repo.created_at # get property of lazy object repo # once a lazy object has been fetched, all properties are available (no more requests) licence = repo.license ``` All PyGithub classes that implement `CompletableGithubObject` support lazy mode (if useful). This is only useful for classes that have methods creating, changing, or getting objects. By default, PyGithub objects are not lazy. ##### PyGithub objects with a paginated property The GitHub API has the "feature" of paginated properties. Some objects returned by the API have a property that allows for pagination. Fetching subsequent pages of that property means fetching the entire object (with all other properties) and the specified page of the paginated property. Iterating over the paginated property means fetching all other properties multiple times. Fortunately, the allowed size of each page (`per_page` is usually 300, in contrast to the "usual" `per_page` maximum of 100). Objects with paginated properties: - Commit.files - Comparison.commits - EnterpriseConsumedLicenses.users This PR makes iterating those paginated properties use the configured `per_page` setting. It further allows to specify an individual `per_page` when either retrieving such objects, or fetching paginated properties. See [Classes with paginated properties](https://pygithub.readthedocs.io/en/stable/utilities.html#utilities-classes-with-paginated-properties) for details. ##### Drop Python 3.8 support due to End-of-Life Python 3.8 reached its end-of-life September 6, 2024. Support has been removed with this release. ##### Deprecations - Method `delete` of `Reaction` is deprecated, use `IssueComment.delete_reaction`, `PullRequestComment.delete_reaction`, `CommitComment.delete_reaction` or `Issue.delete_reaction` instead. - Method `Issue.assignee` and parameter `Issue.edit(assignee=…)` are deprecated, use `Issue.assignees` and `Issue.edit(assignees=…)` instead. - Method `Organization.edit_hook` is deprecated, use `Organization.get_hook(id).edit(…)` instead. If you need to avoid `Organization.get_hook(id)` to fetch the `Hook` object from Github API, use a lazy Github instance: ```python Github(…, lazy=True).get_organization(…).get_hook(id).edit(…) ``` - Methods `Team.add_to_members` and `Team.remove_from_members` are deprecated, use `Team.add_membership` or `Team.remove_membership` instead. ##### New Features - Consider per-page settings when iterating paginated properties by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3377](https://redirect.github.com/PyGithub/PyGithub/pull/3377) - Add Secret Scanning Alerts and Improve Code Scan Alerts by [@matt-davis27](https://redirect.github.com/matt-davis27) in [PyGithub/PyGithub#3307](https://redirect.github.com/PyGithub/PyGithub/pull/3307) ##### Improvements - More lazy objects by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3403](https://redirect.github.com/PyGithub/PyGithub/pull/3403) - Allow for enterprise base url prefixed with `api.` by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3419](https://redirect.github.com/PyGithub/PyGithub/pull/3419) - Add `throw` option to `Workflow.create_dispatch` to raise exceptions by [@dblanchette](https://redirect.github.com/dblanchette) in [PyGithub/PyGithub#2966](https://redirect.github.com/PyGithub/PyGithub/pull/2966) - Use `GET` url or `_links.self` as object url by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3421](https://redirect.github.com/PyGithub/PyGithub/pull/3421) - Add support for `type` parameter to get\_issues by [@nrysk](https://redirect.github.com/nrysk) in [PyGithub/PyGithub#3381](https://redirect.github.com/PyGithub/PyGithub/pull/3381) - Align implemented paths with OpenAPI spec by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3413](https://redirect.github.com/PyGithub/PyGithub/pull/3413) - Add suggested OpenAPI schemas by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3411](https://redirect.github.com/PyGithub/PyGithub/pull/3411) - Apply OpenAPI schemas by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3412](https://redirect.github.com/PyGithub/PyGithub/pull/3412) ##### Bug Fixes - Fix `PaginatedList.totalCount` returning 0 with GitHub deprecation notices by [@odedperezcodes](https://redirect.github.com/odedperezcodes) in [PyGithub/PyGithub#3382](https://redirect.github.com/PyGithub/PyGithub/pull/3382) - Use default type if known type is not supported by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3365](https://redirect.github.com/PyGithub/PyGithub/pull/3365) ##### Maintenance - Deprecate `Reaction.delete` by [@iarspider](https://redirect.github.com/iarspider) in [PyGithub/PyGithub#3435](https://redirect.github.com/PyGithub/PyGithub/pull/3435) - Deprecate `Issue.assignee` by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3366](https://redirect.github.com/PyGithub/PyGithub/pull/3366) - Deprecate `Orginization.edit_hook` by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3404](https://redirect.github.com/PyGithub/PyGithub/pull/3404) - Deprecate `Team.add_to_members` and `Team.remove_from_members` by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3368](https://redirect.github.com/PyGithub/PyGithub/pull/3368) - Various minor OpenAPI fixes by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3375](https://redirect.github.com/PyGithub/PyGithub/pull/3375) - Update test key pair by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3453](https://redirect.github.com/PyGithub/PyGithub/pull/3453) - Pin CI lint Python version to 3.13 by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3406](https://redirect.github.com/PyGithub/PyGithub/pull/3406) - Improve error message on replay data mismatch by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3385](https://redirect.github.com/PyGithub/PyGithub/pull/3385) and [PyGithub/PyGithub#3386](https://redirect.github.com/PyGithub/PyGithub/pull/3386) - Disable sleeps in tests by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3383](https://redirect.github.com/PyGithub/PyGithub/pull/3383) - Update autodoc defaults by [@Aidan-McNay](https://redirect.github.com/Aidan-McNay) in [PyGithub/PyGithub#3369](https://redirect.github.com/PyGithub/PyGithub/pull/3369) - Add Python 3.14 to CI and tox by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3429](https://redirect.github.com/PyGithub/PyGithub/pull/3429) - Restrict PyPi release workflow permissions by [@JLLeitschuh](https://redirect.github.com/JLLeitschuh) in [PyGithub/PyGithub#3418](https://redirect.github.com/PyGithub/PyGithub/pull/3418) - Fix OpenApi workflow by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3389](https://redirect.github.com/PyGithub/PyGithub/pull/3389) - Bump codecov/codecov-action from 3 to 5 by [@dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3284](https://redirect.github.com/PyGithub/PyGithub/pull/3284) - Bump actions/setup-python from 5 to 6 by [@dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3370](https://redirect.github.com/PyGithub/PyGithub/pull/3370) - Bump dawidd6/action-download-artifact from 3.0.0 to 3.1.4 by [@dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3282](https://redirect.github.com/PyGithub/PyGithub/pull/3282) - Bump github/codeql-action from 3 to 4 by [@dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3391](https://redirect.github.com/PyGithub/PyGithub/pull/3391) - Bump actions/upload-artifact from 4 to 5 by [@dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3394](https://redirect.github.com/PyGithub/PyGithub/pull/3394) - Bump actions/download-artifact from 5 to 6 by [@dependabot](https://redirect.github.com/dependabot)\[bot] in [PyGithub/PyGithub#3393](https://redirect.github.com/PyGithub/PyGithub/pull/3393) - Drop Python 3.8 support due to EOL by [@hugovk](https://redirect.github.com/hugovk) in [PyGithub/PyGithub#3191](https://redirect.github.com/PyGithub/PyGithub/pull/3191) - Merge changelog updates from v2.8 release branch by [@EnricoMi](https://redirect.github.com/EnricoMi) in [PyGithub/PyGithub#3367](https://redirect.github.com/PyGithub/PyGithub/pull/3367) #### New Contributors - [@odedperezcodes](https://redirect.github.com/odedperezcodes) made their first contribution in [PyGithub/PyGithub#3382](https://redirect.github.com/PyGithub/PyGithub/pull/3382) - [@Aidan-McNay](https://redirect.github.com/Aidan-McNay) made their first contribution in [PyGithub/PyGithub#3369](https://redirect.github.com/PyGithub/PyGithub/pull/3369) - [@nrysk](https://redirect.github.com/nrysk) made their first contribution in [PyGithub/PyGithub#3381](https://redirect.github.com/PyGithub/PyGithub/pull/3381) - [@matt-davis27](https://redirect.github.com/matt-davis27) made their first contribution in [PyGithub/PyGithub#3307](https://redirect.github.com/PyGithub/PyGithub/pull/3307) **Full Changelog**: <PyGithub/PyGithub@v2.8.0...v2.9.0> </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/lettuce-financial/github-bot-signed-commit).

…#3377 patterns Co-authored-by: EnricoMi <44700269+EnricoMi@users.noreply.github.com> Agent-Logs-Url: https://github.com/PyGithub/PyGithub/sessions/d885b909-c041-400f-9b6a-f3d87aa5118c

The GitHub API has the "feature" of paginated properties. Some objects returned by the API have a property that allows for pagination. Fetching subsequent pages of that property means fetching the entire object (with all other properties) and the specified page of the paginated property. Iterating over the paginated property means fetching all other properties multiple times. Fortunately, the allowed size of each page (`per_page` is usually 300, in contrast to the "usual" `per_page` maximum of 100). Objects with paginated properties: - `Commit.files` - `Comparison.commits` - `EnterpriseConsumedLicenses.users` This PR makes iterating those paginated properties use the configured `per_page` setting. It further allows to specify an individual `per_page` when either retrieving such objects, or fetching paginated properties. Methods returning objects with paginated properties: - `Repository.compare` - `Repository.get_commit` - `Enterprise.get_consumed_licenses` Accessing paginated properties with individual `per_page` value: - `Commit.get_files` - `Comparison.get_commits` - `EnterpriseConsumedLicenses.get_users` Note: setting the `Github(per_page=…)` to a value other than the default of 30 will cause paginated properties to use that setting, unless a different value is specified when retrieving such objects or their paginated properties via methods listed above. For instance, `Commit.files` fetches paged with that number of files. Before, 300 files were fetched with each request. --------- Co-authored-by: iarspider <iarspider@gmail.com> Co-authored-by: EnricoMi <44700269+EnricoMi@users.noreply.github.com>

Agent-Logs-Url: https://github.com/PyGithub/PyGithub/sessions/72084bf8-f285-40db-ac5f-806ab93c2a89 Co-authored-by: EnricoMi <44700269+EnricoMi@users.noreply.github.com>

EnricoMi mentioned this pull request Sep 16, 2025

Commit.files does not respect requester.per_page (in forward mode) #3372

Closed

EnricoMi force-pushed the commit-files-reversed-perpage branch from 34e9af5 to 3bf2c7e Compare September 17, 2025 19:34

EnricoMi changed the title ~~Test reversed commit files with per-page~~ Consider per-page when getting Commit files Sep 17, 2025

EnricoMi marked this pull request as ready for review September 17, 2025 19:35

EnricoMi force-pushed the commit-files-reversed-perpage branch from 3bf2c7e to a64138e Compare September 17, 2025 19:48

iarspider reviewed Oct 2, 2025

View reviewed changes

github/GithubObject.py Outdated Show resolved Hide resolved

iarspider reviewed Oct 7, 2025

View reviewed changes

github/Enterprise.py Outdated Show resolved Hide resolved

iarspider reviewed Oct 7, 2025

View reviewed changes

github/GithubObject.py Outdated Show resolved Hide resolved

iarspider reviewed Oct 7, 2025

View reviewed changes

EnricoMi force-pushed the commit-files-reversed-perpage branch from 9ca6b4b to 2bcfcb1 Compare October 8, 2025 13:00

iarspider mentioned this pull request Oct 21, 2025

Simplify regexes by doing extra preprocessing cms-sw/cms-bot#2551

Merged

EnricoMi added this to the Version 2.9.0 milestone Nov 18, 2025

iarspider mentioned this pull request Nov 18, 2025

Commit.get_combined_status().statuses should be a PaginatedList #3387

Open

Copilot AI mentioned this pull request Nov 19, 2025

Make CommitCombinedStatus.statuses a PaginatedList #3417

Open

8 tasks

EnricoMi added a commit to Felixoid/PyGithub that referenced this pull request Jan 7, 2026

Left TODO on asserting requests once PyGithub#3377 is merged

b703021

EnricoMi added 10 commits January 7, 2026 11:54

Test commit files with per-page

c4b3bb1

Set per-page when getting Commit files

441eeed

Refactor out transformer and apply to Repository.get_commits

3e0b47d

Refactor out logic into CompletableGithubObjectWithPaginatedProperty …

4bf07a2

…class

Make Comparison a CompletableGithubObjectWithPaginatedProperty

3842058

Remove most reversed tests

74efb2b

Make EnterpriseConsumedLicenses a CompletableGithubObjectWithPaginate…

c9f0ba3

…dProperty

Allow to specify per_page for paginated property

c40ac05

Move paginated properties tests into separate test class

804bbce

More tests

37a306f

EnricoMi added 3 commits January 9, 2026 22:02

Run paginated properties tests lazily and eagerly

401ecd4

Simplify getter for paginated property

a369f92

Make order of url parameters deterministic

ab4c942

EnricoMi changed the title ~~Consider per-page when getting Commit files~~ Consider per-page settings when iterating paginated properties Jan 10, 2026

EnricoMi added 3 commits January 10, 2026 17:15

Fix linting

3c4bcdc

Fix method order in CompletableGithubObject

dc69c2a

Remove redundant assertions, add comments

fb367de

EnricoMi force-pushed the commit-files-reversed-perpage branch from 4fdf989 to fb367de Compare January 10, 2026 16:15

Remove redundant code, improve assertion, revert tox change

2473af8

EnricoMi force-pushed the commit-files-reversed-perpage branch 4 times, most recently from 5f7df43 to 8a7e258 Compare January 12, 2026 09:21

EnricoMi added 3 commits January 12, 2026 10:40

Extend documentation

061f2e1

Always set pagination_parameters, not just for incomplete objects

6bc9e30

Fixing tests

cdbb128

EnricoMi force-pushed the commit-files-reversed-perpage branch from 8a7e258 to cdbb128 Compare January 12, 2026 10:54

EnricoMi mentioned this pull request Feb 15, 2026

Deprecate paginated properties in Commit and Comparison #3380

Closed

Merge branch 'main' into commit-files-reversed-perpage

17cf134

EnricoMi merged commit b1a9b7e into main Mar 22, 2026
24 of 25 checks passed

EnricoMi deleted the commit-files-reversed-perpage branch March 22, 2026 11:11

Uh oh!

Conversation

EnricoMi commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iarspider Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

iarspider Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

iarspider commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iarspider commented Oct 9, 2025

Uh oh!

iarspider commented Nov 12, 2025

Uh oh!

EnricoMi commented Nov 18, 2025

Uh oh!

EnricoMi commented Jan 12, 2026

Uh oh!

iarspider commented Jan 14, 2026

Uh oh!

EnricoMi commented Feb 15, 2026

Uh oh!

Uh oh!

EnricoMi commented Mar 22, 2026

Uh oh!

iarspider commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

EnricoMi commented Sep 16, 2025 •

edited

Loading

github-actions bot commented Sep 16, 2025 •

edited

Loading

iarspider commented Oct 7, 2025 •

edited

Loading