What kind of feature are you missing? Where do you notice a shortcoming of PTB?

Creating the changelogs is the part that takes up the most time of the release process. Even though I have already put some time into writing a helper tool for that, several manual steps are still necessary:

• running the helper tool
• checking & adapting the assignment of contributions to categories
• writing HTML-formatted text for the TG Channel announcement
• rerunning (parts of) the script again if something went wrong/was wrongly formatted
• copying the GH release notes to GH (which is currently rather cumbersome b/c I do the building on a separate linux machine)
• copying the TG news to the channel

I would like to automate as much of this process as possible.
Additionally, it was pointed out that it would be benefitial to elaborate the individual entries of the changelog, in particular deprecations and recommended transition steps.

Both goals can be achieved by moving the creation of the release not for the PR along with the PR rather than later at release time.

Benefits that I see

• reduces time required to do a release, which allows for more frequent releases (e.g. faster release after API changes are incorprated)
• an automated build & deploy process doesn't require setup on a local machine such that in case of a maintainer switch, the new maintainer has less to set up
• the time I currently spent on doing releases, I can then use for other work on PTB (or for a nap :))

Possible problems that I see

As PTB has grown to be a rather elaborate package with several cross-connections in the code and intricate test setups, and high standards on documentation, contributions from novices are already not always easy. Adding yet another thing to do for a PR might negatively influence the number of exetrnal PRs we get and the experience of the contributors.

OTOH 1) the majority of contributions comes from the ptb dev team anyway and 2) we can try to automate & help with this step as well.

Describe the solution you'd like

Part 1: Writing the release notes as part of the PR

The naive first approach would be that each PR just adds its contents to the changelog file. However, that will quickly lead to a bunch of merge conflicts.
I've looked at some existing tools:

• towncrier comes from the twisted community
• blurb is used by CPython release managers
• reno was developed by the openstack community

They all have a similar approach in common:

1. for each change, a file ("fragment") with a unique name is created. That file contains information about the change
2. the release notes are then rendered by aggregating the information from those files

They all also provide command line tools for creating such a new file and for the renedring the docs
I'd like to follow the same approach. Whether we can use one of the existing tools depends on what kind of content we want to see in the release notes

Part 1.1 Desired content

Let detail how I would personally envision our release notes to look like:

• mix of info for consumers and "all changes listed". Looking at the reno changelog you won't see entries for internal changes (tests, test-dependencies, code quality, documentation updates) and you also won't see any PR & issue numbers. While we do want to write our release notes for the users of our library, IMO it is fair to both ends to display all the work we do on PTB. I'd like to keep including all PRs that go into a release and also include the relevant PR and issue numbers
• for each change, the commit message/PR title should be the fallback. That suffices for dependabot PRs & such, but for new features & sim, there should be a full-sentence description of what the change does. For deprecations and removals, instructions should be included on how the user should migrate (in a standalone section). BTW, this would also remove the need for our Transition guides in the future :)
• sections with changes of different types. Keep a Changelog has established itself as a relibale standard afaik, so we could use that as base. OTOH sections like "internal", "documentation" and "upgrade instructions" make sense as well
• in terms of transparancy, the author of the PR should be included as well. Currently, we include only external contributors and exclude dependabot & pre-commit-ci. I'd be okay with including them for simplicity.

Part 1.2 evaluation of existing solutions

• blurb is tailored rather strictly to CPython IISC. I don't think this is a good fit.
• towncrier is better, has customizable sections, however it has two downsides IMO:
  • it rather heavily relies on each change coming from an issue (which is not the case for use by far)
  • it deletes all fragments after creating sphinx documentation
• reno goes even further than towncrier IMO and always keeps the fragments. It also has a strong focus on handling supporting different branches with backports - that's explained pretty well in the video on the website. This is not very important for us IMO since we usually only support one branch. The 13.x branch was a rather big exception.

Neither towncrier nor reno have native support for author names and PR numbers.

In conclusion I would say that reno and towncrier are options if we can live without those.
If we don't want to live without that, I guess a custom tool will be the thing to go for. This I would see as a standalone repo (possibly pypi package). Ofc this means non-trivial initial programming effort, but hopefully it saves us a lot of time in the future.

Part 2: Automation of the process

Part 2.1: Helping in PRs

What I'd like to see here is a GH workflow that checks if there is already a fragment file for this PR and if not creates one with some default content (pr title, category based on labels, closing issues, that sort). The user can then ofc easily edit it locally and we can also help with that.
By adding such a workflow we also make sure that we do not merge PRs without a fragment file

Part 2.2: Building changelog

I general, I have come to think that it's wiser to keep the changelog content in a machine readable form rather than a human readable form as that makes it significantly easier to alter the output later on. So I would in fact not significantly aggregate the fragment files on release. Combining all fragments into a single file for the release is okay, but I'd probably stop there.

Displaying the changelog is then only a matter of rendering the machine readable output:

• Sphinx: This should be our primary target IMO so I'd write the fragments already in rst format. reno provides a sphinx-extension with a corresponding directive for that
• GH Releases: Converting rst to md/html is rather easily feasible with pandoc + some custom panflute filters - did that already in earlier versions of ptb-changelog-helper
• TG channel post: This is trickier, since TG has a very limited set of HTML options. pandoc + panflute filters can take care of that, alternatively maybe first convert to HTML and then use something like bleach to remove unsupported tags. In both cases, TBH I somewhat fear that the processing might be a bit unstable making an automation require manual interaction every now and then. Another option would be to simply link to the docs/GH releases page in the channel.

Part 2.3: Release Workflow

Ideally, I'd like the release workflow to be something like this:

1. create PR to bump version number. NEXT.VERSION is replaced either locally or ideally a GH workflow takes care of that.
2. If fragment files need to be combined into a single file and/or CHANGES.rst needs to be touched, a GH workflow does that
3. merge PR. this triggers a workflow (probably in a release environment such that manual approval is required, see [FEATURE] Automated PyPI Release #4318 for more info) that does the following:
   1. create a corresponding tag
   2. converts the releases notes to GH actions format and adds them to the release
   3. creates the TG post and sends it to our channel (via a bot)

Combined with the automated build & publish proposed in #4318 this would mean that a release is a single PR that ideally requires only upaditing the version number.

Describe alternatives you've considered

No response

Additional context

The other part of the release process - building & publishing - is adressed in the separate thread #4318. My goal is that ideally (almost) the complete release & deply process can be automated, maximising the benefits I lsited above :)

Regarding overview over deprecations, I've seen this nice page in the sphinx docs: https://www.sphinx-doc.org/en/master/extdev/deprecated.html
Depending on how we bulid our fragment files, we might be able to auto-generate something like this. I would see this as a secondary goal.