Skip to content

gh-76511: Fix email.Message.as_string() for non-ASCII message with ASCII charset#116125

Merged
serhiy-storchaka merged 2 commits intopython:mainfrom
serhiy-storchaka:email-nonascii-as_string
Mar 5, 2024
Merged

gh-76511: Fix email.Message.as_string() for non-ASCII message with ASCII charset#116125
serhiy-storchaka merged 2 commits intopython:mainfrom
serhiy-storchaka:email-nonascii-as_string

Conversation

@serhiy-storchaka
Copy link
Copy Markdown
Member

@serhiy-storchaka serhiy-storchaka commented Feb 29, 2024 •
edited by bedevere-app bot
Loading

@serhiy-storchaka serhiy-storchaka added needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels Feb 29, 2024
@serhiy-storchaka serhiy-storchaka requested a review from a team as a code owner February 29, 2024 15:09
@bedevere-app bedevere-app bot mentioned this pull request Feb 29, 2024
Closed
@serhiy-storchaka serhiy-storchaka mentioned this pull request Feb 29, 2024
Closed
bitdancer
bitdancer reviewed Mar 1, 2024
View reviewed changes
Misc/NEWS.d/next/Library/2024-02-29-17-06-54.gh-issue-76511.WqjRLP.rst Outdated
@@ -0,0 +1,2 @@
Fix :meth:`email.Message.as_string` for non-ASCII message with ASCII
charset.
Copy link
Copy Markdown
Member

@bitdancer bitdancer Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't really describe the change very clearly. I would say something like "Fix exception in message_as_string that results when a message that claims to be in the ascii character set actually has non-ascii characters by replacing the non-ascii characters with ?, like the 'replace' parameter of encode does." Or something like that.

I don't want this comment to be a blocker, though.

Copy link
Copy Markdown
Member Author

@serhiy-storchaka serhiy-storchaka Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your suggestion @bitdancer.

@serhiy-storchaka serhiy-storchaka merged commit f97f25e into python:main Mar 5, 2024
@miss-islington-app
Copy link
Copy Markdown

miss-islington-app bot commented Mar 5, 2024

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 5, 2024
@serhiy-storchaka @miss-islington
pythongh-76511: Fix email.Message.as_string() for non-ASCII message w…
6b18987 
…ith ASCII charset (pythonGH-116125)

(cherry picked from commit f97f25e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-app
Copy link
Copy Markdown

bedevere-app bot commented Mar 5, 2024

GH-116364 is a backport of this pull request to the 3.12 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.12 only security fixes label Mar 5, 2024
@serhiy-storchaka serhiy-storchaka deleted the email-nonascii-as_string branch March 5, 2024 15:49
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 5, 2024
@serhiy-storchaka @miss-islington
pythongh-76511: Fix email.Message.as_string() for non-ASCII message w…
3d4d722 
…ith ASCII charset (pythonGH-116125)

(cherry picked from commit f97f25e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-app
Copy link
Copy Markdown

bedevere-app bot commented Mar 5, 2024

GH-116365 is a backport of this pull request to the 3.11 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.11 only security fixes label Mar 5, 2024
serhiy-storchaka added a commit that referenced this pull request Mar 5, 2024
@miss-islington @serhiy-storchaka
[3.12] gh-76511: Fix email.Message.as_string() for non-ASCII message …
85c32ad 
…with ASCII charset (GH-116125) (GH-116364)

(cherry picked from commit f97f25e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit that referenced this pull request Mar 5, 2024
@miss-islington @serhiy-storchaka
[3.11] gh-76511: Fix email.Message.as_string() for non-ASCII message …
c75df4b 
…with ASCII charset (GH-116125) (GH-116365)

(cherry picked from commit f97f25e)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 25, 2024
@serhiy-storchaka @adorilson
pythongh-76511: Fix email.Message.as_string() for non-ASCII message w…
a0704b2 
…ith ASCII charset (pythonGH-116125)
diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024
@serhiy-storchaka @diegorusso
pythongh-76511: Fix email.Message.as_string() for non-ASCII message w…
a037ff7 
…ith ASCII charset (pythonGH-116125)
@asottile-sentry
Copy link
Copy Markdown

asottile-sentry commented Apr 24, 2024

fwiw this seems to have caused a user-visible regression in django (at least) -- django/django#17979

oddly enough I don't think surrogates should have been involved at all -- the messages we're seeing were entirely utf-8-encodable and using a utf-8 charset. only internally after this series of changes between 3.11.6 and 3.11.9 they seem to have surrogates 🤔

my gut is something is wrong with this or one of the other recent unicode-related email patches and should probably be looked at. I can spend some time finding a minimal reproduction if that'd be useful

medmunds added a commit to anymail/django-anymail that referenced this pull request May 16, 2024
@medmunds
Tests: fix broken inbound test
4c62f7b 
Django SafeMIMEText corner case
behaves differently in Python
3.11.9+, 3.12.3+, likely due to
python/cpython#116125
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bitdancer bitdancer bitdancer left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@serhiy-storchaka @asottile-sentry @bitdancer