Skip to content

[FW][FIX] *: adapt for noble#160842

Closed
fw-bot wants to merge 8 commits intoodoo:17.0from
odoo-dev:17.0-15.0-noble-xdo-fgE9-fw
Closed

[FW][FIX] *: adapt for noble#160842
fw-bot wants to merge 8 commits intoodoo:17.0from
odoo-dev:17.0-15.0-noble-xdo-fgE9-fw

Conversation

@fw-bot
Copy link
Copy Markdown
Contributor

@fw-bot fw-bot commented Apr 6, 2024

This pr backports some fixed for python3.11 and introduces new fixes for Ubuntu Noble, such as timezones.

Forward-Port-Of: #160449
Forward-Port-Of: #151989

@robodoo
Copy link
Copy Markdown
Contributor

robodoo commented Apr 6, 2024

Pull request status dashboard.

@fw-bot
Copy link
Copy Markdown
Contributor Author

fw-bot commented Apr 6, 2024

@Xavier-Do cherrypicking of pull request #151989 failed.

stdout:

Auto-merging addons/web_editor/controllers/main.py
On branch 17.0-15.0-noble-xdo-fgE9-fw
You are currently cherry-picking commit 25aca4e0fd72.
  (all conflicts fixed: run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

nothing to commit, working tree clean

stderr:

13:44:57.457633 git.c:463               trace: built-in: git cherry-pick 25aca4e0fd72a346c4bfcdbd2ee75b1d2e8b83b1
13:44:57.565762 run-command.c:659       trace: run_command: GIT_REFLOG_ACTION=cherry-pick git commit -n --no-gpg-sign -F .git/MERGE_MSG --cleanup=verbatim --allow-empty-message
13:44:57.568128 git.c:463               trace: built-in: git commit -n --no-gpg-sign -F .git/MERGE_MSG --cleanup=verbatim --allow-empty-message
The previous cherry-pick is now empty, possibly due to conflict resolution.
If you wish to commit it anyway, use:

    git commit --allow-empty

Otherwise, please use 'git cherry-pick --skip'
----------
status:

Either perform the forward-port manually (and push to this branch, proceeding as usual) or close this PR (maybe?).

In the former case, you may want to edit this PR message as well.

⚠️ after resolving this conflict, you will need to merge it via @robodoo.

More info at https://github.com/odoo/odoo/wiki/Mergebot#forward-port

@robodoo robodoo added forwardport This PR was created by @fw-bot conflict There was an error while creating this forward-port PR labels Apr 6, 2024
@C3POdoo C3POdoo added the RD research & development, internal work label Apr 6, 2024
@chapital
Copy link
Copy Markdown

chapital commented Apr 7, 2024

#138728 also gonna have support for python 3.12?

@Xavier-Do
Copy link
Copy Markdown
Contributor

Yes I will work on 3.12 in a followup pr.

pimodoo and others added 8 commits April 7, 2024 10:49
When python expression is evaluated in odoo form an action or qweb, we
are checking the opcodes generated by the evaluation of this code. We do
such a verification, because the code from actions and templates can be
written by someone having not access to the server and we don't want to
let them perform actions out of the scope of their database.

In python 3.11, some opcodes from previous versions of Python have been
renamed, grouped or sepcified. There are also new ones that have been
introduce.

In this PR, we are whitelisting the new ones that are needed by odoo to
properly work in this version of Python.
see lxml/lxml@cc5ddbb

Co-authored-by: Xavier Morel (xmo) <xmo@odoo.com>
In latest versions of werkzeug, the `werkzeug.urls` module has been
reduced to remove feature present in urllib.parse. This commit vendored
the old version to avoid breaking compatibility with older versions of
odoo on ubuntu Noble, without adapting the whole codebase. This
should/may be removed in stable by adaptaing eveything to urlib.

This version of the lib was minimalized to avoid redondunce with feature
still present in werkzeug.urls. Some unused features in odoo are still
vendored for ease but are not exposed on the werkzeug.urls for now.
Some of the non canononical timezones are not present in Ubuntu Noble,
it would be a better practice to only use canonical timezones in data
and tests.

Note that this is not a real fix for all cases since the database that
ran on Ubuntu Jammy and are moved to an ubuntu Noble server will have
the issue with timezones already in database.

One of the possible fix would be to manage that during upgrades, but
this isn't a verry flexible solution since upgrade are meant to manage
chyange of version, not change of server. If an old 17.0 versions needs
to be moved to a Noble server, this won't work.

Another solution would be to install package like tzdata-legacy that may
keep the old timezones but it is not the only think since TAI-10 are
also in this package. This solution is not ideal because non canonical
timezone will still be shown in the dropdown. We would need to filter
them.

A last solution would be to add the support for those old timezones by
monkeypatching the lib. This way, only new timezones would be shown but
non canonical one won't crash when used. This is not ideal either
because we may need to keep this for a while. But in combination with
the upgrade solution, it may work proprely.
In ubuntu noble, some timezone where removed leading to errors when
trying to assign/access them.

This was partially fixed in the code by removing all references to old
timezones but one issue remains: if a database contains timezones that
are not defined in the os, the resolution will fail and break at runtime

This patches proposes to alter timezone to fallback on the new canonical
timezone if the timezone was removed.

This list was generated by checking all symlink in /usr/share/zoneinfo
in ubuntu 22.04 that disapeared in ubuntu 24.04

This solutions will work when moving a database from one server to
another, even without migration.

The all_timezone is not modified on purpose to avoid breaking existing
logic. This list may be used to define if a timezone is known by
postgress, define selection fiels, .... we don"t want to increase the
list in those case.

Some other logic using all_timezone may need to be updated but This
will be done in master.
The main purpose of this test is to ensure that line return are removed,
but all other special character are kept once the file is saved.

Unfortunately, werkzeug versions have different strategies on how to
quote the filename, removing less special character in latest versions
like after pallets/werkzeug@babfc93

This also changes after the changes that removed werkzeug urls methods
replacing them by urllib, making the behaviour different again.

This commit makes the test more robust by checking that the filename
correspond to the expeted one once unquoted, not comparing the quoted
versions.
@Xavier-Do Xavier-Do force-pushed the 17.0-15.0-noble-xdo-fgE9-fw branch from 208158e to 83be6dd Compare April 7, 2024 08:53
@C3POdoo C3POdoo requested review from a team and xmo-odoo and removed request for a team April 7, 2024 08:55
@C3POdoo C3POdoo requested review from a team, BastienFafchamps, kebeclibre and ryv-odoo and removed request for a team April 7, 2024 08:55
@Xavier-Do
Copy link
Copy Markdown
Contributor

@robodoo override=ci/style
@robodoo override=ci/security
(see previous pr)
@robodoo r+

robodoo pushed a commit that referenced this pull request Apr 7, 2024
When python expression is evaluated in odoo form an action or qweb, we
are checking the opcodes generated by the evaluation of this code. We do
such a verification, because the code from actions and templates can be
written by someone having not access to the server and we don't want to
let them perform actions out of the scope of their database.

In python 3.11, some opcodes from previous versions of Python have been
renamed, grouped or sepcified. There are also new ones that have been
introduce.

In this PR, we are whitelisting the new ones that are needed by odoo to
properly work in this version of Python.

Part-of: #160842
robodoo pushed a commit that referenced this pull request Apr 7, 2024
see lxml/lxml@cc5ddbb

Part-of: #160842
Co-authored-by: Xavier Morel (xmo) <xmo@odoo.com>
robodoo pushed a commit that referenced this pull request Apr 7, 2024
robodoo pushed a commit that referenced this pull request Apr 7, 2024
In latest versions of werkzeug, the `werkzeug.urls` module has been
reduced to remove feature present in urllib.parse. This commit vendored
the old version to avoid breaking compatibility with older versions of
odoo on ubuntu Noble, without adapting the whole codebase. This
should/may be removed in stable by adaptaing eveything to urlib.

This version of the lib was minimalized to avoid redondunce with feature
still present in werkzeug.urls. Some unused features in odoo are still
vendored for ease but are not exposed on the werkzeug.urls for now.

Part-of: #160842
robodoo pushed a commit that referenced this pull request Apr 7, 2024
Some of the non canononical timezones are not present in Ubuntu Noble,
it would be a better practice to only use canonical timezones in data
and tests.

Note that this is not a real fix for all cases since the database that
ran on Ubuntu Jammy and are moved to an ubuntu Noble server will have
the issue with timezones already in database.

One of the possible fix would be to manage that during upgrades, but
this isn't a verry flexible solution since upgrade are meant to manage
chyange of version, not change of server. If an old 17.0 versions needs
to be moved to a Noble server, this won't work.

Another solution would be to install package like tzdata-legacy that may
keep the old timezones but it is not the only think since TAI-10 are
also in this package. This solution is not ideal because non canonical
timezone will still be shown in the dropdown. We would need to filter
them.

A last solution would be to add the support for those old timezones by
monkeypatching the lib. This way, only new timezones would be shown but
non canonical one won't crash when used. This is not ideal either
because we may need to keep this for a while. But in combination with
the upgrade solution, it may work proprely.

Part-of: #160842
robodoo pushed a commit that referenced this pull request Apr 7, 2024
In ubuntu noble, some timezone where removed leading to errors when
trying to assign/access them.

This was partially fixed in the code by removing all references to old
timezones but one issue remains: if a database contains timezones that
are not defined in the os, the resolution will fail and break at runtime

This patches proposes to alter timezone to fallback on the new canonical
timezone if the timezone was removed.

This list was generated by checking all symlink in /usr/share/zoneinfo
in ubuntu 22.04 that disapeared in ubuntu 24.04

This solutions will work when moving a database from one server to
another, even without migration.

The all_timezone is not modified on purpose to avoid breaking existing
logic. This list may be used to define if a timezone is known by
postgress, define selection fiels, .... we don"t want to increase the
list in those case.

Some other logic using all_timezone may need to be updated but This
will be done in master.

Part-of: #160842
robodoo pushed a commit that referenced this pull request Apr 7, 2024
The main purpose of this test is to ensure that line return are removed,
but all other special character are kept once the file is saved.

Unfortunately, werkzeug versions have different strategies on how to
quote the filename, removing less special character in latest versions
like after pallets/werkzeug@babfc93

This also changes after the changes that removed werkzeug urls methods
replacing them by urllib, making the behaviour different again.

This commit makes the test more robust by checking that the filename
correspond to the expeted one once unquoted, not comparing the quoted
versions.

Part-of: #160842
@robodoo robodoo closed this in e73094c Apr 7, 2024
@chapital
Copy link
Copy Markdown

chapital commented Apr 8, 2024

Hi @Xavier-Do i have installed PyPDF2 version 3.0.1, I see in the requeriments the correct version to work with odoo is: 2.12.0, what happens if I use this version(3.0.1)?

@Xavier-Do
Copy link
Copy Markdown
Contributor

We didn't test with 3.0.1, the version in ubuntu Noble is 2.12.1 so even for python 3.12 the pinned version will remain 2.12.1

So, it may work, or not, I don't know :)

We may switch to pypdf one day, and adapt it for 4.0 but this would most likely be a task for master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflict There was an error while creating this forward-port PR forwardport This PR was created by @fw-bot RD research & development, internal work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants