Python 3.12 started to report errors when mock assertion is done with a "common" mistake:

mock.not_called()

This does not do what you think it does, and when you run it with Python 3.12 you will get error:

'not_called' is not a valid assertion. Use a spec for the mock if 'not_called' is meant to be an attribute.

The correct way of calling it is:

mock.assert_not_called()

Similar for other methods. It's very easy to miss it during review and for Python < 3.12 that call gets silently ignored (obviously it looks like you are calling not_called method on mock and it will just record the call in the mock.

For projects like Apache Airflow where we optimize most PRs and run most tests with "lowest" supported Python (Python 3.8) this leads sometimes to false-positives where the tests start to fail in Python 3.12 "canary" build where it performs full suite of tests (not mentioning that this test passing on Python 3.8 does not test anything anyway because it does not make any assertion).

Example case where it caused our "canary" builds to fail apache/airflow#42027

It would be great to detect such "mistakes" via static checks with ruff as a new rule.