Skip to content

Conversation

@kulikjak
Copy link
Contributor

@kulikjak kulikjak commented Dec 17, 2018

Solaris uses '646' as an alias for ASCII encoding and thus all the locale coercion tests are failing. This simple patch fixes the problem.

https://bugs.python.org/issue37335

Alexey Izbyshev and others added 30 commits November 8, 2018 23:12
This typo doesn't affect the result because wrong bits are discarded
on implicit conversion to unsigned char, but it trips UBSan
with -fsanitize=implicit-integer-truncation.





https://bugs.python.org/issue35194
Since the n==1 case just returns *max*, it cannot be larger
than the magnitude of the vector entry.
1) Convert weird field name "typ" to the more standard "type".
2) For the NUMBER type, convert the value to an int() or float().
3) Simplify ``group(kind)`` to the shorter and faster ``group()`` call.
4) Simplify logic go a single if-elif chain to make this easier to extend.
5) Reorder the tests to match the order the tokens are specified.
   This isn't necessary for correctness but does make the example
   easier to follow.
6) Move the "column" calculation before the if-elif chain so that
   users have the option of using this value in error messages.
…H-10429)

Gives approx 20% speed-up using clang depending on the number of elements in the set (the less dense the set, the more the speed-up).

Uses the same entry++ logic used elsewhere in the setobject.c code.
Currently, the *n* and *total* variables get converted to floats each time they are multiplied by random().  This minor tweak does the conversion just once and gets a small speedup (approx 3%).
* All internal header files now require Py_BUILD_CORE or
  Py_BUILD_CORE_BUILTIN to be defined.
* _json.c is now compiled with Py_BUILD_CORE_BUILTIN to access
  pycore_accu.h header.
* Add an example to Modules/Setup to show how to build _json
  as a built-in module; it requires non trivial compiler options.
* _PyTuple_ITEMS() gives access to the tuple->ob_item field and cast the
  first argument to PyTupleObject*. This internal macro is only usable if
  Py_BUILD_CORE is defined.
* Replace &PyTuple_GET_ITEM(ob, 0) with _PyTuple_ITEMS(ob).
* Replace PyTuple_GET_ITEM(op, 1) with &_PyTuple_ITEMS(ob)[1].
Co-Authored-By: maggyero <gery.ogam@gmail.com>
The System Preferences Dock "prefer tabs always" setting disables some
IDLE features.  Menus are a bit different than as described for Windows
and Linux.
Remove unneeded assertion (we already know so is a PySetObject *).
…_range(). (pythonGH-10442)

This function may access memory which is mapped but is considered
free by libc allocator. It behaves so by design, therefore we
need to suppress sanitizer reports.

GCC doesn't support MSan, so disable only TSan for it.
* ast.h now includes Python-ast.h and node.h
* parsetok.h now includes node.h and grammar.h
* symtable.h now includes Python-ast.h
* Modify asdl_c.py to enhance Python-ast.h:

  * Add #ifndef/#define Py_PYTHON_AST_H to be able to include the header
    twice
  * Add "extern { ... }" for C++
  * Undefine "Yield" macro conflicting with winbase.h

* Remove "#undef Yield" from C files, it's now done in Python-ast.h
* Remove now useless includes in C files
The documentation was not covering multiple targets enclosed by
parenthesis nor multiple targets enclosed by brackets, adding them all
would be heavy, an else cover them all and is lighter to read.
…honGH-10337)

* universal_newlines defaulting to False would suggest, that not
  specifying universal_newlines explicitly and setting text to True
  should cause an error, which is not the case.
* The run function didn't have the universal_newlines parameter
  documented
* The check_output function didn't have its text parameter documented
Rename Include/internal/ headers:

* pycore_hash.h -> pycore_pyhash.h
* pycore_lifecycle.h -> pycore_pylifecycle.h
* pycore_mem.h -> pycore_pymem.h
* pycore_state.h -> pycore_pystate.h

Add missing headers to Makefile.pre.in and PCbuild:

* pycore_condvar.h.
* pycore_hamt.h
* pycore_pyhash.h
a asynchronous generator -> an asynchronous generator
Adds configure flags for msan and ubsan builds to make it easier to enable.
These also encode the detail that address sanitizer and memory sanitizer
should disable pymalloc.

Define MEMORY_SANITIZER when appropriate at build time and adds workarounds
to existing code to mark things as initialized where the sanitizer is otherwise unable to
determine that.  This lets our build succeed under the memory sanitizer.  not all tests
pass without sanitizer failures yet but we're in pretty good shape after this.
The warnings module now suggests to enable tracemalloc if the source
is specified, tracemalloc module is available, but tracemalloc is not
tracing memory allocations.
clang's MemorySanitizer understand getc() but does not understand
getc_unlocked().  Workaround: Don't use it on msan builds.
ned-deily and others added 7 commits December 11, 2018 01:06
pythonGH-11047)

This speeds up pickling of some iterators.

This fixes also error handling in pickling methods when fail to
look up builtin "getattr".
…11077)

* bpo-35454: Fix miscellaneous minor issues in error handling.

* Fix a null pointer dereference.
The following test fails if a different process creates or removes
a file on the same disk partition between the two lines:

    usage = shutil.disk_usage(os.path.dirname(__file__))
    self.assertEqual(usage, shutil.disk_usage(__file__))

Only test that disk_usage() succeed on a filename, but don't check
the result. Add also tests on the fields type (must be int).
…rkserver on Windows (pythonGH-11086)

Forkserver and fork are not available on Windows and therefore these test must be skipped.
…onGH-11087)

The test only except SIGUSR1Exception inside wait_signal(), but the signal can be sent during subprocess_send_signal() call.
"""
data = data.replace(b"ANSI_X3.4-1968", b"ascii")
data = data.replace(b"US-ASCII", b"ascii")
data = data.replace(b"646", b"ascii")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a alias iso_646. Would not it be convenient to add it?

Copy link
Contributor Author

@kulikjak kulikjak Dec 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it can be done, but there are many ASCII aliases; maybe it's reasonable to add only those that are actually used by platforms.

@zooba
Copy link
Member

zooba commented Jan 10, 2019

Do we have a buildbot that's failing on this? Something we can watch to see whether this fixes the issue?

@csabella csabella requested a review from ncoghlan May 11, 2019 02:19
@ncoghlan
Copy link
Contributor

ncoghlan commented Jun 9, 2019

Huh, I thought we had an OpenIndiana buildbot in the "unstable" set, but it looks like that isn't true anymore (if it was ever actually the case): https://buildbot.python.org/all/#/grid

The '646' alias is present in the main codec table though (see https://docs.python.org/3/library/codecs.html#standard-encodings), so I think this is a genuine bug in the test case.

Given a b.p.o issue number and a testing section NEWS entry, I think it's low risk enough that we can just add the extra case, even if the buildbots aren't exercising it.

@kulikjak
Copy link
Contributor Author

kulikjak commented Jun 12, 2019

I am not sure why, but the test_c_locale_coercion passes without any problem on Python 3.8. So I guess this problem was fixed somewhere between 3.7 and 3.8.

It makes no sense to add it to 3.8, but can it be merged with 3.7 only (if so I will create new issue and NEWS entry for that)?

@ncoghlan
Copy link
Contributor

Yes, we could do this as a 3.7 test-only change.

As far as what changed in 3.8 goes, I suspect the change to get sys.getfilesystemencoding() to report ascii instead of roman8 on HP-UX also got it to stop reporting 646 on Solaris (the startup code now does a codec lookup on whatever the OS reports, which means it correctly translates aliases to their canonical Python codec name, which it didn't previously do).

@kulikjak kulikjak changed the title Add alias for ASCII encoding bpo-37335: Add alias for ASCII encoding Jun 19, 2019
@ncoghlan
Copy link
Contributor

@kulikjak The target branch is going to need to be changed to 3.7, but I'm not sure changing the target of this PR will do the right thing (your branch is presumably based on master, so changing the target could attempt to merge all of Python 3.8+ into the 3.7 maintenance branch).

I'm going to try it anyway, and if it works, cool, but if not, would you be OK creating a new 3.7 based PR with your two commits from this branch added?

@ncoghlan ncoghlan changed the base branch from master to 3.7 June 21, 2019 12:06
@ncoghlan ncoghlan closed this Jun 21, 2019
@ncoghlan
Copy link
Contributor

Ah, that's what I was afraid would happen :(

Copy link
Contributor

@ncoghlan ncoghlan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy with the change, but we're going to need a replacement PR that targets the 3.7 branch, since GitHub wasn't clever enough to only include the PR commits when changing the target branch.

@kulikjak
Copy link
Contributor Author

Ah, I see, I should have made that right away...
I will create a new pull request against the 3.7 branch as soon as possible.

Thanks for the info :)

@kulikjak kulikjak deleted the fix-locale-coercion-alliases branch June 21, 2019 12:30
@kulikjak kulikjak restored the fix-locale-coercion-alliases branch June 21, 2019 12:30
@kulikjak kulikjak deleted the fix-locale-coercion-alliases branch June 21, 2019 12:30
@kulikjak kulikjak restored the fix-locale-coercion-alliases branch June 21, 2019 12:30
@kulikjak kulikjak deleted the fix-locale-coercion-alliases branch June 21, 2019 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting review tests Tests in the Lib/test dir

Projects

None yet

Development

Successfully merging this pull request may close these issues.