Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash when using sqlite3 with subinterpreters #92036

Open
vstinner opened this issue Apr 28, 2022 · 8 comments
Open

Crash when using sqlite3 with subinterpreters #92036

vstinner opened this issue Apr 28, 2022 · 8 comments
Labels
3.9 3.10 type-bug

Comments

@vstinner
Copy link
Member

@vstinner vstinner commented Apr 28, 2022

Running sqlite3_crash.py on Python 3.9 and 3.10 fail randomly with an assertion failure in _PyObject_GC_UNTRACK() called by func_dealloc().

The problem is that a function is allocated in an interpreter and deallocated in another interpreter.

I built Python 3.10 with pyobject_ob_interp.patch. When the bug occurs, I get:

  • func_dealloc(): ob_interp = 0x7fffa4001030
  • In a parent frame (_PyObject_VectorcallTstate), tstate->interp = 0x7fffc0001030
@vstinner vstinner added the type-bug label Apr 28, 2022
@vstinner vstinner mentioned this issue Apr 28, 2022
Open
@vstinner vstinner mentioned this issue Apr 28, 2022
Open
@vstinner
Copy link
Member Author

@vstinner vstinner commented Apr 28, 2022

In the main branch, the _sqlite3 extension was converted to the multi-phase initialization API (PEP 489) which made the extension safer to use with subinterpreters.

@vstinner
Copy link
Member Author

@vstinner vstinner commented Apr 28, 2022

See also #90228 which is similar but has been fixed.

@vstinner
Copy link
Member Author

@vstinner vstinner commented Apr 28, 2022

In the 3.10 branch, the script doesn't crash before my change:

commit e6bb17fe29713368e1fd93d9ac9611017c4f570c (HEAD)
Author: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Date:   Thu Jan 13 10:50:09 2022 -0800

    bpo-46070: _PyGC_Fini() untracks objects (GH-30577)
    
    
    Py_EndInterpreter() now explicitly untracks all objects currently
    tracked by the GC. Previously, if an object was used later by another
    interpreter, calling PyObject_GC_UnTrack() on the object crashed if
    the previous or the next object of the PyGC_Head structure became a
    dangling pointer.
    (cherry picked from commit 1a4d1c1c9b08e75e88aeac90901920938f649832)
    
    Co-authored-by: Victor Stinner <vstinner@python.org>

@erlend-aasland
Copy link
Contributor

@erlend-aasland erlend-aasland commented Apr 28, 2022

I can reproduce this on my Mac on 3.9 and 3.10.

@vstinner
Copy link
Member Author

@vstinner vstinner commented Apr 28, 2022

It seems like the problem comes from the psyco_adapters dict shared by all interpreters: register_adapter(datetime.date, adapt_date) call in register_adapters_and_converters() of Lib/sqlite3/dbapi2.py.

Traceback when the bug occurs:

(gdb) py-bt
Traceback (most recent call first):
  <built-in method register_adapter of module object at remote 0x7fffd8118950>
  File "/home/vstinner/python/3.10/Lib/sqlite3/dbapi2.py", line 80, in register_adapters_and_converters
    register_adapter(datetime.date, adapt_date)
  File "/home/vstinner/python/3.10/Lib/sqlite3/dbapi2.py", line 85, in <module>
    register_adapters_and_converters()
  <built-in method exec of module object at remote 0x7fffe9bd5d30>
  (...)
  File "/home/vstinner/python/3.10/Lib/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  <built-in method exec of module object at remote 0x7fffe9bd5d30>
  (...)
  File "<string>", line 2, in <module>
  <built-in method run_in_subinterp of module object at remote 0x7fffea4a72f0>
  File "/home/vstinner/python/3.10/sqlite3_crash.py", line 12, in doIt
    _testcapi.run_in_subinterp(code)
  (...)

(gdb) frame 10
#10 0x00007fffe80c690d in pysqlite_register_adapter_impl (module=<module at remote 0x7fffd8118950>, type=0x7fffe80f3320 <PyDateTime_DateType>, 
    caster=<function at remote 0x7fffe9bf7330>) at /home/vstinner/python/3.10/Modules/_sqlite/module.c:175
175	    rc = pysqlite_microprotocols_add(type, (PyObject*)pysqlite_PrepareProtocolType, caster);

(gdb) frame 9
#9  0x00007fffe80c5dda in pysqlite_microprotocols_add (type=0x7fffe80f3320 <PyDateTime_DateType>, proto=<type at remote 0x7fffa80a9790>, 
    cast=<function at remote 0x7fffe9bf7330>) at /home/vstinner/python/3.10/Modules/_sqlite/microprotocols.c:68
68	    rc = PyDict_SetItem(psyco_adapters, key, cast);

(gdb) info threads
  Id   Target Id                                   Frame 
  1    Thread 0x7ffff7c7b740 (LWP 332120) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6
  3    Thread 0x7fffe9af8640 (LWP 332122) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6
  5    Thread 0x7fffe8af6640 (LWP 332124) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6
* 6    Thread 0x7fffdbfff640 (LWP 332125) "python" 0x00007ffff7d2454c in __pthread_kill_implementation () from /lib64/libc.so.6
  9    Thread 0x7fffda7fc640 (LWP 332128) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6
  10   Thread 0x7fffd9ffb640 (LWP 332129) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6
  16   Thread 0x7fffb67fc640 (LWP 332135) "python" 0x00007ffff7d1f3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6

vstinner added a commit to vstinner/cpython that referenced this issue Apr 28, 2022
@vstinner
pythongh-92036: gc_fini_untrack() leaks references
97eebfd 
When Py_EndInterpreter() is called to finalize a subinterpreter,
objects still tracked by the GC are untracked. Later, if one of these
object is deleted and its deallocator function expect the object to
be tracked by the GC, the deallocator function can crash.

Leak one strong reference on purpose so the object is never deleted
and its deallocator is not called.
vstinner added a commit to vstinner/cpython that referenced this issue Apr 28, 2022
@vstinner
pythongh-92036: Fix gc_fini_untrack()
7acfea9 
Fix a crash in subinterpreters related to the garbage collector. When
a subinterpreter is deleted, untrack all objects tracked by its GC.
To prevent a crash in deallocator functions expecting objects to be
tracked by the GC, leak a strong reference to these objects on
purpose, so they are never deleted and their deallocator functions
are not called.
@vstinner
Copy link
Member Author

@vstinner vstinner commented Apr 28, 2022

In the 3.10 branch, the script doesn't crash before my change

Well, gc_fini_untrack() solves one problem but not all of them :-( I wrote PR #92037 to fix this issue.

Explanation of this bug:

  • (1) Interpreter A is created
  • (2) Interpreter A creates an adapt_date() function and stores it in sqlite3 adapters using _sqlite3.register_adapter() function
  • (3) Interpreter B is created
  • (4) Interpreter B does the same: create an adapt_date() function and overrides the sqlite3 adapters registered by interperter A using _sqlite3.register_adapter() function
  • (5) Interpreter A is deleted
  • (6) Interpreter B is deleted

In this order, everything is fine: the interpreter A still exists when the interpreter B overrides the datetime adapter.

The problem is when the interpreter A is deleted before the interpreter B registers is its own adapter (override the key in the shared dict). In this case, the function is untracked by the GC. Problem: deleting a function requires the function to be tracked by the GC. Othserwise, _PyObject_GC_UNTRACK() assertion fails: this function must only be called on objects called by the GC.

@vstinner
Copy link
Member Author

@vstinner vstinner commented Apr 28, 2022

By the way, when the bug occurs, Python built in debug mode displays this error:

$ ./python sqlite3_crash.py 
Objects/funcobject.c:641: _PyObject_GC_UNTRACK: Assertion "(((PyGC_Head *)(op)-1)->_gc_next != 0)" failed: object not tracked by the garbage collector
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7faa31e94730
object refcount : 0
object type     : 0x88d660
object type name: function
object repr     :

But then hangs: _PyObject_Dump() tries to acquire the GIL, but another interpreter is holding it...

@vstinner
Copy link
Member Author

@vstinner vstinner commented Apr 28, 2022

cc @ericsnowcurrently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.9 3.10 type-bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vstinner @erlend-aasland