-
-
Notifications
You must be signed in to change notification settings - Fork 4.5k
Prototype async REPL using IPython, take III #11265
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This is a squash and a rebase of a large number of commits from Min and I. For simplicity of managing it, history has been reduced to a single commit, but more historical versions can be found, in particular in PR 11155, or commit aedb5d6 to be more exact.
|
I'm thinking of dropping 3.4, as maintaining compat with it will be painful:
We'll also be supporting 3.5, 3.6 and 3.7. @minrk and @takluyver thoughts ? |
|
asyncio support is pretty valuable and long overdue, so if 3.4 makes it too hard, I think the trade-off is worth it. |
setup.py
Outdated
|
|
||
| extras_require.update({ | ||
| ':python_version == "3.4"': ['typing'], | ||
| ':python_version >= "3.5"': ['trio', 'curio'], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Neither of these are dependencies, right? We shouldn't require these packages.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One of them is necessary if we want to have nested embed(). I can make that optional, I probably added them here to have the test passing.
call async loops synchronously on their own. ipython#11265
call async loops synchronously on their own. ipython#11265 Handle early return conditions from coro
|
@Carreau I ran into an issue running python files that have their own loop calls. So if I have a file that calls Can we change the run_cell_async coro to first yield interactivity so it can call the pseudo sync when async is not required? is a working example. |
|
I have to better understand the implication. It is likely possible, though it will likely work only on terminal IPython. It might also not stay forever, as in the end I want IPython to always run on an event loop. Though, I guess we could run only on the pseudo-sync, or a non-asyncio one. Once that hapend we can figure out how to patch |
|
The way I would imagine it working in the long run is:
Of course there's a ways to go before we use async annotations everywhere like that. But we need to eventually reach a point of keeping the same loop alive over multiple cells, if we want this to be really useful. In any case, I think the basic idea of tracking whether we're in sync or async mode, and turning the magic parser on-or-off as appropriate, is a good one to expose to the user. And that's something that can be done now and would solve @dalejung's problem. |
|
...of course when writing the above I forgot about prompt-toolkit. Besides it needing some work to integrate with all these different loops, I'm not sure that it'll be happy switching between loops mid-stream... Well, the architecture I described above ought to work for the kernel, or for |
|
Thanks ! Much appreciated. |
|
That would close #9166 as well. |
|
Todo, catch exception when the runner crash, EG, when using [EDIT]: Now done. |
Make sure that if the coroutine runner encounter an exception (likely when encountering the wrong coroutine from another aio library), it is set on the result and does not crash the interpreter.
|
Timeit, time, and prun magics will need updates to work with async code. |
| @@ -1,3 +1,6 @@ | |||
| Depreations | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
deprecations
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, Fixed.
| interactivity = 'async' | ||
| # yield interactivity so let run_cell decide whether to use | ||
| # an async loop_runner | ||
| yield interactivity |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding yield interactivity here makes this a not-entirely-valid coroutine, doesn't it? When I try to call await run_cell_async from ipykernel, I get a bad yield: 'last_expr'. How can we communicate this required info without faking that this isn't actually a coroutine? Does it need to be an actual async generator?
|
That's right. You need to do a small dance. I believe I have a patch on my
machine for ipykernel to make this work.
You need to advance the coro by sending None to it once. We could do it in
another way, but at some point we'll always hit the fact that python does
not have a real async-exec and that we need to distinguish 2 path of
running sync code, or asyc one.
…On Tue, Sep 4, 2018, 16:43 Min RK ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In IPython/core/interactiveshell.py
<#11265 (comment)>:
> @@ -2780,9 +2984,14 @@ def error_before_exec(value):
self.displayhook.exec_result = result
# Execute the user code
- interactivity = 'none' if silent else self.ast_node_interactivity
- has_raised = self.run_ast_nodes(code_ast.body, cell_name,
- interactivity=interactivity, compiler=compiler, result=result)
+ interactivity = "none" if silent else self.ast_node_interactivity
+ if _run_async:
+ interactivity = 'async'
+ # yield interactivity so let run_cell decide whether to use
+ # an async loop_runner
+ yield interactivity
Adding yield interactivity here makes this a not-entirely-valid
coroutine, doesn't it? When I try to call await run_cell_async from
ipykernel, I get a bad yield: 'last_expr'. How can we communicate this
required info without faking that this isn't actually a coroutine? Does it
need to be an actual async generator?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#11265 (review)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAUez6-l5irZuy9QnGMdamhrYJsfea77ks5uXpGXgaJpZM4V7qox>
.
|
3156d6d to
85db703
Compare
|
Ok, seem like we've reached quasi static state and most of the latest updates were documentation. Plus ipykernel works with that. So I'm going to merge (thanks all involved !), and we can refine if necessary. |
|
Fair enough. I'm just wondering if there's a better way to check if it should be async explicitly first, rather than relying on the fact that coroutines are actually just a special kind of generators, and having to call it in a special way that no other coro can be called. |
|
#11289 makes run_cell_async a regular coroutine by moving the 'should it be async' check to a separate method. Since in general, running it with a coroutine runner ought to work (I know there are edge cases for now), skipping this check should be safe. |
Alyssa Whitwell (3):
Change signature to be imported from inspect library
Add .python-version to gitignore
Deprecate use of imp library, condense module_paths module to only one function, update tests
Anatol Ulrich (1):
fix #11082 - use dpaste.com, also remove py2 compatibility imports
Benjamin Ragan-Kelley (19):
upgrade pip on appveyor
Merge pull request #11073 from Carreau/remove-io-meth
Merge pull request #11072 from Carreau/remove-profile
pretty-rendering for os.environ by default
Merge pull request #11177 from takluyver/prompt-toolkit-2
Merge pull request #11185 from gpotter2/spec-missing
Merge pull request #11172 from cedric05/master
Merge pull request #11174 from Carreau/close-bg
Merge pull request #11178 from toddrme2178/patch-1
Merge pull request #11250 from Carreau/invalid-escape-sequence
Merge pull request #11278 from Carreau/cleanup-doctest
make `run_cell_async` a regular coroutine
exercise calling run_cell_async as a coroutine
catch BaseException in coroutine runner
Merge pull request #11287 from Carreau/bash-raise
Merge pull request #11307 from Carreau/safe-input-transformer
Merge pull request #11305 from Carreau/more-autoawait-docs
Merge pull request #11311 from Carreau/rst-formatting
Merge pull request #11301 from Carreau/cleanup-manifest
Chaz Reid (1):
Document -o flag for logstart magic
Christoph (5):
Disable qt inputhook backend when DISPLAY is not set on linux, because the qt lib does not handle this correct
apply review comments
type
check also WAYLAND_DISPLAY
apply comments
Dale Jung (1):
Only trigger async loop runner when needed. Allows running files that call async loops synchronously on their own. ipython/ipython#11265
Dave Hirschfeld (1):
Allow specifying the name of the root element of a JSON display object
Fernando Perez (1):
Merge pull request #11107 from zichongkao/repr_doc
Fred Mitchell (1):
Create LICENSE
Gabriel Potter (1):
Fix autogen_shortcuts.py for prompt_toolkit 2.0
Grant Nestor (2):
Drop jquery dependency in IPython.display.Javascript
Take jquery's getScript approach
Hugo (1):
Drop support for Python 3.3
Jonathan Slenders (1):
Upgrade to prompt_toolkit 2.0
Jörg Dietrich (2):
Add ipython_tokens for syntax highlighting following cell magic
add three tests for cell magic syntax highlighting
Kyle Kelley (1):
Merge pull request #11267 from gnestor/display-javascript
M Pacer (1):
create new --raise-error argument to raise an error if nonzero exitcode
Matthew Seal (2):
Added fix for display hook call output format
Added missing imports for test
Matthias Bussonnier (145):
Remove Deprecation Warning, add since when things were deprecated.
Merge pull request #10833 from takluyver/req-py34
remove some py3compat usage
Fix readme: Quote to Codeblock
Update LICENCE and Copying to match.
ignore .editorconfig
Remove deprecated profile line magic.
Deprecate some methods from IPython.utils.io
Merge pull request #10648 from srinivasreddy/srinivasreddy-patch-1
Merge pull request #11110 from takluyver/skip-jedi-omission-test
Merge pull request #11109 from stonebig/patch-1
Merge pull request #11106 from takluyver/displayobjs-metadata
Merge pull request #11101 from MSeal/captureHookFixPy3
What's new in 5.7
What's new in 6.4
Merge pull request #11119 from Carreau/wn64
Refactor of coloring and traceback mechanism.
Traceback printing change `in <module>()` to `in <module>`.
Some other cleaning of ultratb
Remove import indirection and deprecated private function.
Fix timeti test nightly/3.7
Update execution.py
fix tests
Leak less files during test.
Fix test for Python 3.7+
test on 3.7
Merge pull request #11137 from joergdietrich/magic_highlighting
Merge pull request #11131 from Carreau/fix-timeit-nightly
Merge pull request #11135 from Carreau/fix-37
Merge pull request #11134 from Carreau/fix-fleaks
Merge pull request #11129 from Carreau/cleanup
Merge pull request #11127 from Carreau/clean-tb
Merge pull request #11128 from Carreau/non-callable-module
Some cleanup of Pycolorize.
Merge pull request #11130 from Carreau/less-indir
remove unsused utitlity function
Merge pull request #11148 from Carreau/deprecated
Bump Deprecation to 8.0, API has not stabilized.
Autoclose stdin/err for bg scripts, when not assigned to user_ns
Correctly tearDown some test cases with multiple inheritance.
Swap base class instead of redefinign the tearDown method
add docstring, and emit deprecation warnings
Merge pull request #11211 from dhirschfeld/patch-2
Merge pull request #11192 from alphaCTzo7G/fix_typo_ipython_magic
Merge pull request #11199 from mpacer/script_raise_error
Merge pull request #11227 from yen223/11226-3-7-autocomplete-bugfix
Upgrade pip before setuptools for Python compat.
Switch protocol to https
Merge pull request #11173 from Carreau/reswarn
what's new in 6.5
fix release instructions
Merge pull request #11245 from Carreau/fix-release-instructions
Update version6.rst
Merge pull request #11243 from Carreau/whatsnew65
Use raw string when invalid escape sequence present.
Fix more escape sequences
Merge pull request #11255 from alyssawhitwell/add-python-version-to-gitignore
Merge pull request #11254 from alyssawhitwell/11206-signature-warning
Merge pull request #11253 from parente/fix-ipynb
Update deprecation warning message
Merge pull request #11256 from alyssawhitwell/deprecate-imp-import
Merge pull request #11251 from Carreau/deprecated-imports
Prototype async REPL using IPython, take III
fix UnboundLocalError
titleto have doc build
fix docs
Load the asycn ext only on 3.5+
fix runnign on 3.7
remove duplicate WatsNew from bad rebase
test with trio
3.7 still nto on travis
reformat with black
doc, remove appveyor 3.4
move infor to the right place
DeprecationWarning
runblack on new tests
att triio runner at top level now that 3.4 drop + docs
Add pseudo sync mode
no f-strings
docs cleanup, reformat code, remove dead code.
remove trio from test requirement and skipp if not there
fix test
Merge pull request #11271 from luzpaz/misc-typo-and-whitespaces
Test that using wrong runner/coroutine pair does not crash.
run tests with trio and curio on travis
Undeprecate autoindent
Merge remote-tracking branch 'origin/master' into inputtransformer2
add notes some magic don't work yet
Merge remote-tracking branch 'origin/master' into inputtransformer2
Actually run some tests.
Merge pull request #11041 from takluyver/inputtransformer2
Remove Python 2 shim.
Update test_ipdoctest.py
Merge pull request #11277 from Carreau/map-is-lazy
typo
add tests
test both old and new classes
Add some extra metadata for PyPI (in the sidebar)
remove unused utils function in setupbase.py
Switch scripts cell magic default to raise if error happens.
Merge pull request #11283 from oscar6echo/improve-autoreload
Tool to go around sphinx limitation during developpement
run doc fixing tool on travis
Merge pull request #11288 from tkf/inputhook
Merge pull request #11286 from Carreau/unused-utils-in-setupbase
Merge pull request #11285 from Carreau/project_url
Merge pull request #11265 from Carreau/async-repl-aug-2018
Start to updates the what's new / changelog.
update release process
fix comments
build error on traivs
Merge pull request #11289 from minrk/async-real-coroutine
Add debug and fix build hopefully
Merge pull request #11291 from minrk/catch-base-exception
typos
Merge pull request #10974 from boeddeker/master
Merge pull request #11290 from Carreau/wn7
cleanup build process
Remove include from MANIFEST
Merge pull request #11304 from michael-k/python3.7
Merge pull request #11300 from luzpaz/misc-typos
Revert 3.7 AST fix
update docs about autoawait
Add some inputtransformer test cases
Some propose fixes.
Better alternative; try each transformer in a row,
fix Paul's comments
one more rephrasing/plural
one more plural
Rst formatting (typo and decrease level to not be in sidebar)
Fix magic directive and role.
Update autoawait.rst
Merge pull request #11308 from yarko/patch-1
Merge pull request #11313 from Carreau/fix-magic-directive
update instructions: --pre apply to rc as well
Merge pull request #11318 from blink1073/allow-setting-matchers
Merge pull request #11315 from Carreau/instr-rc
Fix the sphinx_ipython directive.
Merge pull request #11321 from Carreau/fix-ipython-directive
Merge pull request #11323 from hongshaoyang/patch-1
release 7.0.0rc1
back to development
Prepare changelog for relese
Update what's new and stats
release 7.0.0
Michael Käufl (1):
[travis] Run tests against Python 3.7 and a non-outdated nightly
Paul Ganssle (5):
Improve async detection mechanism with blacklist
Add tests for SyntaxError with top-level return
Fix indentation problem with _asyncify
Fix async_helpers tests
Refactor and tweak async-in-function tests
Paul Ivanov (8):
updated what's new
fix typos, migrate latest to release notes
add preliminary 7.0 release stats
release 7.0.0b1
back to development
Merge pull request #11299 from Carreau/cleanup-build-process
Merge pull request #11162 from Carreau/revert-37-ast-fix
minor typo and style tweak
Peter Parente (1):
Fix breaking newline
Sourav Singh (1):
Add deprecationwarning with stacklevel
Srinivas Reddy Thatiparthy (1):
remove deprecated variable - default_gui_banner
Steven Silvester (1):
Allow showing dict keys only
Subhendu Ranjan Mishra (2):
Fixes #11068 : Cleanup extra logic that handle python <=3.3 in IPython/core/extensions.py
Issue #11068 : Cleanup extra logic that handle python <=3.3 in IPython/utils/openpy.py
Takafumi Arakaki (1):
Pass inputhook to prompt_toolkit
Thomas A Caswell (1):
FIX: Remove the non-interactive backends from pylabtools.backend2gui
Thomas Kluyver (109):
Working on new input transformation machinery
Start adding tests for inputtransformer2
Add transformation for system assignments
Factor out handling of line continuations
Debugging function to see tokens
Escaped commands
Some more tests for escaped commands
Transformations for 'help?' syntax
Fix cell magic transformation
More tests for line-based input transformers
Start integrating new input transformation machinery into InteractiveShell
Start adding code for checking when input is complete
Start integrating new machinery for checking code completeness
Fix fallback prompt & improve info on test failure
Fix complete checking with space after backslash
Ensure that result is defined in run_cell()
Update test for new transformation API
Document the new input transformation API
Switch some references to input_splitter to input_transformer_manager
Remove unused TerminalMagics.input_splitter
Update sphinxext for new API
Remove unused import
Switch some imports from inputsplitter to inputtransformer2
Mark inputsplitter & inputtransformer as deprecated
Switch inputtransformer2 back to stdlib tokenize module
Drop bundled, outdated copy of the tokenize module
Add shell.check_complete() method
Convert syntax warnings to errors when checking code completeness
This is an incompatible change
Working on new input transformation machinery
Start adding tests for inputtransformer2
Add transformation for system assignments
Factor out handling of line continuations
Debugging function to see tokens
Escaped commands
Some more tests for escaped commands
Transformations for 'help?' syntax
Fix cell magic transformation
More tests for line-based input transformers
Start integrating new input transformation machinery into InteractiveShell
Start adding code for checking when input is complete
Start integrating new machinery for checking code completeness
Fix fallback prompt & improve info on test failure
Fix complete checking with space after backslash
Ensure that result is defined in run_cell()
Update test for new transformation API
Document the new input transformation API
Switch some references to input_splitter to input_transformer_manager
Remove unused TerminalMagics.input_splitter
Update sphinxext for new API
Remove unused import
Switch some imports from inputsplitter to inputtransformer2
Mark inputsplitter & inputtransformer as deprecated
Switch inputtransformer2 back to stdlib tokenize module
Drop bundled, outdated copy of the tokenize module
Add shell.check_complete() method
Convert syntax warnings to errors when checking code completeness
This is an incompatible change
master branch is now 7.0 dev
Add whatsnew development doc back to toc
Drop support for Python 3.3
Bump up Python version on Appveyor too
Simplify some pathlib imports
Update error message in setup.py
Update version numbers for Python support
Merge pull request #11069 from minrk/upgrade-pip
Tweak wording referring to LICENSE
Merge pull request #11067 from Carreau/less-2-compat
Merge pull request #11018 from souravsingh/add-cod
Merge pull request #11071 from Carreau/rdm
Merge pull request #10758 from fm75/master
Disable Jedi completions by default
Enable jedi for tests that apply to that
Move where use_jedi is disabled again back in test
Correct help string for config option
Add changelog for 6.3.1
Merge pull request #11076 from takluyver/disable-jedi
Merge pull request #11087 from tacaswell/fix_noninteractive_backends
Merge pull request #11093 from subhrm/11068-B
Merge pull request #11092 from subhrm/11068-A
Display objects should emit metadata when it exists
Comment out jedi parts of test_omit__names
Merge pull request #11118 from Carreau/wn57
Merge pull request #11123 from luzpaz/fix-typos
Merge pull request #11142 from Carreau/cleanup-pyc
Merge pull request #11147 from charlesreid1/patch-1
Merge pull request #11156 from Carreau/completerdeprecationbump
Merge branch 'inputtransformer2' of github.com:takluyver/ipython into inputtransformer2
Don't let parentheses level go below 0
Prevent infinite loops in input transformation
Merge pull request #11164 from minrk/os-environ-pretty
Arrange imports, add missing, remove unused
Remove commented method
Fix input rewrite prompt
No newline after output prompt
Merge pull request #11181 from apunisal/master
Merge pull request #11182 from spookyvision/master
Merge pull request #11190 from infmagic2047/fix-completion-style
Simpler mechanism to extend input transformations
Merge branch 'master' into inputtransformer2
Fix generating shortcuts docs
Shortcut to transform_cell in magics code
Update tests for input transformer raising SyntaxError
Update transformer test in terminal test case
Add & improve docstrings following @willingc's review
Reformat for readability
Add description of priority system
Merge pull request #11235 from Carreau/https
Merge pull request #11234 from Carreau/pip-first
Todd (1):
Include LICENSE file in wheels
Wei Yen (3):
Add regression test
Ensure `try_import` does not fail on null module.__file__
Reset sys.path after test
Yarko Tymciurak (1):
update documentation to add beta install instructions
Yutao Yuan (1):
Fix config option TerminalInteractiveShell.display_completions
Zi Chong Kao (2):
Document that _repr_*_() can return metadata
Document that _repr_*_() can return metadata
alphaCTzo7G (1):
Fixes default value of -r<R> and updates documentation
apunisal (1):
Changed a misspelled word in tools/git-mpr.py
dhirschf (2):
Update test_json to account for new metadata
Update .gitignore
gpotter2 (1):
Add __spec__ to DummyMod
hongshaoyang (1):
Minor edits to magics doc
luz.paz (4):
Fix documentation typos
Misc. typo fixes
Whitespace fixes
typos
meeseeksdev[bot] (1):
Merge pull request #11273 from Carreau/re-autoindent
oscar6echo (3):
Add new methods in update_class()
Fix inconsequential typo in test
Add file in whatsnew/pr
prasanth (1):
checking weather virtualenv dir exists or not
stonebig (1):
https for all readthedocs.io links
This is a squash and a rebase of a large number of commits from Min and
I. For simplicity of managing it, history has been reduced to a single
commit, but more historical versions can be found, in particular in
PR #11155, or commit aedb5d6 to be more
exact.
--
I'm expecting some errors in the squash and resolve of conflicts, so will iterate on it.