Session: improvements to documentation + handle unpickleable objects#527
Session: improvements to documentation + handle unpickleable objects#527leogama wants to merge 33 commits intouqfoundation:masterfrom
Conversation
3de322f to
2fdd31d
Compare
|
@mmckerns: this docs page affirms that you can push commits directly to the PR branch on my fork's repo. Have you ever tried this procedure? |
Yes. I'm fine with that approach, or fork-PR'ing your fork. Either way. |
9a24624 to
6b55755
Compare
|
I may have successfully solved the problem with unpickleable objects. Added a I've overridden the [Edit: changed this in last commit] Note: the framing trick probably has unhandled corner cases. It may be necessary to disable framing completely (by downgrading the protocol to a maximum of 3 or something else) when using this option. |
b52dce7 to
aac47b5
Compare
I could see this definitely being the case, as the framing used in higher protocols is a good bit more complex. I think the primary reason people will be using this functionality is to "save a session" -- hence the emphasis here should be on being able to save the session without failing... with pickle size a secondary concern (and pickling speed a distant third). |
I'd probably prefer: |
This isn't really about complexity. The problem is that the current frame of an object being pickled may be committed and flushed to file before an unpickleable child object raises an exception and can't even be saved by reference, forcing its siblings that were pickled previously with success to be discarded and one of its parents to be saved by reference. |
I've changed the name, but didn't understand the distinction you made. The parameter signature is identical to the those of the other |
00fa855 to
699f30a
Compare
|
I'm closing this, as it's been merged with #475. |
Other relevant changes in this commit: - Complement and improve the documentation of _dill.py and session.py - Rename the submodule 'logger' back to 'logging'. - Now, the submodule 'logging' is meant to be used also as a proxy to the StdLib module of the same name. - The level for pickling tracing is now 'logging.TRACE', a custom level between INFO and DEBUG - New: if logging level is set to INFO or lower, the variables saved by reference, either by 'refimported' or by 'refonfail', are listed. - New internal functions: _is_imported_module(), _is_stdlib_module() and _module_package() - New private submodule '_utils': new helper _open() context manager for opening streams - More tests added to test_session.py - Etc.
Edit: merged with #475.
Additional changes to session/module saving and restoring based on discussions at #525.
_open()function that returns a file/stream context manageris_pickled_module()function to test whether a pickle file was created bydump_module()and if it's module is importable or not. (It currently returnsTruefor modules saved withdill.dump(module, file).)refonfailoption fordump_module()to save objects that fail to pickle by reference, rolling back the pickle stream and saving one of the parent objects by reference if necessary.To-do
Documentation:
dill.dump(module, file)anddill.dump_module(file, module), and the assumptions and use cases for each alternative — probably in the package's main documentation, or in a separatesession.pymodule's docstringmoduleargument toload_module()load_module()is_pickled_module(filename)(what about its name and signature?)Code:
is_pickled_module()returnFalsefor modules saved by name (and for modules saved by value inside other objects)(likely in a separate PR)@mmckerns ☝🏼 You may start drafting any of these if you feel inspired. Use the "suggested changes" in the review page or modify the code directly (adding new commits) if you are able to. I set the "allow editing by maintainers" option, although I don't know what it means...