Skip to content

cloudpickle 0.6.0#788

Merged
ogrisel merged 3 commits intojoblib:masterfrom
ogrisel:cloudpickle-0.6.0
Oct 10, 2018
Merged

cloudpickle 0.6.0#788
ogrisel merged 3 commits intojoblib:masterfrom
ogrisel:cloudpickle-0.6.0

Conversation

@ogrisel
Copy link
Copy Markdown
Contributor

@ogrisel ogrisel commented Oct 8, 2018

Among other fixes, this will remove the annoying deprecation warning about the imp module.

@ogrisel
Copy link
Copy Markdown
Contributor Author

ogrisel commented Oct 8, 2018

This caused the following failure under Python 3.4:

_____________________ test_nested_exception_dispatch[loky] _____________________
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/travis/build/joblib/joblib/joblib/externals/loky/process_executor.py", line 420, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/travis/build/joblib/joblib/joblib/_parallel_backends.py", line 563, in __call__
    return self.func(*args, **kwargs)
  File "/home/travis/build/joblib/joblib/joblib/parallel.py", line 263, in __call__
    for func, args, kwargs in self.items]
  File "/home/travis/build/joblib/joblib/joblib/parallel.py", line 263, in <listcomp>
    for func, args, kwargs in self.items]
  File "/home/travis/build/joblib/joblib/joblib/test/test_parallel.py", line 518, in nested_function_outer
    delayed(nested_function_inner)(j) for j in range(30))
  File "/home/travis/build/joblib/joblib/joblib/parallel.py", line 998, in __call__
    self.retrieve()
  File "/home/travis/build/joblib/joblib/joblib/parallel.py", line 901, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/travis/miniconda/envs/testenv/lib/python3.4/multiprocessing/pool.py", line 608, in get
    raise self._value
  File "/home/travis/miniconda/envs/testenv/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/travis/build/joblib/joblib/joblib/_parallel_backends.py", line 563, in __call__
    return self.func(*args, **kwargs)
  File "/home/travis/build/joblib/joblib/joblib/parallel.py", line 263, in __call__
    for func, args, kwargs in self.items]
  File "/home/travis/build/joblib/joblib/joblib/parallel.py", line 263, in <listcomp>
    for func, args, kwargs in self.items]
  File "/home/travis/build/joblib/joblib/joblib/test/test_parallel.py", line 514, in nested_function_inner
    delayed(exception_raiser)(j) for j in range(30))
  File "/home/travis/build/joblib/joblib/joblib/parallel.py", line 985, in __call__
    if self.dispatch_one_batch(iterator):
  File "/home/travis/build/joblib/joblib/joblib/parallel.py", line 822, in dispatch_one_batch
    self._pickle_cache)
  File "/home/travis/build/joblib/joblib/joblib/parallel.py", line 253, in __init__
    self.items = list(iterator_slice)
KeyError: 'range'

I am not sure if it's deterministic or not. Let me restart the build.

@ogrisel
Copy link
Copy Markdown
Contributor Author

ogrisel commented Oct 8, 2018

It's deterministic so there is a real regression...

@tomMoral
Copy link
Copy Markdown
Contributor

tomMoral commented Oct 8, 2018

It also makes loky fails with errors of the style:

________ TestsProcessPoolForkserverExecutor.test_memory_leak_protection ________
loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/travis/build/tomMoral/loky/loky/process_executor.py", line 422, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/travis/build/tomMoral/loky/tests/_test_process_executor.py", line 818, in _leak_some_memory
    leaked_size = sum(len(buffer) for buffer in os._loky_leak)
  File "/home/travis/build/tomMoral/loky/tests/_test_process_executor.py", line 818, in <genexpr>
    leaked_size = sum(len(buffer) for buffer in os._loky_leak)
KeyError: 'sum'
"""
The above exception was the direct cause of the following exception:
self = <tests.test_process_executor_forkserver.TestsProcessPoolForkserverExecutor object at 0x7fa0dc462860>
    def test_memory_leak_protection(self):
        self.executor.shutdown(wait=True)
    
        executor = self.executor_type(1, context=self.context)
    
        def _leak_some_memory(size=int(1e6), delay=0.001):
            """function that leaks some memory """
            from loky import process_executor
            process_executor._MEMORY_LEAK_CHECK_DELAY = 0.1
            if getattr(os, '_loky_leak', None) is None:
                os._loky_leak = []
    
            os._loky_leak.append(b"\x00" * size)
    
            # Leave enough time for the memory leak detector to kick-in:
            # by default the process does not check its memory usage
            # more than once per second.
            time.sleep(delay)
    
            leaked_size = sum(len(buffer) for buffer in os._loky_leak)
            return os.getpid(), leaked_size
    
        with pytest.warns(UserWarning, match='memory leak'):
            futures = []
            for i in range(300):
                # Total run time should be 3s which is way over the 1s cooldown
                # period between two consecutive memory checks in the worker.
                futures.append(executor.submit(_leak_some_memory))
    
            executor.shutdown(wait=True)
>           results = [f.result() for f in futures]
_leak_some_memory = <function ExecutorTest.test_memory_leak_protection.<locals>._leak_some_memory at 0x7fa0dd5007b8>
executor   = <loky.process_executor.ProcessPoolExecutor object at 0x7fa0dc4624a8>
futures    = [<Future at 0x7fa0dc452d68 state=finished raised KeyError>, <Future at 0x7fa0dc452278 state=finished raised KeyError>,...ture at 0x7fa0dc449cc0 state=finished raised KeyError>, <Future at 0x7fa0dc449f98 state=finished raised KeyError>, ...]
i          = 299
self       = <tests.test_process_executor_forkserver.TestsProcessPoolForkserverExecutor object at 0x7fa0dc462860>
tests/_test_process_executor.py:829: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/_test_process_executor.py:829: in <listcomp>
    results = [f.result() for f in futures]
/opt/python/3.4.6/lib/python3.4/concurrent/futures/_base.py:395: in result
    return self.__get_result()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <Future at 0x7fa0dc452d68 state=finished raised KeyError>
    def __get_result(self):
        if self._exception:
>           raise self._exception
E           KeyError: 'sum'
self       = <Future at 0x7fa0dc452d68 state=finished raised KeyError>
/opt/python/3.4.6/lib/python3.4/concurrent/futures/_base.py:354: KeyError
---------------------------- Captured stderr setup -----------------------------

@tomMoral
Copy link
Copy Markdown
Contributor

tomMoral commented Oct 8, 2018

this is probably due to a change about __builtins__, as they appear to be missing from the child process.

@codecov
Copy link
Copy Markdown

codecov bot commented Oct 10, 2018

Codecov Report

Merging #788 into master will decrease coverage by 0.09%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##           master     #788     +/-   ##
=========================================
- Coverage   95.43%   95.33%   -0.1%     
=========================================
  Files          44       44             
  Lines        6307     6307             
=========================================
- Hits         6019     6013      -6     
- Misses        288      294      +6
Impacted Files Coverage Δ
joblib/pool.py 91.37% <0%> (-3.45%) ⬇️
joblib/_parallel_backends.py 96% <0%> (-1.21%) ⬇️
joblib/test/test_parallel.py 96.91% <0%> (+0.14%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9db62d5...f32ba26. Read the comment docs.

@ogrisel
Copy link
Copy Markdown
Contributor Author

ogrisel commented Oct 10, 2018

@tomMoral I released cloudpickle 0.6.1 with your fix.

@ogrisel ogrisel merged commit 2f252e3 into joblib:master Oct 10, 2018
@ogrisel ogrisel deleted the cloudpickle-0.6.0 branch October 10, 2018 16:36
yarikoptic added a commit to yarikoptic/joblib that referenced this pull request Apr 12, 2019
* tag '0.13.0':
  Release 0.13.0
  Use https (joblib#805)
  MTN bump loky to 2.4.2 (joblib#804)
  DOC: provide some useful variables for Makefile (joblib#794)
  DOC serialization and processes (joblib#803)
  ENH update loky 2.4.0 (joblib#802)
  FIX backward compatibility for nested backend (joblib#789)
  enable python 3.7 (joblib#795)
  memory: add test for call_and_shelve performance (joblib#791)
  FIX nested backend not changed by SequentialBackend (joblib#792)
  cloudpickle 0.6.0 (joblib#788)
  FIX nested backend setting n_jobs=-1 (joblib#785)
  Raises a more explicit exception when a corrupted MemorizedResult is loaded (joblib#768)
  Back to dev mode
yarikoptic added a commit to yarikoptic/joblib that referenced this pull request Apr 12, 2019
* releases:
  Release 0.13.0
  Use https (joblib#805)
  MTN bump loky to 2.4.2 (joblib#804)
  DOC: provide some useful variables for Makefile (joblib#794)
  DOC serialization and processes (joblib#803)
  ENH update loky 2.4.0 (joblib#802)
  FIX backward compatibility for nested backend (joblib#789)
  enable python 3.7 (joblib#795)
  memory: add test for call_and_shelve performance (joblib#791)
  FIX nested backend not changed by SequentialBackend (joblib#792)
  cloudpickle 0.6.0 (joblib#788)
  FIX nested backend setting n_jobs=-1 (joblib#785)
  Raises a more explicit exception when a corrupted MemorizedResult is loaded (joblib#768)
  Back to dev mode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants