Skip to content

Conversation

@robertnishihara
Copy link
Collaborator

@robertnishihara robertnishihara commented Sep 21, 2018

This brings in a number of recent cloudpickle fixes and may address #2685. This copies cloudpickle.py directly from cloudpipe/cloudpickle@06b18af.

@robertnishihara
Copy link
Collaborator Author

cc @mitar @suquark

@robertnishihara robertnishihara changed the title Update cloudpickle to version 0.5.6. Update cloudpickle to version 0.5.6 (a couple commits after actually). Sep 21, 2018
@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8312/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8311/
Test FAILed.

@robertnishihara
Copy link
Collaborator Author

jenkins, retest this please

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8314/
Test FAILed.

@suquark
Copy link
Member

suquark commented Sep 22, 2018

It's strange that test/runtest.py::GlobalStateAPI::testFlushAPI often fails in various tests. I have seen this many times.

@robertnishihara
Copy link
Collaborator Author

jenkins, retest this please

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8317/
Test FAILed.

@robertnishihara
Copy link
Collaborator Author

jenkins, retest this please

@robertnishihara
Copy link
Collaborator Author

The error appears to be

Runner Thread raised error.
Traceback (most recent call last):
  File "/ray/python/ray/tune/function_runner.py", line 76, in run
    self._entrypoint(*self._entrypoint_args)
  File "/ray/python/ray/tune/examples/tune_mnist_keras.py", line 178, in <lambda>
    lambda cfg, rprtr: train_mnist(args, cfg, rprtr))
  File "/ray/python/ray/tune/examples/tune_mnist_keras.py", line 40, in train_mnist
    intra_op_parallelism_threads=args.threads,
AttributeError: 'Namespace' object has no attribute 'threads'

in docker run -e RAY_USE_XRAY=1 --rm --shm-size=10G --memory=10G 11071230bfdb184c984566048cec671cf536cfbd087806d50f8c4d92ebfdc1a1 python /ray/python/ray/tune/examples/tune_mnist_keras.py --smoke-test

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8324/
Test FAILed.

@richardliaw
Copy link
Contributor

odd; I guess somehow this causes cloudpickle to not capture the scope completely correctly?

@mitar
Copy link
Member

mitar commented Sep 27, 2018

BTW, I am guessing that eventually having also this fix in might saves us some trouble. We are slowly porting code to use typing and it seems with Python 3.6 there could be troubles. So, maybe Ray could update cloudpickle to version which is includes that PR. (But we have not yet had been blocked by this, but we encountered in the past and found some workarounds.)

@mitar
Copy link
Member

mitar commented Oct 14, 2018

@suquark Could you check why tests here are failing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants