Upgrade to cloudpickle 0.9.0.dev0

[pcmoritz]

This includes support for enum classes, see cloudpipe/cloudpickle#246.

This is to test things out, we should wait merging this until the final version of cloudpipe/cloudpickle#246 is merged.

Related issues: #3996 #3917

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12034/
Test FAILed.

[ogrisel]

There are some tests failing at test_actors_and_tasks_with_gpus_version_two which does not seem to be related to cloudpickle.

There are other Python 2.7 tests that fail because the enum module does not exist. This is weird because the Python 2.7 tests pass on my tox -e py27 env for cloudpickle. Maybe older Python 2.7 versions did not have enums?

Anyway, it should be quite easy to do a conditional import for this module in cloudpickle.

[ogrisel]

I've just pushed a new commit to cloudpipe/cloudpickle#246 to make the dependency on enum34 optional (under Python < 3.4).

[ogrisel]

The failure in https://travis-ci.com/ray-project/ray/jobs/178328001 is suspicious (the operation lasts 19s instead of less than 10s). It could be caused by a performance regression in cloudpickle but this test does not seem to involve any dynamic class definition so I am not sure how cloudpipe/cloudpickle#246 could cause this.

[ogrisel]

The failure in https://travis-ci.com/ray-project/ray/jobs/178328007 is weird: no output for 10min (in python/ray/tune/test/trial_runner_test.py::TrainableFunctionApiTest::testBadParams5 ) looks like a deadlock. Has this been observed before in ray or it this the first time?

Using the pytest-timeout plugin would be useful to dump the traceback of all the python threads in case of deadlock.

[ogrisel]

There is also this ArrowIOError: Encountered unexpected EOF in test_multithreading:

https://travis-ci.com/ray-project/ray/jobs/178327998

[pcmoritz]

@ogrisel Thanks for all the help, I just updated to the latest version of the code! The tests look good except the testBadParams5 one which I haven't seen fail. I'll look some more about the cloudpickle changes and the code that is involved here and see if I can spot anything suspicious.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12113/
Test FAILed.

[ogrisel]

Alright thanks @pcmoritz. Let me know if you have any questions on the new semantics for dynamic class defintions introduced in the cloudpickle PR.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12183/
Test FAILed.

[pcmoritz]

Jenkins retest this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12184/
Test FAILed.

[pcmoritz]

Jenkins retest this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12191/
Test FAILed.

[pcmoritz]

@ogrisel The PR looks good to me!

[ogrisel]

@ogrisel The PR looks good to me!

That's excellent news. You mean that the remaining CI failures on Ray are unrelated? If so, please put a comment on the cloudpickle PR :)

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/14/
Test FAILed.

[AmplabJenkins]

Can one of the admins verify this patch?

[simon-mo]

We already updated cloudpickle. Thanks for the PR! Closing