Unable to deploy Python closure: hangs indefinitely on cloudpickle error

[TomNash]

Following the steps for XGBoost deployment, I run into an issue with the command

python_deployer.deploy_python_closure(cl, name='xgboost-model', version=1,
     input_type="integers", func=predict, pkgs_to_install=['xgboost'])

The same behavior occurs with the quickstart guide, running

python_deployer.deploy_python_closure(clipper_conn, name="sum-model", version=1, input_type="doubles", func=feature_sum)

It seems this while loop in docker_container_manager.py will loop infinitely with no error if the container creation fails.

> docker inspect 3dbc59e5619d
[
    {
        "Id": "3dbc59e5619d5f0445cf4b85c3892aad7987ce7bb4a7946a82c89e3a2f440bbf",
        "Created": "2018-09-11T20:31:56.5002835Z",
        "Path": "/container/container_entry.sh",
        "Args": [
            "py-closure-container",
            "/container/python_closure_container.py"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
...

I checked the log of the container to see this

> docker container logs 3dbc59e5619d
Starting Python Closure container
Connecting to Clipper with default port: 7000
Traceback (most recent call last):
  File "/container/python_closure_container.py", line 56, in <module>
    rpc_service.get_input_type())
  File "/container/python_closure_container.py", line 28, in __init__
    self.predict_func = load_predict_func(predict_path)
  File "/container/python_closure_container.py", line 17, in load_predict_func
    return cloudpickle.load(serialized_func_file)
  File "/usr/local/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 1060, in _make_skel_func
    base_globals['__builtins__'] = __builtins__
TypeError: 'str' object does not support item assignment
Encountered error not related to missing packages. Please refer to the container log to diagnose.

Any advice on how to proceed?

[simon-mo]

The quick fix is to pip install cloudpickle==0.5.3 the reason is that your local cloudpickle version is higher than the one used in docker container, making them incompatible.