cloudpickle breaks dill deserialization across servers.

[wmarshall484]

Following up on this issue on Stackoverflow:

http://stackoverflow.com/questions/42960637/python-3-5-dill-pickling-unpickling-on-different-servers-keyerror-classtype/43006034#43006034

In a nutshell, with Python 3.5:

Server A imports cloudpickle this causes types.ClassType to become defined.

>>> import types
>>> dir(types)
  ['BuiltinFunctionType',
   'BuiltinMethodType',
   'ClassType',
   'CodeType',
   ...
  ]

Server B does not import cloudpickle, so types.ClassType is left undefined.

>>> import types
>>> dir(types)
  ['BuiltinFunctionType',
   'BuiltinMethodType',
   'CodeType',
   ...
  ]

Objects which are serialized in server A also seem to serialize a reference to ClassType. Then, when they are deserialized on server B, we encounter the following error:

Traceback (most recent call last):
 File "/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py", line 40, in <module>
   a = dill.loads(base64.b64decode(a.encode()))
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 277, in loads
   return load(file)
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 266, in load
   obj = pik.load()
 File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 524, in _load_type
   return _reverse_typemap[name]
KeyError: 'ClassType'

I've found a workaround, which you can see on Stackoverflow.

Here's my question: types.ClassType was removed in 3.5, yet cloudpickle re-adds it. Is this strictly necessary? It seems to be having side effects.

[tvalentyn]

Bumping this up as users report being hit by these side-effects, for example:

• TFX with Dataflow Python Version Error tensorflow/tfx#1216
• https://lists.apache.org/thread.html/r97bf207cea60ceb3988f26c4dbe8dcb06b05fa10e0a94d74d15a5e1a%40%3Cuser.beam.apache.org%3E

Can cloudpickle maintainers comment on whether it is possible to release a newer version of cloudpickle library that does not create a side-effect after importing cloudpickle?

I think this problem makes it difficult for users to work with libraries that depend on cloudpickle inside interactive environments, when they also interact with other libraries that depend on different picklers in the same session.

[zhitaoli]

+1 this is also affect TFX users.

[tvalentyn]

Sent: #337