Skip to content

can't pickle dict_keys objects #380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
brl0 opened this issue Jun 13, 2020 · 2 comments · Fixed by #384
Closed

can't pickle dict_keys objects #380

brl0 opened this issue Jun 13, 2020 · 2 comments · Fixed by #384

Comments

@brl0
Copy link
Contributor

brl0 commented Jun 13, 2020

Using Dask.distributed, which uses cloudpickle for serialization, I encounter an error when passing a dict_keys object as a parameter to a function. I reported the issue there, where it was suggested that I also report an issue here for consideration.

As a test, I attempted to use cloudpickle directly on this type of object and received the following error:
TypeError: can't pickle dict_keys objects

Would it be possible to add the ability to pickle dict_keys object?

@ogrisel
Copy link
Contributor

ogrisel commented Jun 16, 2020

I wonder why this is not the picklable by default in Python:

>>> import pickle                                                                                  
>>> pickle.dumps({}.keys())                                                                        
Traceback (most recent call last):
  File "<ipython-input-6-6d75fd452bb1>", line 1, in <module>
    pickle.dumps({}.keys())
TypeError: cannot pickle 'dict_keys' object

Maybe I miss something. Maybe because unsuspecting users would risk sending the dict values as part of the pickle without realizing it?

I suppose we could write a reducer in cloudpickle that would ship the whole dict (which should be picklable) and reconstruct the dict_keys view on the other side. Same for dict values and items.

Would you be interested in contributing a PR?

@brl0
Copy link
Contributor Author

brl0 commented Jun 16, 2020

Thanks for your response.

I would be interested in attempting to contribute, but after taking a look at the codebase, I am honestly not sure where/how to start. I am open to guidance if you are willing to point me in the right direction.

My initial thought is that it would be best not to ship the whole dict, because it could be large, complex, or contain sensitive information that is not needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants