Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.mixmin.net!rt.uk.eu.org!newsfeed.xs4all.nl!newsfeed1a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
To: python-list@python.org
From: Terry Reedy <tjreedy@udel.edu>
Subject: Re: Recommended exception for objects that can't be pickled
Date: Mon, 21 Apr 2014 17:08:17 -0400
References: <53551BBA.8000205@sschwarzer.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0
In-Reply-To: <53551BBA.8000205@sschwarzer.net>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.9418.1398114540.18130.python-list@python.org>
Lines: 132
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:70475

On 4/21/2014 9:23 AM, Stefan Schwarzer wrote:
> Hi,
>
> Recently, I got a request [1] to support pickling of
> `FTPHost` instances in my `ftplib` library.
>
> I explained in the ticket why I think it's a bad idea and
> now want to make explicit that `FTPHost` objects can't be
> pickled. The usual way to do this seems to be defining a
> `__getstate__` method and raise an exception there.
>
> Now the question is "which exception?" and my research left
> me a bit confused. I didn't find a recommendation for this
> on the web, not even in the Python documentation for the
> `pickle` module [2]. The only hint is that the documentation
> states:
>
> """
>    The pickle module defines three exceptions:
>
>    exception pickle.PickleError
>
>        Common base class for the other pickling exceptions. It
>        inherits Exception.
>
>    exception pickle.PicklingError
>
>        Error raised when an unpicklable object is encountered
>        by Pickler. It inherits PickleError.

I am going to read this as 'unpicklable as determined by Pickler', 
possibly due to a bug in the objects methods.

>        Refer to What can be pickled and unpickled? to learn
>        what kinds of objects can be pickled.
>
>    exception pickle.UnpicklingError
>
>        Error raised when there is a problem unpickling an
>        object, such as a data corruption or a security
>        violation. It inherits PickleError.
>
>        Note that other exceptions may also be raised during
>        unpickling, including (but not necessarily limited to)
>        AttributeError, EOFError, ImportError, and IndexError.
> """
>
> This sounds like unpicklable objects should raise a
> `PicklingError`. Indeed, if I do this, `pickle.dumps`
> gives me (my own) `PicklingError`:
>
>    Python 3.3.2 (default, Nov  8 2013, 13:38:57)
>    [GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux
>    Type "help", "copyright", "credits" or "license" for more information.
>    >>> import pickle
>    >>> class X:
>    ...   def __getstate__(self):
>    ...     raise pickle.PicklingError("can't pickle X objects")
>    ...
>    >>> x = X()
>    >>> pickle.dumps(x)
>    Traceback (most recent call last):
>      File "<stdin>", line 1, in <module>
>      File "<stdin>", line 3, in __getstate__
>    _pickle.PicklingError: can't pickle X objects
>
> What now confuses me is that most, if not all, objects from
> the standard library that aren't picklable raise a
> `TypeError` when I try to pickle them:
>
>    >>> fobj = open("/etc/passwd")
>    >>> pickle.dumps(fobj)
>    Traceback (most recent call last):
>      File "<stdin>", line 1, in <module>
>    TypeError: cannot serialize '_io.TextIOWrapper' object
>
>    >>> import socket
>    >>> s = socket.socket()
>    >>> pickle.dumps(s)
>    Traceback (most recent call last):
>      File "<stdin>", line 1, in <module>
>      File "/usr/lib64/python3.3/socket.py", line 116, in __getstate__
>        raise TypeError("Cannot serialize socket object")
>    TypeError: Cannot serialize socket object

I would just copy the socket.__getstate__, with a revised message.

> So the documentation for the `pickle` module (to me) implies
> I should raise a `PicklingError` while the standard library
> usually seems to use a `TypeError`. When I grep through the
> library files for `PicklingError`, I get very few hits, most
> of them in `pickle.py`:

This suggests that the exception is intended for use by Pickler, and not 
by user code. The reason is pretty subtle and not explained in the doc. 
See below.

>    $ find /usr/lib64/python3.3 -name "*.py" -exec grep -H PicklingError {} \;
>    /usr/lib64/python3.3/site-packages/numpy/numarray/session.py:        except (pickle.PicklingError, TypeError, SystemError):
>    /usr/lib64/python3.3/pickle.py:__all__ = ["PickleError", "PicklingError", "UnpicklingError", "Pickler",
>    /usr/lib64/python3.3/pickle.py:class PicklingError(PickleError):
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError("Pickler.__init__() was not called by "
>    /usr/lib64/python3.3/pickle.py:                    raise PicklingError("Can't pickle %r object: %r" %
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError("%s must return string or tuple" % reduce)
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError("Tuple returned by %s must have "
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError("args from save_reduce() should be a tuple")
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError("func from save_reduce() should be callable")

Most of the above look like bugs in code intended to work with Pickle. 
In other words, if you user gets a TypeError, the user has made a 
mistake by trying to pickle your object. If you tried to make your 
object picklable, and users got PicklingError, that would indicate a bug 
in your class code, not the users' use of your class.

>    /usr/lib64/python3.3/pickle.py:                raise PicklingError(
>    /usr/lib64/python3.3/pickle.py:                raise PicklingError(
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError(
>    /usr/lib64/python3.3/pickle.py:                raise PicklingError(
>    /usr/lib64/python3.3/pickle.py:                raise PicklingError(
>    /usr/lib64/python3.3/idlelib/rpc.py:        except pickle.PicklingError:
>
> Which exception would you raise for an object that can't be
> pickled and why?

TypeError, as explained above.

> [1] http://ftputil.sschwarzer.net/trac/ticket/75
> [2] https://docs.python.org/3.4/library/pickle.html

-- 
Terry Jan Reedy