Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #70460 > unrolled thread

Recommended exception for objects that can't be pickled

Back to article view | Back to comp.lang.python

Contents

  Recommended exception for objects that can't be pickled Stefan Schwarzer <sschwarzer@sschwarzer.net> - 2014-04-21 15:23 +0200
    Re: Recommended exception for objects that can't be pickled Terry Reedy <tjreedy@udel.edu> - 2014-04-21 17:08 -0400

#70460 — Recommended exception for objects that can't be pickled

Hi,

Recently, I got a request [1] to support pickling of
`FTPHost` instances in my `ftplib` library.

I explained in the ticket why I think it's a bad idea and
now want to make explicit that `FTPHost` objects can't be
pickled. The usual way to do this seems to be defining a
`__getstate__` method and raise an exception there.

Now the question is "which exception?" and my research left
me a bit confused. I didn't find a recommendation for this
on the web, not even in the Python documentation for the
`pickle` module [2]. The only hint is that the documentation
states:

"""
  The pickle module defines three exceptions:

  exception pickle.PickleError

      Common base class for the other pickling exceptions. It
      inherits Exception.

  exception pickle.PicklingError

      Error raised when an unpicklable object is encountered
      by Pickler. It inherits PickleError.

      Refer to What can be pickled and unpickled? to learn
      what kinds of objects can be pickled.

  exception pickle.UnpicklingError

      Error raised when there is a problem unpickling an
      object, such as a data corruption or a security
      violation. It inherits PickleError.

      Note that other exceptions may also be raised during
      unpickling, including (but not necessarily limited to)
      AttributeError, EOFError, ImportError, and IndexError.
"""

This sounds like unpicklable objects should raise a
`PicklingError`. Indeed, if I do this, `pickle.dumps`
gives me (my own) `PicklingError`:

  Python 3.3.2 (default, Nov  8 2013, 13:38:57)
  [GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import pickle
  >>> class X:
  ...   def __getstate__(self):
  ...     raise pickle.PicklingError("can't pickle X objects")
  ...
  >>> x = X()
  >>> pickle.dumps(x)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 3, in __getstate__
  _pickle.PicklingError: can't pickle X objects

What now confuses me is that most, if not all, objects from
the standard library that aren't picklable raise a
`TypeError` when I try to pickle them:

  >>> fobj = open("/etc/passwd")
  >>> pickle.dumps(fobj)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: cannot serialize '_io.TextIOWrapper' object

  >>> import socket
  >>> s = socket.socket()
  >>> pickle.dumps(s)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib64/python3.3/socket.py", line 116, in __getstate__
      raise TypeError("Cannot serialize socket object")
  TypeError: Cannot serialize socket object

So the documentation for the `pickle` module (to me) implies
I should raise a `PicklingError` while the standard library
usually seems to use a `TypeError`. When I grep through the
library files for `PicklingError`, I get very few hits, most
of them in `pickle.py`:

  $ find /usr/lib64/python3.3 -name "*.py" -exec grep -H PicklingError {} \;
  /usr/lib64/python3.3/site-packages/numpy/numarray/session.py:        except (pickle.PicklingError, TypeError, SystemError):
  /usr/lib64/python3.3/pickle.py:__all__ = ["PickleError", "PicklingError", "UnpicklingError", "Pickler",
  /usr/lib64/python3.3/pickle.py:class PicklingError(PickleError):
  /usr/lib64/python3.3/pickle.py:            raise PicklingError("Pickler.__init__() was not called by "
  /usr/lib64/python3.3/pickle.py:                    raise PicklingError("Can't pickle %r object: %r" %
  /usr/lib64/python3.3/pickle.py:            raise PicklingError("%s must return string or tuple" % reduce)
  /usr/lib64/python3.3/pickle.py:            raise PicklingError("Tuple returned by %s must have "
  /usr/lib64/python3.3/pickle.py:            raise PicklingError("args from save_reduce() should be a tuple")
  /usr/lib64/python3.3/pickle.py:            raise PicklingError("func from save_reduce() should be callable")
  /usr/lib64/python3.3/pickle.py:                raise PicklingError(
  /usr/lib64/python3.3/pickle.py:                raise PicklingError(
  /usr/lib64/python3.3/pickle.py:            raise PicklingError(
  /usr/lib64/python3.3/pickle.py:                raise PicklingError(
  /usr/lib64/python3.3/pickle.py:                raise PicklingError(
  /usr/lib64/python3.3/idlelib/rpc.py:        except pickle.PicklingError:

Which exception would you raise for an object that can't be
pickled and why?

[1] http://ftputil.sschwarzer.net/trac/ticket/75
[2] https://docs.python.org/3.4/library/pickle.html

Best regards,
Stefan

[toc] | [next] | [standalone]

#70475

On 4/21/2014 9:23 AM, Stefan Schwarzer wrote:
> Hi,
>
> Recently, I got a request [1] to support pickling of
> `FTPHost` instances in my `ftplib` library.
>
> I explained in the ticket why I think it's a bad idea and
> now want to make explicit that `FTPHost` objects can't be
> pickled. The usual way to do this seems to be defining a
> `__getstate__` method and raise an exception there.
>
> Now the question is "which exception?" and my research left
> me a bit confused. I didn't find a recommendation for this
> on the web, not even in the Python documentation for the
> `pickle` module [2]. The only hint is that the documentation
> states:
>
> """
>    The pickle module defines three exceptions:
>
>    exception pickle.PickleError
>
>        Common base class for the other pickling exceptions. It
>        inherits Exception.
>
>    exception pickle.PicklingError
>
>        Error raised when an unpicklable object is encountered
>        by Pickler. It inherits PickleError.

I am going to read this as 'unpicklable as determined by Pickler', 
possibly due to a bug in the objects methods.

>        Refer to What can be pickled and unpickled? to learn
>        what kinds of objects can be pickled.
>
>    exception pickle.UnpicklingError
>
>        Error raised when there is a problem unpickling an
>        object, such as a data corruption or a security
>        violation. It inherits PickleError.
>
>        Note that other exceptions may also be raised during
>        unpickling, including (but not necessarily limited to)
>        AttributeError, EOFError, ImportError, and IndexError.
> """
>
> This sounds like unpicklable objects should raise a
> `PicklingError`. Indeed, if I do this, `pickle.dumps`
> gives me (my own) `PicklingError`:
>
>    Python 3.3.2 (default, Nov  8 2013, 13:38:57)
>    [GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux
>    Type "help", "copyright", "credits" or "license" for more information.
>    >>> import pickle
>    >>> class X:
>    ...   def __getstate__(self):
>    ...     raise pickle.PicklingError("can't pickle X objects")
>    ...
>    >>> x = X()
>    >>> pickle.dumps(x)
>    Traceback (most recent call last):
>      File "<stdin>", line 1, in <module>
>      File "<stdin>", line 3, in __getstate__
>    _pickle.PicklingError: can't pickle X objects
>
> What now confuses me is that most, if not all, objects from
> the standard library that aren't picklable raise a
> `TypeError` when I try to pickle them:
>
>    >>> fobj = open("/etc/passwd")
>    >>> pickle.dumps(fobj)
>    Traceback (most recent call last):
>      File "<stdin>", line 1, in <module>
>    TypeError: cannot serialize '_io.TextIOWrapper' object
>
>    >>> import socket
>    >>> s = socket.socket()
>    >>> pickle.dumps(s)
>    Traceback (most recent call last):
>      File "<stdin>", line 1, in <module>
>      File "/usr/lib64/python3.3/socket.py", line 116, in __getstate__
>        raise TypeError("Cannot serialize socket object")
>    TypeError: Cannot serialize socket object

I would just copy the socket.__getstate__, with a revised message.

> So the documentation for the `pickle` module (to me) implies
> I should raise a `PicklingError` while the standard library
> usually seems to use a `TypeError`. When I grep through the
> library files for `PicklingError`, I get very few hits, most
> of them in `pickle.py`:

This suggests that the exception is intended for use by Pickler, and not 
by user code. The reason is pretty subtle and not explained in the doc. 
See below.

>    $ find /usr/lib64/python3.3 -name "*.py" -exec grep -H PicklingError {} \;
>    /usr/lib64/python3.3/site-packages/numpy/numarray/session.py:        except (pickle.PicklingError, TypeError, SystemError):
>    /usr/lib64/python3.3/pickle.py:__all__ = ["PickleError", "PicklingError", "UnpicklingError", "Pickler",
>    /usr/lib64/python3.3/pickle.py:class PicklingError(PickleError):
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError("Pickler.__init__() was not called by "
>    /usr/lib64/python3.3/pickle.py:                    raise PicklingError("Can't pickle %r object: %r" %
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError("%s must return string or tuple" % reduce)
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError("Tuple returned by %s must have "
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError("args from save_reduce() should be a tuple")
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError("func from save_reduce() should be callable")

Most of the above look like bugs in code intended to work with Pickle. 
In other words, if you user gets a TypeError, the user has made a 
mistake by trying to pickle your object. If you tried to make your 
object picklable, and users got PicklingError, that would indicate a bug 
in your class code, not the users' use of your class.

>    /usr/lib64/python3.3/pickle.py:                raise PicklingError(
>    /usr/lib64/python3.3/pickle.py:                raise PicklingError(
>    /usr/lib64/python3.3/pickle.py:            raise PicklingError(
>    /usr/lib64/python3.3/pickle.py:                raise PicklingError(
>    /usr/lib64/python3.3/pickle.py:                raise PicklingError(
>    /usr/lib64/python3.3/idlelib/rpc.py:        except pickle.PicklingError:
>
> Which exception would you raise for an object that can't be
> pickled and why?

TypeError, as explained above.

> [1] http://ftputil.sschwarzer.net/trac/ticket/75
> [2] https://docs.python.org/3.4/library/pickle.html

-- 
Terry Jan Reedy

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web