Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #83309

Re: pickle, modules, and ImportErrors

Show key headers only | View raw

On Thu, Jan 8, 2015 at 11:23 AM, John Ladasky
<john_ladasky@sbcglobal.net> wrote:
>> P.S. don't use pickle, it is a security vulnerability equivalent in
>> severity to using exec in your code, and an unversioned opaque
>> schemaless blob that is very difficult to work with when circumstances
>> change.
>
> For all of its shortcomings, I can't live without pickle.  In this case, I am doing data mining.  My TrainingSession class commandeers seven CPU cores via Multiprocessing.Pool.  Still, even my "toy" TrainingSessions take several minutes to run.  I can't afford to re-run TrainingSession every time I need my models.  I need a persistent object.
>
> Besides, the opportunity for mischief is low.  My code is for my own personal use.  And I trust the third-party libraries that I am using.  My SVRModel object wraps the NuSVR object from scikit-learn, which in turn wraps the libsvm binary.

There are several issues, not all of which are easily dodged. Devin cited two:

* Security: it's fundamentally equivalent to using 'exec'
* Unversioned: it's hard to make updates to your code and then load old data

"For your own personal use" dodges the first one, but makes the second
one even more of a concern. You can get much better persistence using
a textual format like JSON, and adding in a simple 'version' member
can make it even easier. Then, when you make changes, you can cope
with old data fairly readily.

Pickle is still there if you want it, but you do have to be aware of
its limitations. If you edit the TrainingSession class, you may well
have to rerun the training... but maybe that's not a bad thing.

ChrisA

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

pickle, modules, and ImportErrors John Ladasky <john_ladasky@sbcglobal.net> - 2015-01-07 12:12 -0800
  Re: pickle, modules, and ImportErrors Devin Jeanpierre <jeanpierreda@gmail.com> - 2015-01-07 14:55 -0600
    Re: pickle, modules, and ImportErrors John Ladasky <john_ladasky@sbcglobal.net> - 2015-01-07 16:23 -0800
      Re: pickle, modules, and ImportErrors Chris Angelico <rosuav@gmail.com> - 2015-01-08 12:06 +1100
  Re: pickle, modules, and ImportErrors Ian Kelly <ian.g.kelly@gmail.com> - 2015-01-07 14:01 -0700

csiph-web