Path: csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed1.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.036 X-Spam-Evidence: '*H*': 0.93; '*S*': 0.00; 'third-party': 0.04; 'class,': 0.07; 'exec': 0.09; 'models.': 0.09; 'subject:modules': 0.09; 'cc:addr:python-list': 0.11; 'jan': 0.12; 'binary.': 0.16; 'easier.': 0.16; 'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16; 'json,': 0.16; 'low.': 0.16; 'wraps': 0.16; 'wrote:': 0.18; 'thu,': 0.19; 'code,': 0.22; 'cc:addr:python.org': 0.22; 'load': 0.23; 'fairly': 0.24; 'cc:2**0': 0.24; 'changes,': 0.26; 'equivalent': 0.26; 'second': 0.26; 'header:In-Reply-To:1': 0.27; 'am,': 0.29; 'message-id:@mail.gmail.com': 0.30; 'code': 0.31; 'concern.': 0.31; 'libraries': 0.31; 'object.': 0.31; 'pickle': 0.31; 'class': 0.32; 'maybe': 0.34; "can't": 0.35; 'case,': 0.35; 'one,': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'doing': 0.36; 'turn': 0.37; 'easily': 0.37; 'bad': 0.39; 'use.': 0.39; 'even': 0.60; 'then,': 0.60; 'issues,': 0.61; 'john': 0.61; 'simple': 0.61; 'first': 0.61; 'personal': 0.63; 'more': 0.64; 'p.s.': 0.66; 'minutes': 0.67; '2015': 0.84; 'blob': 0.84; 'persistent': 0.84; 'afford': 0.91; 'use"': 0.91; 'to:none': 0.92; 'besides,': 0.93 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:cc :content-type:content-transfer-encoding; bh=6EKnRxj8m6itBNKFCu79jWXJUVXqlOFakH6KqD0PexE=; b=r0f0dVYNBqxJbMFoJOaalXf//a9r03Z0BWs9xMkkN3RZ3saypaTigNOLXfOwTEws4B 9ZBCYEr31CyDBCZ8O5M9Cysvu0uylKZh+pYl7k1o8KfxgRE5hsyx+YTIPQkSyUI2QDu5 P0LXgEuH+/GD/Htgv0OZBTgW1TW/W+r7nNxinyXwbTgSEmsW7GzkVrUXxXxO2ETGn5Xb zY/56am6sw10BcdhVH02Y7r3y52okxQVTsObkNR9jPG8Fd2swx8j/6VvWXrXi0ZmIRIG LGSVzS//yHVo9bwGIUNwZ+jw/uSZEJlVo7BUXqumXxI8/9RGN9Bl9XOfr6RHfu0O4q2Y 7/GQ== MIME-Version: 1.0 X-Received: by 10.107.134.39 with SMTP id i39mr6218127iod.53.1420679187771; Wed, 07 Jan 2015 17:06:27 -0800 (PST) In-Reply-To: References: Date: Thu, 8 Jan 2015 12:06:27 +1100 Subject: Re: pickle, modules, and ImportErrors From: Chris Angelico Cc: "python-list@python.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 36 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1420679196 news.xs4all.nl 2838 [2001:888:2000:d::a6]:41706 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:83309 On Thu, Jan 8, 2015 at 11:23 AM, John Ladasky wrote: >> P.S. don't use pickle, it is a security vulnerability equivalent in >> severity to using exec in your code, and an unversioned opaque >> schemaless blob that is very difficult to work with when circumstances >> change. > > For all of its shortcomings, I can't live without pickle. In this case, = I am doing data mining. My TrainingSession class commandeers seven CPU cor= es via Multiprocessing.Pool. Still, even my "toy" TrainingSessions take se= veral minutes to run. I can't afford to re-run TrainingSession every time = I need my models. I need a persistent object. > > Besides, the opportunity for mischief is low. My code is for my own pers= onal use. And I trust the third-party libraries that I am using. My SVRMo= del object wraps the NuSVR object from scikit-learn, which in turn wraps th= e libsvm binary. There are several issues, not all of which are easily dodged. Devin cited t= wo: * Security: it's fundamentally equivalent to using 'exec' * Unversioned: it's hard to make updates to your code and then load old dat= a "For your own personal use" dodges the first one, but makes the second one even more of a concern. You can get much better persistence using a textual format like JSON, and adding in a simple 'version' member can make it even easier. Then, when you make changes, you can cope with old data fairly readily. Pickle is still there if you want it, but you do have to be aware of its limitations. If you edit the TrainingSession class, you may well have to rerun the training... but maybe that's not a bad thing. ChrisA