Path: csiph.com!v102.xanadu-bbs.net!xanadu-bbs.net!news.mixmin.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail Return-Path: X-Original-To: python-list@python.org Delivered-To: python-list@mail.python.org X-Spam-Status: OK 0.001 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'subject:not': 0.03; 'static': 0.04; 'subject:Python': 0.06; 'mentioned,': 0.07; 'subject: -- ': 0.07; 'currently,': 0.09; 'filename': 0.09; 'pgp': 0.09; 'skip:# 30': 0.09; 'subject:files': 0.09; 'works.': 0.09; 'cc:addr:python-list': 0.11; 'python': 0.11; 'suggest': 0.14; 'posted': 0.15; '"with"': 0.16; 'descriptors': 0.16; 'processes.': 0.16; 'skip:q 70': 0.16; '\xa0would': 0.16; 'prevent': 0.16; 'thursday,': 0.16; 'wrote:': 0.18; 'thu,': 0.19; '>>>': 0.22; 'example': 0.22; 'cc:addr:python.org': 0.22; 'convenient': 0.24; 'sorry,': 0.24; 'text,': 0.24; 'fine': 0.24; 'looks': 0.24; 'cc:2**0': 0.24; 'cc:no real name:2**0': 0.24; 'this:': 0.26; 'van': 0.27; 'header:In-Reply-To:1': 0.27; 'function': 0.29; 'thus': 0.29; "doesn't": 0.30; 'statement': 0.30; 'message- id:@mail.gmail.com': 0.30; 'work.': 0.31; 'code': 0.31; "skip:' 10": 0.31; 'you?': 0.31; '3.2': 0.31; '>>>>': 0.31; 'large.': 0.31; 'writes:': 0.31; 'file': 0.32; 'class': 0.32; 'probably': 0.32; 'stuff': 0.32; 'quite': 0.32; 'received:74.125.82': 0.34; 'problem': 0.35; 'something': 0.35; 'but': 0.35; 'received:google.com': 0.35; 'there': 0.35; 'method': 0.36; 'should': 0.36; 'list': 0.37; 'skip:& 10': 0.38; 'skip:[ 10': 0.38; 'issue': 0.38; 'pm,': 0.38; 'received:74.125': 0.39; 'unable': 0.39; 'ensure': 0.60; 'remove': 0.60; 'most': 0.60; 'providing': 0.61; 'here:': 0.62; 'complete': 0.62; 'more': 0.64; 'skip:f 50': 0.65; 'within': 0.65; 'reply': 0.66; 'close': 0.67; 'obvious': 0.74; 'batchelder': 0.84; 'isaac': 0.84; 'subject:skip:M 10': 0.84; '2013': 0.98 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=ZpHp8EWfsj9qFR76OxeZ9vxjDAYG61d+ZRBDoFykjZM=; b=kgNGeY8RqW2PNK+m1z8ng80F1y5nMIrcwgO4Tkec9G6d+10doDKxNPim8IBwA4eBwg powcyIUB1MU7OU+fNpeJy13OVymNuwBI67hrwsztDG8ZUlxFXye59hw4CN0O5lvDzoYo hVtTyfOgIOxkwJuhoEm49RY6cpGeQ9ZA4leOTn0d//XLHWrdP4DTS1ReURIjy2oMeUoX hWoPQAFLNTC65pKFF7yHhOPkaGFrtUgTwy5op7fioOHfLw9PmqsULp/9ZtxmNi488oFU xwQREBhBinfcGFhfmv7KPWv7CxtxfWDJ2n5b18kfGFOmubnh/sEf0j35KUcF0rOtVbWr crNQ== X-Gm-Message-State: ALoCoQnriOFgEpRvhDPCB/PeaOhHmeteDRKNN7wOalRNIzjSv5U1K/n4xF69WZnsOIoy5fPzj+3h X-Received: by 10.194.20.202 with SMTP id p10mr10505497wje.39.1381431666024; Thu, 10 Oct 2013 12:01:06 -0700 (PDT) MIME-Version: 1.0 X-Originating-IP: [146.186.247.30] In-Reply-To: References: <73f03f7c-6a0a-449c-af60-ab204844d578@googlegroups.com> <5256F4C1.4050700@nedbatchelder.com> From: Isaac Gerg Date: Thu, 10 Oct 2013 15:00:45 -0400 Subject: Re: Python 3.2 | WIndows 7 -- Multiprocessing and files not closing To: Ned Batchelder Content-Type: multipart/alternative; boundary=047d7b5d9657bc3f3d04e8679fde Cc: python-list@python.org X-BeenThere: python-list@python.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Newsgroups: comp.lang.python Message-ID: Lines: 229 NNTP-Posting-Host: 2001:888:2000:d::a6 X-Trace: 1381431674 news.xs4all.nl 15975 [2001:888:2000:d::a6]:45494 X-Complaints-To: abuse@xs4all.nl Xref: csiph.com comp.lang.python:56601 --047d7b5d9657bc3f3d04e8679fde Content-Type: text/plain; charset=ISO-8859-1 On Thu, Oct 10, 2013 at 2:49 PM, Isaac Gerg wrote: > > > > On Thu, Oct 10, 2013 at 2:41 PM, Ned Batchelder wrote: > >> On 10/10/13 12:44 PM, Isaac Gerg wrote: >> >>> Sorry, I am just providing pseudo code since I the code i have is quite >>> large. >>> >>> As I mentioned, the code works fine when I remove the multirpcessing >>> stuff so the filename is not the issue (though you are right in your >>> correction). >>> >>> Someone with the same problem posted a smaller, more complete example >>> here: >>> >>> http://stackoverflow.com/**questions/948119/preventing-** >>> file-handle-inheritance-in-**multiprocessing-lib >>> >>> None of the solutions posted work. >>> >> >> (BTW: it's better form to reply beneath the original text, not above it.) >> >> None of the solutions try the obvious thing of closing the file before >> spawning more processes. Would that work for you? A "with" statement is a >> convenient way to do this: >> >> with open(filename,'r') as f: >> data = f.read() >> >> The file is closed automatically when the with statement ends. >> >> --Ned. >> >> >>> On Thursday, October 10, 2013 12:38:19 PM UTC-4, Piet van Oostrum wrote: >>> >>>> Isaac Gerg writes: >>>> >>>> >>>> >>>> I have a function that looks like the following: >>>>> >>>> >>>> >>>> That doesn't look like a function >>>> >>>> >>>> >>>> #-----------------------------**---- >>>>> filename = 'c:\testfile.h5' >>>>> >>>> >>>> >>>> Your filename is most probably wrong. It should be something like: >>>> >>>> >>>> >>>> filename = 'c:/testfile.h5' >>>> >>>> filename = 'c:\\testfile.h5' >>>> >>>> filename = r'c:\testfile.h5' >>>> >>>> -- >>>> >>>> Piet van Oostrum >>>> >>>> WWW: http://pietvanoostrum.com/ >>>> >>>> PGP key: [8DAE142BE17999C4] >>>> >>> >> > I will try what you suggest and see if it works. > > Additionally, is there a place on the web to view this conversation and > reply? Currently, I am only able to access this list through email. > Ned, I am unable to try what you suggest. The multiprocess.Process call is within a class but its target is a static method outside of the class thus no pickling. I cannot close the file and then reopen after the multiprocess.Process call because other threads may be reading from the file during that time. Is there a way in Python 3.2 to prevent the multiprocess.Process from inheriting the file descriptors from the parent process OR, is there a way to ensure that the multiprocess is completely closed and garbaged collected by the time I want to use os.remove()? Isaac --047d7b5d9657bc3f3d04e8679fde Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
On Thu, Oct 10, 2013 at 2:49 PM, Isaac Gerg <isaac.g= erg@gergltd.com> wrote:

=


On Thu, Oct 1= 0, 2013 at 2:41 PM, Ned Batchelder <ned@nedbatchelder.com> wrote:
On 10/10/13 12:44 PM, Isaac Gerg wrote:
Sorry, I am just providing pseudo code since I the code i have is quite lar= ge.

As I mentioned, the code works fine when I remove the multirpcessing stuff = so the filename is not the issue (though you are right in your correction).=

Someone with the same problem posted a smaller, more complete example here:=

http://stackoverflow= .com/questions/948119/preventing-file-handle-inheritance-in-<= u>multiprocessing-lib

None of the solutions posted work.

(BTW: it's better form to reply beneath the original text, not above it= .)

None of the solutions try the obvious thing of closing the file before spaw= ning more processes. =A0Would that work for you? =A0A "with" stat= ement is a convenient way to do this:

=A0 =A0 with open(filename,'r') as f:
=A0 =A0 =A0 =A0 data =3D f.read()

The file is closed automatically when the with statement ends.

--Ned.


On Thursday, October 10, 2013 12:38:19 PM UTC-4, Piet van Oostrum wrote:
Isaac Gerg <= isaac.gerg@gergltd.com> writes:



I have a function that looks like the following:


That doesn't look like a function



#---------------------------------
filename =3D 'c:\testfile.h5'


Your filename is most probably wrong. It should be something like:



filename =3D 'c:/testfile.h5'

filename =3D 'c:\\testfile.h5'

filename =3D r'c:\testfile.h5'

--

Piet van Oostrum <piet@vanoostrum.org>

WWW: http://pietva= noostrum.com/

PGP key: [8DAE142BE17999C4]


I will = try what you suggest and see if it works.
<= br>
Additionally, is there a place on the w= eb to view this conversation and reply? =A0Currently, I am only able to acc= ess this list through email.


Ned, I am unable to try what you suggest. =A0The multipro= cess.Process call is within a class but its target is a static method outsi= de of the class thus no pickling. =A0I cannot close the file and then reope= n after the multiprocess.Process call because other threads may be reading = from the file during that time. =A0Is there a way in Python 3.2 to prevent = the multiprocess.Process from inheriting the file descriptors from the pare= nt process OR, is there a way to ensure that the multiprocess is completely= closed and garbaged collected by the time I want to use os.remove()?

Isaac
=
--047d7b5d9657bc3f3d04e8679fde--