Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #56583 > unrolled thread

Python 3.2 | WIndows 7 -- Multiprocessing and files not closing

Back to article view | Back to comp.lang.python

Contents

  Python 3.2 | WIndows 7 -- Multiprocessing and files not closing Isaac Gerg <isaac.gerg@gergltd.com> - 2013-10-10 08:31 -0700
    Re: Python 3.2 | WIndows 7 -- Multiprocessing and files not closing Piet van Oostrum <piet@vanoostrum.org> - 2013-10-10 12:38 -0400
      Re: Python 3.2 | WIndows 7 -- Multiprocessing and files not closing Isaac Gerg <isaac.gerg@gergltd.com> - 2013-10-10 09:44 -0700
        Re: Python 3.2 | WIndows 7 -- Multiprocessing and files not closing Ned Batchelder <ned@nedbatchelder.com> - 2013-10-10 14:41 -0400
        Re: Python 3.2 | WIndows 7 -- Multiprocessing and files not closing Isaac Gerg <isaac.gerg@gergltd.com> - 2013-10-10 14:49 -0400
        Re: Python 3.2 | WIndows 7 -- Multiprocessing and files not closing Isaac Gerg <isaac.gerg@gergltd.com> - 2013-10-10 15:00 -0400
        Re: Python 3.2 | WIndows 7 -- Multiprocessing and files not closing Piet van Oostrum <piet@vanoostrum.org> - 2013-10-10 15:38 -0400
          Re: Python 3.2 | WIndows 7 -- Multiprocessing and files not closing Isaac Gerg <isaac.gerg@gergltd.com> - 2013-10-10 14:15 -0700
        Re: Python 3.2 | WIndows 7 -- Multiprocessing and files not closing Terry Reedy <tjreedy@udel.edu> - 2013-10-10 21:45 -0400
    Re: Python 3.2 | WIndows 7 -- Multiprocessing and files not closing mapoe <matthias@kill.at> - 2013-10-10 18:16 -0500
      Re: Python 3.2 | WIndows 7 -- Multiprocessing and files not closing mapoe <matthias@kill.at> - 2013-10-10 18:29 -0500

#56583 — Python 3.2 | WIndows 7 -- Multiprocessing and files not closing

I have a function that looks like the following:

#---------------------------------
filename = 'c:\testfile.h5'
f = open(filename,'r')
data = f.read()

q = multiprocessing.Queue()
p = multiprocess.Process(target=myFunction,args=(data,q))
p.start()
result = q.get()
p.join()
q.close()

f.close()

os.remove(filename)
#---------------------------------

When I run this code, I get an error on the last line when I try to remove the file.  It tells me that someone has access to the file.  When I remove the queue and multiprocessing stuff, the function works fine.

What is going on here?

Thanks in advance,
Isaac

[toc] | [next] | [standalone]

#56590

Isaac Gerg <isaac.gerg@gergltd.com> writes:

> I have a function that looks like the following:

That doesn't look like a function

>
> #---------------------------------
> filename = 'c:\testfile.h5'

Your filename is most probably wrong. It should be something like:

filename = 'c:/testfile.h5'
filename = 'c:\\testfile.h5'
filename = r'c:\testfile.h5'
-- 
Piet van Oostrum <piet@vanoostrum.org>
WWW: http://pietvanoostrum.com/
PGP key: [8DAE142BE17999C4]

[toc] | [prev] | [next] | [standalone]

#56591

Sorry, I am just providing pseudo code since I the code i have is quite large.

As I mentioned, the code works fine when I remove the multirpcessing stuff so the filename is not the issue (though you are right in your correction).

Someone with the same problem posted a smaller, more complete example here:

http://stackoverflow.com/questions/948119/preventing-file-handle-inheritance-in-multiprocessing-lib

None of the solutions posted work.

On Thursday, October 10, 2013 12:38:19 PM UTC-4, Piet van Oostrum wrote:
> Isaac Gerg <isaac.gerg@gergltd.com> writes:
> 
> 
> 
> > I have a function that looks like the following:
> 
> 
> 
> That doesn't look like a function
> 
> 
> 
> >
> 
> > #---------------------------------
> 
> > filename = 'c:\testfile.h5'
> 
> 
> 
> Your filename is most probably wrong. It should be something like:
> 
> 
> 
> filename = 'c:/testfile.h5'
> 
> filename = 'c:\\testfile.h5'
> 
> filename = r'c:\testfile.h5'
> 
> -- 
> 
> Piet van Oostrum <piet@vanoostrum.org>
> 
> WWW: http://pietvanoostrum.com/
> 
> PGP key: [8DAE142BE17999C4]

[toc] | [prev] | [next] | [standalone]

#56598

On 10/10/13 12:44 PM, Isaac Gerg wrote:
> Sorry, I am just providing pseudo code since I the code i have is quite large.
>
> As I mentioned, the code works fine when I remove the multirpcessing stuff so the filename is not the issue (though you are right in your correction).
>
> Someone with the same problem posted a smaller, more complete example here:
>
> http://stackoverflow.com/questions/948119/preventing-file-handle-inheritance-in-multiprocessing-lib
>
> None of the solutions posted work.

(BTW: it's better form to reply beneath the original text, not above it.)

None of the solutions try the obvious thing of closing the file before 
spawning more processes.  Would that work for you?  A "with" statement 
is a convenient way to do this:

     with open(filename,'r') as f:
         data = f.read()

The file is closed automatically when the with statement ends.

--Ned.

>
> On Thursday, October 10, 2013 12:38:19 PM UTC-4, Piet van Oostrum wrote:
>> Isaac Gerg <isaac.gerg@gergltd.com> writes:
>>
>>
>>
>>> I have a function that looks like the following:
>>
>>
>> That doesn't look like a function
>>
>>
>>
>>> #---------------------------------
>>> filename = 'c:\testfile.h5'
>>
>>
>> Your filename is most probably wrong. It should be something like:
>>
>>
>>
>> filename = 'c:/testfile.h5'
>>
>> filename = 'c:\\testfile.h5'
>>
>> filename = r'c:\testfile.h5'
>>
>> -- 
>>
>> Piet van Oostrum <piet@vanoostrum.org>
>>
>> WWW: http://pietvanoostrum.com/
>>
>> PGP key: [8DAE142BE17999C4]

[toc] | [prev] | [next] | [standalone]

#56600

[Multipart message — attachments visible in raw view] — view raw

On Thu, Oct 10, 2013 at 2:41 PM, Ned Batchelder <ned@nedbatchelder.com>wrote:

> On 10/10/13 12:44 PM, Isaac Gerg wrote:
>
>> Sorry, I am just providing pseudo code since I the code i have is quite
>> large.
>>
>> As I mentioned, the code works fine when I remove the multirpcessing
>> stuff so the filename is not the issue (though you are right in your
>> correction).
>>
>> Someone with the same problem posted a smaller, more complete example
>> here:
>>
>> http://stackoverflow.com/**questions/948119/preventing-**
>> file-handle-inheritance-in-**multiprocessing-lib<http://stackoverflow.com/questions/948119/preventing-file-handle-inheritance-in-multiprocessing-lib>
>>
>> None of the solutions posted work.
>>
>
> (BTW: it's better form to reply beneath the original text, not above it.)
>
> None of the solutions try the obvious thing of closing the file before
> spawning more processes.  Would that work for you?  A "with" statement is a
> convenient way to do this:
>
>     with open(filename,'r') as f:
>         data = f.read()
>
> The file is closed automatically when the with statement ends.
>
> --Ned.
>
>
>> On Thursday, October 10, 2013 12:38:19 PM UTC-4, Piet van Oostrum wrote:
>>
>>> Isaac Gerg <isaac.gerg@gergltd.com> writes:
>>>
>>>
>>>
>>>  I have a function that looks like the following:
>>>>
>>>
>>>
>>> That doesn't look like a function
>>>
>>>
>>>
>>>  #-----------------------------**----
>>>> filename = 'c:\testfile.h5'
>>>>
>>>
>>>
>>> Your filename is most probably wrong. It should be something like:
>>>
>>>
>>>
>>> filename = 'c:/testfile.h5'
>>>
>>> filename = 'c:\\testfile.h5'
>>>
>>> filename = r'c:\testfile.h5'
>>>
>>> --
>>>
>>> Piet van Oostrum <piet@vanoostrum.org>
>>>
>>> WWW: http://pietvanoostrum.com/
>>>
>>> PGP key: [8DAE142BE17999C4]
>>>
>>
>
I will try what you suggest and see if it works.

Additionally, is there a place on the web to view this conversation and
reply?  Currently, I am only able to access this list through email.

[toc] | [prev] | [next] | [standalone]

#56601

[Multipart message — attachments visible in raw view] — view raw

On Thu, Oct 10, 2013 at 2:49 PM, Isaac Gerg <isaac.gerg@gergltd.com> wrote:

>
>
>
> On Thu, Oct 10, 2013 at 2:41 PM, Ned Batchelder <ned@nedbatchelder.com>wrote:
>
>> On 10/10/13 12:44 PM, Isaac Gerg wrote:
>>
>>> Sorry, I am just providing pseudo code since I the code i have is quite
>>> large.
>>>
>>> As I mentioned, the code works fine when I remove the multirpcessing
>>> stuff so the filename is not the issue (though you are right in your
>>> correction).
>>>
>>> Someone with the same problem posted a smaller, more complete example
>>> here:
>>>
>>> http://stackoverflow.com/**questions/948119/preventing-**
>>> file-handle-inheritance-in-**multiprocessing-lib<http://stackoverflow.com/questions/948119/preventing-file-handle-inheritance-in-multiprocessing-lib>
>>>
>>> None of the solutions posted work.
>>>
>>
>> (BTW: it's better form to reply beneath the original text, not above it.)
>>
>> None of the solutions try the obvious thing of closing the file before
>> spawning more processes.  Would that work for you?  A "with" statement is a
>> convenient way to do this:
>>
>>     with open(filename,'r') as f:
>>         data = f.read()
>>
>> The file is closed automatically when the with statement ends.
>>
>> --Ned.
>>
>>
>>> On Thursday, October 10, 2013 12:38:19 PM UTC-4, Piet van Oostrum wrote:
>>>
>>>> Isaac Gerg <isaac.gerg@gergltd.com> writes:
>>>>
>>>>
>>>>
>>>>  I have a function that looks like the following:
>>>>>
>>>>
>>>>
>>>> That doesn't look like a function
>>>>
>>>>
>>>>
>>>>  #-----------------------------**----
>>>>> filename = 'c:\testfile.h5'
>>>>>
>>>>
>>>>
>>>> Your filename is most probably wrong. It should be something like:
>>>>
>>>>
>>>>
>>>> filename = 'c:/testfile.h5'
>>>>
>>>> filename = 'c:\\testfile.h5'
>>>>
>>>> filename = r'c:\testfile.h5'
>>>>
>>>> --
>>>>
>>>> Piet van Oostrum <piet@vanoostrum.org>
>>>>
>>>> WWW: http://pietvanoostrum.com/
>>>>
>>>> PGP key: [8DAE142BE17999C4]
>>>>
>>>
>>
> I will try what you suggest and see if it works.
>
> Additionally, is there a place on the web to view this conversation and
> reply?  Currently, I am only able to access this list through email.
>


Ned, I am unable to try what you suggest.  The multiprocess.Process call is
within a class but its target is a static method outside of the class thus
no pickling.  I cannot close the file and then reopen after the
multiprocess.Process call because other threads may be reading from the
file during that time.  Is there a way in Python 3.2 to prevent the
multiprocess.Process from inheriting the file descriptors from the parent
process OR, is there a way to ensure that the multiprocess is completely
closed and garbaged collected by the time I want to use os.remove()?

Isaac

[toc] | [prev] | [next] | [standalone]

#56604

Isaac Gerg <isaac.gerg@gergltd.com> writes:

> Sorry, I am just providing pseudo code since I the code i have is quite large.
>
> As I mentioned, the code works fine when I remove the multirpcessing stuff so the filename is not the issue (though you are right in your correction).
>
> Someone with the same problem posted a smaller, more complete example here:
>
Then you should give us real code (a minimal example) that we can try. You use myFunction in your example that isn't defined in your code. And by the way, why don't you close f just after reading? Or even better, use a with statement.
-- 
Piet van Oostrum <piet@vanoostrum.org>
WWW: http://pietvanoostrum.com/
PGP key: [8DAE142BE17999C4]

[toc] | [prev] | [next] | [standalone]

#56608

Hi Piet,

Here is a real code example: http://stackoverflow.com/questions/948119/preventing-file-handle-inheritance-in-multiprocessing-lib

As I said before, I had provide pseudocode.

I cannot close the file after reading because it is part of a class and other threads may be calling member functions which read from the file.

Isaac

[toc] | [prev] | [next] | [standalone]

#56630

On 10/10/2013 2:49 PM, Isaac Gerg wrote:

> Additionally, is there a place on the web to view this conversation and
> reply?  Currently, I am only able to access this list through email.

news.gmane.org newsgroup gmane.lang.python.general
Look at the headers for this message.
The site also has a searchable archive.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#56614

On Thu, 10 Oct 2013 08:31:21 -0700, Isaac Gerg wrote:

> I have a function that looks like the following:
> 
> #---------------------------------
> filename = 'c:\testfile.h5'
> f = open(filename,'r')
> data = f.read()

it seems kind of obvious from your sample:
add: f.close()

> q = multiprocessing.Queue()
> p = multiprocess.Process(target=myFunction,args=(data,q))
> p.start()
> result = q.get()
> p.join()
> q.close()
> 
> f.close()
> 
> os.remove(filename)
> #---------------------------------
> 
> When I run this code, I get an error on the last line when I try to
> remove the file.  It tells me that someone has access to the file.  When
> I remove the queue and multiprocessing stuff, the function works fine.
> 
> What is going on here?
> 
> Thanks in advance,
> Isaac

[toc] | [prev] | [next] | [standalone]

#56616

On Thu, 10 Oct 2013 18:16:07 -0500, mapoe wrote:

> On Thu, 10 Oct 2013 08:31:21 -0700, Isaac Gerg wrote:
> 
>> I have a function that looks like the following:
>> 
>> #---------------------------------
>> filename = 'c:\testfile.h5'
>> f = open(filename,'r')
>> data = f.read()
> 
> it seems kind of obvious from your sample:
> add: f.close()
should have read a little bit further :)


but I would close the file right after I have read all the data



> 
>> q = multiprocessing.Queue()
>> p = multiprocess.Process(target=myFunction,args=(data,q))
>> p.start()
>> result = q.get()
>> p.join()
>> q.close()
>> 
>> f.close()
>> 
>> os.remove(filename)
>> #---------------------------------
>> 
>> When I run this code, I get an error on the last line when I try to
>> remove the file.  It tells me that someone has access to the file. 
>> When I remove the queue and multiprocessing stuff, the function works
>> fine.
>> 
>> What is going on here?
>> 
>> Thanks in advance,
>> Isaac

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web