Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #102993 > unrolled thread

Will file be closed automatically in a "for ... in open..." statement?

Back to article view | Back to comp.lang.python

Contents

  Will file be closed automatically in a "for ... in open..." statement? jfong@ms4.hinet.net - 2016-02-16 00:39 -0800
    Re: Will file be closed automatically in a "for ... in open..." statement? Chris Angelico <rosuav@gmail.com> - 2016-02-16 20:16 +1100
    Re: Will file be closed automatically in a "for ... in open..." statement? Cameron Simpson <cs@zip.com.au> - 2016-02-16 20:24 +1100
    Re: Will file be closed automatically in a "for ... in open..." statement? Terry Reedy <tjreedy@udel.edu> - 2016-02-16 04:41 -0500
    Re: Will file be closed automatically in a "for ... in open..." statement? jfong@ms4.hinet.net - 2016-02-16 20:04 -0800
      Re: Will file be closed automatically in a "for ... in open..." statement? Chris Angelico <rosuav@gmail.com> - 2016-02-17 15:36 +1100
      Re: Will file be closed automatically in a "for ... in open..." statement? Raspberry Aether <raspberryaether@e-s.invalid> - 2016-02-16 23:42 -0500
      Re: Will file be closed automatically in a "for ... in open..." statement? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2016-02-17 16:51 +1100
        Re: Will file be closed automatically in a "for ... in open..." statement? Tim Chase <python.list@tim.thechases.com> - 2016-02-17 20:32 -0600
    Re: Will file be closed automatically in a "for ... in open..." statement? jfong@ms4.hinet.net - 2016-02-17 17:29 -0800
      Re: Will file be closed automatically in a "for ... in open..." statement? Mark Lawrence <breamoreboy@yahoo.co.uk> - 2016-02-18 02:05 +0000
    Re: Will file be closed automatically in a "for ... in open..." statement? Jeremy Leonard <jrmy.lnrd@gmail.com> - 2016-02-18 10:02 -0800

#102993 — Will file be closed automatically in a "for ... in open..." statement?

I know

    with open('foo.txt') as f:
        ...do something...

will close the file automatically when the "with" block ends. 

I also saw codes in a book:

    for line in open('foo.txt'):
	...do something...

but it didn't mention if the file will be closed automatically or not when the "for" block ends. Is there any document talking about this? and how to know if a file is in "open" or not?

--Jach Fong

[toc] | [next] | [standalone]

#102994

On Tue, Feb 16, 2016 at 7:39 PM,  <jfong@ms4.hinet.net> wrote:
> I know
>
>     with open('foo.txt') as f:
>         ...do something...
>
> will close the file automatically when the "with" block ends.
>
> I also saw codes in a book:
>
>     for line in open('foo.txt'):
>         ...do something...
>
> but it didn't mention if the file will be closed automatically or not when the "for" block ends. Is there any document talking about this? and how to know if a file is in "open" or not?
>

The file will be closed when the open file object is disposed of. That
will happen at some point after there are no more references to it.
You're guaranteed that it stays around for the entire duration of the
'for' loop (the loop keeps track of the thing it's iterating over),
but exactly when after that is not guaranteed. In current versions of
CPython, the garbage collector counts references, so the file will be
closed immediately; but other Python interpreters, and future versions
of CPython, may not behave the same way. So the file will *probably*
be closed *reasonably* promptly, but unlike the "with" case, you have
no guarantee that it'll be immediate.

For small scripts, it probably won't even matter, though. You're
unlikely to run out of file handles, and the only time it would matter
is if you're opening, closing, and then reopening the file - for
example:

fn = input("Name a file to frobnosticate: ")
with open(fn) as f:
    data = []
    for line in f:
        data = frobnosticate(data, line)
with open(fn, "w") as f:
    f.writelines(data)

For this to work reliably, the file MUST be closed for reading before
it's opened for writing. The context managers are important. But this
is pretty unusual.

Of course, since it's so little trouble to use the 'with' block, it's
generally worth just using it everywhere. Why run the risk? :)

ChrisA
who often forgets to use 'with' anyway

[toc] | [prev] | [next] | [standalone]

#102995

On 16Feb2016 00:39, jfong@ms4.hinet.net <jfong@ms4.hinet.net> wrote:
>I know
>
>    with open('foo.txt') as f:
>        ...do something...
>
>will close the file automatically when the "with" block ends.

Yes, because open is a context manager - they're great for reliably tidying up 
in the face of exceptions or "direct" departure from the block, such as a 
"return" statement.

>I also saw codes in a book:
>
>    for line in open('foo.txt'):
>	...do something...
>
>but it didn't mention if the file will be closed automatically or not when the 
>"for" block ends. Is there any document talking about this? and how to know if 
>a file is in "open" or not?

This does not reliably close the file.

In CPython (the common implementation, and likely what you are using), objects 
are reference counted and when the interpreter notices their counter go to 
zero, the object's __del__ method is called before releasing the object's 
memory.

For an open file, __del__ _does_ call close if the file is open. However, only 
reference counting Pythons will call __del__ promptly - other systems rely on 
garbage collectors to detect unused objects.

In the for loop above, for interpreter obtains an iterator from the open file 
which returns lines of text. That iterator has a reference to the open file, 
and the for loop has a reference to the iterator. Therefore the file remains 
references while the loop runs. AT the end of the loop the iterator is 
discarded, reducing its references to zero. That in turn triggers releasing the 
open file, dropping its references to zero.

In CPython, that in turn will fire the open file's __del__, which will close 
the file. In other Pythons, not necessarily that promptly.

Also, there are plenty of ways to phrase this where the file reference doesn't 
go to zero. FOr example:

  f = open(...)
  for line in f:
    ...

You can see that it is easy to forget to close the file here (especially if you 
have an exception or exit the function precipitously).

Try to use the "with open(...) as f:" formulation when possible. It is much 
better.

Cheers,
Cameron Simpson <cs@zip.com.au>

[toc] | [prev] | [next] | [standalone]

#102996

On 2/16/2016 3:39 AM, jfong@ms4.hinet.net wrote:
> I know
>
>      with open('foo.txt') as f:
>          ...do something...
>
> will close the file automatically when the "with" block ends.
>
> I also saw codes in a book:
>
>      for line in open('foo.txt'):
> 	...do something...

Some books were originally written before 'with context_manager' was 
added, or a little later, before it became the normal thing to do.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#103041

Thanks for these detailed explanation. Both statements will close file automatically sooner or later and, when considering the exceptions, "with" is better. Hope my understanding is right.

But, just curious, how do you know the "for" will do it? I can't find any document about it from every sources I know. Very depressed:-(

--Jach

[toc] | [prev] | [next] | [standalone]

#103042

On Wed, Feb 17, 2016 at 3:04 PM,  <jfong@ms4.hinet.net> wrote:
> Thanks for these detailed explanation. Both statements will close file automatically sooner or later and, when considering the exceptions, "with" is better. Hope my understanding is right.
>
> But, just curious, how do you know the "for" will do it? I can't find any document about it from every sources I know. Very depressed:-(
>

It's not the 'for' loop that does it. The for loop is kinda like this:

_temp = open("foo.txt")
_temp.read() # do stuff, do stuff
_temp = None

When you stop holding onto an object, Python can get rid of it. When
that happens is not promised, though - and if you have a reference
loop, it might hang around for a long time. But when a file object is
disposed of, the underlying file will get closed.

ChrisA

[toc] | [prev] | [next] | [standalone]

#103043

On 02/16/2016 11:04 PM, jfong@ms4.hinet.net wrote:
> Thanks for these detailed explanation. Both statements will close file automatically sooner or later and, when considering the exceptions, "with" is better. Hope my understanding is right.
> 
> But, just curious, how do you know the "for" will do it? I can't find any document about it from every sources I know. Very depressed:-(
> 
> --Jach
> 

First-- IMO, don't depend on it. Instead, use something like:

with open('foo.txt') as f:
    for line in f:
        pass # do something here

It's one extra indent and one extra line, but it's cleaner.

To answer your question, technically, it might not-- it really
depends upon your implementation of Python. It just so happens
that the most popular version of Python (CPython, the reference
implementation) will garbage collect the file object right away.

HOWEVER. The reason the "for" will PROBABLY result in file
closure is because as soon as the for loop exits, there is no
reason to hold onto the object returned by "open", so it is
disposed. When file objects are disposed, they are closed.

IMO, don't depend on this behaviour; it's bad form.

[toc] | [prev] | [next] | [standalone]

#103045

On Wednesday 17 February 2016 15:04, jfong@ms4.hinet.net wrote:

> Thanks for these detailed explanation. Both statements will close file
> automatically sooner or later and, when considering the exceptions, "with"
> is better. Hope my understanding is right.
> 
> But, just curious, how do you know the "for" will do it? I can't find any
> document about it from every sources I know. Very depressed:-(

This has nothing to do with "for". You would get exactly the same behaviour 
without "for":


f = open("some file", "r")
x = f.read(20)
x = f.read(30)
x = f.read()


So long as the variable "f" is in-scope, the file will stay open. If the 
above code is in a function, "f" goes out of scope when the function 
returns. If the above code is at the top level of the module, "f" will stay 
in scope forever, or until you either delete the variable from the scope:

del f

or re-assign to something else:

f = "hello world"


At this point, after the function has exited, and the variable has 
completely gone out of scope, what happens?

The garbage collector will:

- reclaim the memory used by the object;
- close the file.

BUT there is no promise WHEN the file will be closed. It might be 
immediately, or it might be when the application shuts down.

If you want the file to be closed immediately, you must:

- use a with statement;

- or explicitly call f.close()


otherwise you are at the mercy of the interpreter, which will close the file 
whenever it wants.


-- 
Steve

[toc] | [prev] | [next] | [standalone]

#103081

On 2016-02-17 16:51, Steven D'Aprano wrote:
> If you want the file to be closed immediately, you must:
> 
> - use a with statement;
> 
> - or explicitly call f.close()

I have a lot of pre-"with" code (i.e., Py2.4) that looks like

  f = open(...)
  try:
    do_stuff()
  finally:
    f.close()

To explicitly close() correctly, you still have to pay the
one-level-of-indent cost regardless of whether you use a "with" or
close()

Now that we have the "with" statement, it's the same cost, with no
lost functionality, but with fewer lines of messy code.

So use the "with" unless you're sadly maintaining 2.4 code like
me. :-)

-tkc

[toc] | [prev] | [next] | [standalone]

#103078

The "for ... open ..." is definitely not a good design pattern. It opens a file at "for" block but leaves it closed somewhere in the sky.

> The garbage collector will:
> - reclaim the memory used by the object;
> - close the file. 

I suppose (IMO) that the primitive idea of garbage collection doing the file-close affair is to make itself a final defense to prevent the possible disaster. (after all it's not of his business:-) But some coding may abuse this advantage and forget its hidden danger.

It's better to drop this "for...open..." style completely and stick at "with".

Thanks for all you gurus clarify my puzzle. I deeply appreciated it.

--Jach

[toc] | [prev] | [next] | [standalone]

#103079

On 18/02/2016 01:29, jfong@ms4.hinet.net wrote:
> The "for ... open ..." is definitely not a good design pattern. It opens a file at "for" block but leaves it closed somewhere in the sky.
>

Hardly, as all ready explained, but how about this

handle = open('foo.txt')
for line in handle :
     ...do something...
handle.close()

Is that better?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[toc] | [prev] | [next] | [standalone]

#103143

On Tuesday, February 16, 2016 at 3:39:34 AM UTC-5, jf...@ms4.hinet.net wrote:
> I know
> 
>     with open('foo.txt') as f:
>         ...do something...
> 
> will close the file automatically when the "with" block ends. 
> 
> I also saw codes in a book:
> 
>     for line in open('foo.txt'):
> 	...do something...
> 
> but it didn't mention if the file will be closed automatically or not when the "for" block ends. Is there any document talking about this? and how to know if a file is in "open" or not?
> 
> --Jach Fong

One way of having a file automatically closed is to have something like:

with open("file_name") as foo:
    for x in foo:
       # process line x

Once the with block has completed, the file will be automatically closed. Sorry if this has already been posted, but I wasn't able to see it.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web