Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #28347 > unrolled thread

The opener parameter of Python 3 open() built-in

Back to article view | Back to comp.lang.python

Contents

  The opener parameter of Python 3 open() built-in Marco <marco_u@nsgmail.com> - 2012-09-03 14:32 +0200
    Re: The opener parameter of Python 3 open() built-in Dave Angel <d@davea.name> - 2012-09-03 09:05 -0400
      Re: The opener parameter of Python 3 open() built-in Marco <marco_u@nsgmail.com> - 2012-09-03 15:33 +0200
      Re: The opener parameter of Python 3 open() built-in Marco <marco.buttu@gmail.com> - 2012-09-03 15:33 +0200
    Re: The opener parameter of Python 3 open() built-in Christian Heimes <lists@cheimes.de> - 2012-09-03 15:29 +0200
      Re: The opener parameter of Python 3 open() built-in Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-09-04 01:13 +0000
        Re: The opener parameter of Python 3 open() built-in Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-09-03 23:19 -0400
          Re: The opener parameter of Python 3 open() built-in Ben Finney <ben+python@benfinney.id.au> - 2012-09-04 15:12 +1000
            Re: The opener parameter of Python 3 open() built-in Ben Finney <ben+python@benfinney.id.au> - 2012-09-04 15:20 +1000
            Re: The opener parameter of Python 3 open() built-in Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-09-04 14:44 -0400
          Re: The opener parameter of Python 3 open() built-in Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-09-04 05:25 +0000
            Re: The opener parameter of Python 3 open() built-in Ben Finney <ben+python@benfinney.id.au> - 2012-09-04 15:45 +1000
        Re: The opener parameter of Python 3 open() built-in Serhiy Storchaka <storchaka@gmail.com> - 2012-09-04 15:58 +0300
        Re: The opener parameter of Python 3 open() built-in Terry Reedy <tjreedy@udel.edu> - 2012-09-04 15:16 -0400
        Re: The opener parameter of Python 3 open() built-in Chris Angelico <rosuav@gmail.com> - 2012-09-05 08:18 +1000
        Re: The opener parameter of Python 3 open() built-in Terry Reedy <tjreedy@udel.edu> - 2012-09-04 19:16 -0400
        Re: The opener parameter of Python 3 open() built-in Antoine Pitrou <solipsis@pitrou.net> - 2012-09-06 00:34 +0000
          Re: The opener parameter of Python 3 open() built-in Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-09-06 06:34 +0000
    Re: The opener parameter of Python 3 open() built-in Serhiy Storchaka <storchaka@gmail.com> - 2012-09-03 19:06 +0300
    Re: The opener parameter of Python 3 open() built-in Serhiy Storchaka <storchaka@gmail.com> - 2012-09-04 16:01 +0300

#28347 — The opener parameter of Python 3 open() built-in

Does anyone have an example of utilisation?

[toc] | [next] | [standalone]

#28348

On 09/03/2012 08:32 AM, Marco wrote:
> Does anyone have an example of utilisation?

As of Python 3.2.3, there is no "opener" parameter in the open() function.
    http://docs.python.org/py3k/library/functions.html

I don't know of any such parameter in earlier or later versions, but I
could be wrong there.

-- 

DaveA

[toc] | [prev] | [next] | [standalone]

#28350

On 09/03/2012 03:05 PM, Dave Angel wrote:

>> Does anyone have an example of utilisation?

> As of Python 3.2.3, there is no "opener" parameter in the open() function.
>      http://docs.python.org/py3k/library/functions.html
>
> I don't know of any such parameter in earlier or later versions, but I
> could be wrong there.

It's new in Python 3.3:

http://docs.python.org/dev/library/functions.html#open

[toc] | [prev] | [next] | [standalone]

#28351

On 09/03/2012 03:05 PM, Dave Angel wrote:

>> Does anyone have an example of utilisation?

> As of Python 3.2.3, there is no "opener" parameter in the open() function.
>      http://docs.python.org/py3k/library/functions.html
>
> I don't know of any such parameter in earlier or later versions, but I
> could be wrong there.

It's new in Python 3.3:

http://docs.python.org/dev/library/functions.html#open

[toc] | [prev] | [next] | [standalone]

#28349

Am 03.09.2012 14:32, schrieb Marco:
> Does anyone have an example of utilisation?

The opener argument is a new 3.3 feature. For example you can use the
feature to implement exclusive creation of a file to avoid symlink attacks.

import os

def opener(file, flags):
    return os.open(file, flags | os.O_EXCL)

open("newfile", "w", opener=opener)

[toc] | [prev] | [next] | [standalone]

#28378

On Mon, 03 Sep 2012 15:29:05 +0200, Christian Heimes wrote:

> Am 03.09.2012 14:32, schrieb Marco:
>> Does anyone have an example of utilisation?
> 
> The opener argument is a new 3.3 feature. For example you can use the
> feature to implement exclusive creation of a file to avoid symlink
> attacks.
> 
> import os
> 
> def opener(file, flags):
>     return os.open(file, flags | os.O_EXCL)
> 
> open("newfile", "w", opener=opener)


Why does the open builtin need this added complexity? Why not just call 
os.open directly? Or for more complex openers, just call the opener 
directly?

What is the rationale for complicating open instead of telling people to 
just call their opener directly?


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#28386

On 04 Sep 2012 01:13:09 GMT, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> declaimed the following in
gmane.comp.python.general:


> 
> Why does the open builtin need this added complexity? Why not just call 
> os.open directly? Or for more complex openers, just call the opener 
> directly?
>
	Because os.open() returns a low-level file descriptor, not a Python
file object?

> What is the rationale for complicating open instead of telling people to 
> just call their opener directly?

	To avoid the new syntax would mean coding the example as

	f = os.fdopen(os.open("newfile", flags | os.O_EXCL), "w") 

which does NOT look any cleaner to me... Especially not if "opener" is
to be used in more than one location. Furthermore, using "opener" could
allow for a localized change to affect all open statements in the module
-- change file path, open for string I/O rather than file I/O, etc.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#28387

Dennis Lee Bieber <wlfraed@ix.netcom.com> writes:

> On 04 Sep 2012 01:13:09 GMT, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> declaimed the following in
> gmane.comp.python.general:
> > What is the rationale for complicating [the builtin] open instead of
> > telling people to just call their opener directly?
>
> 	To avoid the new syntax would mean coding the example as
>
> 	f = os.fdopen(os.open("newfile", flags | os.O_EXCL), "w") 
>
> which does NOT look any cleaner to me... Especially not if "opener" is
> to be used in more than one location.

Exactly. That's not what was asked, though. Steven asked why not call
the opener.

So, having written the opener:

> On Mon, 03 Sep 2012 15:29:05 +0200, Christian Heimes wrote:
> > import os
> > 
> > def opener(file, flags):
> >     return os.open(file, flags | os.O_EXCL)

why not call that directly?

    f = opener(file, flags)

It certainly is cleaner than either of the alternatives so far, and it
doesn't add a parameter to the builtin.


> Furthermore, using "opener" could allow for a localized change to
> affect all open statements in the module -- change file path, open for
> string I/O rather than file I/O, etc.

I don't know of any real-life code which would be significantly improved
by that. Can you point us to some?

-- 
 \      “I find the whole business of religion profoundly interesting. |
  `\     But it does mystify me that otherwise intelligent people take |
_o__)                                    it seriously.” —Douglas Adams |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#28390

Ben Finney <ben+python@benfinney.id.au> writes:

> So, having written the opener:
>
> > On Mon, 03 Sep 2012 15:29:05 +0200, Christian Heimes wrote:
> > > import os
> > > 
> > > def opener(file, flags):
> > >     return os.open(file, flags | os.O_EXCL)
>
> why not call that directly?
>
>     f = opener(file, flags)

Ah, because that returns the file descriptor, not the file. I see.

-- 
 \         “If nature has made any one thing less susceptible than all |
  `\    others of exclusive property, it is the action of the thinking |
_o__)              power called an idea” —Thomas Jefferson, 1813-08-13 |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#28420

On Tue, 04 Sep 2012 15:12:48 +1000, Ben Finney
<ben+python@benfinney.id.au> declaimed the following in
gmane.comp.python.general:

> 
> why not call that directly?
> 
>     f = opener(file, flags)
> 
> It certainly is cleaner than either of the alternatives so far, and it
> doesn't add a parameter to the builtin.
>
	But it returns an OS file descriptor... It doesn't return a Python
file object. From what I can tell, (I've just upgraded to Python 2.7
<G>) the opener is meant to replace the low-level function normally used
by Python's open(), and supplies an fd which gets wrapped by Python's
open().
 
>>> import os
>>> fd = os.open("somefile.txt", os.O_CREAT)
>>> type(fd)
<type 'int'>
>>> fo = open("somefile2.txt", "w")
>>> type (fo)
<type 'file'>

	The two are not compatible except by using os.fdopen(fd) to get a
file object, or fo.fileno() to get the low-level file descriptor

>>> fo.fileno()
4
>>> fo1 = os.fdopen(fd, "w")
>>> type (fo1)
<type 'file'>
>>> 


 
> > Furthermore, using "opener" could allow for a localized change to
> > affect all open statements in the module -- change file path, open for
> > string I/O rather than file I/O, etc.
> 
> I don't know of any real-life code which would be significantly improved
> by that. Can you point us to some?
>
	Not really -- but if they went one step further and supplied
"reader" and "writer" operations too, they'd get close to what I once
had to do in FORTRAN 77 under DEC VMS (by hooking in code to do double
buffering when reading data from magtape, while keeping the program
using regular F77 I/O statements; the open statement would do a pre-read
of one buffer and return; subsequent read statements would find a
pre-filled buffer, and issue an non-blocking read to fill the other
buffer -- cut the runtime for the program into a third or less as it was
no longer stuck waiting for slow mag-tape operations each time it did a
read). Implementing something like this in Python would likely require
"opener" to spawn a reader thread to do the I/O asynchronously, using a
limited Queue (1 buffer worth -- the reader thread would be the second
buffer, blocked on Q.put()), and a "reader" that would do Q.get() and
return the result to the Python read() logic for any parsing.

	Okay, in Python, one could probably subclass "file", and override
the read methods -- but one would not be able to use the Python
open()... You'd have to do something like f = myFile(normal, open, args)
instead...


> -- 
>  \      “I find the whole business of religion profoundly interesting. |
>   `\     But it does mystify me that otherwise intelligent people take |
> _o__)                                    it seriously.” —Douglas Adams |
> Ben Finney
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#28391

On Mon, 03 Sep 2012 23:19:51 -0400, Dennis Lee Bieber wrote:

> On 04 Sep 2012 01:13:09 GMT, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> declaimed the following in
> gmane.comp.python.general:
> 
> 
> 
>> Why does the open builtin need this added complexity? Why not just call
>> os.open directly? Or for more complex openers, just call the opener
>> directly?
>>
> 	Because os.open() returns a low-level file descriptor, not a 
> Python file object?

Good point.

But you can wrap the call to os.open, as you mention below. The only 
complication is that you have to give the mode twice, converting between 
low-level O_* integer modes and high-level string modes:

a = os.open('/tmp/foo', os.O_WRONLY | os.O_CREAT)
b = os.fdopen(a, 'w')


But to some degree, you still have to do that with the opener argument, 
at least in your own head.


>> What is the rationale for complicating open instead of telling people
>> to just call their opener directly?
> 
> 	To avoid the new syntax would mean coding the example as
> 
> 	f = os.fdopen(os.open("newfile", flags | os.O_EXCL), "w")
> 
> which does NOT look any cleaner to me... 

Well, I don't know about that. Once you start messing about with low-
level O_* flags, it's never going to exactly be clean no matter what you 
do. But I think a one-liner like the above *is* cleaner than a three-
liner like the original:

def opener(file, flags):
    return os.open(file, flags | os.O_EXCL)

open("newfile", "w", opener=opener)

although I accept that this is a matter of personal taste.

Particularly if the opener is defined far away from where you eventually 
use it. A lambda is arguably better from that perspective:

open("newfile", "w", 
     opener=lambda file, flags: os.open(file, flags | os.O_EXCL)
     )

but none of these solutions are exactly neat or clean. You still have to 
mentally translate between string modes and int modes, and make sure 
you're not passing the wrong mode:

py> open('junk', 'w').write('hello world')
11
py> open('junk', 'r', opener=lambda file, flags: os.open(file, flags | 
os.O_TRUNC)).read()  # oops
''

so it's not exactly a high-level interface.

In my opinion, a cleaner, more Pythonic interface would be either:

* allow built-in open to take numeric modes:

  open(file, os.O_CREAT | os.O_WRONLY | os.O_EXCL)

* or even more Pythonic, expose those numeric modes using strings:

  open(file, 'wx')


That's not as general as an opener, but it covers the common use-case and 
for everything else, write a helper function.


> Especially not if "opener" is to be used in more than one location.

The usual idiom for fixing the "used more than once" is "write a helper", 
not "add a callback function to a builtin" :)


> Furthermore, using "opener" could
> allow for a localized change to affect all open statements in the module
> -- change file path, open for string I/O rather than file I/O, etc.


A common idiom for that is to shadow open in the module, like this:

_open = open
def open(file, *args):
    file = file.lowercase()
    return _open(file, *args)



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#28392

Steven D'Aprano <steve+comp.lang.python@pearwood.info> writes:

> On Mon, 03 Sep 2012 23:19:51 -0400, Dennis Lee Bieber wrote:
> > 	f = os.fdopen(os.open("newfile", flags | os.O_EXCL), "w")
> > 
> > which does NOT look any cleaner to me... 
>
> Well, I don't know about that. Once you start messing about with low-
> level O_* flags, it's never going to exactly be clean no matter what you 
> do. But I think a one-liner like the above *is* cleaner than a three-
> liner like the original:
>
> def opener(file, flags):
>     return os.open(file, flags | os.O_EXCL)
>
> open("newfile", "w", opener=opener)
>
> although I accept that this is a matter of personal taste.

If the opener has an unhelpful name like ‘opener’, yes.

But if it's named as any function should be named – to say what it does
that's special – then I think the result would be much clearer::

    outfile = open("newfile", "w", opener=open_exclusive)

> Particularly if the opener is defined far away from where you
> eventually use it.

Another good reason to name helper functions descriptively.

> * or even more Pythonic, expose those numeric modes using strings:
>
>   open(file, 'wx')

Which is, indeed, another improvement in Python 3.3 – the ‘x’ mode for
‘open’ <URL:http://docs.python.org/dev/library/functions.html#open>.

-- 
 \        “The greatest tragedy in mankind's entire history may be the |
  `\       hijacking of morality by religion.” —Arthur C. Clarke, 1991 |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#28398

On 04.09.12 04:13, Steven D'Aprano wrote:
> Why does the open builtin need this added complexity? Why not just call
> os.open directly? Or for more complex openers, just call the opener
> directly?
>
> What is the rationale for complicating open instead of telling people to
> just call their opener directly?

See http://bugs.python.org/issue12797.

[toc] | [prev] | [next] | [standalone]

#28421

On 9/4/2012 8:58 AM, Serhiy Storchaka wrote:
> On 04.09.12 04:13, Steven D'Aprano wrote:
>> Why does the open builtin need this added complexity? Why not just call
>> os.open directly? Or for more complex openers, just call the opener
>> directly?
>>
>> What is the rationale for complicating open instead of telling people to
>> just call their opener directly?
>
> See http://bugs.python.org/issue12797.

io.open depends on a function the returns an open file descriptor. 
opener exposes that dependency so it can be replaced. (Obviously, one 
could go crazily overboard with this idea.) I believe this is a simple 
form of dependency injection, though it might be hard to discern from 
the Java-inspired verbiage of the Wikipedia article. Part of the 
rationale in the issue is to future-proof io.open from any future needs 
for alternate fd fetching. It could also be used to decouple a test of 
io.open from os.open

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#28429

On Wed, Sep 5, 2012 at 5:16 AM, Terry Reedy <tjreedy@udel.edu> wrote:
> io.open depends on a function the returns an open file descriptor. opener
> exposes that dependency so it can be replaced.

I skimmed the bug report comments but didn't find an answer to this:
Why not just monkey-patch? When a module function calls on a support
function and you want to change that support function's behaviour,
isn't monkey-patching the most usual?

Several possibilities come to mind, but without knowledge of
internals, I have no idea what's actually the case.
* Patching builtins is too confusing or dangerous, and should be avoided?
* You want to narrow the scope of the patch rather than do it globally?
* Explicit is better than implicit?

It just strikes me as something where an API change may not be necessary.

ChrisA

[toc] | [prev] | [next] | [standalone]

#28434

On 9/4/2012 6:18 PM, Chris Angelico wrote:
> On Wed, Sep 5, 2012 at 5:16 AM, Terry Reedy <tjreedy@udel.edu> wrote:
>> io.open depends on a function the returns an open file descriptor. opener
>> exposes that dependency so it can be replaced.
>
> I skimmed the bug report comments but didn't find an answer to this:
> Why not just monkey-patch?

As far as I know, one can only use normal Python code to monkey patch 
modules written in Python. Even then, one can only rebind names stored 
in writable dicts -- the module dict and class attribute dicts. The 
attributes of function code objects are readonly. Replacing a code 
object is not for the faint of heart.

io.py mostly loads _io compiled from C.


-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#28546

Chris Angelico <rosuav <at> gmail.com> writes:
> 
> On Wed, Sep 5, 2012 at 5:16 AM, Terry Reedy <tjreedy <at> udel.edu> wrote:
> > io.open depends on a function the returns an open file descriptor. opener
> > exposes that dependency so it can be replaced.
> 
> I skimmed the bug report comments but didn't find an answer to this:
> Why not just monkey-patch? When a module function calls on a support
> function and you want to change that support function's behaviour,
> isn't monkey-patching the most usual?

Monkey-patching globals is not thread-safe: other threads will see your
modification, which is risky and fragile.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net

[toc] | [prev] | [next] | [standalone]

#28553

On Thu, 06 Sep 2012 00:34:56 +0000, Antoine Pitrou wrote:

> Chris Angelico <rosuav <at> gmail.com> writes:
>> 
>> On Wed, Sep 5, 2012 at 5:16 AM, Terry Reedy <tjreedy <at> udel.edu>
>> wrote:
>> > io.open depends on a function the returns an open file descriptor.
>> > opener exposes that dependency so it can be replaced.
>> 
>> I skimmed the bug report comments but didn't find an answer to this:
>> Why not just monkey-patch? When a module function calls on a support
>> function and you want to change that support function's behaviour,
>> isn't monkey-patching the most usual?
> 
> Monkey-patching globals is not thread-safe: other threads will see your
> modification, which is risky and fragile.

Isn't that assuming that you don't intend the other threads to see the 
modification?

If I have two functions in my module that call "open", and I monkey-patch 
the global (module-level) name "open" to intercept that call, I don't see 
that there is more risk of breakage just because one function is called 
from a thread.

Obviously monkey-patching the builtin module itself is much riskier, 
because it doesn't just effect code in my module, it affects *everything*.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#28363

On 03.09.12 16:29, Christian Heimes wrote:
> Am 03.09.2012 14:32, schrieb Marco:
>> Does anyone have an example of utilisation?
>
> The opener argument is a new 3.3 feature. For example you can use the
> feature to implement exclusive creation of a file to avoid symlink attacks.

Or you can use new dir_fd argument for same reason (or for performance).

for root, dirs, files, rootfd in os.fwalk(topdir):
     def opener(file, flags):
         return os.open(file, flags, dir_fd=rootfd)
     for name in files:
         with open(name, "r", opener=opener) as f:
             ...

[toc] | [prev] | [next] | [standalone]

#28399

On 03.09.12 15:32, Marco wrote:
> Does anyone have an example of utilisation?

http://bugs.python.org/issue13424

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web