Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #22488 > unrolled thread

No os.copy()? Why not?

Back to article view | Back to comp.lang.python

Contents

  No os.copy()?  Why not? John Ladasky <john_ladasky@sbcglobal.net> - 2012-03-28 13:12 -0700
    Re: No os.copy()? Why not? alex23 <wuwei23@gmail.com> - 2012-03-28 21:50 -0700
      Re: No os.copy()? Why not? John Ladasky <john_ladasky@sbcglobal.net> - 2012-03-30 02:25 -0700
    Re: No os.copy()? Why not? Ian Kelly <ian.g.kelly@gmail.com> - 2012-04-02 14:48 -0600
    Re: No os.copy()?  Why not? HoneyMonster <someone@someplace.invalid> - 2012-04-02 21:11 +0000
      Re: No os.copy()?  Why not? Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2012-04-03 08:24 +0200
        Re: No os.copy()? Why not? John Ladasky <john_ladasky@sbcglobal.net> - 2012-04-03 02:34 -0700
          Re: No os.copy()? Why not? Thomas Rachel <nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de> - 2012-04-04 14:30 +0200
        Re: No os.copy()? Why not? Ian Kelly <ian.g.kelly@gmail.com> - 2012-04-03 10:29 -0600
    Re: No os.copy()?  Why not? D'Arcy Cain <darcy@druid.net> - 2012-04-03 15:46 -0400
      Re: No os.copy()?  Why not? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2012-04-04 05:53 +0000
        Re: No os.copy()? Why not? Chris Angelico <rosuav@gmail.com> - 2012-04-04 18:37 +1000
          Re: No os.copy()? Why not? Steve Howell <showell30@yahoo.com> - 2012-04-04 08:15 -0700
        Re: No os.copy()?  Why not? Alain Ketterlin <alain@dpt-info.u-strasbg.fr> - 2012-04-04 11:22 +0200
          Re: No os.copy()?  Why not? Roy Smith <roy@panix.com> - 2012-04-04 08:14 -0400
            Re: No os.copy()?  Why not? Nobody <nobody@nowhere.com> - 2012-04-04 20:37 +0100
        Re: No os.copy()?  Why not? Roy Smith <roy@panix.com> - 2012-04-04 08:08 -0400
          Re: No os.copy()? Why not? Chris Angelico <rosuav@gmail.com> - 2012-04-04 22:17 +1000
    Re: No os.copy()?  Why not? Tycho Andersen <tycho@tycho.ws> - 2012-04-03 15:10 -0500
    Re: Re: No os.copy()?  Why not? Evan Driscoll <driscoll@cs.wisc.edu> - 2012-04-03 15:21 -0500

#22488 — No os.copy()? Why not?

I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
Since the equivalents of "rm" and "mkdir" are in the os module, I
figured I look there.  I haven't found anything in the documentation.
I am also looking through the Python source code in os.py and its
child, posixfile.py.

Any help?  Thanks.

[toc] | [next] | [standalone]

#22489 — Re: No os.copy()? Why not?

On Mar 29, 6:12 am, John Ladasky <john_lada...@sbcglobal.net> wrote:
> I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
> Any help?  Thanks.

Try the shutil module: http://docs.python.org/library/shutil.html

[toc] | [prev] | [next] | [standalone]

#22510 — Re: No os.copy()? Why not?

On Mar 28, 9:50 pm, alex23 <wuwe...@gmail.com> wrote:
> On Mar 29, 6:12 am, John Ladasky <john_lada...@sbcglobal.net> wrote:
>
> > I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
> > Any help?  Thanks.
>
> Try the shutil module:http://docs.python.org/library/shutil.html

Many thanks!  That's what I was looking for.

[toc] | [prev] | [next] | [standalone]

#22539 — Re: No os.copy()? Why not?

On Wed, Mar 28, 2012 at 2:12 PM, John Ladasky
<john_ladasky@sbcglobal.net> wrote:
> I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
> Since the equivalents of "rm" and "mkdir" are in the os module, I
> figured I look there.  I haven't found anything in the documentation.
> I am also looking through the Python source code in os.py and its
> child, posixfile.py.
>
> Any help?  Thanks.

The os module wraps system calls, not shell commands.  You want the
shutil module, not the os module.

[toc] | [prev] | [next] | [standalone]

#22541

On Wed, 28 Mar 2012 13:12:30 -0700, John Ladasky wrote:

> I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
> Since the equivalents of "rm" and "mkdir" are in the os module, I
> figured I look there.  I haven't found anything in the documentation.
> I am also looking through the Python source code in os.py and its child,
> posixfile.py.
> 
> Any help?  Thanks.

One way:
import os

os.system ("cp src sink")

[toc] | [prev] | [next] | [standalone]

#22574

Am 02.04.2012 23:11 schrieb HoneyMonster:

> One way:
> import os
>
> os.system ("cp src sink")

Yes. The worst way you could imagine.

Why not the much much better

from subprocess
subprocess.call(['cp', 'src', 'sink'])

?

Then you can call it with (really) arbitrary file names:


def call_cp(from, to):
     from subprocess
     subprocess.call(['cp', '--', from, to])

Try that with os.system() and from="That's my file"...


Thomas

[toc] | [prev] | [next] | [standalone]

#22580 — Re: No os.copy()? Why not?

I use subprocess.call() for quite a few other things.

I just figured that I should use the tidier modules whenever I can.

[toc] | [prev] | [next] | [standalone]

#22664 — Re: No os.copy()? Why not?

Am 03.04.2012 11:34 schrieb John Ladasky:
> I use subprocess.call() for quite a few other things.
>
> I just figured that I should use the tidier modules whenever I can.

Of course. I only wanted to point out that os.system() is an even worse 
approach. shutils.copy() is by far better, of course.

[toc] | [prev] | [next] | [standalone]

#22615 — Re: No os.copy()? Why not?

On Tue, Apr 3, 2012 at 12:24 AM, Thomas Rachel
<nutznetz-0c1b6768-bfa9-48d5-a470-7603bd3aa915@spamschutz.glglgl.de>
wrote:
> Am 02.04.2012 23:11 schrieb HoneyMonster:
>
>
>> One way:
>> import os
>>
>> os.system ("cp src sink")
>
>
> Yes. The worst way you could imagine.
>
> Why not the much much better
>
> from subprocess
> subprocess.call(['cp', 'src', 'sink'])


In any case, either of these approaches will only work in UNIX,
whereas shutil is cross-platform.

[toc] | [prev] | [next] | [standalone]

#22627

On 03/28/12 16:12, John Ladasky wrote:
> I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
> Since the equivalents of "rm" and "mkdir" are in the os module, I
> figured I look there.  I haven't found anything in the documentation.
> I am also looking through the Python source code in os.py and its
> child, posixfile.py.

cp is not a system command, it's a shell command.  Why not just use the
incredibly simple and portable

   >>>open("outfile", "w").write(open("infile").read())

put it into a method if you find that too much to type:

def cp(infile, outfile):
   open(outfile, "w").write(open(infile).read())

-- 
D'Arcy J.M. Cain <darcy@druid.net>         |  Democracy is three wolves
http://www.druid.net/darcy/                |  and a sheep voting on
+1 416 425 1212     (DoD#0082)    (eNTP)   |  what's for dinner.
IM: darcy@Vex.Net

[toc] | [prev] | [next] | [standalone]

#22649

On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:

> On 03/28/12 16:12, John Ladasky wrote:
>> I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
>> Since the equivalents of "rm" and "mkdir" are in the os module, I
>> figured I look there.  I haven't found anything in the documentation. I
>> am also looking through the Python source code in os.py and its child,
>> posixfile.py.
> 
> cp is not a system command, it's a shell command.  Why not just use the
> incredibly simple and portable
> 
>    >>>open("outfile", "w").write(open("infile").read())
> 
> put it into a method if you find that too much to type:
> 
> def cp(infile, outfile):
>    open(outfile, "w").write(open(infile).read())


Because your cp doesn't copy the FILE, it copies the file's CONTENTS, 
which are not the same thing.

Consider:

* permissions
* access times
* file ownership
* other metadata
* alternate streams and/or resource fork, on platforms that support them
* sparse files


By the time you finish supporting the concept of copying the file itself, 
rather than merely its content, you will have something similar to the 
shutil.copy command -- only less tested.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#22653 — Re: No os.copy()? Why not?

On Wed, Apr 4, 2012 at 3:53 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:
>
>> def cp(infile, outfile):
>>    open(outfile, "w").write(open(infile).read())
>
> Because your cp doesn't copy the FILE, it copies the file's CONTENTS,
> which are not the same thing.

And, as a subtle point: This method can't create the file "at size". I
don't know how it'll end up allocating space, but certainly there's no
opportunity to announce to the OS at file open/create time "please
allocate X bytes for this file". That may be an utterly trivial point,
or a crucially vital one.

ChrisA

[toc] | [prev] | [next] | [standalone]

#22675 — Re: No os.copy()? Why not?

On Apr 4, 1:37 am, Chris Angelico <ros...@gmail.com> wrote:
> On Wed, Apr 4, 2012 at 3:53 PM, Steven D'Aprano
>
> <steve+comp.lang.pyt...@pearwood.info> wrote:
> > On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:
>
> >> def cp(infile, outfile):
> >>    open(outfile, "w").write(open(infile).read())
>
> > Because your cp doesn't copy the FILE, it copies the file's CONTENTS,
> > which are not the same thing.
>
> And, as a subtle point: This method can't create the file "at size". I
> don't know how it'll end up allocating space, but certainly there's no
> opportunity to announce to the OS at file open/create time "please
> allocate X bytes for this file". That may be an utterly trivial point,
> or a crucially vital one.
>
> ChrisA

FWIW shutil.py doesn't do anything particularly fancy with respect to
creating files "at size", unless I'm missing something:

     http://hg.python.org/cpython/file/2.7/Lib/shutil.py

Only one level away from copyfile, you have copyfileobj, which is a
read/write loop:

    46 def copyfileobj(fsrc, fdst, length=16*1024):
    47     """copy data from file-like object fsrc to file-like object
fdst"""
    48     while 1:
    49         buf = fsrc.read(length)
    50         if not buf:
    51             break
    52         fdst.write(buf)

...and that gets called by copyfile, which only does a little bit of
"os"-related stuff:

    66 def copyfile(src, dst):
    67     """Copy data from src to dst"""
    68     if _samefile(src, dst):
    69         raise Error("`%s` and `%s` are the same file" % (src,
dst))
    70
    71     for fn in [src, dst]:
    72         try:
    73             st = os.stat(fn)
    74         except OSError:
    75             # File most likely does not exist
    76             pass
    77         else:
    78             # XXX What about other special files? (sockets,
devices...)
    79             if stat.S_ISFIFO(st.st_mode):
    80                 raise SpecialFileError("`%s` is a named pipe" %
fn)
    81
    82     with open(src, 'rb') as fsrc:
    83         with open(dst, 'wb') as fdst:
    84             copyfileobj(fsrc, fdst)

The "value add" vs. a simple read/write loop depends on whether you
want OSError suppressed.  The _samefile guard is nice to have, but
probably unnecessary for many apps.

I'm sure shutil.copyfile() makes perfect sense for most use cases, and
it's nice that you can see what it does under the covers pretty
easily, but it's not rocket science.

[toc] | [prev] | [next] | [standalone]

#22657

Steven D'Aprano <steve+comp.lang.python@pearwood.info> writes:

> On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:
>
>> On 03/28/12 16:12, John Ladasky wrote:

>>> I'm looking for a Python (2.7) equivalent to the Unix "cp" command.

>>    >>>open("outfile", "w").write(open("infile").read())

> Because your cp doesn't copy the FILE, it copies the file's CONTENTS, 
> which are not the same thing.
> Consider:
> * permissions
> * access times
> * file ownership
> * other metadata
> * alternate streams and/or resource fork, on platforms that support them
> * sparse files
> By the time you finish supporting the concept of copying the file itself, 
> rather than merely its content, you will have something similar to the 
> shutil.copy command -- only less tested.

A minor point, but shutil.copy only "copies" contents and permissions
(no access times, etc.) You probably mean shutil.copy2.

And sparse files are really hard to reproduce, at least on Unix: on
Linux even the system's cp doesn't guarantee sparseness of the copy (the
manual mentions a "crude heuristic").

But of course shutil.copy is the best solution to mimic a raw cp.

-- Alain.

[toc] | [prev] | [next] | [standalone]

#22659

In article <87fwcj4zru.fsf@dpt-info.u-strasbg.fr>,
 Alain Ketterlin <alain@dpt-info.u-strasbg.fr> wrote:

> And sparse files are really hard to reproduce, at least on Unix: on
> Linux even the system's cp doesn't guarantee sparseness of the copy (the
> manual mentions a "crude heuristic").

I imagine the heuristic is to look for blocks of all zeros.  The problem 
is, unless you know the block size of the file system, you can only 
guess as to how many zeros in a row you need to look for.

In the old days, dump/restore used to know about sparse files.  But 
things like dump/restore really get inside the file system's kimono.  In 
today's world of SANs, WANs, and all sorts of virtual file-system-ish 
things, I would expect that's less common.

[toc] | [prev] | [next] | [standalone]

#22677

On Wed, 04 Apr 2012 08:14:18 -0400, Roy Smith wrote:

>> And sparse files are really hard to reproduce, at least on Unix: on
>> Linux even the system's cp doesn't guarantee sparseness of the copy (the
>> manual mentions a "crude heuristic").
> 
> I imagine the heuristic is to look for blocks of all zeros.

Yes. Although it's not really accurate to describe it as a "heuristic".

With --sparse=always, it will try to make the output sparse regardless of
whether the input was sparse, replacing any all-zeros block with a hole.

The default of --sparse=auto will only create a sparse file if the input
itself is sparse, i.e. if the length of the file rounded up to the nearest
block exceeds its disk usage.

Regardless of the --sparse= setting and whether the input was sparse, if
it tries to create a sparse file it will create holes wherever possible
rather than attempting to preserve the exact pattern of holes in a sparse
input file.

[toc] | [prev] | [next] | [standalone]

#22658

On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:
> > cp is not a system command, it's a shell command.  Why not just use the
> > incredibly simple and portable
> > 
> >    >>>open("outfile", "w").write(open("infile").read())

In article <4f7be1e8$0$29999$c3e8da3$5496439d@news.astraweb.com>,
 Steven D'Aprano <steve+comp.lang.python@pearwood.info> wrote:

> Because your cp doesn't copy the FILE, it copies the file's CONTENTS, 
> which are not the same thing.

Not to mention that this will read the entire contents of the file into 
memory at once.  Probably don't want to do that with 100 GB of data.

Slightly off-topic, but are there file systems these days which support 
off-line copying?  If I have a disk at the other end of a network link, 
it would be nice to tell the disk to copy a file and tell me when it's 
done.  As opposed to dragging all that data over the network just so I 
can buffer it in local memory and shove it right back out the network 
port to the same disk.  That kind of stuff used to be standard practice 
in the neanderthalic days of IBM mainframes.

[toc] | [prev] | [next] | [standalone]

#22660 — Re: No os.copy()? Why not?

On Wed, Apr 4, 2012 at 10:08 PM, Roy Smith <roy@panix.com> wrote:
> Slightly off-topic, but are there file systems these days which support
> off-line copying?  If I have a disk at the other end of a network link,
> it would be nice to tell the disk to copy a file and tell me when it's
> done.

Depends on your network protocol. One of the coolest and oldest tricks
with FTP is initiating a file transfer from one remote host to
another; I've never done it but it ought to work with localhost (ie
two sessions to the same host).

ChrisA

[toc] | [prev] | [next] | [standalone]

#22630

On Tue, Apr 03, 2012 at 03:46:31PM -0400, D'Arcy Cain wrote:
> On 03/28/12 16:12, John Ladasky wrote:
> >I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
> >Since the equivalents of "rm" and "mkdir" are in the os module, I
> >figured I look there.  I haven't found anything in the documentation.
> >I am also looking through the Python source code in os.py and its
> >child, posixfile.py.
> 
> cp is not a system command, it's a shell command.  Why not just use the
> incredibly simple and portable
> 
>   >>>open("outfile", "w").write(open("infile").read())

Note, though, that this reads the whole file into memory. As many
others have said, shutil is the most idiomatic option.

\t

[toc] | [prev] | [next] | [standalone]

#22632

On 01/-10/-28163 01:59 PM, Tycho Andersen wrote:
> Note, though, that this reads the whole file into memory. As many
> others have said, shutil is the most idiomatic option.

* most idiomatic
* clearest in terms of showing intent
* potentially fastest
* hardest to screw up (unlike concatenating file names into a
   'system' call)
* has at least a snowball's chance of persisting metadata associated
   with the file (even if shutil doesn't now, it could theoretically
   change)
* portable

There's basically nothing NOT to like about shutil, and at least one 
significant problems with all the other suggestions.

Evan

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web