Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #43435

Re: shutil.copyfile is incomplete (truncated)

Newsgroups comp.lang.python
Date 2013-04-12 02:18 -0700
References <a2ebf26b-b616-4186-84e4-6d88cd94f267@googlegroups.com> <mailman.510.1365755188.3114.python-list@python.org>
Subject Re: shutil.copyfile is incomplete (truncated)
From Rob Schneider <rmschne@gmail.com>
Message-ID <mailman.511.1365758300.3114.python-list@python.org> (permalink)

Show all headers | View raw

On Friday, 12 April 2013 09:26:21 UTC+1, Cameron Simpson  wrote:
> 
> | > > Question: is the size of the incomplete file a round number? (Like
> 
> | > > a multiple of a decent sized power of 2>)
> 
> [...]
> 
> | Source (correct one) is 47,970 bytes. Target after copy of 45,056
> 
> | bytes.  I've tried changing what gets written to change the file
> 
> | size. It is usually this sort of difference.
> 
> 
> 
> 45046 is exactly 11 * 4096. I'd say your I/O is using 4KB blocks,
> 
> and the last partial block (to make it up to 47970) didn't get
> 
> written (at the OS level).
> 
> 
> 
> Earlier you wrote:
> 
> | I have created a file in temp space, then use the function
> 
> | "shutil.copyfile(fn,loc+fname)" from "fn" to "loc+fname".
> 
> and:
> 
> | Yes, there is a close function call  before the copy is launched. No other writes.
> 
> | Does Python wait for file close command to complete before proceeding?
> 
> 
> 
> Please show us the exact code used to make the temp file.
> 
> 
> 
> I would guess the temp file has not been closed (or flushed) before
> 
> the call to copyfile.
> 
> 
> 
> If you're copying data to a tempfile, it will only have complete
> 
> buffers (i.e. multiples of 4096 bytes) in it until the final flush
> 
> or close.
> 
> 
> 
> So I'm imagining something like:
> 
> 
> 
>   tfp = open(tempfilename, "w")
> 
>   ... lots of tfp.write() ...
> 
>   shutil.copyfile(tempfilename, newfilename)
> 
> 
> 
> Note above no flush or close of tfp. So the final incomplete I/O
> 
> buffer is still in Python's memory; it hasn't been actually written
> 
> to the temp file because the buffer has not been filled, and the file
> 
> has not been closed.
> 
> 
> 
> Anyway, can you show us the relevant bits of code involved?
> 
> 
> 
> Cheers,
> 
> -- 
> 
> Cameron Simpson <cs@zip.com.au>
> 
> 
> 
> Processes are like potatoes.    - NCR device driver manual

Thanks for the observation. 

Code (simplified but results in same flaw) (which a close, far as I can tell).

def CreateSpeakerList1():
    import shutil
    import filecmp
    import os.path 

    t=get_template('speaker_list.html')
    fn=TEMP_DIR+SOC_SPEAKER_LIST
    fn=tempfile.gettempdir()+"/"+SOC_SPEAKER_LIST
    f=open(fn,'w')
    speaker_list=Speaker.objects.order_by('status__order','targetmtg__date')
    print "    Creating " + SOC_SPEAKER_LIST +  " ..."
    html=(smart_str(t.render(Context(
        {
        'css_include_file':CSS_INCLUDE_FILE,
        'css_link':False,
        'title': ORG_NAME+" Speaker List",
        'speaker_list': speaker_list,
        }))))
    f.write(html)
    f.close
    print "    Wrote "+fn
    shutil.copyfile(fn,SOC_GENERAL_OUTPUT_FOLDER+SOC_SPEAKER_LIST)
    print "Filecompare :",filecmp.cmp(fn,SOC_GENERAL_OUTPUT_FOLDER+SOC_SPEAKER_LIST)
    print "Statinfo    :"+fn+":\n", os.stat(fn)
    print "Statinfo    :"+SOC_GENERAL_OUTPUT_FOLDER+SOC_SPEAKER_LIST+"\n", os.stat(SOC_GENERAL_OUTPUT_FOLDER+SOC_SPEAKER_LIST)
    return

Output on latest run:

    Creating speakers.htm ...
    Wrote /var/folders/p_/n5lktj2n0r938_46jyqb52g40000gn/T/speakers.htm
Filecompare : True
Statinfo    :/var/folders/p_/n5lktj2n0r938_46jyqb52g40000gn/T/speakers.htm:
posix.stat_result(st_mode=33188, st_ino=32332374, st_dev=16777218L, st_nlink=1, st_uid=501, st_gid=20, st_size=45056, st_atime=1365758139, st_mtime=1365758139, st_ctime=1365758139)
Statinfo    :/Users/rmschne/Documents/ScottishOilClub/Output/speakers.htm
posix.stat_result(st_mode=33188, st_ino=32143886, st_dev=16777218L, st_nlink=1, st_uid=501, st_gid=20, st_size=45056, st_atime=1365758029, st_mtime=1365758139, st_ctime=1365758139)

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-11 11:12 -0700
  Re: shutil.copyfile is incomplete (truncated) Neil Cerutti <neilc@norwich.edu> - 2013-04-11 18:53 +0000
    Re: shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-11 12:07 -0700
      Re: shutil.copyfile is incomplete (truncated) Neil Cerutti <neilc@norwich.edu> - 2013-04-11 19:55 +0000
        Re: shutil.copyfile is incomplete (truncated) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-12 00:06 +0000
          Re: shutil.copyfile is incomplete (truncated) Cameron Simpson <cs@zip.com.au> - 2013-04-12 11:15 +1000
          Re: shutil.copyfile is incomplete (truncated) Ned Deily <nad@acm.org> - 2013-04-11 18:33 -0700
            Re: shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-11 23:32 -0700
              Re: shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-11 23:53 -0700
                Re: shutil.copyfile is incomplete (truncated) Ned Deily <nad@acm.org> - 2013-04-12 00:53 -0700
              Re: shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-11 23:53 -0700
              Re: shutil.copyfile is incomplete (truncated) Cameron Simpson <cs@zip.com.au> - 2013-04-12 18:26 +1000
                Re: shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-12 02:18 -0700
                Re: shutil.copyfile is incomplete (truncated) Chris Angelico <rosuav@gmail.com> - 2013-04-12 19:22 +1000
                Re: shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-12 05:07 -0700
                Re: shutil.copyfile is incomplete (truncated) Mark Lawrence <breamoreboy@yahoo.co.uk> - 2013-04-12 13:18 +0100
                Re: shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-12 05:07 -0700
                Re: shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-12 02:18 -0700
                Re: shutil.copyfile is incomplete (truncated) Roy Smith <roy@panix.com> - 2013-04-12 10:57 -0400
                Re: shutil.copyfile is incomplete (truncated) Roy Smith <roy@panix.com> - 2013-04-12 10:54 -0400
            Re: shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-11 23:32 -0700
              Re: shutil.copyfile is incomplete (truncated) Roy Smith <roy@panix.com> - 2013-04-12 10:47 -0400
          Re: shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-11 23:27 -0700
          Re: shutil.copyfile is incomplete (truncated) Roy Smith <roy@panix.com> - 2013-04-12 10:51 -0400
          Re: shutil.copyfile is incomplete (truncated) 88888 Dihedral <dihedral88888@googlemail.com> - 2013-04-12 08:49 -0700
          Re: shutil.copyfile is incomplete (truncated) Nobody <nobody@nowhere.com> - 2013-04-13 03:33 +0100
            Re: shutil.copyfile is incomplete (truncated) Chris Angelico <rosuav@gmail.com> - 2013-04-13 13:05 +1000
            [OT] Lying hard drives [was Re: shutil.copyfile is incomplete (truncated)] Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-04-13 03:17 +0000
              Re: [OT] Lying hard drives [was Re: shutil.copyfile is incomplete (truncated)] Chris Angelico <rosuav@gmail.com> - 2013-04-13 13:43 +1000
        Re: shutil.copyfile is incomplete (truncated) Rob Schneider <rmschne@gmail.com> - 2013-04-11 23:25 -0700
          Re: shutil.copyfile is incomplete (truncated) Chris Angelico <rosuav@gmail.com> - 2013-04-12 17:32 +1000
          Re: shutil.copyfile is incomplete (truncated) Terry Jan Reedy <tjreedy@udel.edu> - 2013-04-12 10:48 -0400

csiph-web