Path: csiph.com!usenet.pasdenom.info!news.albasani.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Newsgroups: comp.lang.python
Date: Fri, 12 Apr 2013 02:18:17 -0700 (PDT)
In-Reply-To: <mailman.510.1365755188.3114.python-list@python.org>
Complaints-To: groups-abuse@google.com
Injection-Info: glegroupsg2000goo.googlegroups.com; posting-host=46.233.72.51; posting-account=fjrvzQoAAAAF4F9_gcVHWLVVScTZ6_Hn
References: <a2ebf26b-b616-4186-84e4-6d88cd94f267@googlegroups.com> <mailman.510.1365755188.3114.python-list@python.org>
User-Agent: G2/1.0
MIME-Version: 1.0
Subject: Re: shutil.copyfile is incomplete (truncated)
From: Rob Schneider <rmschne@gmail.com>
To: comp.lang.python@googlegroups.com
Content-Type: text/plain; charset=ISO-8859-1
Cc: python-list@python.org, Rob Schneider <rmschne@gmail.com>
Precedence: list
Message-ID: <mailman.511.1365758300.3114.python-list@python.org>
Lines: 134
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:43435

On Friday, 12 April 2013 09:26:21 UTC+1, Cameron Simpson  wrote:
> 
> | > > Question: is the size of the incomplete file a round number? (Like
> 
> | > > a multiple of a decent sized power of 2>)
> 
> [...]
> 
> | Source (correct one) is 47,970 bytes. Target after copy of 45,056
> 
> | bytes.  I've tried changing what gets written to change the file
> 
> | size. It is usually this sort of difference.
> 
> 
> 
> 45046 is exactly 11 * 4096. I'd say your I/O is using 4KB blocks,
> 
> and the last partial block (to make it up to 47970) didn't get
> 
> written (at the OS level).
> 
> 
> 
> Earlier you wrote:
> 
> | I have created a file in temp space, then use the function
> 
> | "shutil.copyfile(fn,loc+fname)" from "fn" to "loc+fname".
> 
> and:
> 
> | Yes, there is a close function call  before the copy is launched. No other writes.
> 
> | Does Python wait for file close command to complete before proceeding?
> 
> 
> 
> Please show us the exact code used to make the temp file.
> 
> 
> 
> I would guess the temp file has not been closed (or flushed) before
> 
> the call to copyfile.
> 
> 
> 
> If you're copying data to a tempfile, it will only have complete
> 
> buffers (i.e. multiples of 4096 bytes) in it until the final flush
> 
> or close.
> 
> 
> 
> So I'm imagining something like:
> 
> 
> 
>   tfp = open(tempfilename, "w")
> 
>   ... lots of tfp.write() ...
> 
>   shutil.copyfile(tempfilename, newfilename)
> 
> 
> 
> Note above no flush or close of tfp. So the final incomplete I/O
> 
> buffer is still in Python's memory; it hasn't been actually written
> 
> to the temp file because the buffer has not been filled, and the file
> 
> has not been closed.
> 
> 
> 
> Anyway, can you show us the relevant bits of code involved?
> 
> 
> 
> Cheers,
> 
> -- 
> 
> Cameron Simpson <cs@zip.com.au>
> 
> 
> 
> Processes are like potatoes.    - NCR device driver manual

Thanks for the observation. 

Code (simplified but results in same flaw) (which a close, far as I can tell).

def CreateSpeakerList1():
    import shutil
    import filecmp
    import os.path 

    t=get_template('speaker_list.html')
    fn=TEMP_DIR+SOC_SPEAKER_LIST
    fn=tempfile.gettempdir()+"/"+SOC_SPEAKER_LIST
    f=open(fn,'w')
    speaker_list=Speaker.objects.order_by('status__order','targetmtg__date')
    print "    Creating " + SOC_SPEAKER_LIST +  " ..."
    html=(smart_str(t.render(Context(
        {
        'css_include_file':CSS_INCLUDE_FILE,
        'css_link':False,
        'title': ORG_NAME+" Speaker List",
        'speaker_list': speaker_list,
        }))))
    f.write(html)
    f.close
    print "    Wrote "+fn
    shutil.copyfile(fn,SOC_GENERAL_OUTPUT_FOLDER+SOC_SPEAKER_LIST)
    print "Filecompare :",filecmp.cmp(fn,SOC_GENERAL_OUTPUT_FOLDER+SOC_SPEAKER_LIST)
    print "Statinfo    :"+fn+":\n", os.stat(fn)
    print "Statinfo    :"+SOC_GENERAL_OUTPUT_FOLDER+SOC_SPEAKER_LIST+"\n", os.stat(SOC_GENERAL_OUTPUT_FOLDER+SOC_SPEAKER_LIST)
    return

Output on latest run:

    Creating speakers.htm ...
    Wrote /var/folders/p_/n5lktj2n0r938_46jyqb52g40000gn/T/speakers.htm
Filecompare : True
Statinfo    :/var/folders/p_/n5lktj2n0r938_46jyqb52g40000gn/T/speakers.htm:
posix.stat_result(st_mode=33188, st_ino=32332374, st_dev=16777218L, st_nlink=1, st_uid=501, st_gid=20, st_size=45056, st_atime=1365758139, st_mtime=1365758139, st_ctime=1365758139)
Statinfo    :/Users/rmschne/Documents/ScottishOilClub/Output/speakers.htm
posix.stat_result(st_mode=33188, st_ino=32143886, st_dev=16777218L, st_nlink=1, st_uid=501, st_gid=20, st_size=45056, st_atime=1365758029, st_mtime=1365758139, st_ctime=1365758139)