Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #6131 > unrolled thread

File access denied after subprocess completion on Windows platform

Back to article view | Back to comp.lang.python

Contents

  File access denied after subprocess completion on Windows platform Claudiu Nicolaie CISMARU <claudiu@virtuamagic.com> - 2011-05-24 13:01 +0300
    Re: File access denied after subprocess completion on Windows platform Claudiu Nicolaie CISMARU <claudiu@virtuamagic.com> - 2011-05-24 20:17 +0300
    Re: File access denied after subprocess completion on Windows platform Claudiu Nicolaie CISMARU <claudiu@virtuamagic.com> - 2011-05-24 23:18 +0300
    Re: File access denied after subprocess completion on Windows platform Terry Reedy <tjreedy@udel.edu> - 2011-05-24 18:03 -0400
    Re: File access denied after subprocess completion on Windows platform Tim Golden <mail@timgolden.me.uk> - 2011-05-25 08:17 +0100
    Re: File access denied after subprocess completion on Windows platform Claudiu Nicolaie CISMARU <claudiu@virtuamagic.com> - 2011-05-25 11:20 +0300

#6131 — File access denied after subprocess completion on Windows platform

[Multipart message — attachments visible in raw view] — view raw

Hello,

I have a program that uses pyside for an QT interface and a thread that 
downloads a lot of files. The thread is created with QThread object. But 
my problem I don't think it's QT related.

The thread retrieves with pycurl a file that contains a list of files 
and start to downloads them. The downloading is done as following:
- instantiate a Curl object
- open the file on local filesystem for write in binary mode (in a try 
block), with the name suffixed with .part.
- pass the description to the curl object for save.
- curl retrieve and save it. It has also a callback function that 
updates the interface, sending a QT signal to the interface.
(1) - use os.rename to rename the file with .part sufix to the final 
file.

On my interface I have 3 buttons. One of the buttons runs an .exe file. 
One button closes the interface and one is deactivated.

On the button that runs the exe I have a callback function that uses 
subprocess.Popen (for not waiting) for running a program (.exe) and 
returns. For now I configured to run calc.exe. The callback is not 
defined inside the downloader thread. It's defined globally (nor in 
QMainWindow object).

The problem appears when I close the called program (in our case 
calc.exe). The (1) part (the call of os.rename) raise an exception:

<type 'exceptions.WindowsError'>
(32, 'The process cannot access the file because it is being used by 
another process')
[Error 32] The process cannot access the file because it is being used 
by another process

Question is why? And how to avoid this issue? The same program on Linux 
works very fine (that's because Linux doesn't has this violation 
access)! If I remove (1) part the program works fine. Somehow after 
closing the spawned process (calc.exe - you see, it has nothing to do 
with a open file somewhere else) the thread losses the acces to the 
current opened file by itself.

-- 
  Claudiu Nicolaie CISMARU
  GNU GPG Key: http://claudiu.targujiu.net/key.gpg
  T: +40 755 135455
  E: claudiu@virtuamagic.com, claudiu.cismaru@gmail.com

[toc] | [next] | [standalone]

#6160

[Multipart message — attachments visible in raw view] — view raw

I'm quoting a message that I received on personal address and wasn't 
sent to list:
> 
> try adding argument close_fds=True to subprocess.Popen
> 
> harish
> 

And Tim's message:
> It's not quite clear from your description above whether you
> can be sure that the called subprocess has closed all its handles
> by the time the os.rename runs.

Seems that close_fds did the trick. Anyway, I read that description on 
the documentation last night but I think I was so tired that I 
understood that in Windows has no effect... :)

Thank you, all.
-- 
  Claudiu Nicolaie CISMARU
  GNU GPG Key: http://claudiu.targujiu.net/key.gpg
  T: +40 755 135455
  E: claudiu@virtuamagic.com, claudiu.cismaru@gmail.com

[toc] | [prev] | [next] | [standalone]

#6171

[Multipart message — attachments visible in raw view] — view raw

> Seems that close_fds did the trick. Anyway, I read that description on 
> the documentation last night but I think I was so tired that I 
> understood that in Windows has no effect... :)

Now. There is one more issue. Seems that on faster computers and/or 
Windows 7 (the Win32 thing I have tested on a HVM Xen machine with 
Windows XP) the os.rename is too fast after fp.close() and generates the 
same Exception. The code follows:

curl.close()
fp.close()
os.rename(tfile, actualfile)

Where, tfile is the .part file, actual file is the real destination, fp 
was opened with open(..., "wb") and the descriptor passed to curl.

I have solved the issue with self.msleep(10) - msleep is a method of 
QThread. But I don't think it's an elegant and normal solution. Did 
fp.close() is delayed, or? I mean, I don't want to rely on a "sleep" in 
order to workaround the access issue.

On this issue there is no more process spawn, nothing, just the 
downloader thread and the main window. And the access denied appears at 
random time.

-- 
  Claudiu Nicolaie CISMARU
  GNU GPG Key: http://claudiu.targujiu.net/key.gpg
  T: +40 755 135455
  E: claudiu@virtuamagic.com, claudiu.cismaru@gmail.com

[toc] | [prev] | [next] | [standalone]

#6179

On 5/24/2011 4:18 PM, Claudiu Nicolaie CISMARU wrote:
>> Seems that close_fds did the trick. Anyway, I read that description on
>> the documentation last night but I think I was so tired that I
>> understood that in Windows has no effect... :)
>
> Now. There is one more issue. Seems that on faster computers and/or
> Windows 7 (the Win32 thing I have tested on a HVM Xen machine with
> Windows XP) the os.rename is too fast after fp.close() and generates the
> same Exception. The code follows:
>
> curl.close()
> fp.close()
> os.rename(tfile, actualfile)
>
> Where, tfile is the .part file, actual file is the real destination, fp
> was opened with open(..., "wb") and the descriptor passed to curl.
>
> I have solved the issue with self.msleep(10) - msleep is a method of
> QThread. But I don't think it's an elegant and normal solution. Did
> fp.close() is delayed, or? I mean, I don't want to rely on a "sleep" in
> order to workaround the access issue.
>
> On this issue there is no more process spawn, nothing, just the
> downloader thread and the main window. And the access denied appears at
> random time.

I would go with what works. In my experience, mysterious and seemingly 
buggy error messages, including Access Denied are not unusual on Windows.

-- 
Terry Jan Reedy

[toc] | [prev] | [next] | [standalone]

#6203

On 24/05/2011 21:18, Claudiu Nicolaie CISMARU wrote:
> Now. There is one more issue. Seems that on faster computers and/or
> Windows 7 (the Win32 thing I have tested on a HVM Xen machine with
> Windows XP) the os.rename is too fast after fp.close() and generates the
> same Exception. The code follows:
>
> curl.close()
> fp.close()
> os.rename(tfile, actualfile)
>
> Where, tfile is the .part file, actual file is the real destination, fp
> was opened with open(..., "wb") and the descriptor passed to curl.
>
> I have solved the issue with self.msleep(10) - msleep is a method of
> QThread. But I don't think it's an elegant and normal solution. Did
> fp.close() is delayed, or? I mean, I don't want to rely on a "sleep" in
> order to workaround the access issue.

There used to be a problem with subprocess fds being held by
a traceback. IIRC, the problem could be triggered by having
an except clause around a subprocess call within which something 
attempted to, eg,
remove one of the affected files. I'm sorry if that's a bit
of a woolly description but if you think this might be
biting you I'll dive in and look at the code. What version
of Python are you using?

(That said, the fact that the behaviour varies between faster
and slower computers makes that cause unlikely. Maybe we're
back to looking at virus checkers and the like...)

TJG

[toc] | [prev] | [next] | [standalone]

#6209

[Multipart message — attachments visible in raw view] — view raw

> There used to be a problem with subprocess fds being held by
> a traceback. IIRC, the problem could be triggered by having
> an except clause around a subprocess call within which something 
> attempted to, eg,
> remove one of the affected files.

I have no subprocess call.. in this last issue. And I didn't triggered 
one (disabled the Button that runs the subprocess).

        try:
            for line in t.splitlines():

                [...]

                ret = self.downloadFileToDisk(filename, do_rename)

                if not ret:
                    print "DEBUG: Problema la download"
                    raise Exception()

(1)     except Exception as inst:
            print type(inst)
            print inst.args

            self.updateText.emit("EROARE: Eroare la descarcare")
            self.updateStatusBar.emit("EROARE: Eroare la descaracare 
fisiere")
            return

Where downloadFileToDisk():

    def downloadFileToDisk(self, filename, final_rename=True):
        dfilename = os.path.join(saveBasePATH, filename)
        sfilename = dfilename + ".part"

        dfolder = os.path.dirname(sfilename)
        if dfolder != "":
            if not os.path.isdir(dfolder):
                os.makedirs(dfolder)

        try:
            fp = open(sfilename, "wb")
        except:
            return False

        curl = pycurl.Curl()

        curl.setopt(pycurl.URL, baseUpdateURL + "/client/" + filename)
        curl.setopt(pycurl.CONNECTTIMEOUT, 30)
        curl.setopt(pycurl.NOPROGRESS, 0)
        curl.setopt(pycurl.FOLLOWLOCATION, 1)
        curl.setopt(pycurl.MAXREDIRS, 5)
        curl.setopt(pycurl.PROGRESSFUNCTION, self.updateFileProgress)
        curl.setopt(pycurl.WRITEDATA, fp)
        curl.setopt(pycurl.BUFFERSIZE, 4194304)
        curl.setopt(pycurl.NOSIGNAL, 1)
        curl.perform()

        retcode = curl.getinfo(pycurl.HTTP_CODE)

        curl.close()

        fp.close()

        if retcode != 200:
(2)         os.unlink(sfilename)
            return False

        self.msleep(10)

        if final_rename:
            os.rename(sfilename, dfilename)

        return True

Without self.msleep(10), (1) catches WindowsError: file access ... blah 
blah. Maybe at (2) can be an access violation, but it wasn't triggered 
yet in tests. I will move the sleep after fp.close(). Yes, I know that 
what it've done with raise Exception() is UGLY, since this program it's 
more a quick hack solution to a problem :). Anyway the Exception that is 
catched is not rised by my code (it's a WindowsError). 

> I'm sorry if that's a bit
> of a woolly description but if you think this might be
> biting you I'll dive in and look at the code. What version
> of Python are you using?

Last 2.7.x. Well, if os.rename is instead a subprocess call, then it's 
subprocess based. I'm new to Python but 99% I think it's a system call 
:)

> (That said, the fact that the behaviour varies between faster
> and slower computers makes that cause unlikely. Maybe we're
> back to looking at virus checkers and the like...)

On that virtual machine there is no virus checker. On the faster machine 
I disabled and closed the antivirus.

-- 
  Claudiu Nicolaie CISMARU
  GNU GPG Key: http://claudiu.targujiu.net/key.gpg
  T: +40 755 135455
  E: claudiu@virtuamagic.com, claudiu.cismaru@gmail.com

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web