Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #99629 > unrolled thread

python response slow when running external DLL

Back to article view | Back to comp.lang.python

Contents

  python response slow when running external DLL jfong@ms4.hinet.net - 2015-11-26 23:51 -0800
    Re: python response slow when running external DLL Peter Otten <__peter__@web.de> - 2015-11-27 10:18 +0100
      Re: python response slow when running external DLL jfong@ms4.hinet.net - 2015-11-27 03:14 -0800
        Re: python response slow when running external DLL Peter Otten <__peter__@web.de> - 2015-11-27 13:20 +0100
          Re: python response slow when running external DLL jfong@ms4.hinet.net - 2015-11-27 23:37 -0800
            Re: python response slow when running external DLL Peter Otten <__peter__@web.de> - 2015-11-28 11:13 +0100
              Re: python response slow when running external DLL jfong@ms4.hinet.net - 2015-11-28 18:55 -0800
                Re: python response slow when running external DLL jfong@ms4.hinet.net - 2015-11-30 17:03 -0800
                Re: python response slow when running external DLL Peter Otten <__peter__@web.de> - 2015-12-01 12:01 +0100
                  Re: python response slow when running external DLL jfong@ms4.hinet.net - 2015-12-01 19:58 -0800
            Re: python response slow when running external DLL Laura Creighton <lac@openend.se> - 2015-11-28 11:51 +0100
              Re: python response slow when running external DLL jfong@ms4.hinet.net - 2015-11-28 19:04 -0800
        Re: python response slow when running external DLL Laura Creighton <lac@openend.se> - 2015-11-27 13:49 +0100

#99629 — python response slow when running external DLL

I am new to Python. As an exercise of it, I try to port a program which was written more than 10 years ago. This program use the Borland C++ Builder as its GUI front end and a DLL does the real work(it will takes a few seconds to complete). I saw a strange phenomenon in the following codes. The "var_status.set('Download...')" statement seems was deferred and didn't show up until the external DLL job was finished and I can only saw the result of "var_status.set('Download OK')" statement which immediately follows. I try to do the DLL function in a thread(selected by using the "test" flag in codes), but it didn't help.

Can anyone tell me what's the point I was missed?

-------------------------
def download():
    global iniFilename
    if test:  global result, busy
    ini = iniFilename
    iniFilename = "c:\\$$temp.in3"
    saveIniFile()
    iniFilename = ini
    #do the rest
    var_status.set('Download...')
    if not test:
        result = mydll.SayHello()
    else:
        busy = True
        _thread.start_new_thread(td_download, ())
        while busy:  pass
    if result:
        var_status.set("Download Fail at %s" % hex(result))
        showerror('Romter', 'Download Fail')
    else:
        var_status.set('Download OK')            
        showinfo('Romter', 'Download OK')

if test:
result = 0x5555
busy = True
def td_download():
    global busy, result
    result = mydll.SayHello()
    busy = False
--------------------------

Best Regards,
Jach

[toc] | [next] | [standalone]

#99630

jfong@ms4.hinet.net wrote:

> I am new to Python. As an exercise of it, I try to port a program which
> was written more than 10 years ago. This program use the Borland C++
> Builder as its GUI front end and a DLL does the real work(it will takes a
> few seconds to complete). I saw a strange phenomenon in the following
> codes. The "var_status.set('Download...')" statement seems was deferred
> and didn't show up until the external DLL job was finished and I can only
> saw the result of "var_status.set('Download OK')" statement which
> immediately follows. I try to do the DLL function in a thread(selected by
> using the "test" flag in codes), but it didn't help.
> 
> Can anyone tell me what's the point I was missed?

What does var_status.set() do? If it writes to stdout you may just need to 
flush().

Do you see the same behaviour when you replace mydll.SayHello() with 
something simple like time.sleep(1)?

> -------------------------
> def download():
>     global iniFilename
>     if test:  global result, busy
>     ini = iniFilename
>     iniFilename = "c:\\$$temp.in3"
>     saveIniFile()
>     iniFilename = ini
>     #do the rest
>     var_status.set('Download...')
>     if not test:
>         result = mydll.SayHello()
>     else:
>         busy = True
>         _thread.start_new_thread(td_download, ())
>         while busy:  pass
>     if result:
>         var_status.set("Download Fail at %s" % hex(result))
>         showerror('Romter', 'Download Fail')
>     else:
>         var_status.set('Download OK')
>         showinfo('Romter', 'Download OK')
> 
> if test:
> result = 0x5555
> busy = True
> def td_download():
>     global busy, result
>     result = mydll.SayHello()
>     busy = False
> --------------------------

As a general remark keep your test scripts as simple as possible. Example: 
If 

import mydll
print("Download...", end="")
mydll.SayHello()
print("OK")

showed the same behaviour it would be the ideal test script.

[toc] | [prev] | [next] | [standalone]

#99636

Peter Otten at 2015/11/27  UTC+8 5:19:17 PM wrote：

Hi! Peter, thanks for your prompt reply.

> What does var_status.set() do? If it writes to stdout you may just need to 
> flush().

   var_status is a StringVar which binds to a lable's textvariable. I use this label as the status bar to show message.

> Do you see the same behaviour when you replace mydll.SayHello() with 
> something simple like time.sleep(1)?

    I use time.sleep(3) to replace mydll.SayHello(), still get the same.

> As a general remark keep your test scripts as simple as possible. Example: 
> If 
> 
> import mydll
> print("Download...", end="")
> mydll.SayHello()
> print("OK")
> 
> showed the same behaviour it would be the ideal test script.

    I run the above statements in a test file, it works as expected. I can see "Download..." and then "OK".

[toc] | [prev] | [next] | [standalone]

#99638

jfong@ms4.hinet.net wrote:

> Peter Otten at 2015/11/27  UTC+8 5:19:17 PM wrote：
> 
> Hi! Peter, thanks for your prompt reply.
> 
>> What does var_status.set() do? If it writes to stdout you may just need
>> to flush().
> 
>    var_status is a StringVar which binds to a lable's textvariable. I use
>    this label as the status bar to show message.

Can I conclude from the above that you are using tkinter? Widgets can only 
update when the event loop gets a chance to run. While a quick fix is to 
invoke update_idletasks() callbacks shouldn't generally contain long-running 
code that -- as you have seen -- will block the complete user interface.
 
>> Do you see the same behaviour when you replace mydll.SayHello() with
>> something simple like time.sleep(1)?
> 
>     I use time.sleep(3) to replace mydll.SayHello(), still get the same.
> 
>> As a general remark keep your test scripts as simple as possible.
>> Example: If
>> 
>> import mydll
>> print("Download...", end="")
>> mydll.SayHello()
>> print("OK")
>> 
>> showed the same behaviour it would be the ideal test script.
> 
>     I run the above statements in a test file, it works as expected. I can
>     see "Download..." and then "OK".
> 
> 

Quick-fix example:

#!/usr/bin/env python3
import tkinter as tk
import time

def download():
    var.set("Starting download...")
    root.update_idletasks()
    time.sleep(3)
    var.set("... done")


root = tk.Tk()
var = tk.StringVar()
label = tk.Label(root, textvariable=var)
label.pack()
button = tk.Button(root, text="Download", command=download)
button.pack()

root.mainloop()

A cleaner solution can indeed involve threads; you might adapt the approach 
from <http://effbot.org/zone/tkinter-threads.htm> (Python 2 code).

[toc] | [prev] | [next] | [standalone]

#99655

Peter Otten at 2015/11/27 UTC+8 8:20:54PM wrote：

> Quick-fix example:
> def download():
>     var.set("Starting download...")
>     root.update_idletasks()
>     time.sleep(3)
>     var.set("... done")

Thanks, Peter, The update_idletasks() works. In my trivial program it's easy to apply for there are only two places call the DLL function.

> A cleaner solution can indeed involve threads; you might adapt the approach 
> from <http://effbot.org/zone/tkinter-threads.htm> (Python 2 code).

Using thread is obviously more logical. I think my mistake was the "while busy:  pass" loop which makes no sense because it blocks the main thread, just as the time.sleep() does. That's why in your link (and Laura's too) the widget.after() scheduling was used for this purpose.

From what I had learned here, the other way I can do is making the codes modified as follows. It will get ride of the "result" and "busy" global variables, but it also makes the codes looks a little ugly. I think I will take the update_idletasks() way in this porting for it seems more simpler, and can be used on thread or non-thread calling. Thank you again.
    .....
    .....
    #do the rest
    var_status.set('Download...')
    _thread.start_new_thread(td_download, ())  #must use threading

def td_download():
    result = mydll.SayHello()
    if result:
        var_status.set("Download Fail at %s" % hex(result))
        showerror('Romter', 'Download Fail')
    else:
        var_status.set('Download OK')            
        showinfo('Romter', 'Download OK')

[toc] | [prev] | [next] | [standalone]

#99656

jfong@ms4.hinet.net wrote:

> Peter Otten at 2015/11/27 UTC+8 8:20:54PM wrote：
> 
>> Quick-fix example:
>> def download():
>>     var.set("Starting download...")
>>     root.update_idletasks()
>>     time.sleep(3)
>>     var.set("... done")
> 
> Thanks, Peter, The update_idletasks() works. In my trivial program it's
> easy to apply for there are only two places call the DLL function.
> 
>> A cleaner solution can indeed involve threads; you might adapt the
>> approach from <http://effbot.org/zone/tkinter-threads.htm> (Python 2
>> code).
> 
> Using thread is obviously more logical. I think my mistake was the "while
> busy:  pass" loop which makes no sense because it blocks the main thread,
> just as the time.sleep() does. That's why in your link (and Laura's too)
> the widget.after() scheduling was used for this purpose.

No, the point of both recipes is that tkinter operations are only ever 
invoked from the main thread. The main thread has polling code that 
repeatedly looks if there are results from the helper thread. As far I 
understand the polling method has the structure

f():
   # did we get something back from the other thread?
   # a queue is used to avoid race conditions

   # if yes react.
   # var_status.set() goes here

   # reschedule f to run again in a few millisecs; 
   # that's what after() does
 
> From what I had learned here, the other way I can do is making the codes
> modified as follows. It will get ride of the "result" and "busy" global
> variables, but it also makes the codes looks a little ugly. I think I will
> take the update_idletasks() way in this porting for it seems more simpler,
> and can be used on thread or non-thread calling. Thank you again.
>     .....
>     .....
>     #do the rest
>     var_status.set('Download...')
>     _thread.start_new_thread(td_download, ())  #must use threading
> 
> def td_download():
>     result = mydll.SayHello()
>     if result:
>         var_status.set("Download Fail at %s" % hex(result))
>         showerror('Romter', 'Download Fail')
>     else:
>         var_status.set('Download OK')
>         showinfo('Romter', 'Download OK')

As td_download() runs in the other thread the var_status.set() methods are 
problematic.

Another complication that inevitably comes with concurrency: what if the 
user triggers another download while one download is already running? If you 
don't keep track of all downloads the message will already switch to 
"Download OK" while one download is still running.

[toc] | [prev] | [next] | [standalone]

#99685

Peter Otten at 2015/11/28 UTC+8 6:14:09PM wrote：
> No, the point of both recipes is that tkinter operations are only ever 
> invoked from the main thread. The main thread has polling code that 
> repeatedly looks if there are results from the helper thread. As far I 
> understand the polling method has the structure
> 
> f():
>    # did we get something back from the other thread?
>    # a queue is used to avoid race conditions
> 
>    # if yes react.
>    # var_status.set() goes here
> 
>    # reschedule f to run again in a few millisecs; 
>    # that's what after() does

Have no idea how the main thread poll on all those events (or it use a queue)?
All I know now is that the main thread(mainloop()?) can be easily blocked by event handlers if the handler didn't run as a separate thread.

> >     .....
> >     .....
> >     #do the rest
> >     var_status.set('Download...')
> >     _thread.start_new_thread(td_download, ())  #must use threading
> > 
> > def td_download():
> >     result = mydll.SayHello()
> >     if result:
> >         var_status.set("Download Fail at %s" % hex(result))
> >         showerror('Romter', 'Download Fail')
> >     else:
> >         var_status.set('Download OK')
> >         showinfo('Romter', 'Download OK')
> 
> As td_download() runs in the other thread the var_status.set() methods are 
> problematic.

No idea what kind of problem it will encounter. Can you explain?

> Another complication that inevitably comes with concurrency: what if the 
> user triggers another download while one download is already running? If you 
> don't keep track of all downloads the message will already switch to 
> "Download OK" while one download is still running.

Hummm...this thought never comes to my mind. After take a quick test I found, you are right, a second "download" was triggered immediately. That's a shock to me. I suppose the same event shouldn't be triggered again, or at least not triggered immediately, before its previous handler was completed. ...I will take a check later on Borland C++ builder to see how it reacts!

Anyway to prevent this happens? if Python didn't take care it for us.

[toc] | [prev] | [next] | [standalone]

#99761

jf...@ms4.hinet.net at 2015/11/29  UTC+8 10:55:28AM wrote：
> > >     .....
> > >     .....
> > >     #do the rest
> > >     var_status.set('Download...')
> > >     _thread.start_new_thread(td_download, ())  #must use threading
> > > 
> > > def td_download():
> > >     result = mydll.SayHello()
> > >     if result:
> > >         var_status.set("Download Fail at %s" % hex(result))
> > >         showerror('Romter', 'Download Fail')
> > >     else:
> > >         var_status.set('Download OK')
> > >         showinfo('Romter', 'Download OK')
> > 
> > As td_download() runs in the other thread the var_status.set() methods are 
> > problematic.
> 
> No idea what kind of problem it will encounter. Can you explain?

It might be not a good idea to update GUI in a spawned thread, according to Mark Lutz's book "Programming Python", section "Using Threads with tkinter GUIs"

"If you do use threads in tkinter programs, however, you need to remember that only the main thread (the one that built the GUI and started the mainloop) should generally make GUI calls. At the least, multiple threads should not attempt to update the GUI at the same time..." 
 
> > Another complication that inevitably comes with concurrency: what if the 
> > user triggers another download while one download is already running? If you 
> > don't keep track of all downloads the message will already switch to 
> > "Download OK" while one download is still running.
> 
> Hummm...this thought never comes to my mind. After take a quick test I found, you are right, a second "download" was triggered immediately. That's a shock to me. I suppose the same event shouldn't be triggered again, or at least not triggered immediately, before its previous handler was completed. ...I will take a check later on Borland C++ builder to see how it reacts!
> 
> Anyway to prevent this happens? if Python didn't take care it for us.

then don't use thread if you prefer the GUI freeze.

[toc] | [prev] | [next] | [standalone]

#99789

jfong@ms4.hinet.net wrote:

> Peter Otten at 2015/11/28 UTC+8 6:14:09PM wrote：
>> No, the point of both recipes is that tkinter operations are only ever
>> invoked from the main thread. The main thread has polling code that
>> repeatedly looks if there are results from the helper thread. As far I
>> understand the polling method has the structure
>> 
>> f():
>>    # did we get something back from the other thread?
>>    # a queue is used to avoid race conditions
>> 
>>    # if yes react.
>>    # var_status.set() goes here
>> 
>>    # reschedule f to run again in a few millisecs;
>>    # that's what after() does
> 
> Have no idea how the main thread poll on all those events (or it use a
> queue)? All I know now is that the main thread(mainloop()?) can be easily
> blocked by event handlers if the handler didn't run as a separate thread.
> 
>> >     .....
>> >     .....
>> >     #do the rest
>> >     var_status.set('Download...')
>> >     _thread.start_new_thread(td_download, ())  #must use threading
>> > 
>> > def td_download():
>> >     result = mydll.SayHello()
>> >     if result:
>> >         var_status.set("Download Fail at %s" % hex(result))
>> >         showerror('Romter', 'Download Fail')
>> >     else:
>> >         var_status.set('Download OK')
>> >         showinfo('Romter', 'Download OK')
>> 
>> As td_download() runs in the other thread the var_status.set() methods
>> are problematic.
> 
> No idea what kind of problem it will encounter. Can you explain?

While the var_status.set() invoked from the second thread modifies some 
internal data the main thread could kick in and modify (parts of) that same 
data, thus bringing tkinter into an broken state. A simple example that 
demonstrates the problem:

import random
import threading
import time

account = 70

def withdraw(delay, amount):
    global account
    if account >= amount:
        print("withdrawing", amount)
        account -= amount
    else:
        print("failed to withdraw", amount)

threads = []
for i in range(10):
    t = threading.Thread(
        target=withdraw,
        kwargs=dict(delay=.1,
                    amount=random.randrange(1, 20)))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

Before every withdrawal there seems to be a check that ensures that there is 
enough money left, but when you run the script a few times you will still 
sometimes end with a negative balance. That happens when thread A finds 
enough money, then execution switches to thread B which also finds enough 
money, then both threads perform a withdrawal -- oops there wasn't enough 
money for both.
 
>> Another complication that inevitably comes with concurrency: what if the
>> user triggers another download while one download is already running? If
>> you don't keep track of all downloads the message will already switch to
>> "Download OK" while one download is still running.
> 
> Hummm...this thought never comes to my mind. After take a quick test I
> found, you are right, a second "download" was triggered immediately.
> That's a shock to me. I suppose the same event shouldn't be triggered
> again, or at least not triggered immediately, before its previous handler
> was completed. ...I will take a check later on Borland C++ builder to see
> how it reacts!
> 
> Anyway to prevent this happens? if Python didn't take care it for us.

A simple measure would be to disable the button until the download has 
ended.

[toc] | [prev] | [next] | [standalone]

#99844

Peter Otten at 2015/12/1  UTC+8 7:01:55PM wrote：
> While the var_status.set() invoked from the second thread modifies some 
> internal data the main thread could kick in and modify (parts of) that same 
> data, thus bringing tkinter into an broken state. A simple example that 
> demonstrates the problem:
> 
> import random
> import threading
> import time
> 
> account = 70
> 
> def withdraw(delay, amount):
>     global account
>     if account >= amount:
>         print("withdrawing", amount)
>         account -= amount
>     else:
>         print("failed to withdraw", amount)
> 
> threads = []
> for i in range(10):
>     t = threading.Thread(
>         target=withdraw,
>         kwargs=dict(delay=.1,
>                     amount=random.randrange(1, 20)))
>     threads.append(t)
>     t.start()
> 
> for t in threads:
>     t.join()

It's a simple and vivid example. You must be in the banking business:-)

In this exercise, I can just use update_idletasks() method to solve the deferred display problem of tkinter, and forget all about thread. But what if I really want to use thread for the long-running DLL function in the download hander? I didn't figure out a correct way of doing it at this moment. Maybe queue is the answer but I am not familiar with it yet.

def download():
    var_status.set(...)  #showing a start-up message
    result = SayHello()  #this is a long-running function
    if result....        #according to the result, display more messages
        ....
> 
> Before every withdrawal there seems to be a check that ensures that there is 
> enough money left, but when you run the script a few times you will still 
> sometimes end with a negative balance. That happens when thread A finds 
> enough money, then execution switches to thread B which also finds enough 
> money, then both threads perform a withdrawal -- oops there wasn't enough 
> money for both.
>  
> >> Another complication that inevitably comes with concurrency: what if the
> >> user triggers another download while one download is already running? If
> >> you don't keep track of all downloads the message will already switch to
> >> "Download OK" while one download is still running.
> > 
> > Hummm...this thought never comes to my mind. After take a quick test I
> > found, you are right, a second "download" was triggered immediately.
> > That's a shock to me. I suppose the same event shouldn't be triggered
> > again, or at least not triggered immediately, before its previous handler
> > was completed. ...I will take a check later on Borland C++ builder to see
> > how it reacts!
> > 
> > Anyway to prevent this happens? if Python didn't take care it for us.
> 
> A simple measure would be to disable the button until the download has 
> ended.

Good! simple and work.

I shouldn't say "Python should take care of it", but "tk" should do. It's annoying the widget's command was re-triggered before it was finished. (my personal opinion:-)

[toc] | [prev] | [next] | [standalone]

#99658

In a message of Sat, 28 Nov 2015 11:13:38 +0100, Peter Otten writes:
>jfong@ms4.hinet.net wrote:
>> Using thread is obviously more logical. I think my mistake was the "while
>> busy:  pass" loop which makes no sense because it blocks the main thread,
>> just as the time.sleep() does. That's why in your link (and Laura's too)
>> the widget.after() scheduling was used for this purpose.

I never saw the reply that Peter is replying to.
The threading module constructs a higher level interface on top of the
low level thread module.  Thus it is the preferred way to go for
standard Python code -- and even Fredrik's recipe contains the
line:
	import thread # should use the threading module instead!

Laura

[toc] | [prev] | [next] | [standalone]

#99686

Laura Creighton at 2015/11/28 UTC+8 6:52:25PM wrote：
> I never saw the reply that Peter is replying to.
> The threading module constructs a higher level interface on top of the
> low level thread module.  Thus it is the preferred way to go for
> standard Python code -- and even Fredrik's recipe contains the
> line:
> 	import thread # should use the threading module instead!
> 
> Laura
Hi! Laura, 

takes the porting of an old BCB GUI program as an exercise in my learning python, I just quickly grab the required tools (mainly the tkinter) in python to complete this "work" and get a feeling of how python performs on doing this. Most of my knowledge of python comes from Mark Lutz's book "Learning python" and "Programming python". I didn't dive into this language deeply yet. There are topics about "_thread" and "threading" modules in the book and I just pick the lower level one for this because the "threading" is based on the "_thread".

Thanks for your note and link.

--Jach

[toc] | [prev] | [next] | [standalone]

#99640

In a message of Fri, 27 Nov 2015 13:20:03 +0100, Peter Otten writes:

>A cleaner solution can indeed involve threads; you might adapt the approach 
>from <http://effbot.org/zone/tkinter-threads.htm> (Python 2 code).

But it is probably better to use threading

http://code.activestate.com/recipes/82965-threads-tkinter-and-asynchronous-io/

Laura

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web