Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #82340 > unrolled thread

PyQt: user interface freezed when using concurrent.futures.ThreadPoolExecutor

Back to article view | Back to comp.lang.python

Contents

  PyQt: user interface freezed when using concurrent.futures.ThreadPoolExecutor iMath <redstone-cold@163.com> - 2014-12-10 20:52 -0800
    Re: PyQt: user interface freezed when using concurrent.futures.ThreadPoolExecutor Michael Torrie <torriem@gmail.com> - 2014-12-10 22:25 -0700
      Re: PyQt: user interface freezed when using concurrent.futures.ThreadPoolExecutor iMath <redstone-cold@163.com> - 2014-12-11 19:20 -0800
        Re: PyQt: user interface freezed when using concurrent.futures.ThreadPoolExecutor Michael Torrie <torriem@gmail.com> - 2014-12-11 23:07 -0700
          Re: PyQt: user interface freezed when using concurrent.futures.ThreadPoolExecutor Marko Rauhamaa <marko@pacujo.net> - 2014-12-12 09:18 +0200
            Re: PyQt: user interface freezed when using concurrent.futures.ThreadPoolExecutor Chris Angelico <rosuav@gmail.com> - 2014-12-12 18:42 +1100
              Re: PyQt: user interface freezed when using concurrent.futures.ThreadPoolExecutor Marko Rauhamaa <marko@pacujo.net> - 2014-12-12 10:43 +0200
    Re: PyQt: user interface freezed when using concurrent.futures.ThreadPoolExecutor Mark Summerfield <list@qtrac.plus.com> - 2014-12-18 03:16 -0800
      Re: PyQt: user interface freezed when using concurrent.futures.ThreadPoolExecutor Michael Torrie <torriem@gmail.com> - 2014-12-18 09:29 -0700

#82340 — PyQt: user interface freezed when using concurrent.futures.ThreadPoolExecutor

I think the user interface shouldn't be freezed when using concurrent.futures.ThreadPoolExecutor here,as it executes asynchronously ,
 but it doesn't meet my expectations,anyone can explain why ? any other solutions here to not let user interface freezed?

code is here
http://stackoverflow.com/questions/27393533/user-interface-freezed-when-using-concurrent-futures-threadpoolexecutor

[toc] | [next] | [standalone]

#82344

On 12/10/2014 09:52 PM, iMath wrote:
> I think the user interface shouldn't be freezed when using
> concurrent.futures.ThreadPoolExecutor here,as it executes
> asynchronously , but it doesn't meet my expectations,anyone can
> explain why ? any other solutions here to not let user interface
> freezed?
> 
> code is here 
> http://stackoverflow.com/questions/27393533/user-interface-freezed-when-using-concurrent-futures-threadpoolexecutor

In most any GUI framework, regardless of your use of threads, your
callbacks must return control to the main loop immediately (whether an
on-click or an on-timer event), or the GUI *will* freeze.

You are spawning threads to download the urls, then sitting there
waiting for them to finish in the callback.  Of course the GUI will
freeze; your callback is blocking it.  What you should be doing is
spawning the threads to do the download, then have the threads (using a
proper QThread mechanism or some other semaphore mechanism) raise a
signal to indicate they are done, which you will then catch in the main
loop as a normal callback.  And really when it comes to I/O and GUIs,
asynchronous calls are always better than threads.  Not sure what Qt has
in the way of asynchronous i/o calls, and I'm not familiar enough with
the various async i/o frameworks in Python to speak to that, though I
did use Python-Twisted once... very powerful once you get your head
wrapped around it.

Look through the list archives because in the last 3 or 4 weeks I and
another Python user talked proper thread use with PyQt or PySide, with
code examples.

[toc] | [prev] | [next] | [standalone]

#82411

在 2014年12月11日星期四UTC+8下午1时25分41秒，Michael Torrie写道：
> On 12/10/2014 09:52 PM, iMath wrote:
> > I think the user interface shouldn't be freezed when using
> > concurrent.futures.ThreadPoolExecutor here,as it executes
> > asynchronously , but it doesn't meet my expectations,anyone can
> > explain why ? any other solutions here to not let user interface
> > freezed?
> > 
> > code is here 
> > http://stackoverflow.com/questions/27393533/user-interface-freezed-when-using-concurrent-futures-threadpoolexecutor
> 
> In most any GUI framework, regardless of your use of threads, your
> callbacks must return control to the main loop immediately (whether an
> on-click or an on-timer event), or the GUI *will* freeze.
> 
> You are spawning threads to download the urls, then sitting there
> waiting for them to finish in the callback.  Of course the GUI will
> freeze; your callback is blocking it.  What you should be doing is
> spawning the threads to do the download, then have the threads (using a
> proper QThread mechanism or some other semaphore mechanism) raise a
> signal to indicate they are done, which you will then catch in the main
> loop as a normal callback.  And really when it comes to I/O and GUIs,
> asynchronous calls are always better than threads.  Not sure what Qt has
> in the way of asynchronous i/o calls, and I'm not familiar enough with
> the various async i/o frameworks in Python to speak to that, though I
> did use Python-Twisted once... very powerful once you get your head
> wrapped around it.
> 
> Look through the list archives because in the last 3 or 4 weeks I and
> another Python user talked proper thread use with PyQt or PySide, with
> code examples.

"when it comes to I/O and GUIs, asynchronous calls are always better than threads."

I cannot grasp your meaning here, IMO, asynchronous calls are done by using threads.

[toc] | [prev] | [next] | [standalone]

#82412

On 12/11/2014 08:20 PM, iMath wrote:
> 在 2014年12月11日星期四UTC+8下午1时25分41秒，Michael Torrie写道：
>> On 12/10/2014 09:52 PM, iMath wrote:
> "when it comes to I/O and GUIs, asynchronous calls are always better than threads."
> 
> I cannot grasp your meaning here, IMO, asynchronous calls are done by using threads.

Not in your code.  All your I/O calls are synchronous; they block until
they are finished.  But if you mean that calling synchronous calls while
in a thread has a similar effect to an asynchronous call, you are
correct.  But the way you're waiting for the threads to finish, you're
blocking your gui.

An asynchronous API lets you start long-running I/O calls and define a
function that is automatically called upon completion.  In other words
it's event-driven.  Qt may provide everything you need already in an
asynchronous form.  Check the docs. And use google.  Here's a link I
found from a google search that illustrates how to fetch a url in Qt
using an asynchronous method.  You could fire off as many of these as
you want, then just wait for signals.

http://qt-project.org/wiki/Download_Data_from_URL

[toc] | [prev] | [next] | [standalone]

#82416

Michael Torrie <torriem@gmail.com>:

> An asynchronous API lets you start long-running I/O calls and define a
> function that is automatically called upon completion. In other words
> it's event-driven. Qt may provide everything you need already in an
> asynchronous form.

GUI developers have been doing event-driven programming for decades.
That's an excellent preparation for network programming as well.
Unfortunately, the minds of a generation of programmers were
contaminated by the thread craze of the 1990's (Java and Windows NT).


Marko

[toc] | [prev] | [next] | [standalone]

#82419

On Fri, Dec 12, 2014 at 6:18 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> GUI developers have been doing event-driven programming for decades.
> That's an excellent preparation for network programming as well.
> Unfortunately, the minds of a generation of programmers were
> contaminated by the thread craze of the 1990's (Java and Windows NT).

Threads predate Java and NT, and with a much MUCH better
implementation, on OS/2. I don't consider my mind to have been
*contaminated* by the OS/2 threading model, which worked superbly.
Like all models, it has its limitations, but it's one worth getting
your head around IMO. Knowing when to use threads, when to use
processes, and when to keep everything in a single thread with some
kind of dispatch loop (and *which* dispatch loop) is a valuable skill.

But yes, the Windows threading model does have a number of annoying
flaws. And I don't remember how Java did things, except that I
struggled to find basic fundamental primitives like semaphores, and
had to use synchronized functions/objects instead.

ChrisA

[toc] | [prev] | [next] | [standalone]

#82426

Chris Angelico <rosuav@gmail.com>:

> And I don't remember how Java did things, except that I struggled to
> find basic fundamental primitives like semaphores, and had to use
> synchronized functions/objects instead.

Java now has a diverse set of synchornization facilities, but the
builtin object synchronization produces excellent idioms and is usually
preferred (by me). Java also has rigorously defined its multithreaded
data model (the Happens-Before Relation).

Java's threads have two major problems:

 * The classic I/O forced you to dedicate a thread for each
   communication context (connection) because there was no multiplexing
   facility and because the sockets were blocking (and buffered IIRC).
   The thread proliferation caused serious scalability issues.
   Latter-day Java's NIO framework addresses this shortcoming to a great
   degree, but even that is a surprisingly tricky beast to program
   for -- it suffers from builtin race conditions.

 * There is no way to interrupt a thread -- except in Solaris! You can
   mark a thread for interruption and there is an associated exception
   but they are not guaranteed to be provided by JVM.

And then there's the inherent problems of thread programming:

 * Deadlocks.

 * Missing synchronization.

which in practice are just too hard for mortals. I've seen it. I've been
complicit.

Now, when it comes to Python, asyncio is its answer to Java's NIO. It
seeks to provide a cross-platform Way to Life, Universe and Everything.
A commendable objective. Unfortunately, I find the coroutine approach
artificial and unintuitive in practice. Select.epoll(EPOLLET) plus a
timer implementation easily beats it (as long as you can limit yourself
to linux).


Marko

[toc] | [prev] | [next] | [standalone]

#82632

On Thursday, December 11, 2014 4:53:04 AM UTC, iMath wrote:
> I think the user interface shouldn't be freezed when using concurrent.futures.ThreadPoolExecutor here,as it executes asynchronously ,
>  but it doesn't meet my expectations,anyone can explain why ? any other solutions here to not let user interface freezed?
> 
> code is here
> http://stackoverflow.com/questions/27393533/user-interface-freezed-when-using-concurrent-futures-threadpoolexecutor

It looks to me that what you are doing is sharing a single core between your GUI and your processing. Threading isn't usually a good approach to Python concurrency that is CPU-bound.

Simply changing ThreadPoolExecutor to ProcessPoolExecutor will improve performance, but will still allow the UI to freeze.

The approach I use for concurrency in GUI applications is for the GUI to run in the main thread (the default, and there's no choice) and to create a "manager" thread that does almost no work. This means that the GUI thread gets almost all the processing on the core on which it runs. Whenever there is work to do it is passed to the manager thread which immediately gives it to someone else. The someone else is a process pool. This ensures that the GUI doesn't freeze even in the face of lots of processing (because the processing is done in one or more separate processes).

In my book "Python in Practice", http://www.qtrac.eu/pipbook.html there's an example of how to do this in chapter 4. The example uses Tkinter but I use exactly the same approach in PyQt and PySide. This chapter also discusses many issues related to Python concurrency.

[toc] | [prev] | [next] | [standalone]

#82638

On 12/18/2014 04:16 AM, Mark Summerfield wrote:
> It looks to me that what you are doing is sharing a single core
> between your GUI and your processing. Threading isn't usually a good
> approach to Python concurrency that is CPU-bound.

Except that his code was not CPU-bound to begin with.  His real problem
is that his callback is starting *and* waiting for all the threads to do
their work without returning to the main loop, thus blocking the GUI.
As for the threads, they are I/O bound--he's simply trying to do
concurrent HTTP downloads.  So blocking in the GIL is not the issue
here.  In fact, in lieu of using proper asynchronous I/O, threading of
some kind is probably not a bad solution here, but he has to do it
within the Qt Framework, using signals to notify the GUI when a thread
is finished, or using a synchronization primitive such as a queue, as
recommended by your book in fact.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web