Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #102691 > unrolled thread

Help using Thread (or other method)

Back to article view | Back to comp.lang.python

Contents

  Help using Thread (or other method) "Brendan Simon (eTRIX)" <brendan.simon@etrix.com.au> - 2016-02-09 07:59 +1100
    Re: Help using Thread (or other method) Marko Rauhamaa <marko@pacujo.net> - 2016-02-08 23:27 +0200

#102691 — Help using Thread (or other method)

Hi.  Need some urgent help.

I have a python app that uses `select` to wait for data from an arm
embedded linux kernel interrupt (every second).  The mainloop of the app
then grabs data from some memory mapped area, processes it and then does
a http post to a server.

The problem is the http post can take multiple seconds to respond and
blocks (using the requests module).  So I've put the posting in a
separate thread (using Thread module) and send data to it via a queue
(using the Queue module).  The http post can send multiple records by
pulling multiple items off the queue.  Sounds good so far, right?

This seems to work ok, but I also need to grab data from some other
serial devices (gps, weather station) to put in the post.  Various
sample apps on the web use Threads to do this, so I am doing the same. 
One of the threads uses `select` (see below), following by readlines()
on the device.

       def __init__(self):
        select_timeout = 1
        serial_timeout = 0.1

    def wait_and_process(self):
        '''wait for data and process it.'''
        r = select.select([self.serial_dev], [], [], self.select_timeout)
        if not r[0]:
            #print("DEBUG: TIMEOUT: wait_and_process")
            return

        #print("DEBUG: Weather Data Captured")
        for s in self.serial_dev.readlines():
            #print("INFO: Data: {}".format(s))

            temperature = extract_temperature(s)

Is using `select` here redundant?  It's used to block the thread until
the first character is ready to be read from the serial device?
Then I use serial_dev.readlines() to read the stream of chars from the
weather station.
Is that ok?

Does readline block until it sees an end-of-line char?  i.e. does it
only wake up the thread when it has a _complete string_ or a timeout of
0.1 seconds?
or will the process block other threads from running while it is
gathering the chars until the newline?
I'm assuming the gathering of chars will happen in the background and
allow my main thread to run, correct?

My application mainloop (not the threads) needs to be high priority and
always process as soon as it gets an interrupt.  Is using select a good
way of doing this?  How can I ensure that no other threads are utilizing
the CPU, etc?  I'm worried about the GIL (I'm using CPython 2.7).

Is there a better way of structuring this to ensure I process the
interrupt and not get interrupt overrun?

Is using select only in the mainloop, with multiple file descriptors, a
better way of doing things, so that I can process the file descriptor of
interest first, before any others if set?

Is using the multiprocessing module a better option?

Thanks for any advice,
Brendan.

[toc] | [next] | [standalone]

#102692

"Brendan Simon (eTRIX)" <brendan.simon@etrix.com.au>:

> Is using `select` here redundant?

Yes, if you go the thread way, select() is usually redundant and just
gets in the way.

> Does readline block until it sees an end-of-line char? i.e. does it
> only wake up the thread when it has a _complete string_ or a timeout
> of 0.1 seconds?

You probably don't want to mix 0.1-second timeouts with readline but let
the readline thread do its job: call readline in an indefinite wait. You
can then have Queue.get(timeout=...) do the timeout if necessary.

> or will the process block other threads from running while it is
> gathering the chars until the newline?

It shouldn't, but I didn't try it out.

> My application mainloop (not the threads) needs to be high priority
> and always process as soon as it gets an interrupt. Is using select a
> good way of doing this? How can I ensure that no other threads are
> utilizing the CPU, etc? I'm worried about the GIL (I'm using CPython
> 2.7).

Here's what you can do:

 * Have your main, high-priority thread have an input Queue coming in.
   Have the peripheral threads put their notifications in that Queue.

 * Have each peripheral thread have an acknowledgement threading.Event
   object. Whenever a peripheral thread has put a notification in the
   Queue, have it wait for the main thread to set() the acknowledgement
   event before looking for more work. That prevents the peripheral
   threads from starving the CPU under any circumstances.

> Is using select only in the mainloop, with multiple file descriptors, a
> better way of doing things, so that I can process the file descriptor of
> interest first, before any others if set?
>
> Is using the multiprocessing module a better option?

I don't know enough of your exact problem to recommend anything. Both of
those are viable options. However, readline() and select() don't mix
well at all because readline() does buffered I/O and select() deals with
raw file descriptors.

If you *can* reduce all functionality to file descriptor operations, use
os.read() and os.write() only. Then, you can use select.select() or
select.epoll() and not bother with threads at all. However, then you'll
have to implement HTTP processing on your own.

(The same goes for 3.5's new async/await scheme: https://docs.python.or
g/3/whatsnew/3.5.html?highlight=asyncio#pep-492-coroutines-with-async-a
nd-await-syntax>.)

Given the nature of your questions, you might want to stick with
threads. Your problem will then be how to shut down those threads in the
end...


Marko

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web