Path: csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From: "Brendan Simon (eTRIX)" <brendan.simon@etrix.com.au>
Newsgroups: comp.lang.python
Subject: Help using Thread (or other method)
Date: Tue, 9 Feb 2016 07:59:27 +1100
Organization: eTRIX pty ltd
Lines: 67
Sender: Brendan Simon <brendanjsimon@gmail.com>
Message-ID: <mailman.111.1454965181.2317.python-list@python.org>
Reply-To: brendan.simon@etrix.com.au
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.1
Precedence: list
Xref: csiph.com comp.lang.python:102691

Hi.  Need some urgent help.

I have a python app that uses `select` to wait for data from an arm
embedded linux kernel interrupt (every second).  The mainloop of the app
then grabs data from some memory mapped area, processes it and then does
a http post to a server.

The problem is the http post can take multiple seconds to respond and
blocks (using the requests module).  So I've put the posting in a
separate thread (using Thread module) and send data to it via a queue
(using the Queue module).  The http post can send multiple records by
pulling multiple items off the queue.  Sounds good so far, right?

This seems to work ok, but I also need to grab data from some other
serial devices (gps, weather station) to put in the post.  Various
sample apps on the web use Threads to do this, so I am doing the same. 
One of the threads uses `select` (see below), following by readlines()
on the device.

       def __init__(self):
        select_timeout = 1
        serial_timeout = 0.1

    def wait_and_process(self):
        '''wait for data and process it.'''
        r = select.select([self.serial_dev], [], [], self.select_timeout)
        if not r[0]:
            #print("DEBUG: TIMEOUT: wait_and_process")
            return

        #print("DEBUG: Weather Data Captured")
        for s in self.serial_dev.readlines():
            #print("INFO: Data: {}".format(s))

            temperature = extract_temperature(s)

Is using `select` here redundant?  It's used to block the thread until
the first character is ready to be read from the serial device?
Then I use serial_dev.readlines() to read the stream of chars from the
weather station.
Is that ok?

Does readline block until it sees an end-of-line char?  i.e. does it
only wake up the thread when it has a _complete string_ or a timeout of
0.1 seconds?
or will the process block other threads from running while it is
gathering the chars until the newline?
I'm assuming the gathering of chars will happen in the background and
allow my main thread to run, correct?

My application mainloop (not the threads) needs to be high priority and
always process as soon as it gets an interrupt.  Is using select a good
way of doing this?  How can I ensure that no other threads are utilizing
the CPU, etc?  I'm worried about the GIL (I'm using CPython 2.7).

Is there a better way of structuring this to ensure I process the
interrupt and not get interrupt overrun?

Is using select only in the mainloop, with multiple file descriptors, a
better way of doing things, so that I can process the file descriptor of
interest first, before any others if set?

Is using the multiprocessing module a better option?

Thanks for any advice,
Brendan.