Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #109293 > unrolled thread

Efficient handling of fast, real-time hex data

Back to article view | Back to comp.lang.python

Contents

  Efficient handling of fast, real-time hex data jladasky@itu.edu - 2016-05-31 17:20 -0700
    Re: Efficient handling of fast, real-time hex data Michael Torrie <torriem@gmail.com> - 2016-05-31 18:35 -0600
      Re: Efficient handling of fast, real-time hex data jladasky@itu.edu - 2016-05-31 18:26 -0700
    Re: Efficient handling of fast, real-time hex data Rob Gaddi <rgaddi@highlandtechnology.invalid> - 2016-06-01 01:01 +0000
      Re: Efficient handling of fast, real-time hex data jladasky@itu.edu - 2016-05-31 18:23 -0700
    Re: Efficient handling of fast, real-time hex data Paul Rubin <no.email@nospam.invalid> - 2016-05-31 21:16 -0700
    Re: Efficient handling of fast, real-time hex data Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2016-06-01 16:37 +1200
      Re: Efficient handling of fast, real-time hex data jladasky@itu.edu - 2016-06-01 11:03 -0700
        Re: Efficient handling of fast, real-time hex data Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2016-06-02 18:50 +1200
          Re: Efficient handling of fast, real-time hex data alister <alister.ware@ntlworld.com> - 2016-06-02 08:13 +0000
            Re: Efficient handling of fast, real-time hex data Gene Heskett <gheskett@wdtv.com> - 2016-06-02 05:41 -0400
              Re: Efficient handling of fast, real-time hex data alister <alister.ware@ntlworld.com> - 2016-06-02 12:30 +0000

#109293 — Efficient handling of fast, real-time hex data

Greetings everyone,

I'm developing small embedded systems.  I can't use Python to program them, I have to program the little CPU's in C.  

I want a peripheral I've designed to talk over USB to a Python program on the host computer.  The peripheral is spewing data at a reasonably high rate, about 5,000 16-bit unsigned integers per second.  For now, I'm trying to print these numbers as they come in.  Even though I'm working with a fast, up-to-date computer, I am finding it difficult for the host computer to keep up with the data stream.

The details of my setup: running Python 3.4 on Ubuntu 15.04 x64.  Using PySerial to handle UART over USB.  Intel Core i7-4790K CPU @ 4.00GHz.

At first I had the peripheral transmit the numbers as base-10 encoded strings, separated by spaces, just so that I could see that I was receiving the right values from the peripheral.  My serial port reads look fine -- until they fall behind.  But in this first case, I was using up to six bytes to send a two-byte value.  Obviously that's wasteful.  So I rewrote the peripheral to send bundles of C uint_16's, with each bundle of three numbers separated by a newline.

Now the challenge is to unpack and display those numbers on the Python side.  I know that print statements themselves are probably slowing things down, but I still want to see that those numbers are correct.  

So, how can I take the byte sequence <0x01 0x02 0x03 0x04 0x05 0x06 \n> that Serial.readline() returns to me, and QUICKLY turn it into three integer values, 258, 772, and 1286?  Better yet, can I write these bytes directly into an array (numpy is an option), since I'm going to have a lot of them?

I never thought I would find a task that's easier for me to imagine doing in C than in Python, but at the moment this is one.  Thanks for any suggestions!

[toc] | [next] | [standalone]

#109294

On 05/31/2016 06:20 PM, jladasky@itu.edu wrote:
> So, how can I take the byte sequence <0x01 0x02 0x03 0x04 0x05 0x06
> \n> that Serial.readline() returns to me, and QUICKLY turn it into
> three integer values, 258, 772, and 1286?  Better yet, can I write
> these bytes directly into an array (numpy is an option), since I'm
> going to have a lot of them?

I think you might want to use the struct module. It's designed for this
kind of packing and unpacking:

https://docs.python.org/3/library/struct.html

> I never thought I would find a task that's easier for me to imagine
> doing in C than in Python, but at the moment this is one.  Thanks for
> any suggestions!

Byte and bit twiddling are things that C has always been fast at.  But I
think Python probably could do it.

[toc] | [prev] | [next] | [standalone]

#109297

On Tuesday, May 31, 2016 at 5:36:10 PM UTC-7, Michael Torrie wrote:
> 
> I think you might want to use the struct module. It's designed for this
> kind of packing and unpacking:
> 
> https://docs.python.org/3/library/struct.html

Hi Michael,

Thanks for pointing me at the struct module.  There appears to be some redundancy between the numpy.frombuffer function and Python's own struct module.  If the latter doesn't work out for me, I'll try the former.

[toc] | [prev] | [next] | [standalone]

#109295

jladasky@itu.edu wrote:

> Greetings everyone,
>
> I'm developing small embedded systems.  I can't use Python to program them, I have to program the little CPU's in C.  
>
> I want a peripheral I've designed to talk over USB to a Python program on the host computer.  The peripheral is spewing data at a reasonably high rate, about 5,000 16-bit unsigned integers per second.  For now, I'm trying to print these numbers as they come in.  Even though I'm working with a fast, up-to-date computer, I am finding it difficult for the host computer to keep up with the data stream.
>
> The details of my setup: running Python 3.4 on Ubuntu 15.04 x64.  Using PySerial to handle UART over USB.  Intel Core i7-4790K CPU @ 4.00GHz.
>
> At first I had the peripheral transmit the numbers as base-10 encoded strings, separated by spaces, just so that I could see that I was receiving the right values from the peripheral.  My serial port reads look fine -- until they fall behind.  But in this first case, I was using up to six bytes to send a two-byte value.  Obviously that's wasteful.  So I rewrote the peripheral to send bundles of C uint_16's, with each bundle of three numbers separated by a newline.
>
> Now the challenge is to unpack and display those numbers on the Python side.  I know that print statements themselves are probably slowing things down, but I still want to see that those numbers are correct.  
>
> So, how can I take the byte sequence <0x01 0x02 0x03 0x04 0x05 0x06 \n> that Serial.readline() returns to me, and QUICKLY turn it into three integer values, 258, 772, and 1286?  Better yet, can I write these bytes directly into an array (numpy is an option), since I'm going to have a lot of them?
>
> I never thought I would find a task that's easier for me to imagine doing in C than in Python, but at the moment this is one.  Thanks for any suggestions!

You'll probably want to process it in blocks.  Allocate a 3kB
bytearray, assign into it from the data coming in off Serial (less
the newlines) and when you fill it, call numpy.from_buffer to rip it.

Then just print the first line of it.  Decimating the data
rate radically will help with the print-induced load, and it's not
like you need to see every sample.

Are you going to be trying to use this data realtime, or are you just
trying to datalog it and deal with it offline?  Because at some point
you'll need to decide, all in, how much data you're willing to try to
hold in memory and what you intend to do with the rest of it.

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com

Email address domain is currently out of order.  See above to fix.

[toc] | [prev] | [next] | [standalone]

#109296

On Tuesday, May 31, 2016 at 6:02:07 PM UTC-7, Rob Gaddi wrote:

> You'll probably want to process it in blocks.  Allocate a 3kB
> bytearray, assign into it from the data coming in off Serial (less
> the newlines) and when you fill it, call numpy.from_buffer to rip it.

Thanks Rob, numpy.frombuffer (no underscore in my version of Numpy, v.1.8) looks like it will be very helpful!

> Then just print the first line of it.  Decimating the data
> rate radically will help with the print-induced load, and it's not
> like you need to see every sample.
> 
> Are you going to be trying to use this data realtime, or are you just
> trying to datalog it and deal with it offline?  Because at some point
> you'll need to decide, all in, how much data you're willing to try to
> hold in memory and what you intend to do with the rest of it.

No, I don't need to print all the data... in fact, I don't plan to print any of it when I'm done.  I do want to save every data packet for offline analysis.  I also want to display real-time histograms of at least some of the data.  I've already cobbled together (but have not optimized) a Matplotlib program that displays mock data at 16 FPS, which is fast enough.

I used to work on flow cytometers.  Look them up if you're interested in multi-dimensional, real-time data display.

[toc] | [prev] | [next] | [standalone]

#109299

jladasky@itu.edu writes:
> high rate, about 5,000 16-bit unsigned integers per second....
> Using PySerial to handle UART over USB.  Intel Core i7-4790K CPU @
> 4.00GHz.

This really should not be an issue.  That's not such a terribly high
speed, and there's enough buffering in the kernel that you shouldn't
lose anything.

[toc] | [prev] | [next] | [standalone]

#109300

jladasky@itu.edu wrote:
> So, how can I take the byte sequence <0x01 0x02 0x03 0x04 0x05 0x06 \n> that
> Serial.readline() returns to me,

Using readline() to read binary data doesn't sound like
a good idea -- what happens if one of the data bytes
happens to be 0x0a?

If you're going binary, it would be better to ditch the
newlines and read a fixed number of bytes each time.

Regarding speed, if struct.unpack or numpy isn't fast
enough, you may need to deal with the data in bigger
chunks.

Although if you're not doing a huge amount of processing
with them, I'd expect it to be plenty fast enough.
I just did a quick test, and I was able to unpack
about 700,000 random 6-byte strings per second on
a 2.8GHz Xeon.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#109314

On Tuesday, May 31, 2016 at 9:37:18 PM UTC-7, Gregory Ewing wrote:
> > So, how can I take the byte sequence <0x01 0x02 0x03 0x04 0x05 0x06 \n> that
> > Serial.readline() returns to me,
> 
> Using readline() to read binary data doesn't sound like
> a good idea -- what happens if one of the data bytes
> happens to be 0x0a?
> 
> If you're going binary, it would be better to ditch the
> newlines and read a fixed number of bytes each time.

Hi Greg,

Of course you're correct.  In my very first version of this system, I piggybacked the transmission of data on the routines that I used to transmit text.  That won't be a permanent state of affairs.

I guess that I should have posted my example string without the newline, which I will of course strip off before parsing the string into 16-bit integers.  Also, now that I've verified that the CPU's at both ends of the line are little-endian, I should have swapped every other byte.  :^)

One common data transmission error I've seen in other systems is added/dropped bytes.  I'm not sure whether to expect this problem with a USB/UART setup.  I may add a CRC-8 error-checking byte in place of the newline.

> Regarding speed, if struct.unpack or numpy isn't fast
> enough, you may need to deal with the data in bigger
> chunks.
> 
> Although if you're not doing a huge amount of processing
> with them, I'd expect it to be plenty fast enough.
> I just did a quick test, and I was able to unpack
> about 700,000 random 6-byte strings per second on
> a 2.8GHz Xeon.

That's an encouraging speed.  I'm sure that I'm not doing everything as efficiently as possible yet.

I've done some Multiprocessing work before.  To speed things up further, I might implement the live graphics in one process and the I/O in another.

[toc] | [prev] | [next] | [standalone]

#109338

jladasky@itu.edu wrote:
> One common data transmission error I've seen in other systems is
> added/dropped bytes. I may add a CRC-8 error-checking byte in place of the
> newline.

Also maybe add a start byte with a known value at the
beginning of each packet to help resynchronise if you
get out of step.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#109339

On Thu, 02 Jun 2016 18:50:34 +1200, Gregory Ewing wrote:

> jladasky@itu.edu wrote:
>> One common data transmission error I've seen in other systems is
>> added/dropped bytes. I may add a CRC-8 error-checking byte in place of
>> the newline.
> 
> Also maybe add a start byte with a known value at the beginning of each
> packet to help resynchronise if you get out of step.

No maybe about it
if you are sending a binary stream you need to be able to reliably signal 
the start AND finish of the data stream (either send the length in the 
message start or have a fixed msg. length)

after a lot of experimenting to ensure reliability you will probably have 
reinvented something like intelhex or x-modem



-- 
The work [of software development] is becoming far easier (i.e. the tools
we're using work at a higher level, more removed from machine, peripheral
and operating system imperatives) than it was twenty years ago, and 
because
of this, knowledge of the internals of a system may become less 
accessible.
We may be able to dig deeper holes, but unless we know how to build taller
ladders, we had best hope that it does not rain much.
		-- Paul Licker

[toc] | [prev] | [next] | [standalone]

#109349

On Thursday 02 June 2016 04:13:51 alister wrote:

> On Thu, 02 Jun 2016 18:50:34 +1200, Gregory Ewing wrote:
> > jladasky@itu.edu wrote:
> >> One common data transmission error I've seen in other systems is
> >> added/dropped bytes. I may add a CRC-8 error-checking byte in place
> >> of the newline.
> >
> > Also maybe add a start byte with a known value at the beginning of
> > each packet to help resynchronise if you get out of step.
>
> No maybe about it
> if you are sending a binary stream you need to be able to reliably
> signal the start AND finish of the data stream (either send the length
> in the message start or have a fixed msg. length)
>
> after a lot of experimenting to ensure reliability you will probably
> have reinvented something like intelhex or x-modem
>
Neither of which can handle that last packet well unless the last packet 
is padded out to be a fill packet and the filler bytes thrown away in 
the receiver.

For that reason alone, zmodem wins because it does both..

zmodem in its present linux implementation must have in window size set 
explicitely on the invocation command line to match the default packet 
size of the receiving device, which is usually the size of its disk 
sector.  That way, when its talking to rzsz on a 256 byte sector box, it 
will be forced to do the checksum checks to match what rzsz can do.

Otherwise the linux version of zmodem will only check the checksums every 
8 kilobytes.  That of course will fail if the line speed overruns the 
receiver, and of course the receiver is sending error restart requests, 
not a good way to make any real speed.  My target box is running nitros9 
and somehow bit rot has destroyed the 7 wire protocol flow controls, so 
I have to restrict the data rate to 4800 baud.  Doing both the window 
size limit and the low baud rate, I have moved an 80 track disk image 
built on this machine, to that machine without any errors several times. 
Slow, and hard on the coffee pot staying awake while effectively 
watching paint dry, but it worked.

Now we have a better method that works at 112 kbaud, much better.  You 
can load a program direct from a virtual disk image "mounted" for use 
with disk descriptors that point at the disk image on this hard drive at 
a load speed approaching loading it from a real floppy on that machine. 

> The work [of software development] is becoming far easier (i.e. the
> tools we're using work at a higher level, more removed from machine,
> peripheral and operating system imperatives) than it was twenty years
> ago, and because
> of this, knowledge of the internals of a system may become less
> accessible.
> We may be able to dig deeper holes, but unless we know how to build
> taller ladders, we had best hope that it does not rain much.
> 		-- Paul Licker

Paul is absolutely correct.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[toc] | [prev] | [next] | [standalone]

#109361

On Thu, 02 Jun 2016 05:41:40 -0400, Gene Heskett wrote:

> On Thursday 02 June 2016 04:13:51 alister wrote:
> 
>> On Thu, 02 Jun 2016 18:50:34 +1200, Gregory Ewing wrote:
>> > jladasky@itu.edu wrote:
>> >> One common data transmission error I've seen in other systems is
>> >> added/dropped bytes. I may add a CRC-8 error-checking byte in place
>> >> of the newline.
>> >
>> > Also maybe add a start byte with a known value at the beginning of
>> > each packet to help resynchronise if you get out of step.
>>
>> No maybe about it if you are sending a binary stream you need to be
>> able to reliably signal the start AND finish of the data stream (either
>> send the length in the message start or have a fixed msg. length)
>>
>> after a lot of experimenting to ensure reliability you will probably
>> have reinvented something like intelhex or x-modem
>>
> Neither of which can handle that last packet well unless the last packet
> is padded out to be a fill packet and the filler bytes thrown away in
> the receiver.

The examples quoted were for examples of binary transfer protocols & not 
intended to be an exhaustive list.

> 
>> The work [of software development] is becoming far easier (i.e. the
>> tools we're using work at a higher level, more removed from machine,
>> peripheral and operating system imperatives) than it was twenty years
>> ago, and because of this, knowledge of the internals of a system may
>> become less accessible.
>> We may be able to dig deeper holes, but unless we know how to build
>> taller ladders, we had best hope that it does not rain much.
>> 		-- Paul Licker
> 
> Paul is absolutely correct.
> 
> Cheers, Gene Heskett
^^^^ like that quote




-- 
Under deadline pressure for the next week.  If you want something, it can 
wait.
Unless it's blind screaming paroxysmally hedonistic...

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web