Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #27880 > unrolled thread

issue with struct.unpack

Back to article view | Back to comp.lang.python

Contents

  issue with struct.unpack 9bizy <a.m.akingbulu-11@student.lboro.ac.uk> - 2012-08-25 11:34 -0700
    Re: issue with struct.unpack Mark Lawrence <breamoreboy@yahoo.co.uk> - 2012-08-25 20:12 +0100
    Re: issue with struct.unpack MRAB <python@mrabarnett.plus.com> - 2012-08-25 20:16 +0100
      Re: issue with struct.unpack 9bizy <a.m.akingbulu-11@student.lboro.ac.uk> - 2012-08-28 15:35 -0700
        Re: issue with struct.unpack MRAB <python@mrabarnett.plus.com> - 2012-08-28 23:53 +0100
        Re: issue with struct.unpack Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-08-28 22:40 -0400
      Re: issue with struct.unpack 9bizy <a.m.akingbulu-11@student.lboro.ac.uk> - 2012-08-28 15:35 -0700
    Re: issue with struct.unpack Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-08-25 19:31 -0400
      Re: issue with struct.unpack Alexander Blinne <news@blinne.net> - 2012-08-26 14:53 +0200
    Re: issue with struct.unpack MRAB <python@mrabarnett.plus.com> - 2012-08-28 23:49 +0100
      Re: issue with struct.unpack 9bizy <a.m.akingbulu-11@student.lboro.ac.uk> - 2012-08-28 16:01 -0700
      Re: issue with struct.unpack MRAB <python@mrabarnett.plus.com> - 2012-08-29 00:36 +0100
        Re: issue with struct.unpack 9bizy <a.m.akingbulu-11@student.lboro.ac.uk> - 2012-08-28 16:55 -0700
        Re: issue with struct.unpack 9bizy <a.m.akingbulu-11@student.lboro.ac.uk> - 2012-08-28 16:55 -0700
    Re: issue with struct.unpack Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2012-08-28 22:31 -0400

#27880 — issue with struct.unpack

I am trying to unpack values from sensor data I am retrieving through a serial cable, but I get errors while using struct.unpack, how can I use struct.unpack to unload the data in a readable format?

I checked the python documentation for struct and I can seen to find any argument for this.

I have data = struct.unpack('char',data) but I still get errors

[toc] | [next] | [standalone]

#27882

On 25/08/2012 19:34, 9bizy wrote:
> I am trying to unpack values from sensor data I am retrieving through a serial cable, but I get errors while using struct.unpack, how can I use struct.unpack to unload the data in a readable format?
>
> I checked the python documentation for struct and I can seen to find any argument for this.
>
> I have data = struct.unpack('char',data) but I still get errors
>

We have two options here.  Either

a) People reading your original request go on a mind reading course or 
similar in an attempt to find out what the errors are, though I'm 
confused as to how you get errors from one line of code.

Or

b) Provide the smallest sample of code that allows the problem to be 
reproduced together with the complete traceback so we can see exactly 
what happened.

-- 
Cheers.

Mark Lawrence.

[toc] | [prev] | [next] | [standalone]

#27883

On 25/08/2012 19:34, 9bizy wrote:
> I am trying to unpack values from sensor data I am retrieving through
> a serial cable, but I get errors while using struct.unpack, how can I
> use struct.unpack to unload the data in a readable format?
>
> I checked the python documentation for struct and I can seen to find
> any argument for this.
>
> I have data = struct.unpack('char',data) but I still get errors
>
The format strings are described here for Python 3:

     http://docs.python.org/3.2/library/struct.html

and here for Python 2:

     http://docs.python.org/2.7/library/struct.html

[toc] | [prev] | [next] | [standalone]

#28027

On Saturday, 25 August 2012 20:16:54 UTC+1, MRAB  wrote:
> On 25/08/2012 19:34, 9bizy wrote:
> 
> > I am trying to unpack values from sensor data I am retrieving through
> 
> > a serial cable, but I get errors while using struct.unpack, how can I
> 
> > use struct.unpack to unload the data in a readable format?
> 
> >
> 
> > I checked the python documentation for struct and I can seen to find
> 
> > any argument for this.
> 
> >
> 
> > I have data = struct.unpack('char',data) but I still get errors
> 
> >
> 
> The format strings are described here for Python 3:
> 
> 
> 
>      http://docs.python.org/3.2/library/struct.html
> 
> 
> 
> and here for Python 2:
> 
> 
> 
>      http://docs.python.org/2.7/library/struct.html

I used this documents but they do not explain or provide an example on how to use struct.unpack for sensor data from an external source or even data from a excel sheet.

[toc] | [prev] | [next] | [standalone]

#28030

On 28/08/2012 23:35, 9bizy wrote:
> On Saturday, 25 August 2012 20:16:54 UTC+1, MRAB  wrote:
>> On 25/08/2012 19:34, 9bizy wrote:
>>
>> > I am trying to unpack values from sensor data I am retrieving through
>>
>> > a serial cable, but I get errors while using struct.unpack, how can I
>>
>> > use struct.unpack to unload the data in a readable format?
>>
>> >
>>
>> > I checked the python documentation for struct and I can seen to find
>>
>> > any argument for this.
>>
>> >
>>
>> > I have data = struct.unpack('char',data) but I still get errors
>>
>> >
>>
>> The format strings are described here for Python 3:
>>
>>
>>
>>      http://docs.python.org/3.2/library/struct.html
>>
>>
>>
>> and here for Python 2:
>>
>>
>>
>>      http://docs.python.org/2.7/library/struct.html
>
> I used this documents but they do not explain or provide an example on how to use struct.unpack for sensor data from an external source or even data from a excel sheet.
>
If you want to read from an Excel file you should be using the 'xlrd'
module. You can find it here: http://www.python-excel.org/

[toc] | [prev] | [next] | [standalone]

#28039

On Tue, 28 Aug 2012 15:35:58 -0700 (PDT), 9bizy
<a.m.akingbulu-11@student.lboro.ac.uk> declaimed the following in
gmane.comp.python.general:

> 
> I used this documents but they do not explain or provide an example on how to use struct.unpack for sensor data from an external source or even data from a excel sheet.

	Why should it?... struct doesn't care WHERE the data came from; it
just takes a /string of bytes/ and interprets the contents as a series
of raw binary fields based upon the format character.

>>> import struct
>>> data = "elevenbytes"
>>> struct.unpack("!B4HH", data)
(101, 27749, 30309, 28258, 31092, 25971)
>>> ndata = [30, 138, 32260, 563, 3238, 23537]
>>> struct.pack("!B4HH", *ndata)
'\x1e\x00\x8a~\x04\x023\x0c\xa6[\xf1'
>>> 
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#28028

On Saturday, 25 August 2012 20:16:54 UTC+1, MRAB  wrote:
> On 25/08/2012 19:34, 9bizy wrote:
> 
> > I am trying to unpack values from sensor data I am retrieving through
> 
> > a serial cable, but I get errors while using struct.unpack, how can I
> 
> > use struct.unpack to unload the data in a readable format?
> 
> >
> 
> > I checked the python documentation for struct and I can seen to find
> 
> > any argument for this.
> 
> >
> 
> > I have data = struct.unpack('char',data) but I still get errors
> 
> >
> 
> The format strings are described here for Python 3:
> 
> 
> 
>      http://docs.python.org/3.2/library/struct.html
> 
> 
> 
> and here for Python 2:
> 
> 
> 
>      http://docs.python.org/2.7/library/struct.html

I used this documents but they do not explain or provide an example on how to use struct.unpack for sensor data from an external source or even data from a excel sheet.

[toc] | [prev] | [next] | [standalone]

#27890

On Sat, 25 Aug 2012 11:34:39 -0700 (PDT), 9bizy
<a.m.akingbulu-11@student.lboro.ac.uk> declaimed the following in
gmane.comp.python.general:

> I am trying to unpack values from sensor data I am retrieving through a serial cable, but I get errors while using struct.unpack, how can I use struct.unpack to unload the data in a readable format?
> 
> I checked the python documentation for struct and I can seen to find any argument for this.
>
	Pardon "... I can seen to find..."?
 
> I have data = struct.unpack('char',data) but I still get errors

	No surprise... You've asked for a single-character, a short integer,
and two unknown field types.

	What about

"""
s char[] string 
"""

and

"""
For the "s" format character, the count is interpreted as the size of
the string, not a repeat count like for the other format characters; for
example, '10s' means a single 10-byte string, while '10c' means 10
characters. For packing, the string is truncated or padded with null
bytes as appropriate to make it fit. For unpacking, the resulting string
always has exactly the specified number of bytes. As a special case,
'0s' means a single, empty string (while '0c' means 0 characters). 
"""

(both from the 2.5 help file).

The struct module relies upon the user knowing the format of the data.
If your problem is that you have some null-terminated string data in a
variable width field, you will have to locate the position of the null
FIRST, and specify the appropriate "count" for the s format.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#27917

On 26.08.2012 01:31, Dennis Lee Bieber wrote:
> The struct module relies upon the user knowing the format of the data.
> If your problem is that you have some null-terminated string data in a
> variable width field, you will have to locate the position of the null
> FIRST, and specify the appropriate "count" for the s format.

This gave me the idea of an Enhancement of the Struct class with an
additional format character (perhaps 'n') which corresponds to a
null-terminated string:

-----
# -*- coding: utf-8 -*-

import struct

class Nstr(object):
    def __init__(self, ncount):
        self.ncount = ncount

    def unpack(self, s):
        s = s.split('\0')
        return s[:self.ncount], '\0'.join(s[self.ncount:])

    def pack(self, *s):
        if len(s)!=self.ncount:
            raise ValueError
        for st in s:
            if '\0' in st:
                raise ValueError
        return '\0'.join(s)+'\0'

    def __repr__(self):
        return 'Nstr('+repr(self.ncount)+')'

class NStruct(object):
    def __init__(self, format):
        self.format = format
        if format[0] in '!=<>@':
            self.endianness = format[0]
            format = format[1:]
        else:
            self.endianness = ''
        self.chunks = []
        while len(format)>0:
            j = format.find('n')
            if j > -1:
                k = j-1
                while format[k].isdigit():
                    k-=1
                chunkformat, ncount, format = format[:k+1],\
                format[k+1:j], format[j+1:]
                ncount = 1 if len(ncount)==0 else int(ncount)
            else:
                chunkformat, ncount, format = format, '', ''
                ncount = 0
            stru = struct.Struct(self.endianness+chunkformat)
            l = len(stru.unpack("0"*stru.size))
            self.chunks.append((stru, l))
            if ncount > 0:
                self.chunks.append((Nstr(ncount), ncount))

    def unpack(self, data):
        res = []
        for sth, n in self.chunks:
            if isinstance(sth, struct.Struct):
                chunk, data = data[:sth.size], data[sth.size:]
                res.extend(sth.unpack(chunk))
            elif isinstance(sth, Nstr):
                chunk, data = sth.unpack(data)
                res.extend(chunk)
        return res

    def pack(self, *data):
        res = []
        for sth, n in self.chunks:
            chunk, data = data[:n], data[n:]
            res.append(sth.pack(*chunk))
        return ''.join(res)

    def __repr__(self):
        return 'NStruct('+repr(self.format)+')'

if __name__=="__main__":
    a = NStruct('h b 2n 2h')
    print repr(a)
    d = 'asdblah blah\0haha\0asdf'
    r = a.unpack(d)
    assert r == [29537, 100, 'blah blah', 'haha', 29537, 26212]
    print repr(d), repr(r)
    dd = a.pack(*r)
    print r, repr(dd)
    assert dd == d

-----

beware of bugs in the above code, i haven't testet it much yet.

Alex

[toc] | [prev] | [next] | [standalone]

#28029

On 28/08/2012 23:34, 9bizy wrote:
> This is what I have to reproduce the challenge I am having below:
>
>
> import csv
> import struct
>
>
> data = []
>
> for Node in csv.reader(file('s_data.xls')):

That tries to read the file as CSV, but, judging from the extension,
it's in Excel's format. You don't even use what is read, i.e. Node.

>      data.append(list((file('s_data.xls'))))
>
That opens the file again and 'list' causes it to read the file as
though it were a series of lines in a text file, which, as I've said,
it looks like it isn't. The list of 'lines' is appended to the list
'data', so that's a list of lists.
>
>      data = struct.unpack('!B4HH', data)
>      print "s_data.csv: ", data
>
> I tries so many format for the struct.unpack but I got this errors:
>
> Traceback (most recent call last):
>
>      data = struct.unpack('!B4HH', data)
> struct.error: unpack requires a string argument of length 11
>
[snip]
It's complaining because it's expecting a string argument but you're
giving it a list instead.

[toc] | [prev] | [next] | [standalone]

#28031

On Tuesday, 28 August 2012 23:49:54 UTC+1, MRAB  wrote:
> On 28/08/2012 23:34, 9bizy wrote:
> 
> > This is what I have to reproduce the challenge I am having below:
> 
> >
> 
> >
> 
> > import csv
> 
> > import struct
> 
> >
> 
> >
> 
> > data = []
> 
> >
> 
> > for Node in csv.reader(file('s_data.xls')):
> 
> 
> 
> That tries to read the file as CSV, but, judging from the extension,
> 
> it's in Excel's format. You don't even use what is read, i.e. Node.
> 
> 
> 
> >      data.append(list((file('s_data.xls'))))
> 
> >
> 
> That opens the file again and 'list' causes it to read the file as
> 
> though it were a series of lines in a text file, which, as I've said,
> 
> it looks like it isn't. The list of 'lines' is appended to the list
> 
> 'data', so that's a list of lists.
> 
> >
> 
> >      data = struct.unpack('!B4HH', data)
> 
> >      print "s_data.csv: ", data
> 
> >
> 
> > I tries so many format for the struct.unpack but I got this errors:
> 
> >
> 
> > Traceback (most recent call last):
> 
> >
> 
> >      data = struct.unpack('!B4HH', data)
> 
> > struct.error: unpack requires a string argument of length 11
> 
> >
> 
> [snip]
> 
> It's complaining because it's expecting a string argument but you're
> 
> giving it a list instead.

How do I then convert data to a string argument in this case?

[toc] | [prev] | [next] | [standalone]

#28033

On 29/08/2012 00:01, 9bizy wrote:> On Tuesday, 28 August 2012 23:49:54 
UTC+1, MRAB  wrote:
 >> On 28/08/2012 23:34, 9bizy wrote:
 >> > This is what I have to reproduce the challenge I am having below:
 >> >
 >> > import csv
 >> > import struct
 >> >
 >> > data = []
 >> >
 >> > for Node in csv.reader(file('s_data.xls')):
 >>
 >> That tries to read the file as CSV, but, judging from the extension,
 >> it's in Excel's format. You don't even use what is read, i.e. Node.
 >>
 >> >      data.append(list((file('s_data.xls'))))
 >> >
 >> That opens the file again and 'list' causes it to read the file as
 >> though it were a series of lines in a text file, which, as I've said,
 >> it looks like it isn't. The list of 'lines' is appended to the list
 >> 'data', so that's a list of lists.
 >> >
 >> >      data = struct.unpack('!B4HH', data)
 >> >      print "s_data.csv: ", data
 >> >
 >> > I tries so many format for the struct.unpack but I got this errors:
 >> >
 >> > Traceback (most recent call last):
 >> >
 >> >      data = struct.unpack('!B4HH', data)
 >> > struct.error: unpack requires a string argument of length 11
 >> >
 >> [snip]
 >> It's complaining because it's expecting a string argument but you're
 >> giving it a list instead.
 >
 > How do I then convert data to a string argument in this case?
 >
The question is: what are you trying to do?

If you're trying to read an Excel file, then you should be trying the
'xlrd' module. You can find it here: http://www.python-excel.org/

If your trying to 'decode' a binary file, then you should open it in
binary mode (with "rb"), read (some of) it as a byte string and then
pass it to struct.unpack.

[toc] | [prev] | [next] | [standalone]

#28034

On Wednesday, 29 August 2012 00:36:40 UTC+1, MRAB  wrote:
> On 29/08/2012 00:01, 9bizy wrote:> On Tuesday, 28 August 2012 23:49:54 
> 
> UTC+1, MRAB  wrote:
> 
>  >> On 28/08/2012 23:34, 9bizy wrote:
> 
>  >> > This is what I have to reproduce the challenge I am having below:
> 
>  >> >
> 
>  >> > import csv
> 
>  >> > import struct
> 
>  >> >
> 
>  >> > data = []
> 
>  >> >
> 
>  >> > for Node in csv.reader(file('s_data.xls')):
> 
>  >>
> 
>  >> That tries to read the file as CSV, but, judging from the extension,
> 
>  >> it's in Excel's format. You don't even use what is read, i.e. Node.
> 
>  >>
> 
>  >> >      data.append(list((file('s_data.xls'))))
> 
>  >> >
> 
>  >> That opens the file again and 'list' causes it to read the file as
> 
>  >> though it were a series of lines in a text file, which, as I've said,
> 
>  >> it looks like it isn't. The list of 'lines' is appended to the list
> 
>  >> 'data', so that's a list of lists.
> 
>  >> >
> 
>  >> >      data = struct.unpack('!B4HH', data)
> 
>  >> >      print "s_data.csv: ", data
> 
>  >> >
> 
>  >> > I tries so many format for the struct.unpack but I got this errors:
> 
>  >> >
> 
>  >> > Traceback (most recent call last):
> 
>  >> >
> 
>  >> >      data = struct.unpack('!B4HH', data)
> 
>  >> > struct.error: unpack requires a string argument of length 11
> 
>  >> >
> 
>  >> [snip]
> 
>  >> It's complaining because it's expecting a string argument but you're
> 
>  >> giving it a list instead.
> 
>  >
> 
>  > How do I then convert data to a string argument in this case?
> 
>  >
> 
> The question is: what are you trying to do?
> 
> 
> 
> If you're trying to read an Excel file, then you should be trying the
> 
> 'xlrd' module. You can find it here: http://www.python-excel.org/
> 
> 
> 
> If your trying to 'decode' a binary file, then you should open it in
> 
> binary mode (with "rb"), read (some of) it as a byte string and then
> 
> pass it to struct.unpack.

Thank you MRAB this was helpful.

[toc] | [prev] | [next] | [standalone]

#28035

On Wednesday, 29 August 2012 00:36:40 UTC+1, MRAB  wrote:
> On 29/08/2012 00:01, 9bizy wrote:> On Tuesday, 28 August 2012 23:49:54 
> 
> UTC+1, MRAB  wrote:
> 
>  >> On 28/08/2012 23:34, 9bizy wrote:
> 
>  >> > This is what I have to reproduce the challenge I am having below:
> 
>  >> >
> 
>  >> > import csv
> 
>  >> > import struct
> 
>  >> >
> 
>  >> > data = []
> 
>  >> >
> 
>  >> > for Node in csv.reader(file('s_data.xls')):
> 
>  >>
> 
>  >> That tries to read the file as CSV, but, judging from the extension,
> 
>  >> it's in Excel's format. You don't even use what is read, i.e. Node.
> 
>  >>
> 
>  >> >      data.append(list((file('s_data.xls'))))
> 
>  >> >
> 
>  >> That opens the file again and 'list' causes it to read the file as
> 
>  >> though it were a series of lines in a text file, which, as I've said,
> 
>  >> it looks like it isn't. The list of 'lines' is appended to the list
> 
>  >> 'data', so that's a list of lists.
> 
>  >> >
> 
>  >> >      data = struct.unpack('!B4HH', data)
> 
>  >> >      print "s_data.csv: ", data
> 
>  >> >
> 
>  >> > I tries so many format for the struct.unpack but I got this errors:
> 
>  >> >
> 
>  >> > Traceback (most recent call last):
> 
>  >> >
> 
>  >> >      data = struct.unpack('!B4HH', data)
> 
>  >> > struct.error: unpack requires a string argument of length 11
> 
>  >> >
> 
>  >> [snip]
> 
>  >> It's complaining because it's expecting a string argument but you're
> 
>  >> giving it a list instead.
> 
>  >
> 
>  > How do I then convert data to a string argument in this case?
> 
>  >
> 
> The question is: what are you trying to do?
> 
> 
> 
> If you're trying to read an Excel file, then you should be trying the
> 
> 'xlrd' module. You can find it here: http://www.python-excel.org/
> 
> 
> 
> If your trying to 'decode' a binary file, then you should open it in
> 
> binary mode (with "rb"), read (some of) it as a byte string and then
> 
> pass it to struct.unpack.

Thank you MRAB this was helpful.

[toc] | [prev] | [next] | [standalone]

#28038

On Tue, 28 Aug 2012 15:34:11 -0700 (PDT), 9bizy
<a.m.akingbulu-11@student.lboro.ac.uk> declaimed the following in
gmane.comp.python.general:

> 
> for Node in csv.reader(file('s_data.xls')):
>     data.append(list((file('s_data.xls'))))
>     
>     
>     data = struct.unpack('!B4HH', data)
>     print "s_data.csv: ", data
> 
> I tries so many format for the struct.unpack but I got this errors:
> 
> Traceback (most recent call last):
>   
>     data = struct.unpack('!B4HH', data)
> struct.error: unpack requires a string argument of length 11
>

	Which seems reasonable -- you've specified 1 byte, and 5 16-bit
integers => 11 bytes needed for input.

	Have you examined what your input really contains? I see a list
given the combination of .append() and list()
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
        wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web