numpy question (fairly basic, I think)

Path	csiph.com!usenet.pasdenom.info!news.franciliens.net!fdn.fr!usenet-fr.net!nerim.net!novso.com!newsfeed.xs4all.nl!newsfeed4a.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path	<fomcl@yahoo.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.047
X-Spam-Evidence	'H': 0.91; 'S': 0.00; "'',": 0.07; 'binary': 0.07; 'initialize': 0.07; "'')": 0.09; 'compact': 0.09; 'repeated': 0.09; 'rows': 0.09; 'to)': 0.09; 'subject:question': 0.10; '(1,': 0.16; '__future__': 0.16; 'advance!': 0.16; 'numpy': 0.16; 'record,': 0.16; 'tuple': 0.16; 'unpacked': 0.16; 'unpacking)': 0.16; "skip:' 30": 0.19; 'import': 0.22; 'this?': 0.23; 'record': 0.27; 'array': 0.29; 'skip:q 20': 0.31; 'probably': 0.32; 'subject: (': 0.35; 'something': 0.35; 'etc': 0.35; 'but': 0.35; 'there': 0.35; 'done': 0.36; 'hi,': 0.36; 'system,': 0.38; 'thank': 0.38; 'filled': 0.38; 'to:addr:python- list': 0.38; 'to:addr:python.org': 0.39; 'read': 0.60; 'received:67.195': 0.60; 'received:98.137': 0.60; 'simple,': 0.60; 'new': 0.61; 'skip:n 10': 0.64; 'more': 0.64; 'header:Reply-To:1': 0.67; 'received:gq1.yahoo.com': 0.68; 'received:mail.gq1.yahoo.com': 0.68; 'apart': 0.72; 'records': 0.73; 'skip:n 40': 0.81; '4.0,': 0.84; 'received:bullet.mail.gq1.yahoo.com': 0.84; 'subject:think': 0.84; 'us?': 0.84; 'medicine,': 0.91; 'education,': 0.96
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1418506751; bh=vzqfF/u3fx2AHYRFLtVKHabQ97y99fWikXwK8s55OnQ=; h=Date:From:Reply-To:To:Subject:From:Subject; b=CTEJVqCnDf81HZJsUc+GWVaaEMHvAnqFsi+7oxlN5P+XKT8WEw86LBGvAErKFrjayiCOyIezvFKET7aHoH2idW3xkLmXr2XkAQJ5x5SIWRsbq5SA+DjAnAcXPfjAo3sB2rua/RCFHRpgB+pJDYg9g6eByRHxh6IHXyh7JuNdUteNWm3I175YNNq1go2xqK91ZIAv9y9neuhAFofPcl6/Az9aQ9wy/HXZBXtZ/c6FX+tjhMkXWCN39QyKvwccPGUPvc431nAHNZDqx2V4S0cqaEcJQDCQZd6Oo+R4grx4Su/LZWCmGrSz4wZ0kqTGrsfa7zc0fs7FcYzUy4g30SSQLA==
DomainKey-Signature	a=rsa-sha1; q=dns; c=nofws; s=s2048; d=yahoo.com; b=i1z7tmcYv7HOA3blW0Srs0qMFCbrLApMqdVkxFee7Nm7egJp2qIqj8oe4bnx2jEoJOE54ysymDJRAeb5OXVUkJ1LX5/mTJlLr8VnBwXlhV7/lLvWl5fd9b5pozBroy/OmHZ97AhPFe1rHeT6tJFPoJo5puiiR1LyIhowzcYZML7bytSw1DgRqb3DdqwFL9bhBx3xapT7r4x6ZmXlmDMOUYktIhWDWFt0p+qQtCw73pfFjhYMdKK0ddWMBevw2btcUNa79DpYPtVwcXbxc54RTyNmv7CHxcH+xNpnFJU2XA9qKW179BNHJQhWdArqbRIbmu0gc7fhzv1OUhjbwb67kw==;
X-Yahoo-Newman-Property	ymail-4
X-Yahoo-Newman-Id	101213.29426.bm@omp1002.mail.gq1.yahoo.com
X-YMail-OSG	3r8XHzYVM1ld31OSSPBKOXyuOHSB15J8PeGK9orCL2ziRzFitkXl6pQ78sqhx89 Q0q7yTsfLwruIw_jH1.6TYnYw.Q29G_hgVsZYNPWCavFwVS3zwlqGZYHGNmNxrpI6DBKI9V9gt3Z oWg132MQTRC62bLTc6eLyHPQVugaosgf9e_1iTBEPUKbAL0OFwXQAfB_.rdUjxXFdcVFb5Wxrs7u 7w6xP_6vKNWaR.CafKrMFk0Sp68135fDIhwDHPVATeorBsezTzuhIJ6L3QinBG8zXknKKzJqnMXP hppPmkIjLOyMnFNyKMqdXFAkormRPCFfBrhnv6IuA_1IigGh7pnG2h3imGYmTDpWIGLt5i91yLso oz9n0.OP3j9Qx1_WSUkRKfo1oaoS4LIPt0Ry5sGAXZWq5vR65AWkJvlBoZ2AYx2fohPa8NasfwBb 4e_kRx8X.5xssfyS0WGjpXr4Ic5L5OiGJkmCRZsJ_7cHyOon1awFoCzHcPyGNtKF11GhYbKj_jmW _gfE_qC34MxKDwg--
Date	Sat, 13 Dec 2014 21:36:10 +0000 (UTC)
From	Albert-Jan Roskam <fomcl@yahoo.com>
To	Python <python-list@python.org>
Subject	numpy question (fairly basic, I think)
MIME-Version	1.0
Content-Type	text/plain; charset=UTF-8
Content-Transfer-Encoding	7bit
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
Reply-To	Albert-Jan Roskam <fomcl@yahoo.com>
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.16937.1418506920.18130.python-list@python.org> (permalink)
Lines	74
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1418506920 news.xs4all.nl 2854 [2001:888:2000:d::a6]:33879
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:82488

Show key headers only | View raw

Hi,

I am new to numpy. I am reading binary data one record at a time (I have to) and I would like to store all the records in a numpy array which I pre-allocate. Below I try to fill the empty array with exactly one record, but it is filled with as many rows as there are columns. Why is this? It is probably something simple, but I am stuck! It is like the original record is not unpacked *as in tuple unpacking) into the array, so it remains one chunk, not an (nrows, ncols) structure.


from __future__ import print_function
import numpy as np


# one binary record
s = '\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00;@\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x80U\xe1@\x00\x00\x00\x00\x80\xd9\xe4@\x00\x00\x00\x00@\xa7\xe3@\xab\xaa\xaa\xaajG\xe3@\x00\x00\x00\x00\x80\xd9\xe4@\x00\x00\x00\x00\x00\x00;@\x00\x00\x00\x00\x00\x00;@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\xa4DI\tBx       qwertyuiopasdfghjklzxcvbnm,./                                                                           \x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00p\x9f@\x00\x00\x00\x00\x00\x00\x10@\x00\x00\x00\x00\x00\x00(@DEC 2012'


# read it into a structured array
formats = ['<d', '<d', '<d', '<d', '<d', '<d', '<d', '<d', '<d', '<d', '<d', '<d', '<d', '<d', '<d', 'a8', 'a104', '<d', '<d', '<d', '<d', 'a8']
names = ["v%02d" % i for i in range(len(formats))]
dt = np.dtype({'formats': formats, "names": names})
record = np.fromstring(s, dtype=dt)


# make it more compact
trunc_formats = ['f4', 'f4', 'f4', 'f4', 'f4', 'f4', 'f4', 'f4', 'f4', 'f4', 'f4', 'f4', 'f4', 'f4', 'f4', 'a1', 'a100', 'f4', 'f4', 'f4', 'f4', 'a8']
trunc_dt = np.dtype({'formats': trunc_formats, "names": names})
record = record.astype(trunc_dt)
print(record.shape)  # (1,), but it needs to be (1, 22)??
#record = np.asarray(*tuple(record.astype(trunc_dt)), dtype=np.object)
#record = np.expand_dims(record, axis=0)


# initialize an empty array and fill it with one record

nrows = 50  # arbitrary number
ncols = len(formats) # 22

array = np.zeros((nrows, ncols), trunc_dt)
array[0,:] = record
print(array)

# output: why is the record repeated ncols times?
[[ (1.0, 27.0, 1.0, 35500.0, 42700.0, 40250.0, 39483.33203125, 42700.0, 27.0, 27.0, 0.0, 1.0, 1.0, 1.0, 13575427072.0, 'x', 'qwertyuiopasdfghjklzxcvbnm,./                                                                       ', 1.0, 2012.0, 4.0, 12.0, 'DEC 2012')
(1.0, 27.0, 1.0, 35500.0, 42700.0, 40250.0, 39483.33203125, 42700.0, 27.0, 27.0, 0.0, 1.0, 1.0, 1.0, 13575427072.0, 'x', 'qwertyuiopasdfghjklzxcvbnm,./                                                                       ', 1.0, 2012.0, 4.0, 12.0, 'DEC 2012')
(1.0, 27.0, 1.0, 35500.0, 42700.0, 40250.0, 39483.33203125, 42700.0, 27.0, 27.0, 0.0, 1.0, 1.0, 1.0, 13575427072.0, 'x', 'qwertyuiopasdfghjklzxcvbnm,./                                                                       ', 1.0, 2012.0, 4.0, 12.0, 'DEC 2012')
...,
(1.0, 27.0, 1.0, 35500.0, 42700.0, 40250.0, 39483.33203125, 42700.0, 27.0, 27.0, 0.0, 1.0, 1.0, 1.0, 13575427072.0, 'x', 'qwertyuiopasdfghjklzxcvbnm,./                                                                       ', 1.0, 2012.0, 4.0, 12.0, 'DEC 2012')
(1.0, 27.0, 1.0, 35500.0, 42700.0, 40250.0, 39483.33203125, 42700.0, 27.0, 27.0, 0.0, 1.0, 1.0, 1.0, 13575427072.0, 'x', 'qwertyuiopasdfghjklzxcvbnm,./                                                                       ', 1.0, 2012.0, 4.0, 12.0, 'DEC 2012')
(1.0, 27.0, 1.0, 35500.0, 42700.0, 40250.0, 39483.33203125, 42700.0, 27.0, 27.0, 0.0, 1.0, 1.0, 1.0, 13575427072.0, 'x', 'qwertyuiopasdfghjklzxcvbnm,./                                                                       ', 1.0, 2012.0, 4.0, 12.0, 'DEC 2012')]
[ (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, '', '', 0.0, 0.0, 0.0, 0.0, '')
(0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, '', '', 0.0, 0.0, 0.0, 0.0, '')
(0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, '', '', 0.0, 0.0, 0.0, 0.0, '')
...,
(0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, '', '', 0.0, 0.0, 0.0, 0.0, '')
(0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, '', '', 0.0, 0.0, 0.0, 0.0, '')
(0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, '', '', 0.0, 0.0, 0.0, 0.0, '')]
etc
etc

 
Thank you in advance!


Regards,

Albert-Jan




~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

All right, but apart from the sanitation, the medicine, education, wine, public order, irrigation, roads, a 

fresh water system, and public health, what have the Romans ever done for us?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

numpy question (fairly basic, I think) Albert-Jan Roskam <fomcl@yahoo.com> - 2014-12-13 21:36 +0000
  Re: numpy question (fairly basic, I think) Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-12-14 10:52 +1100
    Re: numpy question (fairly basic, I think) Albert-Jan Roskam <fomcl@yahoo.com> - 2014-12-14 14:31 +0000
      Re: numpy question (fairly basic, I think) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-12-15 11:26 +1300

csiph-web