RE: looping and searching in numpy array

Path	csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From	Albert-Jan Roskam <sjeik_appie@hotmail.com>
Newsgroups	comp.lang.python
Subject	RE: looping and searching in numpy array
Date	Sun, 13 Mar 2016 13:51:23 +0000
Lines	34
Message-ID	<mailman.69.1457877150.12893.python-list@python.org> (permalink)
References	<0fca7583-c051-48bd-a9be-5e501722fc77@googlegroups.com>
Mime-Version	1.0
Content-Type	text/plain; charset="windows-1256"
Content-Transfer-Encoding	base64
X-Trace	news.uni-berlin.de VwbHlqp/ccTTEuARRLpDDQDHw1PTiqS4GdDReVJGknvA==
Return-Path	<sjeik_appie@hotmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.002
X-Spam-Evidence	'H': 1.00; 'S': 0.00; 'yet.': 0.03; 'python)': 0.05; 'below)': 0.07; 'great.': 0.07; 'iterate': 0.09; 'lookup': 0.09; 'python': 0.10; 'anyway': 0.11; 'index': 0.13; 'size,': 0.13; 'thursday,': 0.13; 'thu,': 0.15; "(i'm": 0.16; '2016': 0.16; 'looping': 0.16; 'numpy': 0.16; 'occurence': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; 'subject:array': 0.16; 'subject:looping': 0.16; 'wrote:': 0.16; '>': 0.18; '(in': 0.18; 'all,': 0.20; 'to:name:python-list@python.org': 0.20; 'fix': 0.21; 'to:2**1': 0.21; 'arrays': 0.22; 'suppose': 0.22; 'bigger': 0.23; 'header:In-Reply-To:1': 0.24; 'script': 0.25; 'example': 0.26; 'correct': 0.28; 'search.': 0.29; 'array': 0.29; 'date:': 0.31; 'run': 0.33; 'values.': 0.33; 'subject:': 0.35; 'reply.': 0.35; 'but': 0.36; 'instead': 0.36; 'there': 0.36; 'lines': 0.36; 'basic': 0.36; 'email addr:python.org': 0.36; 'to:addr:python- list': 0.36; 'subject:: ': 0.37; 'really': 0.37; 'two': 0.37; 'thanks': 0.37; 'mean': 0.38; 'hi,': 0.38; 'from:': 0.39; 'skip:e 20': 0.39; 'to:addr:python.org': 0.40; 'still': 0.40; 'your': 0.60; 'email addr:gmail.com': 0.62; 'above,': 0.63; 'necessarily': 0.63; 'times': 0.63; 'march': 0.64; 'mar': 0.65; 'email name :python-list': 0.67; 'charset:windows-1256': 0.67; 'dear': 0.67; 'otten': 0.84; 'peter,': 0.84; 'url:nabble': 0.84
X-TMN	[0JXHRYDdmx2Hrr2EKJUXk7sU0V/caxFd]
X-Originating-Email	[sjeik_appie@hotmail.com]
Importance	Normal
In-Reply-To	<0fca7583-c051-48bd-a9be-5e501722fc77@googlegroups.com>
X-OriginalArrivalTime	13 Mar 2016 13:51:23.0609 (UTC) FILETIME=[6E30EC90:01D17D2F]
X-Content-Filtered-By	Mailman/MimeDel 2.1.21
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.21
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref	csiph.com comp.lang.python:104767

Show key headers only | View raw


> Date: Thu, 10 Mar 2016 08:48:48 -0800
> Subject: Re: looping and searching in numpy array
> From: hemla21@gmail.com
> To: python-list@python.org
> 
> On Thursday, March 10, 2016 at 2:02:57 PM UTC+1, Peter Otten wrote:
> > Heli wrote:
> > 
> > > Dear all,
> > > 
> > > I need to loop over a numpy array and then do the following search. The
> > > following is taking almost 60(s) for an array (npArray1 and npArray2 in
> > > the example below) with around 300K values.
> > > 
> > > 
> > > for id in np.nditer(npArray1):
> > >                   
> > >        newId=(np.where(npArray2==id))[0][0]
> > > 
> > > 
> > > Is there anyway I can make the above faster? I need to run the script
> > > above on much bigger arrays (50M). Please note that my two numpy arrays in
> > > the lines above, npArray1 and npArray2  are not necessarily the same size,
> > > but they are both 1d.
> > 
> > You mean you are looking for the index of the first occurence in npArray2 
> > for every value of npArray1?
> > 
> > I don't know how to do this in numpy (I'm not an expert), but even basic 
> > Python might be acceptable:
> > 
> > lookup = {}
> > for i, v in enumerate(npArray2):
> >     if v not in lookup:
> >         lookup[v] = i
> > 
> > for v in npArray1:
> >     print(lookup.get(v, "<not found>"))
> > 
> > That way you iterate once (in Python) instead of 2*len(npArray1) times (in 
> > C) over npArray2.
> 
> Dear Peter, 
> 
> Thanks for your reply. This really helped. It reduces the script time from 61(s) to 2(s). 
> 
> I am still very interested in knowing the correct numpy way to do this, but till then your fix works great. 


Hi, I suppose you have seen this already (in particular the first link): http://numpy-discussion.10968.n7.nabble.com/Implementing-a-quot-find-first-quot-style-function-td33085.htmlI don't thonk it's part of numpy yet.
Albert-Jan

Thread

looping and searching in numpy array Heli <hemla21@gmail.com> - 2016-03-10 03:43 -0800
  Re: looping and searching in numpy array Peter Otten <__peter__@web.de> - 2016-03-10 14:02 +0100
    Re: looping and searching in numpy array Heli <hemla21@gmail.com> - 2016-03-10 08:48 -0800
      Re: looping and searching in numpy array Heli <hemla21@gmail.com> - 2016-03-10 08:50 -0800
      RE: looping and searching in numpy array Albert-Jan Roskam <sjeik_appie@hotmail.com> - 2016-03-13 13:51 +0000
      RE: looping and searching in numpy array Albert-Jan Roskam <sjeik_appie@hotmail.com> - 2016-03-13 15:43 +0000
  Re: looping and searching in numpy array Mark Lawrence <breamoreboy@yahoo.co.uk> - 2016-03-10 13:22 +0000
  Re: looping and searching in numpy array srinivas devaki <mr.eightnoteight@gmail.com> - 2016-03-14 10:19 +0530
  Re: looping and searching in numpy array Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2016-03-14 15:22 +0000

csiph-web