Re: looping and searching in numpy array

X-Received	by 10.140.43.75 with SMTP id d69mr3197852qga.5.1457628529667; Thu, 10 Mar 2016 08:48:49 -0800 (PST)
X-Received	by 10.50.87.40 with SMTP id u8mr102607igz.0.1457628529623; Thu, 10 Mar 2016 08:48:49 -0800 (PST)
Path	csiph.com!usenet.blueworldhosting.com!feeder01.blueworldhosting.com!peer01.iad.highwinds-media.com!news.highwinds-media.com!feed-me.highwinds-media.com!w104no5963100qge.1!news-out.google.com!k1ni14616igd.0!nntp.google.com!nt3no477492igb.0!postnews.google.com!glegroupsg2000goo.googlegroups.com!not-for-mail
Newsgroups	comp.lang.python
Date	Thu, 10 Mar 2016 08:48:48 -0800 (PST)
In-Reply-To	<mailman.126.1457614964.15725.python-list@python.org>
Complaints-To	groups-abuse@google.com
Injection-Info	glegroupsg2000goo.googlegroups.com; posting-host=2.139.178.74; posting-account=E2EIjgoAAADrLzUR7q5GCOjCqk9rd0Wd
NNTP-Posting-Host	2.139.178.74
References	<77bd470b-cc05-4117-9ed1-6309d7a5633a@googlegroups.com> <mailman.126.1457614964.15725.python-list@python.org>
User-Agent	G2/1.0
MIME-Version	1.0
Message-ID	<0fca7583-c051-48bd-a9be-5e501722fc77@googlegroups.com> (permalink)
Subject	Re: looping and searching in numpy array
From	Heli <hemla21@gmail.com>
Injection-Date	Thu, 10 Mar 2016 16:48:49 +0000
Content-Type	text/plain; charset=ISO-8859-1
X-Received-Bytes	2608
X-Received-Body-CRC	4008889540
Xref	csiph.com comp.lang.python:104532

Show key headers only | View raw

On Thursday, March 10, 2016 at 2:02:57 PM UTC+1, Peter Otten wrote:
> Heli wrote:
> 
> > Dear all,
> > 
> > I need to loop over a numpy array and then do the following search. The
> > following is taking almost 60(s) for an array (npArray1 and npArray2 in
> > the example below) with around 300K values.
> > 
> > 
> > for id in np.nditer(npArray1):
> >                   
> >        newId=(np.where(npArray2==id))[0][0]
> > 
> > 
> > Is there anyway I can make the above faster? I need to run the script
> > above on much bigger arrays (50M). Please note that my two numpy arrays in
> > the lines above, npArray1 and npArray2  are not necessarily the same size,
> > but they are both 1d.
> 
> You mean you are looking for the index of the first occurence in npArray2 
> for every value of npArray1?
> 
> I don't know how to do this in numpy (I'm not an expert), but even basic 
> Python might be acceptable:
> 
> lookup = {}
> for i, v in enumerate(npArray2):
>     if v not in lookup:
>         lookup[v] = i
> 
> for v in npArray1:
>     print(lookup.get(v, "<not found>"))
> 
> That way you iterate once (in Python) instead of 2*len(npArray1) times (in 
> C) over npArray2.

Dear Peter, 

Thanks for your reply. This really helped. It reduces the script time from 61(s) to 2(s). 

I am still very interested in knowing the correct numpy way to do this, but till then your fix works great. 

Thanks a lot,

Thread

looping and searching in numpy array Heli <hemla21@gmail.com> - 2016-03-10 03:43 -0800
  Re: looping and searching in numpy array Peter Otten <__peter__@web.de> - 2016-03-10 14:02 +0100
    Re: looping and searching in numpy array Heli <hemla21@gmail.com> - 2016-03-10 08:48 -0800
      Re: looping and searching in numpy array Heli <hemla21@gmail.com> - 2016-03-10 08:50 -0800
      RE: looping and searching in numpy array Albert-Jan Roskam <sjeik_appie@hotmail.com> - 2016-03-13 13:51 +0000
      RE: looping and searching in numpy array Albert-Jan Roskam <sjeik_appie@hotmail.com> - 2016-03-13 15:43 +0000
  Re: looping and searching in numpy array Mark Lawrence <breamoreboy@yahoo.co.uk> - 2016-03-10 13:22 +0000
  Re: looping and searching in numpy array srinivas devaki <mr.eightnoteight@gmail.com> - 2016-03-14 10:19 +0530
  Re: looping and searching in numpy array Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2016-03-14 15:22 +0000

csiph-web