Return-Path: <python@mrabarnett.plus.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.011
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'false.': 0.07; 'indices': 0.07; "'w')": 0.09; 'advance': 0.10; "'r')": 0.16; 'entry.': 0.16; 'f1:': 0.16; 'from:addr:mrabarnett.plus.com': 0.16; 'from:addr:python': 0.16; 'from:name:mrab': 0.16; 'message- id:@mrabarnett.plus.com': 0.16; 'mylist': 0.16; 'numpy': 0.16; 'received:192.168.1.4': 0.16; 'run.': 0.16; 'set,': 0.16; 'statement.': 0.16; 'wrote:': 0.16; 'all,': 0.20; 'code.': 0.23; 'help.': 0.23; 'header:In-Reply-To:1': 0.24; 'header:User- Agent:1': 0.26; 'checking': 0.27; 'values': 0.28; 'array': 0.29; 'statement': 0.32; 'optimize': 0.33; 'changing': 0.34; 'equal': 0.34; 'file': 0.34; 'skip:d 20': 0.34; 'list': 0.34; 'replace': 0.35; 'set.': 0.35; 'comment': 0.35; 'list,': 0.36; 'there': 0.36; 'to:addr:python-list': 0.36; 'subject:: ': 0.37; 'list.': 0.37; 'presence': 0.38; 'thank': 0.38; 'data': 0.39; 'takes': 0.39; 'received:192': 0.39; 'to:addr:python.org': 0.40; 'chance': 0.60; 'your': 0.60; 'skip:n 10': 0.62; 'minutes': 0.64; 'percent': 0.66; 'dear': 0.67; 'subject:check': 0.84; 'subject:over': 0.84; 'subject:value': 0.84
X-CM-Score: 0.00
X-CNFS-Analysis: v=2.1 cv=JOrGyJ+b c=1 sm=1 tr=0 a=0nF1XD0wxitMEM03M9B4ZQ==:117 a=0nF1XD0wxitMEM03M9B4ZQ==:17 a=0Bzu9jTXAAAA:8 a=EBOSESyhAAAA:8 a=JAI3OqB5mnwA:10 a=IkcTkHD0fZMA:10 a=4Gtf66luoCpEQ_rREQwA:9 a=94STCxF_mxSOvyTA:21 a=dWQlqTw6tMYcuQE8:21 a=QEXdDO2ut3YA:10
X-AUTH: mrabarnett@:2500
Subject: Re: Optimizing if statement check over a numpy value
To: python-list@python.org
References: <65c45685-dee1-41f8-a16a-7a062f4e7b02@googlegroups.com>
From: MRAB <python@mrabarnett.plus.com>
Date: Thu, 23 Jul 2015 10:55:58 +0100
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0
MIME-Version: 1.0
In-Reply-To: <65c45685-dee1-41f8-a16a-7a062f4e7b02@googlegroups.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.20+
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.906.1437645368.3674.python-list@python.org>
Lines: 37
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1437645368 news.xs4all.nl 2825 [2001:888:2000:d::a6]:45366
X-Complaints-To: abuse@xs4all.nl
Path: csiph.com!usenet.pasdenom.info!news.stben.net!border1.nntp.ams1.giganews.com!nntp.giganews.com!newsfeed.xs4all.nl!newsfeed8.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Xref: csiph.com comp.lang.python:94436

On 2015-07-23 10:21, Heli Nix wrote:
> Dear all,
>
> I have the following piece of code. I am reading a numpy dataset from an hdf5 file and I am changing values to a new value if they equal 1.
>
>   There is 90 percent chance that (if id not in myList:) is true and in 10 percent of time is false.
>
> with h5py.File(inputFile, 'r') as f1:
>      with h5py.File(inputFile2, 'w') as f2:
>          ds=f1["MyDataset"].value
>          myList=[list of Indices that must not be given the new_value]
>
>          new_value=1e-20
>          for index,val in     np.ndenumerate(ds):
>              if val==1.0 :
>                  id=index[0]+1
>                  if id not in myList:
>                      ds[index]=new_value
>
>          dset1 = f2.create_dataset("Cell Ids", data=cellID_ds)
>          dset2 = f2.create_dataset("Porosity", data=poros_ds)
>
> My numpy array has 16M data and it takes 9 hrs to run. If I comment my if statement (if id not in myList:) it only takes 5 minutes to run.
>
> Is there any way that I can optimize this if statement.
>
> Thank you very much in Advance for your help.
>
> Best Regards,
>
When checking for presence in a list, it has to check every entry. The
time taken is proportional to the length of the list.

The time taken to check for presence in a set, however, is a constant.

Replace the list myList with a set.