RE: counting unique numpy subarrays

Path	csiph.com!fu-berlin.de!uni-berlin.de!not-for-mail
From	Albert-Jan Roskam <sjeik_appie@hotmail.com>
Newsgroups	comp.lang.python
Subject	RE: counting unique numpy subarrays
Date	Fri, 4 Dec 2015 22:36:52 +0000
Lines	27
Message-ID	<mailman.208.1449268679.14615.python-list@python.org> (permalink)
References	<Q1m8y.334924$rR1.113623@fx19.iad>
Mime-Version	1.0
Content-Type	text/plain; charset="windows-1256"
Content-Transfer-Encoding	base64
X-Trace	news.uni-berlin.de hO8m9YIjgqIjtfQoolz3LA6SBzll50oFMpZNGqTIhEWA==
Return-Path	<sjeik_appie@hotmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.007
X-Spam-Evidence	'H': 0.99; 'S': 0.00; 'counting': 0.07; '(sorry': 0.09; 'collections': 0.09; 'slow.': 0.09; 'python': 0.10; 'def': 0.13; 'array)': 0.16; 'concatenated': 0.16; 'data)': 0.16; 'numpy': 0.16; 'pointers.': 0.16; 'received:io': 0.16; 'received:psf.io': 0.16; '2015': 0.20; 'to:21': 0.21; 'trying': 0.22; 'bit': 0.23; 'dec': 0.23; 'import': 0.24; 'header:In-Reply- To:1': 0.24; "i've": 0.25; '+0000': 0.27; 'fri,': 0.27; 'skip:# 10': 0.27; 'to:no real name:21': 0.27; 'skip:e 30': 0.27; 'array': 0.29; "i'm": 0.30; 'url:mailman': 0.30; 'code': 0.30; 'date:': 0.31; 'useful': 0.33; 'url:python': 0.33; 'url:listinfo': 0.34; 'returning': 0.35; 'subject:': 0.35; 'but': 0.36; 'url:org': 0.36; 'email addr:python.org': 0.36; 'faster': 0.36; 'to:addr :python-list': 0.36; 'subject:: ': 0.37; 'seem': 0.37; 'data': 0.39; 'along': 0.39; 'from:': 0.39; 'url:mail': 0.40; 'to:addr:python.org': 0.40; 'hello,': 0.40; 'avoid': 0.61; 'skip:n 10': 0.62; 'email name:python-list': 0.67; 'charset:windows-1256': 0.67; 'counts': 0.81
X-TMN	[/d21VXSAM4F/QDLpANZrTKuAumCcWHhC]
X-Originating-Email	[sjeik_appie@hotmail.com]
Importance	Normal
In-Reply-To	<Q1m8y.334924$rR1.113623@fx19.iad>
X-OriginalArrivalTime	04 Dec 2015 22:36:53.0090 (UTC) FILETIME=[45EAEC20:01D12EE4]
X-Content-Filtered-By	Mailman/MimeDel 2.1.20+
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.20+
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<https://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<https://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Xref	csiph.com comp.lang.python:100016

Show key headers only | View raw

Hi

(Sorry for topposting)

numpy.ravel is faster than numpy.flatten (no copy)
numpy.empty is faster than numpy.zeros
numpy.fromiter might be useful to avoid the loop (just a hunch)

Albert-Jan

> From: duncan@invalid.invalid
> Subject: counting unique numpy subarrays
> Date: Fri, 4 Dec 2015 19:43:35 +0000
> To: python-list@python.org
> 
> Hello,
>       I'm trying to find a computationally efficient way of identifying
> unique subarrays, counting them and returning an array containing only
> the unique subarrays and a corresponding 1D array of counts. The
> following code works, but is a bit slow.
> 
> ###############
> 
> from collections import Counter
> import numpy
> 
> def bag_data(data):
>     # data (a numpy array) is bagged along axis 0
>     # returns concatenated array and corresponding array of counts
>     vec_shape = data.shape[1:]
>     counts = Counter(tuple(arr.flatten()) for arr in data)
>     data_out = numpy.zeros((len(counts),) + vec_shape)
>     cnts = numpy.zeros((len(counts,)))
>     for i, (tup, cnt) in enumerate(counts.iteritems()):
>         data_out[i] = numpy.array(tup).reshape(vec_shape)
>         cnts[i] =  cnt
>     return data_out, cnts
> 
> ###############
> 
> I've been looking through the numpy docs, but don't seem to be able to
> come up with a clean solution that avoids Python loops. TIA for any
> useful pointers. Cheers.
> 
> Duncan
> -- 
> https://mail.python.org/mailman/listinfo/python-list

Thread

counting unique numpy subarrays duncan smith <duncan@invalid.invalid> - 2015-12-04 19:43 +0000
  RE: counting unique numpy subarrays Albert-Jan Roskam <sjeik_appie@hotmail.com> - 2015-12-04 22:36 +0000
    Re: counting unique numpy subarrays duncan smith <duncan@invalid.invalid> - 2015-12-05 00:13 +0000
  Re: counting unique numpy subarrays Peter Otten <__peter__@web.de> - 2015-12-05 00:06 +0100
    Re: counting unique numpy subarrays duncan smith <duncan@invalid.invalid> - 2015-12-05 00:18 +0000

csiph-web