Path: csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!newsfeed.xs4all.nl!newsfeed2.news.xs4all.nl!xs4all!post.news.xs4all.nl!not-for-mail
Return-Path: <oscar.j.benjamin@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
X-Spam-Status: OK 0.013
X-Spam-Evidence: '*H*': 0.97; '*S*': 0.00; 'dict': 0.09; 'excluding': 0.09; 'explanation': 0.09; 'cc:addr:python-list': 0.10; 'cc:name:python list': 0.16; 'include.': 0.16; 'integers.': 0.16; 'numpy': 0.16; 'operation.': 0.16; 'robust.': 0.16; 'to:addr:pearwood.info': 0.16; 'to:addr:steve+comp.lang.python': 0.16; "to:name:steven d'aprano": 0.16; 'wrote:': 0.17; 'jan': 0.18; 'cc:2**0': 0.23; 'thus': 0.24; 'cc:addr:python.org': 0.25; 'header:In-Reply-To:1': 0.25; 'values': 0.26; 'message- id:@mail.gmail.com': 0.27; "d'aprano": 0.29; 'far,': 0.29; 'steven': 0.29; 'array': 0.29; 'no,': 0.29; "i'm": 0.29; 'running': 0.32; 'received:google.com': 0.34; 'text': 0.34; 'list': 0.35; 'received:209.85': 0.35; 'really': 0.36; 'received:209': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'sure': 0.38; 'where': 0.40; 'skip:" 10': 0.40; 'header:Received:5': 0.40; 'first': 0.61; 'more': 0.63; '2013': 0.84; 'oscar': 0.84; 'subject:removal': 0.84; 'technique': 0.93
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=VRdcwuSa6q4RwTwiG1URKmx0jTNpinrOGmF3tHcxIVg=; b=NJ3drRO6K57IxDDcfUfbzoSNBfOzNjcwJIVQsToTrIOWpCkbQjHQ2sZF/vat9gStxT Q55SPeNCcvSjE952wUDFnPcSnFmPHAnZxWL7Vu1qewQ4pm0tBmfiv42BFpzfGKbUklvk ngdy/S8ngXzC311+PoXgmjYakp07WZKY0xExe29u4JbGR0SzN8Z1y18d/jdIiQefVPa3 L2IjJlSe7xaFixfbyX1Sw2vdDzOgG88J5NNObmvzt0IZyW2QvFPZOG/A9zDMbUsRy7Sg kaDoKiyu0qCMaZdzeyy8ZX4apprCVC4M8lHpvZ9LuwPp1kudas2JFzg6Ukxh72E0p7xJ 9Cew==
MIME-Version: 1.0
In-Reply-To: <50ea28e7$0$30003$c3e8da3$5496439d@news.astraweb.com>
References: <mailman.179.1357501521.2939.python-list@python.org> <50ea28e7$0$30003$c3e8da3$5496439d@news.astraweb.com>
Date: Mon, 7 Jan 2013 02:29:27 +0000
Subject: Re: Numpy outlier removal
From: Oscar Benjamin <oscar.j.benjamin@gmail.com>
To: "Steven D'Aprano" <steve+comp.lang.python@pearwood.info>
Content-Type: text/plain; charset=ISO-8859-1
Cc: Python List <python-list@python.org>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe: <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <http://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups: comp.lang.python
Message-ID: <mailman.205.1357525775.2939.python-list@python.org>
Lines: 25
NNTP-Posting-Host: 2001:888:2000:d::a6
X-Trace: 1357525775 news.xs4all.nl 6945 [2001:888:2000:d::a6]:34361
X-Complaints-To: abuse@xs4all.nl
Xref: csiph.com comp.lang.python:36316

On 7 January 2013 01:46, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Sun, 06 Jan 2013 19:44:08 +0000, Joseph L. Casale wrote:
>
>> I have a dataset that consists of a dict with text descriptions and
>> values that are integers. If required, I collect the values into a list
>> and create a numpy array running it through a simple routine:
>>
>> data[abs(data - mean(data)) < m * std(data)]
>>
>> where m is the number of std deviations to include.
>
> I'm not sure that this approach is statistically robust. No, let me be
> even more assertive: I'm sure that this approach is NOT statistically
> robust, and may be scientifically dubious.

Whether or not this is "statistically robust" requires more
explanation about the OP's intention. Thus far, the OP has not given
any reason/motivation for excluding data or even for having any data
in the first place! It's hard to say whether any technique applied is
really accurate/robust without knowing *anything* about the purpose of
the operation.


Oscar