Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #36355

Re: Numpy outlier removal

Path csiph.com!newsfeed.hal-mli.net!feeder3.hal-mli.net!newsfeed.hal-mli.net!feeder1.hal-mli.net!nntp-feed.chiark.greenend.org.uk!ewrotcd!news.nosignal.org!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path <oscar.j.benjamin@gmail.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.014
X-Spam-Evidence '*H*': 0.97; '*S*': 0.00; 'correct.': 0.07; 'defined,': 0.09; 'explanation': 0.09; 'robust': 0.09; 'statistical': 0.09; 'subset': 0.09; 'cc:addr:python-list': 0.10; 'cases': 0.15; 'result.': 0.15; '*never*': 0.16; 'benjamin': 0.16; 'cc:name:python list': 0.16; 'robust.': 0.16; 'sense,': 0.16; 'to:addr:pearwood.info': 0.16; 'to:addr:steve+comp.lang.python': 0.16; "to:name:steven d'aprano": 0.16; 'mon,': 0.16; 'wrote:': 0.17; 'jan': 0.18; '>>>': 0.18; 'regardless': 0.21; 'assuming': 0.22; "user's": 0.22; 'cc:2**0': 0.23; 'so.': 0.24; 'cc:addr:python.org': 0.25; 'header:In-Reply-To:1': 0.25; 'fit': 0.26; 'message-id:@mail.gmail.com': 0.27; "doesn't": 0.28; 'fine': 0.28; "d'aprano": 0.29; 'measure': 0.29; 'normal.': 0.29; 'steven': 0.29; 'no,': 0.29; "i'm": 0.29; 'e.g.': 0.30; 'choosing': 0.33; 'ones,': 0.33; 'problem': 0.33; 'received:google.com': 0.34; 'doing': 0.35; 'sometimes': 0.35; 'received:209.85': 0.35; 'there': 0.35; 'but': 0.36; 'depends': 0.36; "wasn't": 0.36; 'useful': 0.36; 'should': 0.36; 'why': 0.37; 'received:209': 0.37; 'data': 0.37; 'subject:: ': 0.38; 'mean': 0.38; 'some': 0.38; 'sure': 0.38; 'where': 0.40; 'skip:" 10': 0.40; 'header:Received:5': 0.40; 'think': 0.40; 'your': 0.60; 'is.': 0.62; 'techniques': 0.63; 'more': 0.63; 'other.': 0.64; 'taking': 0.65; 'medical': 0.66; 'hand': 0.82; '2013': 0.84; 'eye,': 0.84; 'median': 0.84; 'oscar': 0.84; 'subject:removal': 0.84; 'trial)': 0.84; 'mean.': 0.91
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=tuJn4lgvMP6uB8JzTGisX9v2RKaCdPdFDHYQ6wvl5oE=; b=Fj0F9iS2O//Z6OlXOBrBNyvTq/lXOLmoRq56rkL7/0/T8FWL4IYnaMajA/pYkWZsUi psGRdmNx+6cuH6e1Pe5EiI+/CPphUdg22EepVg5JvbNFuamLGbRLtM5pqZBcCh3OieNd WjsIDDgpMT34HcMDF5S4aWb2ceC5Dhh9ByZZdx+DQAfkGZjzKUn1BPw/VPbCMdR/7FIG po27mKuqpPIOwyXHnxsqFjeYjHCCatkeIHqOfahPwVcYJq5H8dPBXh/Dw/waOSowZhob eN/fjseq9eBhxIzL6nSlDEXRvPpFCV4AM+90tu49wiTrJ9V0nUpzbt+7sme2Hq2jrI0O ftMg==
MIME-Version 1.0
In-Reply-To <50ea58f0$0$21851$c3e8da3$76491128@news.astraweb.com>
References <mailman.179.1357501521.2939.python-list@python.org> <50ea28e7$0$30003$c3e8da3$5496439d@news.astraweb.com> <mailman.205.1357525775.2939.python-list@python.org> <50ea58f0$0$21851$c3e8da3$76491128@news.astraweb.com>
Date Mon, 7 Jan 2013 15:20:57 +0000
Subject Re: Numpy outlier removal
From Oscar Benjamin <oscar.j.benjamin@gmail.com>
To "Steven D'Aprano" <steve+comp.lang.python@pearwood.info>
Content-Type text/plain; charset=ISO-8859-1
Cc Python List <python-list@python.org>
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.15
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.223.1357572059.2939.python-list@python.org> (permalink)
Lines 45
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1357572059 news.xs4all.nl 6872 [2001:888:2000:d::a6]:39999
X-Complaints-To abuse@xs4all.nl
Xref csiph.com comp.lang.python:36355

Show key headers only | View raw

On 7 January 2013 05:11, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Mon, 07 Jan 2013 02:29:27 +0000, Oscar Benjamin wrote:
>
>> On 7 January 2013 01:46, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote:
>>> On Sun, 06 Jan 2013 19:44:08 +0000, Joseph L. Casale wrote:
>>>
>>> I'm not sure that this approach is statistically robust. No, let me be
>>> even more assertive: I'm sure that this approach is NOT statistically
>>> robust, and may be scientifically dubious.
>>
>> Whether or not this is "statistically robust" requires more explanation
>> about the OP's intention.
>
> Not really. Statistics robustness is objectively defined, and the user's
> intention doesn't come into it. The mean is not a robust measure of
> central tendency, the median is, regardless of why you pick one or the
> other.

Okay, I see what you mean. I wasn't thinking of robustness as a
technical term but now I see that you are correct.

Perhaps what I should have said is that whether or not this matters
depends on the problem at hand (hopefully this isn't an important
medical trial) and the particular type of data that you have; assuming
normality is fine in many cases even if the data is not "really"
normal.

>
> There are sometimes good reasons for choosing non-robust statistics or
> techniques over robust ones, but some techniques are so dodgy that there
> is *never* a good reason for doing so. E.g. finding the line of best fit
> by eye, or taking more and more samples until you get a statistically
> significant result. Such techniques are not just non-robust in the
> statistical sense, but non-robust in the general sense, if not outright
> deceitful.

There are sometimes good reasons to get a line of best fit by eye. In
particular if your data contains clusters that are hard to separate,
sometimes it's useful to just pick out roughly where you think a line
through a subset of the data is.


Oscar

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Numpy outlier removal "Joseph L. Casale" <jcasale@activenetwerx.com> - 2013-01-06 19:44 +0000
  Re: Numpy outlier removal Hans Mulder <hansmu@xs4all.nl> - 2013-01-06 23:33 +0100
    RE: Numpy outlier removal "Joseph L. Casale" <jcasale@activenetwerx.com> - 2013-01-06 22:50 +0000
    Re: Numpy outlier removal MRAB <python@mrabarnett.plus.com> - 2013-01-06 23:18 +0000
  Re: Numpy outlier removal Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-07 01:46 +0000
    Re: Numpy outlier removal "Paul Simon" <psimon@sonic.net> - 2013-01-06 18:21 -0800
    Re: Numpy outlier removal Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-07 02:29 +0000
      Re: Numpy outlier removal Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-07 05:11 +0000
        Re: Numpy outlier removal Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-07 15:20 +0000
          [Offtopic] Line fitting [was Re: Numpy outlier removal] Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-07 17:58 +0000
            Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Chris Angelico <rosuav@gmail.com> - 2013-01-08 06:43 +1100
              Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-08 02:06 +0000
                Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Chris Angelico <rosuav@gmail.com> - 2013-01-08 17:35 +1100
                Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Robert Kern <robert.kern@gmail.com> - 2013-01-08 15:55 +0000
                Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Chris Angelico <rosuav@gmail.com> - 2013-01-09 07:14 +1100
                Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-09 07:50 +0000
                Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Robert Kern <robert.kern@gmail.com> - 2013-01-08 22:59 +0000
            Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-07 22:32 +0000
              Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-08 01:23 +0000
                Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Terry Reedy <tjreedy@udel.edu> - 2013-01-08 04:07 -0500
                Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Maarten <maarten.sneep@knmi.nl> - 2013-01-08 08:47 -0800
                Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Maarten <maarten.sneep@knmi.nl> - 2013-01-08 08:47 -0800
                Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-01-09 00:02 +0000
                Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-01-08 13:50 +0000
            Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Jason Friedman <jason@powerpull.net> - 2013-01-08 19:22 -0700
            Re: [Offtopic] Line fitting [was Re: Numpy outlier removal] Jason Friedman <jason@powerpull.net> - 2013-01-08 19:23 -0700
        Re: Numpy outlier removal Robert Kern <robert.kern@gmail.com> - 2013-01-07 15:35 +0000
    RE: Numpy outlier removal "Joseph L. Casale" <jcasale@activenetwerx.com> - 2013-01-07 02:12 +0000

csiph-web