Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #52289 > unrolled thread

PEP 450 Adding a statistics module to Python

Back to article view | Back to comp.lang.python

Contents

  PEP 450 Adding a statistics module to Python Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-08-10 01:10 +0000
    Re: PEP 450 Adding a statistics module to Python Skip Montanaro <skip@pobox.com> - 2013-08-09 22:14 -0500
      Re: PEP 450 Adding a statistics module to Python Roy Smith <roy@panix.com> - 2013-08-10 07:50 -0400
        Re: PEP 450 Adding a statistics module to Python Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-08-10 13:23 +0100
          Re: PEP 450 Adding a statistics module to Python Roy Smith <roy@panix.com> - 2013-08-10 08:43 -0400
            Re: PEP 450 Adding a statistics module to Python Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-08-10 14:17 +0100
    Re: PEP 450 Adding a statistics module to Python Ben Finney <ben+python@benfinney.id.au> - 2013-08-10 15:05 +1000
    Re: PEP 450 Adding a statistics module to Python Stefan Behnel <stefan_ml@behnel.de> - 2013-08-10 09:55 +0200
    Re: PEP 450 Adding a statistics module to Python Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2013-08-10 16:19 -0400
    Re: PEP 450 Adding a statistics module to Python Skip Montanaro <skip@pobox.com> - 2013-08-11 06:50 -0500
      Re: PEP 450 Adding a statistics module to Python Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-08-11 13:33 +0000
      Re: PEP 450 Adding a statistics module to Python Roy Smith <roy@panix.com> - 2013-08-11 10:02 -0400
        Re: PEP 450 Adding a statistics module to Python duncan smith <buzzard@invalid.invalid> - 2013-08-11 16:44 +0100
    Re: PEP 450 Adding a statistics module to Python Nicholas Cole <nicholas.cole@gmail.com> - 2013-08-11 13:27 +0100
    Re: PEP 450 Adding a statistics module to Python Wolfgang Keller <feliphil@gmx.net> - 2013-08-13 20:14 +0200
      Re: PEP 450 Adding a statistics module to Python Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-08-13 19:44 +0100
      Re: PEP 450 Adding a statistics module to Python Steven D'Aprano <steve@pearwood.info> - 2013-08-14 06:21 +0000
    Re: PEP 450 Adding a statistics module to Python CM <cmpython@gmail.com> - 2013-08-14 21:26 -0700
      RE: PEP 450 Adding a statistics module to Python "Prasad, Ramit" <ramit.prasad@jpmorgan.com.dmarc.invalid> - 2013-08-16 19:17 +0000
    Re: PEP 450 Adding a statistics module to Python taldcroft@cfa.harvard.edu - 2013-08-16 08:50 -0700
      Re: PEP 450 Adding a statistics module to Python chris.barker@noaa.gov - 2013-08-16 09:31 -0700
        Re: PEP 450 Adding a statistics module to Python Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-08-16 18:15 +0100
          Re: PEP 450 Adding a statistics module to Python chris.barker@noaa.gov - 2013-08-16 12:00 -0700
            Re: PEP 450 Adding a statistics module to Python Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2013-08-16 20:41 +0100
        Re: PEP 450 Adding a statistics module to Python Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-08-16 18:51 +0000
          Re: PEP 450 Adding a statistics module to Python chris.barker@noaa.gov - 2013-08-16 12:48 -0700
      Re: PEP 450 Adding a statistics module to Python Roy Smith <roy@panix.com> - 2013-08-16 22:06 -0400
        Re: PEP 450 Adding a statistics module to Python Josef Pktd <josef.pktd@gmail.com> - 2013-08-17 05:13 -0700
    Re: PEP 450 Adding a statistics module to Python Jason Friedman <jsf80238@gmail.com> - 2013-08-17 21:57 -0600

Page 1 of 2  [1] 2  Next page →

#52289 — PEP 450 Adding a statistics module to Python

I am seeking comments on PEP 450, Adding a statistics module to Python's 
standard library:

http://www.python.org/dev/peps/pep-0450/

Please read the FAQs before asking anything :-)


Also relevant:

http://bugs.python.org/issue18606



-- 
Steven

[toc] | [next] | [standalone]

#52294

On Fri, Aug 9, 2013 at 8:10 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> I am seeking comments on PEP 450, Adding a statistics module to Python's
> standard library:
>
> http://www.python.org/dev/peps/pep-0450/
>
> Please read the FAQs before asking anything :-)

Given that installing numpy or scipy is generally no more difficult
that executing "pip install (scipy|numpy)" I'm not really feeling the
need for a battery here...  (Of course, I use this stuff at work from
time-to-time, so maybe I'm more in the "nuclear reactor of batteries"
camp anyway.)

Skip

[toc] | [prev] | [next] | [standalone]

#52301

In article <mailman.417.1376104455.1251.python-list@python.org>,
 Skip Montanaro <skip@pobox.com> wrote:

> Given that installing numpy or scipy is generally no more difficult
> that executing "pip install (scipy|numpy)" I'm not really feeling the
> need for a battery here...

I just tried installing numpy in a fresh virtualenv on an Ubuntu Precise 
box.  I ran "pip install numpy".  It took 1.5 minutes.  It printed 
almost 1800 lines of build crap, including 383 warnings and 83 errors.  
For a newbie, that can be pretty intimidating.

That's for the case where I've already installed numpy elsewhere on that 
box, so I already had the fortran compiler, and the rest of the build 
chain.  For fun, I just spun up a new Ubuntu Precise instance in AWS.  
It came pre-installed with Python 2.7.3.  I tried "pip install numpy", 
which told me that pip was not installed.

At least it told me what I needed to do to get pip installed.  
Unfortunately, I didn't read the message carefully enough and typed 
"sudo apt-get install pip", which of course got me another error because 
the correct name of the package is python-pip.  Doing "sudo apt-get 
install python-pip" finally got me to the point where I could start to 
install numpy.

Of course, if I didn't have sudo privs on the box (most corporate 
environments), I never would have gotten that far.

At this point, "sudo pip install numpy" got me a bunch of errors 
culminating in "RuntimeError: Broken toolchain: cannot link a simple C 
program", and no indication of how to get any further.

At this point, most people would give up.  I don't remember the full set 
of steps I needed to do the first time.  Obviously, I would start with 
installing gcc, but I seem to remember there were additional steps 
needed to get fortran support.

Having some simple statistics baked into the standard python package 
would be a big win.  As shown above, installing numpy can be an 
insurmountable hurdle for people with insufficient sysadmin-fu.

PEP-450 makes cogent arguments why rolling your own statistics routines 
is fraught with peril.  Looking over our source tree, I see we've 
implemented std deviation in python at least twice. I'm sure they're 
both naive implementations of the sort PEP-450 warns about.

And, yes, backporting to 2.7 would be a big win too.  I know the goal is 
to get everybody onto 3.x, but my pip external dependency list includes 
40 modules.  It's going to be a long and complicated road to get to the 
point where I can move to 3.x, and I imagine most non-trivial projects 
are in a similar situation.

[toc] | [prev] | [next] | [standalone]

#52304

On 10 August 2013 12:50, Roy Smith <roy@panix.com> wrote:
> In article <mailman.417.1376104455.1251.python-list@python.org>,
>  Skip Montanaro <skip@pobox.com> wrote:
>
>> Given that installing numpy or scipy is generally no more difficult
>> that executing "pip install (scipy|numpy)" I'm not really feeling the
>> need for a battery here...
>
> I just tried installing numpy in a fresh virtualenv on an Ubuntu Precise
> box.  I ran "pip install numpy".  It took 1.5 minutes.  It printed
> almost 1800 lines of build crap, including 383 warnings and 83 errors.
> For a newbie, that can be pretty intimidating.
>
> That's for the case where I've already installed numpy elsewhere on that
> box, so I already had the fortran compiler, and the rest of the build
> chain.  For fun, I just spun up a new Ubuntu Precise instance in AWS.
> It came pre-installed with Python 2.7.3.  I tried "pip install numpy",
> which told me that pip was not installed.
>
> At least it told me what I needed to do to get pip installed.
> Unfortunately, I didn't read the message carefully enough and typed
> "sudo apt-get install pip", which of course got me another error because
> the correct name of the package is python-pip.  Doing "sudo apt-get
> install python-pip" finally got me to the point where I could start to
> install numpy.
>
> Of course, if I didn't have sudo privs on the box (most corporate
> environments), I never would have gotten that far.
>
> At this point, "sudo pip install numpy" got me a bunch of errors
> culminating in "RuntimeError: Broken toolchain: cannot link a simple C
> program", and no indication of how to get any further.

You should use apt-get for numpy/scipy on Ubuntu. Although
unfortunately IIRC this doesn't work as well as it should since Ubuntu
doesn't install the appropriate BLAS/LAPACK libraries by default
(leaving you with numpy's fallback libraries).

On Windows you should use the MSI installer (or easy_install).
Hopefully numpy/scipy will start distributing wheels soon and pip
install numpy will actually work.


Oscar

[toc] | [prev] | [next] | [standalone]

#52305

Skip Montanaro <skip@pobox.com> wrote:
> >> installing numpy or scipy is generally no more difficult
> >> that executing "pip install (scipy|numpy)"

I described the problems I had trying to follow that advice.

In article <mailman.425.1376137459.1251.python-list@python.org>,
 Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:

> You should use apt-get for numpy/scipy on Ubuntu. Although
> unfortunately IIRC this doesn't work as well as it should since Ubuntu
> doesn't install the appropriate BLAS/LAPACK libraries by default
> (leaving you with numpy's fallback libraries).

That really kind of proves my point.  It's *not* easy to install.  
Theres' a choice of methods, some of which work in some environments, 
some of which work in others.  And even if apt-get is the preferred 
install method on Ubuntu, it's a method which is unavailable to people 
without root access (and may be undesirable if you rely on virtualenv to 
keep multiple projects cleanly separated).

And, what happens if you don't have the right libraries?  Do you end up 
with an install which is missing some functionality, or one where all 
the calls work, but they're slower, or numerically unstable, or what?

All these questions go away if it's packaged with the standard library.

I'm not sure where the line should be drawn between "basic stuff that 
should be included" and "advanced stuff that you need an add-on to get", 
but certainly mean and std-dev should be in the default distribution.

[toc] | [prev] | [next] | [standalone]

#52306

On 10 August 2013 13:43, Roy Smith <roy@panix.com> wrote:
>
> In article <mailman.425.1376137459.1251.python-list@python.org>,
>  Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
>
>> You should use apt-get for numpy/scipy on Ubuntu. Although
>> unfortunately IIRC this doesn't work as well as it should since Ubuntu
>> doesn't install the appropriate BLAS/LAPACK libraries by default
>> (leaving you with numpy's fallback libraries).
>
> That really kind of proves my point.  It's *not* easy to install.
> Theres' a choice of methods, some of which work in some environments,
> some of which work in others.  And even if apt-get is the preferred
> install method on Ubuntu, it's a method which is unavailable to people
> without root access (and may be undesirable if you rely on virtualenv to
> keep multiple projects cleanly separated).
>
> And, what happens if you don't have the right libraries?  Do you end up
> with an install which is missing some functionality, or one where all
> the calls work, but they're slower, or numerically unstable, or what?

AFAIK not having separate BLAS/LAPACK libraries just means that
certain operations are a lot slower. If there are differences in
accuracy then they aren't significant enough that I've noticed.

I think that the reason Ubuntu doesn't install them by default is
because it's not sure which ones you want to use. Possibly the best
free setup comes from using ATLAS but this is optimised in a
CPU-specific way at build time. Ubuntu doesn't provide binaries for it
as using generic x86 executables would defeat much of the point of the
library (they do make it a lot easier by providing a source package
though).


Oscar

[toc] | [prev] | [next] | [standalone]

#52296

Skip Montanaro <skip@pobox.com> writes:

> Given that installing numpy or scipy is generally no more difficult
> that executing "pip install (scipy|numpy)" I'm not really feeling the
> need for a battery here...

NumPy and SciPy are not available for many Python users, including those
using a Python implementation for which there is no Numpy support
<URL:http://new.scipy.org/faq.html#python-version-support> and those for
whom large, dependency-heavy third-party packages are too much burden.

See the Rationale of PEP 450 for more reasons why “install NumPy” is not
a feasible solution for many use cases, and why having ‘statistics’ as a
pure-Python, standard-library package is desirable.

-- 
 \         “Dad always thought laughter was the best medicine, which I |
  `\    guess is why several of us died of tuberculosis.” —Jack Handey |
_o__)                                                                  |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#52297

Ben Finney, 10.08.2013 07:05:
> Skip Montanaro writes:
>> Given that installing numpy or scipy is generally no more difficult
>> that executing "pip install (scipy|numpy)" I'm not really feeling the
>> need for a battery here...
> 
> See the Rationale of PEP 450 for more reasons why “install NumPy” is not
> a feasible solution for many use cases, and why having ‘statistics’ as a
> pure-Python, standard-library package is desirable.

The rationale suggests that the module is meant as a simple toolset for
non-NumPy users. Are the APIs (class model, function names, etc.) similar
enough to make it easy to switch, preferably in both directions?

It would be good if a stdlib statistics module could be used as a SciPy
fallback for the "simple" things, and if users of the stdlib module could
easily switch their code to SciPy if they need more speed/features/whatever
at some point, without having to relearn the name of each single function.

I'm not asking for compatibility (doesn't sound reasonable without NumPy
arrays), but I think that a similarity in terms of API naming (as far as it
makes sense) should be clearly stated, e.g. in the Design Decisions section.

Stefan

[toc] | [prev] | [next] | [standalone]

#52323

On Fri, 9 Aug 2013 22:14:13 -0500, Skip Montanaro <skip@pobox.com>
declaimed the following:

>
>Given that installing numpy or scipy is generally no more difficult
>that executing "pip install (scipy|numpy)" I'm not really feeling the
>need for a battery here...  (Of course, I use this stuff at work from
>time-to-time, so maybe I'm more in the "nuclear reactor of batteries"
>camp anyway.)
>

	And for the whole nuclear power plant, isn't there an interface module
that lets Python control the R-system? http://rpy.sourceforge.net/ for
example.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#52381

> See the Rationale of PEP 450 for more reasons why “install NumPy” is not
> a feasible solution for many use cases, and why having ‘statistics’ as a
> pure-Python, standard-library package is desirable.

I read that before posting but am not sure I agree. I don't see the
screaming need for this package.  Why can't it continue to live on
PyPI, where, once again, it is available as "pip install ..."?

S

[toc] | [prev] | [next] | [standalone]

#52388

On Sun, 11 Aug 2013 06:50:36 -0500, Skip Montanaro wrote:

>> See the Rationale of PEP 450 for more reasons why “install NumPy” is
>> not a feasible solution for many use cases, and why having ‘statistics’
>> as a pure-Python, standard-library package is desirable.
> 
> I read that before posting but am not sure I agree. I don't see the
> screaming need for this package.  Why can't it continue to live on PyPI,
> where, once again, it is available as "pip install ..."?


The same could be said about any module, really. And indeed, some 
languages have that philosophy, they provide no libraries to speak of, if 
you want anything you have to either write it yourself or get it from 
somebody else.

Not everyone has the luxury of being able, or allowed, to run "pip 
install" to get additional, non-standard packages. E.g. in corporate 
environments. But I've already said that in the PEP.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#52390

In article <mailman.479.1376221844.1251.python-list@python.org>,
 Skip Montanaro <skip@pobox.com> wrote:

> > See the Rationale of PEP 450 for more reasons why “install NumPy” is not
> > a feasible solution for many use cases, and why having ‘statistics’ as a
> > pure-Python, standard-library package is desirable.
> 
> I read that before posting but am not sure I agree. I don't see the
> screaming need for this package.  Why can't it continue to live on
> PyPI, where, once again, it is available as "pip install ..."?

My previous comments on this topic were along the lines of "installing 
numpy is a non-starter if all you need are simple mean/std-dev".  You 
do, however, make a good point here.  Running "pip install statistics" 
is a much lower barrier to entry than getting numpy going, especially if 
statistics is pure python and thus has no dependencies on compiler tool 
chains which may be missing.

Still, I see two classes of function in PEP-450.  Class 1 is the really 
basic stuff:

* mean
* std-dev

Class 2 are the more complicated things like:

* linear regression
* median
* mode
* functions for calculating the probability of random variables
  from the normal, t, chi-squared, and F distributions
* inference on the mean
* anything that differentiates between population and sample

I could see leaving class 2 stuff in an optional pure-python module to 
be installed by pip, but for (as the PEP phrases it), the simplest and 
most obvious statistical functions (into which I lump mean and std-dev), 
having them in the standard library would be a big win.

[toc] | [prev] | [next] | [standalone]

#52392

On 11/08/13 15:02, Roy Smith wrote:
> In article <mailman.479.1376221844.1251.python-list@python.org>,
>   Skip Montanaro <skip@pobox.com> wrote:
>
>>> See the Rationale of PEP 450 for more reasons why “install NumPy” is not
>>> a feasible solution for many use cases, and why having ‘statistics’ as a
>>> pure-Python, standard-library package is desirable.
>>
>> I read that before posting but am not sure I agree. I don't see the
>> screaming need for this package.  Why can't it continue to live on
>> PyPI, where, once again, it is available as "pip install ..."?
>
> My previous comments on this topic were along the lines of "installing
> numpy is a non-starter if all you need are simple mean/std-dev".  You
> do, however, make a good point here.  Running "pip install statistics"
> is a much lower barrier to entry than getting numpy going, especially if
> statistics is pure python and thus has no dependencies on compiler tool
> chains which may be missing.
>
> Still, I see two classes of function in PEP-450.  Class 1 is the really
> basic stuff:
>
> * mean
> * std-dev
>
> Class 2 are the more complicated things like:
>
> * linear regression
> * median
> * mode
> * functions for calculating the probability of random variables
>    from the normal, t, chi-squared, and F distributions
> * inference on the mean
> * anything that differentiates between population and sample
>
> I could see leaving class 2 stuff in an optional pure-python module to
> be installed by pip, but for (as the PEP phrases it), the simplest and
> most obvious statistical functions (into which I lump mean and std-dev),
> having them in the standard library would be a big win.
>

I would probably move other descriptive statistics (median, mode, 
correlation, ...) into Class 1.

I roll my own statistical tests as I need them - simply to avoid having 
a dependency on R. But I generally do end up with a dependency on scipy 
because I need scipy.stats.distributions. So I guess a distinct library 
for probability distributions would be handy - but maybe it should not 
be in the standard library.

Once we move on to statistical modelling (e.g. linear regression) I 
think the case for inclusion in the standard library becomes weaker 
still. Cheers.

Duncan

[toc] | [prev] | [next] | [standalone]

#52384

[Multipart message — attachments visible in raw view] — view raw

On Sun, Aug 11, 2013 at 12:50 PM, Skip Montanaro <skip@pobox.com> wrote:

> > See the Rationale of PEP 450 for more reasons why “install NumPy” is not
> > a feasible solution for many use cases, and why having ‘statistics’ as a
> > pure-Python, standard-library package is desirable.
>
> I read that before posting but am not sure I agree. I don't see the
> screaming need for this package.  Why can't it continue to live on
> PyPI, where, once again, it is available as "pip install ..."?


Well, I *do* think this module would be a wonderful addition to the
standard library.  I've often used python to do analysis of data, nothing
complicated enough to need NumPy, but certainly things where I've needed to
find averages etc. I've rolled my own functions for these projects, and I'm
sure they are fragile.  Besides, it was just a pain to do them.

PyPI is terrific. There are lots of excellent modules on there.  It's a
wonderful resource.  But I think that the standard library is also a
wonderful thing, and where there are clearly defined modules, that serve a
general, well-defined function and where development does not need to be
very rapid, I think they should go into the Standard Library.

I'm aware that my opinion is just that of one user, but I read this PEP and
I thought, "Thank Goodness! That looks great.  About time too."

N.

[toc] | [prev] | [next] | [standalone]

#52468

> I am seeking comments on PEP 450, Adding a statistics module to
> Python's standard library:

I don't think that you want to re-implement RPy.

Sincerely,

Wolfgang

[toc] | [prev] | [next] | [standalone]

#52470

[Multipart message — attachments visible in raw view] — view raw

On Aug 13, 2013 7:22 PM, "Wolfgang Keller" <feliphil@gmx.net> wrote:
>
> > I am seeking comments on PEP 450, Adding a statistics module to
> > Python's standard library:
>
> I don't think that you want to re-implement RPy.

You're right. He doesn't.

Oscar

[toc] | [prev] | [next] | [standalone]

#52496

On Tue, 13 Aug 2013 20:14:55 +0200, Wolfgang Keller wrote:

>> I am seeking comments on PEP 450, Adding a statistics module to
>> Python's standard library:
> 
> I don't think that you want to re-implement RPy.

I never suggested re-implementing RPy. When you read the PEP, you will 
see that this proposal is to have a Python implementation of statistics 
functions, not a thin wrapper around another language.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#52535

On Friday, August 9, 2013 9:10:18 PM UTC-4, Steven D'Aprano wrote:
> I am seeking comments on PEP 450, Adding a statistics module to Python's 
> standard library:

I think it's a very good idea.  Good PEP points, too.  I hope it happens.

[toc] | [prev] | [next] | [standalone]

#52605

CM wrote:
> 
> On Friday, August 9, 2013 9:10:18 PM UTC-4, Steven D'Aprano wrote:
> > I am seeking comments on PEP 450, Adding a statistics module to Python's
> > standard library:
> 
> I think it's a very good idea.  Good PEP points, too.  I hope it happens.
> 

+1 especially for non-Cpython versions of Python.


~Ramit



This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email.  

[toc] | [prev] | [next] | [standalone]

#52593

On Friday, August 9, 2013 9:10:18 PM UTC-4, Steven D'Aprano wrote:
> I am seeking comments on PEP 450, Adding a statistics module to Python's 
> 
> standard library:
> 
> 
> 
> http://www.python.org/dev/peps/pep-0450/
> 
> 
> 
> Please read the FAQs before asking anything :-)
> 

I think this is a super idea.  Python is showing up in high-school and colllege intro programming courses here in the U.S.  Having a solid statistics module built in would work well in that context and make it even more natural as a complement to math courses.  Beyond the educational aspect, having a built-in module to *correctly* handle the frequent light-weight use cases would be useful across many professional disciplines.

I use NumPy on a daily basis and help scientists with installation problems frequently.  I can emphatically state that NumPy is not easy to install for newbies.  Open up a brand new Mac and look, no compilers!  Even experienced users can have problems with gfortran vs. g77 etc.   Anyone that has ever built BLAS/ATLAS from source will also tell you that SciPy is definitely not a simple "pip install" on many platforms (particularly if you don't have root).

- Tom

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web