Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.lang.python > #69792 > unrolled thread
| Started by | Ben Finney <ben+python@benfinney.id.au> |
|---|---|
| First post | 2014-04-07 13:05 +1000 |
| Last post | 2014-04-08 15:19 +0000 |
| Articles | 5 on this page of 105 — 22 participants |
Back to article view | Back to comp.lang.python
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
Re: threading Ben Finney <ben+python@benfinney.id.au> - 2014-04-07 13:05 +1000
Re: threading Roy Smith <roy@panix.com> - 2014-04-06 23:48 -0400
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-07 13:56 +1000
Re: threading Roy Smith <roy@panix.com> - 2014-04-07 08:26 -0400
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-07 22:34 +1000
Re: threading Roy Smith <roy@panix.com> - 2014-04-07 09:22 -0400
Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-07 14:41 +0100
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 16:49 +0300
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 00:27 +1000
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 17:51 +0300
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 01:12 +1000
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 00:24 +1000
Re: threading Rick Johnson <rantingrickjohnson@gmail.com> - 2014-04-08 18:09 -0700
Re: threading "Neil D. Cerutti" <neilc@norwich.edu> - 2014-04-09 09:50 -0400
Re: threading Rick Johnson <rantingrickjohnson@gmail.com> - 2014-04-09 08:51 -0700
Re: threading MRAB <python@mrabarnett.plus.com> - 2014-04-09 18:47 +0100
Re: threading Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-04-10 11:35 +1200
Re: threading Roy Smith <roy@panix.com> - 2014-04-09 19:53 -0400
Re: threading Andrew Berg <robotsondrugs@gmail.com> - 2014-04-09 19:02 -0500
Re: threading Steven D'Aprano <steve@pearwood.info> - 2014-04-10 02:43 +0000
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 13:08 +1000
Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-10 09:23 +0100
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 19:11 +1000
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 04:00 +1000
Re: threading Steven D'Aprano <steve@pearwood.info> - 2014-04-10 03:44 +0000
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 13:54 +1000
Re: threading Ben Finney <ben+python@benfinney.id.au> - 2014-04-07 15:22 +1000
Re: threading Ethan Furman <ethan@stoneleaf.us> - 2014-04-08 11:09 -0700
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 21:41 +0200
Re: threading Grant Edwards <invalid@invalid.invalid> - 2014-04-08 20:30 +0000
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 00:32 +0200
Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-08 19:17 -0700
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 08:10 +0300
Re: threading Paul Rubin <no.email@nospam.invalid> - 2014-04-06 22:39 -0700
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-07 08:46 +0300
Re: threading Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-04-07 19:47 -0400
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-08 08:19 +0300
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 10:47 +0000
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-08 15:10 +0300
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 16:37 +0000
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-08 20:17 +0300
Re: threading Roy Smith <roy@panix.com> - 2014-04-08 09:19 -0400
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 15:44 +0000
Re: threading Paul Rubin <no.email@nospam.invalid> - 2014-04-08 09:38 -0700
Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-09 14:42 +0100
Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-09 15:23 +0200
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-09 16:55 +0300
Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-09 16:46 +0200
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-09 20:31 +0300
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 03:52 +1000
Re: threading Mark H Harris <harrismh777@gmail.com> - 2014-04-10 08:29 -0500
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 19:20 +0000
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-09 23:47 +1000
Re: threading Roy Smith <roy@panix.com> - 2014-04-09 10:44 -0400
Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-09 16:30 +0200
Re: threading Roy Smith <roy@panix.com> - 2014-04-09 10:52 -0400
Re: threading Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2014-04-10 11:19 +1200
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-09 19:48 +0300
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 00:44 +1000
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 15:29 +0000
Re: threading Terry Reedy <tjreedy@udel.edu> - 2014-04-09 12:14 -0400
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 02:25 +1000
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-09 16:32 +0000
Re: threading Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2014-04-09 19:44 -0400
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 11:05 +1000
Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-10 11:17 +0200
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 19:40 +1000
Re: threading "Frank Millman" <frank@chagford.com> - 2014-04-10 13:10 +0200
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 14:43 +0300
Re: threading Roy Smith <roy@panix.com> - 2014-04-10 08:56 -0400
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-10 15:24 +0000
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 19:20 +0300
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-11 01:32 +1000
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 19:25 +0300
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-11 03:08 +1000
Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 11:14 -0700
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 22:44 +0300
Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 13:21 -0700
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-10 23:44 +0300
Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 22:15 -0700
Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-10 23:50 -0700
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-11 18:36 +0300
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-12 01:53 +1000
Re: threading Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-04-11 16:58 +0100
Re: threading Rustom Mody <rustompmody@gmail.com> - 2014-04-11 11:54 -0700
Re: threading Marko Rauhamaa <marko@pacujo.net> - 2014-04-11 22:27 +0300
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-11 01:51 +0200
Re: threading Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2014-04-11 05:35 +0000
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-11 09:26 +0000
Re: threading Roy Smith <roy@panix.com> - 2014-04-11 08:36 -0400
Re: threading Grant Edwards <invalid@invalid.invalid> - 2014-04-11 16:18 +0000
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-11 02:21 +0200
Re: threading Terry Reedy <tjreedy@udel.edu> - 2014-04-10 20:23 -0400
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-10 21:19 +1000
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 02:06 +0000
Re: threading alister <alister.nospam.ware@ntlworld.com> - 2014-04-08 11:07 +0000
Re: threading Roy Smith <roy@panix.com> - 2014-04-08 09:13 -0400
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-08 23:23 +1000
Re: threading alister <alister.nospam.ware@ntlworld.com> - 2014-04-08 14:15 +0000
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 16:06 +0000
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 15:40 +0000
Re: threading Paul Rubin <no.email@nospam.invalid> - 2014-04-08 09:46 -0700
Re: threading Chris Angelico <rosuav@gmail.com> - 2014-04-09 02:46 +1000
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 17:17 +0000
Re: threading Sturla Molden <sturla.molden@gmail.com> - 2014-04-08 15:19 +0000
Page 6 of 6 — ← Prev page 1 2 3 4 5 [6]
| From | Sturla Molden <sturla.molden@gmail.com> |
|---|---|
| Date | 2014-04-08 15:40 +0000 |
| Message-ID | <mailman.9022.1396971675.18130.python-list@python.org> |
| In reply to | #69857 |
Roy Smith <roy@panix.com> wrote: > Let's say I've got a program which consumes 60 GB of RAM, so I'm renting > the 2xlarge instance to run it. My software architect could recode the > program to be more efficient, and fit into just 30 GB, saving me > $3000/year. How much of his time is it worth to do that? He's costing > me about $600/day, so if he can do it in a week, it'll take a year to > recoup my investment. Exactly. That is what I said "just throw more RAM at it". We see this in scientific HPC too. What does it cost of optimizing software compared to just using a bigger computer? It virtually never pays off. Or Python related: Python might be slow, but how much should we value our own time? A simulation which took one week to complete, how long did it take to write the code? When should we use C++ or Fortran instead of Python? Ever? There is a reason scientists are running Python on even the biggest supercomputers today. Hardware might be expensive, but not compared to human resources. And that gap just increases as hardware is getting cheaper. So we should optimize for human resources rather than for hardware. And with 64 bit that is finally possible. (It isn't always possible with 32 bit, so that is a different story.) Sturla
[toc] | [prev] | [next] | [standalone]
| From | Paul Rubin <no.email@nospam.invalid> |
|---|---|
| Date | 2014-04-08 09:46 -0700 |
| Message-ID | <7xzjjvbwge.fsf@ruckus.brouhaha.com> |
| In reply to | #69875 |
Sturla Molden <sturla.molden@gmail.com> writes: > When should we use C++ or Fortran instead of Python? Ever? When performance matters? > There is a reason scientists are running Python on even the biggest > supercomputers today. They use Python as a scripting wrapper around numerics libraries written in compiled languages. > Hardware might be expensive, but not compared to human resources. Sure, and that implies that making people wait for computer results costs you. If a C++ program can run in 2 seconds while the equivalent Python script would take a minute, the Python version loses the user's attention and they have to burn time refocusing it. Or if the C++ version takes 12 hours and the Python version takes a month, they can start the C++ version at the end of their workday and have the results when they arrive at the office the next morning. Then they spend the day examining the results and adjusting the program for the next run. Most of my stuff is in Python but there are times when your stuff just has to run fast.
[toc] | [prev] | [next] | [standalone]
| From | Chris Angelico <rosuav@gmail.com> |
|---|---|
| Date | 2014-04-09 02:46 +1000 |
| Message-ID | <mailman.9028.1396975589.18130.python-list@python.org> |
| In reply to | #69881 |
On Wed, Apr 9, 2014 at 2:46 AM, Paul Rubin <no.email@nospam.invalid> wrote:
> Sure, and that implies that making people wait for computer results
> costs you. If a C++ program can run in 2 seconds while the equivalent
> Python script would take a minute, the Python version loses the user's
> attention and they have to burn time refocusing it. Or if the C++
> version takes 12 hours and the Python version takes a month, they can
> start the C++ version at the end of their workday and have the results
> when they arrive at the office the next morning. Then they spend the
> day examining the results and adjusting the program for the next run.
>
> Most of my stuff is in Python but there are times when your stuff just
> has to run fast.
This is true. However, this applies ONLY when you really are spending
that much computational effort and getting interactive response. Most
programs aren't like that; most interactive programs do their work in
a fraction of a second regardless of the language (the difference
between 10ms and 50ms is nothing to a human), and performance of
non-interactive servers really translates to throughput (can you
handle 100 requests a second or 200 req/s?) When you actually do have
heavy mathematical computation, often you can write your logic in
Python and have the grunt-work done at a lower level; that's what all
those numeric/scientific libraries are doing, and that's also - in
effect - what crypto libraries do for you. You don't implement
cryptography in Python; you call on a lower-level library and let it
do the work.
So the choices aren't really "write it in Python" and "write it in
C++". The choices are actually "write it in some combination of Python
and C/FORTRAN" and "write it in C++ with some libraries written in
whatever". And when you look at it like that, the difference isn't
nearly as stark as it first seemed. I'd much rather write my outer
wrapper code in something like Python than have to fiddle with all
those basics in C++; and that goes even more strongly if I have to
write any sort of UI beyond the most basic console I/O. I've written
my share of GUIs in C++, and Python beats them all hands down. Even
for basic text, it's a lot easier in Python; sure, I have printf() to
do my output, and that's fine, but for input, I'm really not happy
with either cin (C++ iostream style) or gets/scanf (C stdio style). In
Python? >>> input("Enter your name: ") and you get back a nice Unicode
string. Easy.
ChrisA
[toc] | [prev] | [next] | [standalone]
| From | Sturla Molden <sturla.molden@gmail.com> |
|---|---|
| Date | 2014-04-08 17:17 +0000 |
| Message-ID | <mailman.9030.1396977487.18130.python-list@python.org> |
| In reply to | #69881 |
Paul Rubin <no.email@nospam.invalid> wrote: > Sturla Molden <sturla.molden@gmail.com> writes: >> When should we use C++ or Fortran instead of Python? Ever? > > When performance matters? Sometimes, but usually performance only matters in certain types of calculations like matrix algebra or FFTs. But we always use specialized libraries for that, such as Intel MKL, OpenBLAS, AMD ACML, or Apple's Accelerate Framework. "When performance matters" is not really a good answer, because performance might not matter where we think or as much as we think. For examle if I were to write a Levenberg-Marquardt solver, I would e.g. find that 95% of the time is spent solving the linearized least-squares equations (J' J + lambda diag(J' J)) dx = J' error using QR or SVD. If I used LAPACK method DGELS or DGELSS to solve these equations with QR or SVD, I would find that my otherwise idential Fortran and Python code would perform roughly the same. None of the performance libraries I mentioned cares if I am calling them from Python or Fortran. The differences in runtime would thus be in the 5 % spent outside the performance libraries. And if i/o is more efficient in Python, which is very likely, it might even be that the Python version runs faster. So the question is, when does it matter for me? Now and then, perhaps, but very rarely. For example it did matter when we wrote the cKDTree object for SciPy. But it never matters when I use it. I will make the bold claim that 90 % of the use of C++ or Fortran code in scientific computing stems from the misconception that "writing everything" in a compiled language will be faster. I have done this mistake myself, and lost valuable time. I have promised myself I will not do it again, and then realized I have done the same mistake yet again. It is a very dangerous misconception. Always prototype in Python first, then profile, then choose between optimization or using a bigger computer. Starting out by writing C, C++, Cython or Fortran is a trap. Yet many do, myself included, because we errorneously trust ourselves to know when we should just use Python. But the thing is, nobody does, we just think we do. Sturla
[toc] | [prev] | [next] | [standalone]
| From | Sturla Molden <sturla.molden@gmail.com> |
|---|---|
| Date | 2014-04-08 15:19 +0000 |
| Message-ID | <mailman.9018.1396970355.18130.python-list@python.org> |
| In reply to | #69853 |
alister <alister.nospam.ware@ntlworld.com> wrote: > As my only professional coding experience has been with embedded 8 bit > processors with limited resources i naturally abhorrent to the process of > "Just throw more RAM (Or any other resource for that matter)at it". I understand. I do not advocate threads for parallel i/o on 8, 16 or 32 bit systems – nor on 64 bit systems without sufficient RAM. But I do advocate is that buying RAM is cheaper than buying developer time. Sturla
[toc] | [prev] | [standalone]
Page 6 of 6 — ← Prev page 1 2 3 4 5 [6]
Back to top | Article view | comp.lang.python
csiph-web