Re: creating size-limited tar files

Date	2012-11-14 11:33 -0500
From	Dave Angel <d@davea.name>
Subject	Re: creating size-limited tar files
References	(10 earlier) <87fw4clg8b.fsf@nitrogen.i-did-not-set--mail-host-address--so-tickle-me> <CAF_E5JYYEQ77rEQb1tM5tDKbTTSwYNVv1An57bUKn8Y_d-VJUQ@mail.gmail.com> <CAF_E5Jbk2inQbTrurgBKP2DLjzR9=t4XXSGbafuq-E3UvJjaiw@mail.gmail.com> <50A3C287.80707@davea.name> <CAF_E5JaH332cJXko7iLuhNfxyrhE9oeUz3wMZ51XVx5CFE29kw@mail.gmail.com>
Newsgroups	comp.lang.python
Message-ID	<mailman.3694.1352910832.27098.python-list@python.org> (permalink)

Show all headers | View raw

On 11/14/2012 11:16 AM, andrea crotti wrote:
> 2012/11/14 Dave Angel <d@davea.name>:
>> On 11/14/2012 10:56 AM, andrea crotti wrote:
>>> Ok this is all very nice, but:
>>>
>>> [andrea@andreacrotti tar_baller]$ time python2 test_pipe.py > /dev/null
>>>
>>> real  0m21.215s
>>> user  0m0.750s
>>> sys   0m1.703s
>>>
>>> [andrea@andreacrotti tar_baller]$ time ls -lR /home/andrea | cat > /dev/null
>>>
>>> real  0m0.986s
>>> user  0m0.413s
>>> sys   0m0.600s
>>>
>>> <snip>
>>>
>>>
>>> So apparently it's way slower than using this system, is this normal?
>>
>> I'm not sure how this timing relates to the thread, but what it mainly
>> shows is that starting up the Python interpreter takes quite a while,
>> compared to not starting it up.
>>
>>
>> --
>>
>> DaveA
>>
> 
> 
> Well it's related because my program has to be as fast as possible, so
> in theory I thought that using Python pipes would be better because I
> can get easily the PID of the first process.
> 
> But if it's so slow than it's not worth, and I don't think is the
> Python interpreter because it's more or less constantly many times
> slower even changing the size of the input..
> 
> 

Well, as I said, I don't see how the particular timing has anything to
do with the rest of the thread.  If you want to do an ls within a Python
program, go ahead.  But if all you need can be done with ls itself, then
it'll be slower to launch python just to run it.

Your first timing runs python, which runs two new shells, ls, and cat.
Your second timing runs ls and cat.

So the difference is starting up python, plus starting the shell two
extra times.

I'd also be curious if you flushed the system buffers before each
timing, as the second test could be running entirely in system memory.
And no, I don't know offhand how to flush them in Linux, just that
without it, your timings are not at all repeatable.  Note the two
identical runs here.

davea@think:~/temppython$ time ls -lR ~ | cat > /dev/null

real	0m0.164s
user	0m0.020s
sys	0m0.000s
davea@think:~/temppython$ time ls -lR ~ | cat > /dev/null

real	0m0.018s
user	0m0.000s
sys	0m0.010s

real time goes down by 90%, while user time drops to zero.
And on a 3rd and subsequent run, sys time goes to zero as well.

-- 

DaveA

Thread

creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-07 17:13 +0000
  Re: creating size-limited tar files Neil Cerutti <neilc@norwich.edu> - 2012-11-07 18:40 +0000
  Re: creating size-limited tar files Alexander Blinne <news@blinne.net> - 2012-11-07 20:05 +0100
    Re: creating size-limited tar files Roy Smith <roy@panix.com> - 2012-11-07 15:32 -0500
      Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-07 21:52 +0000
      Re: creating size-limited tar files Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-07 23:15 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:11 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:29 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-09 10:39 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-13 10:31 +0000
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:07 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:25 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:30 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 11:35 +0530
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-14 00:22 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 14:21 +0530
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 11:52 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 15:56 +0000
      Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:10 -0500
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 16:16 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:33 -0500
  Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-14 20:43 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 15:57 -0500

csiph-web