Re: creating size-limited tar files

Path	csiph.com!usenet.pasdenom.info!gegeweb.org!de-l.enfer-du-nord.net!feeder1.enfer-du-nord.net!feeds.phibee-telecom.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<d@davea.name>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.013
X-Spam-Evidence	'H': 0.97; 'S': 0.00; 'linux,': 0.05; 'file)': 0.07; 'buffer,': 0.09; 'caching,': 0.09; 'subject:files': 0.09; 'whatever.': 0.09; 'cc:addr:python-list': 0.10; 'anyway': 0.11; 'assume': 0.11; 'cases': 0.15; 'file,': 0.15; 'mapping,': 0.16; 'measurement': 0.16; 'pipe,': 0.16; 'pipes': 0.16; 'subprocess': 0.16; 'think?': 0.16; 'wrote:': 0.17; 'pieces': 0.17; 'memory': 0.18; 'os,': 0.22; 'pipe': 0.22; 'work,': 0.22; 'nearly': 0.23; 'idea': 0.24; 'cc:21': 0.24; 'linux': 0.24; 'cc:addr:python.org': 0.25; 'header:In-Reply-To:1': 0.25; 'header :User-Agent:1': 0.26; '(which': 0.26; 'cc:addr:gmail.com': 0.27; 'run': 0.28; 'piece': 0.29; 'subject:size': 0.29; 'probably': 0.29; 'maybe': 0.29; 'usually': 0.30; 'file': 0.32; 'not.': 0.32; 'running': 0.32; 'allocated': 0.33; 'ram': 0.33; 'doing': 0.35; 'pm,': 0.35; 'but': 0.36; 'wanted': 0.36; 'virtual': 0.37; 'subject:: ': 0.38; 'things': 0.38; 'received:192': 0.39; 'little': 0.39; 'received:192.168': 0.40; 'subject:-': 0.40; 'your': 0.60; 'map': 0.61; 'kind': 0.61; 'between': 0.63; 'more': 0.63; 'choose': 0.65; 'frequently': 0.65; 'header:Reply-To:1': 0.68; 'fact,': 0.69; 'physical': 0.69; 'received:74.208': 0.71; 'reply-to:no real name:20': 0.72; 'special': 0.73; 'andrea': 0.84; 'measure.': 0.84; 'thing,': 0.84; 'why?': 0.84; 'subject:limited': 0.93
Date	Wed, 14 Nov 2012 15:57:20 -0500
From	Dave Angel <d@davea.name>
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121011 Thunderbird/16.0.1
MIME-Version	1.0
To	Andrea Crotti <andrea.crotti.0@gmail.com>
Subject	Re: creating size-limited tar files
References	<mailman.3381.1352308430.27098.python-list@python.org> <CAHVvXxTH7WrfT41GDj15f9FS4eguBF2dj4Khpu_vw2TWMkvPYg@mail.gmail.com> <CAF_E5JbY71J3ix9PiXu7wvvkzc_aG9ZtdwKkuxXXAU-SFeRDsw@mail.gmail.com> <CAF_E5JYJ7Fw4b4z81454k9HwmjZ49qjx48fC_f+skcphCDNUeA@mail.gmail.com> <CAF_E5Jas3ObdX=Mhusj63R_Xcm0e1JwjPEuOgZoggON6-cttjA@mail.gmail.com> <CAF_E5JZFJ+TPmF0LUEFU8AVqbOWfJbH9Yy60CKkcrDNnWATiVg@mail.gmail.com> <CALwzidnrTcSK+Z0cGf5WpTN2sLqVgbpbt2i1DcdUFnGQYEvKQg@mail.gmail.com> <87lie4lnxr.fsf@nitrogen.i-did-not-set--mail-host-address--so-tickle-me> <CALwzidkfARHC5Twpw01xQrOWe4msWdLSH2B=5i4oMwadPyQniA@mail.gmail.com> <87fw4clg8b.fsf@nitrogen.i-did-not-set--mail-host-address--so-tickle-me> <CAF_E5JYYEQ77rEQb1tM5tDKbTTSwYNVv1An57bUKn8Y_d-VJUQ@mail.gmail.com> <CAF_E5Jbk2inQbTrurgBKP2DLjzR9=t4XXSGbafuq-E3UvJjaiw@mail.gmail.com> <50A3C287.80707@davea.name> <CAF_E5JaH332cJXko7iLuhNfxyrhE9oeUz3wMZ51XVx5CFE29kw@mail.gmail.com> <50A3C7D7.7050103@davea.name> <50A4028F.2020805@gmail.com>
In-Reply-To	<50A4028F.2020805@gmail.com>
Content-Type	text/plain; charset=ISO-8859-1
Content-Transfer-Encoding	7bit
X-Provags-ID	V02:K0:f9ASFc0mZ0Ir0OWFhbF52psKG4YCc0lLJFRfLJcvLD1 jaRd8C4/FwqwKlo1k+zM1Gyi8Kq7+khmOyxqrB7zT7NVbPmKC4 qeMSGqt7sWRYwQuNMiBjN1UAzt0AfyRiS5Fvo7sY/jVxtINyTP rCsLMF5wPqKHey7jcwWy5rVYEEv/2pWC+He+/MtCt6UpNafpl1 QVmIRVrJfgjgDPCsgcevlrfJflmV0nT4Mtl5PdmpTqkvHDbwLu Rcv5YD1uxQm1oICOCyzxaAzqnTmN6H8D0TEO6XZ/4tebr2vadP svGCf8q9rSk//CrZCYbLCv/AzOQxmtMguyXiaEZgnZX0kJFkA= =
Cc	Python <python-list@python.org>
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
Reply-To	d@davea.name
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.3699.1352926665.27098.python-list@python.org> (permalink)
Lines	35
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1352926665 news.xs4all.nl 6853 [2001:888:2000:d::a6]:54791
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:33355

Show key headers only | View raw

On 11/14/2012 03:43 PM, Andrea Crotti wrote:
> <SNIP>
> Anyway the only thing I wanted to understand is if using the pipes in
> subprocess is exactly the same as doing
> the Linux pipe, or not.

It's not the same thing, but you can usually assume it's close.  Other
effects will probably dominate any differences.
> 
> And any idea on how to run it in ram?
> Maybe if I create a pipe in tmpfs it might already work, what do you think?
> 
> 

In a good virtual OS, such as Linux, there's very little predictable
difference between running in RAM (which is to say reading and writing
to the swap file) or reading and writing to a file you specify.  In
fact, writing to a file can frequently be quicker, if it's sequential.

Why?  Linux is using any given piece of physical RAM to map a file, or
an allocated buffer, or shared memory, or nearly anything.  About the
only special cases are the kind of RAM that has to be locked into RAM
for hardware reasons.

Linux decides which pieces to keep in memory, whether it calls it
caching, swapping, memory mapping, or whatever.  And frequently,
attempts to "beat the system"  result in counterintuitive results.

If in doubt, measure.  But choose your measures carefully, because lots
more things will change the measurement than you might expect.

-- 

DaveA

Back to comp.lang.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-07 17:13 +0000
  Re: creating size-limited tar files Neil Cerutti <neilc@norwich.edu> - 2012-11-07 18:40 +0000
  Re: creating size-limited tar files Alexander Blinne <news@blinne.net> - 2012-11-07 20:05 +0100
    Re: creating size-limited tar files Roy Smith <roy@panix.com> - 2012-11-07 15:32 -0500
      Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-07 21:52 +0000
      Re: creating size-limited tar files Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-07 23:15 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:11 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:29 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-09 10:39 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-13 10:31 +0000
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:07 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:25 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:30 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 11:35 +0530
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-14 00:22 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 14:21 +0530
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 11:52 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 15:56 +0000
      Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:10 -0500
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 16:16 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:33 -0500
  Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-14 20:43 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 15:57 -0500

csiph-web