Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #32939

Re: creating size-limited tar files

Path csiph.com!usenet.pasdenom.info!weretis.net!feeder1.news.weretis.net!feeder.erje.net!eu.feeder.erje.net!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path <andrea.crotti.0@gmail.com>
X-Original-To python-list@python.org
Delivered-To python-list@mail.python.org
X-Spam-Status OK 0.002
X-Spam-Evidence '*H*': 1.00; '*S*': 0.00; 'python,': 0.02; 'broken': 0.03; 'skip:[ 20': 0.03; 'sys': 0.05; 'skip:/ 10': 0.07; 'filesystem': 0.09; 'subject:files': 0.09; 'cc:addr:python-list': 0.10; '(the': 0.15; 'cc:name:python list': 0.16; 'meanwhile': 0.16; 'reliably': 0.16; 'pipe': 0.22; 'split': 0.23; 'this:': 0.23; 'cc:2**1': 0.24; 'linux': 0.24; 'script': 0.24; 'tried': 0.25; 'cc:addr:python.org': 0.25; 'header:In-Reply-To:1': 0.25; 'message-id:@mail.gmail.com': 0.27; 'run': 0.28; 'subject:size': 0.29; 'tar': 0.29; 'things,': 0.29; 'probably': 0.29; 'received:209.85.210.174': 0.30; 'doubt': 0.33; 'received:google.com': 0.34; 'list': 0.35; 'fail': 0.35; 'faster': 0.35; 'doing': 0.35; 'received:209.85': 0.35; 'but': 0.36; 'should': 0.36; 'bad': 0.37; 'received:209': 0.37; 'subject:: ': 0.38; 'files': 0.38; 'some': 0.38; 'subject:-': 0.40; 'header:Received:5': 0.40; 'real': 0.61; 'more': 0.63; 'saw': 0.75; 'andrea': 0.84; 'easier,': 0.84; 'subject:limited': 0.93
DKIM-Signature v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=c9KxtGj+o6inK4E5zbNBtkv1VUDmoecWo3piGv51WmY=; b=u8+N8Nh2Xa+Dc8zvNYzniRWXk7C8YMMhdOsYjAojJG2Md+JQhkJa0qFzL20IddQBcB 3Tam9lLO4vNJSP7VcqKQrWGPto+wJclDfv5WFux8U9JqhUuDXIKnazqTBk7C1SdV+1as zu5CwGR61MxteIm25Yqmid6Knu/mOMUAj6MBMlXu9QGzU4UQ/ZYe8Y1sPZbpFxCSUVCc QDciahomfQHSUFcn2LjNpnG3u7wz/5bwnt5ViPG8Gy0YgGvkCbR8bSOHMu0/Bu5duPEx /4mmepuLIIcQkWErYYFrJihUCCXr7wb0AUT9Q82AwEpAuWMlZH714w61eDQcz3pGBsZb XbNw==
MIME-Version 1.0
In-Reply-To <CAF_E5JbY71J3ix9PiXu7wvvkzc_aG9ZtdwKkuxXXAU-SFeRDsw@mail.gmail.com>
References <mailman.3381.1352308430.27098.python-list@python.org> <509ab0fa$0$6636$9b4e6d93@newsspool2.arcor-online.net> <roy-2C9B4D.15322307112012@news.panix.com> <509AD812.2060605@gmail.com> <CAHVvXxTH7WrfT41GDj15f9FS4eguBF2dj4Khpu_vw2TWMkvPYg@mail.gmail.com> <CAF_E5JbY71J3ix9PiXu7wvvkzc_aG9ZtdwKkuxXXAU-SFeRDsw@mail.gmail.com>
Date Thu, 8 Nov 2012 10:29:36 +0000
Subject Re: creating size-limited tar files
From andrea crotti <andrea.crotti.0@gmail.com>
To Oscar Benjamin <oscar.j.benjamin@gmail.com>
Content-Type text/plain; charset=ISO-8859-1
Cc Python List <python-list@python.org>, Roy Smith <roy@panix.com>
X-BeenThere python-list@python.org
X-Mailman-Version 2.1.15
Precedence list
List-Id General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe <http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive <http://mail.python.org/pipermail/python-list/>
List-Post <mailto:python-list@python.org>
List-Help <mailto:python-list-request@python.org?subject=help>
List-Subscribe <http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups comp.lang.python
Message-ID <mailman.3429.1352370578.27098.python-list@python.org> (permalink)
Lines 41
NNTP-Posting-Host 2001:888:2000:d::a6
X-Trace 1352370578 news.xs4all.nl 6847 [2001:888:2000:d::a6]:45489
X-Complaints-To abuse@xs4all.nl
Xref csiph.com comp.lang.python:32939

Show key headers only | View raw

2012/11/8 andrea crotti <andrea.crotti.0@gmail.com>:
>
>
>
> Yes yes I saw the answer, but now I was thinking that what I need is
> simply this:
> tar czpvf - /path/to/archive | split -d -b 100M - tardisk
>
> since it should run only on Linux it's probably way easier, my script
> will then only need to create the list of files to tar..
>
> The only doubt is if this is more or less reliably then doing it in
> Python, when can this fail with some bad broken pipe?
> (the filesystem is not very good as I said and it's mounted with NFS)

In the meanwhile I tried a couple of things, and using the pipe on
Linux actually works very nicely, it's even faster than simple tar for
some reasons..

[andrea@andreacrotti isos]$ time tar czpvf - file1.avi file2.avi |
split -d -b 1000M - inchunks
file1.avi
file2.avi

real	1m39.242s
user	1m14.415s
sys	0m7.140s

[andrea@andreacrotti isos]$ time tar czpvf total.tar.gz file1.avi file2.avi
file1.avi
file2.avi

real	1m41.190s
user	1m13.849s
sys	0m5.723s

[andrea@andreacrotti isos]$ time split -d -b 1000M total.tar.gz inchunks

real	0m55.282s
user	0m0.020s
sys	0m3.553s

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-07 17:13 +0000
  Re: creating size-limited tar files Neil Cerutti <neilc@norwich.edu> - 2012-11-07 18:40 +0000
  Re: creating size-limited tar files Alexander Blinne <news@blinne.net> - 2012-11-07 20:05 +0100
    Re: creating size-limited tar files Roy Smith <roy@panix.com> - 2012-11-07 15:32 -0500
      Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-07 21:52 +0000
      Re: creating size-limited tar files Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-07 23:15 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:11 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:29 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-09 10:39 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-13 10:31 +0000
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:07 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:25 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:30 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 11:35 +0530
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-14 00:22 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 14:21 +0530
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 11:52 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 15:56 +0000
      Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:10 -0500
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 16:16 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:33 -0500
  Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-14 20:43 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 15:57 -0500

csiph-web