creating size-limited tar files

Path	csiph.com!usenet.pasdenom.info!news.albasani.net!newsfeed.freenet.ag!news2.euro.net!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
Return-Path	<andrea.crotti.0@gmail.com>
X-Original-To	python-list@python.org
Delivered-To	python-list@mail.python.org
X-Spam-Status	OK 0.023
X-Spam-Evidence	'H': 0.95; 'S': 0.00; 'received:209.85.223': 0.03; 'chunks': 0.09; 'mode,': 0.09; 'subject:files': 0.09; 'to:name:python-list': 0.15; 'compress': 0.16; 'simplest': 0.16; 'tarfile': 0.16; 'replacing': 0.17; '(in': 0.18; 'script': 0.24; 'skip:" 20': 0.26; 'message-id:@mail.gmail.com': 0.27; "doesn't": 0.28; 'actual': 0.28; 'subject:size': 0.29; 'tar': 0.29; "i'm": 0.29; 'file': 0.32; 'to:addr:python-list': 0.33; 'received:google.com': 0.34; 'thanks': 0.34; 'exist': 0.35; 'problem,': 0.35; 'doing': 0.35; 'received:209.85': 0.35; 'there': 0.35; 'but': 0.36; 'should': 0.36; 'moment': 0.37; 'received:209': 0.37; 'data': 0.37; 'to:addr:python.org': 0.39; 'subject:-': 0.40; 'header:Received:5': 0.40; 'skip:n 10': 0.63; 'maybe,': 0.84; 'size?': 0.84; 'subject:limited': 0.93
DKIM-Signature	v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=kDkCVWROWvjQry8OxmGIAh/qVl2EXi/8KG9aC8M2UFc=; b=snwGHpQO7NEfiNmyihdptt3ojEd65yPtRWLNAhcUoLNOXPYA9WlrCRNvBx5AZuAIYz mPnfjPTCqYKv7LCyttlSR2JyhFZVaNvS1F85SlbPc4YXh+wKfyPQ2R9vlpD/uAW/0Q9w DWTSFfH5tVnfEfhsfMHE3CdLwxELtt+jCmAeFJVjmM2ZTSHC7pnBvoPLEm2icYBRIr4t pb3jOmvTfHNzDzwLhYxDFxGb4vIfZ7ZwHQDFUwrRywQ8v4h7uHTwJS7zALSXbDmU+Umj BBEu+Wt4vENzLEfd97WFqhWxSJsmjabXJ0Pq+ZRhaK6wLgGQ8iJ10h8q3E/bWYydJtb4 a4Ww==
MIME-Version	1.0
Date	Wed, 7 Nov 2012 17:13:47 +0000
Subject	creating size-limited tar files
From	andrea crotti <andrea.crotti.0@gmail.com>
To	python-list <python-list@python.org>
Content-Type	text/plain; charset=ISO-8859-1
X-BeenThere	python-list@python.org
X-Mailman-Version	2.1.15
Precedence	list
List-Id	General discussion list for the Python programming language <python-list.python.org>
List-Unsubscribe	<http://mail.python.org/mailman/options/python-list>, <mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive	<http://mail.python.org/pipermail/python-list/>
List-Post	<mailto:python-list@python.org>
List-Help	<mailto:python-list-request@python.org?subject=help>
List-Subscribe	<http://mail.python.org/mailman/listinfo/python-list>, <mailto:python-list-request@python.org?subject=subscribe>
Newsgroups	comp.lang.python
Message-ID	<mailman.3381.1352308430.27098.python-list@python.org> (permalink)
Lines	24
NNTP-Posting-Host	2001:888:2000:d::a6
X-Trace	1352308430 news.xs4all.nl 6970 [2001:888:2000:d::a6]:38167
X-Complaints-To	abuse@xs4all.nl
Xref	csiph.com comp.lang.python:32896

Show key headers only | View raw

Simple problem, given a lot of data in many files/directories, I
should create a tar file splitted in chunks <= a given size.

The simplest way would be to compress the whole thing and then split.

At the moment the actual script which I'm replacing is doing a
"system('split..')", which is not that great, so I would like to do it
while compressing.

So I thought about (in pseudocode)


while remaining_files:
    tar_file.addfile(remaining_files.pop())
    if size(tar_file) >= limit:
         close(tar_file)
         tar_file = new_tar_file()

which might work maybe, but how do I get the current size?  There
should be tarinfo.size but it doesn't exist on a TarFile opened in
write mode, so should I do a stat after each flush?

Any other better ideas otherwise?
thanks

Back to comp.lang.python | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-07 17:13 +0000
  Re: creating size-limited tar files Neil Cerutti <neilc@norwich.edu> - 2012-11-07 18:40 +0000
  Re: creating size-limited tar files Alexander Blinne <news@blinne.net> - 2012-11-07 20:05 +0100
    Re: creating size-limited tar files Roy Smith <roy@panix.com> - 2012-11-07 15:32 -0500
      Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-07 21:52 +0000
      Re: creating size-limited tar files Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-07 23:15 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:11 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:29 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-09 10:39 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-13 10:31 +0000
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:07 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:25 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:30 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 11:35 +0530
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-14 00:22 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 14:21 +0530
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 11:52 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 15:56 +0000
      Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:10 -0500
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 16:16 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:33 -0500
  Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-14 20:43 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 15:57 -0500

csiph-web