Path: csiph.com!usenet.pasdenom.info!aioe.org!news.stack.nl!newsfeed.xs4all.nl!newsfeed6.news.xs4all.nl!xs4all!newsgate.cistron.nl!newsgate.news.xs4all.nl!post.news.xs4all.nl!not-for-mail
MIME-Version: 1.0
In-Reply-To: <CAF_E5Jas3ObdX=Mhusj63R_Xcm0e1JwjPEuOgZoggON6-cttjA@mail.gmail.com>
References: <mailman.3381.1352308430.27098.python-list@python.org> <509ab0fa$0$6636$9b4e6d93@newsspool2.arcor-online.net> <roy-2C9B4D.15322307112012@news.panix.com> <509AD812.2060605@gmail.com> <CAHVvXxTH7WrfT41GDj15f9FS4eguBF2dj4Khpu_vw2TWMkvPYg@mail.gmail.com> <CAF_E5JbY71J3ix9PiXu7wvvkzc_aG9ZtdwKkuxXXAU-SFeRDsw@mail.gmail.com> <CAF_E5JYJ7Fw4b4z81454k9HwmjZ49qjx48fC_f+skcphCDNUeA@mail.gmail.com> <CAF_E5Jas3ObdX=Mhusj63R_Xcm0e1JwjPEuOgZoggON6-cttjA@mail.gmail.com>
Date: Tue, 13 Nov 2012 10:31:32 +0000
Subject: Re: creating size-limited tar files
From: andrea crotti <andrea.crotti.0@gmail.com>
To: Oscar Benjamin <oscar.j.benjamin@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Cc: Python List <python-list@python.org>, Roy Smith <roy@panix.com>
Precedence: list
Newsgroups: comp.lang.python
Message-ID: <mailman.3619.1352802696.27098.python-list@python.org>
Lines: 50
NNTP-Posting-Host: 2001:888:2000:d::a6
Xref: csiph.com comp.lang.python:33232

2012/11/9 andrea crotti <andrea.crotti.0@gmail.com>:
> Anyway in the meanwhile I implemented this tar and split in this way below.
> It works very well and it's probably much faster, but the downside is that
> I give away control to tar and split..
>
> def tar_and_split(inputfile, output, bytes_size=None):
>     """Take the file containing all the files to compress, the bytes
>     desired for the split and the base name of the output file
>     """
>     # cleanup first
>     for fname in glob(output + "*"):
>         logger.debug("Removing old file %s" % fname)
>         remove(fname)
>
>     out = '-' if bytes_size else (output + '.tar.gz')
>     cmd = "tar czpf {} $(cat {})".format(out, inputfile)
>     if bytes_size:
>         cmd += "| split -b {} -d - {}".format(bytes_size, output)
>
>     logger.info("Running command %s" % cmd)
>
>     proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
>     out, err = proc.communicate()
>     if err:
>         logger.error("Got error messages %s" % err)
>
>     logger.info("Output %s" % out)
>
>     if proc.returncode != 0:
>         logger.error("Something failed running %s, need to re-run" % cmd)
>         return False


There is another problem with this solution, if I run something like
this with Popen:
    cmd = "tar {bigc} -czpf - --files-from {inputfile} | split -b
{bytes_size} -d - {output}"

    proc = subprocess.Popen(to_run, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)

the proc.returncode will only be the one from "split", so I lose the
ability to check if tar failed..

A solution would be something like this:
{ ls -dlkfjdsl; echo $? > tar.status; } | split

but it's a bit ugly.  I wonder if I can use the subprocess PIPEs to do
the same thing, is it going to be as fast and work in the same way??