Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #33232

Re: creating size-limited tar files

Show all headers | View raw

2012/11/9 andrea crotti <andrea.crotti.0@gmail.com>:
> Anyway in the meanwhile I implemented this tar and split in this way below.
> It works very well and it's probably much faster, but the downside is that
> I give away control to tar and split..
>
> def tar_and_split(inputfile, output, bytes_size=None):
>     """Take the file containing all the files to compress, the bytes
>     desired for the split and the base name of the output file
>     """
>     # cleanup first
>     for fname in glob(output + "*"):
>         logger.debug("Removing old file %s" % fname)
>         remove(fname)
>
>     out = '-' if bytes_size else (output + '.tar.gz')
>     cmd = "tar czpf {} $(cat {})".format(out, inputfile)
>     if bytes_size:
>         cmd += "| split -b {} -d - {}".format(bytes_size, output)
>
>     logger.info("Running command %s" % cmd)
>
>     proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
>     out, err = proc.communicate()
>     if err:
>         logger.error("Got error messages %s" % err)
>
>     logger.info("Output %s" % out)
>
>     if proc.returncode != 0:
>         logger.error("Something failed running %s, need to re-run" % cmd)
>         return False


There is another problem with this solution, if I run something like
this with Popen:
    cmd = "tar {bigc} -czpf - --files-from {inputfile} | split -b
{bytes_size} -d - {output}"

    proc = subprocess.Popen(to_run, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)

the proc.returncode will only be the one from "split", so I lose the
ability to check if tar failed..

A solution would be something like this:
{ ls -dlkfjdsl; echo $? > tar.status; } | split

but it's a bit ugly.  I wonder if I can use the subprocess PIPEs to do
the same thing, is it going to be as fast and work in the same way??

Back to comp.lang.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-07 17:13 +0000
  Re: creating size-limited tar files Neil Cerutti <neilc@norwich.edu> - 2012-11-07 18:40 +0000
  Re: creating size-limited tar files Alexander Blinne <news@blinne.net> - 2012-11-07 20:05 +0100
    Re: creating size-limited tar files Roy Smith <roy@panix.com> - 2012-11-07 15:32 -0500
      Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-07 21:52 +0000
      Re: creating size-limited tar files Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2012-11-07 23:15 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:11 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-08 10:29 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-09 10:39 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-13 10:31 +0000
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:07 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:25 -0700
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-13 09:30 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 11:35 +0530
      Re: creating size-limited tar files Ian Kelly <ian.g.kelly@gmail.com> - 2012-11-14 00:22 -0700
      Re: creating size-limited tar files Kushal Kumaran <kushal.kumaran+python@gmail.com> - 2012-11-14 14:21 +0530
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 11:52 +0000
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 15:56 +0000
      Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:10 -0500
      Re: creating size-limited tar files andrea crotti <andrea.crotti.0@gmail.com> - 2012-11-14 16:16 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 11:33 -0500
  Re: creating size-limited tar files Andrea Crotti <andrea.crotti.0@gmail.com> - 2012-11-14 20:43 +0000
  Re: creating size-limited tar files Dave Angel <d@davea.name> - 2012-11-14 15:57 -0500

csiph-web