Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #52581 > unrolled thread

How to I do this in Python ?

Back to article view | Back to comp.lang.python

Contents

  How to I do this in Python ? Ganesh Pal <ganesh1pal@gmail.com> - 2013-08-16 11:51 +0530
    Re: How to I do this in Python ? Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-08-16 10:59 +0000
      Re: How to I do this in Python ? Ganesh Pal <ganesh1pal@gmail.com> - 2013-08-18 22:36 +0530
        Re: How to I do this in Python ? Steven D'Aprano <steve@pearwood.info> - 2013-08-19 07:27 +0000
          Re: How to I do this in Python ? Ganesh Pal <ganesh1pal@gmail.com> - 2013-08-28 10:10 +0530
    Re: How to I do this in Python ? Roy Smith <roy@panix.com> - 2013-08-16 07:47 -0400
      Re: How to I do this in Python ? random832@fastmail.us - 2013-08-16 14:17 -0400

#52581 — How to I do this in Python ?

[Multipart message — attachments visible in raw view] — view raw

Hello  Friends ,

Iam a newbie to python , Iam writing a small script that would generate
various kinds of files
in the specified path . Iam using sub process module to achieve this , I
have stuck with few basic problems , any help on this would be great

Case (a) :

The below code creates the  only one spare file named sp1 ,

 # Creating sparse files in the sparse path
     sparse_path = os.path.join(path,'sparsefiles')
     os.makedirs(sparse_path)
     os.chdir(sparse_path)
     sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
     process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)

Current output :
Sp1,

How do I  loop my script to create 100 of files  like sp1 , sp2 ,sp3,..
sp100  .. using the same syntax
 " sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G "


Case (2) :

Is there a better way to create the files in Python other than using  sub
process module and running dd command as shown below ..

Example :

# creating sparse File
 sparse_path = os.path.join(path,'sparsefiles')
     os.makedirs(sparse_path)
     os.chdir(sparse_path)
     sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
     process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)

    # Creating Regular files
     Regular_path = os.path.join(path,'regularfiles')
     os.makedirs(Regular_path)
     os.chdir(Regular_path)
     regularfiles = " dd if=/dev/urandom of=file1 count=0 bs=1 seek=10"
     process_1 = subprocess.Popen(regularfiles, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)


My goal is to create various kinds of files like sparse, regular
,directories, hard and symlinks etc
what would be the best way to do achieve this ?

Regards,
Ganesh

[toc] | [next] | [standalone]

#52584

Hi Ganesh, and welcome!

Unfortunately, you ask your questions in reverse order. The most general 
(and important) question comes last, and the least important first, so 
I'm going to slice-and-dice your post and answer from most general to 
least.


On Fri, 16 Aug 2013 11:51:32 +0530, Ganesh Pal wrote:

> My goal is to create various kinds of files like sparse, regular
> ,directories, hard and symlinks etc
> what would be the best way to do achieve this ?

Use your shell, such as bash or csh or equivalent. For simple tasks like 
that, it will solve the problem much more simply than Python.

There are three good reasons for doing this in Python:

- "This is only a small part of a larger Python application."

- "I'm doing this to learn how to use Python."

- "I really hate my shell."

But of you just want to get the job done, and don't care what language 
you use, use the shell.


Now, having said that, I'm going to assume you have a good reason to use 
Python:


> Case (2) :
> 
> Is there a better way to create the files in Python other than using 
> sub process module and running dd command as shown below ..
> 
> Example :
> 
> # creating sparse File
>  sparse_path = os.path.join(path,'sparsefiles')
>      os.makedirs(sparse_path)
>      os.chdir(sparse_path)
>      sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
>      process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
> stderr=subprocess.PIPE, shell=True)
> 
>     # Creating Regular files
>      Regular_path = os.path.join(path,'regularfiles')
>      os.makedirs(Regular_path)
>      os.chdir(Regular_path)
>      regularfiles = " dd if=/dev/urandom of=file1 count=0 bs=1 seek=10"
>      process_1 = subprocess.Popen(regularfiles, stdout=subprocess.PIPE,
> stderr=subprocess.PIPE, shell=True)

What do you mean by "better"? There's probably nothing that will be 
faster at rapidly copying bytes from one file to another than dd. But not 
with a blocksize of 1 byte at a time. It's more usual to set bs=512 or 
bs=1024.

Oh, I see you're not actually writing anything to the file (count=0). In 
that case, instead of using dd, you should use touch.

I'm not sure that shell=True is a good idea. 

In Python, to create a new empty file, setting its contents to empty if 
it already exists:

open("filename", "w").close()


That will open the file, creating it if it doesn't exist, emptying it if 
it does, then close it.

To touch a file without emptying it:

open("filename", "a").close()


To make a sparse file, assuming your file system supports it, I believe 
you actually have to write at least one byte to the file:

f = open('foo', 'w')
f.seek(10000)
f.write('\0')
f.close()



[...]
> How do I  loop my script to create 100 of files  like sp1 , sp2 ,sp3,..
> sp100  .. using the same syntax

To generate the various file names, you need to loop over a counter from 
1 to 100, and stick the count into a string. Use a for-loop and the range 
function:

for i in range(1, 101):
    filename = "sp" + str(i)
    open(filename, "a").close()


If you are a C programmer, you might prefer this style:

    filename = "sp%d" % i

Or a more object-oriented style:

    filename = "sp{:d}".format(i)



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#52658

[Multipart message — attachments visible in raw view] — view raw

Hi Steven ,

Firstly thanks for responding to the question and also guiding me on how to
post the question in the right order ( general to least important order )

Please find the comments >>> inline

On Fri, 16 Aug 2013 11:51:32 +0530, Ganesh Pal wrote:

> My goal is to create various kinds of files like sparse, regular
> ,directories, hard and symlinks etc
> what would be the best way to do achieve this ?

Use your shell, such as bash or csh or equivalent. For simple tasks like
that, it will solve the problem much more simply than Python.

There are three good reasons for doing this in Python:

- "This is only a small part of a larger Python application."

- "I'm doing this to learn how to use Python."

- "I really hate my shell."

But of you just want to get the job done, and don't care what language
you use, use the shell.


Now, having said that, I'm going to assume you have a good reason to use
Python:

>>>  You  have  hit the nail on the head  - The main goal there was to
learn python and avoid using shell commands as much as possible  , I want
to avoid shell and use Python even if its costly.

        Example : I don' want to create sparse files using  dd   command ( #dd
if=/dev/zero of=sparse-file bs=1 count=1 seek=1024k ) If  the same can be
done  fseek .

         But I guess we might have to d


> Case (2) :
>
> Is there a better way to create the files in Python other than using
> sub process module and running dd command as shown below ..
>
> Example :
>
> # creating sparse File
>  sparse_path = os.path.join(path,'sparsefiles')
>      os.makedirs(sparse_path)
>      os.chdir(sparse_path)
>      sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
>      process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
> stderr=subprocess.PIPE, shell=True)
>
>     # Creating Regular files
>      Regular_path = os.path.join(path,'regularfiles')
>      os.makedirs(Regular_path)
>      os.chdir(Regular_path)
>      regularfiles = " dd if=/dev/urandom of=file1 count=0 bs=1 seek=10"
>      process_1 = subprocess.Popen(regularfiles, stdout=subprocess.PIPE,
> stderr=subprocess.PIPE, shell=True)

What do you mean by "better"? There's probably nothing that will be
faster at rapidly copying bytes from one file to another than dd. But not
with a blocksize of 1 byte at a time. It's more usual to set bs=512 or
bs=1024.

Oh, I see you're not actually writing anything to the file (count=0). In
that case, instead of using dd, you should use touch.

I'm not sure that shell=True is a good idea.

In Python, to create a new empty file, setting its contents to empty if
it already exists:

open("filename", "w").close()


That will open the file, creating it if it doesn't exist, emptying it if
it does, then close it.

To touch a file without emptying it:

open("filename", "a").close()


To make a sparse file, assuming your file system supports it, I believe
you actually have to write at least one byte to the file:

f = open('foo', 'w')
f.seek(10000)
f.write('\0')
f.close()

>>>   Thanks for the suggestion on creating empty files and  they worked
fine . I was today playing with the temp-file  module to create  temporary
files and directories ,
         Iam yet to explore it completely, but in current context , I had a
quick question using temp-file module is its  possible to create empty
 temporary files and save them on disk in the user-defined path ?
         If  " yes "  then can this also be an alternative and does this
have any drawback  ?


On Fri, Aug 16, 2013 at 4:29 PM, Steven D'Aprano <
steve+comp.lang.python@pearwood.info> wrote:

> Hi Ganesh, and welcome!
>
> Unfortunately, you ask your questions in reverse order. The most general
> (and important) question comes last, and the least important first, so
> I'm going to slice-and-dice your post and answer from most general to
> least.
>
>
> On Fri, 16 Aug 2013 11:51:32 +0530, Ganesh Pal wrote:
>
> > My goal is to create various kinds of files like sparse, regular
> > ,directories, hard and symlinks etc
> > what would be the best way to do achieve this ?
>
> Use your shell, such as bash or csh or equivalent. For simple tasks like
> that, it will solve the problem much more simply than Python.
>
> There are three good reasons for doing this in Python:
>
> - "This is only a small part of a larger Python application."
>
> - "I'm doing this to learn how to use Python."
>
> - "I really hate my shell."
>
> But of you just want to get the job done, and don't care what language
> you use, use the shell.
>
>
> Now, having said that, I'm going to assume you have a good reason to use
> Python:
>
>
> > Case (2) :
> >
> > Is there a better way to create the files in Python other than using
> > sub process module and running dd command as shown below ..
> >
> > Example :
> >
> > # creating sparse File
> >  sparse_path = os.path.join(path,'sparsefiles')
> >      os.makedirs(sparse_path)
> >      os.chdir(sparse_path)
> >      sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
> >      process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
> > stderr=subprocess.PIPE, shell=True)
> >
> >     # Creating Regular files
> >      Regular_path = os.path.join(path,'regularfiles')
> >      os.makedirs(Regular_path)
> >      os.chdir(Regular_path)
> >      regularfiles = " dd if=/dev/urandom of=file1 count=0 bs=1 seek=10"
> >      process_1 = subprocess.Popen(regularfiles, stdout=subprocess.PIPE,
> > stderr=subprocess.PIPE, shell=True)
>
> What do you mean by "better"? There's probably nothing that will be
> faster at rapidly copying bytes from one file to another than dd. But not
> with a blocksize of 1 byte at a time. It's more usual to set bs=512 or
> bs=1024.
>
> Oh, I see you're not actually writing anything to the file (count=0). In
> that case, instead of using dd, you should use touch.
>
> I'm not sure that shell=True is a good idea.
>
> In Python, to create a new empty file, setting its contents to empty if
> it already exists:
>
> open("filename", "w").close()
>
>
> That will open the file, creating it if it doesn't exist, emptying it if
> it does, then close it.
>
> To touch a file without emptying it:
>
> open("filename", "a").close()
>
>
> To make a sparse file, assuming your file system supports it, I believe
> you actually have to write at least one byte to the file:
>
> f = open('foo', 'w')
> f.seek(10000)
> f.write('\0')
> f.close()
>
>
>
> [...]
> > How do I  loop my script to create 100 of files  like sp1 , sp2 ,sp3,..
> > sp100  .. using the same syntax
>
> To generate the various file names, you need to loop over a counter from
> 1 to 100, and stick the count into a string. Use a for-loop and the range
> function:
>
> for i in range(1, 101):
>     filename = "sp" + str(i)
>     open(filename, "a").close()
>
>
> If you are a C programmer, you might prefer this style:
>
>     filename = "sp%d" % i
>
> Or a more object-oriented style:
>
>     filename = "sp{:d}".format(i)
>
>
>
> --
> Steven
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[toc] | [prev] | [next] | [standalone]

#52675

On Sun, 18 Aug 2013 22:36:01 +0530, Ganesh Pal wrote:

> Please find the comments >>> inline

Please don't do that!

"Arrows" > are used for quoting in emails. If you prefix your *new* 
comments using >>> it looks like they were quoted *three messages back*.

You should be able to configure your email or news client to prefix 
quoted text with a >, and then you just type your own comments with no 
prefix, like I'm doing here. Even Gmail can do that.

You seem to have copied-and-pasted my response into a new email, and then 
added your comments. Am I right? The normal way to reply to an email is 
to use Reply or Reply All.


[...]
>>>>   Thanks for the suggestion on creating empty files and  they worked
> fine . I was today playing with the temp-file  module to create 
> temporary files and directories ,
>          Iam yet to explore it completely, but in current context , I
>          had a
> quick question using temp-file module is its  possible to create empty
>  temporary files and save them on disk in the user-defined path ?

You shouldn't normally care about where temporary files are stored, since 
they're temporary and will disappear as soon as you are done with them. 
But yes, tempfile has the ability to control where the files are stored. 
Both tempfile.NamedTemporaryFile and tempfile.TemporaryFile take an 
optional directory argument.

To read the documentation, run these two commands at the interactive 
prompt:


import tempfile
help(tempfile)


or read it on the web:

http://docs.python.org/2/library/tempfile.html
http://docs.python.org/3/library/tempfile.html



>          If  " yes "  then can this also be an alternative and does this
> have any drawback  ?

Alternative to what? If you mean, alternative to *not* storing it in the 
user's directory, then yes, it is :-)

Drawbacks -- yes. I hate it when applications dump temporary files in my 
home directory.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#53108

[Multipart message — attachments visible in raw view] — view raw

On Mon, Aug 19, 2013 at 12:57 PM, Steven D'Aprano <steve@pearwood.info>wrote:

> On Sun, 18 Aug 2013 22:36:01 +0530, Ganesh Pal wrote:
>
> > Please find the comments >>> inline
>
> Please don't do that!
>
> "Arrows" > are used for quoting in emails. If you prefix your *new*
> comments using >>> it looks like they were quoted *three messages back*.
>
> You should be able to configure your email or news client to prefix
> quoted text with a >, and then you just type your own comments with no
> prefix, like I'm doing here. Even Gmail can do that.
>
> You seem to have copied-and-pasted my response into a new email, and then
> added your comments. Am I right? The normal way to reply to an email is
> to use Reply or Reply All.
>
>
sure , hence forth will take of the same


>
>
> You shouldn't normally care about where temporary files are stored, since
> they're temporary and will disappear as soon as you are done with them.
> But yes, tempfile has the ability to control where the files are stored.
> Both tempfile.NamedTemporaryFile and tempfile.TemporaryFile take an
> optional directory argument.
>
> To read the documentation, run these two commands at the interactive
> prompt:
>
>
> import tempfile
> help(tempfile)
>
>
> or read it on the web:
>
> http://docs.python.org/2/library/tempfile.html
> http://docs.python.org/3/library/tempfile.html
>
>
>
> >          If  " yes "  then can this also be an alternative and does this
> > have any drawback  ?
>
> Alternative to what? If you mean, alternative to *not* storing it in the
> user's directory, then yes, it is :-)
>
> Drawbacks -- yes. I hate it when applications dump temporary files in my
> home directory.
>
>
>
Thanks for the links  and information on temporary files

[toc] | [prev] | [next] | [standalone]

#52586

In article <mailman.614.1376636762.1251.python-list@python.org>,
 Ganesh Pal <ganesh1pal@gmail.com> wrote:

>  # Creating sparse files in the sparse path
>      sparse_path = os.path.join(path,'sparsefiles')
>      os.makedirs(sparse_path)
>      os.chdir(sparse_path)
>      sparsefiles = "dd if=/dev/zero of=sp1 count=0 bs=1 seek=10G"
>      process_0 = subprocess.Popen(sparsefiles, stdout=subprocess.PIPE,
> stderr=subprocess.PIPE, shell=True)

There is no need to shell out to dd just to do this.  All dd is doing 
for you is seeking to the offset you specify and closing the file.  You 
can do that entirely in Python code.

http://docs.python.org/2.7/library/stdtypes.html#file.seek

[toc] | [prev] | [next] | [standalone]

#52601

On Fri, Aug 16, 2013, at 7:47, Roy Smith wrote:
> There is no need to shell out to dd just to do this.  All dd is doing 
> for you is seeking to the offset you specify and closing the file.  You 
> can do that entirely in Python code.
> 
> http://docs.python.org/2.7/library/stdtypes.html#file.seek

*ahem*

http://docs.python.org/2.7/library/stdtypes.html#file.truncate

The dd recipe was written some time over a decade ago as the easiest way
to achieve this at a shell prompt (before the truncate tool was added to
coreutils), and no-one's ever thought twice about it.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web