Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #99676 > unrolled thread

Generate config file from template using Python search and replace.

Back to article view | Back to comp.lang.python

Contents

  Generate config file from template using Python search and replace. Mr Zaug <matthew.herzog@gmail.com> - 2015-11-28 13:45 -0800
    Re: Generate config file from template using Python search and replace. Mr Zaug <matthew.herzog@gmail.com> - 2015-11-28 14:04 -0800
    Re: Generate config file from template using Python search and replace. Peter Otten <__peter__@web.de> - 2015-11-29 00:07 +0100
      Re: Generate config file from template using Python search and replace. Mr Zaug <matthew.herzog@gmail.com> - 2015-11-29 12:23 -0800
        Re: Generate config file from template using Python search and replace. Peter Otten <__peter__@web.de> - 2015-11-29 23:50 +0100
          Re: Generate config file from template using Python search and replace. Mr Zaug <matthew.herzog@gmail.com> - 2015-11-29 17:28 -0800
            Re: Generate config file from template using Python search and replace. Peter Otten <__peter__@web.de> - 2015-11-30 10:14 +0100
              Re: Generate config file from template using Python search and replace. Mr Zaug <matthew.herzog@gmail.com> - 2015-11-30 20:54 -0800
                Re: Generate config file from template using Python search and replace. Peter Otten <__peter__@web.de> - 2015-12-01 11:49 +0100
                  Re: Generate config file from template using Python search and replace. Mr Zaug <matthew.herzog@gmail.com> - 2015-12-01 05:18 -0800
                  Re: Generate config file from template using Python search and replace. Mr Zaug <matthew.herzog@gmail.com> - 2015-12-01 06:18 -0800
                    Re: Generate config file from template using Python search and replace. Peter Otten <__peter__@web.de> - 2015-12-01 15:41 +0100
                      Re: Generate config file from template using Python search and replace. Mr Zaug <matthew.herzog@gmail.com> - 2015-12-01 07:21 -0800
                        Re: Generate config file from template using Python search and replace. Peter Otten <__peter__@web.de> - 2015-12-01 16:43 +0100
                          Re: Generate config file from template using Python search and replace. Mr Zaug <matthew.herzog@gmail.com> - 2015-12-01 10:43 -0800
          Re: Generate config file from template using Python search and replace. Mr Zaug <matthew.herzog@gmail.com> - 2015-11-29 18:31 -0800
            Re: Generate config file from template using Python search and replace. Rob Hills <rhills@medimorphosis.com.au> - 2015-11-30 10:40 +0800
            Re: Generate config file from template using Python search and replace. MRAB <python@mrabarnett.plus.com> - 2015-11-30 03:23 +0000

#99676 — Generate config file from template using Python search and replace.

I need to generate a config file based on an existing "template" file. I need to replace a set of strings with other strings globally in the generated file.

Here is a snippet of the template file, where CONTENT_PATH and DAMPATH are two "placeholders" or variables. There are several other such placeholders.

      $include "_dispatcher_publish_filters.any"
      /1000 { /type "allow"  /glob "* /CONTENT_PATH/*.html*" }
      /1001 { /type "allow"  /glob "POST /DAMPATH/www/*.html *" }

The script's user will be asked to type in unique values when prompted for DAMPATH or CONTENT_PATH.

Since I know the variables themselves are not going to change (because the contents of the template file don't spontaneously change) should I be using regex to search for them or is there a better way? I was planning on using re.sub but I don't know whether that's the best way. Here's what my script looks like today.

from sys import argv
import re
from os.path import exists

script, template_file = argv
print "Opening the template file..."

in_file = open(template_file)
lines = in_file.readlines()

print "What is the serial number of the site?",
_NNN = raw_input()

print "What is the brand, or product name?",
_BRAND = raw_input()

print "What is the content path?",
_CONTENT_PATH = raw_input()

out_file = open(_nnn + _brand + "_farm.any", 'w')

for line in lines:
   re.sub('NNN', _NNN, line)
   re.sub('BRAND, _BRAND', line)
   re.sub('CONTENT_PATH', _CONTENT_PATH, line)

out_file.close()

[toc] | [next] | [standalone]

#99679

I should mention the template file is small, just 98 lines long and the working config file will be the same size.

[toc] | [prev] | [next] | [standalone]

#99683

Mr Zaug wrote:

> I need to generate a config file based on an existing "template" file. I
> need to replace a set of strings with other strings globally in the
> generated file.
> 
> Here is a snippet of the template file, where CONTENT_PATH and DAMPATH are
> two "placeholders" or variables. There are several other such
> placeholders.
> 
>       $include "_dispatcher_publish_filters.any"
>       /1000 { /type "allow"  /glob "* /CONTENT_PATH/*.html*" }
>       /1001 { /type "allow"  /glob "POST /DAMPATH/www/*.html *" }
> 
> The script's user will be asked to type in unique values when prompted for
> DAMPATH or CONTENT_PATH.
> 
> Since I know the variables themselves are not going to change (because the
> contents of the template file don't spontaneously change) should I be
> using regex to search for them or is there a better way? I was planning on
> using re.sub but I don't know whether that's the best way. Here's what my
> script looks like today.
> 
> from sys import argv
> import re
> from os.path import exists
> 
> script, template_file = argv
> print "Opening the template file..."
> 
> in_file = open(template_file)
> lines = in_file.readlines()
> 
> print "What is the serial number of the site?",
> _NNN = raw_input()
> 
> print "What is the brand, or product name?",
> _BRAND = raw_input()
> 
> print "What is the content path?",
> _CONTENT_PATH = raw_input()
> 
> out_file = open(_nnn + _brand + "_farm.any", 'w')
> 
> for line in lines:
>    re.sub('NNN', _NNN, line)
>    re.sub('BRAND, _BRAND', line)
>    re.sub('CONTENT_PATH', _CONTENT_PATH, line)
> 
> out_file.close()

There are many templating systems out there. Pick the one that suits you 
best. A very basic one is str.format():

>>> "{foo} {bar}".format(foo=1, bar=2)
'1 2'

Here's what your script might become if you choose that one:

$ cat interactive_template.txt
      $include "_dispatcher_publish_filters.any"
      /1000 {{ /type "allow"  /glob "* /{CONTENT_PATH}/*.html*" }}
      /1001 {{ /type "allow"  /glob "POST /{DAMPATH}/www/*.html *" }}
$ cat interactive_template.py

try:
    format_map = str.format_map
except AttributeError:  # python 2 compatibility
    import string

    def format_map(text, lookup):
        return string.Formatter().vformat(text, (), lookup)
    input = raw_input


class Lookup(dict):
    def __missing__(self, key):
        self[key] = value = input("{}: ".format(key))
        return value


def main():
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument("template_file")
    parser.add_argument("generated_file", nargs="?")
    args = parser.parse_args()

    with open(args.template_file) as instream:
        template_text = instream.read()

    lookup = Lookup()
    generated_text = format_map(template_text, lookup)
    generated_file = args.generated_file
    if generated_file is None:
        generated_file = format_map("{nnn}{brand}_farm.any", lookup)
    print("Writing {}".format(generated_file))
    with open(generated_file, "w") as outstream:
        outstream.write(generated_text)


if __name__ == "__main__":
    main()
$ python interactive_template.py interactive_template.txt
CONTENT_PATH: foo
DAMPATH: bar
nnn: baz
brand: ham
Writing bazham_farm.any
$ cat bazham_farm.any 
      $include "_dispatcher_publish_filters.any"
      /1000 { /type "allow"  /glob "* /foo/*.html*" }
      /1001 { /type "allow"  /glob "POST /bar/www/*.html *" }

[toc] | [prev] | [next] | [standalone]

#99698

When I run this script on OS X El Capitan, I see,
  
  # permission sensitive cache
  $include "_dispatcher_shared_auth-checker: 

Was I supposed to incorporate it into the script I posted?

[toc] | [prev] | [next] | [standalone]

#99703

Mr Zaug wrote:

> When I run this script on OS X El Capitan, I see,
>   
>   # permission sensitive cache
>   $include "_dispatcher_shared_auth-checker:
> 
> Was I supposed to incorporate it into the script I posted?

Are you referring to my post? I'm sorry, I can't make sense of your 
question.

[toc] | [prev] | [next] | [standalone]

#99709

On Sunday, November 29, 2015 at 5:50:51 PM UTC-5, Peter Otten wrote:
> Mr Zaug wrote:
> 
> > When I run this script on OS X El Capitan, I see,
> >   
> >   # permission sensitive cache
> >   $include "_dispatcher_shared_auth-checker:
> > 
> > Was I supposed to incorporate it into the script I posted?
> 
> Are you referring to my post? I'm sorry, I can't make sense of your 
> question.

Yes. The snippet you posted went way over my head. When I ran it, it printed 

# permission sensitive cache 
  $include "_dispatcher_shared_auth-checker: 

[toc] | [prev] | [next] | [standalone]

#99734

Mr Zaug wrote:

> On Sunday, November 29, 2015 at 5:50:51 PM UTC-5, Peter Otten wrote:
>> Mr Zaug wrote:
>> 
>> > When I run this script on OS X El Capitan, I see,
>> >   
>> >   # permission sensitive cache
>> >   $include "_dispatcher_shared_auth-checker:
>> > 
>> > Was I supposed to incorporate it into the script I posted?
>> 
>> Are you referring to my post? I'm sorry, I can't make sense of your
>> question.
> 
> Yes. The snippet you posted went way over my head. When I ran it, it
> printed
> 
> # permission sensitive cache
>   $include "_dispatcher_shared_auth-checker:

It's hard to tell from that problem description what might have gone wrong. 
However:

In your template file you have to enclose all words you want to replace in 
braces ("you have foo options" becomes "you have {foo} options"), to replace 
all literal { with {{ and all literal } with }}.

Did you do that?

[toc] | [prev] | [next] | [standalone]

#99767

On Monday, November 30, 2015 at 4:14:48 AM UTC-5, Peter Otten wrote:
> Mr Zaug wrote:
> 
> > On Sunday, November 29, 2015 at 5:50:51 PM UTC-5, Peter Otten wrote:
> >> Mr Zaug wrote:
> >> 
> >> > When I run this script on OS X El Capitan, I see,
> >> >   
> >> >   # permission sensitive cache
> >> >   $include "_dispatcher_shared_auth-checker:
> >> > 
> >> > Was I supposed to incorporate it into the script I posted?
> >> 
> >> Are you referring to my post? I'm sorry, I can't make sense of your
> >> question.
> > 
> > Yes. The snippet you posted went way over my head. When I ran it, it
> > printed
> > 
> > # permission sensitive cache
> >   $include "_dispatcher_shared_auth-checker:
> 
> It's hard to tell from that problem description what might have gone wrong. 
> However:
> 
> In your template file you have to enclose all words you want to replace in 
> braces ("you have foo options" becomes "you have {foo} options"), to replace 
> all literal { with {{ and all literal } with }}.
> 
> Did you do that?

Yup, I sure did. So here is my current script. The only remaining task now is figuring out how to write the contents to a new file.

#!/usr/bin/env python
import os
import sys

script, template_file = sys.argv
print "Opening the template file..."

print "What is the serial number of the site?",
nnn = raw_input()

print "What is the brand, or product name?",
brand = raw_input()

print "What is the (fqdn) ServerName?",
server_name = raw_input()

print "What is the content path?",
content_path = raw_input()

print "What is the DAM path?",
dampath = raw_input()

print "Which environment is this for?",
env = raw_input()

print "What is the cache document root?",
cache_docroot = raw_input()

with open (template_file, "r") as a_string:
    data=a_string.read().replace('{SERVER_NAME}', server_name).replace('{BRAND}', brand).replace('{CONTENT_PATH}', content_path).replace('{DAMPATH}', dampath).replace('{ENV}', env).replace('{CACHE_DOCROOT}', cache_docroot)

[toc] | [prev] | [next] | [standalone]

#99787

Mr Zaug wrote:

> On Monday, November 30, 2015 at 4:14:48 AM UTC-5, Peter Otten wrote:
>> Mr Zaug wrote:
>> 
>> > On Sunday, November 29, 2015 at 5:50:51 PM UTC-5, Peter Otten wrote:
>> >> Mr Zaug wrote:
>> >> 
>> >> > When I run this script on OS X El Capitan, I see,
>> >> >   
>> >> >   # permission sensitive cache
>> >> >   $include "_dispatcher_shared_auth-checker:
>> >> > 
>> >> > Was I supposed to incorporate it into the script I posted?
>> >> 
>> >> Are you referring to my post? I'm sorry, I can't make sense of your
>> >> question.
>> > 
>> > Yes. The snippet you posted went way over my head. When I ran it, it
>> > printed
>> > 
>> > # permission sensitive cache
>> >   $include "_dispatcher_shared_auth-checker:
>> 
>> It's hard to tell from that problem description what might have gone
>> wrong. However:
>> 
>> In your template file you have to enclose all words you want to replace
>> in braces ("you have foo options" becomes "you have {foo} options"), to
>> replace all literal { with {{ and all literal } with }}.
>> 
>> Did you do that?
> 
> Yup, I sure did. So here is my current script. The only remaining task now
> is figuring out how to write the contents to a new file.
> 
> #!/usr/bin/env python
> import os
> import sys
> 
> script, template_file = sys.argv
> print "Opening the template file..."
> 
> print "What is the serial number of the site?",
> nnn = raw_input()

[...]

> with open (template_file, "r") as a_string:
>     data=a_string.read().replace('{SERVER_NAME}',
>     server_name).replace('{BRAND}', brand).replace('{CONTENT_PATH}',
>     content_path).replace('{DAMPATH}', dampath).replace('{ENV}',
>     env).replace('{CACHE_DOCROOT}', cache_docroot)

If all other braces are properly escaped you can use format instead of 
replace:

     data = a_string.read().format(
         SERVER_NAME=server_name,
         BRAND=brand,
         CONTENT_PATH=content_path
         DAMPATH=dampath,
         ENV=env,
         CACHE_DOCROOT=cache_doc_root)

[toc] | [prev] | [next] | [standalone]

#99798

Oh, that's much easier to read. Thanks!

[toc] | [prev] | [next] | [standalone]

#99800

Actually, I don't understand what you mean by "all other braces." What braces are you talking about? The placeholders in the template file (the file being read in) have braces around them but they are not escaped. 

Also, do I really need curly braces to tell Python what my placeholders are?

[toc] | [prev] | [next] | [standalone]

#99801

Mr Zaug wrote:

> Actually, I don't understand what you mean by "all other braces." What
> braces are you talking about? The placeholders in the template file (the
> file being read in) have braces around them but they are not escaped.

The example you provide was

      $include "_dispatcher_publish_filters.any"
      /1000 { /type "allow"  /glob "* /CONTENT_PATH/*.html*" }
      /1001 { /type "allow"  /glob "POST /DAMPATH/www/*.html *" }

If you want to use string formatting to replace CONTENT_PATH and DAMPATH in 
the above excerpt you have to change the template to

      $include "_dispatcher_publish_filters.any"
      /1000 {{ /type "allow"  /glob "* /{CONTENT_PATH}/*.html*" }}
      /1001 {{ /type "allow"  /glob "POST /{DAMPATH}/www/*.html *" }}


> Also, do I really need curly braces to tell Python what my placeholders
> are?

Let's take a slightly modified example to answer that one:

      $include "_dispatcher_publish_filters.any"
      /1000 { /type "allow"  /glob "* /CONTENT_PATH/*.html*" }
      /1001 { /type "allow"  /glob "POST /DAMPATH/www/*.html *" }
      /1002 { /type "allow"  /glob "* /ALTERNATIVE_CONTENT_PATH/*.html*" }


If you try to fill in the CONTENT_PATH with 

    data = data.replace("CONTENT_PATH", "foo")

the result will be

      $include "_dispatcher_publish_filters.any"
      /1000 { /type "allow"  /glob "* /foo/*.html*" }
      /1001 { /type "allow"  /glob "POST /DAMPATH/www/*.html *" }
      /1002 { /type "allow"  /glob "* /ALTERNATIVE_foo/*.html*" }

Look at the butchered line starting with /1002, a problem that MRAB already 
pointed out before. The braces are not necessary if the string you are 
replacing doesn't occur elsewhere in the template, but they make the 
replacement process both flexible and unambiguous.

[toc] | [prev] | [next] | [standalone]

#99802

That makes sense. 

So I still can't see how to write the string object to a file whist naming the file with whatever values I provided for the NNN and BRAND variables.

Printing the contents of the string object is working with all the expected substitutions. Do I need to convert the string object into a file before I write it? Or do I need to open a new file and somehow stuff the string object into it?

[toc] | [prev] | [next] | [standalone]

#99803

Mr Zaug wrote:

> That makes sense.
> 
> So I still can't see how to write the string object to a file whist naming
> the file with whatever values I provided for the NNN and BRAND variables.
> 
> Printing the contents of the string object is working with all the
> expected substitutions. Do I need to convert the string object into a file
> before I write it? Or do I need to open a new file and somehow stuff the
> string object into it?

Yes, open a new file and write the string using the write() method. You can 
use string formatting to build the filename, too:

text = ... # the string you want to write 

nnn = raw_input("What is the serial number of the site? ")
brand = raw_input("What is the brand, or product name? ")

filename = "{NNN}{BRAND}_farm.any".format(BRAND=brand, NNN=nnn)
with open(filename, "w") as outstream:
    outstream.write(text)

[toc] | [prev] | [next] | [standalone]

#99808

Ye, this does work. Many thanks!

filename = "{NNN}_{BRAND}_farm.any".format(BRAND=brand, NNN=nnn)
with open(filename, "w") as outstream:
    outstream.write(data)

[toc] | [prev] | [next] | [standalone]

#99713

On Sunday, November 29, 2015 at 5:50:51 PM UTC-5, Peter Otten wrote:
> Mr Zaug wrote:
> 
> > When I run this script on OS X El Capitan, I see,
> >   
> >   # permission sensitive cache
> >   $include "_dispatcher_shared_auth-checker:
> > 
> > Was I supposed to incorporate it into the script I posted?
> 
> Are you referring to my post? I'm sorry, I can't make sense of your 
> question.

I seem to be heading in this direction.

#!/usr/bin/env python
import re
from os.path import exists

script, template_file = argv
print "Opening the template file..."

with open (template_file, "r") as a_string:
    data=a_string.read().replace('BRAND', 'Fluxotine')
print(data)

So now the challenge is to use the read().replace magic for multiple values.

[toc] | [prev] | [next] | [standalone]

#99715

A program I am writing at present does exactly this and I simply do
multiple calls to string.replace (see below)

On 30/11/15 10:31, Mr Zaug wrote:
> I seem to be heading in this direction.
>
> #!/usr/bin/env python
> import re
> from os.path import exists
>
> script, template_file = argv
> print "Opening the template file..."
>
> with open (template_file, "r") as a_string:
>     data=a_string.read().replace('BRAND', 'Fluxotine')

        data=data.replace('STRING_2', 'New String 2')
        data=data.replace('STRING_3', 'New String 3')

> print(data)
>
> So now the challenge is to use the read().replace magic for multiple values.

It's crude, but it works well for me!

-- 
Rob Hills
Waikiki, Western Australia

[toc] | [prev] | [next] | [standalone]

#99720

On 2015-11-30 02:40, Rob Hills wrote:
> A program I am writing at present does exactly this and I simply do
> multiple calls to string.replace (see below)
>
> On 30/11/15 10:31, Mr Zaug wrote:
>> I seem to be heading in this direction.
>>
>> #!/usr/bin/env python
>> import re
>> from os.path import exists
>>
>> script, template_file = argv
>> print "Opening the template file..."
>>
>> with open (template_file, "r") as a_string:
>>     data=a_string.read().replace('BRAND', 'Fluxotine')
>
>          data=data.replace('STRING_2', 'New String 2')
>          data=data.replace('STRING_3', 'New String 3')
>
>> print(data)
>>
>> So now the challenge is to use the read().replace magic for multiple values.
>
> It's crude, but it works well for me!
>
You do need to watch out for matches that are part of something else.

For example, if you had this:

     template = "REPLACE_THIS but DONT_REPLACE_THIS"

and you did:

     result = template.replace("REPLACE_THIS", "Python")

you would get:

     "Python but DONT_Python"

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web