Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #40568 > unrolled thread

Config & ConfigParser

Back to article view | Back to comp.lang.python

Contents

  Config & ConfigParser Chuck <galois271@gmail.com> - 2013-03-05 12:09 -0800
    Re: Config & ConfigParser Tim Chase <python.list@tim.thechases.com> - 2013-03-05 15:17 -0600
      Re: Config & ConfigParser Chuck <galois271@gmail.com> - 2013-03-05 15:58 -0800
        Re: Config & ConfigParser Tim Chase <python.list@tim.thechases.com> - 2013-03-05 18:15 -0600
          Re: Config & ConfigParser Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-03-06 04:04 +0000
    Re: Config & ConfigParser Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-03-06 03:31 +0000
      Re: Config & ConfigParser Chris Angelico <rosuav@gmail.com> - 2013-03-06 15:05 +1100
        Re: Config & ConfigParser Neil Cerutti <neilc@norwich.edu> - 2013-03-07 16:18 +0000
      Re: Config & ConfigParser Chuck <galois271@gmail.com> - 2013-03-05 20:07 -0800
        Re: Config & ConfigParser Chris Angelico <rosuav@gmail.com> - 2013-03-06 15:19 +1100
          Re: Config & ConfigParser Steven D'Aprano <steve+comp.lang.python@pearwood.info> - 2013-03-06 04:54 +0000
            Re: Config & ConfigParser Chris Angelico <rosuav@gmail.com> - 2013-03-06 17:09 +1100

#40568 — Config & ConfigParser

I'm curious about using configuration files.  Can someone tell me how they are used?   I'm writing a podcast catcher and would like to set up some default configurations, e.g. directory, etc    Other than default directory, what are some of the things that are put in a configuration file?   They don't seem to teach you this kind of stuff in school.  :(

Thanks! 

[toc] | [next] | [standalone]

#40571

On 2013-03-05 12:09, Chuck wrote:
> I'm curious about using configuration files.  Can someone tell me
> how they are used?   I'm writing a podcast catcher and would like
> to set up some default configurations, e.g. directory, etc    Other
> than default directory, what are some of the things that are put in
> a configuration file? 

Just looking at my hpodder config file, you can have all sorts of
things in here.  You might have a root folder, but then each feed
could be configured to a particular sub-folder.  You might have a
cache location you download to, and only move them to the final
destination after they've downloaded completely.  You might have
some check more frequently than others. You might have it execute an
external utility (such as id3v2) to transform various ID3 information
based on information in the feed. If you're downloading in multiple
threads, you might have specify the number of threads it can use.  If
you are using "curl" under the covers, you might limit the
transfer-rate so it doesn't suck up your bandwidth.  If you display
anything, you might allow for suppressing the display, formatting
that display, or controlling whether it uses color.  If you track
download errors, you might specify how many failures constitute a
"don't bother retrying this item" or how many days/hours you need to
wait until you actually retry.

You can even have it act as your data-store for holding the URLs to
the RSS feeds, perhaps a readable name (to override what's in the
feed), along with formatting.  So a more complex .ini might look
something like

  [general]
  root=/home/chuck/Music/Podcasts
  cache=/tmp/podchuck
  threads=4
  rate=50k
  color=auto

  [feed "http://feeds.5by5.tv/webahead"]
  display-name=The Web Ahead"
  location=%(root)s/WebAhead/$FILENAME
  transform=id3v2 -a "WebAhead" -t "$EPTITLE" "$FILENAME"

  [feed "http://www.radiolab.org/feeds/podcast"]
  location=/path/to/someplace/else/$FILENAME
  transform=id3v2 -a "Radio Lab" -t "$EPTITLE" "$FILENAME"

-tkc

[toc] | [prev] | [next] | [standalone]

#40584

Thanks Tim!  So much stuff I haven't thought of before.  Out of curiosity, what's the benefit of caching the download,  instead of downloading to the final destination?   So much stuff they never teach you school.    So much theory & not enough practice.	 :(

[toc] | [prev] | [next] | [standalone]

#40586

On 2013-03-05 15:58, Chuck wrote:
> Thanks Tim!  So much stuff I haven't thought of before.  Out of
> curiosity, what's the benefit of caching the download,  instead of
> downloading to the final destination? 

If your connection gets interrupted, the server goes down, etc, you
have a partial download.  If you've put it directly in the download
path, your other programs see this partial download.  However
if your program can resume the download where it left off, once
it's completed successfully, it can atomically move the file to your
download location.  Thus your other programs only ever see
all-or-nothing in the download directory.

-tkc

[toc] | [prev] | [next] | [standalone]

#40599

On Tue, 05 Mar 2013 18:15:20 -0600, Tim Chase wrote:

> On 2013-03-05 15:58, Chuck wrote:
>> Thanks Tim!  So much stuff I haven't thought of before.  Out of
>> curiosity, what's the benefit of caching the download,  instead of
>> downloading to the final destination?
> 
> If your connection gets interrupted, the server goes down, etc, you have
> a partial download.  If you've put it directly in the download path,
> your other programs see this partial download. However if your program
> can resume the download where it left off, once it's completed
> successfully, it can atomically move the file to your download location.
>  Thus your other programs only ever see all-or-nothing in the download
> directory.

That's not really a *cache* though.

Personally, I find programs that do that sort of cleverness annoying 
rather than helpful. More often than not, they are buggy and fail to 
clean up after themselves if the download is interrupted, so the secret 
download directory ends up filled with junk:

cat-video27~
cat-video27-1~
cat-video27-2~
cat-video27-3~

sort of thing.

Another problem with this tactic is that it makes it unnecessarily 
difficult to watch progress of the download, except via the application's 
official user interface. (If it gives you any interface for watching 
download progress, which is may not.) You have to locate the secret 
download directory, work out what file name the app is using for the 
temporary file (many apps obfuscate the file name), then watch that file 
grow until it suddenly disappears, at which point you then have to change 
directories to see if it reappeared where you actually wanted it to be, 
or was just deleted by something else.

A third problem: instead of having to worry about having enough disk 
space in one location, now you have to worry about having enough disk 
space in *two* locations.

I've even see a program download a large file into the temp location, 
*unsuccessfully* try to copy it into the final location, then delete the 
temp version.

Yet another problem: websites sometimes lie about the size of some files. 
So when you download them, the actual file ends (say) one byte short of 
what the webserver claims. There's nothing wrong with the file, it is 
actually complete, or at least recoverable (many formats, like JPEG, are 
remarkably resistant to damage), but your app will think it never 
completes and either never move it to the final destination, or worse, 
keep trying to download it over and over and over again.

Finally, moving from the temp directory to the final location is not 
necessarily an atomic operation. If it crosses file system boundaries, it 
is a two-step process. 

I think that the KISS principle applies here. If the user tells your app 
to download in some location, your app should download in that location.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#40597

On Tue, 05 Mar 2013 12:09:38 -0800, Chuck wrote:

> I'm curious about using configuration files.  Can someone tell me how
> they are used?   I'm writing a podcast catcher and would like to set up
> some default configurations, e.g. directory, etc    Other than default
> directory, what are some of the things that are put in a configuration
> file?   They don't seem to teach you this kind of stuff in school.  :(

I think that you are doing this backwards. You shouldn't start with a 
question:

"I want a configuration file! What should I put in it?"

and then try to invent a need for configuration settings to put in the 
file. You start with the need, and end with the conclusion:

"I need these configuration settings. I'll put them in a config file."


What configuration settings does your podcast catcher software need? What 
makes you think it needs any? Don't over-engineer your application from 
the start. Begin with the simplest thing that works, and go from there.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#40600

On Wed, Mar 6, 2013 at 2:31 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> What configuration settings does your podcast catcher software need? What
> makes you think it needs any? Don't over-engineer your application from
> the start. Begin with the simplest thing that works, and go from there.

Agreed. The way I generally do these things is to simply gather the
config entries into a block of literal assignments at the top of the
program:

host = "ftp"
port = 21
proxy = "192.168.0.3:81"
user = "transfers"
password = "secret"

>From there, it's easy to decide whether to make them into command-line
parameters, a parseable config file, an importable Python script
("from config import *" is an easy and simple way to make that one),
or whatever.

ChrisA

[toc] | [prev] | [next] | [standalone]

#40797

On 2013-03-06, Chris Angelico <rosuav@gmail.com> wrote:
> On Wed, Mar 6, 2013 at 2:31 PM, Steven D'Aprano
><steve+comp.lang.python@pearwood.info> wrote:
>> What configuration settings does your podcast catcher software
>> need? What makes you think it needs any? Don't over-engineer
>> your application from the start. Begin with the simplest thing
>> that works, and go from there.
>
> Agreed. The way I generally do these things is to simply gather the
> config entries into a block of literal assignments at the top of the
> program:
>
> host = "ftp"
> port = 21
> proxy = "192.168.0.3:81"
> user = "transfers"
> password = "secret"

I find it useful to stuff my config info in a class object; It is
prettier than ALLCAPS, and also makes it simple to create and use
new configurations without erasing or commenting out the old one.

class Config:
  host = "ftp"
  port = 21
  proxy = "192.168.0.3:81"
  user = "transfers"
  password = "secret"

How much to engineer stuff is up to you.

-- 
Neil Cerutti

[toc] | [prev] | [next] | [standalone]

#40601

I guess my question was more what is a config.file & why/how do I use one.
Thanks

[toc] | [prev] | [next] | [standalone]

#40602

On Wed, Mar 6, 2013 at 3:07 PM, Chuck <galois271@gmail.com> wrote:
> I guess my question was more what is a config.file & why/how do I use one.
> Thanks

In its simplest form, a config file is one way to change a program's
behaviour without editing the code. They're helpful when you want to
be able to run the same program in different ways, or when you want to
let someone else change what the program does without having to edit
code (and without the changes getting overwritten by a program
update). There are innumerable file formats that can be used.

ChrisA

[toc] | [prev] | [next] | [standalone]

#40603

On Wed, 06 Mar 2013 15:19:53 +1100, Chris Angelico wrote:

> On Wed, Mar 6, 2013 at 3:07 PM, Chuck <galois271@gmail.com> wrote:
>> I guess my question was more what is a config.file & why/how do I use
>> one. Thanks
> 
> In its simplest form, a config file is one way to change a program's
> behaviour without editing the code.

I don't think that's quite right, because your code has to be changed to 
read the data from the configuration file in the first place. It doesn't 
just happen by magic.

Essentially, a configuration file is a file that holds configuration 
data. That data could be anything that makes sense for your program, but 
you still have to process the data in your program.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#40606

On Wed, Mar 6, 2013 at 3:54 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Wed, 06 Mar 2013 15:19:53 +1100, Chris Angelico wrote:
>
>> On Wed, Mar 6, 2013 at 3:07 PM, Chuck <galois271@gmail.com> wrote:
>>> I guess my question was more what is a config.file & why/how do I use
>>> one. Thanks
>>
>> In its simplest form, a config file is one way to change a program's
>> behaviour without editing the code.
>
> I don't think that's quite right, because your code has to be changed to
> read the data from the configuration file in the first place. It doesn't
> just happen by magic.

Sure, but once you've made your code read from the config file, you
can then edit the file only and it changes the program's actions.

Of course, that's an *incredibly* broad description; amongst its
coverage are such diverse elements as Apache reading an HTML file to
serve, CPython reading a .py file, and the ROM BIOS reading a boot
sector and jumping to it... but based on the OP's question I couldn't
really be any more specific.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web