Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #88034 > unrolled thread

Re: What is elegant way to do configuration on server app

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: What is elegant way to do configuration on server app Ben Finney <ben+python@benfinney.id.au> - 2015-03-26 19:59 +1100
    Re: What is elegant way to do configuration on server app Grant Edwards <invalid@invalid.invalid> - 2015-03-26 15:49 +0000
      Re: What is elegant way to do configuration on server app Jerry OELoo <oyljerry@gmail.com> - 2015-03-27 16:28 +0800
        Re: What is elegant way to do configuration on server app Grant Edwards <invalid@invalid.invalid> - 2015-03-27 16:28 +0000
          Re: What is elegant way to do configuration on server app Chris Angelico <rosuav@gmail.com> - 2015-03-28 03:36 +1100
            Re: What is elegant way to do configuration on server app Grant Edwards <invalid@invalid.invalid> - 2015-03-27 16:44 +0000
      Re: What is elegant way to do configuration on server app Marko Rauhamaa <marko@pacujo.net> - 2015-03-27 10:47 +0200
        Re: What is elegant way to do configuration on server app Chris Angelico <rosuav@gmail.com> - 2015-03-28 00:12 +1100
          Re: What is elegant way to do configuration on server app Marko Rauhamaa <marko@pacujo.net> - 2015-03-27 15:23 +0200
            Re: What is elegant way to do configuration on server app Chris Angelico <rosuav@gmail.com> - 2015-03-28 00:34 +1100
        Re: What is elegant way to do configuration on server app Grant Edwards <invalid@invalid.invalid> - 2015-03-27 16:30 +0000
          Re: What is elegant way to do configuration on server app Marko Rauhamaa <marko@pacujo.net> - 2015-03-27 18:44 +0200

#88034 — Re: What is elegant way to do configuration on server app

Jerry OELoo <oyljerry@gmail.com> writes:

> Currently, I can just think out that I put status into a configure
> file, and service schedule read this file and get status value,

That sounds like a fine start. Some advice:

* You may be tempted to make the configuration file executable (e.g.
  Python code). Resist that temptation; keep it *much* simpler, a
  non-executable data format.

  Python's standard library has the ‘configparser’ module
  <URL:https://docs.python.org/3/library/configparser.html> to parse and
  provide the values from a very common configuration file format.
  Use that unless you have a good reason not to.

* Your program can “poll” the configuration file to see whether it has
  changed. At startup, read the config file's modification timestamp
  <URL:https://docs.python.org/3/library/os.html#os.stat_result.st_mtime>.

  Make a part of your event loop (assuming your server runs an event
  loop) that wakes up every N seconds (e.g. every 60 seconds) and
  checkes the file's modification timestamp again; if it's newer, record
  that value for future comparisons, then re-read the file for its
  values.

Hope that helps.

-- 
 \             “For your convenience we recommend courteous, efficient |
  `\                            self-service.” —supermarket, Hong Kong |
_o__)                                                                  |
Ben Finney

[toc] | [next] | [standalone]

#88073

On 2015-03-26, Ben Finney <ben+python@benfinney.id.au> wrote:
> Jerry OELoo <oyljerry@gmail.com> writes:
>
>> Currently, I can just think out that I put status into a configure
>> file, and service schedule read this file and get status value,
>
> That sounds like a fine start. Some advice:
>
> * You may be tempted to make the configuration file executable (e.g.
>   Python code). Resist that temptation; keep it *much* simpler, a
>   non-executable data format.
>
>   Python's standard library has the ‘configparser’ module
>  <URL:https://docs.python.org/3/library/configparser.html> to parse and
>   provide the values from a very common configuration file format.
>   Use that unless you have a good reason not to.

I second the recommendation for configparser for stuff of simple to
moderate complexity.  If it gets too complex for configparser, you may
want to consider JSON, or for even more complex stuff just use plain
Python code in your config file (this can be very powerful and
expressive, but can also be diffucult to implement safely).

> * Your program can “poll” the configuration file to see whether it has
>   changed. At startup, read the config file's modification timestamp
>  <URL:https://docs.python.org/3/library/os.html#os.stat_result.st_mtime>.
>
>   Make a part of your event loop (assuming your server runs an event
>   loop) that wakes up every N seconds (e.g. every 60 seconds) and
>   checkes the file's modification timestamp again; if it's newer, record
>   that value for future comparisons, then re-read the file for its
>   values.

That sounds rather Windowsesque.  The more-or-less standard way to do
handle the situation on Unix is to reread the config file when you get
a SIGHUP.

-- 
Grant Edwards               grant.b.edwards        Yow! ONE LIFE TO LIVE for
                                  at               ALL MY CHILDREN in ANOTHER
                              gmail.com            WORLD all THE DAYS OF
                                                   OUR LIVES.

[toc] | [prev] | [next] | [standalone]

#88140

Hi Grant:
Why use SIGHUP, Does it has something to do with configure file
modification? I don't get it. Thank you.

On Thu, Mar 26, 2015 at 11:49 PM, Grant Edwards <invalid@invalid.invalid> wrote:
> On 2015-03-26, Ben Finney <ben+python@benfinney.id.au> wrote:
>> Jerry OELoo <oyljerry@gmail.com> writes:
>>
>>> Currently, I can just think out that I put status into a configure
>>> file, and service schedule read this file and get status value,
>>
>> That sounds like a fine start. Some advice:
>>
>> * You may be tempted to make the configuration file executable (e.g.
>>   Python code). Resist that temptation; keep it *much* simpler, a
>>   non-executable data format.
>>
>>   Python's standard library has the ‘configparser’ module
>>  <URL:https://docs.python.org/3/library/configparser.html> to parse and
>>   provide the values from a very common configuration file format.
>>   Use that unless you have a good reason not to.
>
> I second the recommendation for configparser for stuff of simple to
> moderate complexity.  If it gets too complex for configparser, you may
> want to consider JSON, or for even more complex stuff just use plain
> Python code in your config file (this can be very powerful and
> expressive, but can also be diffucult to implement safely).
>
>> * Your program can “poll” the configuration file to see whether it has
>>   changed. At startup, read the config file's modification timestamp
>>  <URL:https://docs.python.org/3/library/os.html#os.stat_result.st_mtime>.
>>
>>   Make a part of your event loop (assuming your server runs an event
>>   loop) that wakes up every N seconds (e.g. every 60 seconds) and
>>   checkes the file's modification timestamp again; if it's newer, record
>>   that value for future comparisons, then re-read the file for its
>>   values.
>
> That sounds rather Windowsesque.  The more-or-less standard way to do
> handle the situation on Unix is to reread the config file when you get
> a SIGHUP.
>
> --
> Grant Edwards               grant.b.edwards        Yow! ONE LIFE TO LIVE for
>                                   at               ALL MY CHILDREN in ANOTHER
>                               gmail.com            WORLD all THE DAYS OF
>                                                    OUR LIVES.
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Rejoice,I Desire!

[toc] | [prev] | [next] | [standalone]

#88171

On 2015-03-27, Jerry OELoo <oyljerry@gmail.com> wrote:

>>>   Make a part of your event loop (assuming your server runs an event
>>>   loop) that wakes up every N seconds (e.g. every 60 seconds) and
>>>   checkes the file's modification timestamp again; if it's newer, record
>>>   that value for future comparisons, then re-read the file for its
>>>   values.
>>
>> That sounds rather Windowsesque.  The more-or-less standard way to do
>> handle the situation on Unix is to reread the config file when you get
>> a SIGHUP.
>
> Why use SIGHUP,

It's just tradition.  Some other seldom-used signal could have been
chosen, but way back when somebody chose SIGHUP, and other people
followed suit.

> Does it has something to do with configure file modification?

Not intrinsically.  It's just a convention in the Unix world that
sending SIGHUP to a daemon will cause it to re-read its configuration
files.

I presume that automagically reading them any time they changed was
both too much hassle and possibly dangerous: if a file is being
edited, it might get saved in intermediate (broken) states during the
editing session.

-- 
Grant Edwards               grant.b.edwards        Yow! An air of FRENCH FRIES
                                  at               permeates my nostrils!!
                              gmail.com            

[toc] | [prev] | [next] | [standalone]

#88173

On Sat, Mar 28, 2015 at 3:28 AM, Grant Edwards <invalid@invalid.invalid> wrote:
> I presume that automagically reading them any time they changed was
> both too much hassle and possibly dangerous: if a file is being
> edited, it might get saved in intermediate (broken) states during the
> editing session.

Even more so if you have multiple files (look, for instance, at
PostgreSQL configs, where you might make simultaneous and connected
changes to several related files); if you change one and it's
activated before you change another, it might even have security
implications - although more likely, it'd cause an outage (when legit
connections get rejected because you haven't yet added the permission
lines).

The same problem occurs with PHP-based web sites, but there you don't
get the option of holding over for a SIGHUP, so you're just stuck with
uploading a new version of your site file-by-file, or *maybe*
attempting an atomic rename of a directory. If you're lucky.

ChrisA

[toc] | [prev] | [next] | [standalone]

#88175

On 2015-03-27, Chris Angelico <rosuav@gmail.com> wrote:

> The same problem occurs with PHP-based web sites, but there you don't
> get the option of holding over for a SIGHUP, so you're just stuck with
> uploading a new version of your site file-by-file, or *maybe*
> attempting an atomic rename of a directory. If you're lucky.

I don't think that anybody using PHP can be considered "lucky". ;)

-- 
Grant Edwards               grant.b.edwards        Yow! ANN JILLIAN'S HAIR
                                  at               makes LONI ANDERSON'S
                              gmail.com            HAIR look like RICARDO
                                                   MONTALBAN'S HAIR!

[toc] | [prev] | [next] | [standalone]

#88142

Grant Edwards <invalid@invalid.invalid>:

> That sounds rather Windowsesque. The more-or-less standard way to do
> handle the situation on Unix is to reread the config file when you get
> a SIGHUP.

That, indeed, is the classic Unix way. However, Linux has now moved to
systemd:

   ExecReload=

       Commands to execute to trigger a configuration reload in the
       service. This argument takes multiple command lines, following
       the same scheme as described for ExecStart= above. Use of this
       setting is optional. Specifier and environment variable
       substitution is supported here following the same scheme as for
       ExecStart=.

       One additional, special environment variable is set: if known,
       $MAINPID is set to the main process of the daemon, and may be
       used for command lines like the following:

       /bin/kill -HUP $MAINPID

       Note however that reloading a daemon by sending a signal (as with
       the example line above) is usually not a good choice, because
       this is an asynchronous operation and hence not suitable to order
       reloads of multiple services against each other. It is strongly
       recommended to set ExecReload= to a command that not only
       triggers a configuration reload of the daemon, but also
       synchronously waits for it to complete.

   <URL: http://www.freedesktop.org/software/systemd/ma
   n/systemd.service.html>


Marko

[toc] | [prev] | [next] | [standalone]

#88147

On Fri, Mar 27, 2015 at 7:47 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Grant Edwards <invalid@invalid.invalid>:
>
>> That sounds rather Windowsesque. The more-or-less standard way to do
>> handle the situation on Unix is to reread the config file when you get
>> a SIGHUP.
>
> That, indeed, is the classic Unix way. However, Linux has now moved to
> systemd:
>
>        Note however that reloading a daemon by sending a signal (as with
>        the example line above) is usually not a good choice, because
>        this is an asynchronous operation and hence not suitable to order
>        reloads of multiple services against each other. It is strongly
>        recommended to set ExecReload= to a command that not only
>        triggers a configuration reload of the daemon, but also
>        synchronously waits for it to complete.

The number of cases where this matters is fairly low. Doing the reload
asynchronously is generally sufficient. And even if you have something
that waits for the reload to finish, you'll usually fire-and-forget
that command anyway.

For maximum portability, most programs are going to want to continue
to respond to SIGHUP, even if they do a systemd-triggered reload some
other way.

ChrisA

[toc] | [prev] | [next] | [standalone]

#88151

Chris Angelico <rosuav@gmail.com>:

> The number of cases where this matters is fairly low. Doing the reload
> asynchronously is generally sufficient.

Not sure. Not sure at all. Issues like this occupy a great part of my
office hours.

> And even if you have something that waits for the reload to finish,
> you'll usually fire-and-forget that command anyway.

The command

   systemctl reload xyz.service

is supposed to return when the service is happily running with the
updated its configuration.

Any failures should be reported with a nonzero exit code from
"systemctl".

> For maximum portability, most programs are going to want to continue
> to respond to SIGHUP, even if they do a systemd-triggered reload some
> other way.

I disagree. You can't just assume you can send a SIGHUP to a daemon for
the LOLs. You must first discover the trick in the documentation.

The default reaction to SIGHUP is to crash. A daemon (which has detached
from the controlling terminal) is within its rights to do just that.


Marko

[toc] | [prev] | [next] | [standalone]

#88154

On Sat, Mar 28, 2015 at 12:23 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> The number of cases where this matters is fairly low. Doing the reload
>> asynchronously is generally sufficient.
>
> Not sure. Not sure at all. Issues like this occupy a great part of my
> office hours.

Only because your office hours are occupied by problems, not by the
cases that work flawlessly. You don't see the innumerable times that
fire-and-forget works fine :)

>> And even if you have something that waits for the reload to finish,
>> you'll usually fire-and-forget that command anyway.
>
> The command
>
>    systemctl reload xyz.service
>
> is supposed to return when the service is happily running with the
> updated its configuration.
>
> Any failures should be reported with a nonzero exit code from
> "systemctl".

Sure. But how often do you actually need that, compared to the number
of times when you deploy an updated config that was working fine on
your test system, signal a reload, and off you go? You don't need to
wait for it to finish; you'll see errors in the log, if there are any,
and most likely there won't (your test system IS virtually identical
to live, right?). I have scripts that edit config files and pop off
SIGHUPs, usually in response to external events, and they neither need
nor want to wait. It's pretty common.

>> For maximum portability, most programs are going to want to continue
>> to respond to SIGHUP, even if they do a systemd-triggered reload some
>> other way.
>
> I disagree. You can't just assume you can send a SIGHUP to a daemon for
> the LOLs. You must first discover the trick in the documentation.
>
> The default reaction to SIGHUP is to crash. A daemon (which has detached
> from the controlling terminal) is within its rights to do just that.

Sure it can, and you do need to know what you're sending signals to.
But if you're unsure how to reload a daemon, the easiest way to probe
it is to send it a SIGHUP (on your test system, of course), and see if
it's still running afterwards. And once you know that this process
responds thusly to SIGHUP, you can do this many *many* times more,
never needing to worry about it. I would guess that there are
hundreds, maybe thousands, of times when you can safely
fire-and-forget, for every one that you actually need to wait on it -
where you need some other action to be delayed until the reload is
complete.

ChrisA

[toc] | [prev] | [next] | [standalone]

#88172

On 2015-03-27, Marko Rauhamaa <marko@pacujo.net> wrote:
> Grant Edwards <invalid@invalid.invalid>:
>
>> That sounds rather Windowsesque. The more-or-less standard way to do
>> handle the situation on Unix is to reread the config file when you get
>> a SIGHUP.
>
> That, indeed, is the classic Unix way. However, Linux has now moved to
> systemd:

That's a bit of an overstatement.  _Some_ distros have switched to
systemd.  None of my machines use it, and it's very rare in embedded
systems.

-- 
Grant Edwards               grant.b.edwards        Yow! Are we THERE yet?
                                  at               My MIND is a SUBMARINE!!
                              gmail.com            

[toc] | [prev] | [next] | [standalone]

#88174

Grant Edwards <invalid@invalid.invalid>:

> On 2015-03-27, Marko Rauhamaa <marko@pacujo.net> wrote:
>> That, indeed, is the classic Unix way. However, Linux has now moved
>> to systemd:
>
> That's a bit of an overstatement. _Some_ distros have switched to
> systemd. None of my machines use it, and it's very rare in embedded
> systems.

Much less of an overstatement than saying that the Python world has now
moved to Python3.

RHEL/CentOS/Fedora are there. OpenSuSE is there. Debian is there. Ubuntu
is going there.

I'm not an apologist of systemd. Just stating the situation.


Marko

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web