Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #26397

Re: Dynamic upgrading/Hot-swapping a service

Show all headers | View raw

On 4/21/2017 12:45 AM, upsidedown@downunder.com wrote:
> On Thu, 20 Apr 2017 16:05:00 -0700, Don Y
> <blockedofcourse@foo.invalid> wrote:
>
>> My application runs 24/7/365.  There's no "reset" switch.  So,
>> components (hardware AND software) that are upgraded are done
>> so while the system is live.
>>
>> [I suspect there are six nines services on-line that behave
>> similarly.  But, if inspected "in the small", I wonder how
>> clean the switchovers are?  And, how durable the connections?]
>
> Before (re)inventing the wheel, take a look how VAXcluster (now
> VMScluster) has done it since the 1980's.

In cluster environments, nodes tend to be indivisible entities.
When you update the software on a node, you update the node,
itself.  You "kick off" the processes that are running on the
node just prior to the upgrade (even if that means migrating them
to another node -- with an OLD copy of the service they are using
at the time) and summarily replace the node (and the services it
provides).

[If you've migrated the existing connections to services that WERE
running on that node to some other node, then you still have those
clients running on that other node -- potentially indefinitely with
the OLD server code!]

Imagine, instead, that some processes are using the "file service"
(whatever THAT is!) on *a* node.  You want to upgrade the file service
code (without affecting any of the other services that are running
on the node) WHILE the file service remains in use on that node.

I.e., install the new service and start it running.  Change the
service registry to reference the new server instance for *new*
service requests (i.e., any files that are accessed AFTER this
point will be handled by the NEW file service).  Allow the old
file service instance to remain active to finish servicing any
existing connections.

*Eventually*, the preexisting connections will be completely serviced
(those files closed, etc.).  Because all NEW requests are handled by
the NEW service, the old service will eventually find itself with no
work to do -- no active connections (clients).  At that point, it
can terminate itself with no deleterious impact on the system.

The problem comes with clients that "linger" on the old service longer
than you'd like.  E.g., imagine a process that opens a dribble file and
leaves it open FOREVER.  That would stake a continuous claim on the
old service preventing it from ever being "replaced".

Or, you might be in a *hurry* to replace a service -- before the
clients currently using it are naturally *done* with it.

So, you need a way of migrating the *active* connections to another
server ALONG WITH THE INTERNAL STATE associated with each of those
connections.

For a file service, that state might include an inode number, access
mode (R/W), current file offset (for read or write), any buffered data
(to be written or already read-ahead), any media I/O actions "in progress",
etc.

But, the *new* service may associate different state with each connection
as dictated by *its* implementation.  So, simply "copying" the state
from the old service to the new service won't suffice; there needs to
be some "state translation" that takes place to ensure the client's
connection remains semantically intact across the transition between
servers.

Or, a modification of the server contract that allows any server to
simply state, "I quit" and let the clients figure out how to recover
or restore their use of that service (boo, hiss!).

I'll be meeting up with some local colleagues, tonight, for 12oz curls.
I'll see if any of the guys who work in "enterprises" can shed some light
on the approach they take to this sort of thing.  Though I suspect their
users are more "transient" than persistent.  So, will leave a service in
short order as a natural consequence of their operation (in which case,
just registering the new service and waiting would suffice).

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-20 16:05 -0700
  Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-20 16:11 -0700
  Re: Dynamic upgrading/Hot-swapping a service upsidedown@downunder.com - 2017-04-21 10:45 +0300
    Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-21 10:43 -0700
      Re: Dynamic upgrading/Hot-swapping a service upsidedown@downunder.com - 2017-04-21 23:06 +0300
        Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-21 17:51 -0700
          Re: Dynamic upgrading/Hot-swapping a service upsidedown@downunder.com - 2017-04-22 14:09 +0300
            Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-22 08:55 -0700
              Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-23 13:01 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-23 10:52 -0700
                Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-24 10:48 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-24 12:33 -0700
                Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-24 22:02 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-24 21:37 -0700
                Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-25 12:19 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-25 13:54 -0700

csiph-web