Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #26399

Re: Dynamic upgrading/Hot-swapping a service

Show all headers | View raw

On 4/21/2017 1:06 PM, upsidedown@downunder.com wrote:
> On Fri, 21 Apr 2017 10:43:28 -0700, Don Y
> <blockedofcourse@foo.invalid> wrote:
>
>> On 4/21/2017 12:45 AM, upsidedown@downunder.com wrote:
>>> On Thu, 20 Apr 2017 16:05:00 -0700, Don Y
>>> <blockedofcourse@foo.invalid> wrote:
>>>
>>>> My application runs 24/7/365.  There's no "reset" switch.  So,
>>>> components (hardware AND software) that are upgraded are done
>>>> so while the system is live.
>>>>
>>>> [I suspect there are six nines services on-line that behave
>>>> similarly.  But, if inspected "in the small", I wonder how
>>>> clean the switchovers are?  And, how durable the connections?]
>>>
>>> Before (re)inventing the wheel, take a look how VAXcluster (now
>>> VMScluster) has done it since the 1980's.
>>
>> In cluster environments, nodes tend to be indivisible entities.
>> When you update the software on a node, you update the node,
>> itself.  You "kick off" the processes that are running on the
>> node just prior to the upgrade (even if that means migrating them
>> to another node -- with an OLD copy of the service they are using
>> at the time) and summarily replace the node (and the services it
>> provides).
>>
>> [If you've migrated the existing connections to services that WERE
>> running on that node to some other node, then you still have those
>> clients running on that other node -- potentially indefinitely with
>> the OLD server code!]
>>
>> Imagine, instead, that some processes are using the "file service"
>> (whatever THAT is!) on *a* node.  You want to upgrade the file service
>> code (without affecting any of the other services that are running
>> on the node) WHILE the file service remains in use on that node.
>>
>> I.e., install the new service and start it running.  Change the
>> service registry to reference the new server instance for *new*
>> service requests (i.e., any files that are accessed AFTER this
>> point will be handled by the NEW file service).  Allow the old
>> file service instance to remain active to finish servicing any
>> existing connections.
>>
>> *Eventually*, the preexisting connections will be completely serviced
>> (those files closed, etc.).  Because all NEW requests are handled by
>> the NEW service, the old service will eventually find itself with no
>> work to do -- no active connections (clients).  At that point, it
>> can terminate itself with no deleterious impact on the system.
>>
>> The problem comes with clients that "linger" on the old service longer
>> than you'd like.  E.g., imagine a process that opens a dribble file and
>> leaves it open FOREVER.  That would stake a continuous claim on the
>> old service preventing it from ever being "replaced".
>>
>> Or, you might be in a *hurry* to replace a service -- before the
>> clients currently using it are naturally *done* with it.
>>
>> So, you need a way of migrating the *active* connections to another
>> server ALONG WITH THE INTERNAL STATE associated with each of those
>> connections.
>>
>> For a file service, that state might include an inode number, access
>> mode (R/W), current file offset (for read or write), any buffered data
>> (to be written or already read-ahead), any media I/O actions "in progress",
>> etc.
>>
>> But, the *new* service may associate different state with each connection
>> as dictated by *its* implementation.  So, simply "copying" the state
>>from the old service to the new service won't suffice; there needs to
>> be some "state translation" that takes place to ensure the client's
>> connection remains semantically intact across the transition between
>> servers.
>>
>> Or, a modification of the server contract that allows any server to
>> simply state, "I quit" and let the clients figure out how to recover
>> or restore their use of that service (boo, hiss!).
>>
>> I'll be meeting up with some local colleagues, tonight, for 12oz curls.
>> I'll see if any of the guys who work in "enterprises" can shed some light
>> on the approach they take to this sort of thing.  Though I suspect their
>> users are more "transient" than persistent.  So, will leave a service in
>> short order as a natural consequence of their operation (in which case,
>> just registering the new service and waiting would suffice).
>
> It is more than a quarter of a century since I have been running a
> large VAXcluster with  a dozen cabinet size CPUs, but I try to
> remember some of the details.
>
> If you have multiple CPUs with shared (and mirrored) disks, switching
> from an active process from one CPU to an other is quite easy. As long
> as the OS supports process checkpointing or swapping out a complete
> process to disk, things are easy. Instead of swapping in a process
> from disk back into the original CPU, just swap it in to an another
> CPU :-)

That's not the same thing.   That's *migrating* a process to a different
CPU.  You're moving the entire state of the process to resume execution
on another CPU.  All the "variables" AND all the instructions that
interpret those variables!

I want to "alter the executable" while it is running -- change the
instructions and (somehow) tweak the variables so their current
values "make sense" when interpreted by a different set of instructions!

I'm typing a "followup" to your message using Thunderbird.  I
(the human) can be regarded as a client of Thunderbird.  I am engaged
in an interaction with it -- my CONNECTION to it persists continuously
as I am typing this message.

WHILE I AM TYPING, I want something to be able to sneak in and
REPLACE the copy of Thunderbird that is executing in my computer's
memory -- not just the copy that resides on the disk (which Windows
won't examine until the next time I *load* Thunderbird) -- and to
do so such that this message ends up intact as it is eventually
posted to the NNTP server.

I.e., to do this, you'd need to capture a copy of what I've typed
up to the instant the upgrade is switched in *under* me.  It
would have to know how the windows that the old Thunderbird
instance was using were maintained by the OS, and the source
of keystrokes and other user interface events.

It would be messy and tedious to get it "right" -- but not impossible.

A far easier goal would be to swap the executable bound to "Thunderbird.exe"
so that the next time I invoked Thunderbird, I'd get the NEW executable;
let my current interaction run to completion with the *old* executable!

But, there's no guarantee that I will terminate this Thunderbird session
anytime soon.  Or *ever*!

The OS can forcibly move the user interface connections to another
process running on the same -- or different -- node.  But, that doesn't
mean the client's (i.e., user's) experience will be "continuous"
or coherent.

> In a VAX cluster, application programs refer to resources, such as
> disks by logical names. It is the responsibility of the system manager
> to maintain the day to day mapping between the logical disk names and
> the physical disk names.  Some logical name lists (one logical name
> translates to multiple physical resources and the OS selected the
> first physical device available from the logical name list).
>
> In those days, dumb terminals were used. With the Ethernet/serial
> converters (DecServer xxx) running the LAT protocol, it was quite easy
> to automatically connect a dumb terminal user from one CPU to an
> other.

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-20 16:05 -0700
  Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-20 16:11 -0700
  Re: Dynamic upgrading/Hot-swapping a service upsidedown@downunder.com - 2017-04-21 10:45 +0300
    Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-21 10:43 -0700
      Re: Dynamic upgrading/Hot-swapping a service upsidedown@downunder.com - 2017-04-21 23:06 +0300
        Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-21 17:51 -0700
          Re: Dynamic upgrading/Hot-swapping a service upsidedown@downunder.com - 2017-04-22 14:09 +0300
            Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-22 08:55 -0700
              Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-23 13:01 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-23 10:52 -0700
                Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-24 10:48 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-24 12:33 -0700
                Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-24 22:02 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-24 21:37 -0700
                Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-25 12:19 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-25 13:54 -0700

csiph-web