Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #26394

Dynamic upgrading/Hot-swapping a service

Show all headers | View raw

My application runs 24/7/365.  There's no "reset" switch.  So,
components (hardware AND software) that are upgraded are done
so while the system is live.

[I suspect there are six nines services on-line that behave
similarly.  But, if inspected "in the small", I wonder how
clean the switchovers are?  And, how durable the connections?]

I can instantiate a new instance of a server and replace the
bindings to the old server with bindings to the new.  So, any
*new* attempts to use the service are automatically routed away
from the old server and handled, instead, by the new instance.

Any connections/transactions to the old server can continue to
be handled, there.  When each client closes its connection to
the server, the old server will see one less client to service.
This is sort of equivalent to a zombie process (even though the
process is actually still functioning as intended)

And, no *new* clients will ever appear (because the new instance
is handling them).  So, eventually, the server will detect "no
more clients" and exit().  This frees up the resources that the
old service was using.

[This is true even if an "old" client tries to reopen a connection
to the service!]

The problem comes with connections that are more durable/persistent.
There comes a time when you want to more aggressively *kill* the
"zombie", even though it is performing as intended, etc. -- if only
to reclaim those resources!

Doing so WITHOUT INTERRUPTING THE SERVICE is, of course, the desired
way of meeting this goal (crashing the service is almost always a
sign that you got lazy in the implementation).  So, the old service
needs to be told to "migrate clients" to the new instance of the service.

Of course, the mechanics of migrating a specific client for a specific
service are highly dependent on that service, etc.  Any internal state
associated with the client has to be abstracted to a form that the
new server can interpret and map to its new implementation.  This, then,
has to be conveyed to the new service along with the endpoints for the
existing client(s).

The first question:  should this be implemented as a "force clients
to the new service" (i.e., shed ALL clients)?  Or, as "force THIS client
to the new service" -- iterated over the set of existing current clients?

[The net result in each case should be the same]

The former gives finer-grained control over the reallocation of
resources/connections/activities.  But, I can't really see how it
would be used with *less* than the full set of current clients
(so, is it needless detail?).

The second question goes to who should affect this decision.
Imposing it "from on high" ignores the particular requirements
of the service in question -- it assumes all services are
equally easy/difficult from which to migrate clients.  OTOH,
the only place that has global knowledge of what's happening
(resource-wise) in the system is "on high".

My current thinking is between two alternatives:
- signal the new service that it should *acquire* the old
   clients (from OLD_SERVICE) and let "it" sort out the most
   expeditious way of doing so;
- signal the old service that it should *shed* the old clients
   (to NEW_SERVICE) and let it figure out how best to affect that
   change

I think I prefer the former as it lets the new service assume the
agency implied by "on high"; *it* acts to affect the desired
change but with a better idea of what is involved in doing that
("on high" being ignorant of the specifics of this service).
So, it can ensure its own house is in order before contacting
the old service and imposing the directive on it.

In this way, the old service can make decisions as to which
clients should be migrated first (there is always the threat
of an unceremonious process shutdown imposed without consent
"at some time", hereafter).  And, it can negotiate with the
new service as to the best way of exchanging state information
for those connections (e.g., define the protocol as well as
the mechanism)

Back to comp.arch.embedded | Previous | Next — Next in thread | Find similar | Unroll thread

Thread

Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-20 16:05 -0700
  Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-20 16:11 -0700
  Re: Dynamic upgrading/Hot-swapping a service upsidedown@downunder.com - 2017-04-21 10:45 +0300
    Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-21 10:43 -0700
      Re: Dynamic upgrading/Hot-swapping a service upsidedown@downunder.com - 2017-04-21 23:06 +0300
        Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-21 17:51 -0700
          Re: Dynamic upgrading/Hot-swapping a service upsidedown@downunder.com - 2017-04-22 14:09 +0300
            Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-22 08:55 -0700
              Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-23 13:01 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-23 10:52 -0700
                Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-24 10:48 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-24 12:33 -0700
                Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-24 22:02 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-24 21:37 -0700
                Re: Dynamic upgrading/Hot-swapping a service Phil Martel <pomartel@comcast.net> - 2017-04-25 12:19 -0400
                Re: Dynamic upgrading/Hot-swapping a service Don Y <blockedofcourse@foo.invalid> - 2017-04-25 13:54 -0700

csiph-web