Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.misc > #5621 > unrolled thread

Sometimes Kill -9 Isn't Enough

Back to article view | Back to comp.misc

Contents

  Sometimes Kill -9 Isn't Enough Rich <rich@example.invalid> - 2014-11-13 02:27 +0000
    Re:Sometimes Kill -9 Isn't Enough RS Wood <rsw@therandymon.com> - 2014-11-14 16:23 +0000
      Re: Sometimes Kill -9 Isn't Enough Marko Rauhamaa <marko@pacujo.net> - 2014-11-14 19:31 +0200
        Re: Sometimes Kill -9 Isn't Enough RS Wood <rsw@therandymon.com> - 2014-11-14 17:33 +0000

#5621 — Sometimes Kill -9 Isn't Enough

http://www.bravenewgeek.com/sometimes-kill-9-isnt-enough/

   If there's one thing to know about distributed systems, it's that they
   have to be designed with the expectation of failure. It's also safe to
   say that most software these days is, in some form, distributed -
   whether it's a database, mobile app, or enterprise SaaS. If you have two
   different processes talking to each other, you have a distributed
   system, and it doesn't matter if those processes are local or
   intergalactically displaced.

   Marc Hedlund recently had a great post on Stripe's game-day exercises
   where they block off an afternoon, take a blunt instrument to their
   servers, and see what happens. We're talking like abruptly killing
   instances here - kill -9, ec2-terminate-instances, yanking on the damn
   power cord - that sort of thing. Everyone should be doing this type of
   stuff. You really don't know how your system behaves until you see it
   under failure conditions.

   ...

[toc] | [next] | [standalone]

#5633

 We're talking like abruptly killing
>    instances here - kill -9, ec2-terminate-instances, yanking on the damn
>    power cord - that sort of thing. Everyone should be doing this type of
>    stuff. You really don't know how your system behaves until you see it
>    under failure conditions.
> 
>    ...
> 

That's actually a great idea.  It's so counterintuitive though
 it's a frightening proposition. 
-- 

[toc] | [prev] | [next] | [standalone]

#5634

RS Wood <rsw@therandymon.com>:

>>  We're talking like abruptly killing instances here - kill -9,
>>  ec2-terminate-instances, yanking on the damn power cord - that sort
>>  of thing. Everyone should be doing this type of stuff. You really
>>  don't know how your system behaves until you see it under failure
>>  conditions.
>
> That's actually a great idea. It's so counterintuitive though it's a
> frightening proposition.

In cluster computing, you have this thing called fencing: excluding
faulty nodes from the cluster. There are numerous techniques for
fencing, but one of the cleanest techniques is turning off the power.

In a two-node cluster, no quorum can form that could exclude an oddball
node. Instead, when the internode communication is lost, each node
assumes the partner node is misbehaving, immediately pulls its weapon
and shoots the other node dead, old-western-style.

This is accomplished using a special power supply that takes
assassination orders from the nodes. It arbitrates between the
competing requests and turns off the power to the node who arrived
later.

The process is supposed to be immediate and complete. No graceful
shutdown. No committing transactions, flushing the cache or ensuring the
consistency of metadata. No last will and testament. It's an immediate
execution without the chance of appeal. That way, the cluster minimizes
the chance of the damage spreading.


Marko

[toc] | [prev] | [next] | [standalone]

#5635

Marko Rauhamaa <marko@pacujo.net> Wrote in message:
> 
> This is accomplished using a special power supply that takes
> assassination orders from the nodes. It arbitrates between the
> competing requests and turns off the power to the node who arrived
> later.
 

I'll have to read into this - it's endlessly fascinating.
-- 

[toc] | [prev] | [standalone]

Back to top | Article view | comp.misc

csiph-web