Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #12718

Re: Resource revocation

Cross-posted to 2 groups.

Show all headers | View raw

Hi Paul,

On 7/25/2013 11:37 PM, Paul Rubin wrote:
> Don Y <this@isnotme.com> writes:
>> On the other hand, it might be acceptable for the task to *think*
>> it still owns the resource and have it "waste" CPU acting on that
>> misbelief.
>
> This type of program typically doesn't compute very much.  It's either
> acting on some message, or sleeping til the next message arrives.

Actually, the irrigation program "computes" a fair amount, given
how "slow" it is expected to operate.  E.g., it has to identify
the "plants" ("water consumers") that are serviced by its zone;
identify their individual water needs; identify the "emitters"
that service their respective root systems and the flow rates
associated with each of those emitters; track the water available
to them "recently" (rainfall, water from other nearby irrigation
zones that happen to overlap their root systems, supplemental water
"injected" by the user manually; etc.); the amount of sunshine
falling on them (some might be shaded during some seasons while
others are in "full/reflected sun") as well as the desicating (sp?)
effects of the wind (again, noting the individual "exposure" to
wind from particular directions;  etc.

And, it has to continue to update these data while waiting for
the "water resource".  Or, waiting for its *return* (if it has
been revoked).  Along the way, it may have to escalate its
request as the *hard* deadline approaches.  ("Hey, if you don't
let me water these things soon, they will die -- in which case,
there is no point in my continuing to execute as a task!")

[I.e., this is a soft realtime problem layered inside a hard realtime
shell.  There *is* a point where a missed dealine results in a failure]

> I think overall it's preferable to not confuse the issue by moving stuff
> around between processes without the processes knowing.

The question becomes one of whether you inform the process *before*
you take action (or, even let the process itself "relinquish" the
resource) or, if you inform the process after-the-fact.  (or, if you
just kill the process and don't even worry about informing it!  :> )

> The resource
> should be under control of one process, and relinquished by 1) sending a
> message asking for the process to give it back; or 2) killing the
> process, preferably with automatic cleanup actions when the process
> dies.

If you "request" the process to relinquish the resource, then the
system (i.e., all other consumers of that resource) are at the
mercy of the developer who coded that application.  If he fails to
relinquish it (perhaps even failing to notice the notification!)
or intentionally delays relinquishing it (like a youngster trying
to postpone bed-time), then other consumers suffer.

I.e., if everyone adopts that sort of attitude, then you've got
a sluggish system.

And, you *still* would need a kill switch so a stubborn consumer
could be forced to relinquish the resource, regardless of his wishes.

[Note this all ignores the timeliness issues involved.  How *quickly*
must a task relinquish a resource when commanded?  What happens if the
task isn't a high enough priority to even be granted a significant
slice of the CPU to process that request?]

I've taken the other approach.  A process owns (permanently?) the
resources.  It then doles them out, on request, to other consumers.
When it wants/needs to give the resource to another consumer, it
does so -- and notifies the previous owner that it has LOST the
resource.  (of course, it can also *request* a current consumer
to release a resource... but, it has to be capable of withdrawing
them in the presence of uncooperative consumers!)

This allows me to ensure "policy" over how a resource is managed
is centralized in one place:  the (permanent) "owner" of the resource.

Ponder:

We don't notify a task that we are going to take the *CPU* away from
it (timeslice) and expect the task to respond, "OK, you can have it".
Instead, we just *take* the processor and give it to <whatever>
*we* (scheduler) decide is the most important use for that resource.
There are no guarantees that the interrupted task will ever regain
the CPU.  Nor any notification that it has *lost* the CPU!

Yet, this is something we are comfortable with...

>> Everything in my system is message based ... Signals
>> are even delivered in this way
>
> OK, the usual sense of signals that I thought was reflected in your code
> sample, is basically delivering a simulated hardware interrupt to a
> running task, so it needs locks, critical sections and all that messy
> stuff.

Yes, the signal *is* delivered as a simulated hardware interrupt
(targeted towards that task).  But, it is passed to the "task"
as a message from <whatever> task (the one who raises the signal)
*through* the kernel and to the scheduler as it (eventually) prepares
to resume that signalled task.  (I.e., I need to be able to raise
a signal on one physical processor and have the task that it affects
reside on *another* physical processor).

>> Yes.  Everything has the weight of a system call, effectively.
>
> In the case of your lawn sprinkler application I think that is fine.
> IMHO in this day and age, it's only worth dealing with low-level
> approaches if you're doing hard-real-time or have to run on 10-cent
> processors or something like that.

I chose the irrigation example because it avoids the issues of
timescale.  So, we're not distracted by the efficiency of the
delivery mechanism, etc.

But, just because its "slow" and "computationally simple" (compared
to rendering video), that doesn't make it any less of a concern.
E.g., if there is only a few KIPs of spare capacity in the processor
(since processors do more than just control water valves), then
this can be just as constrained as trying to implement a mouse in
200 bytes of FLASH...

>> Well, the "resource" in question has already been passed along
>> to someone else.
>
> Yeah, the way I'm imagining, I wouldn't do it that way, as described
> above.
>
>> So, its more a question of do you let the task continue to consume
>> *other* resources in the mistaken belief that it still owns the
>> significant resource?
>
> In this case I'd say just kill the task, so it can restart in a
> completely known state.  Admittedly I am somewhat under the influence of
> Erlang right now, and this is a core tenet of Erlang philosophy.

I had a friend who coded like that.  Spawn hundreds of processes...
then, kill of the ones he decided weren;t important.  :-/

>> Yes, but this approach assumes there are no associated activities
>> that must be unwound.  I.e., that everything the task has done
>> can be discarded and recreated.
>
> The task would periodically post updates saying how far it has gotten
> (how much water has been dispensed, or whatever).  When it's killed and
> restarts, it can take up where it left off.

Yes.  All of these approaches are just juggling "responsibilities".
E.g., in my case, a task only checkpoint when it knows the resource
has been revoked *and* the nature of the task requires remembering
state (vs. simply restarting from scratch).  If you require the
task to *periodically* checkpoint itself, then it has to come to
some sort of balance between spending all of its time checkpointing
(so it has very fine-grained resumption capability) vs. very little
checkpointing (so it doesn't waste its time keeping track of where
it was).

[Recall, the checkpointing must be done in a medium that is more
persistent than the task's context -- since the task's execution
environment can be torn down (completely) at any time.  So, now
the system must provide a service for this -- and, one that is
sufficiently lightweight that invoking it OFTEN doesn't affect
performance... or, the performance of other tasks.]

(remember, you don't necessarily have a big disk sitting there
or scads of RAM... how much is a task granted access to?  What
if it requires *more* to preserve its "significant" state?)

>> Imagine if the task is machining a piece of metal and the resource
>> that it wanted has been withdrawn.
>
> I don't understand this example--what would the "resource" be?  In

Maybe a cutting tool.  Maybe a power source because the next operation
puts significant demands on the available power supply (which is
shared by other machines in the facility).  Maybe a coolant system
(what happens if you withdraw the coolant before it has had a
chance to achieve its intended goal... is the "piece" ruined?)

The point is, you may not be able to "resume" the operation.
You've just made an expensive piece of "scrap".  And, even if
this is unavoidable, you have to *know* that it is scrap and
must be disposed of... not "resumed".

> general terms I'd say kill the process and let the crash handler park
> the tool in a safe position.  But in this machining example, I'm
> imagining some kind of low level PID loop that would keep checking a
> flag to know if it had to bail out.  In either case, the idea is to get
> to a place where you can restart later.
>
>> The "advantage" to an asynchronous notification *like* a signal is
>> it gives the developer as much rope as he wants
>
> The only reason for such "rope" is to push the limits of the hardware
> because more modular approaches are too slow or whatever.  Computers are
> ridiculously powerful these days, so unless you're doing something
> extremely demanding (basically something that would have been impossible
> or economically unfeasible 10 years ago), seeking "rope" is probably a
> sign of doing something wrong.

Dealing with consumer markets, *everything* is economically unfeasible!
(unless you are catering to consumers who are not cost conscious).
E.g., I would imagine most irrigation controllers are implemented
with little PICs -- because their approach to the problem is much
more naive:  turn the water on for X minutes, then advance to the
next zone.  They don't look at the *needs* of their "consumers"
(plants/landscape).  Nor do they worry about the availability of the
resource they are using.  I.e., only a single zone is active at a time
(often, though not necessarily) and they assume someone else has
ensured an adequate supply *to* the valve manifold.

Here, for example, if I turned on all of the irrigation valves
simultaneously, several things would happen:
- household water pressure would drop noticeably
- the implied flow rates of each irrigation "emitter" would not
   be correct (because the water pressure in the irrigation
   system would have fallen below nominal)
- some of the irrigation loads probably wouldn't "work" at all
   (i.e., not having enough static head to meet the required rise)

But, no particular "zone" should have to worry about this.  It's
a system constraint.  One that should be enforced by whatever
doles out the "water resource".  If someone decides to "run a bath",
the individual irrigation zones shouldn't need to know that their
water use will interfere with that activity.  (OTOH, something
"higher" in the system design should be able to enforce that
policy on the irrigation system)

>> The problem lies in the nature of asynchronous events and how
>> we think of them as developers.
>
> Right, they are messy and it's preferable to avoid them.  E.g. by using
> message passing instead of signals.

But messages only "exist" when they are being examined.  If, for
example, you issue a query to the database, you either assume the
query happens "fast enough" (whatever that means in your application)
*or* spawn a separate thread to process that query so you can
keep watching for messages.  You now have yet another synchronization
issue to contend with, etc.

>> The other half of this is that it is REALLY HARD to test for these
>> things.  You can't just inject an asynchronous event at every possible
>> place where it could occur and verify proper operation.  :(
>
> You should probably look into model-checking tools if you absolutely
> have to pursue this approach.  Dawson Engler's papers on using such
> tools to find crash bugs in Unix file systems might be of interest.
>
> Actually Tom Hawkins' "ImProve" program might be of some use to check
> that you got all your watering stuff right, in terms of turning correct
> combinations of valves on and off etc., if you're interested in

I've avoided the "combinations" issue entirely (at least in the
"zone controller tasks").  An individual zone deals only with the
needs of its "consumers" (plants).  It knows that it may not have
access to the resources that it requires at all times (water,
"information", etc.)  So, it knows how to deal with these deficits
on its own.

Similarly, the "water controller" only has to worry about the needs
of *its* consumers (several of which are the individual irrigation
zone controllers).  And, how it can recover from those cases where
it can not supply the needs of those consumers (loss of water
supply, master valve failure, etc.)

> experimenting with high-tech approaches.  I haven't used it but have
> been interested in it for a while:
>
>     https://github.com/tomahawkins/improve/wiki/ImProve
>
> I did play around with Atom (a hard realtime DSL written by the same
> guy) and I think the approach is pretty powerful.
>
>> I really dislike telling people they "can't run with scissors" --
>> since those folks who are adept at doing so shouldn't be penalized by
>> the "clumsiness" of those who can't!
>
> Even someone capable of running with scissors without stabbing himself
> every time shouldn't do it outside of some dire emergency.

It boils down to what you consider an "emergency".  And, how much
"spare capacity" you have available.  E.g., if you can afford to
*walk* from place to place with those scissors, then there is
no need to take on the risk of running!

OTOH, if you don't have the luxury of being able to take a leisurely
stroll with them, then you either *break* (fail to meet your goals)
or you learn to run, safely!

> In this watering application you have (presumably) rather loose timing
> constraints, and roughly unlimited CPU resources.  So I think you can do
> fine using safe, simple methods instead of running with scissors.

Again, irrigation is a trivial example.  Imagine, instead, the
resources are physical processors and you are rendering video
in real time for distribution over the network.  If one of those
resources becomes unavailable (crashes or is repurposed to put out
something more important than watching TV), how much spare
capacity do you design into the system so you can *leisurely*
go about recovering?  (Remember, if the next frame isn't there
in a few tens of milliseconds, the user will perceive a visual
artifact in the rendered image!)

I.e., I would like to find *one* way of dealing with this sort
of thing instead of one way for "fast" things and another for
"slow" things, etc. (because that leaves "fast" and "slow" up
to debate and requires developers to appreciate the differences)

--don

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Resource revocation Don Y <this@isnotme.com> - 2013-07-25 12:23 -0700
  Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-25 12:51 -0700
    Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-25 15:00 -0700
      Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-25 19:38 -0700
        Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-25 22:12 -0700
          Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-25 23:37 -0700
            Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-26 01:13 -0700
              Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-26 02:48 -0700
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-26 04:19 -0700
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-26 09:46 -0700
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-26 10:21 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-26 19:30 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-26 11:56 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-26 20:08 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-26 13:11 -0700
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-26 12:31 -0700
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-27 11:08 -0700
                Re: Resource revocation Rob Gaddi <rgaddi@technologyhighland.invalid> - 2013-07-29 09:16 -0700
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-29 10:22 -0700
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-29 11:22 -0700
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-27 08:56 -0700
                Re: Resource revocation Hans-Bernhard Bröker <HBBroeker@t-online.de> - 2013-07-26 19:25 +0200
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-26 10:51 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-26 19:21 +0100
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-26 11:50 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-26 19:42 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-26 12:43 -0700
                Does the Buddha have a real time nature? ;)  (was Resource revocation) Roberto Waltman <usenet@rwaltman.com> - 2013-07-26 17:06 -0400
                Re: Does the Buddha have a real time nature? ;)  (was Resource revocation) Don Y <this@isnotme.com> - 2013-07-26 17:12 -0700
                Re: Does the Buddha have a real time nature? ;)  (was Resource revocation) Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-27 09:44 +0100
                Re: Does the Buddha have a real time nature? ;)  (was Resource revocation) Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-27 09:45 +0100
                Re: Does the Buddha have a real time nature? ;)  (was Resource revocation) Don Y <this@isnotme.com> - 2013-07-27 12:28 -0700
                Re: Does the Buddha have a real time nature? ;)  (was Resource revocation) Hans-Bernhard Bröker <HBBroeker@t-online.de> - 2013-07-27 14:11 +0200
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-26 12:40 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-26 21:29 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-26 13:52 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-26 22:55 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-26 17:22 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-27 10:02 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-27 09:29 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-27 18:20 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-27 12:48 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-27 22:57 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-27 16:18 -0700
                Re: Resource revocation Roberto Waltman <usenet@rwaltman.com> - 2013-07-31 16:33 -0400
                Re: Resource revocation upsidedown@downunder.com - 2013-07-27 22:53 +0300
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-27 22:42 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-27 16:41 -0700
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-27 16:49 -0700
                Re: Resource revocation upsidedown@downunder.com - 2013-07-28 08:39 +0300
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-27 23:11 -0700
                Re: Resource revocation Roberto Waltman <usenet@rwaltman.com> - 2013-07-31 16:15 -0400
                Re: Resource revocation upsidedown@downunder.com - 2013-08-01 00:13 +0300
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-27 15:16 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-27 23:05 +0100
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-27 15:37 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-27 23:38 +0100
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-27 16:08 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-28 01:16 +0100
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-27 18:43 -0700
                Re: Resource revocation upsidedown@downunder.com - 2013-07-28 09:01 +0300
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-27 15:21 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-27 23:07 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-26 12:00 -0700
                Re: Resource revocation stephenXXX@mpeforth.com (Stephen Pelc) - 2013-07-27 16:49 +0000
                Re: Resource revocation Richard Damon <Richard@Damon-Family.org> - 2013-07-28 18:31 -0400
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-28 15:51 -0700
                Re: Resource revocation Richard Damon <Richard@Damon-Family.org> - 2013-07-28 20:12 -0400
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-28 18:35 -0700
                Re: Resource revocation Richard Damon <Richard@Damon-Family.org> - 2013-07-29 00:10 -0400
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-29 00:23 -0700
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-29 01:05 -0700
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-29 12:07 -0700
                Re: Resource revocation upsidedown@downunder.com - 2013-07-30 01:11 +0300
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-29 15:20 -0700
                Re: Resource revocation Rob Gaddi <rgaddi@technologyhighland.invalid> - 2013-07-29 15:42 -0700
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-29 16:41 -0700
                Re: Resource revocation upsidedown@downunder.com - 2013-07-30 08:51 +0300
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-30 09:09 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-30 01:30 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-30 10:04 +0100
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-30 02:55 -0700
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-30 07:12 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-30 16:59 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-30 13:17 -0700
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-30 10:12 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-30 18:48 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-30 20:18 -0700
                Re: Resource revocation upsidedown@downunder.com - 2013-07-31 09:27 +0300
                Re: Resource revocation [long] Don Y <this@isnotme.com> - 2013-07-31 00:47 -0700
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-30 02:50 -0700
                Re: Resource revocation upsidedown@downunder.com - 2013-07-30 13:22 +0300
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-30 07:24 -0700
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-30 01:17 -0700
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-29 19:28 -0700
                Re: Resource revocation upsidedown@downunder.com - 2013-07-29 11:40 +0300
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-29 11:44 -0700
                Re: Resource revocation Tom Gardner <spamjunk@blueyonder.co.uk> - 2013-07-29 20:56 +0100
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-29 13:16 -0700
                Re: Resource revocation Richard Damon <Richard@Damon-Family.org> - 2013-07-30 00:08 -0400
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-30 23:41 -0700
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-31 00:58 -0700
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-31 10:18 -0700
                Re: Resource revocation George Neuner <gneuner2@comcast.net> - 2013-08-02 06:19 -0400
                Re: Resource revocation Richard Damon <Richard@Damon-Family.org> - 2013-08-03 02:26 -0400
                Re: Resource revocation Paul Rubin <no.email@nospam.invalid> - 2013-07-28 19:31 -0700
                Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-29 00:44 -0700
  Re: Resource revocation Richard Damon <Richard@Damon-Family.org> - 2013-07-25 22:42 -0400
    Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-25 22:12 -0700
  Re: Resource revocation "Boudewijn Dijkstra" <sp4mtr4p.boudewijn@indes.com> - 2013-07-31 13:41 +0200
    Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-31 07:51 -0700
      Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-31 08:07 -0700
        Re: Resource revocation Anders.Montonen@kapsi.spam.stop.fi.invalid - 2013-07-31 15:55 +0000
          Re: Resource revocation Don Y <this@isnotme.com> - 2013-07-31 10:18 -0700
      Re: Resource revocation "Boudewijn Dijkstra" <sp4mtr4p.boudewijn@indes.com> - 2013-08-01 14:42 +0200
        Re: Resource revocation Don Y <this@isnotme.com> - 2013-08-01 14:34 -0700
      Re: Resource revocation upsidedown@downunder.com - 2013-08-02 09:08 +0300
        Re: Resource revocation Don Y <this@isnotme.com> - 2013-08-02 07:26 -0700

csiph-web