Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > comp.arch.embedded > #25896 > unrolled thread
| Started by | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| First post | 2017-03-05 16:03 -0700 |
| Last post | 2017-03-11 14:21 -0700 |
| Articles | 20 on this page of 202 — 22 participants |
Back to article view | Back to comp.arch.embedded
Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-05 16:03 -0700
Re: Language feature selection Wouter van Ooijen <wouter@voti.nl> - 2017-03-06 08:04 +0100
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-07 10:31 -0700
Re: Language feature selection Wouter van Ooijen <wouter@voti.nl> - 2017-03-07 23:29 +0100
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-08 02:49 -0700
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-08 13:18 +0100
Re: Language feature selection Wouter van Ooijen <wouter@voti.nl> - 2017-03-08 16:58 +0100
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-08 10:35 -0700
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-08 22:07 +0100
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-06 08:40 +0100
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-06 10:31 +0100
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-06 08:47 -0700
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-06 21:33 +0100
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-06 13:44 -0700
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-07 12:24 +0100
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-07 09:34 -0700
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-08 10:14 +0100
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-08 10:37 -0700
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-08 22:09 +0100
Re: Language feature selection Tauno Voipio <tauno.voipio@notused.fi.invalid> - 2017-03-06 22:50 +0200
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-07 12:31 +0100
Re: Language feature selection rickman <gnuarm@gmail.com> - 2017-03-07 14:45 -0500
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-06 08:39 +0000
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-06 08:39 -0700
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-06 17:27 +0000
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-06 13:30 -0700
Re: Language feature selection Dimiter_Popoff <dp@tgi-sci.com> - 2017-03-06 11:11 +0200
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-06 13:38 -0700
Re: Language feature selection Dimiter_Popoff <dp@tgi-sci.com> - 2017-03-07 12:18 +0200
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-07 10:28 -0700
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-06 13:33 -0500
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-06 13:36 -0700
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-06 22:15 -0500
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-06 22:22 -0500
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-07 00:27 -0700
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-08 12:20 -0500
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-08 11:23 -0700
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-10 13:43 -0500
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-10 16:49 -0700
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-11 04:17 -0500
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-11 12:33 -0700
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-12 18:08 -0400
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-12 17:25 -0700
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-15 21:05 -0400
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-16 17:23 -0700
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-17 23:59 -0400
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-17 23:25 -0700
Re: Language feature selection Tauno Voipio <tauno.voipio@notused.fi.invalid> - 2017-03-18 12:05 +0200
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-18 04:04 -0700
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-18 04:39 -0700
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-18 03:55 -0700
Re: Language feature selection upsidedown@downunder.com - 2017-03-18 16:00 +0200
Re: Language feature selection Les Cargill <lcargill99@comcast.com> - 2017-03-11 16:35 -0600
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-11 15:39 -0700
Re: Language feature selection Grant Edwards <invalid@invalid.invalid> - 2017-03-06 18:59 +0000
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-06 12:36 -0700
Re: Language feature selection Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> - 2017-03-07 19:02 +0000
Re: Language feature selection Grant Edwards <invalid@invalid.invalid> - 2017-03-07 19:18 +0000
Re: Language feature selection Grant Edwards <invalid@invalid.invalid> - 2017-03-07 19:19 +0000
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-07 12:38 -0700
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-07 22:10 -0600
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-07 20:35 -0800
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-08 20:43 -0600
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-07 23:48 -0700
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-08 20:48 -0600
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-08 21:15 -0700
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-09 01:33 -0800
Re: Language feature selection Grant Edwards <invalid@invalid.invalid> - 2017-03-09 14:47 +0000
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-09 16:40 +0100
Re: Language feature selection Grant Edwards <invalid@invalid.invalid> - 2017-03-09 16:15 +0000
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-09 22:04 +0100
Re: Language feature selection John Devereux <john@devereux.me.uk> - 2017-03-10 07:46 +0000
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-10 13:21 +0100
Re: Language feature selection Les Cargill <lcargill99@comcast.com> - 2017-03-09 17:57 -0600
Re: Language feature selection Tauno Voipio <tauno.voipio@notused.fi.invalid> - 2017-03-10 11:16 +0200
Re: Language feature selection Jacob Sparre Andersen <jacob@jacob-sparre.dk> - 2017-03-10 20:39 +0100
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-10 13:32 -0800
Re: Language feature selection Niklas Holsti <niklas.holsti@tidorum.invalid> - 2017-03-11 14:58 +0200
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-13 13:02 +0100
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-08 10:46 +0100
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-08 20:33 -0600
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-09 11:08 +0100
Re: Language feature selection Britt <britt.snodgrass@gmail.com> - 2017-03-09 07:04 -0800
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-09 16:36 +0100
Re: Language feature selection Grant Edwards <invalid@invalid.invalid> - 2017-03-09 16:11 +0000
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-09 12:06 -0600
Re: Language feature selection Hans-Bernhard Bröker <HBBroeker@t-online.de> - 2017-03-09 19:26 +0100
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-09 22:08 +0100
Re: Language feature selection Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> - 2017-03-09 19:04 +0000
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-09 15:34 -0700
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-09 15:40 -0700
Re: Language feature selection Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> - 2017-03-10 14:19 +0000
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-10 11:10 -0700
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-10 11:49 -0700
Re: Language feature selection Les Cargill <lcargill99@comcast.com> - 2017-03-09 18:00 -0600
Re: Language feature selection Les Cargill <lcargill99@comcast.com> - 2017-03-09 17:54 -0600
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-09 18:54 -0600
Re: Language feature selection Grant Edwards <invalid@invalid.invalid> - 2017-03-10 04:07 +0000
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-10 08:07 -0600
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-10 13:59 +0100
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-10 08:17 +0000
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-10 11:24 -0700
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-10 19:09 +0000
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-10 12:33 -0700
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-10 23:25 +0000
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-10 20:22 -0500
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-11 01:32 +0000
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-10 18:16 -0800
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-11 09:59 +0000
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-11 02:23 -0800
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-11 11:13 +0000
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-11 14:28 -0500
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-11 17:20 -0500
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-12 16:26 -0400
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-12 21:46 +0100
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-12 23:15 +0000
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-13 14:14 -0400
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-13 18:48 +0000
Re: Language feature selection "Brian G. Lucas" <bagel99@gmail.com> - 2017-03-13 15:17 -0500
Re: Language feature selection Niklas Holsti <niklas.holsti@tidorum.invalid> - 2017-03-13 23:17 +0200
Re: Language feature selection "Brian G. Lucas" <bagel99@gmail.com> - 2017-03-13 17:27 -0500
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-13 17:10 -0700
Re: Language feature selection Niklas Holsti <niklas.holsti@tidorum.invalid> - 2017-03-14 10:11 +0200
Re: Language feature selection "Brian G. Lucas" <bagel99@gmail.com> - 2017-03-14 12:23 -0500
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-14 13:27 +0100
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-14 13:30 -0400
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-14 22:03 +0100
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-14 17:24 -0400
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-11 07:19 -0600
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-11 19:43 +0000
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-12 09:28 -0600
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-12 15:15 -0400
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-12 19:38 +0000
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-10 18:08 -0800
Re: Language feature selection George Neuner <gneuner2@comcast.net> - 2017-03-11 04:22 -0500
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-11 10:02 +0000
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-11 02:22 -0800
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-11 11:07 +0000
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-10 14:30 -0600
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-10 13:37 -0800
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-10 16:30 -0600
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-10 18:14 -0800
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-11 07:46 -0600
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-11 12:53 -0700
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-12 09:45 -0600
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-12 11:56 -0700
Re: Language feature selection Dimiter_Popoff <dp@tgi-sci.com> - 2017-03-12 21:05 +0200
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-12 14:40 -0700
Re: Language feature selection Dimiter_Popoff <dp@tgi-sci.com> - 2017-03-13 00:36 +0200
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-12 17:00 -0700
Re: Language feature selection Robert Wessel <robertwessel2@yahoo.com> - 2017-03-13 16:50 -0500
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-13 17:08 -0700
Re: Language feature selection Robert Wessel <robertwessel2@yahoo.com> - 2017-03-13 20:51 -0500
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-13 20:31 -0700
Re: Language feature selection Robert Wessel <robertwessel2@yahoo.com> - 2017-03-14 21:45 -0500
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-14 21:16 -0700
Re: Language feature selection Les Cargill <lcargill99@comcast.com> - 2017-03-11 16:40 -0600
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-10 15:10 -0700
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-10 16:42 -0600
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-10 17:02 -0700
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-11 00:46 +0000
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-11 13:05 -0700
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-11 21:58 +0000
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-11 15:19 -0700
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-11 23:27 +0000
Re: Language feature selection Walter Banks <walter@bytecraft.com> - 2017-03-10 20:14 -0600
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-11 12:16 -0700
Re: Language feature selection Hans-Bernhard Bröker <HBBroeker@t-online.de> - 2017-03-10 20:40 +0100
Re: Language feature selection Wouter van Ooijen <wouter@voti.nl> - 2017-03-10 22:28 +0100
Re: Language feature selection Jacob Sparre Andersen <jacob@jacob-sparre.dk> - 2017-03-10 20:03 +0100
Re: Language feature selection Grant Edwards <invalid@invalid.invalid> - 2017-03-10 19:48 +0000
Re: Language feature selection Jacob Sparre Andersen <jacob@jacob-sparre.dk> - 2017-03-13 21:25 +0100
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-10 13:03 -0700
Re: Language feature selection Jacob Sparre Andersen <jacob@jacob-sparre.dk> - 2017-03-13 21:40 +0100
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-13 14:35 -0700
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-14 01:06 +0000
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-13 19:23 -0700
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-14 08:48 +0000
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-14 02:14 -0700
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-14 09:33 +0000
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-14 09:21 -0700
Re: Language feature selection Les Cargill <lcargill99@comcast.com> - 2017-03-17 06:01 -0500
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-17 11:39 +0000
Re: Language feature selection Les Cargill <lcargill99@comcast.com> - 2017-03-17 07:16 -0500
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-17 13:49 +0000
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-17 11:03 -0700
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-17 19:21 +0100
Re: Language feature selection Paul Rubin <no.email@nospam.invalid> - 2017-03-17 11:34 -0700
Re: Language feature selection David Brown <david.brown@hesbynett.no> - 2017-03-17 20:26 +0100
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-17 20:07 +0000
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-17 20:29 +0000
Re: Language feature selection Les Cargill <lcargill99@comcast.com> - 2017-03-17 19:19 -0500
Re: Language feature selection Tom Gardner <spamjunk@blueyonder.co.uk> - 2017-03-18 08:18 +0000
Re: Language feature selection Hans-Bernhard Bröker <HBBroeker@t-online.de> - 2017-03-14 21:20 +0100
Re: Language feature selection jim.brakefield@ieee.org - 2017-03-10 18:46 -0800
Re: Language feature selection Niklas Holsti <niklas.holsti@tidorum.invalid> - 2017-03-11 15:41 +0200
Re: Language feature selection jim.brakefield@ieee.org - 2017-03-11 14:20 -0800
Re: Language feature selection upsidedown@downunder.com - 2017-03-12 00:54 +0200
Re: Language feature selection jim.brakefield@ieee.org - 2017-03-12 18:08 -0700
Re: Language feature selection upsidedown@downunder.com - 2017-03-13 23:36 +0200
Re: Language feature selection Robert Wessel <robertwessel2@yahoo.com> - 2017-03-13 17:03 -0500
Re: Language feature selection Don Y <blockedofcourse@foo.invalid> - 2017-03-11 14:21 -0700
Page 3 of 11 — ← Prev page 1 2 [3] 4 5 … 11 Next page →
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2017-03-11 12:33 -0700 |
| Message-ID | <oa1j82$tja$1@dont-email.me> |
| In reply to | #26003 |
Hi George,
On 3/11/2017 2:17 AM, George Neuner wrote:
> On Fri, 10 Mar 2017 16:49:21 -0700, Don Y
> <blockedofcourse@foo.invalid> wrote:
>
>> On 3/10/2017 11:43 AM, George Neuner wrote:
>>>
>>> You asked if closures introduced a GC penalty - the answer to which is
>>> "not necessarily". Now you are complaining that functions referenced
>>> by a closure need to remain available for its entire lifetime.
>>
>> Yes -- so how does the tool and/or developer ensure it provides the
>> intended functionality when required? How do you imbue the tool
>> with the "smarts" to be able to analyze these cases (e.g., impose
>> a "persistent" implementation) in light of the different ways
>> that the developer can opt to use it?
>
> Dependency (import) analysis shows what is needed - but not when it is
> needed. *Safe* dynamic unloading requires some form of GC - but the
> form GC takes depends on many factors.
>
> E.g., using stack-based closures, or in the absence of closures
> altogether, it is relatively simple for a compiler to identify uses of
> a module and affect OS-based reference counting to control its
> residency. This works transitively if all modules obey the same
> conventions.
>
> [This is just like using COM or CORBA interfaces, but the mechanism
> would be built into the primary language compiler instead of being a
> separate tool. More later.]
>
> Once you introduce heap-based closures the lifetime of closures
> becomes indefinite: it can be *estimated* by a compiler - e.g., using
> region analysis - but runtime GC generally is better at noticing
> disuse in a timely fashion. Module reference counting by the compiler
> is no longer sufficient and must be augmented by runtime GC.
This (as with other aspects below) suggests the compiler really
needs to see the entire "application" ("system"). And, even then,
can be tripped up because it doesn't know what will come and go
(dynamic "applet" loading) or when. Add another "applet" and
the apple-cart has been toppled and needs to be re-sorted...
>> Alternatively, how do you give the developer tools that let *him*
>> convey those dependencies to the tool?
>>
>> Unless you restrict how they can be used, I don't see how you can
>> address each possibility...
>
> It is up to the programmer to structure her modules so that dynamic
> (un)loading is most effective. Modules have management overhead, and
> many small modules could be worse than fewer, larger ones.
>
> There is no formulae for figuring out the best structure. There may
> be systemic usage patterns which suggest certain structurings, but
> such patterns necessarily are usage dependent.
I see this as just making coding more "artful" -- contrary to the
current trend in language design ("trained monkeys protected from
shooting themselves in the foot")
>>> A module has not "served its purpose" if its functions may yet be
>>> called in the future. That, in general, is undecidable.
>>>
>>> The GC standard of "reachability" is conservative - GC does not
>>> consider whether an object will be used again because it can't know
>>> that. A compiler is in a better position to figure out usage, but
>>> even there the only way to know for certain is to simulate execution
>>> of the program and observe that it terminates without ever again
>>> referencing the object in question.
>>> [Hint: indefinate loops can't be guaranteed to terminate.]
>>
>> Exactly. I "avoid" this issue by requiring the developer to
>> handle the "resource reclamation". This means he can make a mistake
>> and shoot off both feet (so the OS has to know to catch these
>> sorts of problems, indirectly).
>
> If modules are stateless - iow, all state is in the client - then
> modules can come and go at the whim of the OS memory manager. Not
> necessarily very performant, but always correct.
Technically, even stateful modules can be swapped out -- if the
state is preserved in the process.
> However, stateless modules likely will seem unnatural to many
> programmers, who may find them difficult to create even if there is a
> language (or other mechanism) that makes them [relatively] easy to
> use.
I haven't settled on a consistent way of handling this (which is why
I am having trouble imagining how a compiler/tool could uniformly
enforce it). I look at each instance and think about how it is
intended to be used (cuz *I* am using it!) and let that information
dictate how I'll implement. What seems natural in some cases would be
piss-poor in others.
>>>> [If initialize_heap RETURNS "memory manager"s that are then used to
>>>> manage each individual heap, then initialize_heap() itself is still
>>>> dynamically bound to those objects. If you unload the module
>>>> containing it, then the memory manager objects (in this example) that
>>>> it created for the callers also disappears.]
>>>
>>> Not necessarily - it depends on the module structure.
>>> E.g.,
>>>
>>> +----------------+
>>> -| heap functions |
>>> / +----------------+
>>> / |
>>> / v
>>> / +-----------------+
>>> <closure> <-- | initialize_heap |
>>> +-----------------+
>>>
>>> initialize_heap can be in a module separate from other heap control
>>> functions. Once the (heap interface) closure is created, the module
>>> containing initialize_heap is unneeded and could be unloaded.
>>
>> That's how I "manually" implement these things. But *I* have to
>> keep track of whether the functions are loaded and *where* they
>> are loaded. E.g., if I want to migrate something to another
>> node, then I need a handle by which the heap functions (and the
>> bindings made at "initialization") can be invoked "later".
>>
>> I just don't see how a compiler can be aware of this sort of thing
>> (without PREVENTING me from doing these sorts of things).
>
> In the simple case it is just a matter of identifying call sites and
> tracking which modules are supplying the called function.
I do that at runtime -- by letting the OS track handles and handle
references.
> The main difficulty stems from reachability of code in the face of
> indeterminant loops. The compiler pretty much has to assume that any
> call made to the module will actually happen.
>
> The analysis can be simplified if the language has constructs that
> guarantee termination even in the face of errors or exceptions: e.g.,
> Lisp's unwind-protect or Scheme's dynamic-wind.
>
>> I think a lot of these sorts of things have an underlying assumption that
>> they are part of a persistent, single "program unit" so the compiler doesn't
>> have to "worry" about lifespan.
>
> Most compilers assume libraries/modules are always available *because*
> that's what they assume. There is no reason that they must assume
> this - it's simply easier to do so.
>
> Consider the ancient overlay compilers ... dynamic module handling is
> not really any different [unless you have to consider the bin-packing
> aspect of fitting arbitrary load sequences into memory].
Yes, but with overlays you have to engineer their partitioning into your
solution. The tool didn't do that on your behalf (by analyzing your code
and playing slice-and-dice)
>>>> Does the language magically enforce this dependence? Or, does the developer
>>>> have to be aware of the potential "gotcha"? Or, does something else reference
>>>> count, etc.?
>
> Modular languages handle dependence - they arrange to load "library"
> modules in the same way a C program loads a DLL. However, most
> systems have no provisions to automatically unload unreferenced
> modules.
>
> E.g., COM and CORBA, when used via IDL, normally hold a reference to
> the library until the program ends (secondary references are added and
> deleted in the course of using the library).
>
> With COM (not sure about CORBA) you can step outside the IDL and
> explicitly release the initial reference, which - if there are no
> other references held by the program - will unlink the library and
> remove it from the program's address space. But that doesn't affect
> other clients using the library - the COM server won't terminate the
> library until all clients have released it.
In my case, "no more references" causes the OS to delete the resource
("unreachable") -- whether that is a dynamically created object (e.g., a
heap, in this example) or a static one (e.g., a loaded module -- that might
find itself RE-loaded any time now)
We just think of things in different ways. You think in terms of the
compiler's work, I think in terms of the OS/runtime's.
[toc] | [prev] | [next] | [standalone]
| From | George Neuner <gneuner2@comcast.net> |
|---|---|
| Date | 2017-03-12 18:08 -0400 |
| Message-ID | <scdbcclcc152sh9g66cni5emsqjggkniv3@4ax.com> |
| In reply to | #26017 |
On Sat, 11 Mar 2017 12:33:04 -0700, Don Y
<blockedofcourse@foo.invalid> wrote:
>On 3/11/2017 2:17 AM, George Neuner wrote:
>>
>> Once you introduce heap-based closures the lifetime of closures
>> becomes indefinite: it can be *estimated* by a compiler - e.g., using
>> region analysis - but runtime GC generally is better at noticing
>> disuse in a timely fashion. Module reference counting by the compiler
>> is no longer sufficient and must be augmented by runtime GC.
>
>This (as with other aspects below) suggests the compiler really
>needs to see the entire "application" ("system"). And, even then,
>can be tripped up because it doesn't know what will come and go
>(dynamic "applet" loading) or when. Add another "applet" and
>the apple-cart has been toppled and needs to be re-sorted...
Operationally there is little difference vs manual DLL use.
(un)Loading the module is under control of the execution environment:
the OS or VM under which the application is running. The
compiler/runtime only tells the environment whether or not the
application is currently using the module.
Overall memory use is more or less the same regardless of whether
module handling is manual or automatic.
>I haven't settled on a consistent way of handling this (which is why
>I am having trouble imagining how a compiler/tool could uniformly
>enforce it). I look at each instance and think about how it is
>intended to be used (cuz *I* am using it!) and let that information
>dictate how I'll implement. What seems natural in some cases would be
>piss-poor in others.
On the face of it that seems strange: if the module mechanism is
"unnatural" in some usage, that indicates to me that the problem is
not the mechanism per se, but rather that the application itself is
not well suited to modules.
>In my case, "no more references" causes the OS to delete the resource
>("unreachable") -- whether that is a dynamically created object (e.g., a
>heap, in this example) or a static one (e.g., a loaded module -- that might
>find itself RE-loaded any time now)
>
>We just think of things in different ways. You think in terms of the
>compiler's work, I think in terms of the OS/runtime's.
No. I just can't think in terms of *your* system because it is unlike
anything in my experience.
You are aware that my background includes study of both operating
systems and virtual machine environments. I always consider the
runtime environment and how the program interacts with it.
George
[toc] | [prev] | [next] | [standalone]
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2017-03-12 17:25 -0700 |
| Message-ID | <oa4oon$s0r$1@dont-email.me> |
| In reply to | #26040 |
On 3/12/2017 3:08 PM, George Neuner wrote:
> On Sat, 11 Mar 2017 12:33:04 -0700, Don Y
> <blockedofcourse@foo.invalid> wrote:
>
>> On 3/11/2017 2:17 AM, George Neuner wrote:
>>>
>>> Once you introduce heap-based closures the lifetime of closures
>>> becomes indefinite: it can be *estimated* by a compiler - e.g., using
>>> region analysis - but runtime GC generally is better at noticing
>>> disuse in a timely fashion. Module reference counting by the compiler
>>> is no longer sufficient and must be augmented by runtime GC.
>>
>> This (as with other aspects below) suggests the compiler really
>> needs to see the entire "application" ("system"). And, even then,
>> can be tripped up because it doesn't know what will come and go
>> (dynamic "applet" loading) or when. Add another "applet" and
>> the apple-cart has been toppled and needs to be re-sorted...
>
> Operationally there is little difference vs manual DLL use.
>
> (un)Loading the module is under control of the execution environment:
> the OS or VM under which the application is running. The
> compiler/runtime only tells the environment whether or not the
> application is currently using the module.
The problem is *knowing* when it can be unloaded. In my case,
the runtime (OS) actively tracks references as a side-effect
of the implementation. E.g., each time a handle is deleted/released,
the OS does that work. So, it knows when there are no more
active handles to a particular object. And, can reclaim the
object itself.
[Likewise, it knows "who" to notify if an object is prematurely deleted
by noticing the outstanding handles still in existence at that time]
> Overall memory use is more or less the same regardless of whether
> module handling is manual or automatic.
>
>> I haven't settled on a consistent way of handling this (which is why
>> I am having trouble imagining how a compiler/tool could uniformly
>> enforce it). I look at each instance and think about how it is
>> intended to be used (cuz *I* am using it!) and let that information
>> dictate how I'll implement. What seems natural in some cases would be
>> piss-poor in others.
>
> On the face of it that seems strange: if the module mechanism is
> "unnatural" in some usage, that indicates to me that the problem is
> not the mechanism per se, but rather that the application itself is
> not well suited to modules.
Assume applet, memory_manager_module, applet_parent.
Applet_parent (for example) instantiates a memory object for applet. Part
of that requires binding parameters to that memory object (size, granularity,
allocation_policy, release_policy, etc.). Applet_parent need not persist
beyond this point (assume its job is done).
Where do the bound parameters get stored? In the applet? In the memory
object? Obviously, can't be part of applet_parent because that won't
persist (unless you make other arrangements for it or portions of it).
[I.e., applet_parent can't define an allocation handler in its body.]
Now, migrate applet to another node. Another (shared) instance of
memory_manager_module likely already exists on that node. If the
allocation handler was a "stock" handler (contained in the
memory_manager_module), can you simply copy the state over to
the new node?
What if the allocation handler was resident in some OTHER module
on the original node? Do you migrate that other module, as well?
Or, instantiate another copy if the module has outstanding references
on the first node??
Or, do you require declarations as to what *can* migrate and what
is bound in place? Then, contend with the consequences of
cascaded dependencies?
The "consistent" solution is to make all of these first class
objects and let the OS manage them and their locations/instantiations.
But, that adds considerably to the cost of *using* them.
[I'm currently exploring short-circuit paths to automatically
expedite local cases. But, that's even more complexity...]
Its a lot easier to deal with implementations where "everything" is
a cohesive "blob" instead of being able to slice-and-dice it at
run-time and redistribute it based on evolving workloads, resources,
etc.
[Do you kill off a task/applet so you can reinstantiate it, from scratch,
when new hardware is made available to the system? Or, do you let it
continue to run as that hardware's resources come on-line?]
Last 25# of oranges... yippee!
[toc] | [prev] | [next] | [standalone]
| From | George Neuner <gneuner2@comcast.net> |
|---|---|
| Date | 2017-03-15 21:05 -0400 |
| Message-ID | <inljcc1n15d869k10l8i0ve9rfhbgp8run@4ax.com> |
| In reply to | #26044 |
Hi Don, On Sun, 12 Mar 2017 17:25:43 -0700, Don Y <blockedofcourse@foo.invalid> wrote: >Assume applet, memory_manager_module, applet_parent. > >Applet_parent (for example) instantiates a memory object for applet. Part >of that requires binding parameters to that memory object (size, granularity, >allocation_policy, release_policy, etc.). Applet_parent need not persist >beyond this point (assume its job is done). > >Where do the bound parameters get stored? In the applet? In the memory >object? Obviously, can't be part of applet_parent because that won't >persist (unless you make other arrangements for it or portions of it). > >[I.e., applet_parent can't define an allocation handler in its body.] > >Now, migrate applet to another node. Another (shared) instance of >memory_manager_module likely already exists on that node. If the >allocation handler was a "stock" handler (contained in the >memory_manager_module), can you simply copy the state over to >the new node? > >What if the allocation handler was resident in some OTHER module >on the original node? Do you migrate that other module, as well? >Or, instantiate another copy if the module has outstanding references >on the first node?? For one "applet" (process / security context) to allocate memory on behalf of another, that memory has to be supplied by something that exists outside of either of them. Thus the system memory manager can't be a dynamic "module" loaded locally into a process. Its *interface* could be, but not the manager itself. Leaving that aside. Ok, so you have a block of system memory, and you created one (or more) heaps having certain properties within the block - heaps which are "managed" locally in the process by code from a dynamic module. And now you want to rehost the process. 1) The obvious: store the heap properties together with the heap so that "copying" (serializing) the contents of the memory block to another host replicates the heap state for the equivalent "manager" module on that system. 2) Rehosting a running process necessarily requires checkpointing, copying or reconstructing its entire dynamic state: heaps, stacks, globals, loaded modules, kernel [meta]data, etc. - far more than one memory block. Every piece of distributed state must be identifiable as belonging to the process - regardless of what "module" may have created or is currently managing it. You have to copy all data belonging to the process, so having heap properties stored separately from the heap itself is not a problem. [Unless the systems involved do not have MMU/VMM such that the process address space can be faithfully reconstructed ... but if you really need to do it, it can be done using a software based virtual machine solution.] >Or, do you require declarations as to what *can* migrate and what >is bound in place? Then, contend with the consequences of >cascaded dependencies? > >The "consistent" solution is to make all of these first class >objects and let the OS manage them and their locations/instantiations. >But, that adds considerably to the cost of *using* them. Yes, that certainly is the ridiculous, overcompensating solution. The "consistent" solution is the one done already by any decent OS: keep a record of all the kernel structures and memory pages belonging to the process. Program objects then are just ordinary data in memory blocks "owned" by the process, or by the kernel on behalf of the process. Ensuring that a rehosted program will work is another matter. But as I understand (???) your system, there is location transparent naming and equivalence[*] hosts will expose the same set of service APIs. [*] capable of running the same applications. >Its a lot easier to deal with implementations where "everything" is >a cohesive "blob" instead of being able to slice-and-dice it at >run-time and redistribute it based on evolving workloads, resources, >etc. You can somewhat mitigate the problem by making large allocations piece wise: abusing VMM to make them appear address contiguous. E.g., if a process asks for 100MB, reserve the address space but instantiate just a few megabytes at a time, on demand as the process touches unbacked addresses. [Because (I assume) you don't want to overcommit memory, you need to reserve requested address space both locally in the process, and globally so that other processes can't accidentally grab it.] You need to reconstruct the address space on the new host, but you need only copy as much physical memory as the process is actually using. Aside: one of the nice features of "moving" GC is that it compacts live data into a (usually) smaller address range. You don't want to have to copy a large block to get at a much smaller volume of data contained within it. There are a number of tricks that involve using VMM to implement GC. You can't completely avoid data copying if you want to compact the live data, but using VMM techniques together with clustered BiBoP allocation, you can reduce the amount of copying to bare minimum. It also is possible to use VMM and just the tracing step of GC to identify the actual set of pages containing live data ... you don't need to copy any more than that even if the process has much more memory instantiated. And regardless of whether you (would want to) use GC in normal operation, if it is supported by the language/runtime, a compacting collection executed before rehosting a process would minimize both what needs to be copied and the physical memory that must be instantiated on the target. >Last 25# of oranges... yippee! 10 inches of snow, followed by sleet, followed by freezing temps. The crust is ~3/4 of an inch thick - my 185 lbs can walk on top of it. Had to use the heavy coal shovel to break it up so it could be removed. George
[toc] | [prev] | [next] | [standalone]
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2017-03-16 17:23 -0700 |
| Message-ID | <oafa48$kmn$1@dont-email.me> |
| In reply to | #26100 |
Hi George, On 3/15/2017 6:05 PM, George Neuner wrote: >> Assume applet, memory_manager_module, applet_parent. >> >> Applet_parent (for example) instantiates a memory object for applet. Part >> of that requires binding parameters to that memory object (size, granularity, >> allocation_policy, release_policy, etc.). Applet_parent need not persist >> beyond this point (assume its job is done). >> >> Where do the bound parameters get stored? In the applet? In the memory >> object? Obviously, can't be part of applet_parent because that won't >> persist (unless you make other arrangements for it or portions of it). >> >> [I.e., applet_parent can't define an allocation handler in its body.] >> >> Now, migrate applet to another node. Another (shared) instance of >> memory_manager_module likely already exists on that node. If the >> allocation handler was a "stock" handler (contained in the >> memory_manager_module), can you simply copy the state over to >> the new node? >> >> What if the allocation handler was resident in some OTHER module >> on the original node? Do you migrate that other module, as well? >> Or, instantiate another copy if the module has outstanding references >> on the first node?? > > For one "applet" (process / security context) to allocate memory on > behalf of another, that memory has to be supplied by something that > exists outside of either of them. Actually, the memory can exist anywhere: in some global space, in the "allocating entity" or in the "client entity". As can the code that does the actual allocating (at the behest of the allocator). [Consider client environment not supporting arbitrary pointers. Unless something *gives* you a reference/handle to a piece of memory, there's no way you can access its contents (short of an exploit). E.g., no way you can go looking up/down the stack even though you know there *is* a stack supporting the implementation language] > Thus the system memory manager can't be a dynamic "module" loaded > locally into a process. Its *interface* could be, but not the manager > itself. > > Leaving that aside. > > Ok, so you have a block of system memory, and you created one (or > more) heaps having certain properties within the block - heaps which > are "managed" locally in the process by code from a dynamic module. > And now you want to rehost the process. Objects (heaps in this example) are managed by <something>. That something may be the entity "owning" the object, a proxy or the OS acting as the "proxy of last resort". The actions involved in management may reside in the OS, a service (created by <something> -- including the client in question!) or the client. At the direct or *implied* request of the client, etc. > 1) The obvious: store the heap properties together with the heap so > that "copying" (serializing) the contents of the memory block to > another host replicates the heap state for the equivalent "manager" > module on that system. This potentially exposes those "trusted" parameters to abuse by the entiti(es) having access to the underlying object. E.g., if client can tweek the "heapsize" parameter stored there, then it can potentially trick the "manager" on the remote system to instantiate a larger heap than it was originally allowed to create (unless you rerun the "create_heap" request as part of the migration effort) Likewise, if some of the "properties" reside *in* the client (e.g., the algorithm by which allocations will be performed -- under the supervision of some remote/system service that actually *does* them), then you have to drag that/those things along with you. [Imagine these things reside in a *third* entity] > 2) Rehosting a running process necessarily requires checkpointing, > copying or reconstructing its entire dynamic state: heaps, stacks, > globals, loaded modules, kernel [meta]data, etc. - far more than one > memory block. > > Every piece of distributed state must be identifiable as belonging to > the process - regardless of what "module" may have created or is > currently managing it. > > You have to copy all data belonging to the process, so having heap > properties stored separately from the heap itself is not a problem. In my "portable" case, this is handled by moving the object handles. The system can then opt to "optimize" execution by (later?) moving the actual object instances (to minimize communication delays or take better advantage of processing power on some particular node -- which may differ from the source or destination nodes) But, you can (currently!) create "non-portable" objects. And, objects that can't (easily) be shared. (e.g., for a "conventional" heap, you can let the default policies available in a "heap manager" govern the way the heap operates. *But*, you can't take advantage of any "enhanced capabilities" in much the same way that you can't in a more conventional process container, etc.) The problem with this (apparently arbitrary) distinction between portable/shareable objects and "legacy" variety implementations is that the developer has to explicitly decide what can be migrated, shared, and how (because the developer has to take extra steps at compile and link time to make those capabilities available). So, where these sorts of bindings get stored (and how they get tracked) varies based on this "other" information that the developer supplies. [You don't want to have to change the sources to support these abilities as you want to be able to experiment with letting them migrate, be shared, etc. without having to undertake a rewrite/refactoring. And, I can't see an easy way for the build tools to determine that without explicit directives from the developer (hence my related comments in this thread)] > [Unless the systems involved do not have MMU/VMM such that the process > address space can be faithfully reconstructed ... but if you really > need to do it, it can be done using a software based virtual machine > solution.] > >> Or, do you require declarations as to what *can* migrate and what >> is bound in place? Then, contend with the consequences of >> cascaded dependencies? >> >> The "consistent" solution is to make all of these first class >> objects and let the OS manage them and their locations/instantiations. >> But, that adds considerably to the cost of *using* them. > > Yes, that certainly is the ridiculous, overcompensating solution. > > The "consistent" solution is the one done already by any decent OS: > keep a record of all the kernel structures and memory pages belonging > to the process. > > Program objects then are just ordinary data in memory blocks "owned" > by the process, or by the kernel on behalf of the process. But that ignores all of the "other" dependencies that can be in play at any given time (i.e., my "solution" being these portable handles that the OS tracks for the application/developer). So, <something> knows that there is an aspect of the current applet's instantiation that relies upon something else *or* that is relied upon BY something else. [My approach lets the run-time know of these possible external references by the presence of object handles held -- or exported -- by the applet in question. *Absent* these, the task is just a block of memory and "processor state" that can be copied anywhere and "resumed".] > Ensuring that a rehosted program will work is another matter. But as > I understand (???) your system, there is location transparent naming > and equivalence[*] hosts will expose the same set of service APIs. > > [*] capable of running the same applications. > >> Its a lot easier to deal with implementations where "everything" is >> a cohesive "blob" instead of being able to slice-and-dice it at >> run-time and redistribute it based on evolving workloads, resources, >> etc. > > You can somewhat mitigate the problem by making large allocations > piece wise: abusing VMM to make them appear address contiguous. > > E.g., if a process asks for 100MB, reserve the address space but > instantiate just a few megabytes at a time, on demand as the process > touches unbacked addresses. > > [Because (I assume) you don't want to overcommit memory, you need to > reserve requested address space both locally in the process, and > globally so that other processes can't accidentally grab it.] Resource constraints vary with the resource and the consumer/provider. You can overcommit many resources because many "jobs" don't exploit their worst-case resource needs. But, you do so at the risk of delaying the availability of those resources to the job in question. Or, some other job interested in those resources. [E.g., if your job is diarising a previously recorded telephone conversation, it can *probably* afford to block waiting on memory that it needs to complete that task -- perhaps even indefinitely! It shouldn't be prevented from starting just because the MAX memory that MIGHT be required isn't presently available. Nor should all of that memory be wired down for its benefit without knowing that it *will* need it -- or, when!] > You need to reconstruct the address space on the new host, but you > need only copy as much physical memory as the process is actually > using. > > Aside: one of the nice features of "moving" GC is that it compacts > live data into a (usually) smaller address range. You don't want to > have to copy a large block to get at a much smaller volume of data > contained within it. But you need a way of tying a "handle" to the data involved -- and, later, resolving that reference. > There are a number of tricks that involve using VMM to implement GC. > You can't completely avoid data copying if you want to compact the > live data, but using VMM techniques together with clustered BiBoP > allocation, you can reduce the amount of copying to bare minimum. > > It also is possible to use VMM and just the tracing step of GC to > identify the actual set of pages containing live data ... you don't > need to copy any more than that even if the process has much more > memory instantiated. > > And regardless of whether you (would want to) use GC in normal > operation, if it is supported by the language/runtime, a compacting > collection executed before rehosting a process would minimize both > what needs to be copied and the physical memory that must be > instantiated on the target. > >> Last 25# of oranges... yippee! > > 10 inches of snow, followed by sleet, followed by freezing temps. The Yeah, caught a glimpse of that on the national news... > crust is ~3/4 of an inch thick - my 185 lbs can walk on top of it. Had > to use the heavy coal shovel to break it up so it could be removed. 90's, here. I think 95 expected this weekend. Being outdoors is a chore as it's either too warm during the daylight hours *or* the air too "smelly" (everything is in bloom) in the cooler hours when the breeze has died down. I'd (personally) appreciate a cold spell (but not *now* as the new crop of fruit is setting in place) to slow the growth process. I suspect it will be a bad year for insects! [Normally, we don't see 95 until May; and 90 until mid-April]
[toc] | [prev] | [next] | [standalone]
| From | George Neuner <gneuner2@comcast.net> |
|---|---|
| Date | 2017-03-17 23:59 -0400 |
| Message-ID | <bp3pccl1jtv66f4jfb036lv1cnkdn8quuq@4ax.com> |
| In reply to | #26103 |
On Thu, 16 Mar 2017 17:23:11 -0700, Don Y <blockedofcourse@foo.invalid> wrote: >On 3/15/2017 6:05 PM, George Neuner wrote: > >> For one "applet" (process / security context) to allocate memory on >> behalf of another, that memory has to be supplied by something that >> exists outside of either of them. > >Actually, the memory can exist anywhere: in some global space, in >the "allocating entity" or in the "client entity". As can the code >that does the actual allocating (at the behest of the allocator). > >[Consider client environment not supporting arbitrary pointers. >Unless something *gives* you a reference/handle to a piece of memory, >there's no way you can access its contents (short of an exploit). >E.g., no way you can go looking up/down the stack even though you >know there *is* a stack supporting the implementation language] Moving the goal-post again. From previous discussions, I know your system works similarly to Mach ... and while I don't know your system, I *do* know Mach. You're conflating things when you talk about "allocating entities" (aka Mach servers), and introducing complexity where none should exist. E.g., The Mach memory manager (MM) grabs most of the RAM for itself and creates a heap from which it parcels blocks to other processes. An user allocated memory block physically exists IN THE MM's HEAP, but semantically it *belongs* to the user process: it is mapped into the address space of the user process, else that process could not use it. When you go to rehost this process, you need to consult the MM only so far as to know the attributes of the block. The block itself is IN THE USER PROCESS, and what it contains has to be replicated at the same VMM "linear" address in the new process on the new host. The MM itself need not be involved in the replication. In fact you probably would not want it to be because you only want to copy live data and not the entire address space of the rehosting process. The access/call and identity mechanisms: i.e. the "handles" by which the MM is called, and by which the block is known to the MM - mostly are beside the point. >> Ok, so you have a block of system memory, and you created one (or >> more) heaps having certain properties within the block - heaps which >> are "managed" locally in the process by code from a dynamic module. >> And now you want to rehost the process. > >Objects (heaps in this example) are managed by <something>. That >something may be the entity "owning" the object, a proxy or the >OS acting as the "proxy of last resort". Now you are getting completely away from the "memory" question you initially asked and into objects managed by servers on behalf of clients. Again with the goal post. An IN-PROCESS server allocates from memory already owned by the client process. The client DOES NOT OWN an object which it can't touch directly, and which it can only affect via RPC to another process. Replicating the memory state of the process on the new host automagically handles the case of an in-process server. It also replicates client side state for anything controlled on its behalf by an out-of-process server. The state of an out-of-process server does not factor directly into rehosting a client. All the server needs to know (if even this much) is that the client's "session" is to be suspended pending reconnection rather than terminated. >The actions involved in management may reside in the OS, a service >(created by <something> -- including the client in question!) or the >client. At the direct or *implied* request of the client, etc. You are failing to realize that it doesn't matter who manages the data: it matters only who *owns* the data. >Likewise, if some of the "properties" reside *in* the client (e.g., >the algorithm by which allocations will be performed -- under the >supervision of some remote/system service that actually *does* them), >then you have to drag that/those things along with you. Or simply provide them in the new setting. E.g., there is no reason to copy in-memory code if the new host has access to it via a file system. If the new host has a different architecture, you can't copy the code anyway and would have to provide checkpoint mechanisms whereby the *program* (not "process") can be stopped and restarted cleanly. >> 2) Rehosting a running process necessarily requires checkpointing, >> copying or reconstructing its entire dynamic state: heaps, stacks, >> globals, loaded modules, kernel [meta]data, etc. - far more than one >> memory block. >> >> Every piece of distributed state must be identifiable as belonging to >> the process - regardless of what "module" may have created or is >> currently managing it. >> >> You have to copy all data belonging to the process, so having heap >> properties stored separately from the heap itself is not a problem. > >In my "portable" case, this is handled by moving the object handles. >The system can then opt to "optimize" execution by (later?) moving the >actual object instances (to minimize communication delays or take >better advantage of processing power on some particular node -- which >may differ from the source or destination nodes) > >But, you can (currently!) create "non-portable" objects. So they are lost. So what? You knew they were non-portable when you created them. >And, objects >that can't (easily) be shared. (e.g., for a "conventional" heap, you >can let the default policies available in a "heap manager" govern >the way the heap operates. *But*, you can't take advantage of any >"enhanced capabilities" in much the same way that you can't in a >more conventional process container, etc.) There's no reason code can't use "enhanced capabilities" - as long as the result is the same (for some definition). E.g., if you can ignore timing, then debug code is interchangeable with optimized code. >The problem with this (apparently arbitrary) distinction between >portable/shareable objects and "legacy" variety implementations >is that the developer has to explicitly decide what can be migrated, >shared, and how (because the developer has to take extra steps at >compile and link time to make those capabilities available). It is completely arbitrary, and IMO unnecessary. If there is something - a device possibly - that only exists on one host, then the module that interacts DIRECTLY with that <something> is not portable. Anything objects it creates as a server on behalf of a client likewise are not portable. So if this module is to be used by portable applications, it should NOT be implemented as an in-process server. Q.E.D. >So, where these sorts of bindings get stored (and how they get >tracked) varies based on this "other" information that the developer >supplies. This is where you and I have parted company in the past. You like the kernel capability approach whereas I prefer the user capability approach ... precisely BECAUSE user capabilities avoid many of these semantic problems by virtue of being ordinary data. Everything other than kernel level process and address space control can be done in user space, and rehosting then is simply a matter of replicating the memory state [and reconstructing the kernel state]. >>> The "consistent" solution is to make all of these first class >>> objects and let the OS manage them and their locations/instantiations. >>> But, that adds considerably to the cost of *using* them. >> >> Yes, that certainly is the ridiculous, overcompensating solution. >> >> The "consistent" solution is the one done already by any decent OS: >> keep a record of all the kernel structures and memory pages belonging >> to the process. >> >> Program objects then are just ordinary data in memory blocks "owned" >> by the process, or by the kernel on behalf of the process. > >But that ignores all of the "other" dependencies that can be in >play at any given time (i.e., my "solution" being these portable >handles that the OS tracks for the application/developer). So, ><something> knows that there is an aspect of the current applet's >instantiation that relies upon something else *or* that is relied >upon BY something else. No it doesn't. You have created a whole set of artificial dependencies through your (ab)use of kernel capabilities. Nothing in your system is - or even can be - self contained. >[My approach lets the run-time know of these possible external >references by the presence of object handles held -- or exported -- by >the applet in question. *Absent* these, the task is just a block of >memory and "processor state" that can be copied anywhere and "resumed".] But by doing that, you've inserted the operating system into all kinds of things where IMO it has no business being. You've created the MCC from "Tron". >> You can somewhat mitigate [rehosting] by making large allocations >> piece wise: abusing VMM to make them appear address contiguous. >> >> E.g., if a process asks for 100MB, reserve the address space but >> instantiate just a few megabytes at a time, on demand as the process >> touches unbacked addresses. >> >> [Because (I assume) you don't want to overcommit memory, you need to >> reserve requested address space both locally in the process, and >> globally so that other processes can't accidentally grab it.] > >Resource constraints vary with the resource and the consumer/provider. >You can overcommit many resources because many "jobs" don't exploit >their worst-case resource needs. > >But, you do so at the risk of delaying the availability of those resources >to the job in question. Or, some other job interested in those resources. No, you are misunderstanding. The process owns the space ... but in the scenario I presented, the physical memory backing the space is mapped on demand - which gives the OS more visibility into the process if/when it needs to rehost the process. Remember that there are 3 forms of VMM addresses: "linear", "virtual" and "physical". - "Linear" space is what is seen by an application. - "Physical" space is the actual RAM as seen by the system. - "Virtual" space is what ties the other 2 together. In a system using MMU only for process isolation, the "virtual" and "physical" spaces would be identical - mapped 1:1. Overcommitting memory is only possible when "virtual" space > "physical" space. When an application requests 100MB [e.g., (s)brk] from the OS, then [assuming the request can be granted] the OS adds 100MB to the linear space of the process, and *reserves* 100MB in the virtual space of the system. Now the reserved space belongs to the requesting process - no other process can take it. However, the OS can choose whether or not to immediately map the new linear space to physical space. By mapping physical space on demand as it is touched by the process, the OS has a much better picture of the *actual* memory use of the process. When the time comes to replicate the process's memory on another host, only mapped physical space needs to be copied - not the entire virtual address space reservation. George
[toc] | [prev] | [next] | [standalone]
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2017-03-17 23:25 -0700 |
| Message-ID | <oaijmu$8f3$1@dont-email.me> |
| In reply to | #26128 |
On 3/17/2017 8:59 PM, George Neuner wrote:
> On Thu, 16 Mar 2017 17:23:11 -0700, Don Y
> <blockedofcourse@foo.invalid> wrote:
>
>> On 3/15/2017 6:05 PM, George Neuner wrote:
>>
>>> For one "applet" (process / security context) to allocate memory on
>>> behalf of another, that memory has to be supplied by something that
>>> exists outside of either of them.
>>
>> Actually, the memory can exist anywhere: in some global space, in
>> the "allocating entity" or in the "client entity". As can the code
>> that does the actual allocating (at the behest of the allocator).
>>
>> [Consider client environment not supporting arbitrary pointers.
>> Unless something *gives* you a reference/handle to a piece of memory,
>> there's no way you can access its contents (short of an exploit).
>> E.g., no way you can go looking up/down the stack even though you
>> know there *is* a stack supporting the implementation language]
>
> Moving the goal-post again.
>
> From previous discussions, I know your system works similarly to Mach
> ... and while I don't know your system, I *do* know Mach. You're
> conflating things when you talk about "allocating entities" (aka Mach
> servers), and introducing complexity where none should exist.
>
> E.g.,
>
> The Mach memory manager (MM) grabs most of the RAM for itself and
> creates a heap from which it parcels blocks to other processes.
>
> An user allocated memory block physically exists IN THE MM's HEAP, but
> semantically it *belongs* to the user process: it is mapped into the
> address space of the user process, else that process could not use it.
> When you go to rehost this process, you need to consult the MM only so
> far as to know the attributes of the block. The block itself is IN
> THE USER PROCESS, and what it contains has to be replicated at the
> same VMM "linear" address in the new process on the new host.
>
> The MM itself need not be involved in the replication. In fact you
> probably would not want it to be because you only want to copy live
> data and not the entire address space of the rehosting process.
>
> The access/call and identity mechanisms: i.e. the "handles" by which
> the MM is called, and by which the block is known to the MM - mostly
> are beside the point.
(sigh) You're fixating on AN EXAMPLE of an OBJECT that can need things
like "allocation_policy" bound to it. (rewind the thread to see where I
introduced a "heap"). Reread the thread with "potato" substituting
for all instances of "heap". And, "mashing_policy" for "allocation_policy",
"weight" for "size", etc.
THEN, lecture me as to how the "OS" owns this object and how *its*
primitives (VMM in your example) apply to its management.
[Should I, in the future, talk STRICTLY in terms of abstractions and avoid
ANY pleas to reify those concepts? For fear that a reader will seize on an
example as representative of the entire universe of possibilities? How
much "imagination" should I expect of readers?]
>>> Ok, so you have a block of system memory, and you created one (or
>>> more) heaps having certain properties within the block - heaps which
>>> are "managed" locally in the process by code from a dynamic module.
>>> And now you want to rehost the process.
>>
>> Objects (heaps in this example) are managed by <something>. That
>> something may be the entity "owning" the object, a proxy or the
>> OS acting as the "proxy of last resort".
>
> Now you are getting completely away from the "memory" question you
> initially asked and into objects managed by servers on behalf of
> clients.
>
> Again with the goal post.
"Imagination". Remember, we're talking about potatoes...
> An IN-PROCESS server allocates from memory already owned by the client
> process. The client DOES NOT OWN an object which it can't touch
> directly, and which it can only affect via RPC to another process.
An in-process server allocates RESOURCES from resources that it
can REFERENCE from ANY OTHER SERVICE (which may be the OS or any
other object server to which the "in-process server" has rights.
The potato server can call upon a "masher" supplied by a "mashing service"
(and the poor OS is completely clueless as to the concept of "mashing").
The handle for the object that it then instantiates is all the client
ever needs to manipulate this particular potato.
The potato server can migrate to another node. The client referencing
the potato can migrate. The client can give the potato to someone else.
The potato server can pass management of the potato to a different
instance of a *compatible* potato server.
The potato can be "serialized" to a persistent medium and later
reconstituted ("instant potatoes"! :> )
The application need be aware of none of these things.
> Replicating the memory state of the process on the new host
> automagically handles the case of an in-process server. It also
> replicates client side state for anything controlled on its behalf by
> an out-of-process server.
>
> The state of an out-of-process server does not factor directly into
> rehosting a client. All the server needs to know (if even this much)
> is that the client's "session" is to be suspended pending reconnection
> rather than terminated.
Replicating "memory" doesn't replicate the state of the machine because
other machines are involved.
Copy the memory image of *my* PC's "shell", at this moment, onto your PC.
Now, tell me which files I have open. Which TCP connections are live. etc.
The "state" is spread around in several places *besides* the process
itself. Including the mail server at the other end of this SMTP connection
(when I click SEND).
[I.e., unless the passphrase for my SMTP server persists somewhere in
this process's memory, your copy of it's memory won't help you access
that connection that exists *now*]
>> The actions involved in management may reside in the OS, a service
>> (created by <something> -- including the client in question!) or the
>> client. At the direct or *implied* request of the client, etc.
>
> You are failing to realize that it doesn't matter who manages the
> data: it matters only who *owns* the data.
>
>> Likewise, if some of the "properties" reside *in* the client (e.g.,
>> the algorithm by which allocations will be performed -- under the
>> supervision of some remote/system service that actually *does* them),
>> then you have to drag that/those things along with you.
>
> Or simply provide them in the new setting. E.g., there is no reason
> to copy in-memory code if the new host has access to it via a file
> system. If the new host has a different architecture, you can't copy
> the code anyway and would have to provide checkpoint mechanisms
> whereby the *program* (not "process") can be stopped and restarted
> cleanly.
The applets run in a virtual machine. So, I can pick them up and
move them to another heterogeneous host. (The "servers" are currently
designed for a homogeneous environment. But, the same concepts apply)
>>> 2) Rehosting a running process necessarily requires checkpointing,
>>> copying or reconstructing its entire dynamic state: heaps, stacks,
>>> globals, loaded modules, kernel [meta]data, etc. - far more than one
>>> memory block.
>>>
>>> Every piece of distributed state must be identifiable as belonging to
>>> the process - regardless of what "module" may have created or is
>>> currently managing it.
>>>
>>> You have to copy all data belonging to the process, so having heap
>>> properties stored separately from the heap itself is not a problem.
>>
>> In my "portable" case, this is handled by moving the object handles.
>> The system can then opt to "optimize" execution by (later?) moving the
>> actual object instances (to minimize communication delays or take
>> better advantage of processing power on some particular node -- which
>> may differ from the source or destination nodes)
>>
>> But, you can (currently!) create "non-portable" objects.
>
> So they are lost. So what? You knew they were non-portable when you
> created them.
But the apps that contain them can be portable! You might have
*chosen* to make a non-portable potato (by implementing a potato
server as one of the threads in your process). The potato is
bound to the process. But, the process isn't (necessarily) bound
to the *node* (or process group, etc.).
E.g., I can create a mutex to allow the local threads to cooperatively
access <something> (maybe just a 4 byte counter). As long as the threads
remain bound in that process container AND the counter (or it's accessor)
AND the mutex, the whole entity (process) can be moved at will.
If some other process/agent needs to access that mutex, then it
needs a handle that is visible from outside of the local process.
If that other process can migrate to a different node, then it
can retain the ability to interact with the mutex in this first
process still residing on the original node.
So, you end up with very different use/performance cases:
- two or more colocated threads accessing the object
- two or more LOCAL *processes* accessing the object
- two or more REMOTE processes accessing the object
[Note, of course, that any other *object* can also indirectly be accessing
that object/potato]
In the first case, you can conceivably optimize the access to that of
a conventional mutex (in this example -- remember potatoes!) implementation.
In the second case, one process might be able to access the object
"from within" (a direct call to the functions that implement the
object) while another process (same node) accesses it "from without".
In the third case, one may be from within, or without, or "remote".
I don't consider it a "good idea" to be forced to make these "mobility"
decisions at compile time. For the same reason that you don't decide where in
memory a piece of code should execute. Or, whether it WILL be optimized by
the compiler, or not (compiler command line options aren't reflected in
the sources)
Just like I don't define which functions/methods should be accessible
remotely (let the IDL compiler address that based on what I tell *it* to do)
>> And, objects
>> that can't (easily) be shared. (e.g., for a "conventional" heap, you
>> can let the default policies available in a "heap manager" govern
>> the way the heap operates. *But*, you can't take advantage of any
>> "enhanced capabilities" in much the same way that you can't in a
>> more conventional process container, etc.)
>
> There's no reason code can't use "enhanced capabilities" - as long as
> the result is the same (for some definition). E.g., if you can ignore
> timing, then debug code is interchangeable with optimized code.
That's been my goal! But, it means the code has to reconfigure itself AT
RUNTIME. Because the compiler doesn't know where an instance of a potato
will reside at *compile* time.
>> The problem with this (apparently arbitrary) distinction between
>> portable/shareable objects and "legacy" variety implementations
>> is that the developer has to explicitly decide what can be migrated,
>> shared, and how (because the developer has to take extra steps at
>> compile and link time to make those capabilities available).
>
> It is completely arbitrary, and IMO unnecessary.
>
> If there is something - a device possibly - that only exists on one
> host, then the module that interacts DIRECTLY with that <something> is
> not portable. Anything objects it creates as a server on behalf of a
> client likewise are not portable.
No. A "something" that CAN only exist on one host (node) is not portable.
A "something" that HAPPENS TO only exist on one node but CAN be migrated
to another node doesn't suffer those same restrictions.
I put a thin layer immediately above the hardware devices on each node
(think of it as a sort of HAL). At the top interface of this layer, I
make a deliberate choice as to whether or not that interface shall be
exportable across nodes.
For example, there is little need to bind anything that uses the
"irrigation valve" hardware (HAL layer) to the node that actually
has the hammer drivers for those valve actuators. They are inherently
slow and largely sacrificial. So, that interface is exported. The
"irrigation control task" can reside anywhere in the system -- including
nowhere (zombie).
OTOH, the interface to each of the video cameras is relatively high
bandwidth. The task(s) that directly interact with them *should*
reside on those nodes (*if* you assume the camera output will see
much consumption). So, those interfaces are not exported.
OToOH, the interface to the *compressed* video streams provided by those
tasks *can* be exported. Their bandwidth requirements might be
considerably less than the "raw" video coming off each individual
camera. So, a "motion detection" task can run on the IRRIGATION
NODE. Or, mode to the LAUNDRY NODE if/when the irrigation node
is taxed with some other responsibility (perhaps it is taken out of
service as the valves -- and the proxies that run them -- are no longer
needed)
I.e., you gain the most flexibility by making everything exportable.
But, that means you either incur the costs associated with that
overhead -- even for the "local" case (e.g., motion detection task
running on the same node as the camera!) -- *or* do run-time
optimization of the communication path so that the remote costs
are only incurred when necessary (and factored into the workload
manager's reassignment of tasks to nodes)
> So if this module is to be used by portable applications, it should
> NOT be implemented as an in-process server. Q.E.D.
>
>> So, where these sorts of bindings get stored (and how they get
>> tracked) varies based on this "other" information that the developer
>> supplies.
>
> This is where you and I have parted company in the past. You like the
> kernel capability approach whereas I prefer the user capability
> approach ... precisely BECAUSE user capabilities avoid many of these
> semantic problems by virtue of being ordinary data.
The value of the in-kernel capability is that inherently tracks
references. If the capability is "just data", then you have to
add a second mechanism to track *where* every copy of the data
happens to reside! There's no way for the system to know that
you've just handed a COPY of a capability to something, deleted
YOUR copy, then retrieved a copy of that copy! (i.e., the
"reference" is still "live", even though it looks like you
deleted it). The "system" has no idea who has outstanding
references (handles/capabilities) nor how MANY.
By contrast, if the kernel manages the capabilities, then they
can't (by definition) be copied or transferred or deleted
without the kernel being an active part of that operation.
The kernel (and the manager for that particular capability)
can mediate how each is used ("No, you can't give that
capability away! If you try to do so, I will abend your
process and/or just silently DELETE the capability and let
you and the other party wonder why that operation that
you intended to have performed *isn't*!")
"Is anyone using this video camera? No, there are no active
capabilities in existence ANYWHERE in the system so we can
shut it down. Is anyone (process) using the node that hosts
that camera? No, so we can shut down the node, as well!"
> Everything other than kernel level process and address space control
> can be done in user space, and rehosting then is simply a matter of
> replicating the memory state [and reconstructing the kernel state].
>
>>>> The "consistent" solution is to make all of these first class
>>>> objects and let the OS manage them and their locations/instantiations.
>>>> But, that adds considerably to the cost of *using* them.
>>>
>>> Yes, that certainly is the ridiculous, overcompensating solution.
>>>
>>> The "consistent" solution is the one done already by any decent OS:
>>> keep a record of all the kernel structures and memory pages belonging
>>> to the process.
>>>
>>> Program objects then are just ordinary data in memory blocks "owned"
>>> by the process, or by the kernel on behalf of the process.
>>
>> But that ignores all of the "other" dependencies that can be in
>> play at any given time (i.e., my "solution" being these portable
>> handles that the OS tracks for the application/developer). So,
>> <something> knows that there is an aspect of the current applet's
>> instantiation that relies upon something else *or* that is relied
>> upon BY something else.
>
> No it doesn't.
>
> You have created a whole set of artificial dependencies through your
> (ab)use of kernel capabilities. Nothing in your system is - or even
> can be - self contained.
A process need never export a handle to an object that it creates
and manages -- for itself. The OS needn't even be aware that the
object exists. Does the OS know that I have a 4 byte counter
*here*? Or here?
OTOH, if something *else* DEPENDS on that counter (potato), then
you want that dependency to be visible to the system. You don't
want the potato to silently disappear without the dependent entity
knowing about it, *now* (not at some later date when it expects it
to STILL exist).
Likewise, if it "moves", you don't want the dependent entity to have to
scramble to *find* it when it needs it -- for much the same reason as
above.
>> [My approach lets the run-time know of these possible external
>> references by the presence of object handles held -- or exported -- by
>> the applet in question. *Absent* these, the task is just a block of
>> memory and "processor state" that can be copied anywhere and "resumed".]
>
> But by doing that, you've inserted the operating system into all kinds
> of things where IMO it has no business being. You've created the MCC
> from "Tron".
An OS manages resources. I consider dependencies to be resources.
And, the permissions associated with them.
I don't want a user-level task to have to keep track of where the
things that it needs happen to be, *now* -- or if they even happen
to EXIST!
Nor do I want to have to require the developer to never try to do
something with a particular thing/object (potato) that it *shouldn't*.
Just like the OS shouldn't let process A peek into process B's memory.
Or, access unmapped portions of memory, etc.
You'd not advocate allowing a user to "add 3" to an arbitrary string.
You'd expect the *compiler* to enforce the rules of the language on
his *use* of the language.
I expect the OS to enforce the rules of the *system* on the users
of that system in much the same way!
>>> You can somewhat mitigate [rehosting] by making large allocations
>>> piece wise: abusing VMM to make them appear address contiguous.
>>>
>>> E.g., if a process asks for 100MB, reserve the address space but
>>> instantiate just a few megabytes at a time, on demand as the process
>>> touches unbacked addresses.
>>>
>>> [Because (I assume) you don't want to overcommit memory, you need to
>>> reserve requested address space both locally in the process, and
>>> globally so that other processes can't accidentally grab it.]
>>
>> Resource constraints vary with the resource and the consumer/provider.
>> You can overcommit many resources because many "jobs" don't exploit
>> their worst-case resource needs.
>>
>> But, you do so at the risk of delaying the availability of those resources
>> to the job in question. Or, some other job interested in those resources.
>
> No, you are misunderstanding. The process owns the space ... but in
> the scenario I presented, the physical memory backing the space is
> mapped on demand - which gives the OS more visibility into the process
> if/when it needs to rehost the process.
The "handles" give the workload manager (via the OS) insight into how and
where it can/should rehost the process (or any other object!). It knows
what the process "talks to" because it manages the capabilities FOR that
process. And, it knows the other (more tangible) resources that the
process requires (RAM, MIPS) by examining its current ledger and historical
performance. So, it can balance the costs of "stretching" certain
communication channels (RPC/RMI being more expensive than the local case)
against the costs that some *other* process/object (a process is just a
different sort of object, managed by similar handles) would incur if
*this* process is not migrated.
[You can't come up with an "optimal" solution. But, you can remember the
solutions you've tried and how they performed -- because the OS is
involved in these communications and can measure them! So, you can
avoid solutions that performed poorly in favor of those that performed
*well* or that "have yet to be tried.]
> Remember that there are 3 forms of VMM addresses: "linear", "virtual"
> and "physical".
> - "Linear" space is what is seen by an application.
"Logical" -- it need not be linear/contiguous.
> - "Physical" space is the actual RAM as seen by the system.
> - "Virtual" space is what ties the other 2 together.
And, how does this pertain to potatoes? :>
> In a system using MMU only for process isolation, the "virtual" and
> "physical" spaces would be identical - mapped 1:1. Overcommitting
> memory is only possible when "virtual" space > "physical" space.
Exactly. Recall that there are multiple objects sharing that physical
space on a particular node. The physical space determines the maximum
amount that can be mapped at a given time to the set of objects
currently making demands on it. Those demands can (do) change, over time.
E.g., a process may load many modules and, by executing code *in*
each of them, cause pages to be mapped to instantiate those "opcodes"
JIT. If a portion of a module is never actively referenced, then
the memory required to back it never gets mapped -- not taxing the
physical resources currently available. If a module gets UNloaded,
then all of the mapped pages that were referenced can be released for
other uses (other potatoes). Likewise, for the pages that were
never actually mapped!
E := load Encyclopaedia
...
article := E.lookup(Da Vinci)
...
unload(E)
You wouldn't foolishly load the entire encyclopaedia into physical
(or virtual) memory to do this. Instead, you'd load some accessor
code that could, itself, load PORTIONS of the "index" *when* you
initiated a search. This would require some number of pages of
physical memory to be mapped to hold the resulting "article".
Anything beyond that would be waste!
E := load Encyclopaedia
...
article := E.lookup(Da Vinci)
...
article := E.lookup(R Crumb)
...
unload(E)
Would only require resources proportional to the LARGER of the Da Vinci
and R Crumb entries (the resources held by the first "article" are
freed when the second "article" is defined).
> When an application requests 100MB [e.g., (s)brk] from the OS, then
> [assuming the request can be granted] the OS adds 100MB to the linear
> space of the process, and *reserves* 100MB in the virtual space of the
> system.
No. I don't *impose* that requirement! Because it makes the system
brittle. The application handles this at a different level dictated
by the CM system.
If, in your example, you ultimately *use* all 100MB of that memory
(which may be mmap'd in page-at-a-time increments) and then need
(or, someone else needs) 50MB *more*, the system decides how best to
provide those resources based on parameters from the CM system.
If "more resources" are available (though not necessarily "50" MB),
the system can opt to satisfy the request and let you continue to
mmap (by your usage patterns) additional pages.
If there are NO more resources available *when* you mmap another
page (indirectly by your actions), the system decides how best to
handle that request.
Can I unmap some portion of a loaded module (knowing that I can
always reload that portion of the module AS IF it hadn't *yet*
been referenced/mmap'd)? Can I pageout some "really low" portion
of the stack (on the assumption that it won't be accessed RSN)?
Can I *migrate* some other task off of this node and reclaim the
physical resources that it currently holds? Can I migrate *this*
task to some other node that has more resources available? Can
I bring another node on-line and migrate "something" to it? Can
I rely on some redundant (potato!) server to assume responsibility
for managing some objects that happen to reside on the local system
and, as such, free up the space they occupy *and* the space
required for the code to manage them? Can I *kill* some task
(and associated objects) that are just running opportunistically,
presently (let's do some commercial detection in that OTA TV
broadcast we recorded two hours ago so its ready for viewing before
the user comes home and wants to see it!)? etc.
You don't want to make those decisions at compile time. Because you
don't know what will be active in the system at *run* time! And,
you want to build as much flexibility in how/where resources can
reside so that you have more run-time options. You don't want to
overprovision just to handle a "worst case" that, in practice, will
never occur!
[Repeat this substituting "MIPS" for "memory", etc.]
> Now the reserved space belongs to the requesting process - no other
> process can take it.
See above. If a process (potato) *needs* to be sure that resource
(memory, MIPS, etc.) *is* available, "sitting and waiting", then it
has to place an active "reservation" through the CM system. The
CM system dictates limits for what individual tasks/"users" can
consume.
Reserves are A Bad Thing as they tie up those resources regardless
of what other objects might happen to place similar demands on the
system's resources. What's to stop "everyone" from demanding EVERYTHING
that they *might* need "just in case"?
If, instead, you allow elasticity in how the resources are supplied,
you can afford to loosen the constraints on individual "consumers"
because they have less ACTUAL impact on other consumers.
Joe User (not Bob Developer) could write a sloppy script that
consumes "limitless" resources. Should he be allowed to bring the
system to its knees -- because he is the "owner" of the device?
Similarly, should Bob Developer be allowed free reign at the risk
of compromising core services (implemented by *me*)?
It's a lot easier to ensure your system can meet all its goals if
you provision *for* those goals. But, with an open/expandable system
that sees huge variations between actual and potential use, that
translates into dollars that consumers aren't keen on spending.
(why do you have more VM configured in your PC than physical memory?)
> However, the OS can choose whether or not to immediately map the new
> linear space to physical space. By mapping physical space on demand
> as it is touched by the process, the OS has a much better picture of
> the *actual* memory use of the process.
Yes. Though that can vary with other "input data" that this instance
of the process (potato) sees. E.g., the resources required for that
lookup(R Crumb) are probably far less than the earlier lookup(Da Vinci).
Do you *know* what the user will opt to want to "lookup" when you
instantiate that task? :>
You can either say "you can lookup ANYTHING" -- and provision for it.
Or, "you can look up anything as long as the article doesn't exceed X"
(and impose a disconcerting constraint on the user: "How do I know
how big the article is LIKELY to be?"). Or, you can make a best
effort and trade your response based on your current needs and those
just added by the user.
> When the time comes to replicate the process's memory on another host,
> only mapped physical space needs to be copied - not the entire virtual
> address space reservation.
Again, that assumes the (active/mapped) memory represents the entire
"state" of the process and all of its dependencies. (what about any
network connections -- managed by the OS -- that may be active at the
time? the contents of any NIC buffers AS the process is in transit?
file handles, my "object handles", etc.)
[Think about it. Send me an ISO with an image of your PC's memory
at this point in time. I'll let you include the OS's state in that
image, as well! Will I be able to read this USENET post at exactly
THIS --> . <-- point when I load the image into my machine's memory?
What if you mark the thread as read before the ISO gets loaded on
my machine? Or, if YOUR server expires the article? Or, if
your NNTP server happens to be offline? That's all state that is part
of your "reading this USENET post" activity -- yet, you've not captured
it well enough for me to pick up where you left off! :> ]
[toc] | [prev] | [next] | [standalone]
| From | Tauno Voipio <tauno.voipio@notused.fi.invalid> |
|---|---|
| Date | 2017-03-18 12:05 +0200 |
| Message-ID | <oaj0j5$doq$1@dont-email.me> |
| In reply to | #26129 |
On 18.3.17 08:25, Don Y wrote:
> On 3/17/2017 8:59 PM, George Neuner wrote:
>> On Thu, 16 Mar 2017 17:23:11 -0700, Don Y
>> <blockedofcourse@foo.invalid> wrote:
>>
>>> On 3/15/2017 6:05 PM, George Neuner wrote:
>>>
>>>> For one "applet" (process / security context) to allocate memory on
>>>> behalf of another, that memory has to be supplied by something that
>>>> exists outside of either of them.
>>>
>>> Actually, the memory can exist anywhere: in some global space, in
>>> the "allocating entity" or in the "client entity". As can the code
>>> that does the actual allocating (at the behest of the allocator).
>>>
>>> [Consider client environment not supporting arbitrary pointers.
>>> Unless something *gives* you a reference/handle to a piece of memory,
>>> there's no way you can access its contents (short of an exploit).
>>> E.g., no way you can go looking up/down the stack even though you
>>> know there *is* a stack supporting the implementation language]
>>
>> Moving the goal-post again.
>>
>> From previous discussions, I know your system works similarly to Mach
>> ... and while I don't know your system, I *do* know Mach. You're
>> conflating things when you talk about "allocating entities" (aka Mach
>> servers), and introducing complexity where none should exist.
>>
>> E.g.,
>>
>> The Mach memory manager (MM) grabs most of the RAM for itself and
>> creates a heap from which it parcels blocks to other processes.
>>
>> An user allocated memory block physically exists IN THE MM's HEAP, but
>> semantically it *belongs* to the user process: it is mapped into the
>> address space of the user process, else that process could not use it.
>> When you go to rehost this process, you need to consult the MM only so
>> far as to know the attributes of the block. The block itself is IN
>> THE USER PROCESS, and what it contains has to be replicated at the
>> same VMM "linear" address in the new process on the new host.
>>
>> The MM itself need not be involved in the replication. In fact you
>> probably would not want it to be because you only want to copy live
>> data and not the entire address space of the rehosting process.
>>
>> The access/call and identity mechanisms: i.e. the "handles" by which
>> the MM is called, and by which the block is known to the MM - mostly
>> are beside the point.
>
> (sigh) You're fixating on AN EXAMPLE of an OBJECT that can need things
> like "allocation_policy" bound to it. (rewind the thread to see where I
> introduced a "heap"). Reread the thread with "potato" substituting
> for all instances of "heap". And, "mashing_policy" for
> "allocation_policy",
> "weight" for "size", etc.
>
> THEN, lecture me as to how the "OS" owns this object and how *its*
> primitives (VMM in your example) apply to its management.
>
> [Should I, in the future, talk STRICTLY in terms of abstractions and avoid
> ANY pleas to reify those concepts? For fear that a reader will seize on an
> example as representative of the entire universe of possibilities? How
> much "imagination" should I expect of readers?]
>
>>>> Ok, so you have a block of system memory, and you created one (or
>>>> more) heaps having certain properties within the block - heaps which
>>>> are "managed" locally in the process by code from a dynamic module.
>>>> And now you want to rehost the process.
>>>
>>> Objects (heaps in this example) are managed by <something>. That
>>> something may be the entity "owning" the object, a proxy or the
>>> OS acting as the "proxy of last resort".
>>
>> Now you are getting completely away from the "memory" question you
>> initially asked and into objects managed by servers on behalf of
>> clients.
>>
>> Again with the goal post.
>
> "Imagination". Remember, we're talking about potatoes...
>
>> An IN-PROCESS server allocates from memory already owned by the client
>> process. The client DOES NOT OWN an object which it can't touch
>> directly, and which it can only affect via RPC to another process.
>
> An in-process server allocates RESOURCES from resources that it
> can REFERENCE from ANY OTHER SERVICE (which may be the OS or any
> other object server to which the "in-process server" has rights.
>
> The potato server can call upon a "masher" supplied by a "mashing service"
> (and the poor OS is completely clueless as to the concept of "mashing").
> The handle for the object that it then instantiates is all the client
> ever needs to manipulate this particular potato.
>
> The potato server can migrate to another node. The client referencing
> the potato can migrate. The client can give the potato to someone else.
> The potato server can pass management of the potato to a different
> instance of a *compatible* potato server.
>
> The potato can be "serialized" to a persistent medium and later
> reconstituted ("instant potatoes"! :> )
>
> The application need be aware of none of these things.
>
>> Replicating the memory state of the process on the new host
>> automagically handles the case of an in-process server. It also
>> replicates client side state for anything controlled on its behalf by
>> an out-of-process server.
>>
>> The state of an out-of-process server does not factor directly into
>> rehosting a client. All the server needs to know (if even this much)
>> is that the client's "session" is to be suspended pending reconnection
>> rather than terminated.
>
> Replicating "memory" doesn't replicate the state of the machine because
> other machines are involved.
>
> Copy the memory image of *my* PC's "shell", at this moment, onto your PC.
> Now, tell me which files I have open. Which TCP connections are live.
> etc.
> The "state" is spread around in several places *besides* the process
> itself. Including the mail server at the other end of this SMTP connection
> (when I click SEND).
>
> [I.e., unless the passphrase for my SMTP server persists somewhere in
> this process's memory, your copy of it's memory won't help you access
> that connection that exists *now*]
>
>>> The actions involved in management may reside in the OS, a service
>>> (created by <something> -- including the client in question!) or the
>>> client. At the direct or *implied* request of the client, etc.
>>
>> You are failing to realize that it doesn't matter who manages the
>> data: it matters only who *owns* the data.
>>
>>> Likewise, if some of the "properties" reside *in* the client (e.g.,
>>> the algorithm by which allocations will be performed -- under the
>>> supervision of some remote/system service that actually *does* them),
>>> then you have to drag that/those things along with you.
>>
>> Or simply provide them in the new setting. E.g., there is no reason
>> to copy in-memory code if the new host has access to it via a file
>> system. If the new host has a different architecture, you can't copy
>> the code anyway and would have to provide checkpoint mechanisms
>> whereby the *program* (not "process") can be stopped and restarted
>> cleanly.
>
> The applets run in a virtual machine. So, I can pick them up and
> move them to another heterogeneous host. (The "servers" are currently
> designed for a homogeneous environment. But, the same concepts apply)
>
>>>> 2) Rehosting a running process necessarily requires checkpointing,
>>>> copying or reconstructing its entire dynamic state: heaps, stacks,
>>>> globals, loaded modules, kernel [meta]data, etc. - far more than one
>>>> memory block.
>>>>
>>>> Every piece of distributed state must be identifiable as belonging to
>>>> the process - regardless of what "module" may have created or is
>>>> currently managing it.
>>>>
>>>> You have to copy all data belonging to the process, so having heap
>>>> properties stored separately from the heap itself is not a problem.
>>>
>>> In my "portable" case, this is handled by moving the object handles.
>>> The system can then opt to "optimize" execution by (later?) moving the
>>> actual object instances (to minimize communication delays or take
>>> better advantage of processing power on some particular node -- which
>>> may differ from the source or destination nodes)
>>>
>>> But, you can (currently!) create "non-portable" objects.
>>
>> So they are lost. So what? You knew they were non-portable when you
>> created them.
>
> But the apps that contain them can be portable! You might have
> *chosen* to make a non-portable potato (by implementing a potato
> server as one of the threads in your process). The potato is
> bound to the process. But, the process isn't (necessarily) bound
> to the *node* (or process group, etc.).
>
> E.g., I can create a mutex to allow the local threads to cooperatively
> access <something> (maybe just a 4 byte counter). As long as the threads
> remain bound in that process container AND the counter (or it's accessor)
> AND the mutex, the whole entity (process) can be moved at will.
>
> If some other process/agent needs to access that mutex, then it
> needs a handle that is visible from outside of the local process.
> If that other process can migrate to a different node, then it
> can retain the ability to interact with the mutex in this first
> process still residing on the original node.
>
> So, you end up with very different use/performance cases:
> - two or more colocated threads accessing the object
> - two or more LOCAL *processes* accessing the object
> - two or more REMOTE processes accessing the object
>
> [Note, of course, that any other *object* can also indirectly be accessing
> that object/potato]
>
> In the first case, you can conceivably optimize the access to that of
> a conventional mutex (in this example -- remember potatoes!)
> implementation.
>
> In the second case, one process might be able to access the object
> "from within" (a direct call to the functions that implement the
> object) while another process (same node) accesses it "from without".
>
> In the third case, one may be from within, or without, or "remote".
>
> I don't consider it a "good idea" to be forced to make these "mobility"
> decisions at compile time. For the same reason that you don't decide
> where in
> memory a piece of code should execute. Or, whether it WILL be optimized by
> the compiler, or not (compiler command line options aren't reflected in
> the sources)
>
> Just like I don't define which functions/methods should be accessible
> remotely (let the IDL compiler address that based on what I tell *it* to
> do)
>
>>> And, objects
>>> that can't (easily) be shared. (e.g., for a "conventional" heap, you
>>> can let the default policies available in a "heap manager" govern
>>> the way the heap operates. *But*, you can't take advantage of any
>>> "enhanced capabilities" in much the same way that you can't in a
>>> more conventional process container, etc.)
>>
>> There's no reason code can't use "enhanced capabilities" - as long as
>> the result is the same (for some definition). E.g., if you can ignore
>> timing, then debug code is interchangeable with optimized code.
>
> That's been my goal! But, it means the code has to reconfigure itself AT
> RUNTIME. Because the compiler doesn't know where an instance of a potato
> will reside at *compile* time.
>
>>> The problem with this (apparently arbitrary) distinction between
>>> portable/shareable objects and "legacy" variety implementations
>>> is that the developer has to explicitly decide what can be migrated,
>>> shared, and how (because the developer has to take extra steps at
>>> compile and link time to make those capabilities available).
>>
>> It is completely arbitrary, and IMO unnecessary.
>>
>> If there is something - a device possibly - that only exists on one
>> host, then the module that interacts DIRECTLY with that <something> is
>> not portable. Anything objects it creates as a server on behalf of a
>> client likewise are not portable.
>
> No. A "something" that CAN only exist on one host (node) is not portable.
> A "something" that HAPPENS TO only exist on one node but CAN be migrated
> to another node doesn't suffer those same restrictions.
>
> I put a thin layer immediately above the hardware devices on each node
> (think of it as a sort of HAL). At the top interface of this layer, I
> make a deliberate choice as to whether or not that interface shall be
> exportable across nodes.
>
> For example, there is little need to bind anything that uses the
> "irrigation valve" hardware (HAL layer) to the node that actually
> has the hammer drivers for those valve actuators. They are inherently
> slow and largely sacrificial. So, that interface is exported. The
> "irrigation control task" can reside anywhere in the system -- including
> nowhere (zombie).
>
> OTOH, the interface to each of the video cameras is relatively high
> bandwidth. The task(s) that directly interact with them *should*
> reside on those nodes (*if* you assume the camera output will see
> much consumption). So, those interfaces are not exported.
>
> OToOH, the interface to the *compressed* video streams provided by those
> tasks *can* be exported. Their bandwidth requirements might be
> considerably less than the "raw" video coming off each individual
> camera. So, a "motion detection" task can run on the IRRIGATION
> NODE. Or, mode to the LAUNDRY NODE if/when the irrigation node
> is taxed with some other responsibility (perhaps it is taken out of
> service as the valves -- and the proxies that run them -- are no longer
> needed)
>
> I.e., you gain the most flexibility by making everything exportable.
> But, that means you either incur the costs associated with that
> overhead -- even for the "local" case (e.g., motion detection task
> running on the same node as the camera!) -- *or* do run-time
> optimization of the communication path so that the remote costs
> are only incurred when necessary (and factored into the workload
> manager's reassignment of tasks to nodes)
>
>> So if this module is to be used by portable applications, it should
>> NOT be implemented as an in-process server. Q.E.D.
>>
>>> So, where these sorts of bindings get stored (and how they get
>>> tracked) varies based on this "other" information that the developer
>>> supplies.
>>
>> This is where you and I have parted company in the past. You like the
>> kernel capability approach whereas I prefer the user capability
>> approach ... precisely BECAUSE user capabilities avoid many of these
>> semantic problems by virtue of being ordinary data.
>
> The value of the in-kernel capability is that inherently tracks
> references. If the capability is "just data", then you have to
> add a second mechanism to track *where* every copy of the data
> happens to reside! There's no way for the system to know that
> you've just handed a COPY of a capability to something, deleted
> YOUR copy, then retrieved a copy of that copy! (i.e., the
> "reference" is still "live", even though it looks like you
> deleted it). The "system" has no idea who has outstanding
> references (handles/capabilities) nor how MANY.
>
> By contrast, if the kernel manages the capabilities, then they
> can't (by definition) be copied or transferred or deleted
> without the kernel being an active part of that operation.
> The kernel (and the manager for that particular capability)
> can mediate how each is used ("No, you can't give that
> capability away! If you try to do so, I will abend your
> process and/or just silently DELETE the capability and let
> you and the other party wonder why that operation that
> you intended to have performed *isn't*!")
>
> "Is anyone using this video camera? No, there are no active
> capabilities in existence ANYWHERE in the system so we can
> shut it down. Is anyone (process) using the node that hosts
> that camera? No, so we can shut down the node, as well!"
>
>> Everything other than kernel level process and address space control
>> can be done in user space, and rehosting then is simply a matter of
>> replicating the memory state [and reconstructing the kernel state].
>>
>>>>> The "consistent" solution is to make all of these first class
>>>>> objects and let the OS manage them and their locations/instantiations.
>>>>> But, that adds considerably to the cost of *using* them.
>>>>
>>>> Yes, that certainly is the ridiculous, overcompensating solution.
>>>>
>>>> The "consistent" solution is the one done already by any decent OS:
>>>> keep a record of all the kernel structures and memory pages belonging
>>>> to the process.
>>>>
>>>> Program objects then are just ordinary data in memory blocks "owned"
>>>> by the process, or by the kernel on behalf of the process.
>>>
>>> But that ignores all of the "other" dependencies that can be in
>>> play at any given time (i.e., my "solution" being these portable
>>> handles that the OS tracks for the application/developer). So,
>>> <something> knows that there is an aspect of the current applet's
>>> instantiation that relies upon something else *or* that is relied
>>> upon BY something else.
>>
>> No it doesn't.
>>
>> You have created a whole set of artificial dependencies through your
>> (ab)use of kernel capabilities. Nothing in your system is - or even
>> can be - self contained.
>
> A process need never export a handle to an object that it creates
> and manages -- for itself. The OS needn't even be aware that the
> object exists. Does the OS know that I have a 4 byte counter
> *here*? Or here?
>
> OTOH, if something *else* DEPENDS on that counter (potato), then
> you want that dependency to be visible to the system. You don't
> want the potato to silently disappear without the dependent entity
> knowing about it, *now* (not at some later date when it expects it
> to STILL exist).
>
> Likewise, if it "moves", you don't want the dependent entity to have to
> scramble to *find* it when it needs it -- for much the same reason as
> above.
>
>>> [My approach lets the run-time know of these possible external
>>> references by the presence of object handles held -- or exported -- by
>>> the applet in question. *Absent* these, the task is just a block of
>>> memory and "processor state" that can be copied anywhere and "resumed".]
>>
>> But by doing that, you've inserted the operating system into all kinds
>> of things where IMO it has no business being. You've created the MCC
>> from "Tron".
>
> An OS manages resources. I consider dependencies to be resources.
> And, the permissions associated with them.
>
> I don't want a user-level task to have to keep track of where the
> things that it needs happen to be, *now* -- or if they even happen
> to EXIST!
>
> Nor do I want to have to require the developer to never try to do
> something with a particular thing/object (potato) that it *shouldn't*.
> Just like the OS shouldn't let process A peek into process B's memory.
> Or, access unmapped portions of memory, etc.
>
> You'd not advocate allowing a user to "add 3" to an arbitrary string.
> You'd expect the *compiler* to enforce the rules of the language on
> his *use* of the language.
>
> I expect the OS to enforce the rules of the *system* on the users
> of that system in much the same way!
>
>>>> You can somewhat mitigate [rehosting] by making large allocations
>>>> piece wise: abusing VMM to make them appear address contiguous.
>>>>
>>>> E.g., if a process asks for 100MB, reserve the address space but
>>>> instantiate just a few megabytes at a time, on demand as the process
>>>> touches unbacked addresses.
>>>>
>>>> [Because (I assume) you don't want to overcommit memory, you need to
>>>> reserve requested address space both locally in the process, and
>>>> globally so that other processes can't accidentally grab it.]
>>>
>>> Resource constraints vary with the resource and the consumer/provider.
>>> You can overcommit many resources because many "jobs" don't exploit
>>> their worst-case resource needs.
>>>
>>> But, you do so at the risk of delaying the availability of those
>>> resources
>>> to the job in question. Or, some other job interested in those
>>> resources.
>>
>> No, you are misunderstanding. The process owns the space ... but in
>> the scenario I presented, the physical memory backing the space is
>> mapped on demand - which gives the OS more visibility into the process
>> if/when it needs to rehost the process.
>
> The "handles" give the workload manager (via the OS) insight into how and
> where it can/should rehost the process (or any other object!). It knows
> what the process "talks to" because it manages the capabilities FOR that
> process. And, it knows the other (more tangible) resources that the
> process requires (RAM, MIPS) by examining its current ledger and historical
> performance. So, it can balance the costs of "stretching" certain
> communication channels (RPC/RMI being more expensive than the local case)
> against the costs that some *other* process/object (a process is just a
> different sort of object, managed by similar handles) would incur if
> *this* process is not migrated.
>
> [You can't come up with an "optimal" solution. But, you can remember the
> solutions you've tried and how they performed -- because the OS is
> involved in these communications and can measure them! So, you can
> avoid solutions that performed poorly in favor of those that performed
> *well* or that "have yet to be tried.]
>
>> Remember that there are 3 forms of VMM addresses: "linear", "virtual"
>> and "physical".
>> - "Linear" space is what is seen by an application.
>
> "Logical" -- it need not be linear/contiguous.
>
>> - "Physical" space is the actual RAM as seen by the system.
>> - "Virtual" space is what ties the other 2 together.
>
> And, how does this pertain to potatoes? :>
>
>> In a system using MMU only for process isolation, the "virtual" and
>> "physical" spaces would be identical - mapped 1:1. Overcommitting
>> memory is only possible when "virtual" space > "physical" space.
>
> Exactly. Recall that there are multiple objects sharing that physical
> space on a particular node. The physical space determines the maximum
> amount that can be mapped at a given time to the set of objects
> currently making demands on it. Those demands can (do) change, over time.
>
> E.g., a process may load many modules and, by executing code *in*
> each of them, cause pages to be mapped to instantiate those "opcodes"
> JIT. If a portion of a module is never actively referenced, then
> the memory required to back it never gets mapped -- not taxing the
> physical resources currently available. If a module gets UNloaded,
> then all of the mapped pages that were referenced can be released for
> other uses (other potatoes). Likewise, for the pages that were
> never actually mapped!
>
> E := load Encyclopaedia
> ...
> article := E.lookup(Da Vinci)
> ...
> unload(E)
>
> You wouldn't foolishly load the entire encyclopaedia into physical
> (or virtual) memory to do this. Instead, you'd load some accessor
> code that could, itself, load PORTIONS of the "index" *when* you
> initiated a search. This would require some number of pages of
> physical memory to be mapped to hold the resulting "article".
> Anything beyond that would be waste!
>
> E := load Encyclopaedia
> ...
> article := E.lookup(Da Vinci)
> ...
> article := E.lookup(R Crumb)
> ...
> unload(E)
>
> Would only require resources proportional to the LARGER of the Da Vinci
> and R Crumb entries (the resources held by the first "article" are
> freed when the second "article" is defined).
>
>> When an application requests 100MB [e.g., (s)brk] from the OS, then
>> [assuming the request can be granted] the OS adds 100MB to the linear
>> space of the process, and *reserves* 100MB in the virtual space of the
>> system.
>
> No. I don't *impose* that requirement! Because it makes the system
> brittle. The application handles this at a different level dictated
> by the CM system.
>
> If, in your example, you ultimately *use* all 100MB of that memory
> (which may be mmap'd in page-at-a-time increments) and then need
> (or, someone else needs) 50MB *more*, the system decides how best to
> provide those resources based on parameters from the CM system.
>
> If "more resources" are available (though not necessarily "50" MB),
> the system can opt to satisfy the request and let you continue to
> mmap (by your usage patterns) additional pages.
>
> If there are NO more resources available *when* you mmap another
> page (indirectly by your actions), the system decides how best to
> handle that request.
>
> Can I unmap some portion of a loaded module (knowing that I can
> always reload that portion of the module AS IF it hadn't *yet*
> been referenced/mmap'd)? Can I pageout some "really low" portion
> of the stack (on the assumption that it won't be accessed RSN)?
> Can I *migrate* some other task off of this node and reclaim the
> physical resources that it currently holds? Can I migrate *this*
> task to some other node that has more resources available? Can
> I bring another node on-line and migrate "something" to it? Can
> I rely on some redundant (potato!) server to assume responsibility
> for managing some objects that happen to reside on the local system
> and, as such, free up the space they occupy *and* the space
> required for the code to manage them? Can I *kill* some task
> (and associated objects) that are just running opportunistically,
> presently (let's do some commercial detection in that OTA TV
> broadcast we recorded two hours ago so its ready for viewing before
> the user comes home and wants to see it!)? etc.
>
> You don't want to make those decisions at compile time. Because you
> don't know what will be active in the system at *run* time! And,
> you want to build as much flexibility in how/where resources can
> reside so that you have more run-time options. You don't want to
> overprovision just to handle a "worst case" that, in practice, will
> never occur!
>
> [Repeat this substituting "MIPS" for "memory", etc.]
>
>> Now the reserved space belongs to the requesting process - no other
>> process can take it.
>
> See above. If a process (potato) *needs* to be sure that resource
> (memory, MIPS, etc.) *is* available, "sitting and waiting", then it
> has to place an active "reservation" through the CM system. The
> CM system dictates limits for what individual tasks/"users" can
> consume.
>
> Reserves are A Bad Thing as they tie up those resources regardless
> of what other objects might happen to place similar demands on the
> system's resources. What's to stop "everyone" from demanding EVERYTHING
> that they *might* need "just in case"?
>
> If, instead, you allow elasticity in how the resources are supplied,
> you can afford to loosen the constraints on individual "consumers"
> because they have less ACTUAL impact on other consumers.
>
> Joe User (not Bob Developer) could write a sloppy script that
> consumes "limitless" resources. Should he be allowed to bring the
> system to its knees -- because he is the "owner" of the device?
>
> Similarly, should Bob Developer be allowed free reign at the risk
> of compromising core services (implemented by *me*)?
>
> It's a lot easier to ensure your system can meet all its goals if
> you provision *for* those goals. But, with an open/expandable system
> that sees huge variations between actual and potential use, that
> translates into dollars that consumers aren't keen on spending.
>
> (why do you have more VM configured in your PC than physical memory?)
>
>> However, the OS can choose whether or not to immediately map the new
>> linear space to physical space. By mapping physical space on demand
>> as it is touched by the process, the OS has a much better picture of
>> the *actual* memory use of the process.
>
> Yes. Though that can vary with other "input data" that this instance
> of the process (potato) sees. E.g., the resources required for that
> lookup(R Crumb) are probably far less than the earlier lookup(Da Vinci).
> Do you *know* what the user will opt to want to "lookup" when you
> instantiate that task? :>
>
> You can either say "you can lookup ANYTHING" -- and provision for it.
> Or, "you can look up anything as long as the article doesn't exceed X"
> (and impose a disconcerting constraint on the user: "How do I know
> how big the article is LIKELY to be?"). Or, you can make a best
> effort and trade your response based on your current needs and those
> just added by the user.
>
>> When the time comes to replicate the process's memory on another host,
>> only mapped physical space needs to be copied - not the entire virtual
>> address space reservation.
>
> Again, that assumes the (active/mapped) memory represents the entire
> "state" of the process and all of its dependencies. (what about any
> network connections -- managed by the OS -- that may be active at the
> time? the contents of any NIC buffers AS the process is in transit?
> file handles, my "object handles", etc.)
>
> [Think about it. Send me an ISO with an image of your PC's memory
> at this point in time. I'll let you include the OS's state in that
> image, as well! Will I be able to read this USENET post at exactly
> THIS --> . <-- point when I load the image into my machine's memory?
> What if you mark the thread as read before the ISO gets loaded on
> my machine? Or, if YOUR server expires the article? Or, if
> your NNTP server happens to be offline? That's all state that is part
> of your "reading this USENET post" activity -- yet, you've not captured
> it well enough for me to pick up where you left off! :> ]
Just get the book
Andrew S. Tanenbaum, Operating Systems: Design and Implementation,
read and understand it, then come back.
--
-TV
[toc] | [prev] | [next] | [standalone]
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2017-03-18 04:04 -0700 |
| Message-ID | <oaj434$oeb$1@dont-email.me> |
| In reply to | #26131 |
On 3/18/2017 3:05 AM, Tauno Voipio wrote: > Just get the book > > Andrew S. Tanenbaum, Operating Systems: Design and Implementation, > > read and understand it, then come back. AST doesn't address distributed OS's in that tome concentrating only on a toy OS. Get his _Distributed Operating Systems_. Understand *it*. Make sure you can understand why a distributed OS is a very different animal from a monolithic one. Pay particular attention to the case studies of Mach, Amoeba and Chorus and the tradeoffs made in their *different* implementations. Then, go chase down the *technical* information on each of those. Boykin et al. would be a good place for you to get some exposure to what you could do with the original Mach. Amoeba will require a fair bit more digging (no source code available). Then, you can look into the Inferno docs to see how they handle location transparency and followup issues addressed in their Grid implementation. From there, imagine how you'd support resource migration in each of the above -- and how you'd expose it to the developer. Finally, sort out how to extend all of these concepts to real-time programming. [All this assumes you've already been through Organick, McKusick, etc.] Don't bother coming back -- been there, done that, T-shirt, etc. :>
[toc] | [prev] | [next] | [standalone]
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2017-03-18 04:39 -0700 |
| Message-ID | <oaj63n$u5o$1@dont-email.me> |
| In reply to | #26133 |
On 3/18/2017 4:04 AM, Don Y wrote: > On 3/18/2017 3:05 AM, Tauno Voipio wrote: >> Just get the book >> >> Andrew S. Tanenbaum, Operating Systems: Design and Implementation, >> >> read and understand it, then come back. > > AST doesn't address distributed OS's in that tome concentrating only > on a toy OS. Get his _Distributed Operating Systems_. Understand *it*. > Make sure you can understand why a distributed OS is a very different > animal from a monolithic one. Pay particular attention to the case > studies of Mach, Amoeba and Chorus and the tradeoffs made in their > *different* implementations. > > Then, go chase down the *technical* information on each of those. > Boykin et al. would be a good place for you to get some exposure to > what you could do with the original Mach. Amoeba will require a fair bit > more digging (no source code available). Then, you can look into the Sorry, Amoeba *is* available; *Alpha* (Jensen et al.) isn't -- but has similarly applicable concepts (particularly capability handling). You'll need to understand both approaches, and their consequences, before you can appreciate how they can be leveraged for security and reliability (esp redundancy). > Inferno docs to see how they handle location transparency and followup > issues addressed in their Grid implementation. From there, imagine how > you'd support resource migration in each of the above -- and how > you'd expose it to the developer. > > Finally, sort out how to extend all of these concepts to real-time > programming. > > [All this assumes you've already been through Organick, McKusick, etc.] > > Don't bother coming back -- been there, done that, T-shirt, etc. :> >
[toc] | [prev] | [next] | [standalone]
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2017-03-18 03:55 -0700 |
| Message-ID | <oaj3hb$ml2$1@dont-email.me> |
| In reply to | #26129 |
On 3/17/2017 11:25 PM, Don Y wrote:
>> When the time comes to replicate the process's memory on another host,
>> only mapped physical space needs to be copied - not the entire virtual
>> address space reservation.
>
> Again, that assumes the (active/mapped) memory represents the entire
> "state" of the process and all of its dependencies. (what about any
> network connections -- managed by the OS -- that may be active at the
> time? the contents of any NIC buffers AS the process is in transit?
> file handles, my "object handles", etc.)
ProcessA:
ThreadA1();
ThreadA2() {
...
// some sort of synchronization primitive with ThreadA3
send_resource(my_potato, ProcessC)
...
exit
}
ThreadA3() {
...
my_potato = create_potato(size, flavor, &my_masher)
...
exit
};
my_masher();
ProcessB:
ThreadB1();
ThreadB2();
create_potato();
default_masher();
ProcessC:
ThreadC1();
ThreadC2() {
...
delete(received_potato)
...
};
Normally, when all ThreadAi have exited, ProcessA would die and
its resources be reclaimed.
If create_potato() was hosted by ProcessA *and* the send_resource()
targeted ProcessA, then everything is self-contained; when everything
dies/exits, the potato can be deleted (because nothing outside the
ProcessA references it). There is no visibility outside of that
process container for any of the objects contained within.
If we assume ProcessB has the "potato server" (i.e., if create_potato()
is an aspect of that SERVICE), then ProcessB must remain active for
the potato to be created *and*, once created, must continue to remain
active to service requests made of the "my_potato" object (because
my_potato is just a handle to the potato abstraction reified by
the potato server in ProcessB). Note that this dependency is extended
to ProcessC once the potato resource has been transfered to ProcessC
by ThreadA2!
In addition to ProcessB remaining active while the outstanding reference
to the potato exists, ProcessA must remain active even after the potato
has been passed to ProcessC -- because of the dependency on "my_masher"
that is bound to the instantiation of the potato as it occurred in
ThreadA3!
[Maybe "my_masher" leaves a certain amount of chunkiness in the potatoes
after the mash method is applied -- by the potato server -- while the
default_masher ensures a more uniform, pasty consistency]
<something> has to track these dependencies. We (I) don't trust the
developer to write (bug-free) code to "manually" do that -- not for EVERY
such "distributed object". The compiler can't do it because it
doesn't know where the objects will reside.
By assigning capabilities/handles to each of these objects (a masher
is an object just like a potato is an object and a potato server is an
object and a process is an object and a thread is an object and a...),
the OS can keep track of:
- where each object "resides"
- where each reference to a particular object resides
- how many outstanding references to an object exist
So, when ThreadC2 finally deletes the potato (which *it* knows by the
name "received_potato"), the resources bound up in that potato instance
can be freed (from the "potato server" in ProcessB as well as the
"masher service" in ProcessA -- in addition to the handle in ProcessC!
If ProcessA (and/or ProcessB and/or ProcessC) now has no remaining
active resources (e.g., all threads have exited), then *they* can
be reclaimed along with their resources.
If the system opts to migrate ProcessA (containing my_masher()) to
another node *while* that potato is still active, then the resources
previously used for that *instance* of ProcessA (on NodeX) can be
freed -- but, an outstanding claim levied against the resources that
ProcessA (now known as ProcessQ) *still* requires while being hosted
on NodeY.
So, if received_potato->mash() is invoked by a thread in ProcessC
*after* it has accepted the transfer of "received_potato" but before
it has been deleted, the potato server in ProcessB will receive that
RMI and, as part of its *mechanism*, will invoke the masher member
that will be implemented by my_masher() in ProcessA (if ProcessA
hasn't yet migrated) *or* by ProcessQ (if ProcessA *has* already
migrated).
There's no clear cut concept of the potato's "ownership". The
process that has "rights" to it changes (from A to C in this example).
The process that *implements* it morphs (from B assisted by A to
B assisted by Q). A single object has tied up resources in three
different process containers.
For a concrete EXAMPLE of similar objects, imagine the potato is a
video_clip and there are "compress", "store", "display", "motion_detect"
and "commercial_detect" methods available. Further, that the user
can *bind* (returning to the concept of closures) particular functions
and criteria to each of these.
So, maybe an "MP4" functionality is bound to a Compressor object and
that compressor object bound to the clip. I.e., my_clip->compress()
effectively causes an MP4 version of the clip to be created. The
MP4 functionality is provided by an MP4 *module* in ProcessD running
on nodeI (because nodeI has the resources available to expedite that
processing).
Meanwhile, a Region object (polygon) may be created and bound to the
motion_detector object that is created to determine which portion(s)
of my_clip to examine for signs of motion in the motion_detect method.
And, criteria for a "commercial_detector" bound to a detector that
is eventually bound to that video_clip and subsequently invoked
by the my_clip->elide_commercials() method running in ProcessL on
nodeW.
Why force every compression algorithm to be implemented in a single
"Compressors" module? Or, be supported by set of resources defined
by the UNION of their requirements? Why force all commercial
detection algorithms to co-reside with these compressors? And, all
to reside in a "video_clip" module? Or, execute on the same hardware
node, concurrently?
Why force all of these bindings to persist if the client(s) that
*had* permission to invoke them have terminated? (i.e., if there
is only one outstanding capability for the compress method and
the process holding that capability terminates, then NOTHING can
invoke that compressor! So, the *binding* can be freed in my_clip.
And, in return, the service that implements it (in ProcessD) can be
allowed to die an orderly death (if no other outstanding references).
Put *all* of this in a single process and the OS has no way to
know when certain resources are no longer required -- because
the containing object (the process) can't inform the OS of "things"
of which the OS has no knowledge (this is A Good Thing because it
means the OS doesn't HAVE TO know about everything!)
[toc] | [prev] | [next] | [standalone]
| From | upsidedown@downunder.com |
|---|---|
| Date | 2017-03-18 16:00 +0200 |
| Message-ID | <4oeqccp48p35e887212dvq7b92mpiiheml@4ax.com> |
| In reply to | #26103 |
On Thu, 16 Mar 2017 17:23:11 -0700, Don Y <blockedofcourse@foo.invalid> wrote: >Hi George, > >On 3/15/2017 6:05 PM, George Neuner wrote: > >>> Assume applet, memory_manager_module, applet_parent. >>> >>> Applet_parent (for example) instantiates a memory object for applet. Part >>> of that requires binding parameters to that memory object (size, granularity, >>> allocation_policy, release_policy, etc.). Applet_parent need not persist >>> beyond this point (assume its job is done). >>> >>> Where do the bound parameters get stored? In the applet? In the memory >>> object? Obviously, can't be part of applet_parent because that won't >>> persist (unless you make other arrangements for it or portions of it). >>> >>> [I.e., applet_parent can't define an allocation handler in its body.] >>> >>> Now, migrate applet to another node. Another (shared) instance of >>> memory_manager_module likely already exists on that node. If the >>> allocation handler was a "stock" handler (contained in the >>> memory_manager_module), can you simply copy the state over to >>> the new node? >>> >>> What if the allocation handler was resident in some OTHER module >>> on the original node? Do you migrate that other module, as well? >>> Or, instantiate another copy if the module has outstanding references >>> on the first node?? >> >> For one "applet" (process / security context) to allocate memory on >> behalf of another, that memory has to be supplied by something that >> exists outside of either of them. > >Actually, the memory can exist anywhere: in some global space, in >the "allocating entity" or in the "client entity". As can the code >that does the actual allocating (at the behest of the allocator). > >[Consider client environment not supporting arbitrary pointers. >Unless something *gives* you a reference/handle to a piece of memory, >there's no way you can access its contents (short of an exploit). >E.g., no way you can go looking up/down the stack even though you >know there *is* a stack supporting the implementation language] > >> Thus the system memory manager can't be a dynamic "module" loaded >> locally into a process. Its *interface* could be, but not the manager >> itself. >> >> Leaving that aside. >> >> Ok, so you have a block of system memory, and you created one (or >> more) heaps having certain properties within the block - heaps which >> are "managed" locally in the process by code from a dynamic module. >> And now you want to rehost the process. > >Objects (heaps in this example) are managed by <something>. That >something may be the entity "owning" the object, a proxy or the >OS acting as the "proxy of last resort". > >The actions involved in management may reside in the OS, a service >(created by <something> -- including the client in question!) or the >client. At the direct or *implied* request of the client, etc. > >> 1) The obvious: store the heap properties together with the heap so >> that "copying" (serializing) the contents of the memory block to >> another host replicates the heap state for the equivalent "manager" >> module on that system. > >This potentially exposes those "trusted" parameters to abuse by >the entiti(es) having access to the underlying object. E.g., if client >can tweek the "heapsize" parameter stored there, then it can potentially >trick the "manager" on the remote system to instantiate a larger heap than >it was originally allowed to create (unless you rerun the "create_heap" >request as part of the migration effort) > >Likewise, if some of the "properties" reside *in* the client (e.g., >the algorithm by which allocations will be performed -- under the >supervision of some remote/system service that actually *does* them), >then you have to drag that/those things along with you. > >[Imagine these things reside in a *third* entity] > >> 2) Rehosting a running process necessarily requires checkpointing, >> copying or reconstructing its entire dynamic state: heaps, stacks, >> globals, loaded modules, kernel [meta]data, etc. - far more than one >> memory block. >> >> Every piece of distributed state must be identifiable as belonging to >> the process - regardless of what "module" may have created or is >> currently managing it. >> >> You have to copy all data belonging to the process, so having heap >> properties stored separately from the heap itself is not a problem. > >In my "portable" case, this is handled by moving the object handles. >The system can then opt to "optimize" execution by (later?) moving the >actual object instances (to minimize communication delays or take >better advantage of processing power on some particular node -- which >may differ from the source or destination nodes) > >But, you can (currently!) create "non-portable" objects. And, objects >that can't (easily) be shared. (e.g., for a "conventional" heap, you >can let the default policies available in a "heap manager" govern >the way the heap operates. *But*, you can't take advantage of any >"enhanced capabilities" in much the same way that you can't in a >more conventional process container, etc.) > >The problem with this (apparently arbitrary) distinction between >portable/shareable objects and "legacy" variety implementations >is that the developer has to explicitly decide what can be migrated, >shared, and how (because the developer has to take extra steps at >compile and link time to make those capabilities available). > >So, where these sorts of bindings get stored (and how they get >tracked) varies based on this "other" information that the developer >supplies. > >[You don't want to have to change the sources to support these >abilities as you want to be able to experiment with letting them migrate, >be shared, etc. without having to undertake a rewrite/refactoring. >And, I can't see an easy way for the build tools to determine that >without explicit directives from the developer (hence my related >comments in this thread)] > >> [Unless the systems involved do not have MMU/VMM such that the process >> address space can be faithfully reconstructed ... but if you really >> need to do it, it can be done using a software based virtual machine >> solution.] >> >>> Or, do you require declarations as to what *can* migrate and what >>> is bound in place? Then, contend with the consequences of >>> cascaded dependencies? >>> >>> The "consistent" solution is to make all of these first class >>> objects and let the OS manage them and their locations/instantiations. >>> But, that adds considerably to the cost of *using* them. >> >> Yes, that certainly is the ridiculous, overcompensating solution. >> >> The "consistent" solution is the one done already by any decent OS: >> keep a record of all the kernel structures and memory pages belonging >> to the process. >> >> Program objects then are just ordinary data in memory blocks "owned" >> by the process, or by the kernel on behalf of the process. > >But that ignores all of the "other" dependencies that can be in >play at any given time (i.e., my "solution" being these portable >handles that the OS tracks for the application/developer). So, ><something> knows that there is an aspect of the current applet's >instantiation that relies upon something else *or* that is relied >upon BY something else. > >[My approach lets the run-time know of these possible external >references by the presence of object handles held -- or exported -- by >the applet in question. *Absent* these, the task is just a block of >memory and "processor state" that can be copied anywhere and "resumed".] > >> Ensuring that a rehosted program will work is another matter. But as >> I understand (???) your system, there is location transparent naming >> and equivalence[*] hosts will expose the same set of service APIs. >> >> [*] capable of running the same applications. >> >>> Its a lot easier to deal with implementations where "everything" is >>> a cohesive "blob" instead of being able to slice-and-dice it at >>> run-time and redistribute it based on evolving workloads, resources, >>> etc. >> >> You can somewhat mitigate the problem by making large allocations >> piece wise: abusing VMM to make them appear address contiguous. >> >> E.g., if a process asks for 100MB, reserve the address space but >> instantiate just a few megabytes at a time, on demand as the process >> touches unbacked addresses. >> >> [Because (I assume) you don't want to overcommit memory, you need to >> reserve requested address space both locally in the process, and >> globally so that other processes can't accidentally grab it.] > >Resource constraints vary with the resource and the consumer/provider. >You can overcommit many resources because many "jobs" don't exploit >their worst-case resource needs. > >But, you do so at the risk of delaying the availability of those resources >to the job in question. Or, some other job interested in those resources. > >[E.g., if your job is diarising a previously recorded telephone conversation, >it can *probably* afford to block waiting on memory that it needs to complete >that task -- perhaps even indefinitely! It shouldn't be prevented from >starting just because the MAX memory that MIGHT be required isn't presently >available. Nor should all of that memory be wired down for its benefit >without knowing that it *will* need it -- or, when!] > >> You need to reconstruct the address space on the new host, but you >> need only copy as much physical memory as the process is actually >> using. >> >> Aside: one of the nice features of "moving" GC is that it compacts >> live data into a (usually) smaller address range. You don't want to >> have to copy a large block to get at a much smaller volume of data >> contained within it. > >But you need a way of tying a "handle" to the data involved -- and, >later, resolving that reference. > >> There are a number of tricks that involve using VMM to implement GC. >> You can't completely avoid data copying if you want to compact the >> live data, but using VMM techniques together with clustered BiBoP >> allocation, you can reduce the amount of copying to bare minimum. >> >> It also is possible to use VMM and just the tracing step of GC to >> identify the actual set of pages containing live data ... you don't >> need to copy any more than that even if the process has much more >> memory instantiated. >> >> And regardless of whether you (would want to) use GC in normal >> operation, if it is supported by the language/runtime, a compacting >> collection executed before rehosting a process would minimize both >> what needs to be copied and the physical memory that must be >> instantiated on the target. >> >>> Last 25# of oranges... yippee! >> >> 10 inches of snow, followed by sleet, followed by freezing temps. The > >Yeah, caught a glimpse of that on the national news... > >> crust is ~3/4 of an inch thick - my 185 lbs can walk on top of it. Had >> to use the heavy coal shovel to break it up so it could be removed. > >90's, here. I think 95 expected this weekend. Being outdoors is a >chore as it's either too warm during the daylight hours *or* the air >too "smelly" (everything is in bloom) in the cooler hours when the >breeze has died down. > >I'd (personally) appreciate a cold spell (but not *now* as the new >crop of fruit is setting in place) to slow the growth process. I >suspect it will be a bad year for insects! > >[Normally, we don't see 95 until May; and 90 until mid-April] I am not sure where you did move the goal posts this time, but at least the VAX/VMS VAXcluster addressed some of these issues in the 1980s. Switching processes between (cabinet size) CPUs was routine via a shared (redundant) disk pool (checkpointing and virtual memory page faulting). These days the disk pool could be part of the L3 (or L4) cache hierarchy.
[toc] | [prev] | [next] | [standalone]
| From | Les Cargill <lcargill99@comcast.com> |
|---|---|
| Date | 2017-03-11 16:35 -0600 |
| Message-ID | <oa1tm8$2o8$1@dont-email.me> |
| In reply to | #26003 |
George Neuner wrote: > On Fri, 10 Mar 2017 16:49:21 -0700, Don Y <snip> > > Once you introduce heap-based closures the lifetime of closures > becomes indefinite: it can be *estimated* by a compiler - e.g., using > region analysis - but runtime GC generally is better at noticing > disuse in a timely fashion. Module reference counting by the compiler > is no longer sufficient and must be augmented by runtime GC. > > IMO, this is and always will be difficult, and it's never been that clear to me that simply constructing systems of frames of objects that are explicit in lifetime isn't more rigorous and more deterministic. I have also found that having all closures have the same lifetime as the main program/thread that owns them to not be that big of a burden. If something needs a transient lifespan, I'd be happier if it were made explicit and transactional. But I like the 50,000 foot view, which is declining in popularity. <snip> > > George > -- Les Cargill
[toc] | [prev] | [next] | [standalone]
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2017-03-11 15:39 -0700 |
| Message-ID | <oa1u4t$6te$1@dont-email.me> |
| In reply to | #26026 |
On 3/11/2017 3:35 PM, Les Cargill wrote: > George Neuner wrote: >> Once you introduce heap-based closures the lifetime of closures >> becomes indefinite: it can be *estimated* by a compiler - e.g., using >> region analysis - but runtime GC generally is better at noticing >> disuse in a timely fashion. Module reference counting by the compiler >> is no longer sufficient and must be augmented by runtime GC. > > IMO, this is and always will be difficult, and it's never been that > clear to me that simply constructing systems of frames of objects > that are explicit in lifetime isn't more rigorous and more > deterministic. The problem then shifts to handling different lifetime extents. E.g., it's not uncommon for a "program" to "run forever" (c.a.e). At the same time, it is very possible for portions of it to "come and go" with some frequency. How do you address the differences in resource constrained (regardless of the magnitude of those resources) systems and those that are more open? E.g., zombie processes are no big deal on a desktop machine. But, in a machine with tighter constraints, you want to harvest the zombie earlier, rather than later. > I have also found that having all closures have the same lifetime as the main > program/thread that owns them to not be that big of a burden. How do you define ownership? Does init() own the objects that it instantiates? Does init() then need to persist beyond the cumulative lifespans of all of those surrogates? Does init() transfer ownership to some other entity after instantiating it (or an object that it uses)? > If something needs a transient lifespan, I'd be happier if it were made > explicit and transactional. Why not *implicit*? Let some active entity (e.g., OS) take responsibility for the clean-up? > But I like the 50,000 foot view, which is declining > in popularity.
[toc] | [prev] | [next] | [standalone]
| From | Grant Edwards <invalid@invalid.invalid> |
|---|---|
| Date | 2017-03-06 18:59 +0000 |
| Message-ID | <o9kbjc$kfb$1@reader1.panix.com> |
| In reply to | #25896 |
On 2017-03-05, Don Y <blockedofcourse@foo.invalid> wrote:
> A quick/informal/UNSCIENTIFIC poll:
>
> What *single* (non-traditional) language feature
What's the definition of "non-traditional"?
--
Grant Edwards grant.b.edwards Yow! My haircut is totally
at traditional!
gmail.com
[toc] | [prev] | [next] | [standalone]
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2017-03-06 12:36 -0700 |
| Message-ID | <o9kdjh$1uj$1@dont-email.me> |
| In reply to | #25906 |
On 3/6/2017 11:59 AM, Grant Edwards wrote:
> On 2017-03-05, Don Y <blockedofcourse@foo.invalid> wrote:
>> A quick/informal/UNSCIENTIFIC poll:
>>
>> What *single* (non-traditional) language feature
>
> What's the definition of "non-traditional"?
Whatever you want ("unscientific"). The question is meant to draw
attention to that aspect of <your_ideal_language> that you consider
to be most valuable (using *your* definition of "valuable").
[toc] | [prev] | [next] | [standalone]
| From | Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> |
|---|---|
| Date | 2017-03-07 19:02 +0000 |
| Message-ID | <o9n04v$cks$1@dont-email.me> |
| In reply to | #25896 |
On 2017-03-05, Don Y <blockedofcourse@foo.invalid> wrote: > A quick/informal/UNSCIENTIFIC poll: > > What *single* (non-traditional) language feature do you find most > valuable in developing code? (and, applicable language if unique > to *a* language or class of languages) Non-traditional in today's environment only or non-traditional over the whole of computing history ? You sound like you are asking for language features which would be considered non-traditional in today's world only. As such I would offer you the strong typing as seen in the Wirth languages or (especially) Ada. A person who only knows Javascript and C would probably consider C to be strongly typed. A person with Wirth languages (or especially Ada) in their background would have a different viewpoint. I think we are losing something if we continue the move away from strongly typed languages for various application domains. I really like the discipline that strongly typed languages force upon you as I think you produce better code as a result. Simon. PS: I'll let you decide if you consider Ada to be a Wirth language or not. He didn't design it but it's very strongly based on his language concepts. -- Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP Microsoft: Bringing you 1980s technology to a 21st century world
[toc] | [prev] | [next] | [standalone]
| From | Grant Edwards <invalid@invalid.invalid> |
|---|---|
| Date | 2017-03-07 19:18 +0000 |
| Message-ID | <o9n11g$9d$1@reader1.panix.com> |
| In reply to | #25923 |
On 2017-03-07, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:
> On 2017-03-05, Don Y <blockedofcourse@foo.invalid> wrote:
>> A quick/informal/UNSCIENTIFIC poll:
>>
>> What *single* (non-traditional) language feature do you find most
>> valuable in developing code? (and, applicable language if unique
>> to *a* language or class of languages)
>
> Non-traditional in today's environment only or non-traditional
> over the whole of computing history ?
To me, "traditional" means COBOL, FORTRAN and assembly.
--
Grant Edwards grant.b.edwards Yow! Life is a POPULARITY
at CONTEST! I'm REFRESHINGLY
gmail.com CANDID!!
[toc] | [prev] | [next] | [standalone]
| From | Grant Edwards <invalid@invalid.invalid> |
|---|---|
| Date | 2017-03-07 19:19 +0000 |
| Message-ID | <o9n13n$lp2$1@reader1.panix.com> |
| In reply to | #25924 |
On 2017-03-07, Grant Edwards <invalid@invalid.invalid> wrote:
> On 2017-03-07, Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:
>> On 2017-03-05, Don Y <blockedofcourse@foo.invalid> wrote:
>>> A quick/informal/UNSCIENTIFIC poll:
>>>
>>> What *single* (non-traditional) language feature do you find most
>>> valuable in developing code? (and, applicable language if unique
>>> to *a* language or class of languages)
>>
>> Non-traditional in today's environment only or non-traditional
>> over the whole of computing history ?
>
> To me, "traditional" means COBOL, FORTRAN and assembly.
On punch cards or paper tape. :)
--
Grant Edwards grant.b.edwards Yow! I'm ZIPPY the PINHEAD
at and I'm totally committed
gmail.com to the festive mode.
[toc] | [prev] | [next] | [standalone]
| From | Don Y <blockedofcourse@foo.invalid> |
|---|---|
| Date | 2017-03-07 12:38 -0700 |
| Message-ID | <o9n21q$nm9$1@dont-email.me> |
| In reply to | #25923 |
On 3/7/2017 12:02 PM, Simon Clubley wrote:
> On 2017-03-05, Don Y <blockedofcourse@foo.invalid> wrote:
>> A quick/informal/UNSCIENTIFIC poll:
>>
>> What *single* (non-traditional) language feature do you find most
>> valuable in developing code? (and, applicable language if unique
>> to *a* language or class of languages)
>
> Non-traditional in today's environment only or non-traditional
> over the whole of computing history ?
>
> You sound like you are asking for language features which would be
> considered non-traditional in today's world only.
No. I'm letting the respondent(s) define what they consider to be
"traditional" -- as pertaining to *their* historical experiences.
I.e., what features were "wow" when *you* encountered them and
that *you* have decided are "invaluable" (which is yet another
layer of subjectivity as it is driven by YOUR application
domains, etc.)
If you learned your (basic) skills using procedural languages, you
might consider OO features as a simple REPACKAGING of the same -- *or*
something "revolutionary". Moving from ASM to a HLL might leave
you giddy that you can write an infix arithmetic expression instead
of moving "bits" around "memory locations" with special "operators".
> As such I would offer you the strong typing as seen in the Wirth
> languages or (especially) Ada. A person who only knows Javascript
> and C would probably consider C to be strongly typed. A person
> with Wirth languages (or especially Ada) in their background
> would have a different viewpoint.
>
> I think we are losing something if we continue the move away from
> strongly typed languages for various application domains. I really
> like the discipline that strongly typed languages force upon you
> as I think you produce better code as a result.
Agreed. Note, however, that you can still use type information
in other "informal" ways even if the compiler refuses to enforce
*strict* type checking. E.g., defining a synonym for a particular
"built-in" type and using that consistently AS A DOCUMENTATION AID.
"No, this isn't an 'integer' -- even though the underlying
type FOR THIS IMPLEMENTATION happens to be an 'int'. Instead,
it is a 'color' and you should think of it as such when you
operate on it as well as take note of how it is used..."
[When I code in poorly typed languages like C, my code is littered
with superfluous casts to make it clear to the reader that the
object in question needs to (magically) be converted from one type
to some other type.]
The problem with TOO strong of type casting is that it can get in
the way of efficiency (even if there are no run-time costs, it
can add to the cognitive load as readers are forced to perpetually
consider: why is this type conversion needed, here?) as folks
tend to resist thinking PURELY in the abstract (and, thus, can
not avoid thinking that converting a 'color' to an 'integer' is
just a waste of glyphs on the page!)
> PS: I'll let you decide if you consider Ada to be a Wirth language
> or not. He didn't design it but it's very strongly based on his
> language concepts.
Ada is a great example of how type checking can get in the way
of "efficiency". Too few practitioners, too bulky validation efforts,
etc. -- so, too little code placed into use. And, it works *hard* to
keep you from doing things that you "shouldn't do" -- even if you *need*
to do them! ("OK, but you will HAVE TO do them *this* way...")
[One can argue that *released* Ada code may be of higher quality.
But, if it doesn't make it out of the lab, it's LESS useful than
some buggier piece of crapware that *almost* works! You don't see a whole
lot of FOSS *Ada*-built products! :> ]
[toc] | [prev] | [next] | [standalone]
Page 3 of 11 — ← Prev page 1 2 [3] 4 5 … 11 Next page →
Back to top | Article view | comp.arch.embedded
csiph-web