Groups | Search | Server Info | Login | Register

Groups > comp.arch.embedded > #32460

Re: Return of the Overlays!

Show all headers | View raw

On Thu, 15 Jan 2026 03:25:30 -0700, Don Y
<blockedofcourse@foo.invalid> wrote:


>>>>> But, there is nothing in the traditional build process that
>>>>> ensures references from "some point" in the code don't refer
>>>>> back to some *other* point (that you thought you were done with).
>> 
>> With discipline it (miserably) can be done.
>> See below.
>
>In addition to knowing what MIGHT be accessible (exported), you
>also have to track down every reference to each of those and
>identify when, in time, they occur (or might occur) relative to
>your explicitly declaring them to be "no longer needed".
>
>THAT is the tough part -- a link map tells you "what talks to
>what" but says nothing about when, in time, those interactions occur.
>If FOO calls BAR and BAR calls BAZ, then anything that calls
>BAZ, BAR *or* FOO, at a point in time AFTER you have declared
>BAZ to no longer be needed will SIGSEGV.
>
>You have to keep track of these interdependencies when making
>decisions declaring particular "identifiers" (code or data)
>to be no longer needed.

No ... the module simply need to reference count active calls.
[Unless there can be circular dependencies among modules - which you
should disallow (or see below).]


If you load the module explicitly, calls into it will have to be made
through mapped function pointers, so if you null[*] the pointers when
first *requesting* an unload then you can prevent new calls while
waiting for active calls to complete.

When you load the module you initialize its count to  1.
When you unload you subtract 1 from the count (allowing it to zero).

Each *exported* function increments the count on entry and decrements
it on exit.  When the last active call exits the count will to go to
zero, and at that point the module is not in use and can be unloaded. 
You can track the count inside the module as cleanup to each function,
or you can poll it from outside the module.  Works either way.

This is the way COM controls work, and the way Corba used to work.
[Haven't looked at Corba for quite a while.]


[*] or point them to an error handler




>The "Initialization()" example I provided is easily to conceive of
>occurring exactly once in a "program's" execution.  So, as long as
>it is only invoked once AND THERE ARE NO PATHS BACK TO Main() -- in
>my example -- there is no way it can be reaccessed/referenced after
>the "extinctify" invocation that follows it.
>
>Can you say that about string(3C) functions -- can you decide
>when those library functions are no longer needed, thereby
>freeing up the resources that they require?  Or, some function
>that you wrote to handle a particular error condition?

Yes, I can.  BTDTGTTS.


>>> Think in terms of FORTRAN's "COMMON" and "CHAIN"; you want to be able
>>> to put any shared data created in "section 1" into COMMON and then
>>> fall into the code in the next "section", having the benefit of
>>> all that "common" data.
>>>
>>> Knowing that the code from the first section is forever gone.
>> 
>> Data produced by X and consumed later by Y is not a problem unless X
>> and Y disagree on the data's format.
>
>The format isn't the issue.  Rather, that X /no longer exists/.
>References to the address(es) that it occupied aren't mapped to
>anything, anymore.

If you're following the Fortran metaphor, there won't be any
references to missing modules BECAUSE there won't be any function
pointers contained in the data.  

Modules should export only functions - not data objects.

Data pointers can be allowed because the data exists outside of any
transient module.  The "main" program has to provide the storage.




>You must be sure Y (and every other X consumer) have made use
>of X before you unmap its resources.  THAT is what is hard to track
>because tools don't lay out the temporal relationships of various
>identifiers.
>
>> Problems are only possible when there can exist stored references to
>> things which no longer exist.
>
>The references need not be explicitly "stored" but, rather, can be
>part of an instruction stream generated by a compiler.

No they can't be.  The only way to reference a module is via function
pointers that are explicitly mapped.  Yes, the compiler could try to
call a function through an unmapped pointer ... and that would be bad.

So what?  That's a bug in the program.


>>  In Fortran that was not possible: a
>> COMMON block could not include pointers [even in the Fortrans that had
>> pointers], and so CHAIN with COMMON could not fail in that way.
>
>I mention COMMON and CHAIN because they implement the mechanisms that
>must be present for one part of a "program" to do work and pass those
>results to a followup part (via COMMON) while execution is passed
>to that followup part (via CHAIN).
>
>You (typ) need data and code bridges to connect separate parts of a program
>together, esp if you are deliberately trying to cut a program into smaller
>pieces to reduce its "current" footprint (for any value of "current")

Remember DOS?  Again, BTDTGTTS. 

But realize that, even with Fortran, you could invoke the same
subprogram repeatedly if the COMMON data structure it expected was
there.

With Fortran90's modules it became easier, but could be done with
COMMON and CHAIN also.


>> Similarly, creating a data structure stored by the main program with
>> overlay X and then swapping in overlay Y to process the data will not
>> be a problem so long as the data structure contains no references to
>> anything inside X.
>
>That doesn't address having the data structure overlaid (or, in
>my case, unmapped) because *IT* was considered "no longer needed".

That is up to the loader code, which ALWAYS should preserve the
program's heap.  Again, no data should be stored BY the modules on
behalf of the program. Data for the main program always should be
returned to the main program for storage.

The main program doesn't necessarily need to understand the data ...
it can be treated as opaque and require some module(s) to do anything
with it ... but the main program has to be responsible for the
storage.


>At some point in the "earlier" execution of the "program", that data
>structure had meaning.  But, one the developer declares that data to
>no longer be needed, it disappears.  Any code that later references it
>(in error!) crashes.  No, it doesn't get the wrong values for the data
>or interpret the values incorrectly.  The memory reference simply FAILS.

Again, that's a bug in the program.



>> With explicit DLL management, the programmer has to deliberately write
>> code to load a DLL and map its exported API to function pointers.  If
>> the DLL then is unloaded, the mapped pointers become garbage: trying
>> to call the functions will _NOT_ reload the DLL - rather it will,
>> almost certainly, crash the program.
>
>That is EXACTLY the situation I am describing.  The developer explicitly
>decided the DLL was NO LONGER NEEDED.  *He* unloaded it.  If he wasn't
>disciplined enough to know that he wasn't yet done with it, then his
>program crashes.

So what?


>However, he can choose to unload the DLL (to reduce his resource
>usage) and then, at some later time, explicitly REload it as if
>for the first time and make continued use of it.
>
>With CHAIN/COMMON, the past is past.  The code that preceded is no longer
>available (unless you rerun the program).

Maybe.  Again, a Fortran subprogram (or module) can be invoked any
time its initial conditions are met.



>... my deliberate marking of portions of that memory as "no longer
>needed" (mapped) discards them and their contents, irretrievably.
>The only way to recreate the data and code is to restart the
>program.

Then you are doing it wrong.  Transient modules can't be allowed to
store data.



>You have some codebase.  Tools will tell you where a particular
>identifier is referenced.  You can use that to build a list
>of other objects that implicitly reference that identifier.
>
>*What* is going to tell you WHEN any of those  objects are
>invoked, relative to a particular statement that is executed
>at a specific point in time:
>     unmap the memory used by identifier X

Reference count.  Better yet a tracing GC, but that would have to be
embedded into the runtime or operating system.  Refcount can be done
by the modules themselves.

>------
>X4()
>{
>    X7()
>}
>------
>X5()
>{
>    F2()
>}
>-----
>X6()
>{
>    if (blah)
>       X4()
>}
>-----
>X7()
>{
>    F1()
>}
>----
>X8()
>{
>     X4()
>}
>
>[This isn't really far-fetched with libraries that can be interrelated or
>interact in other ways]
>
>At some point in time, in some module, I decide to unmap the resources that
>F1 consumes.  Maybe that happens conditionally.
>
>But, at some later point in time, X8() is invoked.  Or, X4.  Or, X7,  Or,
>X6 with blah being true.
>
>You have to track the dependency hierarchy for all of these "objects"
>and ensure that none of them are invoked *after* you have unmapped F1.
>Because you can't recreate F1, its resources or its actual "value"/meaning.
>
>And, you can't exhaustively test to ensure that every possible path through
>the application is exercised.  You could easily have a latent bug that
>only surfaces in some particular set of circumstances that you didn't
>consider, test or encounter, before.

You don't need to test other than the program runs, but if you have
circular dependencies you need to TRACE/TRACK them AT RUNTIME because
refcount won't work (at least not without a tracing backup to break
circles).


>We aren't accustomed to having objects (code or data) disappear during
>the course of developing or executing a piece of code.  If "foo"
>existed at some point in the program -- and you are executing in
>the same context and scope, then why would you NOT expect it to still
>be there?

If I loaded it and haven't yet unloaded it, it ought to be there
regardless of what any other code has done with it.  If this is not
true, then the system is unstable.


>You can do an RPC and then be surprised that the NEXT invocation
>fails -- because the remote host disappeared.  You code against
>that possibility.

If you are treating EVERY function call as if RPC, then your programs
will be 2% functionality and 98% error handling.  And few people ever
will be able to write code for your system.


>I just delete objects and whatever they contain/represent goes away
>in that action.  Whether it is a collection of data that I no longer
>need, a group of functions, a subassembly, etc.  The developer
>has to decide how to group "things" for their utility and temporal
>pertinence.

And so you have to guide programmers toward a structure that works -
not just let them do whatever they wish.

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-13 17:03 -0700
  Re: Return of the Overlays! George Neuner <gneuner2@comcast.net> - 2026-01-14 01:11 -0500
    Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-14 13:39 -0700
      Re: Return of the Overlays! George Neuner <gneuner2@comcast.net> - 2026-01-15 03:06 -0500
        Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-15 03:25 -0700
          Re: Return of the Overlays! George Neuner <gneuner2@comcast.net> - 2026-01-17 01:23 -0500
  Re: Return of the Overlays! John Levine <johnl@taugh.com> - 2026-01-14 20:39 +0000
    Re: Return of the Overlays! Grant Edwards <invalid@invalid.invalid> - 2026-01-14 21:00 +0000
      Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-14 18:55 -0700
    Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-14 18:48 -0700
  Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-25 01:46 -0700
    Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-25 02:10 -0700

csiph-web