Groups | Search | Server Info | Login | Register

Groups > comp.arch.embedded > #32453

Re: Return of the Overlays!

Show all headers | View raw

Hi George,

Hoping all is well (Mom?)

>> I'd like to be able to shed resources that are no longer
>> needed.  But, have some assurances that they truly *aren't*
>> needed (referenced) going forward.  This lets the system free
>> up those resources for use by other applications (and runtime
>> diagnostics, once you know "no one" is using a resource).
>>
>> [I can do this automatically or let the developer do it "on demand"
>> by lumping resources in specific sections with judicious use
>> of linker scripts, etc.  Applications developed with this
>> in mind will tend to be more resilient because they will tend
>> to be allowed to continue execution when the system is running
>> in overload; applications that hold onto unneeded resources
>> will look "more wasteful" and selected for removal.]
>>
>> But, there is nothing in the traditional build process that
>> ensures references from "some point" in the code don't refer
>> back to some *other* point (that you thought you were done with).
> 
> Granted most existing compilers do not understand the concept of
> "overlay" per se, but they do certainly understand both definition and
> execution scope [in many languages these are identical, but in some
> they can be different].
> 
> The problems with overlays almost all are related to closures and
> [escape] continuations - in C terms: stored function pointers and
> exceptions (or longjmp).

I am hoping to assume "modules" can't be split (so the compiler
always knows where local targets reside) and confine the "problem"
to the linkage editor -- assuming the targeted module can "need
help" being accessed.

I further assume that "overlay" is a misnomer; that the address space
is large enough that all targets have unique addresses and just need
assistance being accessed.  (or, prohibitions against FURTHER access
if they have been "discarded"/marked "extinct")

>> The overlay build process had to ensure THIS overlay didn't refer to
>> anything in THAT overlay.  (Bankswitching code had to rely on "trampoline"
>> logic in some shared/persistent area to get from one bank to another,
>> but, there was nothing prohibiting such references!)
> 
> Just scope handling.

If they can coexist in a shared address space, that's an issue.
But, if you're just trying to ensure nothing "here" ever refers
to something "there" (marked extinct), I don't think it is as
much of an issue.

The problem becomes one of discipline -- "inherently" knowing how to
organize your modules/references so you can be "sure" when you execute
one specific line of code (that marks a section as "extinct") that you
will NEVER reference anything that it containED (past tense because it
no longer exists)

Think in terms of FORTRAN's "COMMON" and "CHAIN"; you want to be able
to put any shared data created in "section 1" into COMMON and then
fall into the code in the next "section", having the benefit of
all that "common" data.

Knowing that the code from the first section is forever gone.

[Ideally, I would like to be able ot section off more varied instances
of resources instead of just chopping a program into discrete CONSECUTIVE
sections.]

> The issue with (classic) C was it had only global and local scopes.
> Supporting overlays required the addition of non-global-non-local
> scopes [at the load module level at least], and a bit of runtime magic
> behind the scenes to load/unload them >
>> Does anyone still develop with overlays ...
> 
> I haven't seen an overlay compiler since 8-bit.  Of course, 8-bit
> cpus/mpus still are used in the embedded world.
> 
> More broadly: explicity loaded DLLs [ using dlopen()/dlsym(),
> LoadLibrary()/GetProcAddress(), etc. ] could be considered modern
> moral equivalents of overlays.

Yes.  Though you can REload a DLL if you decide you need it
later.  The CHAIN/COMMON distinction was that prior sections
are gone -- until you restart the program.

Imagine constant pressure on pages to be swapped out.  So,
they effectively end up swapped out once they are no longer
being referenced.  Now, at some point, disable the backing
store so they can never be faulted back *in*.

This is fine -- if there is never a NEED to re-reference
them.  But, a disaster (SIGSEGV) if something forgot that
there was one more outstanding reference.

If you test, thoroughly, you should be able to catch all
such references.  But, chances are, you'll miss some exception
(not "Exception") that requires access to something in that
"old" section.  But, the runtime won't miss it -- it will
segfault and crash your "program".

> Though they may not share memory (or other resources) and their
> coming/going is not handled automagically by the compiler runtime,
> they are similar [in the geometric meaning] in that: they might not be
> loaded/mapped when needed, they may be nested, and storing references
> to anything defined or created by them can be frought with danger.

Exactly.  And, you can "never" be sure that you have examined/tested
all such use cases.  How many little utility/helper routines do you
reference throughout your code without ever remembering to place them
in the "SHARED" section so they are *always* present?

> Granted, few developers ever muck with explicit DLL management ...
> most will just let the linker/runtime/loader handle it.

And, loading a DLL still, conceptually, requires a trip to the
disk subsystem (even if cached).

>> ... (where the overlay has to
>> replace the existing overlay at run time)?  Besides personal
>> familiarity with the codebase, how do you select code portions
>> to place in each overlay?
> 
> Deliberately ignoring namespaces and "static", the problem with C is
> that functions all are defined globally, so any function potentially
> can be called from anywhere.
> 
> Think instead how you would code it in Pascal ... or any language
> having nested functions but without using closures.  Functions that
> support specific computations (as opposed to generally useful) should
> be defined / in scope only where they need to be callable.
> 
> Disjoint definition scopes then are exactly analogous to overlays, and
> the functions defined within them can be packaged appropriately.

But, that assumes those objects are NEVER accessed -- despite the
obvious need to access them /at least once/ to gain entry to that domain.

> Note that grafting this onto a language lacking the notion of nested
> functions will not be easy.

I can handle some cases easily:

main() {
     Initialize()

     Extinctify(&Initialize)

     DoWork()
}

and place "Initialize" in its own section, commanding the linker to locate
it "conveniently" (likely on a page-frame boundary soas to maximize the
amount of usable space in that page-frame).

But, aside from such obvious choices, I think it is hard to mentally
subdivide a piece of code for such a partitioning.  And, then, remembering
to "extinctify" portions that you no longer need.  You'd have to be
keenly aware of the cost of each such "portion" of the algorithm so
you could identify things that could/should be excised.

[Remember, you can group several small resources into a larger
resource -- e.g., page -- that would merit excision whereas
any of the smaller resources would seem insignificant.  Look
at command line parsing -- and ALL the related code -- to see
how quickly this adds up (don't forget crt0.s!)]

The link map could help MANUALLY verify that there are no other, LATER
references to something in one of those extinct-ed areas.  But, I
don't see an easy way (language agnostic) to automate this to provide
a high degree of confidence that you won't throw a SIGSEGV at some
*distant* future date when some combination of events conspires to
invoke an identifier (data or text) that you'd previously thought
"no longer of use".

>> Note, this is different than paged VMM where individual pages are
>> swapped in and out on demand (I have no backing store so once "out",
>> coming back *in* is costly).
> 
> George

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-13 17:03 -0700
  Re: Return of the Overlays! George Neuner <gneuner2@comcast.net> - 2026-01-14 01:11 -0500
    Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-14 13:39 -0700
      Re: Return of the Overlays! George Neuner <gneuner2@comcast.net> - 2026-01-15 03:06 -0500
        Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-15 03:25 -0700
          Re: Return of the Overlays! George Neuner <gneuner2@comcast.net> - 2026-01-17 01:23 -0500
  Re: Return of the Overlays! John Levine <johnl@taugh.com> - 2026-01-14 20:39 +0000
    Re: Return of the Overlays! Grant Edwards <invalid@invalid.invalid> - 2026-01-14 21:00 +0000
      Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-14 18:55 -0700
    Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-14 18:48 -0700
  Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-25 01:46 -0700
    Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-25 02:10 -0700

csiph-web