Groups | Search | Server Info | Login | Register

Groups > comp.arch.embedded > #32458

Re: Return of the Overlays!

Show all headers | View raw

On Wed, 14 Jan 2026 13:39:07 -0700, Don Y
<blockedofcourse@foo.invalid> wrote:

>Hi George,
>
>Hoping all is well (Mom?)

Past expiration date.



>>> But, there is nothing in the traditional build process that
>>> ensures references from "some point" in the code don't refer
>>> back to some *other* point (that you thought you were done with).

With discipline it (miserably) can be done. 
See below.

 
>> Granted most existing compilers do not understand the concept of
>> "overlay" per se, but they do certainly understand both definition and
>> execution scope [in many languages these are identical, but in some
>> they can be different].
>> 
>> The problems with overlays almost all are related to closures and
>> [escape] continuations - in C terms: stored function pointers and
>> exceptions (or longjmp).
>
>I am hoping to assume "modules" can't be split (so the compiler
>always knows where local targets reside) and confine the "problem"
>to the linkage editor -- assuming the targeted module can "need
>help" being accessed.
>
>I further assume that "overlay" is a misnomer; that the address space
>is large enough that all targets have unique addresses and just need
>assistance being accessed.  (or, prohibitions against FURTHER access
>if they have been "discarded"/marked "extinct")

Technically, "overlay" does not require sharing of addresses.  It was
often necessary to reuse address space in the limited memory available
to early micros, but the real defining aspect of "overlay" is that the
code is transient and so may not be there (have to be loaded) when it
is called.

Actually, one quite common use of overlays involved customization such
that calling a particular function x() did something different
depending on configuration or user settings.  A lot of early word
processors relied on this for formatting and printing - bringing in
different code depending on what they wanted to do. Today we'd do this
with DLLs - but they (often) did it with code overlays.


>>> The overlay build process had to ensure THIS overlay didn't refer to
>>> anything in THAT overlay.  (Bankswitching code had to rely on "trampoline"
>>> logic in some shared/persistent area to get from one bank to another,
>>> but, there was nothing prohibiting such references!)
>> 
>> Just scope handling.
>
>If they can coexist in a shared address space, that's an issue.
>But, if you're just trying to ensure nothing "here" ever refers
>to something "there" (marked extinct), I don't think it is as
>much of an issue.
>
>The problem becomes one of discipline -- "inherently" knowing how to
>organize your modules/references so you can be "sure" when you execute
>one specific line of code (that marks a section as "extinct") that you
>will NEVER reference anything that it containED (past tense because it
>no longer exists)
>
>Think in terms of FORTRAN's "COMMON" and "CHAIN"; you want to be able
>to put any shared data created in "section 1" into COMMON and then
>fall into the code in the next "section", having the benefit of
>all that "common" data.
>
>Knowing that the code from the first section is forever gone.

Data produced by X and consumed later by Y is not a problem unless X
and Y disagree on the data's format.

Problems are only possible when there can exist stored references to
things which no longer exist.  In Fortran that was not possible: a
COMMON block could not include pointers [even in the Fortrans that had
pointers], and so CHAIN with COMMON could not fail in that way.

Similarly, creating a data structure stored by the main program with
overlay X and then swapping in overlay Y to process the data will not
be a problem so long as the data structure contains no references to
anything inside X.


>[Ideally, I would like to be able ot section off more varied instances
>of resources instead of just chopping a program into discrete CONSECUTIVE
>sections.]
>
>> :
>> More broadly: explicity loaded DLLs [ using dlopen()/dlsym(),
>> LoadLibrary()/GetProcAddress(), etc. ] could be considered modern
>> moral equivalents of overlays.
>
>Yes.  Though you can REload a DLL if you decide you need it
>later.  The CHAIN/COMMON distinction was that prior sections
>are gone -- until you restart the program.

You're confused.

With explicit DLL management, the programmer has to deliberately write
code to load a DLL and map its exported API to function pointers.  If
the DLL then is unloaded, the mapped pointers become garbage: trying
to call the functions will _NOT_ reload the DLL - rather it will,
almost certainly, crash the program.

If the program was written in a CHAIN/COMMON fashion, using and
disgarding a progression of DLLs, then there will be NO programmer
supplied code to "go back" and reload one of them.



>>> ... (where the overlay has to
>>> replace the existing overlay at run time)?  Besides personal
>>> familiarity with the codebase, how do you select code portions
>>> to place in each overlay?
>> 
>> Deliberately ignoring namespaces and "static", the problem with C is
>> that functions all are defined globally, so any function potentially
>> can be called from anywhere.
>> 
>> Think instead how you would code it in Pascal ... or any language
>> having nested functions but without using closures.  Functions that
>> support specific computations (as opposed to generally useful) should
>> be defined / in scope only where they need to be callable.
>> 
>> Disjoint definition scopes then are exactly analogous to overlays, and
>> the functions defined within them can be packaged appropriately.
>
>But, that assumes those objects are NEVER accessed -- despite the
>obvious need to access them /at least once/ to gain entry to that domain.

Again confused.  Consider the following and [for simplicity] assume
Pascal scoping rules:

  MAIN
    F1
      G1, H1
    F2
      G2, H2

For the purposes of illustration here we will disallow recursive calls
to the F's.  With that stipulation, it should be obvious that  G1/H1
and  G2/H2  exist in disjoint scopes - unable to see each other.  

Eg., G2 can't call G1 or H1, or reference any data internal to F1.  
[In Pascal, G2 could _call_ F1, but we're disallowing that here.]



All that is needed for an overlay is the entry point. There may be
more than one, but that is not relevant here.  F1 is the entry point
that gets you to G1 and H1.  (see below).


The analogy is not perfect: Pascal doesn't understand overlays. But
its scope rules make overlays relatively easy to support.

You'd have even more control using scope rules from Scheme or ML, but
I assume [perhaps wrongly] that - in this forum - readers likely would
be more familiar with Pascal.


>> Note that grafting this onto a language lacking the notion of nested
>> functions will not be easy.
>
>I can handle some cases easily:
>
>main() {
>     Initialize()
>
>     Extinctify(&Initialize)
>
>     DoWork()
>}
>
>and place "Initialize" in its own section, commanding the linker to locate
>it "conveniently" (likely on a page-frame boundary soas to maximize the
>amount of usable space in that page-frame).
>
>But, aside from such obvious choices, I think it is hard to mentally
>subdivide a piece of code for such a partitioning.  And, then, remembering
>to "extinctify" portions that you no longer need.  You'd have to be
>keenly aware of the cost of each such "portion" of the algorithm so
>you could identify things that could/should be excised.

It's hard because you work in languages that make it hard(er).


The most straightforward way to arrange the code in C would be to
separate the disjoint "overlay" scopes by source file / compilation
unit.  

pseudo'ing the example above in C:

  ----------------
  F1();
  F2();
  main()
  {
     F1();
     F2();
  }
  ----------------
  static G1() {}
  static H1() {}
  F1()
  {
   ...
  }   
  ----------------
  static G2() {}
  static H2() {}
  F2()
  {
   ...
  }   
  ----------------

where the horizontal lines denote source file / compilation unit
boundaries.  

This way the compiler will catch most problems, and the code will be
in modules that potentially could be loaded/unloaded independently
(assuming you have a way to do that).

But, as you said, it takes some dicipline.

The best way is to just use DLLs if you can.  Export only the entry
points and everything else will be hidden.

George

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-13 17:03 -0700
  Re: Return of the Overlays! George Neuner <gneuner2@comcast.net> - 2026-01-14 01:11 -0500
    Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-14 13:39 -0700
      Re: Return of the Overlays! George Neuner <gneuner2@comcast.net> - 2026-01-15 03:06 -0500
        Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-15 03:25 -0700
          Re: Return of the Overlays! George Neuner <gneuner2@comcast.net> - 2026-01-17 01:23 -0500
  Re: Return of the Overlays! John Levine <johnl@taugh.com> - 2026-01-14 20:39 +0000
    Re: Return of the Overlays! Grant Edwards <invalid@invalid.invalid> - 2026-01-14 21:00 +0000
      Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-14 18:55 -0700
    Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-14 18:48 -0700
  Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-25 01:46 -0700
    Re: Return of the Overlays! Don Y <blockedofcourse@foo.invalid> - 2026-01-25 02:10 -0700

csiph-web