Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.msdos.programmer > #1390 > unrolled thread

huge memory model, malloc() and sytem()

Back to article view | Back to comp.os.msdos.programmer

Contents

  huge memory model, malloc() and sytem() "Alexei A. Frounze" <alexfrunews@gmail.com> - 2014-06-03 01:01 -0700
    Re: huge memory model, malloc() and sytem() "Rod Pemberton" <dont_use_email@xnothavet.cqm> - 2014-06-03 06:11 -0400
      Re: huge memory model, malloc() and sytem() "Alexei A. Frounze" <alexfrunews@gmail.com> - 2014-06-03 03:46 -0700
        Re: huge memory model, malloc() and sytem() Ross Ridge <rridge@csclub.uwaterloo.ca> - 2014-06-03 18:49 -0400
        Re: huge memory model, malloc() and sytem() "Rod Pemberton" <dont_use_email@xnothavet.cqm> - 2014-06-04 03:14 -0400

#1390 — huge memory model, malloc() and sytem()

I'm thinking of a heap implementation in a compiler targeting DOS and I'm a bit unsure how to do it.

You see, if the compiled program uses either the tiny or the small memory model (with all pointers being near), it's confined to 64-128KB, within which the heap resides, and there's plenty of memory left for this program to execute another program (via system() and the like).

Now, if the program uses the huge memory model (with all pointers being far), its heap can occupy whatever is left after loading the program into the RAM. If I set the EXE header's maximum alloc field to the max, then all (or almost all of) the memory will be allocated at load time and there will be nothing left to execute another program. So, that's definitely not a good thing to do.

Let's see what the options are.

First of all, I could, of course, set up the EXE header appropriately to limit the amount of memory DOS would reserve at load time and thus limit the size of the heap. But whichever value I choose, it might be too little or too large, depending on what the program is actually going to do. I'd like to avoid hard coding the value or having to specify it explicitly at compile time.

Next, I could pre-allocate nothing per the EXE header and instead simply ask DOS to allocate memory for me and have malloc() and free() pretty much thin wrappers around the appropriate int 21h functions. While that would definitely work, I will be at the mercy of DOS and I'll have to swallow the overhead of the memory control block (AKA MCB) preceding every allocated block of memory. The MCB, if you didn't know, is 16 bytes long, and that is at least twice as big as necessary. Also, there's the overhead of allocating anything smaller than 16 bytes because int 21h memory functions work with 16-byte chunks. I'd like to avoid these overheads as well.

Which got me thinking...

How acceptable would be (was something like it a common practice in other compilers?) to allocate all the memory from DOS, manage it manually, and then, inside system() free whatever memory isn't used by creating artificial MCBs for DOS and then get it back from DOS when system() is over?

Basically, I want to be flexible w.r.t. default/hard-coded sizes and yet with as little MCB overhead as possible. All that without driving DOS crazy.

Any other options?

Thanks,
Alex

[toc] | [next] | [standalone]

#1391

On Tue, 03 Jun 2014 04:01:17 -0400, Alexei A. Frounze  
<alexfrunews@gmail.com> wrote:

> I'm thinking of a heap implementation in a compiler targeting
> DOS and I'm a bit unsure how to do it.
>

May I take it that you're not wanting me to mention Doug Lea's
dlmalloc, John Walker's BGET, and Richard Harter's DSA, yet again.
Yes?  (i.e., a.o.d. 2006, 2007, 2007 ...)

> You see, if the compiled program uses either the tiny or the small  
> memory model (with all pointers being near), it's confined to 64-128KB,  
> within which the heap resides, and there's plenty of memory left for  
> this program to execute another program (via system() and the like).
>
> Now, if the program uses the huge memory model (with all pointers being  
> far), its heap can occupy whatever is left after loading the program  
> into the RAM. If I set the EXE header's maximum alloc field to the max,  
> then all (or almost all of) the memory will be allocated at load time  
> and there will be nothing left to execute another program. So, that's  
> definitely not a good thing to do.
>
> Let's see what the options are.
>
> First of all, I could, of course, set up the EXE header appropriately to  
> limit the amount of memory DOS would reserve at load time and thus limit  
> the size of the heap. But whichever value I choose, it might be too  
> little or too large, depending on what the program is actually going to  
> do. I'd like to avoid hard coding the value or having to specify it  
> explicitly at compile time.
>
> Next, I could pre-allocate nothing per the EXE header and instead simply  
> ask DOS to allocate memory for me and have malloc() and free() pretty  
> much thin wrappers around the appropriate int 21h functions. While that  
> would definitely work, I will be at the mercy of DOS and I'll have to  
> swallow the overhead of the memory control block (AKA MCB) preceding  
> every allocated block of memory. The MCB, if you didn't know, is 16  
> bytes long, and that is at least twice as big as necessary. Also,  
> there's the overhead of allocating anything smaller than 16 bytes  
> because int 21h memory functions work with 16-byte chunks. I'd like to  
> avoid these overheads as well.
>
> Which got me thinking...
>
> How acceptable would be (was something like it a common practice in  
> other compilers?) to allocate all the memory from DOS, manage it  
> manually, and then, inside system() free whatever memory isn't used by  
> creating artificial MCBs for DOS and then get it back from DOS when  
> system() is over?
>
> Basically, I want to be flexible w.r.t. default/hard-coded sizes and yet  
> with as little MCB overhead as possible. All that without driving DOS  
> crazy.
>
> Any other options?
>

Well, I came late to the programming DOS thing, around 2004.  AIUI,
DOS memory allocation calls will only allow you to allocate below 1MB.
That's acceptable for small memory models, but really, you want to
access all that wonderful, uncluttered, bountiful memory above 1MB
without any restrictions.  Yes?

There is EMS, XMS, VCPI, DPMI, etc used for memory allocation and
PM environments in DOS.  However, I like the combination of XMS and
DPMI.  XMS 3.0 can allow you to access upto 4GB.  Many DPMI hosts will
use XMS for their memory allocation calls when XMS is present.  Some
XMS clients will use BIOS E820h call instead of E801h or earlier calls.
So, I'd suggest either XMS or DPMI, if you're wanting to use an
existing memory manager.  If you use Microsoft's HIMEM, then Windows
98/SE can "inherit" the mapped XMS memory via a special interface.

If you're not wanting a memory manager, then I'd suggest the
memory allocators mentioned above and on a.o.d. in 2007 or so.

Of course, the *fun* is to create your own.  ;-)  That's when the
insanity or genious is expressed in code.  That's when you get to
test out your ideas for speed or effectiveness or simplicity.


Japheth HIMEMX
http://www.japheth.de/Jemm.html


Rod Pemberton

[toc] | [prev] | [next] | [standalone]

#1392

On Tuesday, June 3, 2014 3:11:30 AM UTC-7, Rod Pemberton wrote:
> On Tue, 03 Jun 2014 04:01:17 -0400, Alexei A. Frounze  
> 
> <alexfrunews@gmail.com> wrote:
> 
> > I'm thinking of a heap implementation in a compiler targeting
> > DOS and I'm a bit unsure how to do it.
> >
> 
> May I take it that you're not wanting me to mention Doug Lea's
> dlmalloc, John Walker's BGET, and Richard Harter's DSA, yet again.
> Yes?  (i.e., a.o.d. 2006, 2007, 2007 ...)

Nope. :) I'm perfectly aware of Doug's memory manager.

> > You see, if the compiled program uses either the tiny or the small  
> > memory model (with all pointers being near), it's confined to 64-128KB,  
> > within which the heap resides, and there's plenty of memory left for  
> > this program to execute another program (via system() and the like).
> >
> > Now, if the program uses the huge memory model (with all pointers being  
> > far), its heap can occupy whatever is left after loading the program  
> > into the RAM. If I set the EXE header's maximum alloc field to the max,  
> > then all (or almost all of) the memory will be allocated at load time  
> > and there will be nothing left to execute another program. So, that's  
> > definitely not a good thing to do.
> >
> > Let's see what the options are.
> >
> > First of all, I could, of course, set up the EXE header appropriately to  
> > limit the amount of memory DOS would reserve at load time and thus limit  
> > the size of the heap. But whichever value I choose, it might be too  
> > little or too large, depending on what the program is actually going to  
> > do. I'd like to avoid hard coding the value or having to specify it  
> > explicitly at compile time.
> >
> > Next, I could pre-allocate nothing per the EXE header and instead simply  
> > ask DOS to allocate memory for me and have malloc() and free() pretty  
> > much thin wrappers around the appropriate int 21h functions. While that  
> > would definitely work, I will be at the mercy of DOS and I'll have to  
> > swallow the overhead of the memory control block (AKA MCB) preceding  
> > every allocated block of memory. The MCB, if you didn't know, is 16  
> > bytes long, and that is at least twice as big as necessary. Also,  
> > there's the overhead of allocating anything smaller than 16 bytes  
> > because int 21h memory functions work with 16-byte chunks. I'd like to  
> > avoid these overheads as well.
> >
> > Which got me thinking...
> >
> > How acceptable would be (was something like it a common practice in  
> > other compilers?) to allocate all the memory from DOS, manage it  
> > manually, and then, inside system() free whatever memory isn't used by  
> > creating artificial MCBs for DOS and then get it back from DOS when  
> > system() is over?
> >
> > Basically, I want to be flexible w.r.t. default/hard-coded sizes and yet  
> > with as little MCB overhead as possible. All that without driving DOS  
> > crazy.
> >
> > Any other options?
> >
> 
> Well, I came late to the programming DOS thing, around 2004.  AIUI,
> DOS memory allocation calls will only allow you to allocate below 1MB.
> That's acceptable for small memory models, but really, you want to
> access all that wonderful, uncluttered, bountiful memory above 1MB
> without any restrictions.  Yes?

No. The question is about a memory manager working in real (or virtual 8086) mode and its interaction with DOS.

> There is EMS, XMS, VCPI, DPMI, etc used for memory allocation and
> PM environments in DOS.  However, I like the combination of XMS and
> DPMI.  XMS 3.0 can allow you to access upto 4GB.  Many DPMI hosts will
> use XMS for their memory allocation calls when XMS is present.  Some
> XMS clients will use BIOS E820h call instead of E801h or earlier calls.
> So, I'd suggest either XMS or DPMI, if you're wanting to use an
> existing memory manager.  If you use Microsoft's HIMEM, then Windows
> 98/SE can "inherit" the mapped XMS memory via a special interface.
> 
> If you're not wanting a memory manager, then I'd suggest the
> memory allocators mentioned above and on a.o.d. in 2007 or so.
> 
> Of course, the *fun* is to create your own.  ;-)  That's when the
> insanity or genious is expressed in code.  That's when you get to
> test out your ideas for speed or effectiveness or simplicity.
> 
> Japheth HIMEMX
> http://www.japheth.de/Jemm.html

I've written a couple of simple memory managers, it's not a problem. Nor is having a very-very optimal memory manager.

The specific question I'm having is how to best marry memory management with DOS in the absence of DPMI, VCPI, protected mode, himem.sys and all the rest, in real mode (or virtual 8086, as it should be transparent and unimportant).

I think I described the problem rather clearly in the original post. But if it's the intent that is unclear, it's simple. I want my C compiler to target DOS and make it possible for programs compiled with it to use all the conventional memory (that is, under 1MB) via malloc() while staying in the same CPU mode (real or v86). Despite all the shortcomings if the setup/scenario, there's a subtle benefit here, btw. You know that there are few good DPMI hosts. You should also know that DPMI/NTVDM is broken in Windows XP and because of that DJGPP had (still has?) issues with nested DPMI programs when run under Windows XP and later. Well, if you don't need all the memory, you can stay in 16-bit mode and run nested programs as long as everything (including DOS) fits into ~640 KB. But I digress. Anyway, here's the intent. Should be clear now, too.

Can you help solving this particular problem and not another, unspoken one?

Alex

[toc] | [prev] | [next] | [standalone]

#1399

Alexei A. Frounze <alexfrunews@gmail.com> wrote:
>Can you help solving this particular problem and not another, unspoken one?

Ahem.

I believe most 16-bit compilers set the max memory required in the EXE
header to FFFFh so that all of conventional memory (or at least the
largest contiguous chunk) was allocated to the program at start up.
However, the startup code would then resize this initial allocation to
the size actually needed.

After that I'm not entirely sure how memory allocation was handled.
Near mallocs were allocated out of a heap maintained by the C runtime by
necessity, but I wouldn't be surprised if in at least some implementations
far mallocs weren't managed by the C runtime and just called the MS-DOS
allocation routines each time memory was allocated or freed.

It looks like Borland C manages its own far heap, but it allocates memory
from DOS as needed.  It doesn't allocate all of memory at startup.

If I were you I wouldn't worry about the overhead of MCBs and just punt
to MS-DOS for your far heap implementation.  I'd only implement my own
heap if the overhead proved to be significant enough problem in practice.

					Ross Ridge

-- 
 l/  //	  Ross Ridge -- The Great HTMU
[oo][oo]  rridge@csclub.uwaterloo.ca
-()-/()/  http://www.csclub.uwaterloo.ca/~rridge/ 
 db  //	  

[toc] | [prev] | [next] | [standalone]

#1404

On Tue, 03 Jun 2014 06:46:10 -0400, Alexei A. Frounze  
<alexfrunews@gmail.com> wrote:

> The specific question I'm having is how to best marry memory management  
> with DOS in the absence of DPMI, VCPI, protected mode, himem.sys and all  
> the rest, in real mode (or virtual 8086, as it should be transparent and  
> unimportant).
>
> [...]
>
> I want my C compiler to target DOS and make it possible for programs
> compiled with it to use all the conventional memory (that is, under 1MB)
> via malloc() while staying in the same CPU mode (real or v86).

Well, I think allocating all the memory from DOS and managing it yourself
as you mentioned is probably the place to start.  I.e., you can control
the header size used to manage each memory region, or not even use a
header, e.g., tables, stacks, linked-lists, etc.  If you can create a
more efficient method than DOS, it's probably worth doing.  I'm not
familiar with how DOS internally manages memory.  If you don't need
historic DOS memory management, that's a bonus, but very few use DOS
without XMS.

As for creating "artificial MCBs for DOS" and somehow telling DOS about
them, I'm not sure how that would work, or if it could.  I know Windows 98
has a special interface to "inherit" XMS memory structures, but I've never
heard of a method to tell DOS about MCBs created outside of DOS.  Of  
course,
there are examples of programs which patch DOS structures.

As for intending to use "all the conventional memory".  C needs contiguous
memory.  You can use most of the conventional memory, i.e., below 640K,
or below 512K to be safe since early machines stopped at 512K, minus
in-use low memory IVT, BDA, etc.  You could also allocate C objects in
the free memory blocks between 640KB and 1MB, but then you might need
to start keeping track of the ranges of valid memory so you don't
allocate something too large for the upper block, or resize the memory
and exceed the block size.  If you're going to start managing blocks
and not use existing DOS solutions, then perhaps a version of XMS could
be implemented within compiler...  No XMS, your version goes into effect.
XMS present, external XMS used.  This way, you don't have to write the
code to call two different functions.  I.e., you don't have to select
between calling your non-XMS memory manager and calling XMS.

As for allocating vie EXE header, I would probably set that very
conservatively, but not zero it, e.g., 1KB, 2KB, 4KB, and hope that
I got the dynamic heap memory allocation working correctly.  You might
test out how much you really need for a typical minimal heap.  It might
be far less than you think, e.g., 512 bytes.

Was some aspect of this about dealing with differently sized segments,
or 16-bit segment:offset versus 32-bit offsets?


Rod Pemberton

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.msdos.programmer

csiph-web