Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #25398 > unrolled thread

g++ on Cortex-M with no dynamic memory

Back to article view | Back to comp.arch.embedded

Contents

  g++ on Cortex-M with no dynamic memory Dave Nadler <drn@nadler.com> - 2016-11-05 14:09 -0700
    Re: g++ on Cortex-M with no dynamic memory antispam@math.uni.wroc.pl - 2016-11-07 02:55 +0000
      Re: g++ on Cortex-M with no dynamic memory Tom Gardner <spamjunk@blueyonder.co.uk> - 2016-11-07 07:52 +0000
        Re: g++ on Cortex-M with no dynamic memory Wouter van Ooijen <wouter@voti.nl> - 2016-11-07 09:11 +0100
          Re: g++ on Cortex-M with no dynamic memory Tim Wescott <tim@seemywebsite.com> - 2016-11-07 10:08 -0600
        Re: g++ on Cortex-M with no dynamic memory David Brown <david.brown@hesbynett.no> - 2016-11-07 10:02 +0100
        Re: g++ on Cortex-M with no dynamic memory Dave Nadler <drn@nadler.com> - 2016-11-08 12:01 -0800
      Re: g++ on Cortex-M with no dynamic memory Dave Nadler <drn@nadler.com> - 2016-11-07 04:13 -0800
        Re: g++ on Cortex-M with no dynamic memory Tim Wescott <tim@seemywebsite.com> - 2016-11-07 10:15 -0600
          Re: g++ on Cortex-M with no dynamic memory Wouter van Ooijen <wouter@voti.nl> - 2016-11-07 18:39 +0100
            Re: g++ on Cortex-M with no dynamic memory Tim Wescott <seemywebsite@myfooter.really> - 2016-11-07 11:59 -0600
              Re: g++ on Cortex-M with no dynamic memory Wouter van Ooijen <wouter@voti.nl> - 2016-11-07 19:40 +0100
              Re: g++ on Cortex-M with no dynamic memory Dave Nadler <drn@nadler.com> - 2016-11-08 11:59 -0800
                Re: g++ on Cortex-M with no dynamic memory Wouter van Ooijen <wouter@voti.nl> - 2016-11-08 23:30 +0100
                  Re: g++ on Cortex-M with no dynamic memory David Brown <david.brown@hesbynett.no> - 2016-11-09 09:23 +0100
                    Re: g++ on Cortex-M with no dynamic memory Wouter van Ooijen <wouter@voti.nl> - 2016-11-10 07:34 +0100
                      Re: g++ on Cortex-M with no dynamic memory David Brown <david.brown@hesbynett.no> - 2016-11-10 10:43 +0100
                        Re: g++ on Cortex-M with no dynamic memory Dave Nadler <drn@nadler.com> - 2016-11-11 16:53 -0800
                          Re: g++ on Cortex-M with no dynamic memory David Brown <david.brown@hesbynett.no> - 2016-11-12 14:21 +0100
                          Re: g++ on Cortex-M with no dynamic memory Jack <jack4747@gmail.com> - 2016-11-21 04:08 -0800
            Re: g++ on Cortex-M with no dynamic memory upsidedown@downunder.com - 2016-11-07 23:01 +0200
              Re: g++ on Cortex-M with no dynamic memory Don Y <blockedofcourse@foo.invalid> - 2016-11-07 14:14 -0700
                Re: g++ on Cortex-M with no dynamic memory upsidedown@downunder.com - 2016-11-08 10:57 +0200
                  Re: g++ on Cortex-M with no dynamic memory Don Y <blockedofcourse@foo.invalid> - 2016-11-08 02:31 -0700
                    Re: g++ on Cortex-M with no dynamic memory upsidedown@downunder.com - 2016-11-08 13:52 +0200
                      Re: g++ on Cortex-M with no dynamic memory Dimiter_Popoff <dp@tgi-sci.com> - 2016-11-08 15:11 +0200
                        Re: g++ on Cortex-M with no dynamic memory Don Y <blockedofcourse@foo.invalid> - 2016-11-08 08:01 -0700
                          Re: g++ on Cortex-M with no dynamic memory Dimiter_Popoff <dp@tgi-sci.com> - 2016-11-09 08:26 +0200
                            Re: g++ on Cortex-M with no dynamic memory Don Y <blockedofcourse@foo.invalid> - 2016-11-09 00:55 -0700
                              Re: g++ on Cortex-M with no dynamic memory Dimiter_Popoff <dp@tgi-sci.com> - 2016-11-10 10:27 +0200
                                Re: g++ on Cortex-M with no dynamic memory Don Y <blockedofcourse@foo.invalid> - 2016-11-10 02:40 -0700
                      Re: g++ on Cortex-M with no dynamic memory Don Y <blockedofcourse@foo.invalid> - 2016-11-08 06:58 -0700
              Re: g++ on Cortex-M with no dynamic memory Wouter van Ooijen <wouter@voti.nl> - 2016-11-07 23:01 +0100
                Re: g++ on Cortex-M with no dynamic memory Dave Nadler <drn@nadler.com> - 2016-11-08 11:38 -0800
          Re: g++ on Cortex-M with no dynamic memory Tim Wescott <tim@seemywebsite.com> - 2016-11-07 20:02 -0600
      Re: g++ on Cortex-M with no dynamic memory upsidedown@downunder.com - 2016-11-07 22:56 +0200
        Re: g++ on Cortex-M with no dynamic memory Don Y <blockedofcourse@foo.invalid> - 2016-11-07 14:11 -0700
          Re: g++ on Cortex-M with no dynamic memory David Brown <david.brown@hesbynett.no> - 2016-11-08 10:37 +0100
            Re: g++ on Cortex-M with no dynamic memory upsidedown@downunder.com - 2016-11-08 13:38 +0200
              Re: g++ on Cortex-M with no dynamic memory David Brown <david.brown@hesbynett.no> - 2016-11-08 13:32 +0100
              Re: g++ on Cortex-M with no dynamic memory Don Y <blockedofcourse@foo.invalid> - 2016-11-08 07:09 -0700
    Re: g++ on Cortex-M with no dynamic memory Wouter van Ooijen <wouter@voti.nl> - 2016-11-07 08:10 +0100

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#25440

On Mon, 7 Nov 2016 18:39:49 +0100, Wouter van Ooijen <wouter@voti.nl>
wrote:

>Op 07-Nov-16 om 5:15 PM schreef Tim Wescott:
>> On Mon, 07 Nov 2016 04:13:31 -0800, Dave Nadler wrote:
>>
>>> On Sunday, November 6, 2016 at 9:55:33 PM UTC-5,
>>> anti...@math.uni.wroc.pl wrote:
>>>> Dave Nadler <drn@nadler.com> wrote:
>>>>> For years *one* of the *many* ways we've made reliable C++
>>>>> embedded is by forbidding dynamic storage (except heap).
>>>>                                                     ^^^^
>>>> You mean stack?
>>>
>>> Sorry, stack only (no malloc/free/new/delete).
>>> And of course static ctors.
>>>
>>> I was hoping for a standard way to:
>>> - prune all memory allocation functions from libraries - stub any RTL
>>> bits as you mention
>>>
>>> Thanks,
>>> Best Regards,Dave
>>
>> I don't know if there's a standard way, but there are various ways.  It's
>> been a while since I've done it, but search on topics like "porting newlib
>> to a bace-metal platform" or some such.  Newlib is the run-time library
>> for no-OS or RTOS operation, and it leaves about a dozen functions un-
>> functioned.
>>
>> Personally, I allow for one-time object creation off of the heap, as long
>> as it's all before the tasks start up and as long as nothing gets deleted
>> and recreated.  But that makes it hard to be strict about not allowing it.
>>
>> IIRC, newlib calls the "OS base memory allocation" procedure -- I don't
>> think you can get rid of it entirely, but you can stub this off to an
>> assert.  If you _do_ get rid of all allocation off of the heap, then just
>> leaving it out of your collection -o- stubs will make the link fail when
>> someone does something that calls 'new'.
>
>That's exactly what I do:
>
>// using malloc must cause a linker error
>void * malloc( size_t n ){
>    void if_you_see_this_you_are_using_new_or_malloc();
>    if_you_see_this_you_are_using_new_or_malloc();
>    return NULL;
>}

What is wrong with malloc(), I use it during the first millisecond of
the program startup.

>
>// using free must cause a linker error
>void free( void * p ){
>    void if_you_see_this_you_are_using_free_or_delete();
>    if_you_see_this_you_are_using_free_or_delete();
>}

free() is a no-no, I disable it after the first millisecond and let
the program run for 10-40 years.

>
>And btw I hate global ctor's (ctor ordering problem), so my linkerscript 
>has:
>
>.IF_YOU_SEE_THIS_YOU_HAVE_ONE_OR_MORE_GLOBAL_OBJECT_CONSTRUCTORS : 	
>	{
>		KEEP(*(.init));
>		KEEP(*(.preinit_array));
>		KEEP(*(SORT(.init_array.*)));
>		KEEP(*(.init_array));	
>	} > nul
>
>Wouter "Objects? No Thanks!" van Ooijen
>
>

[toc] | [prev] | [next] | [standalone]

#25442

On 11/7/2016 2:01 PM, upsidedown@downunder.com wrote:
>> That's exactly what I do:
>>
>> // using malloc must cause a linker error
>> void * malloc( size_t n ){
>>    void if_you_see_this_you_are_using_new_or_malloc();
>>    if_you_see_this_you_are_using_new_or_malloc();
>>    return NULL;
>> }
>
> What is wrong with malloc(), I use it during the first millisecond of
> the program startup.

If that's the only time you use it, you can *probably* replace each
instance with a suitable variable/array declaration (global)

>> // using free must cause a linker error
>> void free( void * p ){
>>    void if_you_see_this_you_are_using_free_or_delete();
>>    if_you_see_this_you_are_using_free_or_delete();
>> }
>
> free() is a no-no, I disable it after the first millisecond and let
> the program run for 10-40 years.

So, all of your objects are persistent?  (e.g., no "temporary/transient"
tasks that come and go -- instantiating then releasing their stacks, etc.)

How would you, for example, build a set of cascaded menus?  In
artificially introduced function layers?

[toc] | [prev] | [next] | [standalone]

#25448

On Mon, 7 Nov 2016 14:14:50 -0700, Don Y <blockedofcourse@foo.invalid>
wrote:

>On 11/7/2016 2:01 PM, upsidedown@downunder.com wrote:
>>> That's exactly what I do:
>>>
>>> // using malloc must cause a linker error
>>> void * malloc( size_t n ){
>>>    void if_you_see_this_you_are_using_new_or_malloc();
>>>    if_you_see_this_you_are_using_new_or_malloc();
>>>    return NULL;
>>> }
>>
>> What is wrong with malloc(), I use it during the first millisecond of
>> the program startup.
>
>If that's the only time you use it, you can *probably* replace each
>instance with a suitable variable/array declaration (global)

And recompile the source code for each application and each customer
with different parameters. No thanks.

>
>>> // using free must cause a linker error
>>> void free( void * p ){
>>>    void if_you_see_this_you_are_using_free_or_delete();
>>>    if_you_see_this_you_are_using_free_or_delete();
>>> }
>>
>> free() is a no-no, I disable it after the first millisecond and let
>> the program run for 10-40 years.
>
>So, all of your objects are persistent?  (e.g., no "temporary/transient"
>tasks that come and go -- instantiating then releasing their stacks, etc.)

The nice thing about stacks is that the worst case stack allocation
can be determined statically. Even if recursion is used, the worst
case allocation as long as you have a strict limit to the recursion
depth.

The problem with heaps is the fragmentation of the heap in a long
running system. It might seem running OK for the first year, but will
it run for the next ten years ? One can not test a system for 10 years
and then release it. With properly designed systems, running it for a
few months should be enough. If it is going to fail in the far future,
it will fail during the early test period (infant mortality).

>How would you, for example, build a set of cascaded menus?  In
>artificially introduced function layers?

What prevents using a stack based allocation system ?

[toc] | [prev] | [next] | [standalone]

#25450

On 11/8/2016 1:57 AM, upsidedown@downunder.com wrote:
>>>> // using free must cause a linker error
>>>> void free( void * p ){
>>>>    void if_you_see_this_you_are_using_free_or_delete();
>>>>    if_you_see_this_you_are_using_free_or_delete();
>>>> }
>>>
>>> free() is a no-no, I disable it after the first millisecond and let
>>> the program run for 10-40 years.
>>
>> So, all of your objects are persistent?  (e.g., no "temporary/transient"
>> tasks that come and go -- instantiating then releasing their stacks, etc.)
>
> The nice thing about stacks is that the worst case stack allocation
> can be determined statically. Even if recursion is used, the worst
> case allocation as long as you have a strict limit to the recursion
> depth.
>
> The problem with heaps is the fragmentation of the heap in a long
> running system. It might seem running OK for the first year, but will
> it run for the next ten years ?

That depends on how that memory is physically implemented.
E.g., slip a VMM under it and issues change (albeit at the
granularity of pages)

Also, it assumes your memory allocation strategy is fixed and
not under the control of the memory system's clients -- who
*know* how the memory is likely to be used (allocated AND
released).

[IMO, that's the problem with most allocators:  one arena and
one black-box allocation strategy.]

E.g., folks have no problem dealing with "buffer pools"
("partitions", etc.) -- and PREFER them to "dynamic
memory allocation".  But, isn't this just a special case
of dynamic memory allocation?  One chosen where all
requests are of a size determined a priori?

If, for example, I implemented malloc() to allocate pools in exactly
those same sizes, I could replace your "buffer_alloc()" with
"malloc()" and your code would perform identically!

> One can not test a system for 10 years
> and then release it. With properly designed systems, running it for a
> few months should be enough. If it is going to fail in the far future,
> it will fail during the early test period (infant mortality).

If I allocate task stacks from an arena that I, BY CONVENTION,
manage as a set of "stack sized units" -- or, that is backed
by a set of VMM pages that correspond efficiently with
submultiples of stack sizes -- then I can allocate and free in
damn near any order for all eternity without ever worrying
about fragmentation.

[It's just "buffer pools" at a different level in the
implementation!]

"Don't run with scissors" is unnecessarily pessimistic.

>> How would you, for example, build a set of cascaded menus?  In
>> artificially introduced function layers?
>
> What prevents using a stack based allocation system ?

Why force them to be auto variables in (layered) *function* stack frames
when you can use an (intrinsic) stacking allocation strategy on a heap?
(because that is the very essence of the user interface!)

I.e., there are characteristics of the application of which the DEVELOPER
is aware -- that a compiler nor stdlib writer will *never* be aware.
Why tie the developers' hands needlessly?  Are the standard libraries
THAT intimidating that you can't "look under the hood" and FIX them
to behave in a more opportune manner?  To provide guarantees that
SOME PARTICULAR IMPLEMENTATION doesn't?

[Preface each function's name with "my_" if you're afraid someone
won't understand that there are differences]

[toc] | [prev] | [next] | [standalone]

#25455

On Tue, 8 Nov 2016 02:31:42 -0700, Don Y <blockedofcourse@foo.invalid>
wrote:

>On 11/8/2016 1:57 AM, upsidedown@downunder.com wrote:
>>>>> // using free must cause a linker error
>>>>> void free( void * p ){
>>>>>    void if_you_see_this_you_are_using_free_or_delete();
>>>>>    if_you_see_this_you_are_using_free_or_delete();
>>>>> }
>>>>
>>>> free() is a no-no, I disable it after the first millisecond and let
>>>> the program run for 10-40 years.
>>>
>>> So, all of your objects are persistent?  (e.g., no "temporary/transient"
>>> tasks that come and go -- instantiating then releasing their stacks, etc.)
>>
>> The nice thing about stacks is that the worst case stack allocation
>> can be determined statically. Even if recursion is used, the worst
>> case allocation as long as you have a strict limit to the recursion
>> depth.
>>
>> The problem with heaps is the fragmentation of the heap in a long
>> running system. It might seem running OK for the first year, but will
>> it run for the next ten years ?
>
>That depends on how that memory is physically implemented.
>E.g., slip a VMM under it and issues change (albeit at the
>granularity of pages)

If you have some virtual memory system available that helps a lot 

Unfortunately a virtual memory system requires a backup storage to
hold the "dirty" pages, a page file. 

If rotating fans and rotating disk can't be used, the only alternative
is a Flash disk with limited number of write cycles.
 

[toc] | [prev] | [next] | [standalone]

#25457

On 08.11.2016 г. 13:52, upsidedown@downunder.com wrote:
> On Tue, 8 Nov 2016 02:31:42 -0700, Don Y <blockedofcourse@foo.invalid>
> wrote:
>
>> On 11/8/2016 1:57 AM, upsidedown@downunder.com wrote:
>>>>>> // using free must cause a linker error
>>>>>> void free( void * p ){
>>>>>>     void if_you_see_this_you_are_using_free_or_delete();
>>>>>>     if_you_see_this_you_are_using_free_or_delete();
>>>>>> }
>>>>>
>>>>> free() is a no-no, I disable it after the first millisecond and let
>>>>> the program run for 10-40 years.
>>>>
>>>> So, all of your objects are persistent?  (e.g., no "temporary/transient"
>>>> tasks that come and go -- instantiating then releasing their stacks, etc.)
>>>
>>> The nice thing about stacks is that the worst case stack allocation
>>> can be determined statically. Even if recursion is used, the worst
>>> case allocation as long as you have a strict limit to the recursion
>>> depth.
>>>
>>> The problem with heaps is the fragmentation of the heap in a long
>>> running system. It might seem running OK for the first year, but will
>>> it run for the next ten years ?
>>
>> That depends on how that memory is physically implemented.
>> E.g., slip a VMM under it and issues change (albeit at the
>> granularity of pages)
>
> If you have some virtual memory system available that helps a lot
>
> Unfortunately a virtual memory system requires a backup storage to
> hold the "dirty" pages, a page file.
>
> If rotating fans and rotating disk can't be used, the only alternative
> is a Flash disk with limited number of write cycles.
>
>

While generally so VM does not necessarily need to swap memory.
If your physical memory is say 64M and your logical space is set to
say 1G _and_ you know the 64M will never be all used up, the benefit
of the VM is that fragmentation will be much less of a problem.
Then if memory is allocated using worst-fit strategy fragmentation
is not much of an issue even without VM.

To me (writing under DPS) stack usage and general allocation
are just both in use, stacks are nice because of what you already
stated, generic allocation (not sure how you'd call that, when
you do a system call to allocate that much memory and you get
returned the address and the actually allocated size (the latter
probably somewhat more than requested because of the granularity))
is good because it can be deallocated "out of order" (unlike stack
frames).

Dimiter

------------------------------------------------------
Dimiter Popoff, TGI             http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/

[toc] | [prev] | [next] | [standalone]

#25460

Hi Dimiter,

On 11/8/2016 6:11 AM, Dimiter_Popoff wrote:
>>>> The problem with heaps is the fragmentation of the heap in a long
>>>> running system. It might seem running OK for the first year, but will
>>>> it run for the next ten years ?
>>>
>>> That depends on how that memory is physically implemented.
>>> E.g., slip a VMM under it and issues change (albeit at the
>>> granularity of pages)
>>
>> If you have some virtual memory system available that helps a lot
>>
>> Unfortunately a virtual memory system requires a backup storage to
>> hold the "dirty" pages, a page file.
>>
>> If rotating fans and rotating disk can't be used, the only alternative
>> is a Flash disk with limited number of write cycles.
>
> While generally so VM does not necessarily need to swap memory.
> If your physical memory is say 64M and your logical space is set to
> say 1G _and_ you know the 64M will never be all used up, the benefit
> of the VM is that fragmentation will be much less of a problem.

It also lets you "write protect" memory, trap on writes to
memory that shouldn't be written, trap on writes to memory
that *should* be written, provide protected address spaces,
share memory with different processes (and in different places),
move large blocks of memory in constant time, etc.

Put the stack "someplace special" (in that large, "empty" address
space) and let the hardware tell you when you need to allocate
another page for the stack -- or, when you've *underrun* it, etc.

[Things that are hard to do with pure software mechanisms]

> Then if memory is allocated using worst-fit strategy fragmentation
> is not much of an issue even without VM.
>
> To me (writing under DPS) stack usage and general allocation
> are just both in use, stacks are nice because of what you already
> stated, generic allocation (not sure how you'd call that, when
> you do a system call to allocate that much memory and you get
> returned the address and the actually allocated size (the latter
> probably somewhat more than requested because of the granularity))
> is good because it can be deallocated "out of order" (unlike stack
> frames).

The problem with most memory allocators is they are used for a
hodge-podge of unstructured/unordered requests.  E.g., override
new() and you'll have a better chance at a "well behaved" memory
subsystem.

[toc] | [prev] | [next] | [standalone]

#25465

On 08.11.2016 г. 17:01, Don Y wrote:
> Hi Dimiter,
>
> On 11/8/2016 6:11 AM, Dimiter_Popoff wrote:
>>>>> The problem with heaps is the fragmentation of the heap in a long
>>>>> running system. It might seem running OK for the first year, but will
>>>>> it run for the next ten years ?
>>>>
>>>> That depends on how that memory is physically implemented.
>>>> E.g., slip a VMM under it and issues change (albeit at the
>>>> granularity of pages)
>>>
>>> If you have some virtual memory system available that helps a lot
>>>
>>> Unfortunately a virtual memory system requires a backup storage to
>>> hold the "dirty" pages, a page file.
>>>
>>> If rotating fans and rotating disk can't be used, the only alternative
>>> is a Flash disk with limited number of write cycles.
>>
>> While generally so VM does not necessarily need to swap memory.
>> If your physical memory is say 64M and your logical space is set to
>> say 1G _and_ you know the 64M will never be all used up, the benefit
>> of the VM is that fragmentation will be much less of a problem.
>
> It also lets you "write protect" memory, trap on writes to
> memory that shouldn't be written, trap on writes to memory
> that *should* be written, provide protected address spaces,
> share memory with different processes (and in different places),
> move large blocks of memory in constant time, etc.

All of the protection etc., of course. When I write something new
to run under dps many of the mistakes I make are captured by the
"attempted access to non-allocated space, press CR to kill task" :-).

There are also inconveniences associated with VM though.
Your interrupt handlers cannot be in page translated memory without
killing the system's latency figures (get an IRQ while fixing a
page fault and hhave the IRQ handled create a fault of its own....
ooops, here is the reset button :). So - like it is in dps - the
interrupt handlers must be in BAT (block array translated) memory
which has to be set upon boot time such that it can fit all
allocation requests it will get during a boot session.


> Put the stack "someplace special" (in that large, "empty" address
> space) and let the hardware tell you when you need to allocate
> another page for the stack -- or, when you've *underrun* it, etc.

Yes, in dps a task gets started with an allocated data section
and its (user) stack pointer is put on top of it, command line
data is on its bottom. Needs just to be large enough, this is
"logical memory" being allocated, physical gets there just after
page faults. Well and then there is the same - but much shorter - for
the system stack and then there is the initialized data section etc.

>
> [Things that are hard to do with pure software mechanisms]
>
>> Then if memory is allocated using worst-fit strategy fragmentation
>> is not much of an issue even without VM.
>>
>> To me (writing under DPS) stack usage and general allocation
>> are just both in use, stacks are nice because of what you already
>> stated, generic allocation (not sure how you'd call that, when
>> you do a system call to allocate that much memory and you get
>> returned the address and the actually allocated size (the latter
>> probably somewhat more than requested because of the granularity))
>> is good because it can be deallocated "out of order" (unlike stack
>> frames).
>
> The problem with most memory allocators is they are used for a
> hodge-podge of unstructured/unordered requests.  E.g., override
> new() and you'll have a better chance at a "well behaved" memory
> subsystem.

In dps I call "allocation" the act when memory is allocated to a
requestor based on a bitmap - a bit per cluster. The lowest level
is the system allocation - a cluster being a page, 4k for power; so
there is the LCAT (logical memory cluster allocation table) and the
PCAT (physical memory CAT). Then there is the page translation table.

Tasks get memory allocated by requesting some and getting back
an address and allocated size (size being a multiple of the cluster
size, i.e. 4k on 32 bit power). The allocated piece can be recorded
in a "task history record" so it is deallocated by the system upon
task kill or it can be not put there (the task can still have put a call
point in its history records which is called upon kill).

On top of that there is a dps object (dps maintains a runtime objects
system) called memory_pool; it can have a cluster size as small as just
a few bytes and be of variable size, can be "made" even from within
a script, can be "removed" from there etc. Then the pools can get
automatically chained if needed - i.e. if some object requests
more memory than there is in any of the pools of a chain a new
pool is "made" etc.

Since having all this I have not had any necessity to think "how
do I allocate this or that", and well, I have never thought of
stack growth as of "allocation" (though I do write "allocate a
new stack frame" when I make one :-)).

Dimiter

------------------------------------------------------
Dimiter Popoff, TGI             http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/

[toc] | [prev] | [next] | [standalone]

#25466

On 11/8/2016 11:26 PM, Dimiter_Popoff wrote:
>>>> If rotating fans and rotating disk can't be used, the only alternative
>>>> is a Flash disk with limited number of write cycles.
>>>
>>> While generally so VM does not necessarily need to swap memory.
>>> If your physical memory is say 64M and your logical space is set to
>>> say 1G _and_ you know the 64M will never be all used up, the benefit
>>> of the VM is that fragmentation will be much less of a problem.
>>
>> It also lets you "write protect" memory, trap on writes to
>> memory that shouldn't be written, trap on writes to memory
>> that *should* be written, provide protected address spaces,
>> share memory with different processes (and in different places),
>> move large blocks of memory in constant time, etc.
>
> All of the protection etc., of course. When I write something new
> to run under dps many of the mistakes I make are captured by the
> "attempted access to non-allocated space, press CR to kill task" :-).

Yes, handy at debug time but also handy for (VERY) long-lived
processes;  "CAN'T HAPPEN" had really better NOT happen!

> There are also inconveniences associated with VM though.
> Your interrupt handlers cannot be in page translated memory without
> killing the system's latency figures (get an IRQ while fixing a
> page fault and hhave the IRQ handled create a fault of its own....
> ooops, here is the reset button :). So - like it is in dps - the
> interrupt handlers must be in BAT (block array translated) memory
> which has to be set upon boot time such that it can fit all
> allocation requests it will get during a boot session.

There are LOTS of "inconveniences"!

But, there are also lots of *opportunities* that are simply
unavailable without that hardware.

E.g., when I release virtual memory, it need not immediately
be made available to satisfy another request -- ANY "free"
page can suffice!  So, *this* free'd page can be scrubbed clean
(so process A doesn't leak information to process B through this
uncontrolled channel).  That "scrubbing" can happen when the
system has time to do it -- instead of immediately on the
release of the page (as long as it is scrubbed prior to
reallocation)

>> Put the stack "someplace special" (in that large, "empty" address
>> space) and let the hardware tell you when you need to allocate
>> another page for the stack -- or, when you've *underrun* it, etc.
>
> Yes, in dps a task gets started with an allocated data section
> and its (user) stack pointer is put on top of it, command line
> data is on its bottom. Needs just to be large enough, this is
> "logical memory" being allocated, physical gets there just after
> page faults. Well and then there is the same - but much shorter - for
> the system stack and then there is the initialized data section etc.

Part of a thread's configuration specifies how much stack MUST
be resident (in which case, it is allocated/mmap'ed when the task
is spawned) and a maximum size to which it will be allowed to
dynamically grow.  Other parameters govern how aggressively
the system tries to reclaim no-longer-in-use stack space.
(you don't want to foster pathological behaviors wherein you reclaim
a page only to see it faulted back in as soon as the task resumes!)

>> [Things that are hard to do with pure software mechanisms]
>>
>>> Then if memory is allocated using worst-fit strategy fragmentation
>>> is not much of an issue even without VM.
>>>
>>> To me (writing under DPS) stack usage and general allocation
>>> are just both in use, stacks are nice because of what you already
>>> stated, generic allocation (not sure how you'd call that, when
>>> you do a system call to allocate that much memory and you get
>>> returned the address and the actually allocated size (the latter
>>> probably somewhat more than requested because of the granularity))
>>> is good because it can be deallocated "out of order" (unlike stack
>>> frames).
>>
>> The problem with most memory allocators is they are used for a
>> hodge-podge of unstructured/unordered requests.  E.g., override
>> new() and you'll have a better chance at a "well behaved" memory
>> subsystem.
>
> In dps I call "allocation" the act when memory is allocated to a
> requestor based on a bitmap - a bit per cluster. The lowest level
> is the system allocation - a cluster being a page, 4k for power; so
> there is the LCAT (logical memory cluster allocation table) and the
> PCAT (physical memory CAT). Then there is the page translation table.
>
> Tasks get memory allocated by requesting some and getting back
> an address and allocated size (size being a multiple of the cluster
> size, i.e. 4k on 32 bit power). The allocated piece can be recorded
> in a "task history record" so it is deallocated by the system upon
> task kill or it can be not put there (the task can still have put a call
> point in its history records which is called upon kill).
>
> On top of that there is a dps object (dps maintains a runtime objects
> system) called memory_pool; it can have a cluster size as small as just
> a few bytes and be of variable size, can be "made" even from within
> a script, can be "removed" from there etc. Then the pools can get
> automatically chained if needed - i.e. if some object requests
> more memory than there is in any of the pools of a chain a new
> pool is "made" etc.

I support multiple "heaps" that run in the logical memory space.
Depending on the policies the caller uses to manage them, they
can end up as classic heaps, buffer pools, FIFO's, etc.  Let
the developer figure out how memory will behave in his application
and let the heap implementation simply be the mechanisms to make
that happen (the developer sets *policy* in how he uses each heap).

The VMM runs beneath all this.

There are also bridge objects that essentially map the heaps
conveniently onto VMM pages.

As damn near everything in my world is IPC/RPC, this lets
me move objects and sets of objects (e.g., parameters)
between processes just by twiddling the underlying VMM
system.  Of course, silly for small objects that can
simply be passed by copy.

And, for asynchronous operations, it provides immutability
without "burdening" the caller with a copy operation.
E.g., I can pass a frame of raw video to a remote process
and not worry about the caller altering the frame *after*
the call (but before it has actually been "transmitted").
If the caller doesn't fiddle with the frame's contents
until AFTER it has been transmitted, then the call was
"free".  OTOH, if the caller attempts to alter it's contents
BEFORE it is actually transmitted, then the VMM system ensures
a clean copy is transmitted without the caller's "corruption".

[The system then knows to "consume" such an extra copy
after it has been "used"]

> Since having all this I have not had any necessity to think "how
> do I allocate this or that", and well, I have never thought of
> stack growth as of "allocation" (though I do write "allocate a
> new stack frame" when I make one :-)).

Most VMM operations happen without the developer's knowledge,
in my case.  The OS takes care of keeping track of all the
magic so the developer sees a "clean" interface free of
hazzards, etc.

Its tricky to provide these hooks without EXPOSING them
(which would require a more sophisticated class of developer!)

[toc] | [prev] | [next] | [standalone]

#25471

On 09.11.2016 г. 09:55, Don Y wrote:
> On 11/8/2016 11:26 PM, Dimiter_Popoff wrote:
>>>>> If rotating fans and rotating disk can't be used, the only alternative
>>>>> is a Flash disk with limited number of write cycles.
>>>>
>>>> While generally so VM does not necessarily need to swap memory.
>>>> If your physical memory is say 64M and your logical space is set to
>>>> say 1G _and_ you know the 64M will never be all used up, the benefit
>>>> of the VM is that fragmentation will be much less of a problem.
>>>
>>> It also lets you "write protect" memory, trap on writes to
>>> memory that shouldn't be written, trap on writes to memory
>>> that *should* be written, provide protected address spaces,
>>> share memory with different processes (and in different places),
>>> move large blocks of memory in constant time, etc.
>>
>> All of the protection etc., of course. When I write something new
>> to run under dps many of the mistakes I make are captured by the
>> "attempted access to non-allocated space, press CR to kill task" :-).
>
> Yes, handy at debug time but also handy for (VERY) long-lived
> processes;  "CAN'T HAPPEN" had really better NOT happen!

Hi Don,
in my experience long-lived tasks (months) just do not fail. It
appears that if some piece of code survives a day or so chances
are it is pretty stable and "bug free" (to the extent it will
not fail at all). Which is not to say your scenario is not
thinkable of course, just how things are with me.
Where I do see failures is when doing new things, e.g. a user
using some relatively new feature. Had this recently during a
demo - user being me. Right click on a button crashed the task,
got that "attempted access to non-allocated space" message.
The buttons had been there for a few years... had never seen
that. The cause turned out to be a non-initialized pointer of
sorts; had lay dormant (pointing to some allocated piece to
be read to no further consequences, what was read was irrelevant)
until some other global system change or whatever woke it up.
Good thing the entire demo was a shiny plug-and-play affair
(with a HPGe detector people are used to anything but that)
so I could just laugh the bug off.

>
> Most VMM operations happen without the developer's knowledge,
> in my case.  The OS takes care of keeping track of all the
> magic so the developer sees a "clean" interface free of
> hazzards, etc.

Well if the developer has to allocate memory he has to say so,
no way around that. I can't think of a cleaner and simpler way
than have him say "give me that much" and in return "here you
are, at this address, that much, rounded up a little".
The developer (even if this is me) just does not think of
physical memory at this level, just of logical.
Then if you write at a higher level - e.g. write a dps shell
script - allocate/deallocate is completely hidden.
You "make" an object which requests memory from its "container"
object, logs that so it will deallocate upon "remove" etc.
It is a highly sophisticated/interconnected piece of underlying
code really.

> Its tricky to provide these hooks without EXPOSING them
> (which would require a more sophisticated class of developer!)

I just do not think of the limitations of the user. If I can
learn to do something he/she also can (if not he is just messing
with things he is not suppose to mess with. If something is too
much for me to process in the time I want to spend on a given
task it will be for someone else, too; so I just pack things
into higher and higher level as needed, thus reducing the level
of competence one needs to have involved in the task at hand.
I.e. if I want to make some search utility I just write a dps
script, I do not use my knowledge on allocate/deallocate, vpa
language etc.

Generally I think the assumption that a user is too stupid to
be handed this or that so we have to stupefy the product is a
bad approach, I have never seen anything useful come out of it
(I don't think this is what you mean but it brought the association,
it is one of my red buttons I suppose).

Dimiter

------------------------------------------------------
Dimiter Popoff, TGI             http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/

[toc] | [prev] | [next] | [standalone]

#25472

Hi Dimiter,

On 11/10/2016 1:27 AM, Dimiter_Popoff wrote:
> On 09.11.2016 г. 09:55, Don Y wrote:
>> On 11/8/2016 11:26 PM, Dimiter_Popoff wrote:
>>>>>> If rotating fans and rotating disk can't be used, the only alternative
>>>>>> is a Flash disk with limited number of write cycles.
>>>>>
>>>>> While generally so VM does not necessarily need to swap memory.
>>>>> If your physical memory is say 64M and your logical space is set to
>>>>> say 1G _and_ you know the 64M will never be all used up, the benefit
>>>>> of the VM is that fragmentation will be much less of a problem.
>>>>
>>>> It also lets you "write protect" memory, trap on writes to
>>>> memory that shouldn't be written, trap on writes to memory
>>>> that *should* be written, provide protected address spaces,
>>>> share memory with different processes (and in different places),
>>>> move large blocks of memory in constant time, etc.
>>>
>>> All of the protection etc., of course. When I write something new
>>> to run under dps many of the mistakes I make are captured by the
>>> "attempted access to non-allocated space, press CR to kill task" :-).
>>
>> Yes, handy at debug time but also handy for (VERY) long-lived
>> processes;  "CAN'T HAPPEN" had really better NOT happen!
>
> in my experience long-lived tasks (months) just do not fail. It
> appears that if some piece of code survives a day or so chances
> are it is pretty stable and "bug free" (to the extent it will
> not fail at all). Which is not to say your scenario is not
> thinkable of course, just how things are with me.

It depends on what portions of the code are exercised in those
"months".   E.g., if something only runs on the 5th Monday
of each month (and this is February) there may not *be* a
5th monday in the near future.  Or, an action that the user
typically seldom invokes ("compact mailboxes"), etc.

Or, effectively exposing a hazzard that might not be encountered
in short term use (e.g., someone checking mail and then quitting
will rarely encounter a competitor process accessing the
mailbox)

I.e., the user interface can be running "forever" and still
not tickle a latent bug.

> Where I do see failures is when doing new things, e.g. a user
> using some relatively new feature. Had this recently during a
> demo - user being me. Right click on a button crashed the task,
> got that "attempted access to non-allocated space" message.
> The buttons had been there for a few years... had never seen
> that. The cause turned out to be a non-initialized pointer of
> sorts; had lay dormant (pointing to some allocated piece to
> be read to no further consequences, what was read was irrelevant)
> until some other global system change or whatever woke it up.
> Good thing the entire demo was a shiny plug-and-play affair
> (with a HPGe detector people are used to anything but that)
> so I could just laugh the bug off.
>
>> Most VMM operations happen without the developer's knowledge,
>> in my case.  The OS takes care of keeping track of all the
>> magic so the developer sees a "clean" interface free of
>> hazzards, etc.
>
> Well if the developer has to allocate memory he has to say so,
> no way around that. I can't think of a cleaner and simpler way
> than have him say "give me that much" and in return "here you
> are, at this address, that much, rounded up a little".
> The developer (even if this is me) just does not think of
> physical memory at this level, just of logical.

I try to hide as much unnecessary detail as possible at
each "level" in my design.  And, the "standard" interfaces
that you would expect to encounter tend to behave as you'd
expect of them.

But, a knowledgeable user can usually access the same mechanisms
via a more versatile interface for finer control over the implementation.
(However, the API doesn't REQUIRE this of all users!)

> Then if you write at a higher level - e.g. write a dps shell
> script - allocate/deallocate is completely hidden.

Yes.  I use a similar philosophy throughout the design.  I
expose what a developer coding at a particular level in the
design expects/needs to meet his design goals.

Someone writing for bare metal sees far more detail than
a "user" (end user/consumer) who writes nothing OR, at
best, "scripts".

Someone writing a device driver sees a different level of
abstractions and services.  Writing atop the RTOS exposes
yet another API -- that of the proxy/service.

At the very top (least capable, most hand-holding) are
user "scripts".  Here, the user doesn't worry about
numeric representations, memory allocation/garbage collection,
communication channels, permissions (assuming he HAS them),
crash recovery, etc.

> You "make" an object which requests memory from its "container"
> object, logs that so it will deallocate upon "remove" etc.
> It is a highly sophisticated/interconnected piece of underlying
> code really.
>
>> Its tricky to provide these hooks without EXPOSING them
>> (which would require a more sophisticated class of developer!)
>
> I just do not think of the limitations of the user. If I can
> learn to do something he/she also can (if not he is just messing
> with things he is not suppose to mess with. If something is too
> much for me to process in the time I want to spend on a given
> task it will be for someone else, too; so I just pack things
> into higher and higher level as needed, thus reducing the level
> of competence one needs to have involved in the task at hand.

Exactly.  In my case, I don't want developers to have to be
concerned with the VMM and how it ties into the RTOS, etc.
If a developer (a sort of "user") wants to send a message
to a particular host/process, I don't want him to have to worry
about the mechanism for doing that:  create the message and enqueue
it on the desired destination -- let the system figure out
*where* the target lies WHEN THE MESSAGE IS ACTUALLY SENT (a
target can MOVE while the message is enqueued -- why bother
the developer with handling that case?!)

Likewise, my IDL lets me define the contracts between clients and
the proxy/stub libraries for each service.  E.g., if you are
MOVE-ing an object to another process (local or remote), then
the semantics of that IPC/RPC imply that the object will cease to
exist on the originating process/node.  The stub generated by
the IDL will unmap the object from the caller's address space
when the routine is invoked. However, the object will persist
within the stub until it can actually be "transferred" to the
destination -- at which point, it will be deleted from the
node (if the destination was remote) or deleted from the stub's
memory space.

A whole class of errors are avoided -- those where the object is not
unmapped by the caller AFTER the RPC returns (as well as ensuring
the caller doesn't try to alter the cached object while it is waiting
to be transmitted).

OTOH, if you are *copying* the object, then it's never unmapped from
the caller's memory space -- BUT, is protected from alteration
(zero copy semantics) while the stub is waiting to copy it out
to the destination.

Again, another class of errors avoided:  changing the object while
it is conceptually immutable (protecting against that would normally
require a copyout() -- even if the caller doesn't elect to alter
the data before the actual transmission occurs!)

*BUT*, the caller needs to provide a page-aligned object in order
for the system to efficiently implement this magic!  (Yet I
don't want to directly expose the VMM to the caller)

> I.e. if I want to make some search utility I just write a dps
> script, I do not use my knowledge on allocate/deallocate, vpa
> language etc.

Yes, but you can do so because the machinery beneath you is already
in place and automated.

> Generally I think the assumption that a user is too stupid to
> be handed this or that so we have to stupefy the product is a
> bad approach, I have never seen anything useful come out of it
> (I don't think this is what you mean but it brought the association,
> it is one of my red buttons I suppose).

I want users (at their respective "levels" in the implementation
hierarchy) to concentrate on their objectives and not my mechanisms.

E.g., if a user wants to figure out (roughly) how many 18x18" ceramic
tiles with 1/4" grout lines are needed to tile a 19'5" x 12'2" room
(assuming a 3/8 inch border along each edge):
    (19 ft 5 in - (2 * 3/8 in)) * (12 ft 2 in - (2 * 3/8 in))
   / (18 in + 1/4 in) ^ 2
will yield a dimensionless value (area/area).  OTOH, omitting the
divisor would lead to an area result -- which the user could request
in any suitable "area units".  It's silly to force the user to
do a bunch of trivial but error prone normalizations when the API
can handle those more reliably.

[toc] | [prev] | [next] | [standalone]

#25458

On 11/8/2016 4:52 AM, upsidedown@downunder.com wrote:
> On Tue, 8 Nov 2016 02:31:42 -0700, Don Y <blockedofcourse@foo.invalid>
> wrote:
>
>> On 11/8/2016 1:57 AM, upsidedown@downunder.com wrote:
>>>>>> // using free must cause a linker error
>>>>>> void free( void * p ){
>>>>>>    void if_you_see_this_you_are_using_free_or_delete();
>>>>>>    if_you_see_this_you_are_using_free_or_delete();
>>>>>> }
>>>>>
>>>>> free() is a no-no, I disable it after the first millisecond and let
>>>>> the program run for 10-40 years.
>>>>
>>>> So, all of your objects are persistent?  (e.g., no "temporary/transient"
>>>> tasks that come and go -- instantiating then releasing their stacks, etc.)
>>>
>>> The nice thing about stacks is that the worst case stack allocation
>>> can be determined statically. Even if recursion is used, the worst
>>> case allocation as long as you have a strict limit to the recursion
>>> depth.
>>>
>>> The problem with heaps is the fragmentation of the heap in a long
>>> running system. It might seem running OK for the first year, but will
>>> it run for the next ten years ?
>>
>> That depends on how that memory is physically implemented.
>> E.g., slip a VMM under it and issues change (albeit at the
>> granularity of pages)
>
> If you have some virtual memory system available that helps a lot
>
> Unfortunately a virtual memory system requires a backup storage to
> hold the "dirty" pages, a page file.

No, it doesn't. You only need backing store if your total virtual
memory (i.e., in use at any particular instant) will exceed your
"real" physical memory.

There are many reasons to implement VMM and benefits that come with
it.

> If rotating fans and rotating disk can't be used, the only alternative
> is a Flash disk with limited number of write cycles.

[toc] | [prev] | [next] | [standalone]

#25443

Op 07-Nov-16 om 10:01 PM schreef upsidedown@downunder.com:
> On Mon, 7 Nov 2016 18:39:49 +0100, Wouter van Ooijen <wouter@voti.nl>
> wrote:
>
>> Op 07-Nov-16 om 5:15 PM schreef Tim Wescott:
>>> On Mon, 07 Nov 2016 04:13:31 -0800, Dave Nadler wrote:
>>>
>>>> On Sunday, November 6, 2016 at 9:55:33 PM UTC-5,
>>>> anti...@math.uni.wroc.pl wrote:
>>>>> Dave Nadler <drn@nadler.com> wrote:
>>>>>> For years *one* of the *many* ways we've made reliable C++
>>>>>> embedded is by forbidding dynamic storage (except heap).
>>>>>                                                     ^^^^
>>>>> You mean stack?
>>>>
>>>> Sorry, stack only (no malloc/free/new/delete).
>>>> And of course static ctors.
>>>>
>>>> I was hoping for a standard way to:
>>>> - prune all memory allocation functions from libraries - stub any RTL
>>>> bits as you mention
>>>>
>>>> Thanks,
>>>> Best Regards,Dave
>>>
>>> I don't know if there's a standard way, but there are various ways.  It's
>>> been a while since I've done it, but search on topics like "porting newlib
>>> to a bace-metal platform" or some such.  Newlib is the run-time library
>>> for no-OS or RTOS operation, and it leaves about a dozen functions un-
>>> functioned.
>>>
>>> Personally, I allow for one-time object creation off of the heap, as long
>>> as it's all before the tasks start up and as long as nothing gets deleted
>>> and recreated.  But that makes it hard to be strict about not allowing it.
>>>
>>> IIRC, newlib calls the "OS base memory allocation" procedure -- I don't
>>> think you can get rid of it entirely, but you can stub this off to an
>>> assert.  If you _do_ get rid of all allocation off of the heap, then just
>>> leaving it out of your collection -o- stubs will make the link fail when
>>> someone does something that calls 'new'.
>>
>> That's exactly what I do:
>>
>> // using malloc must cause a linker error
>> void * malloc( size_t n ){
>>    void if_you_see_this_you_are_using_new_or_malloc();
>>    if_you_see_this_you_are_using_new_or_malloc();
>>    return NULL;
>> }
>
> What is wrong with malloc(), I use it during the first millisecond of
> the program startup.

If you need it, use it. I have an alternative version that I prefer not 
to use that supports malloc but not free. You can guess that its 
implementation is trivial.

But if possible, I prefer not to use malloc at all, because it proves 
beyond doubt that

- malloc isn't used after that initial phase
- apart from stack size(s), the application will fit in memory

This approach isn't suitable for all applications, but when it is 
suitable it a joy to use.

Wouter "Objects? No Thanks!" van Ooijen

[toc] | [prev] | [next] | [standalone]

#25461

On Monday, November 7, 2016 at 5:01:19 PM UTC-5, Wouter van Ooijen wrote:
> ...if possible, I prefer not to use malloc at all,
> because it proves 
> beyond doubt that:
> - malloc isn't used after that initial phase
> - apart from stack size(s), the application will fit in memory

Exactly...

[toc] | [prev] | [next] | [standalone]

#25444

On Mon, 07 Nov 2016 10:15:10 -0600, Tim Wescott wrote:

> On Mon, 07 Nov 2016 04:13:31 -0800, Dave Nadler wrote:
> 
>> On Sunday, November 6, 2016 at 9:55:33 PM UTC-5,
>> anti...@math.uni.wroc.pl wrote:
>>> Dave Nadler <drn@nadler.com> wrote:
>>> > For years *one* of the *many* ways we've made reliable C++
>>> > embedded is by forbidding dynamic storage (except heap).
>>>                                                     ^^^^
>>> You mean stack?
>> 
>> Sorry, stack only (no malloc/free/new/delete).
>> And of course static ctors.
>> 
>> I was hoping for a standard way to:
>> - prune all memory allocation functions from libraries - stub any RTL
>> bits as you mention
>> 
>> Thanks,
>> Best Regards,Dave
> 
> I don't know if there's a standard way, but there are various ways. 
> It's been a while since I've done it, but search on topics like "porting
> newlib to a bace-metal platform" or some such.  Newlib is the run-time
> library for no-OS or RTOS operation, and it leaves about a dozen
> functions un- functioned.
> 
> Personally, I allow for one-time object creation off of the heap, as
> long as it's all before the tasks start up and as long as nothing gets
> deleted and recreated.  But that makes it hard to be strict about not
> allowing it.
> 
> IIRC, newlib calls the "OS base memory allocation" procedure -- I don't
> think you can get rid of it entirely, but you can stub this off to an
> assert.  If you _do_ get rid of all allocation off of the heap, then
> just leaving it out of your collection -o- stubs will make the link fail
> when someone does something that calls 'new'.

_sbrk is the function I was thinking of.  I have a functioning one for 
the reasons given above.  But, if you're going to do your own board-
support package for newlib, there's about a dozen others that you need to 
address.

-- 
Tim Wescott
Control systems, embedded software and circuit design
I'm looking for work!  See my website if you're interested
http://www.wescottdesign.com

[toc] | [prev] | [next] | [standalone]

#25439

On Mon, 7 Nov 2016 02:55:30 +0000 (UTC), antispam@math.uni.wroc.pl
wrote:

>Dave Nadler <drn@nadler.com> wrote:
>> For years *one* of the *many* ways we've made reliable C++
>> embedded is by forbidding dynamic storage (except heap).
>                                                    ^^^^
>You mean stack?  Allocating memory at startup is reasonably
>safe: each time you allocate the same amount so can easily
>test if you stay within limit.  Such allocation can be
>useful for static constructors in library functions.

Slightly off topic, why are C-programs so keen of using heap
(malloc/free) instead of allocating  temporary table variables
(malloca) on stack as in e.g. Pascal ?

Even some FORTRAN IV compilers supported something like

SUBROUTINE FOO (N,M)
DIMENSION TEMP (N,M)

In which the floating point two dimensional TEMP array on the
subroutine stack had different size, depending on which parameters
were used to call FOO.

Did this have something to do with the situation that some of the
early processors used in C implementation did not support stack (or
index register) relative addressing and only supported some kind of
return address stack ?

[toc] | [prev] | [next] | [standalone]

#25441

On 11/7/2016 1:56 PM, upsidedown@downunder.com wrote:
> On Mon, 7 Nov 2016 02:55:30 +0000 (UTC), antispam@math.uni.wroc.pl
> wrote:
>
>> Dave Nadler <drn@nadler.com> wrote:
>>> For years *one* of the *many* ways we've made reliable C++
>>> embedded is by forbidding dynamic storage (except heap).
>>                                                    ^^^^
>> You mean stack?  Allocating memory at startup is reasonably
>> safe: each time you allocate the same amount so can easily
>> test if you stay within limit.  Such allocation can be
>> useful for static constructors in library functions.
>
> Slightly off topic, why are C-programs so keen of using heap
> (malloc/free) instead of allocating  temporary table variables
> (malloca) on stack as in e.g. Pascal ?
>
> Even some FORTRAN IV compilers supported something like
>
> SUBROUTINE FOO (N,M)
> DIMENSION TEMP (N,M)
>
> In which the floating point two dimensional TEMP array on the
> subroutine stack had different size, depending on which parameters
> were used to call FOO.
>
> Did this have something to do with the situation that some of the
> early processors used in C implementation did not support stack (or
> index register) relative addressing and only supported some kind of
> return address stack ?

IIRC, variable length *auto* arrays were not supported prior to C99.

There is also the issue of whether the array's size is part of
its *type* -- or *value*.  (and, in the latter case, if it is
accessible as such!)

[toc] | [prev] | [next] | [standalone]

#25451

On 07/11/16 22:11, Don Y wrote:
> On 11/7/2016 1:56 PM, upsidedown@downunder.com wrote:
>> On Mon, 7 Nov 2016 02:55:30 +0000 (UTC), antispam@math.uni.wroc.pl
>> wrote:
>>
>>> Dave Nadler <drn@nadler.com> wrote:
>>>> For years *one* of the *many* ways we've made reliable C++
>>>> embedded is by forbidding dynamic storage (except heap).
>>>                                                    ^^^^
>>> You mean stack?  Allocating memory at startup is reasonably
>>> safe: each time you allocate the same amount so can easily
>>> test if you stay within limit.  Such allocation can be
>>> useful for static constructors in library functions.
>>
>> Slightly off topic, why are C-programs so keen of using heap
>> (malloc/free) instead of allocating  temporary table variables
>> (malloca) on stack as in e.g. Pascal ?

Pascal and C both use the heap and the stack for the same purpose -
there is no significant difference.  Local variables are kept on the
stack, allocated memory is on the heap.  Allocating on the stack is
usually a good deal faster, but the memory is deallocated whenever you
leave the function that allocated it, and on some targets there may be
more restrictions on the stack space than heap space.  So stack
allocation is great for short-lived dynamic memory, especially if it is
small, but you can't hold on to the data after the function returns.

Also, there is no standard C stack space allocation function.  Many
(most?) C libraries have alloca(), which does the job - but it is not
part of the standard.  And while overloading the heap gives clear errors
(malloc returns 0), overloading the stack is just undefined behaviour.

VLAs give you a standardised way to allocate space on the stack in C.
There is no standard way to do it in C++, though many compilers (such as
gcc) support both VLAs and alloca() in C++.

Of course, if you know the size you want at compile-time, you can freely
create whatever local variables you want on the stack.

It is a good while since I have programmed in Pascal - I can't remember
any way of getting stack memory other than local variables whose sizes
are known at compile time.

>>
>> Even some FORTRAN IV compilers supported something like
>>
>> SUBROUTINE FOO (N,M)
>> DIMENSION TEMP (N,M)
>>
>> In which the floating point two dimensional TEMP array on the
>> subroutine stack had different size, depending on which parameters
>> were used to call FOO.
>>
>> Did this have something to do with the situation that some of the
>> early processors used in C implementation did not support stack (or
>> index register) relative addressing and only supported some kind of
>> return address stack ?
> 
> IIRC, variable length *auto* arrays were not supported prior to C99.

Correct - VLA's were standardised in C99.  (They were a gcc extension
before that, and alloca() was commonly implemented.)

> 
> There is also the issue of whether the array's size is part of
> its *type* -- or *value*.  (and, in the latter case, if it is
> accessible as such!)
> 

If the size of an array is known at compile time, then it's size is part
of its type (to the extent that C distinguishes between different array
sizes as types).  If it is a VLA, then the size is not fixed at compile
time (though it might be known by the compiler through optimisation, or
just because the VLA has constant size.  "const int n = 10; xs[n];" is a
VLA in C, and an ordinary array in C++).  The compiler knows the size,
however, because it needed the size when creating the VLA - thus
"sizeof" works fine on VLAs.  This is also the one situation where
sizeof may evaluate its arguments for side-effects - but if that makes
any difference to your program, you have really messed things up!

When you pass an array as a parameter to a function, however, the
parameter is merely a pointer to the first element of the array - you
lose all size information regardless of how the array was defined.  (C++
std::array<> types are different.)

[toc] | [prev] | [next] | [standalone]

#25454

On Tue, 08 Nov 2016 10:37:51 +0100, David Brown
<david.brown@hesbynett.no> wrote:

>On 07/11/16 22:11, Don Y wrote:
>> On 11/7/2016 1:56 PM, upsidedown@downunder.com wrote:
>>> On Mon, 7 Nov 2016 02:55:30 +0000 (UTC), antispam@math.uni.wroc.pl
>>> wrote:
>>>
>>>> Dave Nadler <drn@nadler.com> wrote:
>>>>> For years *one* of the *many* ways we've made reliable C++
>>>>> embedded is by forbidding dynamic storage (except heap).
>>>>                                                    ^^^^
>>>> You mean stack?  Allocating memory at startup is reasonably
>>>> safe: each time you allocate the same amount so can easily
>>>> test if you stay within limit.  Such allocation can be
>>>> useful for static constructors in library functions.
>>>
>>> Slightly off topic, why are C-programs so keen of using heap
>>> (malloc/free) instead of allocating  temporary table variables
>>> (malloca) on stack as in e.g. Pascal ?
>
>Pascal and C both use the heap and the stack for the same purpose -
>there is no significant difference.  Local variables are kept on the
>stack, allocated memory is on the heap.  Allocating on the stack is
>usually a good deal faster, but the memory is deallocated whenever you
>leave the function that allocated it, and on some targets there may be
>more restrictions on the stack space than heap space.  So stack
>allocation is great for short-lived dynamic memory, especially if it is
>small, but you can't hold on to the data after the function returns.

If that data needs a longer lifetime, allocate it at a higher level in
the call stack and pass a pointer to the lower levels.

>Also, there is no standard C stack space allocation function.  Many
>(most?) C libraries have alloca(), which does the job - but it is not
>part of the standard.  And while overloading the heap gives clear errors
>(malloc returns 0), overloading the stack is just undefined behaviour.

Look at this issue from an application system upper level designer,
what do you do if the malloc() returns NULL in some deep function ?
What do you do ? 

The simplest case is to restart the program and hope that the problem
go away ? 

However, if the system must continue to work in a sensible way even if
malloc fails due to heap fragmentation, what do you do ?

[toc] | [prev] | [next] | [standalone]

#25456

On 08/11/16 12:38, upsidedown@downunder.com wrote:
> On Tue, 08 Nov 2016 10:37:51 +0100, David Brown
> <david.brown@hesbynett.no> wrote:
> 
>> On 07/11/16 22:11, Don Y wrote:
>>> On 11/7/2016 1:56 PM, upsidedown@downunder.com wrote:
>>>> On Mon, 7 Nov 2016 02:55:30 +0000 (UTC), antispam@math.uni.wroc.pl
>>>> wrote:
>>>>
>>>>> Dave Nadler <drn@nadler.com> wrote:
>>>>>> For years *one* of the *many* ways we've made reliable C++
>>>>>> embedded is by forbidding dynamic storage (except heap).
>>>>>                                                    ^^^^
>>>>> You mean stack?  Allocating memory at startup is reasonably
>>>>> safe: each time you allocate the same amount so can easily
>>>>> test if you stay within limit.  Such allocation can be
>>>>> useful for static constructors in library functions.
>>>>
>>>> Slightly off topic, why are C-programs so keen of using heap
>>>> (malloc/free) instead of allocating  temporary table variables
>>>> (malloca) on stack as in e.g. Pascal ?
>>
>> Pascal and C both use the heap and the stack for the same purpose -
>> there is no significant difference.  Local variables are kept on the
>> stack, allocated memory is on the heap.  Allocating on the stack is
>> usually a good deal faster, but the memory is deallocated whenever you
>> leave the function that allocated it, and on some targets there may be
>> more restrictions on the stack space than heap space.  So stack
>> allocation is great for short-lived dynamic memory, especially if it is
>> small, but you can't hold on to the data after the function returns.
> 
> If that data needs a longer lifetime, allocate it at a higher level in
> the call stack and pass a pointer to the lower levels.

Sure, that can be done.  But that won't help you implement a function
called "allocateNewArray" :-)

(For my own code, I try to minimise the use of dynamic memory,
preferring to make everything statically allocated where possible and
occasionally using VLAs on the stack.  However, sometimes dynamic memory
is almost unavoidable.)

> 
>> Also, there is no standard C stack space allocation function.  Many
>> (most?) C libraries have alloca(), which does the job - but it is not
>> part of the standard.  And while overloading the heap gives clear errors
>> (malloc returns 0), overloading the stack is just undefined behaviour.
> 
> Look at this issue from an application system upper level designer,
> what do you do if the malloc() returns NULL in some deep function ?
> What do you do ? 
> 
> The simplest case is to restart the program and hope that the problem
> go away ? 
> 
> However, if the system must continue to work in a sensible way even if
> malloc fails due to heap fragmentation, what do you do ?
> 

And that is one of the key reasons to avoid malloc and dynamic memory in
embedded systems!

Memory pools, sized allocation groups, etc., can reduce the potential
problem somewhat - but it is still a danger.

About the only time when I do use dynamic memory is for Ethernet
networking - it is hard to completely avoid it.  So the typical reaction
is simply to drop that particular network packet or connection.  It is
messy to keep it all leak-free, but usually that's the least bad
reaction to memory allocation failure.

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.arch.embedded

csiph-web