Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #29187 > unrolled thread

newlib, FreeRTOS, reentrancy, heap and related questions

Back to article view | Back to comp.arch.embedded

Contents

  newlib, FreeRTOS, reentrancy, heap and related questions pozz <pozzugno@gmail.com> - 2020-02-06 13:26 +0100
    Re: newlib, FreeRTOS, reentrancy, heap and related questions Richard Damon <Richard@Damon-Family.org> - 2020-02-06 07:37 -0500
      Re: newlib, FreeRTOS, reentrancy, heap and related questions pozz <pozzugno@gmail.com> - 2020-02-06 13:57 +0100
        Re: newlib, FreeRTOS, reentrancy, heap and related questions David Brown <david.brown@hesbynett.no> - 2020-02-06 14:58 +0100
          Re: newlib, FreeRTOS, reentrancy, heap and related questions pozz <pozzugno@gmail.com> - 2020-02-06 16:06 +0100
            Re: newlib, FreeRTOS, reentrancy, heap and related questions David Brown <david.brown@hesbynett.no> - 2020-02-06 21:41 +0100
              Re: newlib, FreeRTOS, reentrancy, heap and related questions pozz <pozzugno@gmail.com> - 2020-02-06 23:23 +0100
                Re: newlib, FreeRTOS, reentrancy, heap and related questions Richard Damon <Richard@Damon-Family.org> - 2020-02-06 23:44 -0500
                  Re: newlib, FreeRTOS, reentrancy, heap and related questions David Brown <david.brown@hesbynett.no> - 2020-02-07 09:51 +0100
                    Re: newlib, FreeRTOS, reentrancy, heap and related questions Paul Rubin <no.email@nospam.invalid> - 2020-02-07 01:04 -0800
                      Re: newlib, FreeRTOS, reentrancy, heap and related questions David Brown <david.brown@hesbynett.no> - 2020-02-07 10:48 +0100
                        Re: newlib, FreeRTOS, reentrancy, heap and related questions Paul Rubin <no.email@nospam.invalid> - 2020-02-07 08:00 -0800
                          Re: newlib, FreeRTOS, reentrancy, heap and related questions David Brown <david.brown@hesbynett.no> - 2020-02-08 16:19 +0100
                            Re: newlib, FreeRTOS, reentrancy, heap and related questions Paul Rubin <no.email@nospam.invalid> - 2020-02-08 19:42 -0800
                              Re: newlib, FreeRTOS, reentrancy, heap and related questions David Brown <david.brown@hesbynett.no> - 2020-02-09 12:11 +0100
                Re: newlib, FreeRTOS, reentrancy, heap and related questions David Brown <david.brown@hesbynett.no> - 2020-02-07 10:35 +0100
        Re: newlib, FreeRTOS, reentrancy, heap and related questions Dave Nadler <drn@nadler.com> - 2020-02-06 07:21 -0800
          Re: newlib, FreeRTOS, reentrancy, heap and related questions pozz <pozzugno@gmail.com> - 2020-02-06 17:14 +0100
          Re: newlib, FreeRTOS, reentrancy, heap and related questions pozz <pozzugno@gmail.com> - 2020-02-06 23:26 +0100
            Re: newlib, FreeRTOS, reentrancy, heap and related questions Richard Damon <Richard@Damon-Family.org> - 2020-02-06 23:29 -0500
        Re: newlib, FreeRTOS, reentrancy, heap and related questions Richard Damon <Richard@Damon-Family.org> - 2020-02-06 23:23 -0500
          Re: newlib, FreeRTOS, reentrancy, heap and related questions pozz <pozzugno@gmail.com> - 2020-02-07 09:19 +0100
    Re: newlib, FreeRTOS, reentrancy, heap and related questions upsidedown@downunder.com - 2020-02-07 11:36 +0200
      Re: newlib, FreeRTOS, reentrancy, heap and related questions pozz <pozzugno@gmail.com> - 2020-02-07 14:31 +0100
        Re: newlib, FreeRTOS, reentrancy, heap and related questions Dave Nadler <drn@nadler.com> - 2020-02-07 05:55 -0800
    Re: newlib, FreeRTOS, reentrancy, heap and related questions Dave Nadler <drn@nadler.com> - 2020-04-05 15:05 -0700

Page 1 of 2  [1] 2  Next page →

#29187 — newlib, FreeRTOS, reentrancy, heap and related questions

Usually arm gcc compiler uses newlib (or newlib-nano) for standard C 
libraries (memset, malloc, printf, time and so on).

I sometimes replace newlib functions, because I don't like them. First 
of all, I replace snprintf because newlib implementation uses malloc and 
I don't like to use malloc, mostly if it can be avoided.
And for printf-like functions, there are a few implementations that 
don't use malloc.

When newlib is used with FreeRTOS, there are two heaps: one used by 
FreeRTOS and one used by newlib. Dave[1] suggests to replace FreeRTOS 
heap with newlib heap. Why don't do the contrary? newlib malloc is not 
reentrant, you should implement malloc_lock/unlock. If we're able to 
force newlib to use FreeRTOS heap, I think it would be better, because 
FreeRTOS malloc is natively multitasking safe.

Many times I need to have a date and time. time() and gettimeofday() 
from newlib are good, but I need to implement a _gettimeofday() function 
for my platform. So the value added by newlib is very little (I could 
implement a time() or gettimeofday() myself).

So the final question. If I remove printf/malloc/time/... from newlib, 
is newlib(-nano) needed yet? Is it possible to avoid using newlib at all?
I know I could need memset, strchr, strtok, but those functions can be 
implemented if needed (I think there are many implementations available, 
even in newlib project).

Of course, I don't need stdio features (open, close, read, write, exit, 
...).

I know newlib is for resource constrained embedded devices, so it is 
already small, but I have the feeling that it's bigger than I need, 
mostly with FreeRTOS.


[1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html

[toc] | [next] | [standalone]

#29188

On 2/6/20 7:26 AM, pozz wrote:
> Usually arm gcc compiler uses newlib (or newlib-nano) for standard C 
> libraries (memset, malloc, printf, time and so on).
> 
> I sometimes replace newlib functions, because I don't like them. First 
> of all, I replace snprintf because newlib implementation uses malloc and 
> I don't like to use malloc, mostly if it can be avoided.
> And for printf-like functions, there are a few implementations that 
> don't use malloc.
> 
> When newlib is used with FreeRTOS, there are two heaps: one used by 
> FreeRTOS and one used by newlib. Dave[1] suggests to replace FreeRTOS 
> heap with newlib heap. Why don't do the contrary? newlib malloc is not 
> reentrant, you should implement malloc_lock/unlock. If we're able to 
> force newlib to use FreeRTOS heap, I think it would be better, because 
> FreeRTOS malloc is natively multitasking safe.
> 
The FreeRTOS heap functions implement a smaller set of fuctionality than 
do the standard C library as implemented in newlib, in particular the 
realloc function, which is used (I am pretty sure) in some of the library.

It is fairly trivial to write a 'heapc.c' memory allocator file that has 
FreeRTOS use the native malloc/free with also provides teh needed 
malloc_lock/unlock to make them thread safe.

> Many times I need to have a date and time. time() and gettimeofday() 
> from newlib are good, but I need to implement a _gettimeofday() function 
> for my platform. So the value added by newlib is very little (I could 
> implement a time() or gettimeofday() myself).
> 
> So the final question. If I remove printf/malloc/time/... from newlib, 
> is newlib(-nano) needed yet? Is it possible to avoid using newlib at all?
> I know I could need memset, strchr, strtok, but those functions can be 
> implemented if needed (I think there are many implementations available, 
> even in newlib project).
> 
> Of course, I don't need stdio features (open, close, read, write, exit, 
> ...).
> 
> I know newlib is for resource constrained embedded devices, so it is 
> already small, but I have the feeling that it's bigger than I need, 
> mostly with FreeRTOS.
> 
> 
> [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html

Yes, it is quite possible to totally implement you own 'standard 
library' yourself and totally replace newlib-nano. The question comes 
does doing so buy you enough to be worth the time.

[toc] | [prev] | [next] | [standalone]

#29190

Il 06/02/2020 13:37, Richard Damon ha scritto:
> On 2/6/20 7:26 AM, pozz wrote:
>> Usually arm gcc compiler uses newlib (or newlib-nano) for standard C 
>> libraries (memset, malloc, printf, time and so on).
>>
>> I sometimes replace newlib functions, because I don't like them. First 
>> of all, I replace snprintf because newlib implementation uses malloc 
>> and I don't like to use malloc, mostly if it can be avoided.
>> And for printf-like functions, there are a few implementations that 
>> don't use malloc.
>>
>> When newlib is used with FreeRTOS, there are two heaps: one used by 
>> FreeRTOS and one used by newlib. Dave[1] suggests to replace FreeRTOS 
>> heap with newlib heap. Why don't do the contrary? newlib malloc is not 
>> reentrant, you should implement malloc_lock/unlock. If we're able to 
>> force newlib to use FreeRTOS heap, I think it would be better, because 
>> FreeRTOS malloc is natively multitasking safe.
>>
> The FreeRTOS heap functions implement a smaller set of fuctionality than 
> do the standard C library as implemented in newlib, in particular the 
> realloc function, which is used (I am pretty sure) in some of the library.

Do you mean that realloc(), that is used by some newlib functions, isn't 
implemented by FreeRTOS heap?
I have the sensation that I don't use those newlib functions that use 
realloc(), because I avoid to use functions that use heap at all.

In this case, I think using FreeRTOS heap only instead of newlib heap 
has some sense.


> It is fairly trivial to write a 'heapc.c' memory allocator file that has 
> FreeRTOS use the native malloc/free with also provides teh needed 
> malloc_lock/unlock to make them thread safe.

Why not the contrary? Why don't use FreeRTOS heap management only? Is it 
impossible for some reason to completely avoid newlib heap functions?


>> Many times I need to have a date and time. time() and gettimeofday() 
>> from newlib are good, but I need to implement a _gettimeofday() 
>> function for my platform. So the value added by newlib is very little 
>> (I could implement a time() or gettimeofday() myself).
>>
>> So the final question. If I remove printf/malloc/time/... from newlib, 
>> is newlib(-nano) needed yet? Is it possible to avoid using newlib at all?
>> I know I could need memset, strchr, strtok, but those functions can be 
>> implemented if needed (I think there are many implementations 
>> available, even in newlib project).
>>
>> Of course, I don't need stdio features (open, close, read, write, 
>> exit, ...).
>>
>> I know newlib is for resource constrained embedded devices, so it is 
>> already small, but I have the feeling that it's bigger than I need, 
>> mostly with FreeRTOS.
>>
>>
>> [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html
> 
> Yes, it is quite possible to totally implement you own 'standard 
> library' yourself and totally replace newlib-nano. The question comes 
> does doing so buy you enough to be worth the time.

I'm wondering how many standard functions I need to implement without 
newlib. 1, 10 or 100? How complex would be to implement them?
Of course, only me can answer.

[toc] | [prev] | [next] | [standalone]

#29191

On 06/02/2020 13:57, pozz wrote:
> Il 06/02/2020 13:37, Richard Damon ha scritto:
>> On 2/6/20 7:26 AM, pozz wrote:

>>> So the final question. If I remove printf/malloc/time/... from
>>> newlib, is newlib(-nano) needed yet? Is it possible to avoid using
>>> newlib at all?
>>> I know I could need memset, strchr, strtok, but those functions can
>>> be implemented if needed (I think there are many implementations
>>> available, even in newlib project).
>>>
>>> Of course, I don't need stdio features (open, close, read, write,
>>> exit, ...).
>>>
>>> I know newlib is for resource constrained embedded devices, so it is
>>> already small, but I have the feeling that it's bigger than I need,
>>> mostly with FreeRTOS.
>>>
>>>
>>> [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html
>>
>> Yes, it is quite possible to totally implement you own 'standard
>> library' yourself and totally replace newlib-nano. The question comes
>> does doing so buy you enough to be worth the time.
> 
> I'm wondering how many standard functions I need to implement without
> newlib. 1, 10 or 100? How complex would be to implement them?
> Of course, only me can answer.

It depends a lot on what you need to do.  A good many of the standard C
functions are rarely useful in small embedded systems (hands up those
who need the locale functions or wide character handling).  Most systems
can also do without the maths functions - perhaps also implementing
their own for a more appropriate balance between speed and accuracy.

There are a few oddities to watch out for.  Technically, you can't
actually implement malloc() in standard C, nor can you implement memcpy
or memmove, due to the type aliasing and effective type rules.  If you
want to be sure of problem-free code that is safe regardless of
optimisation, link-time optimisation, new generations of compilers,
etc., then you'll be quite careful and make good use of gcc attributes.

Another possibility here is to move to C++.  It has better support for
memory allocation functions, memory pools, etc.  And you can avoid one
of the silliest aspects of malloc/free - the fact that the memory
allocator implementation has to store the size of the allocated block
somewhere.  Usually when you are freeing memory, you know the size
already - storing it in the block just makes things less efficient.  C++
also has functions for "washing" pointers so that you can keep the
compiler informed about aliasing and re-use of memory, for better
efficiency and safety.  (I haven't tried much of this myself as yet, but
I expect to do so before long.)

[toc] | [prev] | [next] | [standalone]

#29193

Il 06/02/2020 14:58, David Brown ha scritto:
> On 06/02/2020 13:57, pozz wrote:
>> Il 06/02/2020 13:37, Richard Damon ha scritto:
>>> On 2/6/20 7:26 AM, pozz wrote:
> 
>>>> So the final question. If I remove printf/malloc/time/... from
>>>> newlib, is newlib(-nano) needed yet? Is it possible to avoid using
>>>> newlib at all?
>>>> I know I could need memset, strchr, strtok, but those functions can
>>>> be implemented if needed (I think there are many implementations
>>>> available, even in newlib project).
>>>>
>>>> Of course, I don't need stdio features (open, close, read, write,
>>>> exit, ...).
>>>>
>>>> I know newlib is for resource constrained embedded devices, so it is
>>>> already small, but I have the feeling that it's bigger than I need,
>>>> mostly with FreeRTOS.
>>>>
>>>>
>>>> [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html
>>>
>>> Yes, it is quite possible to totally implement you own 'standard
>>> library' yourself and totally replace newlib-nano. The question comes
>>> does doing so buy you enough to be worth the time.
>>
>> I'm wondering how many standard functions I need to implement without
>> newlib. 1, 10 or 100? How complex would be to implement them?
>> Of course, only me can answer.
> 
> It depends a lot on what you need to do.  A good many of the standard C
> functions are rarely useful in small embedded systems (hands up those
> who need the locale functions or wide character handling).  Most systems
> can also do without the maths functions - perhaps also implementing
> their own for a more appropriate balance between speed and accuracy.

Yes, indeed those are my assumptions.


> There are a few oddities to watch out for.  Technically, you can't
> actually implement malloc() in standard C, 

If I really need heap, I would use FreeRTOS implementation that is 
thread-safe and already available.


> nor can you implement memcpy
> or memmove, due to the type aliasing and effective type rules.  If you
> want to be sure of problem-free code that is safe regardless of
> optimisation, link-time optimisation, new generations of compilers,
> etc., then you'll be quite careful and make good use of gcc attributes.

What about copying byte by byte?
Here[1] you can see newlib memcpy implementation. If 
PREFER_SIZE_OVER_SPEED or __OPTIMIZE_SIZE__ is defined, the 
implementation is really copying byte by byte.

I don't really know how newlib used by my compiler (CubeIDE from ST) was 
compiled, maybe I'm using dumb version of memcpy already.

This is an extract from a listing:

08025850 <memcpy>:
  8025850:	b510      	push	{r4, lr}
  8025852:	1e43      	subs	r3, r0, #1
  8025854:	440a      	add	r2, r1
  8025856:	4291      	cmp	r1, r2
  8025858:	d100      	bne.n	802585c <memcpy+0xc>
  802585a:	bd10      	pop	{r4, pc}
  802585c:	f811 4b01 	ldrb.w	r4, [r1], #1
  8025860:	f803 4f01 	strb.w	r4, [r3, #1]!
  8025864:	e7f7      	b.n	8025856 <memcpy+0x6>

I'm not an expert of assembly, but it seems to me it is implemented in 
the simple and not optimized way.

> Another possibility here is to move to C++.  It has better support for
> memory allocation functions, memory pools, etc.  And you can avoid one
> of the silliest aspects of malloc/free - the fact that the memory
> allocator implementation has to store the size of the allocated block
> somewhere.  Usually when you are freeing memory, you know the size
> already - storing it in the block just makes things less efficient.  C++
> also has functions for "washing" pointers so that you can keep the
> compiler informed about aliasing and re-use of memory, for better
> efficiency and safety.  (I haven't tried much of this myself as yet, but
> I expect to do so before long.)
> 

[1] https://github.com/bminor/newlib/blob/master/newlib/libc/string/memcpy.c

[toc] | [prev] | [next] | [standalone]

#29201

On 06/02/2020 16:06, pozz wrote:
> Il 06/02/2020 14:58, David Brown ha scritto:

> 
>> nor can you implement memcpy
>> or memmove, due to the type aliasing and effective type rules.  If you
>> want to be sure of problem-free code that is safe regardless of
>> optimisation, link-time optimisation, new generations of compilers,
>> etc., then you'll be quite careful and make good use of gcc attributes.
> 
> What about copying byte by byte?
> Here[1] you can see newlib memcpy implementation. If 
> PREFER_SIZE_OVER_SPEED or __OPTIMIZE_SIZE__ is defined, the 
> implementation is really copying byte by byte.
> 
> I don't really know how newlib used by my compiler (CubeIDE from ST) was 
> compiled, maybe I'm using dumb version of memcpy already.
> 
> This is an extract from a listing:
> 
> 08025850 <memcpy>:
>   8025850:    b510          push    {r4, lr}
>   8025852:    1e43          subs    r3, r0, #1
>   8025854:    440a          add    r2, r1
>   8025856:    4291          cmp    r1, r2
>   8025858:    d100          bne.n    802585c <memcpy+0xc>
>   802585a:    bd10          pop    {r4, pc}
>   802585c:    f811 4b01     ldrb.w    r4, [r1], #1
>   8025860:    f803 4f01     strb.w    r4, [r3, #1]!
>   8025864:    e7f7          b.n    8025856 <memcpy+0x6>
> 
> I'm not an expert of assembly, but it seems to me it is implemented in 
> the simple and not optimized way.
> 

It is not the actual copying that is the problem - copying by char is 
simple and safe (though often inefficient).  The issue is that the C 
standards say memcpy also copies the effective type in certain 
circumstances - there is no way to specify that in C, and it is 
therefore a special feature of the library memcpy.  A homemade memcpy 
does not have that same feature.  (In a similar vain, there is no way to 
get memory in standard C that has "no declared type" except via the 
library malloc and friends - a homemade malloc won't do.)  I am not sure 
what the best solution is here.

Anyway, for memcpy make sure the compiler can use the builtin versions 
where possible (avoid -ffreestanding, or use -fbuiltin) as this will 
give far better code.

[toc] | [prev] | [next] | [standalone]

#29205

Il 06/02/2020 21:41, David Brown ha scritto:
> On 06/02/2020 16:06, pozz wrote:
>> Il 06/02/2020 14:58, David Brown ha scritto:
> 
>>
>>> nor can you implement memcpy
>>> or memmove, due to the type aliasing and effective type rules.  If you
>>> want to be sure of problem-free code that is safe regardless of
>>> optimisation, link-time optimisation, new generations of compilers,
>>> etc., then you'll be quite careful and make good use of gcc attributes.
>>
>> What about copying byte by byte?
>> Here[1] you can see newlib memcpy implementation. If 
>> PREFER_SIZE_OVER_SPEED or __OPTIMIZE_SIZE__ is defined, the 
>> implementation is really copying byte by byte.
>>
>> I don't really know how newlib used by my compiler (CubeIDE from ST) 
>> was compiled, maybe I'm using dumb version of memcpy already.
>>
>> This is an extract from a listing:
>>
>> 08025850 <memcpy>:
>>   8025850:    b510          push    {r4, lr}
>>   8025852:    1e43          subs    r3, r0, #1
>>   8025854:    440a          add    r2, r1
>>   8025856:    4291          cmp    r1, r2
>>   8025858:    d100          bne.n    802585c <memcpy+0xc>
>>   802585a:    bd10          pop    {r4, pc}
>>   802585c:    f811 4b01     ldrb.w    r4, [r1], #1
>>   8025860:    f803 4f01     strb.w    r4, [r3, #1]!
>>   8025864:    e7f7          b.n    8025856 <memcpy+0x6>
>>
>> I'm not an expert of assembly, but it seems to me it is implemented in 
>> the simple and not optimized way.
>>
> 
> It is not the actual copying that is the problem - copying by char is 
> simple and safe (though often inefficient).  The issue is that the C 
> standards say memcpy also copies the effective type in certain 
> circumstances - there is no way to specify that in C, and it is 
> therefore a special feature of the library memcpy.  

Could you make an example? I didn't understand.

Anyway as you can see, newlib just implements memcpy in pure C language 
when compiled without optimizations. Are you saying it's bugged?


> A homemade memcpy 
> does not have that same feature.  (In a similar vain, there is no way to 
> get memory in standard C that has "no declared type" except via the 
> library malloc and friends - a homemade malloc won't do.)  I am not sure 
> what the best solution is here.
> 
> Anyway, for memcpy make sure the compiler can use the builtin versions 
> where possible (avoid -ffreestanding, or use -fbuiltin) as this will 
> give far better code.
> 

I don't use -ffreestanding, but I don't know if I'm using -fbuiltin. 
Anyway you are suggesting to use builtin functions that are functions 
built *in* the compiler and not in the newlib.

Another reason to consider useless newlib.

[toc] | [prev] | [next] | [standalone]

#29211

On 2/6/20 5:23 PM, pozz wrote:
> Il 06/02/2020 21:41, David Brown ha scritto:
> 
> Could you make an example? I didn't understand.
> 
> Anyway as you can see, newlib just implements memcpy in pure C language 
> when compiled without optimizations. Are you saying it's bugged?
> 
> 

The issue isn't that the code needs to do something special, but that 
the code being implementation code is allowed to do something special.

The rules on effective types provides limits on what 'standard' code is 
able to do and stay within the rules. Code that doesn't follow those 
rules might not work right, or it might work.

memcpy as pure C, breaks the rules, so the langague doesn't promise that 
it will work. The writers of newlib understand the implementations, and 
know what they need to do so that for that implementation, it will work. 
Sometimes this means adding an special implementation provide construct 
to make the code work right, sometimes it is just making sure the 
compiler is in the right mode when compiling that code so as to avoid 
creating the problem, in particular do what is needed to keep the 
optimizer from looking into the function and see the effective type rule 
violation.

[toc] | [prev] | [next] | [standalone]

#29216

On 07/02/2020 05:44, Richard Damon wrote:
> On 2/6/20 5:23 PM, pozz wrote:
>> Il 06/02/2020 21:41, David Brown ha scritto:
>>
>> Could you make an example? I didn't understand.
>>
>> Anyway as you can see, newlib just implements memcpy in pure C
>> language when compiled without optimizations. Are you saying it's bugged?
>>
>>
> 
> The issue isn't that the code needs to do something special, but that
> the code being implementation code is allowed to do something special.
> 
> The rules on effective types provides limits on what 'standard' code is
> able to do and stay within the rules. Code that doesn't follow those
> rules might not work right, or it might work.
> 
> memcpy as pure C, breaks the rules, so the langague doesn't promise that
> it will work. The writers of newlib understand the implementations, and
> know what they need to do so that for that implementation, it will work.
> Sometimes this means adding an special implementation provide construct
> to make the code work right, sometimes it is just making sure the
> compiler is in the right mode when compiling that code so as to avoid
> creating the problem, in particular do what is needed to keep the
> optimizer from looking into the function and see the effective type rule
> violation.

The compile can also know that the memcpy in newlib is the library
memcpy, and give it special privileges without it being marked in any
special way or being compiled in particular modes.  Many library
functions are in some way special or privileged in C, simply because
they are C standard library functions, and in theory you can't duplicate
that with your own plain C code.  Basically, the compiler knows a great
deal about what a library function will do, even if it does not have
direct access to the source code.  This can be used for optimisation and
static analysis.  If you write your own versions of these functions and
don't follow these rules, all sorts of bad things can happen.  In
practice, of course, your won alternative C code will often work exactly
the same - and sometimes there are compiler features like attributes or
pragmas that can give the compiler the extra information.

An example of this, consider this code:

int sum(const int * p, int n) {
    int s = 0;
    while (n--) s += *p++;
    return s;
}

int test(void) {
    const int N = 4;

    int * p = malloc(N);
    for (int i = 0; i < N; i++) {
        p[i] = i;
    }
    int s = sum(p, N);
    free(p);
    return s;
}

We allocate space for 4 ints with "malloc", fill the array, pass it on
to a calculation function, free the resources with "free", and return
the result.

gcc compiles test to:

test:
  mov eax, 6
  ret


It knows what malloc and free do, and can eliminate them entirely.  It
could not do that with hand-made memory allocation functions.

It also knows that malloc returns either 0, or a pointer that cannot
alias any existing memory, and it can use that for optimisation.  But if
the source of memory that a hand-made malloc uses is a C-defined array,
then this knowledge will be wrong as the malloc-returned pointer will
alias an existing array.  Not good.

[toc] | [prev] | [next] | [standalone]

#29217

David Brown <david.brown@hesbynett.no> writes:
>     int * p = malloc(N);

(cough) that allocates N bytes, not N ints.

> gcc compiles test to:
>
> test:
>   mov eax, 6
>   ret

Wow!  I think it saw the consts and basically ran the code at compile
time.

[toc] | [prev] | [next] | [standalone]

#29222

On 07/02/2020 10:04, Paul Rubin wrote:
> David Brown <david.brown@hesbynett.no> writes:
>>     int * p = malloc(N);
> 
> (cough) that allocates N bytes, not N ints.

Just checking that you were paying attention :-)

> 
>> gcc compiles test to:
>>
>> test:
>>   mov eax, 6
>>   ret
> 
> Wow!  I think it saw the consts and basically ran the code at compile
> time.
> 

Yes, exactly.

The point is that because the compiler knows what malloc and free do -
they are specified in the standards - it can use that knowledge for
optimisation.

(The exact point at which it will change from run-time calculation to
compile-time calculation is dependent on the compiler, target, options,
etc.)

[toc] | [prev] | [next] | [standalone]

#29227

David Brown <david.brown@hesbynett.no> writes:
> The point is that because the compiler knows what malloc and free do -
> they are specified in the standards - it can use that knowledge for
> optimisation.

In this case it has accessed unallocated memory.  I wonder if there is
an exploit.  Hmm.

[toc] | [prev] | [next] | [standalone]

#29233

On 07/02/2020 17:00, Paul Rubin wrote:
> David Brown <david.brown@hesbynett.no> writes:
>> The point is that because the compiler knows what malloc and free do -
>> they are specified in the standards - it can use that knowledge for
>> optimisation.
> 
> In this case it has accessed unallocated memory.  I wonder if there is
> an exploit.  Hmm.
> 

No, it hasn't accessed unallocated memory at all.  It did not access 
anything.

The compiler could see that either the malloc worked fine and the result 
would be the value 6, or the malloc would fail (and return 0) in which 
case the program would have undefined behaviour (accessing a null 
pointer).  The compiler can assume that the programmer doesn't care what 
happens when executing undefined behaviour, and thus giving a result of 
6 is perfectly acceptable there too.  So the best code is simply to 
return 6 without any work at run time.

Ironically, if I had checked the result of malloc() for a null pointer, 
it could not have made this optimisation!

[toc] | [prev] | [next] | [standalone]

#29255

David Brown <david.brown@hesbynett.no> writes:
> No, it hasn't accessed unallocated memory at all.  It did not access
> anything.  The compiler could see that either the malloc worked fine
> and the result would be the value 6,

It's a question of how the compiler implemented the compile-time
execution.  I hope that it didn't really allocate 6 bytes of memory at
compile time, and then write past it.  But it makes me wonder.

[toc] | [prev] | [next] | [standalone]

#29260

On 09/02/2020 04:42, Paul Rubin wrote:
> David Brown <david.brown@hesbynett.no> writes:
>> No, it hasn't accessed unallocated memory at all.  It did not access
>> anything.  The compiler could see that either the malloc worked fine
>> and the result would be the value 6,
> 
> It's a question of how the compiler implemented the compile-time
> execution.  I hope that it didn't really allocate 6 bytes of memory at
> compile time, and then write past it.  But it makes me wonder.
> 

I'm lost here.

Are you talking about the mistake I made in allocating 4 bytes, rather 
than 4 * sizeof(int) ?  It makes no difference to the code generated 
when that is corrected, and I don't see where "6 bytes" comes from.

The use of "malloc" in the source code bears no direct relation to 
having to allocate dynamic memory in the compiler.  Compile-time 
execution means figuring out what the effect of the code is, and 
simulating it at compile time - it does /not/ mean executing it 
directly.  And in particular, baring bugs in the compiler it does not 
mean executing undefined behaviour in the compiler - though it can mean 
ignoring it in the source code.

(It would have been nice if the compiler had spotted my mistake and told 
me about it, however.)

[toc] | [prev] | [next] | [standalone]

#29218

On 06/02/2020 23:23, pozz wrote:
> Il 06/02/2020 21:41, David Brown ha scritto:
>> On 06/02/2020 16:06, pozz wrote:
>>> Il 06/02/2020 14:58, David Brown ha scritto:
>>
>>>
>>>> nor can you implement memcpy
>>>> or memmove, due to the type aliasing and effective type rules.  If you
>>>> want to be sure of problem-free code that is safe regardless of
>>>> optimisation, link-time optimisation, new generations of compilers,
>>>> etc., then you'll be quite careful and make good use of gcc attributes.
>>>
>>> What about copying byte by byte?
>>> Here[1] you can see newlib memcpy implementation. If
>>> PREFER_SIZE_OVER_SPEED or __OPTIMIZE_SIZE__ is defined, the
>>> implementation is really copying byte by byte.
>>>
>>> I don't really know how newlib used by my compiler (CubeIDE from ST)
>>> was compiled, maybe I'm using dumb version of memcpy already.
>>>
>>> This is an extract from a listing:
>>>
>>> 08025850 <memcpy>:
>>>   8025850:    b510          push    {r4, lr}
>>>   8025852:    1e43          subs    r3, r0, #1
>>>   8025854:    440a          add    r2, r1
>>>   8025856:    4291          cmp    r1, r2
>>>   8025858:    d100          bne.n    802585c <memcpy+0xc>
>>>   802585a:    bd10          pop    {r4, pc}
>>>   802585c:    f811 4b01     ldrb.w    r4, [r1], #1
>>>   8025860:    f803 4f01     strb.w    r4, [r3, #1]!
>>>   8025864:    e7f7          b.n    8025856 <memcpy+0x6>
>>>
>>> I'm not an expert of assembly, but it seems to me it is implemented
>>> in the simple and not optimized way.
>>>
>>
>> It is not the actual copying that is the problem - copying by char is
>> simple and safe (though often inefficient).  The issue is that the C
>> standards say memcpy also copies the effective type in certain
>> circumstances - there is no way to specify that in C, and it is
>> therefore a special feature of the library memcpy.  
> 
> Could you make an example? I didn't understand.

Suppose you have a block "b" of memory allocated on the heap with malloc
- it has no "declared type" because it was not part of a C-defined
object.  You are free to store data of any kind in "b", and it takes on
a type based on the access you used to store to "b" (unless you use
character type access, which leaves it untyped).  Let's say you treat
"b" as an array of floats and fill it up - now its effective type is
float[].

Suppose you have another C object or array "s" with a specific type from
somewhere - such as an array of char* pointers.

You want to copy the contents of "s" into "b".

You could do this in several ways:

1. Read from "s" as char* pointers, converting to a float using a union,
and write it to "b".  Then "b" is still an array of floats and the
compiler knows that any access to it as an array of char* pointers is
undefined behaviour - it can assume it can't happen.  Beware the nasal
demons!

2. Make a pointer "char* * p = (char* *) b" and use that as the
destination when copying from "s" to "b".  The compiler knows that "b"
is now an array char* pointers, and can be accessed as such.  Everything
works, but you need a specific copying function each time.

3. Make a generic function that copies using unsigned char, and call
that to copy from s to b.  Then b takes on the effective type of s, and
so b is now an array of char* pointers.  Everything works, but copying
is inefficient.

4. Make a generic function that copies using uint32_t for speed, and
call that to copy from s to b.  Then b becomes an array of uint32_t, and
accessing it as an array of char* pointers is undefined behaviour.
Nasal demons again.

5. Call the standard library memcpy.  Then b gets the effective type of
s, and everything works.  This is true whether the compiler generates a
local loop, or calls the library function, and it is true whether the
copying is done by byte or in larger lumps.  The library memcpy is
special here - you cannot duplicate that behaviour in standard C.


This kind of thing - type based alias analysis and the effective type
rules in C - is difficult to get right.  And it is not often that the
compiler can use this extra information for optimisation.  But sometimes
it can.  And sometimes it uses it for an optimisation that is correct
according to the C code you wrote, but not according to what you wanted.

Understanding the rules is hard, and sometimes playing by the rules is
even harder, so one solution is to change the rules.  The
"-fno-strict-aliasing" flag in gcc changes the semantics of C to say
that the effective type of an object is always the type used to access
it - this simplifies things a lot here, at the cost of occasionally
missed optimisation opportunities.  For example, the Linux kernel is
always compiled with "-fno-strict-aliasing".

(Note that this flag does not help with the aliasing issue with
home-made malloc, as that's a different thing entirely.)

> 
> Anyway as you can see, newlib just implements memcpy in pure C language
> when compiled without optimizations. Are you saying it's bugged?

No - it can be treated as special because it is the standard library for
your implementation.

> 
> 
>> A homemade memcpy does not have that same feature.  (In a similar
>> vain, there is no way to get memory in standard C that has "no
>> declared type" except via the library malloc and friends - a homemade
>> malloc won't do.)  I am not sure what the best solution is here.
>>
>> Anyway, for memcpy make sure the compiler can use the builtin versions
>> where possible (avoid -ffreestanding, or use -fbuiltin) as this will
>> give far better code.
>>
> 
> I don't use -ffreestanding, but I don't know if I'm using -fbuiltin.
> Anyway you are suggesting to use builtin functions that are functions
> built *in* the compiler and not in the newlib.
> 
> Another reason to consider useless newlib.
> 

When you use one of the common "small" functions in the C standard
library, like memcpy, memset, strcat, etc., the compiler knows what they
do without knowing the source.  If it can make smaller or faster code
inline with the same effect as specified in the standards, then it may
do so.  Typically for memcpy that means the compiler knows the size of
the copy and the alignments at compiler time.  For example:


uint32_t rawfloat(float f) {
    uint32_t u;

    memcpy(&u, &f, sizeof(u));
    return u;
}

This will be turned into a register move (if needed, depending on the
cpu), with nothing stored in memory and no library calls made.  And
unlike faffing around with pointer casts, it is correct C code.  And
unlike using a type-punning union, it is correct C++ code as well as
correct C code.

But more general calls to memcpy will be passed on to the library function.

[toc] | [prev] | [next] | [standalone]

#29195

On Thursday, February 6, 2020 at 7:57:57 AM UTC-5, pozz wrote:
> Do you mean that realloc(), that is used by some newlib functions, isn't 
> implemented by FreeRTOS heap?

realloc() use within newlib is documented on my web page you referenced.

> >> [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html

Whatever you do, check the map and make sure you aren't accidentally
using memory management functions you haven't covered.

Hope that helps,
Best Regards, Dave

[toc] | [prev] | [next] | [standalone]

#29196

Il 06/02/2020 16:21, Dave Nadler ha scritto:
> On Thursday, February 6, 2020 at 7:57:57 AM UTC-5, pozz wrote:
>> Do you mean that realloc(), that is used by some newlib functions, isn't
>> implemented by FreeRTOS heap?
> 
> realloc() use within newlib is documented on my web page you referenced.

You're right, I didn't notice:

 > Another option is wrap newlib’s malloc-family to use FreeRTOS free
 > storage (ie heap_4.c), and specify newlib support for FreeRTOS. Tell
 > the linker to wrap all newlib's malloc-family functions (using
 > -Xlinker --wrap=malloc etc.), and provide a wrapper function that
 > calls the FreeRTOS functions. I tried that, but newlib's printf family
 > uses realloc, which is not supported in FreeRTOS heap implementations.

So my sensation was correct, replacing printf-like functions with others 
that don't use malloc is sufficient to avoid using newlib heap at all.
Most probably there are other newlib functions that need realloc(), but 
I think they can be replaced as well as printf.

With those assumptions, are there any other drawbacks in using FreeRTOS 
heap only?

> 
>>>> [1] http://www.nadler.com/embedded/newlibAndFreeRTOS.html
> 
> Whatever you do, check the map and make sure you aren't accidentally
> using memory management functions you haven't covered.

Yes, good suggestion.

[toc] | [prev] | [next] | [standalone]

#29206

Dave, in your page you suggest to use -Wrap compiler option to replace 
newlib malloc with FreeRTOS malloc.

I think it's not necessary. If you define a malloc() function, the 
linker should use it instead of newlib malloc, without emitting any 
warning or errors about duplicate definition.

I use this to replace printf-like functions.

Linker searches for a function in the object files that you're linking 
and only if it isn't able to find it will try with libraries.

[toc] | [prev] | [next] | [standalone]

#29210

On 2/6/20 5:26 PM, pozz wrote:
> Dave, in your page you suggest to use -Wrap compiler option to replace 
> newlib malloc with FreeRTOS malloc.
> 
> I think it's not necessary. If you define a malloc() function, the 
> linker should use it instead of newlib malloc, without emitting any 
> warning or errors about duplicate definition.
> 
> I use this to replace printf-like functions.
> 
> Linker searches for a function in the object files that you're linking 
> and only if it isn't able to find it will try with libraries.

One big problem with that is that much of newlib internally doesn't call 
malloc, but another function that malloc also calls, so replacing malloc 
doesn't work. This is one reason the Standard doesn't tell you that you 
can override library functions like this, you really need to understand 
the implementation, and put on your 'implementer' hat to make that sort 
of change.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.arch.embedded

csiph-web