Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #10511 > unrolled thread

Benchmarking C compilers for embedded systems

Back to article view | Back to comp.lang.c

Contents

  Benchmarking C compilers for embedded systems Philipp Klaus Krause <pkk@spth.de> - 2011-09-19 19:43 +0200
    Re: Benchmarking C compilers for embedded systems David Brown <david.brown@removethis.hesbynett.no> - 2011-09-19 22:10 +0200
      Re: Benchmarking C compilers for embedded systems Philipp Klaus Krause <pkk@spth.de> - 2011-09-22 09:16 +0200
        Re: Benchmarking C compilers for embedded systems David Brown <david@westcontrol.removethisbit.com> - 2011-09-22 10:26 +0200
    Re: Benchmarking C compilers for embedded systems Abhishek <abhishekmathur.10d@gmail.com> - 2011-09-20 00:50 -0700
      Re: Benchmarking C compilers for embedded systems Philipp Klaus Krause <pkk@spth.de> - 2011-09-21 09:40 +0200
        Re: Benchmarking C compilers for embedded systems Ian Collins <ian-news@hotmail.com> - 2011-09-21 21:06 +1200
          Re: Benchmarking C compilers for embedded systems Philipp Klaus Krause <pkk@spth.de> - 2011-09-22 09:12 +0200
            Re: Benchmarking C compilers for embedded systems Ian Collins <ian-news@hotmail.com> - 2011-09-22 19:41 +1200
    Re: Benchmarking C compilers for embedded systems Arlet Ottens <usenet+5@c-scape.nl> - 2011-09-22 10:34 +0200
    Re: Benchmarking C compilers for embedded systems Walter Banks <walter@bytecraft.com> - 2011-09-30 08:11 -0400

#10511 — Benchmarking C compilers for embedded systems

I'd like to compare C compilers targetting embedded systems - the kind
that only have a few KB of RAM, etc.

However there seems to be a lack of software I could use. Waht I'm
looking for should be free, written (mostly) in standard C, be not
totally different from typical embedded applications.

So far I see:
- dhrystone
- The Contiki OS
- the files I've been using to track sdcc over revisions at
https://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list

Are there any other suitable benchmarks I could look at? They don't have
to be designed as benchmarks, other software, like the Contiki OS
mentioned above is OK, too. THey don't even have to "run", so libraries
would be OK, too.

Philipp

[toc] | [next] | [standalone]

#10521

On 19/09/11 19:43, Philipp Klaus Krause wrote:
> I'd like to compare C compilers targetting embedded systems - the kind
> that only have a few KB of RAM, etc.
>
> However there seems to be a lack of software I could use. Waht I'm
> looking for should be free, written (mostly) in standard C, be not
> totally different from typical embedded applications.
>
> So far I see:
> - dhrystone
> - The Contiki OS
> - the files I've been using to track sdcc over revisions at
> https://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>
> Are there any other suitable benchmarks I could look at? They don't have
> to be designed as benchmarks, other software, like the Contiki OS
> mentioned above is OK, too. THey don't even have to "run", so libraries
> would be OK, too.
>
> Philipp

There are no good general benchmarks for small embedded systems - partly 
because it would be a great deal of work (standard C is not standard for 
many compilers - you need to use compiler-specific features to get the 
best out of them), and partly because it would be of very little use.

What you want to know is which tools are the best for /your/ code, not 
some general code written for completely different uses.  Most tool 
vendors provide free evaluation software - test them out on your own code.

[toc] | [prev] | [next] | [standalone]

#10691

Am 19.09.2011 22:10, schrieb David Brown:
> On 19/09/11 19:43, Philipp Klaus Krause wrote:
>> I'd like to compare C compilers targetting embedded systems - the kind
>> that only have a few KB of RAM, etc.
>>
>> However there seems to be a lack of software I could use. Waht I'm
>> looking for should be free, written (mostly) in standard C, be not
>> totally different from typical embedded applications.
>>
>> So far I see:
>> - dhrystone
>> - The Contiki OS
>> - the files I've been using to track sdcc over revisions at
>> https://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>>
>> Are there any other suitable benchmarks I could look at? They don't have
>> to be designed as benchmarks, other software, like the Contiki OS
>> mentioned above is OK, too. THey don't even have to "run", so libraries
>> would be OK, too.
>>
>> Philipp
> 
> There are no good general benchmarks for small embedded systems - partly
> because it would be a great deal of work (standard C is not standard for
> many compilers - you need to use compiler-specific features to get the
> best out of them), and partly because it would be of very little use.
> 
> What you want to know is which tools are the best for /your/ code, not
> some general code written for completely different uses.  Most tool
> vendors provide free evaluation software - test them out on your own code.
> 

I don't just want to know which tool is best for my code. I want to
know, which tool is best for typical code.
I am one of the developers of the free sdcc compiler and currently
working on register and stack allcoation. I see that what I implemented
results in better code when compiling my own programs, Contiki,
dhrystone, Coremark. But I'd like to know if there is any code for which
the new approach performs worse than the on epreviously used in sdcc.
And I'd like to know it know, instead of having to wait for bug reports
about code size regressions coming in after the next release.

Philipp

[toc] | [prev] | [next] | [standalone]

#10693

On 22/09/2011 09:16, Philipp Klaus Krause wrote:
> Am 19.09.2011 22:10, schrieb David Brown:
>> On 19/09/11 19:43, Philipp Klaus Krause wrote:
>>> I'd like to compare C compilers targetting embedded systems - the kind
>>> that only have a few KB of RAM, etc.
>>>
>>> However there seems to be a lack of software I could use. Waht I'm
>>> looking for should be free, written (mostly) in standard C, be not
>>> totally different from typical embedded applications.
>>>
>>> So far I see:
>>> - dhrystone
>>> - The Contiki OS
>>> - the files I've been using to track sdcc over revisions at
>>> https://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>>>
>>> Are there any other suitable benchmarks I could look at? They don't have
>>> to be designed as benchmarks, other software, like the Contiki OS
>>> mentioned above is OK, too. THey don't even have to "run", so libraries
>>> would be OK, too.
>>>
>>> Philipp
>>
>> There are no good general benchmarks for small embedded systems - partly
>> because it would be a great deal of work (standard C is not standard for
>> many compilers - you need to use compiler-specific features to get the
>> best out of them), and partly because it would be of very little use.
>>
>> What you want to know is which tools are the best for /your/ code, not
>> some general code written for completely different uses.  Most tool
>> vendors provide free evaluation software - test them out on your own code.
>>
>
> I don't just want to know which tool is best for my code. I want to
> know, which tool is best for typical code.
> I am one of the developers of the free sdcc compiler and currently
> working on register and stack allcoation. I see that what I implemented
> results in better code when compiling my own programs, Contiki,
> dhrystone, Coremark. But I'd like to know if there is any code for which
> the new approach performs worse than the on epreviously used in sdcc.
> And I'd like to know it know, instead of having to wait for bug reports
> about code size regressions coming in after the next release.
>

I understand what you are trying to achieve - I just don't think it is 
possible.  There is no such thing as "typcial code" for embedded 
systems, and often "benchmark code" is so artificial that it doesn't 
represent the real world (though it can be useful for identifying 
particular cases).

What you are doing now, testing with a sample of your own real-world 
programs and other real-world embedded code (Contiki), is the best you 
can do.  If you want to do more, then collect more samples in a similar 
vein.  If sdcc has a collection of example code, then that would be a 
good start - you may also find that some users will give you programs 
you can use.  The toughest test is real-world code that is written for 
portability - good examples there would be FreeRTOS and LWIP.  These are 
code bases that are written to work on a wide range of targets, and 
can't take advantage of any target or compiler-specific tricks and 
optimisations, and they are projects that are used by a great many people.

[toc] | [prev] | [next] | [standalone]

#10559

On Sep 19, 10:43 pm, Philipp Klaus Krause <p...@spth.de> wrote:
> I'd like to compare C compilers targetting embedded systems - the kind
> that only have a few KB of RAM, etc.
>
> However there seems to be a lack of software I could use. Waht I'm
> looking for should be free, written (mostly) in standard C, be not
> totally different from typical embedded applications.
>
> So far I see:
> - dhrystone
> - The Contiki OS
> - the files I've been using to track sdcc over revisions athttps://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>
> Are there any other suitable benchmarks I could look at? They don't have
> to be designed as benchmarks, other software, like the Contiki OS
> mentioned above is OK, too. THey don't even have to "run", so libraries
> would be OK, too.
>
> Philipp

Hi Philipp,

Have you tried Coremark from EEMBC (www.coremark.org)?
It is available for free.
The other benchmarking suites from EEMBC require you to pay small
membership fee i guess.

Regards,
Abhishek

[toc] | [prev] | [next] | [standalone]

#10630

Am 20.09.2011 09:50, schrieb Abhishek:

> 
> Have you tried Coremark from EEMBC (www.coremark.org)?
> It is available for free.
> The other benchmarking suites from EEMBC require you to pay small
> membership fee i guess.

I now did, however the license is a problem:
1) The license. I can understand when benchmarking organizations don't
distribute older versions of their benchmarks and no longer accept
reults generated using an old version, like SPEC does. However the
Coremark license explicitly forbids you from just running or even
keeping a copy of older versions: You loose all rights to them each time
a new version is released.

I'll probably use it anyway and just hope they don't release a new
version anytime soon. Still, I'm looking for further benchmarks. The
Contiki OS seems perfect, but I'd like to find more of that kind.

Philipp

[toc] | [prev] | [next] | [standalone]

#10632

On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
> Am 20.09.2011 09:50, schrieb Abhishek:
>
>>
>> Have you tried Coremark from EEMBC (www.coremark.org)?
>> It is available for free.
>> The other benchmarking suites from EEMBC require you to pay small
>> membership fee i guess.
>
> I now did, however the license is a problem:
> 1) The license. I can understand when benchmarking organizations don't
> distribute older versions of their benchmarks and no longer accept
> reults generated using an old version, like SPEC does. However the
> Coremark license explicitly forbids you from just running or even
> keeping a copy of older versions: You loose all rights to them each time
> a new version is released.
>
> I'll probably use it anyway and just hope they don't release a new
> version anytime soon. Still, I'm looking for further benchmarks. The
> Contiki OS seems perfect, but I'd like to find more of that kind.

What about your own code?

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10690

Am 21.09.2011 11:06, schrieb Ian Collins:
> On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
>> Am 20.09.2011 09:50, schrieb Abhishek:
>>
>>>
>>> Have you tried Coremark from EEMBC (www.coremark.org)?
>>> It is available for free.
>>> The other benchmarking suites from EEMBC require you to pay small
>>> membership fee i guess.
>>
>> I now did, however the license is a problem:
>> 1) The license. I can understand when benchmarking organizations don't
>> distribute older versions of their benchmarks and no longer accept
>> reults generated using an old version, like SPEC does. However the
>> Coremark license explicitly forbids you from just running or even
>> keeping a copy of older versions: You loose all rights to them each time
>> a new version is released.
>>
>> I'll probably use it anyway and just hope they don't release a new
>> version anytime soon. Still, I'm looking for further benchmarks. The
>> Contiki OS seems perfect, but I'd like to find more of that kind.
> 
> What about your own code?
> 

I'd like to get results that are useful to other people as well. I
intend to use some of my code as one of the benchmarks benchmark, but in
general I'd prefer to use more generally accepted stuff.

Philipp

[toc] | [prev] | [next] | [standalone]

#10692

On 09/22/11 07:12 PM, Philipp Klaus Krause wrote:
> Am 21.09.2011 11:06, schrieb Ian Collins:
>> On 09/21/11 07:40 PM, Philipp Klaus Krause wrote:
>>> Am 20.09.2011 09:50, schrieb Abhishek:
>>>
>>>>
>>>> Have you tried Coremark from EEMBC (www.coremark.org)?
>>>> It is available for free.
>>>> The other benchmarking suites from EEMBC require you to pay small
>>>> membership fee i guess.
>>>
>>> I now did, however the license is a problem:
>>> 1) The license. I can understand when benchmarking organizations don't
>>> distribute older versions of their benchmarks and no longer accept
>>> reults generated using an old version, like SPEC does. However the
>>> Coremark license explicitly forbids you from just running or even
>>> keeping a copy of older versions: You loose all rights to them each time
>>> a new version is released.
>>>
>>> I'll probably use it anyway and just hope they don't release a new
>>> version anytime soon. Still, I'm looking for further benchmarks. The
>>> Contiki OS seems perfect, but I'd like to find more of that kind.
>>
>> What about your own code?
>>
>
> I'd like to get results that are useful to other people as well. I
> intend to use some of my code as one of the benchmarks benchmark, but in
> general I'd prefer to use more generally accepted stuff.

Every target and code base combination will behave differently, even on 
different members of the same processor family.  This makes general 
benchmarks useless for embedded systems.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#10694

On 09/19/2011 07:43 PM, Philipp Klaus Krause wrote:
> I'd like to compare C compilers targetting embedded systems - the kind
> that only have a few KB of RAM, etc.
>
> However there seems to be a lack of software I could use. Waht I'm
> looking for should be free, written (mostly) in standard C, be not
> totally different from typical embedded applications.
>
> So far I see:
> - dhrystone
> - The Contiki OS
> - the files I've been using to track sdcc over revisions at
> https://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>
> Are there any other suitable benchmarks I could look at? They don't have
> to be designed as benchmarks, other software, like the Contiki OS
> mentioned above is OK, too. THey don't even have to "run", so libraries
> would be OK, too.

Elm's FAT filesystem implementation:

http://elm-chan.org/fsw/ff/00index_e.html

[toc] | [prev] | [next] | [standalone]

#11105

Philipp Klaus Krause wrote:

> I'd like to compare C compilers targetting embedded systems - the kind
> that only have a few KB of RAM, etc.
>
> However there seems to be a lack of software I could use. Waht I'm
> looking for should be free, written (mostly) in standard C, be not
> totally different from typical embedded applications.
>
> So far I see:
> - dhrystone
> - The Contiki OS
> - the files I've been using to track sdcc over revisions at
> https://sourceforge.net/apps/trac/sdcc/wiki/Philipp%27s%20TODO%20list
>
> Are there any other suitable benchmarks I could look at? They don't have
> to be designed as benchmarks, other software, like the Contiki OS
> mentioned above is OK, too. THey don't even have to "run", so libraries
> would be OK, too.
>

Philipp,

There are tons of embedded systems benchmarks on the internet. In our
compiler work we have found the most informative benchmarks are application
type benchmarks close to the actual intended application.

The application fragment type benchmarks (fft, filters, math packs) are
useful generally is seeing how focused a compiler is in optimizing a
specific application area.

Generic benchmarks would likely need to also have some definition of the
intended application area.

There isn't much meaning in comparing the performance of processors used in
mice and processors used in home appliance controllers. Both are non hosted
embedded controllers with a few K of code but very different requirements.
They may even be members of the same family.

Walter..

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web