Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #103355 > unrolled thread

Instrumenting binaries with meta-gatherers for subsequent recompilation optimization

Back to article view | Back to comp.lang.c

Contents

  Instrumenting binaries with meta-gatherers for subsequent recompilation optimization "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2017-02-07 06:34 -0800
    Re: Instrumenting binaries with meta-gatherers for subsequent recompilation optimization David Brown <david.brown@hesbynett.no> - 2017-02-07 16:14 +0100
      Re: Instrumenting binaries with meta-gatherers for subsequent recompilation optimization "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2017-02-07 07:22 -0800
        Re: Instrumenting binaries with meta-gatherers for subsequent recompilation optimization "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2017-02-07 07:30 -0800
          Re: Instrumenting binaries with meta-gatherers for subsequent recompilation optimization David Brown <david.brown@hesbynett.no> - 2017-02-08 09:40 +0100
            Re: Instrumenting binaries with meta-gatherers for subsequent recompilation optimization "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2017-02-08 03:14 -0800
              Re: Instrumenting binaries with meta-gatherers for subsequent recompilation optimization David Brown <david.brown@hesbynett.no> - 2017-02-08 14:20 +0100
                Re: Instrumenting binaries with meta-gatherers for subsequent recompilation optimization "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2017-02-08 05:24 -0800

#103355 — Instrumenting binaries with meta-gatherers for subsequent recompilation optimization

I had the idea today of instrumenting the generated binary with meta
data capture abilities.  These would be little counters and flags
added by the compiler to collect runtime information, the purpose of
which would be to help it determine how best to optimize code in
future compiles.

By instrumenting the binary with this information, it generates a
live meta file at runtime based on a data set being processed, or a
use pattern if it's involves human interaction.  These can be fed
back in to the compiler for subsequent compilations, helping it make
choices which best serve that data set, or that user.

In such a case, and with the speed of computers growing by leaps and
bounds, and with the open-source nature of source code becoming far
more widely used, it may be such that a user could download the
instrumented version of their software, run it for a while, and then
submit their live meta file to a build bot, which then generates for
them a custom build tailored to their data set or use pattern.

Multiple live meta files could be used as input to create multiple
static versions people could download without having to give their
own feedback, etc.

-----
You know, it wasn't until the last couple of years that I've really
begun to realize how much control we have over these machines.  I've
always had the mindset of a computer being used by the tools that
people give us, but we really are the people who are here to make
the future what we desire it to be.

I am moving forward with these goals of giving this offering of my
RDC framework, CAlive compiler, Visual FreePro database manager and
GUI designer, Arxoda hardware, and so much more, to the Lord, and to
men.  It is how I want to proceed with my life.  I want people to
receive the fruits of my labors for their use in this world.  I want
people to know that Christians are not their enemy.  That we desire
with great passion to help everyone around us, and that help is not
just in tools like these, but it is in the real recognition of what
our lives mean, of who we are eternally, and what it means to be the
unique and special creations of God we are here on this planet, even
being distinct from every other form by leaps and bounds.

We are amazing people collectively.  We need to work together for the
Lord, for one another, so that we can become all that He's created us
to be here in this world, and in the world to come.

Thank you,
Rick C. Hodgin

[toc] | [next] | [standalone]

#103363

On 07/02/17 15:34, Rick C. Hodgin wrote:
> I had the idea today of instrumenting the generated binary with meta
> data capture abilities.  These would be little counters and flags
> added by the compiler to collect runtime information, the purpose of
> which would be to help it determine how best to optimize code in
> future compiles.
> 
> By instrumenting the binary with this information, it generates a
> live meta file at runtime based on a data set being processed, or a
> use pattern if it's involves human interaction.  These can be fed
> back in to the compiler for subsequent compilations, helping it make
> choices which best serve that data set, or that user.
> 
> In such a case, and with the speed of computers growing by leaps and
> bounds, and with the open-source nature of source code becoming far
> more widely used, it may be such that a user could download the
> instrumented version of their software, run it for a while, and then
> submit their live meta file to a build bot, which then generates for
> them a custom build tailored to their data set or use pattern.
> 
> Multiple live meta files could be used as input to create multiple
> static versions people could download without having to give their
> own feedback, etc.
> 

<https://en.wikipedia.org/wiki/Profile-guided_optimization>

It's been available in compilers such as MSVC and gcc for decades.  Some
people find it useful, others find it to inconvenient to use.  A
particular problem is that it can severely limit the optimisation and
code generation - it is very hard to instrumentalise functions without
affecting inlining, cloning, and other optimisations.  It gets even
worse if you want to instrumentalise loops, blocks, or other parts of
the code.  Thus your profiling could give data that simply does not
match what happens in optimised code.

Alternatives to instrumenting code are to use sampling to build up
statistics (use a timer-based interrupt to sample the program counter in
the main program), and "hardware performance counters" that can count
the number of times a particular piece of code is run without needing
software changes.  These methods have their own advantages and
disadvantages, of course.

All in all, profiling like this can be useful for some types of
software, but it is an art - it is not something you can just "switch
on" and it will result in better code.

[toc] | [prev] | [next] | [standalone]

#103364

On Tuesday, February 7, 2017 at 10:14:37 AM UTC-5, David Brown wrote:
> <https://en.wikipedia.org/wiki/Profile-guided_optimization>

I've heard of this before, but I never knew what it was.  Makes sense.

I had always assumed it was an externally defined set of #if..#endif
type conditions which then generated a particular type of binary.

Thank you,
Rick C. Hodgin

[toc] | [prev] | [next] | [standalone]

#103366

On Tuesday, February 7, 2017 at 10:22:59 AM UTC-5, Rick C. Hodgin wrote:
> On Tuesday, February 7, 2017 at 10:14:37 AM UTC-5, David Brown wrote:
> > <https://en.wikipedia.org/wiki/Profile-guided_optimization>
> I've heard of this before, but I never knew what it was.  Makes sense.

It gives me a great sense of pride and satisfaction to know that I am
coming up with things other real developers have come up with and have
in use, because these ones I post are created independently within my
own thinking, most of which just occur to me from time to time as I am
writing other code ... the idea of what's needed in that particular
scenario pops into my head as a fully-formed thought in that moment.

Thank you,
Rick C. Hodgin

[toc] | [prev] | [next] | [standalone]

#103430

On 07/02/17 16:30, Rick C. Hodgin wrote:
> On Tuesday, February 7, 2017 at 10:22:59 AM UTC-5, Rick C. Hodgin wrote:
>> On Tuesday, February 7, 2017 at 10:14:37 AM UTC-5, David Brown wrote:
>>> <https://en.wikipedia.org/wiki/Profile-guided_optimization>
>> I've heard of this before, but I never knew what it was.  Makes sense.
> 
> It gives me a great sense of pride and satisfaction to know that I am
> coming up with things other real developers have come up with and have
> in use, because these ones I post are created independently within my
> own thinking, most of which just occur to me from time to time as I am
> writing other code ... the idea of what's needed in that particular
> scenario pops into my head as a fully-formed thought in that moment.
> 

The other way to think about this is that it makes sense to learn more
about programming languages, programming language design, and
programming language implementation before making your own language and
tools.  That way you don't have to re-invent the wheel all the time, and
you can benefit from other people's experiences.

The idea of profile-guided optimisation is 60 years old, according to
Wikipedia - it's pretty much as old as programming languages.  It's not
really a difficult thing to think of - though implementing it well is
far from trivial.

[toc] | [prev] | [next] | [standalone]

#103447

David Brown wrote:
> The other way to think about this is that it makes sense to learn
> more about programming languages, programming language design,
> and programming language implementation before making your
> own language and tools. That way you don't have to re-invent the
> wheel all the time, and you can benefit from other people's experiences 

I purposefully stay away from such formal learning because it
maintains my unique perspective which, coupled to my creativity,
has produced some really great features thus far.  I'm actually quite
stunned I've been able to do what I have done.  But, as I've said, it's
not me.  These thoughts pop into my head fully formed.  I rarely have
to "think through" any of them.

If I learned more about languages, I might pigeonhole my own
internal vision.  And I have another reason I won't share publicly.

I've really only thought as I am doing now in the last few years.  And
in that time I've come up with a lot on my own.  I look forward to seeing
what the future holds as my labor is producing maturing tools, and
more necessary foundation.

Thank you,
Rick C. Hodgin

[toc] | [prev] | [next] | [standalone]

#103458

On 08/02/17 12:14, Rick C. Hodgin wrote:
> David Brown wrote:
>> The other way to think about this is that it makes sense to learn
>> more about programming languages, programming language design,
>> and programming language implementation before making your
>> own language and tools. That way you don't have to re-invent the
>> wheel all the time, and you can benefit from other people's experiences 
> 
> I purposefully stay away from such formal learning because it
> maintains my unique perspective which, coupled to my creativity,
> has produced some really great features thus far.  I'm actually quite
> stunned I've been able to do what I have done.  But, as I've said, it's
> not me.  These thoughts pop into my head fully formed.  I rarely have
> to "think through" any of them.
> 
> If I learned more about languages, I might pigeonhole my own
> internal vision.  And I have another reason I won't share publicly.

I can appreciate that you might feel that learning more about what
others have done might affect your creativity and the directions you
might take.  It will affect them positively, for the most part - but
perhaps sometimes negatively.  There is a balance to be found.  (As for
your "hidden" reason - it's entirely up to you to decide its importance.)

However, it is regularly clear to me that ideas that "stun" you are not
usually very impressive to others with a bit more understanding of
programming language history and concepts, and that most of your ideas
are very far from "fully formed".  (Other things, such as your
enthusiasm, energy, and ambition are more impressive or "stunning", as
are your abilities to learn new ideas and fields of study.)

If you were more open to external ideas, suggestions, and (constructive)
criticism then I think you would get on far better.  Accept that some of
your ideas are good, but some are bad.  Some are new, some have been
thought of long ago.  Some are inspired, some are obvious.  And accept
that there is /nothing/ that you can think of, or that "springs into
your mind", that cannot be improved by discussing it with others - that
way you will let your ideas be improved by discussion, instead of always
trying to defend them as though you had already decided on the "best"
way to do things.

> 
> I've really only thought as I am doing now in the last few years.  And
> in that time I've come up with a lot on my own.  I look forward to seeing
> what the future holds as my labor is producing maturing tools, and
> more necessary foundation.
> 
> Thank you,
> Rick C. Hodgin
> 

[toc] | [prev] | [next] | [standalone]

#103459

On Wednesday, February 8, 2017 at 8:20:13 AM UTC-5, David Brown wrote:
> [snip]

I appreciate your efforts to help me all these years, David.

Thank you,
Rick C. Hodgin

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.c

csiph-web