Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #18463 > unrolled thread

What is the right way to do this simple thing

Back to article view | Back to comp.lang.c

Contents

  What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-10 19:26 +0000
    Re: What is the right way to do this simple thing Don Y <this@isnotme.com> - 2012-03-10 12:57 -0700
    Re: What is the right way to do this simple thing ImpalerCore <jadill33@gmail.com> - 2012-03-10 14:12 -0800
      Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-10 23:31 +0000
        Re: What is the right way to do this simple thing Ian Collins <ian-news@hotmail.com> - 2012-03-11 13:06 +1300
          Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-11 00:44 +0000
            Re: What is the right way to do this simple thing Ian Collins <ian-news@hotmail.com> - 2012-03-11 14:15 +1300
              Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-11 04:46 +0000
                Re: What is the right way to do this simple thing Ian Collins <ian-news@hotmail.com> - 2012-03-11 21:27 +1300
                  Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-11 17:23 +0000
                    Re: What is the right way to do this simple thing Ian Collins <ian-news@hotmail.com> - 2012-03-12 10:01 +1300
                      Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-13 21:18 +0000
                        Re: What is the right way to do this simple thing Ian Collins <ian-news@hotmail.com> - 2012-03-14 10:23 +1300
                Re: What is the right way to do this simple thing Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-03-11 11:59 +0000
                  Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-11 16:51 +0000
                    Re: What is the right way to do this simple thing Gene <gene.ressler@gmail.com> - 2012-03-13 07:32 -0700
                      Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-13 21:46 +0000
        Re: What is the right way to do this simple thing ImpalerCore <jadill33@gmail.com> - 2012-03-12 07:28 -0700
          Re: What is the right way to do this simple thing ImpalerCore <jadill33@gmail.com> - 2012-03-12 08:13 -0700
          Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-13 21:51 +0000
    Re: What is the right way to do this simple thing Tim Rentsch <txr@alumni.caltech.edu> - 2012-03-10 14:21 -0800
      Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-11 01:40 +0000
        Re: What is the right way to do this simple thing Tim Rentsch <txr@alumni.caltech.edu> - 2012-03-11 14:11 -0700
          Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-13 21:56 +0000
            Re: What is the right way to do this simple thing Tim Rentsch <txr@alumni.caltech.edu> - 2012-03-14 14:55 -0700
              Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-15 21:55 +0000
                Re: What is the right way to do this simple thing Tim Rentsch <txr@alumni.caltech.edu> - 2012-03-19 13:33 -0700
    Re: What is the right way to do this simple thing Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-03-10 15:21 -0800
      Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-11 00:47 +0000
    Re: What is the right way to do this simple thing Philip Lantz <prl@canterey.us> - 2012-03-11 01:07 -0800
      Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-11 16:14 +0000
    Re: What is the right way to do this simple thing Markus Wichmann <nullplan@gmx.net> - 2012-03-11 13:47 +0100
      Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-11 16:24 +0000
    Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-11 21:01 +0000
      Re: What is the right way to do this simple thing Anders Wegge Keller <wegge@wegge.dk> - 2012-03-12 00:05 +0100
        Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-13 21:57 +0000
      Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-11 23:06 +0000
        Re: What is the right way to do this simple thing Geoff <geoff@invalid.invalid> - 2012-03-11 17:34 -0700
          Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-13 22:00 +0000
            Re: What is the right way to do this simple thing Geoff <geoff@invalid.invalid> - 2012-03-13 18:12 -0700
              Re: What is the right way to do this simple thing cri@tiac.net (Richard Harter) - 2012-03-14 03:38 +0000
                Re: What is the right way to do this simple thing Geoff <geoff@invalid.invalid> - 2012-03-14 00:49 -0700

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#18489

cri@tiac.net (Richard Harter) writes:

> Suppose that I have some data that is pointer free.  It isn't
> necessarily serialized in some format such as JSON but I would like to
> be able to copy it from one from one machine to another compatible
> machine and use it without trouble.  Also I don't want to pass around
> a collection of items and I want everything to be properly aligned.
>
> Naturally I think of using the struct hack and a structure something
> like this:
>
>    struct package {
>        message_type type;       /* The type has access info    */
>        size_t       total_len;  /* Total length of the package */
>        size_t       data_len;   /* Length of the data          */
>        char[1]      data;       /* The actual data             */
>    };
>
> Presumably at some point I will copy said data into an instance of the
> package which can then be passed around.  Also I really would like to
> be able to use the data upon receipt without copying it out, e.g.,
>
>    local_data = (data_type *)(pkg->data);
>
> It occurs to me that this might not work there is no guarantee that
> pkg->data is correctly aligned.  There are a whole bunch of other
> potential gotchas that I can foresee.  This is particularly troubling
> to me because my PC has four cores, each with a different DS9000 chip.
> What's worse my boss is on a "compiler of the week" binge.  Every week
> we get to test a new compiler.  I really want to get this code
> bullet-proof.
>
> Help!  What is the right way to do this simple thing?

I was in the middle of a somewhat longer reply when I changed
my mind and wrote this instead.

It's hard to give an answer without more information.  Some
of the key questions are:

  1. are the package struct's being passed through shared
     memory or through some sort of serial interface (such
     as file writing/reading or a network socket)?

  2. is it guaranteed that the raw data bytes will work on
     all the different machines used provided the base is
     suitably aligned?

  3. is 'message_type' a scalar type, or is it something
     more complicated that could be affected by changing
     between processors or compilers?

  4. I assume that all the different machines used will
     work identically on integer data types as long as
     the types have the same length and the data values
     have the same alignment from one machine to the next.
     Is this assumption correct, or might it be incorrect?
     If incorrect, then please say more.

  5. if shared memory is going to be used - is malloc()
     guaranteed to return a memory block that is suitably
     aligned in, and for, all the different execution
     environments used?  If not, please say more.

[toc] | [prev] | [next] | [standalone]

#18510

On Sat, 10 Mar 2012 14:21:23 -0800, Tim Rentsch
<txr@alumni.caltech.edu> wrote:

>cri@tiac.net (Richard Harter) writes:
>
>> Suppose that I have some data that is pointer free.  It isn't
>> necessarily serialized in some format such as JSON but I would like to
>> be able to copy it from one from one machine to another compatible
>> machine and use it without trouble.  Also I don't want to pass around
>> a collection of items and I want everything to be properly aligned.
>>
>> Naturally I think of using the struct hack and a structure something
>> like this:
>>
>>    struct package {
>>        message_type type;       /* The type has access info    */
>>        size_t       total_len;  /* Total length of the package */
>>        size_t       data_len;   /* Length of the data          */
>>        char[1]      data;       /* The actual data             */
>>    };
>>
>> Presumably at some point I will copy said data into an instance of the
>> package which can then be passed around.  Also I really would like to
>> be able to use the data upon receipt without copying it out, e.g.,
>>
>>    local_data = (data_type *)(pkg->data);
>>
>> It occurs to me that this might not work there is no guarantee that
>> pkg->data is correctly aligned.  There are a whole bunch of other
>> potential gotchas that I can foresee.  This is particularly troubling
>> to me because my PC has four cores, each with a different DS9000 chip.
>> What's worse my boss is on a "compiler of the week" binge.  Every week
>> we get to test a new compiler.  I really want to get this code
>> bullet-proof.
>>
>> Help!  What is the right way to do this simple thing?
>
>I was in the middle of a somewhat longer reply when I changed
>my mind and wrote this instead.
>
>It's hard to give an answer without more information.  Some
>of the key questions are:

All good questions.  It is quite unfair of you to ask astute
questions.

>
>  1. are the package struct's being passed through shared
>     memory or through some sort of serial interface (such
>     as file writing/reading or a network socket)?

I get to change the specs if needful.  That said, the presumption is
that there is a base level in which there is a single copy of the
package.  The only thing that is passed around at that level are
references.  An important fact is that there can only be one active
reference at any one one.  (I'll expand on that if needful.)

Separate threads can communicate through restricted shared memory.  At
the serial interface level there may or may not be a need for this
kind of packaging - the presumption is that the overhead of a full
serializing package becomes inconsequential. 
>
>  2. is it guaranteed that the raw data bytes will work on
>     all the different machines used provided the base is
>     suitably aligned?

No.  If they will work, we can use the cheaper method.  If not
serialize.

>
>  3. is 'message_type' a scalar type, or is it something
>     more complicated that could be affected by changing
>     between processors or compilers?

At present message type is scalar.  I already know how to handle
different message types in different processors.

>
>  4. I assume that all the different machines used will
>     work identically on integer data types as long as
>     the types have the same length and the data values
>     have the same alignment from one machine to the next.
>     Is this assumption correct, or might it be incorrect?
>     If incorrect, then please say more.

Either this assumption is correct or this type of packaging will not
be used.

>
>  5. if shared memory is going to be used - is malloc()
>     guaranteed to return a memory block that is suitably
>     aligned in, and for, all the different execution
>     environments used?  If not, please say more.

Malloc is expected to return a memory block that is suitably aligned
for the shared memory.  I am prepared to rule out memory shared by
execution environments that have different alignment requirements.

I hope this helps.

[toc] | [prev] | [next] | [standalone]

#18552

cri@tiac.net (Richard Harter) writes:

> On Sat, 10 Mar 2012 14:21:23 -0800, Tim Rentsch
> <txr@alumni.caltech.edu> wrote:
>
>>cri@tiac.net (Richard Harter) writes:
>>
>>> Suppose that I have some data that is pointer free.  It isn't
>>> necessarily serialized in some format such as JSON but I would like to
>>> be able to copy it from one from one machine to another compatible
>>> machine and use it without trouble.  Also I don't want to pass around
>>> a collection of items and I want everything to be properly aligned.
>>>
>>> Naturally I think of using the struct hack and a structure something
>>> like this:
>>>
>>>    struct package {
>>>        message_type type;       /* The type has access info    */
>>>        size_t       total_len;  /* Total length of the package */
>>>        size_t       data_len;   /* Length of the data          */
>>>        char[1]      data;       /* The actual data             */
>>>    };
>>>
>>> Presumably at some point I will copy said data into an instance of the
>>> package which can then be passed around.  Also I really would like to
>>> be able to use the data upon receipt without copying it out, e.g.,
>>>
>>>    local_data = (data_type *)(pkg->data);
>>>
>>> It occurs to me that this might not work there is no guarantee that
>>> pkg->data is correctly aligned.  There are a whole bunch of other
>>> potential gotchas that I can foresee.  This is particularly troubling
>>> to me because my PC has four cores, each with a different DS9000 chip.
>>> What's worse my boss is on a "compiler of the week" binge.  Every week
>>> we get to test a new compiler.  I really want to get this code
>>> bullet-proof.
>>>
>>> Help!  What is the right way to do this simple thing?
>>
>>I was in the middle of a somewhat longer reply when I changed
>>my mind and wrote this instead.
>>
>>It's hard to give an answer without more information.  Some
>>of the key questions are:
>
> All good questions.  It is quite unfair of you to ask astute
> questions.  

I try.
 
> [snip q's and a's]

Let me try a very pragmatic suggestion, first motivating it, and
afterwards talk about how to mitigate a potential objection.

It seems likely that all the processors you'll be running on
will share two characteristics (please tell me the first one
might NOT be true of all your target platforms):

    CHAR_BIT == 8
    sizes of all "elemental" data types are powers of two

If we assume for a moment that these two conditions are true, then
we might try something like this:

    struct meta_stuff {
        ... whatever you want in here as long as you can get
            it out reliably, and the size isn't "too big" ...
    };

    #define DATA_OFFSET 64   /* or 128, 256, etc, as appropriate */

    union package {
        struct meta_stuff meta;
        unsigned char bytes[ DATA_OFFSET * 2 ];
    };

    ... make sure 'sizeof (struct meta_data) <= DATA_OFFSET' ...

    local_data = (data_type *) &package->bytes[ DATA_OFFSET ];

Obviously we might want to wrap the RHS of that expression in
a macro call or something similar.  That's just a style issue,
no need to delve into it further.

Now, as long as we use malloc() to reserve space for packages,
and all the elemental types have sizes that are powers of two no
larger than 64, this approach will work.  (Note:  I am also
assuming that stuff in the data area isn't some weird mapped
memory or something similar that has some exotic (and probably
extra-linguistic) alignment requirement.  If there is then
Heaven help you.)  As long as the sizes are all powers of two,
we don't have to care about their alignments.

Practically speaking I expect this approach will work for all
your current or future target platforms.  But now let's suppose
that it doesn't;  suppose it works for some but not all, or
may fail on some future platform.  This possibility is easy
to protect against:

    local_data =
        CHAR_BIT != 8  ||
	DATA_OFFSET % offsetof( struct { char c; data_type t;}, t )
	    ?  (data_type *) weird_case_fallback( package, ...etc... )
            :  (data_type *) &package->bytes[ DATA_OFFSET ];

Again we probably want to wrap this in a macro or some similar
scheme.  I wrote it out in full so the details would be evident.
The 'weird_case_fallback()' function would be responsible for
extracting out the relevant bytes appropriately, eg, copying into
some suitably aligned structure.  That shouldn't be too hard to do,
especially since the weird_case_fallback() function can be made
platform specific (perhaps by doing a platform check and setting a
pointer-to-function variable during initialization).

Obviously there needs to be some similar accommodatation in the
code that "creates" packages.  I leave the details of that to you.

If it were me, I would propose a scheme like this one and suggest
running with it until there was some evidence that a different
scheme would be better.  That lets us get something running now
that probably will work for every target we're actually going to
run on but still leaves an out for "difficult" targets.  And if
later we need to switch to a different scheme, we will then have a
better idea of why that is, and what a better scheme might be.

Also, one other minor point - the alignment check shown in the last
code fragment can also be done during initialization of each
process (or processor, perhaps, depending) to give an early warning
of potential problems.  Doing these checks, for all potential
payload types if possible, is probably a good idea no matter what
scheme is used to align the payload data.

[toc] | [prev] | [next] | [standalone]

#18706

On Sun, 11 Mar 2012 14:11:50 -0700, Tim Rentsch
<txr@alumni.caltech.edu> wrote:

>cri@tiac.net (Richard Harter) writes:
>
>> On Sat, 10 Mar 2012 14:21:23 -0800, Tim Rentsch
>> <txr@alumni.caltech.edu> wrote:
>>
>>>cri@tiac.net (Richard Harter) writes:

[snip lots of good stuff]

Basically what you seem to be saying is to use an appropriate power of
two to force alignment.  Sounds good to me, but it's not quite squeaky
clean bullet proof.  Just a crochet.

[toc] | [prev] | [next] | [standalone]

#18782

cri@tiac.net (Richard Harter) writes:

> On Sun, 11 Mar 2012 14:11:50 -0700, Tim Rentsch
> <txr@alumni.caltech.edu> wrote:
>
>>cri@tiac.net (Richard Harter) writes:
>>
>>> On Sat, 10 Mar 2012 14:21:23 -0800, Tim Rentsch
>>> <txr@alumni.caltech.edu> wrote:
>>>
>>>>cri@tiac.net (Richard Harter) writes:
>
> [snip lots of good stuff]
>
> Basically what you seem to be saying is to use an appropriate power of
> two to force alignment.  Sounds good to me, but it's not quite squeaky
> clean bullet proof.  Just a crochet.

In that case I would want to ask for a more complete description
of what constitutes, using your phrase, a squeaky clean bullet
proof solution.  An arbitrarily perverse implementation can
screw up basically any scheme imaginable for this problem.  What
simplifying assumptions are you willing to grant about the
various implementations involved, or what parts of the language
can be excluded (one of those is not having pointers in the data
area; what others)?  If there aren't any then I think the
problem is pretty much intractable.

Related to that, unless I am misunderstanding something, there
is a significant flaw in the approach you outlined in another
message, where a struct is used to place and align the payload
data within a larger package structure, and an offset for
locating the data is put in the package header.  The problem is,
the alignment guarantees for the processor (and implementation)
that _produces_ the package may not be sufficient to meet the
alignment requirements for the processor (and implementation)
that _receives_ the package.  Whatever alignment guarantees are
made must suffice to meet the requirements for all processors
and all implementations simultaneously.  This stipulation is
what takes the problem out of the realm of just writing
"portable code".  Getting the alignment exactly right just in
the package-creating process doesn't solve the problem.

[toc] | [prev] | [next] | [standalone]

#18814

On Wed, 14 Mar 2012 14:55:01 -0700, Tim Rentsch
<txr@alumni.caltech.edu> wrote:

>cri@tiac.net (Richard Harter) writes:
>
>> On Sun, 11 Mar 2012 14:11:50 -0700, Tim Rentsch
>> <txr@alumni.caltech.edu> wrote:
>>
>>>cri@tiac.net (Richard Harter) writes:
>>>
>>>> On Sat, 10 Mar 2012 14:21:23 -0800, Tim Rentsch
>>>> <txr@alumni.caltech.edu> wrote:
>>>>
>>>>>cri@tiac.net (Richard Harter) writes:
>>
>> [snip lots of good stuff]
>>
>> Basically what you seem to be saying is to use an appropriate power of
>> two to force alignment.  Sounds good to me, but it's not quite squeaky
>> clean bullet proof.  Just a crochet.
>
>In that case I would want to ask for a more complete description
>of what constitutes, using your phrase, a squeaky clean bullet
>proof solution.  An arbitrarily perverse implementation can
>screw up basically any scheme imaginable for this problem.  What
>simplifying assumptions are you willing to grant about the
>various implementations involved, or what parts of the language
>can be excluded (one of those is not having pointers in the data
>area; what others)?  If there aren't any then I think the
>problem is pretty much intractable.

Agreed that the general problem is intractable.  Here is some
background.

The relevant context is a message passing system written in C.
Messages are passed between "mini-processes" which behave as though
they were processes (suspend/resume logic, input ports and output
ports, etc).  The architecture is

Mini-processes run within process managers
Multiple process managers can run within a single OS thread.
Process managers can run in different OS threads.
Process managers can run on multiple networked processers.
Process managers can run on processors connected by the internet.

Key facts are:  Process managers are single threaded; they are
(supposed to be) fast.  Mini-processes are light-weight, processing
isn't much heavier than function calls.  Mini-processes don't know
where data comes from or where it goes.

One of the issues in these kind of systems is what the inputs look
like.  In shell programming the input ('the' is a problem in shell
programming) is in the form of a stream of characters.  In Erlang it
is in the form of a list.  

In this system inputs and outputs are typed.  Moreover inputs to a
mini-process DO NOT HAVE TO BE DECODED.  They can be used much as
though they were arguments to a function.  Similarly outputs do not
have to be encoded.

Clearly, we can move data as is between the mini-processes running
under a single process manager.  Similarly we should be able to move
data as is from one OS-thread to another in a single processor.  After
that it gets a little fuzzy.

Connecting process managers requires the cooperation of magic network
software.  The magic network software is in charge of moving messages
from one process manager to another.  It decides whether the data must
be serialized and deserialized using something JSON.  (Actually it
tells the process managers to do it.)  Don't ask about addressing.
You really don't want to know.

So it comes down to this.  We want a solid data format that is usable
within process managers.  We would like it to be usable between
process managers if the connection are otherwise lightware and they
are consistent implementations.  Otherwise we serialize.  (And
lesnerize.) However that decision is left to matic network manager.

All clear?  If, not I have more mud.
      




>
>Related to that, unless I am misunderstanding something, there
>is a significant flaw in the approach you outlined in another
>message, where a struct is used to place and align the payload
>data within a larger package structure, and an offset for
>locating the data is put in the package header.  The problem is,
>the alignment guarantees for the processor (and implementation)
>that _produces_ the package may not be sufficient to meet the
>alignment requirements for the processor (and implementation)
>that _receives_ the package.  Whatever alignment guarantees are
>made must suffice to meet the requirements for all processors
>and all implementations simultaneously.  This stipulation is
>what takes the problem out of the realm of just writing
>"portable code".  Getting the alignment exactly right just in
>the package-creating process doesn't solve the problem.

No.  You can play the same game at the receiving end to see what the
offset is at the receiving end.  They had better be the same.  If not
that is prima facie evidence that the implementations are
inconsistent.

[toc] | [prev] | [next] | [standalone]

#18877

cri@tiac.net (Richard Harter) writes:

> On Wed, 14 Mar 2012 14:55:01 -0700, Tim Rentsch
> <txr@alumni.caltech.edu> wrote:
>
>>cri@tiac.net (Richard Harter) writes:
>>
>>> On Sun, 11 Mar 2012 14:11:50 -0700, Tim Rentsch
>>> <txr@alumni.caltech.edu> wrote:
>>>
>>>>cri@tiac.net (Richard Harter) writes:
>>>>
>>>>> On Sat, 10 Mar 2012 14:21:23 -0800, Tim Rentsch
>>>>> <txr@alumni.caltech.edu> wrote:
>>>>>
>>>>>>cri@tiac.net (Richard Harter) writes:
>>>
>>> [snip lots of good stuff]
>>>
>>> Basically what you seem to be saying is to use an appropriate power of
>>> two to force alignment.  Sounds good to me, but it's not quite squeaky
>>> clean bullet proof.  Just a crochet.
>>
>>In that case I would want to ask for a more complete description
>>of what constitutes, using your phrase, a squeaky clean bullet
>>proof solution.  An arbitrarily perverse implementation can
>>screw up basically any scheme imaginable for this problem.  What
>>simplifying assumptions are you willing to grant about the
>>various implementations involved, or what parts of the language
>>can be excluded (one of those is not having pointers in the data
>>area; what others)?  If there aren't any then I think the
>>problem is pretty much intractable.
>
> Agreed that the general problem is intractable.  Here is some
> background.
>
> The relevant context is a message passing system written in C.
> Messages are passed between "mini-processes" which behave as though
> they were processes (suspend/resume logic, input ports and output
> ports, etc).  The architecture is
>
> Mini-processes run within process managers
> Multiple process managers can run within a single OS thread.
> Process managers can run in different OS threads.
> Process managers can run on multiple networked processers.
> Process managers can run on processors connected by the internet.
>
> Key facts are:  Process managers are single threaded; they are
> (supposed to be) fast.  Mini-processes are light-weight, processing
> isn't much heavier than function calls.  Mini-processes don't know
> where data comes from or where it goes.
>
> One of the issues in these kind of systems is what the inputs look
> like.  In shell programming the input ('the' is a problem in shell
> programming) is in the form of a stream of characters.  In Erlang it
> is in the form of a list.  
>
> In this system inputs and outputs are typed.  Moreover inputs to a
> mini-process DO NOT HAVE TO BE DECODED.  They can be used much as
> though they were arguments to a function.  Similarly outputs do not
> have to be encoded.
>
> Clearly, we can move data as is between the mini-processes running
> under a single process manager.  Similarly we should be able to move
> data as is from one OS-thread to another in a single processor.  After
> that it gets a little fuzzy.
>
> Connecting process managers requires the cooperation of magic network
> software.  The magic network software is in charge of moving messages
> from one process manager to another.  It decides whether the data must
> be serialized and deserialized using something JSON.  (Actually it
> tells the process managers to do it.)  Don't ask about addressing.
> You really don't want to know.
>
> So it comes down to this.  We want a solid data format that is usable
> within process managers.  We would like it to be usable between
> process managers if the connection are otherwise lightware and they
> are consistent implementations.  Otherwise we serialize.  (And
> lesnerize.) However that decision is left to matic network manager.
>
> All clear?  If, not I have more mud.

What I think you're saying is:

Although the data (inputs/outputs) are typed, they are treated
basically as raw bytes.  Structure imposed on top of those raw
bytes is done as part of the Erlang environment, ie, they are not
C structs or whatever.

A bundle of data ("payload") is enclosed in a C struct (or
"struct-like", eg, a struct with variable-length last member) for
the purpose of being able to reference such things from C.

The data bundles are moved and/or copied and/or "remapped" (eg,
may be in the same address space if there are process managers in
sibling threads) from one process manager to another.  This is
always guaranteed to work intra-PM because a single PM is always
done on a single implementation.  The question is how to transmit
data bundles inter-PM, because those may be implemented in very
different implementations.

Having said all that. my main take-away is that all that's
important in terms of C issues is to get the top-level enclosing
struct's to be "right", because what's inside the structs will
work as long as the enclosing structs can be accessed definedly in
the C environments in question.  Is that right?  If not, then I
have missed something or misunderstood.


>>Related to that, unless I am misunderstanding something, there
>>is a significant flaw in the approach you outlined in another
>>message, where a struct is used to place and align the payload
>>data within a larger package structure, and an offset for
>>locating the data is put in the package header.  The problem is,
>>the alignment guarantees for the processor (and implementation)
>>that _produces_ the package may not be sufficient to meet the
>>alignment requirements for the processor (and implementation)
>>that _receives_ the package.  Whatever alignment guarantees are
>>made must suffice to meet the requirements for all processors
>>and all implementations simultaneously.  This stipulation is
>>what takes the problem out of the realm of just writing
>>"portable code".  Getting the alignment exactly right just in
>>the package-creating process doesn't solve the problem.
>
> No.  You can play the same game at the receiving end to see
> what the offset is at the receiving end.  They had better be
> the same.  If not that is prima facie evidence that the
> implementations are inconsistent.

In that case I don't see any significant difference between this
approach and the constant-but-checked 64-byte aligned approach I
described earlier, except that, practically speaking, using a
fixed 64-byte alignment is likely to work (ie, and not require any
serialization) across a wider variety of implementations than the
"sender-and-receiver-must-exactly-agree" approach.  In particular,
based on your explanation I believe the two approaches are equally
"squeaky clean" in terms of not screwing up, but the fixed offset
will probably require a fallback to serialization less often.

[toc] | [prev] | [next] | [standalone]

#18499

On Mar 10, 7:26 pm, c...@tiac.net (Richard Harter) wrote:
> Suppose that I have some data that is pointer free.  It isn't
> necessarily serialized in some format such as JSON but I would like to
> be able to copy it from one from one machine to another compatible
> machine and use it without trouble.
> Help!  What is the right way to do this simple thing?
>
It's not formally guaranteed that structs have the same layout in
memory on different compilers. But on modern systems there is usally a
rich collection of libraries that take structs as parameters, and if
the assumption that all structs are the same doesn't hold, programs
will break.
The main potential problem I see with your code is that the struct is
aligned on a 12-byte boundary (presumbaly) whilst the struct might
contains doubles which might need 8 byte alignment. That's easily
enough fixed, at the cost of four bytes.

[toc] | [prev] | [next] | [standalone]

#18505

On Sat, 10 Mar 2012 15:21:22 -0800 (PST), Malcolm McLean
<malcolm.mclean5@btinternet.com> wrote:

>On Mar 10, 7:26=A0pm, c...@tiac.net (Richard Harter) wrote:
>> Suppose that I have some data that is pointer free. =A0It isn't
>> necessarily serialized in some format such as JSON but I would like to
>> be able to copy it from one from one machine to another compatible
>> machine and use it without trouble.
>> Help! =A0What is the right way to do this simple thing?
>>
>It's not formally guaranteed that structs have the same layout in
>memory on different compilers. But on modern systems there is usally a
>rich collection of libraries that take structs as parameters, and if
>the assumption that all structs are the same doesn't hold, programs
>will break.
>The main potential problem I see with your code is that the struct is
>aligned on a 12-byte boundary (presumbaly) whilst the struct might
>contains doubles which might need 8 byte alignment. That's easily
>enough fixed, at the cost of four bytes.

The trouble with that sort of answer is that you are assuming that the
4 byte and 8 byte alignments suffice.  Wouldn't it be better to use a
solution that does not make alignment assumptions?

[toc] | [prev] | [next] | [standalone]

#18523

Richard Harter wrote:
> 
> Suppose that I have some data that is pointer free.  It isn't
> necessarily serialized in some format such as JSON but I would like to
> be able to copy it from one from one machine to another compatible
> machine and use it without trouble.  Also I don't want to pass around
> a collection of items and I want everything to be properly aligned.
> 
> Naturally I think of using the struct hack and a structure something
> like this:
> 
>    struct package {
>        message_type type;       /* The type has access info    */
>        size_t       total_len;  /* Total length of the package */
>        size_t       data_len;   /* Length of the data          */
>        char[1]      data;       /* The actual data             */
>    };
> 
> Presumably at some point I will copy said data into an instance of the
> package which can then be passed around.  Also I really would like to
> be able to use the data upon receipt without copying it out, e.g.,
> 
>    local_data = (data_type *)(pkg->data);
> 
> It occurs to me that this might not work there is no guarantee that
> pkg->data is correctly aligned.  There are a whole bunch of other
> potential gotchas that I can foresee.  This is particularly troubling
> to me because my PC has four cores, each with a different DS9000 chip.
> What's worse my boss is on a "compiler of the week" binge.  Every week
> we get to test a new compiler.  I really want to get this code
> bullet-proof.
> 
> Help!  What is the right way to do this simple thing?


I'm not sure why you're only concerned about the alignment of pkg->data. 
When passing a structure to a different processor, you also have to be 
concerned about the size, alignment, padding, and semantics of every 
field within the structure. The whole idea of passing a struct is 
futile. You need to convert the struct to a processor-independent byte 
stream, and convert the byte stream back to a struct on the receiving 
end.

[toc] | [prev] | [next] | [standalone]

#18535

On Sun, 11 Mar 2012 01:07:53 -0800, Philip Lantz <prl@canterey.us>
wrote:

>Richard Harter wrote:
>> 
>> Suppose that I have some data that is pointer free.  It isn't
>> necessarily serialized in some format such as JSON but I would like to
>> be able to copy it from one from one machine to another compatible
>> machine and use it without trouble.  Also I don't want to pass around
>> a collection of items and I want everything to be properly aligned.
>> 
>> Naturally I think of using the struct hack and a structure something
>> like this:
>> 
>>    struct package {
>>        message_type type;       /* The type has access info    */
>>        size_t       total_len;  /* Total length of the package */
>>        size_t       data_len;   /* Length of the data          */
>>        char[1]      data;       /* The actual data             */
>>    };
>> 
>> Presumably at some point I will copy said data into an instance of the
>> package which can then be passed around.  Also I really would like to
>> be able to use the data upon receipt without copying it out, e.g.,
>> 
>>    local_data = (data_type *)(pkg->data);
>> 
>> It occurs to me that this might not work there is no guarantee that
>> pkg->data is correctly aligned.  There are a whole bunch of other
>> potential gotchas that I can foresee.  This is particularly troubling
>> to me because my PC has four cores, each with a different DS9000 chip.
>> What's worse my boss is on a "compiler of the week" binge.  Every week
>> we get to test a new compiler.  I really want to get this code
>> bullet-proof.
>> 
>> Help!  What is the right way to do this simple thing?
>
>
>I'm not sure why you're only concerned about the alignment of pkg->data. 
>When passing a structure to a different processor, you also have to be 
>concerned about the size, alignment, padding, and semantics of every 
>field within the structure. The whole idea of passing a struct is 
>futile. You need to convert the struct to a processor-independent byte 
>stream, and convert the byte stream back to a struct on the receiving 
>end.

Did you see the word "compatible"?  Clearly if the processors are not
compatible, i.e., do not have the same size etc then, yes, you need a
processor-independent byte stream.  However you can also have multiple
processors that are the same, multiple cores, and multiple "processes"
running within a single processor.  If you are doing cheap message
passing the cost of serializing/de-serializing is high overhead.

[toc] | [prev] | [next] | [standalone]

#18530

On 10.03.2012 20:26, Richard Harter wrote:
> Suppose that I have some data that is pointer free.  It isn't
> necessarily serialized in some format such as JSON but I would like to
> be able to copy it from one from one machine to another compatible
> machine and use it without trouble.  Also I don't want to pass around
> a collection of items and I want everything to be properly aligned.
> 
> [...]

JSON makes an object into a string representation that can be passed
around and parsed. Note, however, that parsing is expensive; more so,
the more complicated a language gets.

One simple way to send a structure over the network would be to define a
packet format _to_the_byte_ and then use that. For instance, if you have
to pass the structure:

struct foo {
    unsigned int foo;
    unsigned int bar;
    char* string;
}

You could define a packet structure:

- All data is sent in big endian (as is custom in the network)

Offset |	Size |	Description
0		4	foo
4		4	bar
8		4	len
12		len	string

Neither len nor string include a terminating NULL character.

Then you would send the stuff:

#include <limits.h>
#if CHAR_BIT > 8
#error Sorry, this code only works on platforms with 8 bit bytes.
#endif

static void uint_to_buf_be(unsigned char* start, unsigned int data)
{
    start[0] = (data >> 24) & 255;
    start[1] = (data >> 16) & 255;
    start[2] = (data >>  8) & 255;
    start[3] = (data >>  0) & 255;
}

void send_struct_foo(struct foo* data)
{
    static char buf[1500];
    size_t len;
    len = strlen(data->string);
    uint_to_buf_be(buf + 0, data->foo);
    uint_to_buf_be(buf + 4, data->bar);
    uint_to_buf_be(buf + 8, len);
    memcpy(buf + 12, data->string, len);
    send(out_socket, buf, len + 12);
}

And receive it:

#include <limits.h>
#if CHAR_BIT > 8
#error Sorry, this code only works on platforms with 8 bit bytes.
#endif

static unsigned int buf_be_to_uint(unsigned char* start)
{
    unsigned int r;
    r = start[0] * 16777216
      + start[1] *    65536
      + start[2] *      256
      + start[3];
    return r;
}

void recv_struct_foo(struct foo* data)
{
    static unsigned char buf[1500];
    size_t len;
    recv(in_socket, buf, sizeof buf);
    data->foo = buf_be_to_uint(buf + 0);
    data->bar = buf_be_to_uint(buf + 4);
    len       = buf_be_to_uint(buf + 8);

    data->string = malloc(len+1);
    if (!data->string) abort();
    memcpy(data->string, buf + 12, len);
    data->string[len] = 0;
}

Another, oft employed and "more efficient" solution is simply using
memcpy(). However, that will only work when sending the data to a
machine with the same ABI. That is, same endianess, same rules for
alignment in aggregates, same rules for alignment _of_ aggregates; when
sending them, same representation of floating-point numbers and so on.
There are very many ways that can go wrong. You experienced some of them
on your own.

HTH,
Markus

[toc] | [prev] | [next] | [standalone]

#18536

On Sun, 11 Mar 2012 13:47:52 +0100, Markus Wichmann <nullplan@gmx.net>
wrote:

>On 10.03.2012 20:26, Richard Harter wrote:
>> Suppose that I have some data that is pointer free.  It isn't
>> necessarily serialized in some format such as JSON but I would like to
>> be able to copy it from one from one machine to another compatible
>> machine and use it without trouble.  Also I don't want to pass around
>> a collection of items and I want everything to be properly aligned.
>> 
>> [...]
>
>JSON makes an object into a string representation that can be passed
>around and parsed. Note, however, that parsing is expensive; more so,
>the more complicated a language gets.

Just so.
>
>One simple way to send a structure over the network would be to define a
>packet format _to_the_byte_ and then use that. For instance, if you have
>to pass the structure:

[Snip byte packing technique and code]

It's an interesting technique.  However it looks to be incredibly
cumbersome if one has many different message types.

>Another, oft employed and "more efficient" solution is simply using
>memcpy(). However, that will only work when sending the data to a
>machine with the same ABI. That is, same endianess, same rules for
>alignment in aggregates, same rules for alignment _of_ aggregates; when
>sending them, same representation of floating-point numbers and so on.
>There are very many ways that can go wrong. You experienced some of them
>on your own.

I did?

Thank you for the comments and suggestions.

[toc] | [prev] | [next] | [standalone]

#18550

On Sat, 10 Mar 2012 19:26:15 GMT, cri@tiac.net (Richard Harter) wrote:

I'm going to try this again with some background material.  Maybe that
will help.

Imagine a network of mini-processes similar to Erlang's "processes".
There are several possible levels of networking.  Thus all of the
mini-processes may run within an OS thread.  They may run within
several OS threads on a single machine.  They may run on a network of
similar machines.  They may run on a network of different machines.
In addition to all of the mini-processes there is process-manager that
takes care of communication, storage allocation, scheduling, and other
necessary functions.

A reasonable attempt is made for the network to be efficient.
Basically this implies that most of the communication is local.

Mini-processes communicate by message passing.  Life is much simpler
if all of the message packets that the mini-processes see use a single
standard format.  This is a requirement.  It should be simple.  That
is another requirement.  It should be bullet proof.  That is yet
another requirement.

There can be many types of messages with different formats.  There can
be hundreds or even thousands of different types.  The scheme I am
using is to use integers for message types.

What does that format look like?  One possibility is to encode the
messages in JSON or some other serialization protocol.  This is
expensive.  Better is to use a cheap format for input and output from
the mini-processes.  Let the process manager determine whether
serialization is needed and let it take care of the details.  It can
convert from JSON to local format as needed.

Here is a tentative package format; it consists of a header and a data
payload.  There is the header.

   struct package_header {
      message_type type;
      size_t       total_length;
      size_t       data_length;
      int          offset;
   };
 
And here is the package in the received and sent format:

   struct package {
      struct package_header header;
      char[1] data;
   } pkg;

The actual data payload begins at pkg.data[pkg.header.offset].  To
load data we use a second structure thusly,

    struct load_package {
      struct package_header header;
      payload_t data;
   } pkg_load;

The payload is copied into data.  We get the offset by subtracting the
address of pkg_load from pkg_load.data.  Finally we cast pkg_load to
struct package when we send it.

There may be a gotcha in there that I haven't spotted.  There may be a
fundamentally cleaner way to do it.  If so, I'd love to hear about it.


 

Within any particular
            
    

[toc] | [prev] | [next] | [standalone]

#18559

cri@tiac.net (Richard Harter) writes:

> On Sat, 10 Mar 2012 19:26:15 GMT, cri@tiac.net (Richard Harter) wrote:
> 
> I'm going to try this again with some background material.  Maybe that
> will help.
> 
> Imagine a network of mini-processes similar to Erlang's "processes".
> There are several possible levels of networking.  Thus all of the
> mini-processes may run within an OS thread.  They may run within
> several OS threads on a single machine.  They may run on a network of
> similar machines.  They may run on a network of different machines.
> In addition to all of the mini-processes there is process-manager that
> takes care of communication, storage allocation, scheduling, and other
> necessary functions.

 I happen to maintain a codebase that's almost as you describe. Most
of the processes run as their own OS process, the architecture is
identical enough (X86 all the way) that we don't have to worry too
much about alignment and ordering. So this real-world example have
some limitations as a general solution.

 ...

> There may be a gotcha in there that I haven't spotted.  There may be a
> fundamentally cleaner way to do it.  If so, I'd love to hear about it.

 All of the messages passed between the tasks have an identical header:

typedef struct head {
    word                length ;        /* Length of message            */
    word                dcode ;         /* Destination code             */
    word                acode ;         /* Answer destination code      */
    byte                rcode ;         /* Request code                 */
    byte                ccode ;         /* Completion code              */
} Head_t ;

 Each individual message starts off with the header:

typedef struct  turnoff_mes
{
    Head_t              head ;
    word                node ;
    word                c_type ;
} Turnoff_mes ;

 A process that receives a messag start by looking at the rcode value
in head. Each (rcode,dcode)-tuple identifies a unique message
structure, so the receiving process can cast the received message to
the appropriate struct, and process it.

 Sending a message to a specific process is as simple as including the
header with the message definitions, just like calling a function in
another translation unit, requires including the appropriate prototype
header.

 I find this pretty simple, but I've been working with this for more
than a decade, so for all I know, I could suffer from Helsinki syndrome. 

-- 
/Wegge

Leder efter redundant peering af dk.*,linux.debian.*

[toc] | [prev] | [next] | [standalone]

#18707

On 12 Mar 2012 00:05:30 +0100, Anders Wegge Keller <wegge@wegge.dk>
wrote:

>cri@tiac.net (Richard Harter) writes:
>
>> On Sat, 10 Mar 2012 19:26:15 GMT, cri@tiac.net (Richard Harter) wrote:
>> 
>> I'm going to try this again with some background material.  Maybe that
>> will help.
>> 
>> Imagine a network of mini-processes similar to Erlang's "processes".
>> There are several possible levels of networking.  Thus all of the
>> mini-processes may run within an OS thread.  They may run within
>> several OS threads on a single machine.  They may run on a network of
>> similar machines.  They may run on a network of different machines.
>> In addition to all of the mini-processes there is process-manager that
>> takes care of communication, storage allocation, scheduling, and other
>> necessary functions.
>
> I happen to maintain a codebase that's almost as you describe. Most
>of the processes run as their own OS process, the architecture is
>identical enough (X86 all the way) that we don't have to worry too
>much about alignment and ordering. So this real-world example have
>some limitations as a general solution.
>
> ...
>
>> There may be a gotcha in there that I haven't spotted.  There may be a
>> fundamentally cleaner way to do it.  If so, I'd love to hear about it.
>
> All of the messages passed between the tasks have an identical header:
>
>typedef struct head {
>    word                length ;        /* Length of message            */
>    word                dcode ;         /* Destination code             */
>    word                acode ;         /* Answer destination code      */
>    byte                rcode ;         /* Request code                 */
>    byte                ccode ;         /* Completion code              */
>} Head_t ;
>
> Each individual message starts off with the header:
>
>typedef struct  turnoff_mes
>{
>    Head_t              head ;
>    word                node ;
>    word                c_type ;
>} Turnoff_mes ;
>
> A process that receives a messag start by looking at the rcode value
>in head. Each (rcode,dcode)-tuple identifies a unique message
>structure, so the receiving process can cast the received message to
>the appropriate struct, and process it.
>
> Sending a message to a specific process is as simple as including the
>header with the message definitions, just like calling a function in
>another translation unit, requires including the appropriate prototype
>header.
>
> I find this pretty simple, but I've been working with this for more
>than a decade, so for all I know, I could suffer from Helsinki syndrome. 

Interesting code.  Thanks for the example.

[toc] | [prev] | [next] | [standalone]

#18561

On Sun, 11 Mar 2012 21:01:23 GMT, cri@tiac.net (Richard Harter) wrote:

>On Sat, 10 Mar 2012 19:26:15 GMT, cri@tiac.net (Richard Harter) wrote:
>
>I'm going to try this again with some background material.  Maybe that
>will help.
[snip]

There is an obvious bug.  All are invited to point it out.

[toc] | [prev] | [next] | [standalone]

#18564

On Sun, 11 Mar 2012 23:06:16 GMT, cri@tiac.net (Richard Harter) wrote:

>On Sun, 11 Mar 2012 21:01:23 GMT, cri@tiac.net (Richard Harter) wrote:
>
>>On Sat, 10 Mar 2012 19:26:15 GMT, cri@tiac.net (Richard Harter) wrote:
>>
>>I'm going to try this again with some background material.  Maybe that
>>will help.
>[snip]
>
>There is an obvious bug.  All are invited to point it out.
>
How about process IDs of source and destination of the message?
A unique message ID? A in-reply-to-message ID?

[toc] | [prev] | [next] | [standalone]

#18709

On Sun, 11 Mar 2012 17:34:41 -0700, Geoff <geoff@invalid.invalid>
wrote:

>On Sun, 11 Mar 2012 23:06:16 GMT, cri@tiac.net (Richard Harter) wrote:
>
>>On Sun, 11 Mar 2012 21:01:23 GMT, cri@tiac.net (Richard Harter) wrote:
>>
>>>On Sat, 10 Mar 2012 19:26:15 GMT, cri@tiac.net (Richard Harter) wrote:
>>>
>>>I'm going to try this again with some background material.  Maybe that
>>>will help.
>>[snip]
>>
>>There is an obvious bug.  All are invited to point it out.
>>
>How about process IDs of source and destination of the message?
>A unique message ID? A in-reply-to-message ID?

None of these have to be in the message.  The processing manager may
keep this kind of information, but not the individual mini-processes.

Incidentally the bug was in the calculation of the offset of the
payload from char[1] array.

[toc] | [prev] | [next] | [standalone]

#18738

On Tue, 13 Mar 2012 22:00:15 GMT, cri@tiac.net (Richard Harter) wrote:

>>How about process IDs of source and destination of the message?
>>A unique message ID? A in-reply-to-message ID?
>
>None of these have to be in the message.  The processing manager may
>keep this kind of information, but not the individual mini-processes.
>

If the process manager keeps this information, how will it know a
process is awaiting a reply? How will it know two processes are
maintaining an open channel? What you are proposing seems to be an
address-less, stateless version of TCP/IP.

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.c

csiph-web