Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #29205 > unrolled thread

Serialization library, request for feedback

Back to article view | Back to comp.lang.c

Contents

  Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 13:46 -0800
    Re: Serialization library, request for feedback Keith Thompson <kst-u@mib.org> - 2012-12-13 13:57 -0800
      Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 14:30 -0800
        Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-14 19:38 +1300
          Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 03:47 -0600
            Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-14 23:23 +1300
              Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 11:29 -0600
            Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 03:51 -0800
              Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 12:02 -0600
          Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 12:18 +0000
            Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 14:27 +0000
              Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 15:22 +0000
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 13:00 -0600
                  Re: Serialization library, request for feedback Greg Martin <greg@softsprocket.com> - 2012-12-14 11:58 -0800
                    Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 15:08 -0600
                    Re: Serialization library, request for feedback William Ahern <william@wilbur.25thandClement.com> - 2012-12-14 20:07 -0800
                    Re: Serialization library, request for feedback Nick Keighley <nick_keighley_nospam@hotmail.com> - 2012-12-16 03:49 -0800
                      Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-16 13:17 -0600
                        Re: Serialization library, request for feedback Greg Martin <greg@softsprocket.com> - 2012-12-17 09:07 -0800
                          Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-17 13:25 -0600
                Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 09:23 +1300
                  Re: Serialization library, request for feedback James Kuyper <jameskuyper@verizon.net> - 2012-12-14 15:31 -0500
                    Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 15:26 -0600
                      Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 12:25 +1300
                        Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 18:41 -0600
                          Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 14:56 +1300
                            Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 23:36 -0600
                              Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 20:13 +1300
                      Re: Serialization library, request for feedback Shao Miller <sha0.miller@gmail.com> - 2012-12-14 21:42 -0500
                        Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 23:47 -0600
                Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 21:12 +0000
                  Re: Serialization library, request for feedback Johann Klammer <klammerj@NOSPAM.a1.net> - 2012-12-15 05:06 +0100
      Re: Serialization library, request for feedback Willem <willem@turtle.stack.nl> - 2012-12-14 22:20 +0000
    Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-13 23:11 +0000
      Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 15:57 -0800
        Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 02:53 +0000
          Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 04:25 -0800
        Re: Serialization library, request for feedback Bart van Ingen Schenau <bart@ingen.ddns.info.invalid> - 2012-12-14 08:04 +0000
    Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 14:25 +0000
      Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 13:58 -0600
        Re: Serialization library, request for feedback "BartC" <bc@freeuk.com> - 2012-12-14 20:44 +0000
          Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 16:58 -0600
          Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 17:39 -0800
      Re: Serialization library, request for feedback Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-12-14 14:19 -0800

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#29243

Ben Bacarisse wrote:
> Rui Maciel <rui.maciel@gmail.com> writes:
>
>> Ben Bacarisse wrote:
>>
>>> I don't think JSON is a good fit for the OP's task.  There are lots of C
>>> data structures that have no obvious JSON representation at all.
>>
>> What data structures do you have in mind?
>
> My ring_buffer example is one.  In fact, any structure where pointers are
> "shared" would seem to be a bad fit with JSON.

How often would you want to serialise a structure that contains 
pointers?  Pointer values are only useful within the current executing 
program, so you have to use some form of conversion.

> Maybe "no obvious JSON representation" is too strong because you can
> probably always map pointers so some sort of index or string label, but
> it's not a natural fit.

Arrays and indexes would be one choice.  But this is more of a general 
issue of how to serialise a structure with pointers than a JSON one.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#29244

On 12/14/2012 03:23 PM, Ian Collins wrote:
> Ben Bacarisse wrote:
>> Rui Maciel <rui.maciel@gmail.com> writes:
>>
>>> Ben Bacarisse wrote:
>>>
>>>> I don't think JSON is a good fit for the OP's task.  There are lots of C
>>>> data structures that have no obvious JSON representation at all.
>>>
>>> What data structures do you have in mind?
>>
>> My ring_buffer example is one.  In fact, any structure where pointers are
>> "shared" would seem to be a bad fit with JSON.
> 
> How often would you want to serialise a structure that contains 
> pointers?  Pointer values are only useful within the current executing 
> program, so you have to use some form of conversion.

Yes, and ideally the serialization system should handle that conversion
for you, if you give it enough information to do so. I don't claim to
know of any serialization system that meets that ideal.

[toc] | [prev] | [next] | [standalone]

#29250

On 12/14/2012 2:31 PM, James Kuyper wrote:
> On 12/14/2012 03:23 PM, Ian Collins wrote:
>> Ben Bacarisse wrote:
>>> Rui Maciel <rui.maciel@gmail.com> writes:
>>>
>>>> Ben Bacarisse wrote:
>>>>
>>>>> I don't think JSON is a good fit for the OP's task.  There are lots of C
>>>>> data structures that have no obvious JSON representation at all.
>>>>
>>>> What data structures do you have in mind?
>>>
>>> My ring_buffer example is one.  In fact, any structure where pointers are
>>> "shared" would seem to be a bad fit with JSON.
>>
>> How often would you want to serialise a structure that contains
>> pointers?  Pointer values are only useful within the current executing
>> program, so you have to use some form of conversion.
>
> Yes, and ideally the serialization system should handle that conversion
> for you, if you give it enough information to do so. I don't claim to
> know of any serialization system that meets that ideal.
>


if I understand what is meant here, this case would likely a *very* hard 
problem for a general-purpose serialization API...


simpler and more well-behaved cases can be more easily addressed, say:
requiring a specific memory-manager;
requiring use of explicit type-tagging;
requiring structures to be allocated individually;
banning physically nested structures or the use of unions;
...

[toc] | [prev] | [next] | [standalone]

#29254

BGB wrote:
> On 12/14/2012 2:31 PM, James Kuyper wrote:
>> On 12/14/2012 03:23 PM, Ian Collins wrote:
>>> Ben Bacarisse wrote:
>>>> Rui Maciel <rui.maciel@gmail.com> writes:
>>>>
>>>>> Ben Bacarisse wrote:
>>>>>
>>>>>> I don't think JSON is a good fit for the OP's task.  There are lots of C
>>>>>> data structures that have no obvious JSON representation at all.
>>>>>
>>>>> What data structures do you have in mind?
>>>>
>>>> My ring_buffer example is one.  In fact, any structure where pointers are
>>>> "shared" would seem to be a bad fit with JSON.
>>>
>>> How often would you want to serialise a structure that contains
>>> pointers?  Pointer values are only useful within the current executing
>>> program, so you have to use some form of conversion.
>>
>> Yes, and ideally the serialization system should handle that conversion
>> for you, if you give it enough information to do so. I don't claim to
>> know of any serialization system that meets that ideal.
>
> if I understand what is meant here, this case would likely a *very* hard
> problem for a general-purpose serialization API...

Don't loose sight of the difference between the serialisation technique 
and the representation of the serialised data.  The same problems (such 
as what to do with pointers) exist with any general purpose 
serialisation technique or sufficiently expressive data representation.

How the data types are described is another, independent, problem.  Even 
in C code, a structure object requires meta-data (the struct 
declaration) to describe its type.

In my experience serialising data containing pointers is a very unusual 
occurrence.  The only time I have done this is where a set of 
cooperating processes are using data in shared memory.  In this case, 
pointers can either be serialised as an offset from the segment base or 
as plain integer values if all processes use the same base address for 
the segment.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#29255

On 12/14/2012 5:25 PM, Ian Collins wrote:
> BGB wrote:
>> On 12/14/2012 2:31 PM, James Kuyper wrote:
>>> On 12/14/2012 03:23 PM, Ian Collins wrote:
>>>> Ben Bacarisse wrote:
>>>>> Rui Maciel <rui.maciel@gmail.com> writes:
>>>>>
>>>>>> Ben Bacarisse wrote:
>>>>>>
>>>>>>> I don't think JSON is a good fit for the OP's task.  There are
>>>>>>> lots of C
>>>>>>> data structures that have no obvious JSON representation at all.
>>>>>>
>>>>>> What data structures do you have in mind?
>>>>>
>>>>> My ring_buffer example is one.  In fact, any structure where
>>>>> pointers are
>>>>> "shared" would seem to be a bad fit with JSON.
>>>>
>>>> How often would you want to serialise a structure that contains
>>>> pointers?  Pointer values are only useful within the current executing
>>>> program, so you have to use some form of conversion.
>>>
>>> Yes, and ideally the serialization system should handle that conversion
>>> for you, if you give it enough information to do so. I don't claim to
>>> know of any serialization system that meets that ideal.
>>
>> if I understand what is meant here, this case would likely a *very* hard
>> problem for a general-purpose serialization API...
>
> Don't loose sight of the difference between the serialisation technique
> and the representation of the serialised data.  The same problems (such
> as what to do with pointers) exist with any general purpose
> serialisation technique or sufficiently expressive data representation.
>

usually, the matter of pointers is transformed into some sort of object 
index, but the problem becomes, what if the pointer doesn't point at a 
well-defined object?...

say, a person is pointing to raw memory gained via "mmap()" or 
"malloc()" or similar.


a lot of my stuff doesn't work correctly in these cases, and if given a 
pointer to such memory, it will have NULL as the dynamic type.


my scripting language can deal with it, so far as it is also capable of 
seeing the types "C-style" (and so will work with the data according to 
its struct declaration and/or pointer types). (when doing so, it will no 
longer have any notion of array-bounds though, since there may not be 
any information available to tell the VM where the beginning or end of 
the array are located, ...).

but, the serialization code will have no idea, and will generally encode 
it as an "UNDEFINED" value.



> How the data types are described is another, independent, problem.  Even
> in C code, a structure object requires meta-data (the struct
> declaration) to describe its type.
>

even as such, C has many holes in its natural type representations:
information which may be needed to serialize an object with a given type 
may not be present within the types as understood by the language itself.

hence, why some information may be needed in terms of special 
annotations, and run-time type-tags, because even though the tool can 
understand the types (struct declarations, ...), some information may 
still not be knowable from the declarations themselves.


like, say:
"int *a;"

given the 'a' pointer, how do you know where the beginning or end of its 
associated memory object is at?... run-time information may be needed in 
this case (in the sense that the memory manager may know where the array 
is located, and how big it is).

but, the memory manager may only know this if the memory for 'a' was 
allocated via the appropriate functions. (if it came directly from 
"mmap()" or similar, the memory manager may not have any idea).

granted, yes, the memory manager gets its own memory initially from 
"mmap()" or "VirtualAlloc()" or similar, but it remembers which memory 
chunks exist, and can identify individual memory objects within these 
chunks.


a problem though is that, usually, there is not sufficient information 
to work with a union, either within C, or within run-time tags, since if 
all we know is that it is a union, it doesn't help much knowing which 
possibility regarding its contents is the "correct" one.


> In my experience serialising data containing pointers is a very unusual
> occurrence.  The only time I have done this is where a set of
> cooperating processes are using data in shared memory.  In this case,
> pointers can either be serialised as an offset from the segment base or
> as plain integer values if all processes use the same base address for
> the segment.
>

most often, IME, the pointers are actually object references (either 
pointing to another structure, or an array of values, ...).

in other cases, there may be other problems.


granted, in my case, nearly all of my program's memory is allocated via 
the special memory manager (I very rarely use "malloc()" or similar), 
so, in this case, there is usually sufficient information to work with.

[toc] | [prev] | [next] | [standalone]

#29257

BGB wrote:
> On 12/14/2012 5:25 PM, Ian Collins wrote:
>> BGB wrote:
>>> On 12/14/2012 2:31 PM, James Kuyper wrote:
>>>> On 12/14/2012 03:23 PM, Ian Collins wrote:.
>>>>>
>>>>> How often would you want to serialise a structure that contains
>>>>> pointers?  Pointer values are only useful within the current executing
>>>>> program, so you have to use some form of conversion.
>>>>
>>>> Yes, and ideally the serialization system should handle that conversion
>>>> for you, if you give it enough information to do so. I don't claim to
>>>> know of any serialization system that meets that ideal.
>>>
>>> if I understand what is meant here, this case would likely a *very* hard
>>> problem for a general-purpose serialization API...
>>
>> Don't loose sight of the difference between the serialisation technique
>> and the representation of the serialised data.  The same problems (such
>> as what to do with pointers) exist with any general purpose
>> serialisation technique or sufficiently expressive data representation.
>
> usually, the matter of pointers is transformed into some sort of object
> index, but the problem becomes, what if the pointer doesn't point at a
> well-defined object?...

Then the object isn't a suitable candidate for serialisation.

I don't agree with your "usually".  Unless you are dealing with the very 
rare case where a pointer value (even expressed as a base + offset) is 
meaningful to the reader of the serialised data as well as the writer, 
the only sensible thing to do is the equivalent of a deep copy and 
serialise the data pointed to and not the pointer value.

>> How the data types are described is another, independent, problem.  Even
>> in C code, a structure object requires meta-data (the struct
>> declaration) to describe its type.
>>
>
> even as such, C has many holes in its natural type representations:
> information which may be needed to serialize an object with a given type
> may not be present within the types as understood by the language itself.

Eh?

> hence, why some information may be needed in terms of special
> annotations, and run-time type-tags, because even though the tool can
> understand the types (struct declarations, ...), some information may
> still not be knowable from the declarations themselves.
>
> like, say:
> "int *a;"
>
> given the 'a' pointer, how do you know where the beginning or end of its
> associated memory object is at?... run-time information may be needed in
> this case (in the sense that the memory manager may know where the array
> is located, and how big it is).

Ah-ha.  See above.

>> In my experience serialising data containing pointers is a very unusual
>> occurrence.  The only time I have done this is where a set of
>> cooperating processes are using data in shared memory.  In this case,
>> pointers can either be serialised as an offset from the segment base or
>> as plain integer values if all processes use the same base address for
>> the segment.
>>
>
> most often, IME, the pointers are actually object references (either
> pointing to another structure, or an array of values, ...).
>
> in other cases, there may be other problems.
>
> granted, in my case, nearly all of my program's memory is allocated via
> the special memory manager (I very rarely use "malloc()" or similar),
> so, in this case, there is usually sufficient information to work with.

Not if you are sending the data to another language....

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#29268

On 12/14/2012 7:56 PM, Ian Collins wrote:
> BGB wrote:
>> On 12/14/2012 5:25 PM, Ian Collins wrote:
>>> BGB wrote:
>>>> On 12/14/2012 2:31 PM, James Kuyper wrote:
>>>>> On 12/14/2012 03:23 PM, Ian Collins wrote:.
>>>>>>
>>>>>> How often would you want to serialise a structure that contains
>>>>>> pointers?  Pointer values are only useful within the current
>>>>>> executing
>>>>>> program, so you have to use some form of conversion.
>>>>>
>>>>> Yes, and ideally the serialization system should handle that
>>>>> conversion
>>>>> for you, if you give it enough information to do so. I don't claim to
>>>>> know of any serialization system that meets that ideal.
>>>>
>>>> if I understand what is meant here, this case would likely a *very*
>>>> hard
>>>> problem for a general-purpose serialization API...
>>>
>>> Don't loose sight of the difference between the serialisation technique
>>> and the representation of the serialised data.  The same problems (such
>>> as what to do with pointers) exist with any general purpose
>>> serialisation technique or sufficiently expressive data representation.
>>
>> usually, the matter of pointers is transformed into some sort of object
>> index, but the problem becomes, what if the pointer doesn't point at a
>> well-defined object?...
>
> Then the object isn't a suitable candidate for serialisation.
>

I thought someone here was objecting here to serialization not working 
with structures mapped into a ring-buffer?... (IOW: a big glob of memory 
treated like a ring, with structures allocated "around" the inside of 
the ring...).

I was mostly noting that this sort of thing is a hard problem for a 
serialization API to deal with (and, in my case, I don't even try to 
make this work).


though, I may have misunderstood, and the reference may have just been 
structures with cyclic linking (say, a ring composed of linked 
structures), which is an easier problem (mostly this one involves 
mapping objects to indices, and the "ring" will fall away, as any prior 
object will already have been mapped).


> I don't agree with your "usually".  Unless you are dealing with the very
> rare case where a pointer value (even expressed as a base + offset) is
> meaningful to the reader of the serialised data as well as the writer,
> the only sensible thing to do is the equivalent of a deep copy and
> serialise the data pointed to and not the pointer value.
>

the "usually" is because there are some ways to implement persistence 
mechanisms which do depend on pointer addresses... (usually by 
implementing it via "mmap()" or "CreateFileMapping()" or similar...).

whether this is good or not is a separate manner.
(I don't do this, as it is just too ugly and broken IME to really be all 
that useful).


in most other cases, any pointer is mapped to an object index or 
similar, and the original address is not preserved. (hence, "usually").


>>> How the data types are described is another, independent, problem.  Even
>>> in C code, a structure object requires meta-data (the struct
>>> declaration) to describe its type.
>>>
>>
>> even as such, C has many holes in its natural type representations:
>> information which may be needed to serialize an object with a given type
>> may not be present within the types as understood by the language itself.
>
> Eh?
>

like, array bounds for raw-memory arrays.

if a person is like:
int *a;
a=malloc(256*sizeof(int));

it isn't (necessarily) the case that a person can get back the base and 
size of the pointed-to memory (absent compiler-specific extensions, such 
as "_msize()" and similar).


other possibilities include a pointer to a struct physically embedded 
within another struct.

typedef struct
{
int x, y;
}Foo;

typedef struct
{
int w;
Foo foo;
int z;
}Bar;

Bar *bar;
Foo *foo;
bar=malloc(sizeof(Bar));
foo=&(bar->foo);

if we try to serialize with nothing more than 'foo', there is not enough 
information available (within the C type-system), for a serialization 
API to realize that 'foo' is contained within an instance of Bar, and 
that the object pointed to by 'bar' should be serialized instead.

another similar example, would be allocating a buffer for the contents 
of a file, then casting various parts of this buffer to various structs 
pointers.

a serializer working simply by walking the pointers would likely be 
entirely unaware that the structs are held within a common buffer (and 
would not preserve their physical spatial relationship within this buffer).


granted, a lot of these are cases which my APIs don't really address, 
nor would I really want to try to address them...

this is part of the reason for some of the "restrictions" mentioned 
before, which mostly serve to place more limits of what can be called 
"well defined" data structures.


>> hence, why some information may be needed in terms of special
>> annotations, and run-time type-tags, because even though the tool can
>> understand the types (struct declarations, ...), some information may
>> still not be knowable from the declarations themselves.
>>
>> like, say:
>> "int *a;"
>>
>> given the 'a' pointer, how do you know where the beginning or end of its
>> associated memory object is at?... run-time information may be needed in
>> this case (in the sense that the memory manager may know where the array
>> is located, and how big it is).
>
> Ah-ha.  See above.
>

ok, fair enough...


>>> In my experience serialising data containing pointers is a very unusual
>>> occurrence.  The only time I have done this is where a set of
>>> cooperating processes are using data in shared memory.  In this case,
>>> pointers can either be serialised as an offset from the segment base or
>>> as plain integer values if all processes use the same base address for
>>> the segment.
>>>
>>
>> most often, IME, the pointers are actually object references (either
>> pointing to another structure, or an array of values, ...).
>>
>> in other cases, there may be other problems.
>>
>> granted, in my case, nearly all of my program's memory is allocated via
>> the special memory manager (I very rarely use "malloc()" or similar),
>> so, in this case, there is usually sufficient information to work with.
>
> Not if you are sending the data to another language....
>

a lot of this is used when sharing data with my own language 
(BGBScript), which is built on the same custom memory manager as most of 
my C code.


functionally though, BGBScript can (optionally) deal with most of the 
same types and memory-management practices as in C.

apparently, I just sort of took it for granted that these sorts of 
things "should" work, for such a scripting language to be "practically 
useful"...

granted, yes, this isn't really the normal way of writing code in the 
language...

[toc] | [prev] | [next] | [standalone]

#29270

BGB wrote:
> On 12/14/2012 7:56 PM, Ian Collins wrote:
>> BGB wrote:

>>> usually, the matter of pointers is transformed into some sort of object
>>> index, but the problem becomes, what if the pointer doesn't point at a
>>> well-defined object?...
>>
>> Then the object isn't a suitable candidate for serialisation.
>>
>
> I thought someone here was objecting here to serialization not working
> with structures mapped into a ring-buffer?... (IOW: a big glob of memory
> treated like a ring, with structures allocated "around" the inside of
> the ring...).

My point was an object that contains a pointer to a blob of unspecified 
size isn't suitable for serialisation.  If the object is designed to be 
serialised (of if the requirement is retrofitted), it would have an 
embedded size, or provide a means to determine the size of the blob.  In 
practice the conditions for serialising are pretty much the same as 
those for copying an object.

>> I don't agree with your "usually".  Unless you are dealing with the very
>> rare case where a pointer value (even expressed as a base + offset) is
>> meaningful to the reader of the serialised data as well as the writer,
>> the only sensible thing to do is the equivalent of a deep copy and
>> serialise the data pointed to and not the pointer value.
>>
>
> the "usually" is because there are some ways to implement persistence
> mechanisms which do depend on pointer addresses... (usually by
> implementing it via "mmap()" or "CreateFileMapping()" or similar...).

That's basically what I described up thread as the case of a set of 
cooperating processes are using data in shared memory.

> in most other cases, any pointer is mapped to an object index or
> similar, and the original address is not preserved. (hence, "usually").

Again, I disagree.  it makes more sense (in a general scheme) to 
serialise the data pointed to, not the pointer.  Unless you also intend 
including the entire memory space in the serialised data.  The only time 
I see that is in a core file.

>>>> How the data types are described is another, independent, problem.  Even
>>>> in C code, a structure object requires meta-data (the struct
>>>> declaration) to describe its type.
>>>>
>>>
>>> even as such, C has many holes in its natural type representations:
>>> information which may be needed to serialize an object with a given type
>>> may not be present within the types as understood by the language itself.
>>
>> Eh?
>>
>
> like, array bounds for raw-memory arrays.
>
> if a person is like:
> int *a;
> a=malloc(256*sizeof(int));

If you wanted to serialise a, you would hang on to the size.

> other possibilities include a pointer to a struct physically embedded
> within another struct.
>
> typedef struct
> {
> int x, y;
> }Foo;
>
> typedef struct
> {
> int w;
> Foo foo;
> int z;
> }Bar;
>
> Bar *bar;
> Foo *foo;
> bar=malloc(sizeof(Bar));
> foo=&(bar->foo);
>
> if we try to serialize with nothing more than 'foo', there is not enough
> information available (within the C type-system), for a serialization
> API to realize that 'foo' is contained within an instance of Bar, and
> that the object pointed to by 'bar' should be serialized instead.

That sounds like a programming error rather than a deficiency in the C 
type system.

> another similar example, would be allocating a buffer for the contents
> of a file, then casting various parts of this buffer to various structs
> pointers.
>
> a serializer working simply by walking the pointers would likely be
> entirely unaware that the structs are held within a common buffer (and
> would not preserve their physical spatial relationship within this buffer).

If you want a portable, persistent representation of the data, that is 
probably a good thing!

> granted, a lot of these are cases which my APIs don't really address,
> nor would I really want to try to address them...
>
> this is part of the reason for some of the "restrictions" mentioned
> before, which mostly serve to place more limits of what can be called
> "well defined" data structures.

In most of what I do the serialisation API is "convert to JSON, stream". 
  So if something can't be represented as JSON, it can't be serialised. 
  This rule came about in order to share data between JavaScript clients 
and C++, C or PHP servers.  I have subsequently found internal JSON 
objects incredibly useful for building and passing dynamic types, 
especially as I can use almost identical code in JavaScript, PHP and C++ 
(but alas, not C) to manipulate them.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#29258

On 12/14/2012 16:26, BGB wrote:
>
> if I understand what is meant here, this case would likely a *very* hard
> problem for a general-purpose serialization API...
>

Well seeing as how "the slow way" is the only portable way to test two 
pointers for pointing into the same object (one step at a time with != 
or ==), "the slow way" could be adopted for assigning indices to 
pointers, or referring to previously-established indices for the same 
pointee.  No?  At least deserializing wouldn't be so slow.  - Shao

[toc] | [prev] | [next] | [standalone]

#29269

On 12/14/2012 8:42 PM, Shao Miller wrote:
> On 12/14/2012 16:26, BGB wrote:
>>
>> if I understand what is meant here, this case would likely a *very* hard
>> problem for a general-purpose serialization API...
>>
>
> Well seeing as how "the slow way" is the only portable way to test two
> pointers for pointing into the same object (one step at a time with !=
> or ==), "the slow way" could be adopted for assigning indices to
> pointers, or referring to previously-established indices for the same
> pointee.  No?  At least deserializing wouldn't be so slow.  - Shao

could be...


in my case, I am using a customized memory manager, and it is possible 
to use special calls, say:
int Foo_CheckPtrAisB(void *a, void *b)
{ return((gcgetbase(a)!=NULL) && (gcgetbase(a)==gcgetbase(b))); }

where any two pointers into the same object, will produce the same base 
address for "gcgetbase()", which returns the starting address for an 
allocated object, or NULL if the pointer isn't into the GC's heap.


but, yeah, otherwise, it is a harder problem.


or such...

[toc] | [prev] | [next] | [standalone]

#29249

Ben Bacarisse wrote:

> My ring_buffer example is one.  In fact, any structure where pointers are
> "shared" would seem to be a bad fit with JSON.
> 
> Maybe "no obvious JSON representation" is too strong because you can
> probably always map pointers so some sort of index or string label, but
> it's not a natural fit.


Granted, JSON doesn't offer explicit support for pointers.  Nonetheless, 
pointers are essentially a map between a reference number and an object. 
Therefore,if it's possible to represent a map between a number and an 
object,  it's possible to represent a pointer, and JSON supports those 
types.


Rui Maciel

[toc] | [prev] | [next] | [standalone]

#29262

Rui Maciel wrote:
> Ben Bacarisse wrote:
>
>> My ring_buffer example is one.  In fact, any structure where pointers are
>> "shared" would seem to be a bad fit with JSON.
>>
>> Maybe "no obvious JSON representation" is too strong because you can
>> probably always map pointers so some sort of index or string label, but
>> it's not a natural fit.
>
>
> Granted, JSON doesn't offer explicit support for pointers.  Nonetheless,
> pointers are essentially a map between a reference number and an object.
> Therefore,if it's possible to represent a map between a number and an
> object,  it's possible to represent a pointer, and JSON supports those
> types.
>
>
> Rui Maciel
>
A one-on one mapping will not do. There may be multiple pointers 
pointing into the same area, and they need not point at the start of it.
You'd need at least some kind of interval table/tree, to check the 
pointers that fall into a certain range and try to associate them with 
some 'base objects', into which they point... Otherwise the read back 
structure will end up differently than what was written.

[toc] | [prev] | [next] | [standalone]

#29252

Keith Thompson wrote:
) Ulf ??str??m <ulf.astrom@gmail.com> writes:
) [...]
)> The user needs to set up translators (composed of one or more fields)
)> for each type they wish to serialize. Internally it works by
)> flattening each type into a list of void pointers and replacing them
)> with numeric indices. The output format has a simple "<tag> <value>;"
)> format; it is similar to JSON but it is not a reimplementation of it.
) [...]
)
) Why not just use JSON?  It would make the flattened files accessible by
) other tools.

I think YAML is a much better choice in this case, because it has
some extra features such as object descriptions and object references.


SaSW, Willem
-- 
Disclaimer: I am in no way responsible for any of the statements
            made in the above text. For all I know I might be
            drugged or something..
            No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT

[toc] | [prev] | [next] | [standalone]

#29211

Ulf Åström <ulf.astrom@gmail.com> writes:

> I'm writing a serialization library for C. It can convert binary data
> (a struct or array, for example) into human-readable text and of
> course also read serialized data back and reconstruct it in memory,
> including nested pointers. The purpose is to provide a quick way to
> save/load in applications and reduce the effort needed to keep file
> formats in sync.
>
> The user needs to set up translators (composed of one or more fields)
> for each type they wish to serialize. Internally it works by
> flattening each type into a list of void pointers and replacing them
> with numeric indices. The output format has a simple "<tag> <value>;"
> format; it is similar to JSON but it is not a reimplementation of it.
>
> I'm looking for feedback how I can make this most useful to other
> people. The code and documentation can be found at:
> http://www.happyponyland.net/serialize.php
>
> I'm wondering if there is anything crucial I have missed. It can deal
> with most primitive C types, strings, arrays and almost any
> arrangement of linked structures.

The most striking omission are support for bool, wide strings and
complex types.  This is not a criticism, just an observation.

The other area that I could not understand is how to serialise multiple
pointers into a single object.  For example, how does one serialise

  struct ring_buffer {
     T buffer[BSIZE];
     T *head, *tail;
  };

where 'head' and 'tail' both point to an element of 'buffer'?  This may
be just because there's no documentation yet -- I had a look at the API
and did not immediately see how this could be done.

As for the design itself, I got stuck on an issue in the first example.
You seem to require a dummy object in order to pass pointers to it's
members to the set-up functions, and a comment purports to explain why
this is needed but I just don't see it.  The comment includes a
statement that is wrong (if I understand it) that a pointer to the first
member of a field may not equal (when suitably converted) a pointer to
the structure object itself.  Even if this were true, I don't see why
that means you can't just pass the offset of the field.  At the least,
more explanation is required.

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#29214

On Dec 14, 12:11 am, Ben Bacarisse <ben.use...@bsb.me.uk> wrote:
> Ulf Åström <ulf.ast...@gmail.com> writes:
> > I'm wondering if there is anything crucial I have missed. It can deal
> > with most primitive C types, strings, arrays and almost any
> > arrangement of linked structures.
>
> The most striking omission are support for bool, wide strings and
> complex types.  This is not a criticism, just an observation.

Ok! I will have a look at this. I don't use them much myself so I
haven't thought of them, but they should be supported.

> The other area that I could not understand is how to serialise multiple
> pointers into a single object.  For example, how does one serialise
>
>   struct ring_buffer {
>      T buffer[BSIZE];
>      T *head, *tail;
>   };
>
> where 'head' and 'tail' both point to an element of 'buffer'?  This may
> be just because there's no documentation yet -- I had a look at the API
> and did not immediately see how this could be done.

Here is a quick example (assuming int for T):

  struct ring_buffer rbuf;
  tra = ser_new_tra("ring_buffer", sizeof(struct ring_buffer), NULL);
  field = ser_new_field(tra, "int", 0, "buffer", &rbuf, &rbuf.buffer);
  field->repeat = BSIZE;
  ser_new_field(tra, "int", 1, "head", &rbuf, &rbuf.head);
  ser_new_field(tra, "int", 1, "tail", &rbuf, &rbuf.tail);

The 1s to ser_new_field indicate that it is a pointer.

This exposed a bug, however; the int pointers will only be repointed
correctly (during inflation) if they point to rbuf.buffer[0]; other
addresses would be duplicated in memory. This is because it will only
check for pointers that exactly match a field offset. It should also
check all elements of an array.

> As for the design itself, I got stuck on an issue in the first example.
> You seem to require a dummy object in order to pass pointers to it's
> members to the set-up functions, and a comment purports to explain why
> this is needed but I just don't see it.  The comment includes a
> statement that is wrong (if I understand it) that a pointer to the first
> member of a field may not equal (when suitably converted) a pointer to
> the structure object itself.  Even if this were true, I don't see why
> that means you can't just pass the offset of the field.  At the least,
> more explanation is required.

It's a typo; it should say "pos must be within st".

Using a dummy isn't strictly necessary, you may as well pass zero and
an offset. I designed it this way so people won't hardcode offset
values, then change their structures (or start porting their program
to a different architecture!) but forget to update the translator.
There doesn't seem to be any other reliable way to get the offset of a
member (offsetof() with null cast is undefined behaviour, according to
Wikipedia). Anyway, the dummy is only needed when setting up the
translator. Perhaps I could change it to only take an offset but
suggest using a dummy to calculate it, e.g. (&rbuf.head - &rbuf).

Good suggestions, thanks a lot.

/Ulf

[toc] | [prev] | [next] | [standalone]

#29217

Ulf Åström <ulf.astrom@gmail.com> writes:

> On Dec 14, 12:11 am, Ben Bacarisse <ben.use...@bsb.me.uk> wrote:
<snip>
>> The other area that I could not understand is how to serialise multiple
>> pointers into a single object.  For example, how does one serialise
>>
>>   struct ring_buffer {
>>      T buffer[BSIZE];
>>      T *head, *tail;
>>   };
>>
>> where 'head' and 'tail' both point to an element of 'buffer'?  This may
>> be just because there's no documentation yet -- I had a look at the API
>> and did not immediately see how this could be done.
>
> Here is a quick example (assuming int for T):
>
>   struct ring_buffer rbuf;
>   tra = ser_new_tra("ring_buffer", sizeof(struct ring_buffer), NULL);
>   field = ser_new_field(tra, "int", 0, "buffer", &rbuf, &rbuf.buffer);
>   field->repeat = BSIZE;
>   ser_new_field(tra, "int", 1, "head", &rbuf, &rbuf.head);
>   ser_new_field(tra, "int", 1, "tail", &rbuf, &rbuf.tail);
>
> The 1s to ser_new_field indicate that it is a pointer.

What about pointers to pointers?

> This exposed a bug, however; the int pointers will only be repointed
> correctly (during inflation) if they point to rbuf.buffer[0]; other
> addresses would be duplicated in memory. This is because it will only
> check for pointers that exactly match a field offset. It should also
> check all elements of an array.

Thanks.  I think I see what's happening.  As I understand it, this
suggests there must be a "root" structure for all of the data, but maybe
what you are doing it more subtle than that.

As an example, imagine a binary tree of nodes and a separate linked list
of node pointers.  Can I serialise this?  Must I serialise the tree
first?

>> As for the design itself, I got stuck on an issue in the first example.
>> You seem to require a dummy object in order to pass pointers to it's
>> members to the set-up functions, and a comment purports to explain why
>> this is needed but I just don't see it.  The comment includes a
>> statement that is wrong (if I understand it) that a pointer to the first
>> member of a field may not equal (when suitably converted) a pointer to
>> the structure object itself.  Even if this were true, I don't see why
>> that means you can't just pass the offset of the field.  At the least,
>> more explanation is required.
>
> It's a typo; it should say "pos must be within st".

I think that a reference to the API documentation ("Note: pos must be
within pos").  I was talking about the comment in example1.c:

  "In this example &thing_dummy and &thing_dummy.a will most probably be
  the same location. &thing_dummy.b on the other hand will typically end
  up 2 to 8 bytes from the base pointer (depending on the computer
  architecture). For this reason we do not pass the offset as a number
  but rely on the compiler to tell us exactly where the members will
  be."

In your example, &thing_dummy.a and &thing_dummy must *always* be the
same location so the phrase "will most probably be" just looks odd.

That aside, the following bit is what threw me: "For this reason we do
not pass the offset as a number".  Nothing previous to this seems to
explain why you pass two pointers rather than an offset.

> Using a dummy isn't strictly necessary, you may as well pass zero and
> an offset. I designed it this way so people won't hardcode offset
> values, then change their structures (or start porting their program
> to a different architecture!) but forget to update the translator.

That is a reason but not, to my mind, a strong one.  To complicate the
API because it's possible to so grossly abuse a simpler one seems to be
contrary to the spirit if C.  What's more, if you are serious about
this, then you should not permit sizeof(myType) in the earlier call,
since a programmer might hard code the size.  Instead, they should be
obliged to pass two pointers: &thing_dummy and &thing_dummy + 1.  If you
accept a simple size in one case, permit a simple offset in the other.

> There doesn't seem to be any other reliable way to get the offset of a
> member (offsetof() with null cast is undefined behaviour, according to
> Wikipedia). Anyway, the dummy is only needed when setting up the
> translator. Perhaps I could change it to only take an offset but
> suggest using a dummy to calculate it, e.g. (&rbuf.head - &rbuf).

What's wrong with offsetof?  I don't follow what you mean about a "null
cast" being undefined.

BTW, writing &rbuf.head - &rbuf won't work -- the pointers are of the
wrong type.  You'd need to cast to char * but that's so messy.  I think
offsetof is the way to go.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#29230

On Dec 14, 3:53 am, Ben Bacarisse <ben.use...@bsb.me.uk> wrote:
> Ulf Åström <ulf.ast...@gmail.com> writes:
> > On Dec 14, 12:11 am, Ben Bacarisse <ben.use...@bsb.me.uk> wrote:
> <snip>
> >> The other area that I could not understand is how to serialise multiple
> >> pointers into a single object.  For example, how does one serialise
>
> >>   struct ring_buffer {
> >>      T buffer[BSIZE];
> >>      T *head, *tail;
> >>   };
>
> >> where 'head' and 'tail' both point to an element of 'buffer'?  This may
> >> be just because there's no documentation yet -- I had a look at the API
> >> and did not immediately see how this could be done.
>
> > Here is a quick example (assuming int for T):
>
> >   struct ring_buffer rbuf;
> >   tra = ser_new_tra("ring_buffer", sizeof(struct ring_buffer), NULL);
> >   field = ser_new_field(tra, "int", 0, "buffer", &rbuf, &rbuf.buffer);
> >   field->repeat = BSIZE;
> >   ser_new_field(tra, "int", 1, "head", &rbuf, &rbuf.head);
> >   ser_new_field(tra, "int", 1, "tail", &rbuf, &rbuf.tail);
>
> > The 1s to ser_new_field indicate that it is a pointer.
>
> What about pointers to pointers?

It can be done; have a look at test_single_ptr() in demo.c (I still
need to write up a clean example for this). It requires an extra
translator for "pointer to type", so other translators can reference
the pointer type itself. In this case it also needs the _int wrapper
(with a single int at offset 0) since I haven't found a satisfactory
way yet to translate primitives on their own.

> > This exposed a bug, however; the int pointers will only be repointed
> > correctly (during inflation) if they point to rbuf.buffer[0]; other
> > addresses would be duplicated in memory. This is because it will only
> > check for pointers that exactly match a field offset. It should also
> > check all elements of an array.
>
> Thanks.  I think I see what's happening.  As I understand it, this
> suggests there must be a "root" structure for all of the data, but maybe
> what you are doing it more subtle than that.
>
> As an example, imagine a binary tree of nodes and a separate linked list
> of node pointers.  Can I serialise this?  Must I serialise the tree
> first?

This is just the kind of problem I'm hoping to solve.

It needs translators for the tree nodes and the list nodes. It is true
it only works on hierarchal data. The only way with the current API
would be to make a wrapper struct (and a translator for it) that
points to both the first tree and list nodes; this should be passed to
ser_ialize(). I can imagine using a variadric function to pass an
arbitrary number of type/pointer pairs and omit the wrapper, but
another problem would arise in the other end since ser_parse() will
only return a single pointer.

> >> As for the design itself, I got stuck on an issue in the first example.
> >> You seem to require a dummy object in order to pass pointers to it's
> >> members to the set-up functions, and a comment purports to explain why
> >> this is needed but I just don't see it.  The comment includes a
> >> statement that is wrong (if I understand it) that a pointer to the first
> >> member of a field may not equal (when suitably converted) a pointer to
> >> the structure object itself.  Even if this were true, I don't see why
> >> that means you can't just pass the offset of the field.  At the least,
> >> more explanation is required.
>
> > It's a typo; it should say "pos must be within st".
>
> I think that a reference to the API documentation ("Note: pos must be
> within pos").  I was talking about the comment in example1.c:
>
>   "In this example &thing_dummy and &thing_dummy.a will most probably be
>   the same location. &thing_dummy.b on the other hand will typically end
>   up 2 to 8 bytes from the base pointer (depending on the computer
>   architecture). For this reason we do not pass the offset as a number
>   but rely on the compiler to tell us exactly where the members will
>   be."
>
> In your example, &thing_dummy.a and &thing_dummy must *always* be the
> same location so the phrase "will most probably be" just looks odd.

True; I was probably overthinking things and imagining a larger
structure.

> That aside, the following bit is what threw me: "For this reason we do
> not pass the offset as a number".  Nothing previous to this seems to
> explain why you pass two pointers rather than an offset.
>
> > Using a dummy isn't strictly necessary, you may as well pass zero and
> > an offset. I designed it this way so people won't hardcode offset
> > values, then change their structures (or start porting their program
> > to a different architecture!) but forget to update the translator.
>
> That is a reason but not, to my mind, a strong one.  To complicate the
> API because it's possible to so grossly abuse a simpler one seems to be
> contrary to the spirit if C.  What's more, if you are serious about
> this, then you should not permit sizeof(myType) in the earlier call,
> since a programmer might hard code the size.  Instead, they should be
> obliged to pass two pointers: &thing_dummy and &thing_dummy + 1.  If you
> accept a simple size in one case, permit a simple offset in the other.

I agree; I should trust the programmer to know what they are doing. I
will be changing it to take only an offset.

> --
> Ben.

/Ulf

[toc] | [prev] | [next] | [standalone]

#29223

On Thu, 13 Dec 2012 15:57:27 -0800, Ulf Åström wrote:
>
>Using a dummy isn't strictly necessary, you may as well pass zero and
>an offset. I designed it this way so people won't hardcode offset
>values, then change their structures (or start porting their program
>to a different architecture!) but forget to update the translator.
>There doesn't seem to be any other reliable way to get the offset of a
>member (offsetof() with null cast is undefined behaviour, according to
>Wikipedia). Anyway, the dummy is only needed when setting up the
>translator. Perhaps I could change it to only take an offset but
>suggest using a dummy to calculate it, e.g. (&rbuf.head - &rbuf).

You could also setup the interface like this:

    ser_field_t * ser_new_field_impl(ser_tra_t * tra, const char * type, 
                                     const int ref,  const char * tag, 
                                     const int offset);
   #define ser_new_field(tra,type,ref,tag,src_type,field) \
       ser_new_field_impl(tra, type, ref, tag, offsetof(src_type, field))

Also note that offsetof must be defined by the compiler/implementation 
and they can use whatever magic that gets the job done, including the 
trick with dereferencing a null pointer.
If you do the same, you incur Undefined Behaviour, but the compiler 
itself is above the law in this respect.
That makes offsetof() the easiest and reliable way to obtain the offset 
of a member of a structure and it is in the standard just because you 
need compiler magic to get the offset without incurring UB or the 
overhead of a dummy object.

>
>/Ulf

Bart v Ingen Schenau

[toc] | [prev] | [next] | [standalone]

#29234

Ulf Åström wrote:

> I'm wondering if there is anything crucial I have missed. It can deal
> with most primitive C types, strings, arrays and almost any
> arrangement of linked structures. There are still a few pieces
> missing; for example I would like to have a clean interface to let the
> user provide custom read/write functions for complex data types. I
> will also work on the type safety, so structures can't be linked
> incorrectly by erroneous or malicious input.

I believe that's a poor approach to a common problem which also has a very 
common and simple solution:  for the output, a single custom 
operator/function that takes in a data type/struct and calls fprintf() to 
dump the relevant text following a specific data format, and for the input, 
a custom parser.  That's it.  No need to complicate things.

The "let's blindly dump everything" approach is fundamentally broken, 
because it actually doesn't describe the information.  Instead, it dumps the 
data as how it was stored in a particular data structure and omits any 
semantics associated with it.  This causes significant problems, because it 
doesn't perform any integrity checks on the information and essentially 
eliminates the possibility of adding the necessary sanity checks 
to the parsing process, other than testing if a particular primitive data 
type is actually valid.  Therefore, dumping data makes it impossible to test 
if the data which has been imported actually represents valid information.  
And this is a bad thing.

The snags you've stumbled on are a natural consequence of trying to dump 
data instead of exporting information, which happen to be trivial to fix 
with the parser/document dump combo approach.  Complex data types are never 
a problem because the programmer is responsible to write the necessary I/O 
routines, and all the necessary sanity checks are automatically performed, 
not only with regards to type safety but also in the form of semantics 
checks, including the ability to infer if a problem is fatal or recoverable.


Rui Maciel

[toc] | [prev] | [next] | [standalone]

#29242

On 12/14/2012 8:25 AM, Rui Maciel wrote:
> Ulf Åström wrote:
>
>> I'm wondering if there is anything crucial I have missed. It can deal
>> with most primitive C types, strings, arrays and almost any
>> arrangement of linked structures. There are still a few pieces
>> missing; for example I would like to have a clean interface to let the
>> user provide custom read/write functions for complex data types. I
>> will also work on the type safety, so structures can't be linked
>> incorrectly by erroneous or malicious input.
>
> I believe that's a poor approach to a common problem which also has a very
> common and simple solution:  for the output, a single custom
> operator/function that takes in a data type/struct and calls fprintf() to
> dump the relevant text following a specific data format, and for the input,
> a custom parser.  That's it.  No need to complicate things.
>
> The "let's blindly dump everything" approach is fundamentally broken,
> because it actually doesn't describe the information.  Instead, it dumps the
> data as how it was stored in a particular data structure and omits any
> semantics associated with it.  This causes significant problems, because it
> doesn't perform any integrity checks on the information and essentially
> eliminates the possibility of adding the necessary sanity checks
> to the parsing process, other than testing if a particular primitive data
> type is actually valid.  Therefore, dumping data makes it impossible to test
> if the data which has been imported actually represents valid information.
> And this is a bad thing.
>
> The snags you've stumbled on are a natural consequence of trying to dump
> data instead of exporting information, which happen to be trivial to fix
> with the parser/document dump combo approach.  Complex data types are never
> a problem because the programmer is responsible to write the necessary I/O
> routines, and all the necessary sanity checks are automatically performed,
> not only with regards to type safety but also in the form of semantics
> checks, including the ability to infer if a problem is fatal or recoverable.
>

makes sense IMO...


personally, I far more often use specialized file-formats, rather than 
dumping data structures, and usually if serialization is used, it is 
usually limited mostly to certain sets of dedicated "safe to serialize" 
data-types, which in my case are typically "lists" and 
"dynamically-typed objects".

most types provide dedicated handlers for serializing and reading back 
in their serialized forms, many often using a small "version" tag, as a 
way to indicate what fields are present in the data (this itself has 
flaws, but basically works), or sometimes the handlers use individual 
tags to indicate various items within the serialized structure.

types which don't bother with version-tags are typically things like 
integer values or strings, which are basically "too simple to justify a 
version tag".


actually, given my frequent use of sequential read/write functions, is a 
major part of why, very often, I end up using variable-length integer 
representations.

for example, the vast majority of integer-values are fairly small 
(basically, forming a bell-curve with its center around 0), making it so 
that, on-average, small integers may only require 1 byte, and most 
others are 2 or 3 bytes, independent of the physical storage-type of the 
integer (32 or 64 bits).

like, although a variable-length integer isn't very good for 
random-access data, it is pretty good for data that is read/written 
sequentially.

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.c

csiph-web