Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #29239

Re: Serialization library, request for feedback

Show all headers | View raw

On 12/14/2012 5:51 AM, Ulf Åström wrote:
> On Dec 14, 10:47 am, BGB <cr88...@hotmail.com> wrote:
>> On 12/14/2012 12:38 AM, Ian Collins wrote:
>>> Ulf str m wrote:
>>>> On Dec 13, 10:57 pm, Keith Thompson <ks...@mib.org> wrote:
>>>>> Ulf str m <ulf.ast...@gmail.com> writes:
>>
>>>>>> The user needs to set up translators (composed of one or more fields)
>>>>>> for each type they wish to serialize. Internally it works by
>>>>>> flattening each type into a list of void pointers and replacing them
>>>>>> with numeric indices. The output format has a simple "<tag> <value>;"
>>>>>> format; it is similar to JSON but it is not a reimplementation of it.
>>
>>>>> Why not just use JSON?  It would make the flattened files accessible by
>>>>> other tools.
>>
>>>> At a glance they seem to be mostly compatible but I don't know how
>>>> well it would mesh with my memory offset <-> value layout or how I
>>>> would perform pointer substitution. I haven't decided if I want to
>>>> pursue a fully compliant JSON implementation, or take the format in
>>>> another direction entirely.
>>
>>> Why make things complex?  JSON is an ideal candidate for representing
>>> structure and array types.  It is after all designed as an object notation.
>>
>> yep, JSON makes sense.
>>
>> one possible downside though is that it doesn't normally identify object
>> types, which means some means may be needed to identify what sort of
>> struct is being serialized, and/or the type of array, ...
>>
>> it is either that, or deal with data in a dynamically-typed manner,
>> rather than directly mapping it to raw C structs.
>
> I can see a couple of problems with JSON. It is really just an
> associative array and doesn't specify the type of objects, so I would
> either need to guess (from the fields provided) or add a wrapper of
> some kind telling what each element is. It also seems JSON has no
> built-in way to do object references.
>

actually, it can do both, just it is maybe not entirely obvious.
it helps more to imaging how JavaScript itself works, and JSON more are 
a direct mapping to associated JavaScript objects.


> I can imagine something like
>
> {
>    "some_struct #1"    : { "property" : "value" },
>    "another_struct #2" : { "reference" : "#1" }
> }
>
> where another_struct.reference would be defined as a some_struct
> pointer, so it seems I can get around this. I'm not sure how useful
> the end result would really be, though; even if it is technically
> valid as JSON data and other applications can parse it, they still
> wouldn't know what to make of it.
>
> Another (minor) problem is that JSON doesn't allow comments, while my
> format does.
>

this is all, not exactly right...


however, how it actually works gets a little less trivial to explain.

handling it more "effectively" would involve essentially throwing 
together dynamic type facilities on top of C.

a way of imagining this, is if you have some sort of opaque type, which 
can hold a value of any type (no, it is *not* a string).

a popular solution (though not the one I am using) is basically to treat 
the low bits of a pointer as a "magic tag", for example:
low 3 bits:
0=raw pointer
1/5=integer (30-bit)
2/6=float (30-bit)
3, 4=reserved
7=misc values (grabs another 5 bits as an expanded tag, with a 24-bit 
value).

(my system is based instead on "magic address ranges" for some types, 
and encoding the object type in the memory-manager's header, for other 
types).


you might have functions, then, to wrap and unwrap values, say (using my 
API as an example):
dyt dyint(s64 v);		//wrap integer (or "long")
s64 dyintv(dyt v);		//unwrap integer (or "long")
dyt dylong(s64 v);		//wrap "long"
s64 dylongv(dyt v);		//unwrap "long"
dyt dyfloat(float v);		//wrap float
float dyfloatv(dyt v);		//unwrap float
dyt dydouble(double v);		//wrap double
double dydoublev(dyt v);	//unwrap double
...

as well, as functions to perform "type-checks":
int dyintp(dyt v);	//returns non-zero if value is an integer
int dyfloatp(dyt v);
...


and, so then, a person can write logic like:
if(dyintp(v))
{
	i=dyintv(v);
}else if(dyfloatp(v))
{
	f=dyfloatv(v);
}else if(dystringp(v))
{
	...
}else if(dyObjectP(v))
{
	...
}else ...


as for parsing JSON, it will generally involve looking at the syntax for 
every token to identify what it is.

part of this is done in the "lexer"/"tokenizer", with the idea being 
that the logic for splitting apart the input text into tokens, will 
identify what sort of token it is looking at: numbers vs strings vs 
braces vs operators vs ...

the parser logic will then build a representation of the data via the 
tokens it sees.

a common strategy for hand-written parsers is "recursive descent", where 
basically the code looks at the current tokens and sees what to do, and 
then may call itself recursively in order to parse sub-items:
if(tokenty==TOK_INTEGER)
{
	value=dyint(atoi(token));
}else if(tokenty==TOK_REAL)
{
	value=dydouble(atof(token));
}else if(!strcmp(token, "{") && (tokenty==TOK_BRACE))
{
	... parse object ...
}else if(!strcmp(token, "[") && (tokenty==TOK_BRACE))
{
	... parse array ...
}else ...


a possible representation for a "naive" object, could be something like:
struct Object_s
{
char **vars;
dyt *vals;
int n_vars;
int m_vars;
};

with vars holding a list of member names, vals holding the values, and 
n_vars holding a count for the number of members in an object, and 
m_vars holding the max number of members (how many members can be added 
before the arrays would need to be reallocated).


there are potentially more efficient ways to handle all this, but this 
is a "general idea".

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 13:46 -0800
  Re: Serialization library, request for feedback Keith Thompson <kst-u@mib.org> - 2012-12-13 13:57 -0800
    Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 14:30 -0800
      Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-14 19:38 +1300
        Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 03:47 -0600
          Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-14 23:23 +1300
            Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 11:29 -0600
          Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 03:51 -0800
            Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 12:02 -0600
        Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 12:18 +0000
          Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 14:27 +0000
            Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 15:22 +0000
              Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 13:00 -0600
                Re: Serialization library, request for feedback Greg Martin <greg@softsprocket.com> - 2012-12-14 11:58 -0800
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 15:08 -0600
                Re: Serialization library, request for feedback William Ahern <william@wilbur.25thandClement.com> - 2012-12-14 20:07 -0800
                Re: Serialization library, request for feedback Nick Keighley <nick_keighley_nospam@hotmail.com> - 2012-12-16 03:49 -0800
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-16 13:17 -0600
                Re: Serialization library, request for feedback Greg Martin <greg@softsprocket.com> - 2012-12-17 09:07 -0800
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-17 13:25 -0600
              Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 09:23 +1300
                Re: Serialization library, request for feedback James Kuyper <jameskuyper@verizon.net> - 2012-12-14 15:31 -0500
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 15:26 -0600
                Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 12:25 +1300
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 18:41 -0600
                Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 14:56 +1300
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 23:36 -0600
                Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 20:13 +1300
                Re: Serialization library, request for feedback Shao Miller <sha0.miller@gmail.com> - 2012-12-14 21:42 -0500
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 23:47 -0600
              Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 21:12 +0000
                Re: Serialization library, request for feedback Johann Klammer <klammerj@NOSPAM.a1.net> - 2012-12-15 05:06 +0100
    Re: Serialization library, request for feedback Willem <willem@turtle.stack.nl> - 2012-12-14 22:20 +0000
  Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-13 23:11 +0000
    Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 15:57 -0800
      Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 02:53 +0000
        Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 04:25 -0800
      Re: Serialization library, request for feedback Bart van Ingen Schenau <bart@ingen.ddns.info.invalid> - 2012-12-14 08:04 +0000
  Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 14:25 +0000
    Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 13:58 -0600
      Re: Serialization library, request for feedback "BartC" <bc@freeuk.com> - 2012-12-14 20:44 +0000
        Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 16:58 -0600
        Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 17:39 -0800
    Re: Serialization library, request for feedback Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-12-14 14:19 -0800

csiph-web