Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #29205 > unrolled thread

Serialization library, request for feedback

Back to article view | Back to comp.lang.c

Contents

  Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 13:46 -0800
    Re: Serialization library, request for feedback Keith Thompson <kst-u@mib.org> - 2012-12-13 13:57 -0800
      Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 14:30 -0800
        Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-14 19:38 +1300
          Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 03:47 -0600
            Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-14 23:23 +1300
              Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 11:29 -0600
            Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 03:51 -0800
              Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 12:02 -0600
          Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 12:18 +0000
            Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 14:27 +0000
              Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 15:22 +0000
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 13:00 -0600
                  Re: Serialization library, request for feedback Greg Martin <greg@softsprocket.com> - 2012-12-14 11:58 -0800
                    Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 15:08 -0600
                    Re: Serialization library, request for feedback William Ahern <william@wilbur.25thandClement.com> - 2012-12-14 20:07 -0800
                    Re: Serialization library, request for feedback Nick Keighley <nick_keighley_nospam@hotmail.com> - 2012-12-16 03:49 -0800
                      Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-16 13:17 -0600
                        Re: Serialization library, request for feedback Greg Martin <greg@softsprocket.com> - 2012-12-17 09:07 -0800
                          Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-17 13:25 -0600
                Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 09:23 +1300
                  Re: Serialization library, request for feedback James Kuyper <jameskuyper@verizon.net> - 2012-12-14 15:31 -0500
                    Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 15:26 -0600
                      Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 12:25 +1300
                        Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 18:41 -0600
                          Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 14:56 +1300
                            Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 23:36 -0600
                              Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 20:13 +1300
                      Re: Serialization library, request for feedback Shao Miller <sha0.miller@gmail.com> - 2012-12-14 21:42 -0500
                        Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 23:47 -0600
                Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 21:12 +0000
                  Re: Serialization library, request for feedback Johann Klammer <klammerj@NOSPAM.a1.net> - 2012-12-15 05:06 +0100
      Re: Serialization library, request for feedback Willem <willem@turtle.stack.nl> - 2012-12-14 22:20 +0000
    Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-13 23:11 +0000
      Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 15:57 -0800
        Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 02:53 +0000
          Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 04:25 -0800
        Re: Serialization library, request for feedback Bart van Ingen Schenau <bart@ingen.ddns.info.invalid> - 2012-12-14 08:04 +0000
    Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 14:25 +0000
      Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 13:58 -0600
        Re: Serialization library, request for feedback "BartC" <bc@freeuk.com> - 2012-12-14 20:44 +0000
          Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 16:58 -0600
          Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 17:39 -0800
      Re: Serialization library, request for feedback Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-12-14 14:19 -0800

Page 1 of 3  [1] 2 3  Next page →

#29205 — Serialization library, request for feedback

Hello,

I'm writing a serialization library for C. It can convert binary data
(a struct or array, for example) into human-readable text and of
course also read serialized data back and reconstruct it in memory,
including nested pointers. The purpose is to provide a quick way to
save/load in applications and reduce the effort needed to keep file
formats in sync.

The user needs to set up translators (composed of one or more fields)
for each type they wish to serialize. Internally it works by
flattening each type into a list of void pointers and replacing them
with numeric indices. The output format has a simple "<tag> <value>;"
format; it is similar to JSON but it is not a reimplementation of it.

I'm looking for feedback how I can make this most useful to other
people. The code and documentation can be found at:
http://www.happyponyland.net/serialize.php

I'm wondering if there is anything crucial I have missed. It can deal
with most primitive C types, strings, arrays and almost any
arrangement of linked structures. There are still a few pieces
missing; for example I would like to have a clean interface to let the
user provide custom read/write functions for complex data types. I
will also work on the type safety, so structures can't be linked
incorrectly by erroneous or malicious input.

How should I structure the documentation and what level of detail does
it need (I'm guessing more is always better)? Is the API consistent?

I'm not asking anyone to proofread my 3000 lines of spaghetti, but of
course I would also appreciate to hear thoughts on the safety and
portability of the source itself. I'm open to suggestions how to
improve the conversion process.

Finally, do you think this could be genuinely useful? I'm doing it
just for fun and for use in my personal projects, but is there any
niche in programming where it would fit a need?

/Ulf

[toc] | [next] | [standalone]

#29206

Ulf Åström <ulf.astrom@gmail.com> writes:
[...]
> The user needs to set up translators (composed of one or more fields)
> for each type they wish to serialize. Internally it works by
> flattening each type into a list of void pointers and replacing them
> with numeric indices. The output format has a simple "<tag> <value>;"
> format; it is similar to JSON but it is not a reimplementation of it.
[...]

Why not just use JSON?  It would make the flattened files accessible by
other tools.

-- 
Keith Thompson (The_Other_Keith) kst-u@mib.org  <http://www.ghoti.net/~kst>
    Will write code for food.
"We must do something.  This is something.  Therefore, we must do this."
    -- Antony Jay and Jonathan Lynn, "Yes Minister"

[toc] | [prev] | [next] | [standalone]

#29207

On Dec 13, 10:57 pm, Keith Thompson <ks...@mib.org> wrote:
> Ulf Åström <ulf.ast...@gmail.com> writes:
>
> > The user needs to set up translators (composed of one or more fields)
> > for each type they wish to serialize. Internally it works by
> > flattening each type into a list of void pointers and replacing them
> > with numeric indices. The output format has a simple "<tag> <value>;"
> > format; it is similar to JSON but it is not a reimplementation of it.
>
> Why not just use JSON?  It would make the flattened files accessible by
> other tools.

I'm considering it, but I haven't read the JSON specs in detail yet.
At a glance they seem to be mostly compatible but I don't know how
well it would mesh with my memory offset <-> value layout or how I
would perform pointer substitution. I haven't decided if I want to
pursue a fully compliant JSON implementation, or take the format in
another direction entirely.

Also, there is always the question: Why not use C#, Java, or anything
else that has these things built in? Hm.

/Ulf

[toc] | [prev] | [next] | [standalone]

#29220

Ulf Åström wrote:
> On Dec 13, 10:57 pm, Keith Thompson <ks...@mib.org> wrote:
>> Ulf Åström <ulf.ast...@gmail.com> writes:
>>
>>> The user needs to set up translators (composed of one or more fields)
>>> for each type they wish to serialize. Internally it works by
>>> flattening each type into a list of void pointers and replacing them
>>> with numeric indices. The output format has a simple "<tag> <value>;"
>>> format; it is similar to JSON but it is not a reimplementation of it.
>>
>> Why not just use JSON?  It would make the flattened files accessible by
>> other tools.
>
> I'm considering it, but I haven't read the JSON specs in detail yet.

There isn't a great deal to read...

> At a glance they seem to be mostly compatible but I don't know how
> well it would mesh with my memory offset <-> value layout or how I
> would perform pointer substitution. I haven't decided if I want to
> pursue a fully compliant JSON implementation, or take the format in
> another direction entirely.

Why make things complex?  JSON is an ideal candidate for representing 
structure and array types.  It is after all designed as an object notation.

> Also, there is always the question: Why not use C#, Java, or anything
> else that has these things built in? Hm.

Other languages have better support for manipulating JSON objects, but 
at least one of them (PHP) uses a C library under the hood.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#29224

On 12/14/2012 12:38 AM, Ian Collins wrote:
> Ulf Åström wrote:
>> On Dec 13, 10:57 pm, Keith Thompson <ks...@mib.org> wrote:
>>> Ulf Åström <ulf.ast...@gmail.com> writes:
>>>
>>>> The user needs to set up translators (composed of one or more fields)
>>>> for each type they wish to serialize. Internally it works by
>>>> flattening each type into a list of void pointers and replacing them
>>>> with numeric indices. The output format has a simple "<tag> <value>;"
>>>> format; it is similar to JSON but it is not a reimplementation of it.
>>>
>>> Why not just use JSON?  It would make the flattened files accessible by
>>> other tools.
>>
>> I'm considering it, but I haven't read the JSON specs in detail yet.
>
> There isn't a great deal to read...
>
>> At a glance they seem to be mostly compatible but I don't know how
>> well it would mesh with my memory offset <-> value layout or how I
>> would perform pointer substitution. I haven't decided if I want to
>> pursue a fully compliant JSON implementation, or take the format in
>> another direction entirely.
>
> Why make things complex?  JSON is an ideal candidate for representing
> structure and array types.  It is after all designed as an object notation.
>

yep, JSON makes sense.

one possible downside though is that it doesn't normally identify object 
types, which means some means may be needed to identify what sort of 
struct is being serialized, and/or the type of array, ...

it is either that, or deal with data in a dynamically-typed manner, 
rather than directly mapping it to raw C structs.

JSON is generally better IMO for serializing dynamically-typed data, 
than for doing data-binding against structs.

another minor problem with JSON is that, in its pure form, it has no 
good way to deal with cyclic data (where a referenced object may refer 
back to a prior object), but an extended form could allow this.



in my case, one mechanism I have serializes a wider range of data into a 
binary format, but in my case, only deals with dynamically type-tagged 
data (which basically means that it was allocated via my GC API, and 
with a type-name supplied), and also that the typedefs have any relevant 
annotations (it depends on data gathered by using a tool to parse the 
headers in order to work correctly).

it may try to use a special serialization handler if one is registered, 
but will otherwise fall back to using key/value serialization of the 
structs.


the basic format (for each member) is:
<typeindex:VLI> <size:VLI> <data:byte[size]>

where 'VLI' is a special encoding for variable-length-integers.
in my case:
00-7F 		0-127
80-BF XX	128-16383
C0-DF XX XX	16384-2097151
...

a slight VLI variant (SVLI) encoding signed values by folding the sign 
into the LSB, so, the values follow a pattern:
0, -1, 1, -2, 2, ...


the format was basically just a flat-array of serialized members, and 
these members were linked via index (and allowed both backwards and 
forwards references). it has a limited form of value-identity preservation.

the typeindex members are basically indices into this array, giving the 
elements which give the type-name (which, as a simplifying assumption, 
are assumed to be ASCII strings).

index 0 is reserved for NULL, and is not encoded. index 1 is the first 
encoded index, and serves as the "root member" (basically, the "thing 
that the program asked the serializer to serialize").

as a later addition, if the typeindex is 0, then the member is a comment 
(and does not appear in the member array). a comment member immediately 
at the start of the file is used to indicate the "type" of the file 
(basically, it is a "magic string"), which is then followed by the root 
member.


a partial drawback was that the format doesn't have any good way to 
indicate "how" the data is encoded, making it potentially more subject 
to versioning issues (consider, for example, if a structure-layout 
changes, ...). I have tried previously to develop self-describing 
serialization formats in the past, but trying to make a format fully 
self-describing tends to make working with it unreasonably complex. (the 
basic idea here would be that not only would the format identify the 
types in use, but it would also encode information to describe all of 
the data encodings used by the format, down to the level of some number 
of "atomic" types, ...).


however, my Script-VM's bytecode serialization format is based on the 
above mechanism (the bytecode is actually just the result of serializing 
the output from compiling a source-module).



some amount of stuff also uses an S-Expression based notation:
999			//integer number
3.14159			//real number
"text"			//string
name			//symbol (identifier, used to identify something)
:name			//keyword (special type of literal identifier)
name:			//field or item name
...
( values )		//list of items (composed of "cons cells")
#( values )		//array of items (dynamically typed)
{ key: value ... }	//object (dynamically-typed)
#A<sig> ( ... )		//array (statically-typed)
#X<name> { key: value ... }	//struct
#L<name> { key: value ... }	//instance of a class
...
#idx#	//object index (declaration or reference)
#z	//null
#u	//undefined
#t	//true
#f	//false
...


so, for example, a struct like:
typdef dytname("foo_t") as_variant //(magic annotations, 1)
	struct Foo_s Foo;

struct Foo_s {
Foo *next;
char *name;
int x;
float y;
double z[16];
}


1: these annotations are no-op macros in a normal C compiler (and mostly 
expand to special attributes used by the header-processing tool).

"dytname()" basically gives the type-name that will be used when 
allocating instances of this struct-type (it is used to key the 
type-name back to the struct).

"as_variant" is basically a hint for how it should be handled by my 
scripting language. this modifier asserts that the type should be 
treated as a self-defined VM type (potentially opaque), rather than be 
treated as a boxed-struct or as a boxed "pointer to a struct" (more 
literally mapping the struct and/or struct pointer to the scripting 
language, causing script code to see it more like how C would see it).


with the structs being created like:
Foo *obj;
obj=gctalloc("foo_t", sizeof(Foo));	//allocate object with type-tag
obj->name=dystrdup("foo_instance_13");	//make a new tagged string
...


might be serialized as:
#0# = #X<foo_t> { next: #1# name: "foo_instance_13" x: 99 y: 4.9 z: 
#A<d> ( 2.0 3.0 ... ) }
#1# = #X<foo_t> { ... }

where this format works, but isn't really, exactly, pretty...



I also have a network protocol I call "BSXRP", which basically works 
very similar to the above (same data model, ...), just it uses Huffman 
coding and predictive context modeling of the data, and "clever" ways of 
VLC coding what data-values are sent. (compression is favorable to that 
of S-Expressions + Deflate, typically being around 25% the size, whereas 
Deflate by itself was reducing the S-Expressions to around 10% their 
original size, or IOW: around 2.5% the size of the textual 
serialization). (basically, if similar repeating structures are sent, 
prior structures may be used as "templates" for sending later 
structures, allowing them to be encoded in fewer bits, essentially 
working sort of like a dynamically-built schema).

as-before, it has special cases to allow encoding cyclic data, but the 
protocol does not generally preserve "value-identity" (value-identity or 
data-identity is its own hairy set of issues, and in my case I leave 
matters of identity to higher-level protocols).

some tweaks to the format also allow it to give modest compression 
improvements over Deflate when being used for delivering lots of short 
plaintext or binary data messages (it essentially includes a Deflate64 
like compressor as a sub-mode, but addresses some "weak areas" regarding 
Deflate).

a minor drawback though is that the context models can eat up a lot of 
memory (the memory costs are considerably higher than those of Deflate).

(it was originally written partly as a "proof of concept", but is, 
technically, pretty much overkill).


>> Also, there is always the question: Why not use C#, Java, or anything
>> else that has these things built in? Hm.
>
> Other languages have better support for manipulating JSON objects, but
> at least one of them (PHP) uses a C library under the hood.
>

yeah...

I use variants of both JSON and S-Expressions, but mostly for 
dynamically typed data.

not depending on the use of type-tags and data mined from headers would 
require a rather different implementation strategy.

most of my code is C, but I make fairly extensive use of dynamic-type 
tagging.


so, yeah, all this isn't really a "general purpose" set of solutions for 
the data-serialization process.

I suspect that though that there may not actually be any sort of 
entirely "general purpose" solution to this problem though...

and, as-is, to use my implementation would probably require dragging 
around roughly about 400 kloc of code, and it is very likely that many 
people would object to needing to use a special memory manager and 
code-processing tools to be able to use these facilities...


or, IOW:
if you allocate the data with "malloc()" or via a raw "mmap()" or 
similar, a lot of my code will have no idea what it is looking at (yes, 
a lame limitation, I know).

granted, the scripting language can partly work around it:
if you don't use "as_variant" modifier, the struct will map literally, 
and ironically, this allows script-code to still use "malloc()" for 
these types.

however, the data serialization code generally isn't this clever...


or such...

[toc] | [prev] | [next] | [standalone]

#29225

BGB wrote:
> On 12/14/2012 12:38 AM, Ian Collins wrote:
>>
>> Other languages have better support for manipulating JSON objects, but
>> at least one of them (PHP) uses a C library under the hood.
>
> yeah...
>
> I use variants of both JSON and S-Expressions, but mostly for
> dynamically typed data.
>
> not depending on the use of type-tags and data mined from headers would
> require a rather different implementation strategy.
>
> most of my code is C, but I make fairly extensive use of dynamic-type
> tagging.

I originally wrote a JSON library to enable my (C++) sever side web 
application it interact with client side JavaScript.  I soon found the 
objects extremely useful for building dynamic type objects in general 
programming.  I doubt the same would be true in C, not elegantly at least.

-- 
Ian Collins

[toc] | [prev] | [next] | [standalone]

#29238

On 12/14/2012 4:23 AM, Ian Collins wrote:
> BGB wrote:
>> On 12/14/2012 12:38 AM, Ian Collins wrote:
>>>
>>> Other languages have better support for manipulating JSON objects, but
>>> at least one of them (PHP) uses a C library under the hood.
>>
>> yeah...
>>
>> I use variants of both JSON and S-Expressions, but mostly for
>> dynamically typed data.
>>
>> not depending on the use of type-tags and data mined from headers would
>> require a rather different implementation strategy.
>>
>> most of my code is C, but I make fairly extensive use of dynamic-type
>> tagging.
>
> I originally wrote a JSON library to enable my (C++) sever side web
> application it interact with client side JavaScript.  I soon found the
> objects extremely useful for building dynamic type objects in general
> programming.  I doubt the same would be true in C, not elegantly at least.
>


I actually use a fair amount of dynamically-typed stuff in C.

another big area has to do with my scripting language (largely itself 
based on JavaScript and ActionScript).


the main downsides have to do with performance, namely that 
dynamically-typed code just doesn't perform anywhere near as well as 
statically-typed code.

in many areas, this isn't a big issue, and/or leads to a hybrid 
strategy, with some occasional type-checking, but much of the code 
remains statically typed (for example, much of my 3D renderer largely 
works this way).

typically, this would be using dynamic type-checking more for things 
like classifying "what type of object is this?" and dispatching more to 
the appropriate part of the renderer (or, IOW: type-checking on the 
level of structures). a few places are "optimized" though, mostly to try 
to avoid using dynamically-typed operations in tight-loops (such as 
scene-graph object lookup operations, ...).


side note (main project):
http://www.youtube.com/user/BGBTech

http://www.youtube.com/watch?v=hhdq87CXL0U

(note, a lot of parts of the video are from much earlier versions of my 
3D engine...).


(although the 3D engine has some amount of code written in my 
script-language, I generally don't use script-code within the 3D 
renderer, as the languages' performance is a bit weak, and the 3D 
renderer is a lot more performance-sensitive).

sadly, performance is kind of a problem area sometimes, but much of this 
more has to do with getting everything fed through OpenGL.


in my case, as-noted, the type-tagging is largely transparent to C code, 
apart from the use of special allocation and freeing functions, and 
supplying type-names, ...

these type-names can be retrieved via API calls, like:
BGBDY_API char *dygettype(dyt val);

or checked with functions like:
BGBDY_API int dyTypeP(dyt val, char *ty);


many operations on dynamically typed values may check types using these 
mechanisms, but many internal operations essentially bypass this 
(usually, because if/else chains of "strcmp()" calls can get expensive...).


elsewhere, for many core operations rather than fetching the type-name 
from the object, the code will instead fetch a "type vtable":

BGBDY_API dyt dyCopyValue(dyt p)
{
	BGBDY_ObjVTab *ty;
	if(dyCheckValueNoCopyP(p))
		return(p);
	ty=BGBDY_GetObjVTabFast(p);
	if(ty && ty->copyValue)
		return(ty->copyValue(p));
	return(p);
}

BGBDY_API int dyDropValue(dyt p)
{
	BGBDY_ObjVTab *ty;
	if(dyCheckValueNoCopyP(p))
		return(0);
	ty=BGBDY_GetObjVTabFast(p);
	if(ty && ty->dropValue)
		return(ty->dropValue(p));
	return(0);
}

where BGBDY_GetObjVTabFast basically calls off to code for some 
(slightly long/ugly) logic for fetching the object-base, fetching the 
"ObjType", and returning its "vtable" member.



in a few places (due mostly to performance issues), some of this has 
lead to frequent use of things like type-signatures and untagged union 
types:

#ifndef DYTVALUE_T
#define DYTVALUE_T
typedef union
{
s32 i;				//int
u32 ui;				//uint
s64 l;				//long
u64 ul;				//ulong
f32 f;				//float
f64 d;				//double
nlint a;			//address
void *pv;			//raw pointer
dyt r;				//variant reference
dytf cr;			//variant reference (double)
dycClass cls;			//class
dycObject obj;			//object
dycArray arr;			//array
struct { int x, y; }k;		//lexical env index
}dycValue;
#endif


the main reason being that these allow higher performance.

for the untagged union, it is because one can access values like:
dycValue a, b, c;
...
c.i=a.i+b.i;

and, apparently the compiler is smart enough to figure out this at least 
(and generate vaguely efficient code).

the reason for type-signatures is that unlike full dynamic typing, 
signatures are more readily usable with the "build stuff with function 
pointers" strategy.

say, for example:
void DYLL_CopyValueBufI(dycValue *v, void *p)
	{ *(s32 *)p=v->i; }
void DYLL_CopyValueBufL(dycValue *v, void *p)
	{ *(s64 *)p=v->l; }
...

typedef struct {
void (*CopyValueBuf)(dycValue *v, void *p);
void (*CopyBufValue)(void *p, dycValue *v);
...
int offs;
}DYLL_ValueTransfer;

typedef struct {
DYLL_ValueTransfer *ret;
DYLL_ValueTransfer *args;
int nargs;
void (*CopyValuesBuf)(DYLL_ArgList *ctx, dycValue *args, byte *buf);
void (*CopyBufValues)(DYLL_ArgList *ctx, byte *buf, dycValue *args);
...
}DYLL_ArgList;

...

void DYLL_CopyValuesBuf3(DYLL_ArgList *ctx, dycValue *args, byte *buf)
{
	ctx->args[0].CopyValueBuf(args+0, buf+ctx->args[0].offs);
	ctx->args[1].CopyValueBuf(args+1, buf+ctx->args[1].offs);
	ctx->args[2].CopyValueBuf(args+2, buf+ctx->args[2].offs);
}

BGBDY_API void DYLL_CopyValuesBuf(DYLL_ArgList *ctx,
	dycValue *args, byte *buf)
	{ ctx->CopyValuesBuf(ctx, args, buf); }

where the "DYLL_ArgList" structure may be built via a function which 
parses the signature.

...

which, although arguably not exactly efficient, can always be a good 
deal worse...

(there is a good deal more of this sort of logic for working with 
"va_list" argument lists as well, typically for indirect calls 
originating within C land, a lot of this stuff has to deal with calls 
between natively compiled and interpreted code).

generally the "buffer" is a packed buffer-like representation of an 
argument list, and is generally what is used by the ASM parts of the 
call-handling machinery (which may in-turn repack these buffer into 
ABI-specific representations, and then initiate the call).

partly, this is because, it isn't really desirable to have to deal with 
the various possible argument-list representations and ASM code at the 
same time.


in some places, the function-pointer funkiness serves to bypass older 
machinery which often involves lots of sometimes long if/else chains.


a downside though is that a lot of this is, admittedly, a bit more 
painful to work with, and the code can get very long/winding and ugly 
(so, the spread of this sort of code in many places within the VM core 
has both good and bad points).

however, a lot of these sorts of mechanisms end up somewhat abstracted, 
and so are generally unseen by "user code" (luckily, it all looks much 
nicer from the outside).


or such...

[toc] | [prev] | [next] | [standalone]

#29226

On Dec 14, 10:47 am, BGB <cr88...@hotmail.com> wrote:
> On 12/14/2012 12:38 AM, Ian Collins wrote:
> > Ulf str m wrote:
> >> On Dec 13, 10:57 pm, Keith Thompson <ks...@mib.org> wrote:
> >>> Ulf str m <ulf.ast...@gmail.com> writes:
>
> >>>> The user needs to set up translators (composed of one or more fields)
> >>>> for each type they wish to serialize. Internally it works by
> >>>> flattening each type into a list of void pointers and replacing them
> >>>> with numeric indices. The output format has a simple "<tag> <value>;"
> >>>> format; it is similar to JSON but it is not a reimplementation of it.
>
> >>> Why not just use JSON?  It would make the flattened files accessible by
> >>> other tools.
>
> >> At a glance they seem to be mostly compatible but I don't know how
> >> well it would mesh with my memory offset <-> value layout or how I
> >> would perform pointer substitution. I haven't decided if I want to
> >> pursue a fully compliant JSON implementation, or take the format in
> >> another direction entirely.
>
> > Why make things complex?  JSON is an ideal candidate for representing
> > structure and array types.  It is after all designed as an object notation.
>
> yep, JSON makes sense.
>
> one possible downside though is that it doesn't normally identify object
> types, which means some means may be needed to identify what sort of
> struct is being serialized, and/or the type of array, ...
>
> it is either that, or deal with data in a dynamically-typed manner,
> rather than directly mapping it to raw C structs.

I can see a couple of problems with JSON. It is really just an
associative array and doesn't specify the type of objects, so I would
either need to guess (from the fields provided) or add a wrapper of
some kind telling what each element is. It also seems JSON has no
built-in way to do object references.

I can imagine something like

{
  "some_struct #1"    : { "property" : "value" },
  "another_struct #2" : { "reference" : "#1" }
}

where another_struct.reference would be defined as a some_struct
pointer, so it seems I can get around this. I'm not sure how useful
the end result would really be, though; even if it is technically
valid as JSON data and other applications can parse it, they still
wouldn't know what to make of it.

Another (minor) problem is that JSON doesn't allow comments, while my
format does.

/Ulf

[toc] | [prev] | [next] | [standalone]

#29239

On 12/14/2012 5:51 AM, Ulf Åström wrote:
> On Dec 14, 10:47 am, BGB <cr88...@hotmail.com> wrote:
>> On 12/14/2012 12:38 AM, Ian Collins wrote:
>>> Ulf str m wrote:
>>>> On Dec 13, 10:57 pm, Keith Thompson <ks...@mib.org> wrote:
>>>>> Ulf str m <ulf.ast...@gmail.com> writes:
>>
>>>>>> The user needs to set up translators (composed of one or more fields)
>>>>>> for each type they wish to serialize. Internally it works by
>>>>>> flattening each type into a list of void pointers and replacing them
>>>>>> with numeric indices. The output format has a simple "<tag> <value>;"
>>>>>> format; it is similar to JSON but it is not a reimplementation of it.
>>
>>>>> Why not just use JSON?  It would make the flattened files accessible by
>>>>> other tools.
>>
>>>> At a glance they seem to be mostly compatible but I don't know how
>>>> well it would mesh with my memory offset <-> value layout or how I
>>>> would perform pointer substitution. I haven't decided if I want to
>>>> pursue a fully compliant JSON implementation, or take the format in
>>>> another direction entirely.
>>
>>> Why make things complex?  JSON is an ideal candidate for representing
>>> structure and array types.  It is after all designed as an object notation.
>>
>> yep, JSON makes sense.
>>
>> one possible downside though is that it doesn't normally identify object
>> types, which means some means may be needed to identify what sort of
>> struct is being serialized, and/or the type of array, ...
>>
>> it is either that, or deal with data in a dynamically-typed manner,
>> rather than directly mapping it to raw C structs.
>
> I can see a couple of problems with JSON. It is really just an
> associative array and doesn't specify the type of objects, so I would
> either need to guess (from the fields provided) or add a wrapper of
> some kind telling what each element is. It also seems JSON has no
> built-in way to do object references.
>

actually, it can do both, just it is maybe not entirely obvious.
it helps more to imaging how JavaScript itself works, and JSON more are 
a direct mapping to associated JavaScript objects.


> I can imagine something like
>
> {
>    "some_struct #1"    : { "property" : "value" },
>    "another_struct #2" : { "reference" : "#1" }
> }
>
> where another_struct.reference would be defined as a some_struct
> pointer, so it seems I can get around this. I'm not sure how useful
> the end result would really be, though; even if it is technically
> valid as JSON data and other applications can parse it, they still
> wouldn't know what to make of it.
>
> Another (minor) problem is that JSON doesn't allow comments, while my
> format does.
>

this is all, not exactly right...


however, how it actually works gets a little less trivial to explain.

handling it more "effectively" would involve essentially throwing 
together dynamic type facilities on top of C.

a way of imagining this, is if you have some sort of opaque type, which 
can hold a value of any type (no, it is *not* a string).

a popular solution (though not the one I am using) is basically to treat 
the low bits of a pointer as a "magic tag", for example:
low 3 bits:
0=raw pointer
1/5=integer (30-bit)
2/6=float (30-bit)
3, 4=reserved
7=misc values (grabs another 5 bits as an expanded tag, with a 24-bit 
value).

(my system is based instead on "magic address ranges" for some types, 
and encoding the object type in the memory-manager's header, for other 
types).


you might have functions, then, to wrap and unwrap values, say (using my 
API as an example):
dyt dyint(s64 v);		//wrap integer (or "long")
s64 dyintv(dyt v);		//unwrap integer (or "long")
dyt dylong(s64 v);		//wrap "long"
s64 dylongv(dyt v);		//unwrap "long"
dyt dyfloat(float v);		//wrap float
float dyfloatv(dyt v);		//unwrap float
dyt dydouble(double v);		//wrap double
double dydoublev(dyt v);	//unwrap double
...

as well, as functions to perform "type-checks":
int dyintp(dyt v);	//returns non-zero if value is an integer
int dyfloatp(dyt v);
...


and, so then, a person can write logic like:
if(dyintp(v))
{
	i=dyintv(v);
}else if(dyfloatp(v))
{
	f=dyfloatv(v);
}else if(dystringp(v))
{
	...
}else if(dyObjectP(v))
{
	...
}else ...


as for parsing JSON, it will generally involve looking at the syntax for 
every token to identify what it is.

part of this is done in the "lexer"/"tokenizer", with the idea being 
that the logic for splitting apart the input text into tokens, will 
identify what sort of token it is looking at: numbers vs strings vs 
braces vs operators vs ...

the parser logic will then build a representation of the data via the 
tokens it sees.

a common strategy for hand-written parsers is "recursive descent", where 
basically the code looks at the current tokens and sees what to do, and 
then may call itself recursively in order to parse sub-items:
if(tokenty==TOK_INTEGER)
{
	value=dyint(atoi(token));
}else if(tokenty==TOK_REAL)
{
	value=dydouble(atof(token));
}else if(!strcmp(token, "{") && (tokenty==TOK_BRACE))
{
	... parse object ...
}else if(!strcmp(token, "[") && (tokenty==TOK_BRACE))
{
	... parse array ...
}else ...


a possible representation for a "naive" object, could be something like:
struct Object_s
{
char **vars;
dyt *vals;
int n_vars;
int m_vars;
};

with vars holding a list of member names, vals holding the values, and 
n_vars holding a count for the number of members in an object, and 
m_vars holding the max number of members (how many members can be added 
before the arrays would need to be reallocated).


there are potentially more efficient ways to handle all this, but this 
is a "general idea".

[toc] | [prev] | [next] | [standalone]

#29228

Ian Collins <ian-news@hotmail.com> writes:

> Ulf Åström wrote:
>> On Dec 13, 10:57 pm, Keith Thompson <ks...@mib.org> wrote:
>>> Ulf Åström <ulf.ast...@gmail.com> writes:
>>>
>>>> The user needs to set up translators (composed of one or more fields)
>>>> for each type they wish to serialize. Internally it works by
>>>> flattening each type into a list of void pointers and replacing them
>>>> with numeric indices. The output format has a simple "<tag> <value>;"
>>>> format; it is similar to JSON but it is not a reimplementation of it.
>>>
>>> Why not just use JSON?  It would make the flattened files accessible by
>>> other tools.
>>
>> I'm considering it, but I haven't read the JSON specs in detail yet.
>
> There isn't a great deal to read...
>
>> At a glance they seem to be mostly compatible but I don't know how
>> well it would mesh with my memory offset <-> value layout or how I
>> would perform pointer substitution. I haven't decided if I want to
>> pursue a fully compliant JSON implementation, or take the format in
>> another direction entirely.
>
> Why make things complex?  JSON is an ideal candidate for representing
> structure and array types.  It is after all designed as an object
> notation.

I don't think JSON is a good fit for the OP's task.  There are lots of C
data structures that have no obvious JSON representation at all.

<snip>
-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#29235

Ben Bacarisse wrote:

> I don't think JSON is a good fit for the OP's task.  There are lots of C
> data structures that have no obvious JSON representation at all.

What data structures do you have in mind?


Rui Maciel

[toc] | [prev] | [next] | [standalone]

#29236

Rui Maciel <rui.maciel@gmail.com> writes:

> Ben Bacarisse wrote:
>
>> I don't think JSON is a good fit for the OP's task.  There are lots of C
>> data structures that have no obvious JSON representation at all.
>
> What data structures do you have in mind?

My ring_buffer example is one.  In fact, any structure where pointers are
"shared" would seem to be a bad fit with JSON.

Maybe "no obvious JSON representation" is too strong because you can
probably always map pointers so some sort of index or string label, but
it's not a natural fit.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#29240

On 12/14/2012 9:22 AM, Ben Bacarisse wrote:
> Rui Maciel <rui.maciel@gmail.com> writes:
>
>> Ben Bacarisse wrote:
>>
>>> I don't think JSON is a good fit for the OP's task.  There are lots of C
>>> data structures that have no obvious JSON representation at all.
>>
>> What data structures do you have in mind?
>
> My ring_buffer example is one.  In fact, any structure where pointers are
> "shared" would seem to be a bad fit with JSON.
>
> Maybe "no obvious JSON representation" is too strong because you can
> probably always map pointers so some sort of index or string label, but
> it's not a natural fit.
>

yeah, these are general problems with data serialization, namely that 
there is no good way to deal with "everything" within any reasonable 
level of complexity.

it is easier then to find a reasonable subset of things that can be 
serialized via a given mechanism, and use this.


more careful design of a serialization interface would be needed, for 
example, to be able to deal with cyclic data structures, and more 
in-turn to deal with preserving the identity of serialized objects (1).

1: if obja==objb, will this be true or false after the data is saved, 
and re-loaded?...


some options will not preserve any identity, so if the same object is 
reached via multiple paths, each may become separate objects when the 
same image is read-in later.

other options may in-turn create false identity, where pretty much any 
object which "looks the same" will have the same identity, regardless of 
whether or not it was the same object prior to serialization.

identity becomes a particularly ugly issue if it needs to be dealt with 
across the same object being saved across multiple images, or if network 
gets involved (currently, I just don't deal with this problem). (this 
part gets much more important if dealing with trying to serialize things 
like program-state, say, where a "running program image" may be saved to 
disk, and reloaded later, and lots of little "subtle details" will need 
to be preserved, or else things will often go "critically wrong" when 
the program state gets reloaded later on).


or, at least, short of much simpler but very ugly solutions, like doing 
a raw memory-dump of the program's heap (I have done this before, but it 
"doesn't really scale well"), but more practical solutions to general 
program heap serialization are overly complicated, and it is easier just 
to deal with smaller scale serialization and reloading (such as 
"serialize this particular data structure").

a basic idea for how the heap-dump serialization works is:
imagine, if you will, that on startup the application uses a 
memory-mapped file as the main VM heap, and uses special magic to 
preserve any "roots" (generally global variables). on exiting and 
reloading, any objects in this heap can be preserved, with the "roots" 
automatically restored to their prior values.

yeah, long ago, one of my prior VMs actually did this, although the heap 
wasn't a single giant file, but rather a bunch of several-MB "heap chunks".

I later generally abandoned this strategy, as it had some particularly 
nasty technical issues (every exit/reload cycle leaked memory that could 
not be reclaimed, nasty "bit rot" problems, structure-layout problems, ...).

it is IMO better to actually just try to "serialize the contents of the 
data", rather than just "stupidly dump the memory to the HDD and try to 
reload it later".



but, in general, dealing with these sorts of issues requires more 
careful attention to detail, and sadly, this isn't really something JSON 
does very well.

not that this makes JSON useless though.


more generally though, I just use plain files for data storage, as they 
just work so-much better and more reliably.


one possible (partial) solution here for cyclic structures/... though is 
an "extended and non-standard" JSON variant, say, where everything is 
indexed.

#1 = { name: "foo", link: #2 };
#2 = { name: "bar", link: #3 };
#3 = { name: "baz", link: #4, first: #1 };
...
{ name: "someobject", listHead: #1 }

but, dealing effectively with a more comprehensive type-system would 
require further somewhat extending the syntax, say, adding some 
mechanism to identify up-front the type of object being serialized, ...


this is partly what my "somewhat hacked-up S-Expressions" do, even if 
they are a little out of place (long ago, I had a Scheme implementation, 
and even now, vestiges of it still remain, floating around in my 
codebase...).


or such...

[toc] | [prev] | [next] | [standalone]

#29241

On 12-12-14 11:00 AM, BGB wrote:
> On 12/14/2012 9:22 AM, Ben Bacarisse wrote:
>> Rui Maciel <rui.maciel@gmail.com> writes:
>>
>>> Ben Bacarisse wrote:
>>>
>>>> I don't think JSON is a good fit for the OP's task.  There are lots
>>>> of C
>>>> data structures that have no obvious JSON representation at all.
>>>
>>> What data structures do you have in mind?
>>
>> My ring_buffer example is one.  In fact, any structure where pointers are
>> "shared" would seem to be a bad fit with JSON.
>>
>> Maybe "no obvious JSON representation" is too strong because you can
>> probably always map pointers so some sort of index or string label, but
>> it's not a natural fit.
>>
>
> yeah, these are general problems with data serialization, namely that
> there is no good way to deal with "everything" within any reasonable
> level of complexity.
>
> it is easier then to find a reasonable subset of things that can be
> serialized via a given mechanism, and use this.
>
>

At one point I did a lot of work with ASN.1, which is a specification 
for defining protocols but is implemented using a variety of encoding 
schemes. It actually is an effective means of serializing data and can 
be very compact and, as I recall, extended to covering most data 
encodings that I ran into but it never seemed trivial to me. Debugging a 
stream of encoded data can lead to premature blindness and post 
traumatic drunkenness. Well, maybe that was just me. To interpret the 
data correctly you need the specification for encoding, BER for example, 
and also the ASN.1 specification.

Lately I've been embedding the V8 engine in an application. I haven't 
looked at how they've implemented the JSON objects but it might be worth 
a peek for anyone interested in serialization techniques. I'm very 
impressed with the V8 JavaScript engine.

[toc] | [prev] | [next] | [standalone]

#29248

On 12/14/2012 1:58 PM, Greg Martin wrote:
> On 12-12-14 11:00 AM, BGB wrote:
>> On 12/14/2012 9:22 AM, Ben Bacarisse wrote:
>>> Rui Maciel <rui.maciel@gmail.com> writes:
>>>
>>>> Ben Bacarisse wrote:
>>>>
>>>>> I don't think JSON is a good fit for the OP's task.  There are lots
>>>>> of C
>>>>> data structures that have no obvious JSON representation at all.
>>>>
>>>> What data structures do you have in mind?
>>>
>>> My ring_buffer example is one.  In fact, any structure where pointers
>>> are
>>> "shared" would seem to be a bad fit with JSON.
>>>
>>> Maybe "no obvious JSON representation" is too strong because you can
>>> probably always map pointers so some sort of index or string label, but
>>> it's not a natural fit.
>>>
>>
>> yeah, these are general problems with data serialization, namely that
>> there is no good way to deal with "everything" within any reasonable
>> level of complexity.
>>
>> it is easier then to find a reasonable subset of things that can be
>> serialized via a given mechanism, and use this.
>>
>>
>
> At one point I did a lot of work with ASN.1, which is a specification
> for defining protocols but is implemented using a variety of encoding
> schemes. It actually is an effective means of serializing data and can
> be very compact and, as I recall, extended to covering most data
> encodings that I ran into but it never seemed trivial to me. Debugging a
> stream of encoded data can lead to premature blindness and post
> traumatic drunkenness. Well, maybe that was just me. To interpret the
> data correctly you need the specification for encoding, BER for example,
> and also the ASN.1 specification.
>

ASN.1 BER isn't particularly compact.

it is about the same as my "typical" binary serialization format, where 
both will typically require around 2-bytes for a marker, and 0 or more 
bytes for payload data.

granted, yes, it is much more compact than more naive binary formats, or 
simpler TLV formats like IFF or RIFF, but, either way...


ASN.1 PER is a bit more compact.
basically, it maps data to fixed-size bit-fields.


part of the "proof of concept" part of my "BSXRP" protocol was that it 
is possible to generate a message stream on-average more compact than 
ASN.1 PER, and without needing to make use of a dedicated schema. they 
work in different ways though, whereas PER uses fixed bit-packing, my 
protocol borrowed more heavily from Deflate and JPEG, basically working 
to reduce the data mostly to near-0 integer values (via "prediction"), 
which can be relatively efficiently coded via a VLC scheme (a 
Huffman-coded value followed by 0 or more "extra bits").

say, one can encode one of:
a VLC value indicating which "prediction" is correct (regarding the next 
item);
a VLC value indicating an item within an MRU-list of "recently seen items";
a direct representation of the item in question, in the case where it 
hasn't been seen before, or was "forgotten" (fell off the end of the MRU).

Huffman coding these choice-values will tend to use a near optimal 
number of bits (vs a fixed-field encoding, which will tend to allow 
every possibility with a near equal weight regardless of the relative 
probability of a given choice).

say, there are 6 possible choices.
a fixed-width encoding would use 3 bits here.

but, what if one choice is much more common than another:
000 = 90%
001 = 8%
others = 2%

then, 3 bits are no longer optimal, and instead we might want something 
like:
000 -> 0
001 -> 10
others -> 11xxx

now, 90% of the time, we only need 1 bit.
this is part of the advantage of Huffman coding.


note that an Arithmetic coder can compress better than Huffman coding 
(because it can deal with "fractional bits"), but tends to be a little 
slower to encode/decode (note, however, that the H.264 video-codec is 
built around an arithmetic coder).

in my case things like strings and byte-arrays are compressed using an 
LZ77 scheme very similar to Deflate (but using a 64kB window rather than 
a 32kB one), and also does not need to resend the Huffman table with 
each message.


personally though, I don't really like ASN.1, mostly on the grounds that 
its design requires a schema in order to work correctly, and I much more 
prefer formats which will handle "whatever you throw at them" even 
without any sort of schema.


> Lately I've been embedding the V8 engine in an application. I haven't
> looked at how they've implemented the JSON objects but it might be worth
> a peek for anyone interested in serialization techniques. I'm very
> impressed with the V8 JavaScript engine.
>

yeah, V8 is probably fairly good...


I looked at it before, but in my case decided to stick with my own 
Scripting-VM (the VM is sufficiently tightly integrated with my project, 
that it isn't really an "easily replaceable" component).

both have it in common though that they run ECMAScript / JavaScript 
variants though (but, have different language extensions...).

but, mine has a fancier FFI, albeit worse performance...
but, it is easier to try to optimize things as-needed, than try to rip 
out and rewrite around 2/3's of the 3D engine's codebase.


or such...

[toc] | [prev] | [next] | [standalone]

#29264

Greg Martin <greg@softsprocket.com> wrote:
<snip>
> At one point I did a lot of work with ASN.1, which is a specification 
> for defining protocols but is implemented using a variety of encoding 
> schemes. It actually is an effective means of serializing data and can 
> be very compact and, as I recall, extended to covering most data 
> encodings that I ran into but it never seemed trivial to me. Debugging a 
> stream of encoded data can lead to premature blindness and post 
> traumatic drunkenness. Well, maybe that was just me. To interpret the 
> data correctly you need the specification for encoding, BER for example, 
> and also the ASN.1 specification.

BER is complex because you're supposed to use an ASN.1 compiler to generate
the parser as well as the code to build the data structure. The problem is
that most software just includes a very simple BER decoder, and then code
everything else ad hoc. The value proposition of ASN.1/BER is totally lost.
That's why there's been no end to certificate parsing bugs in OpenSSL and
similar software.

If you're using a compiler, you're not trying to debug the on-wire format.
Once the bugs are squeezed out of the compiler, everything should work
seamlessly, for old and new message definitions. See, e.g.,
http://lionet.info/asn1c/blog/

[toc] | [prev] | [next] | [standalone]

#29325

On Dec 14, 7:58 pm, Greg Martin <g...@softsprocket.com> wrote:
> On 12-12-14 11:00 AM, BGB wrote:
>
>
>
>
>
>
>
>
>
> > On 12/14/2012 9:22 AM, Ben Bacarisse wrote:
> >> Rui Maciel <rui.mac...@gmail.com> writes:
>
> >>> Ben Bacarisse wrote:
>
> >>>> I don't think JSON is a good fit for the OP's task.  There are lots
> >>>> of C
> >>>> data structures that have no obvious JSON representation at all.
>
> >>> What data structures do you have in mind?
>
> >> My ring_buffer example is one.  In fact, any structure where pointers are
> >> "shared" would seem to be a bad fit with JSON.
>
> >> Maybe "no obvious JSON representation" is too strong because you can
> >> probably always map pointers so some sort of index or string label, but
> >> it's not a natural fit.
>
> > yeah, these are general problems with data serialization, namely that
> > there is no good way to deal with "everything" within any reasonable
> > level of complexity.
>
> > it is easier then to find a reasonable subset of things that can be
> > serialized via a given mechanism, and use this.
>
> At one point I did a lot of work with ASN.1, which is a specification
> for defining protocols but is implemented using a variety of encoding
> schemes. It actually is an effective means of serializing data and can
> be very compact and, as I recall, extended to covering most data
> encodings that I ran into but it never seemed trivial to me. Debugging a
> stream of encoded data can lead to premature blindness and post
> traumatic drunkenness.

I wrote a vary basic ASN.1 decoder. It didn't seem that hard...


Well, maybe that was just me. To interpret the
> data correctly you need the specification for encoding, BER for example,
> and also the ASN.1 specification.
>
> Lately I've been embedding the V8 engine in an application. I haven't
> looked at how they've implemented the JSON objects but it might be worth
> a peek for anyone interested in serialization techniques. I'm very
> impressed with the V8 JavaScript engine.

[toc] | [prev] | [next] | [standalone]

#29336

On 12/16/2012 5:49 AM, Nick Keighley wrote:
> On Dec 14, 7:58 pm, Greg Martin <g...@softsprocket.com> wrote:
>> On 12-12-14 11:00 AM, BGB wrote:
>>
>>
>>> On 12/14/2012 9:22 AM, Ben Bacarisse wrote:
>>>> Rui Maciel <rui.mac...@gmail.com> writes:
>>
>>>>> Ben Bacarisse wrote:
>>
>>>>>> I don't think JSON is a good fit for the OP's task.  There are lots
>>>>>> of C
>>>>>> data structures that have no obvious JSON representation at all.
>>
>>>>> What data structures do you have in mind?
>>
>>>> My ring_buffer example is one.  In fact, any structure where pointers are
>>>> "shared" would seem to be a bad fit with JSON.
>>
>>>> Maybe "no obvious JSON representation" is too strong because you can
>>>> probably always map pointers so some sort of index or string label, but
>>>> it's not a natural fit.
>>
>>> yeah, these are general problems with data serialization, namely that
>>> there is no good way to deal with "everything" within any reasonable
>>> level of complexity.
>>
>>> it is easier then to find a reasonable subset of things that can be
>>> serialized via a given mechanism, and use this.
>>
>> At one point I did a lot of work with ASN.1, which is a specification
>> for defining protocols but is implemented using a variety of encoding
>> schemes. It actually is an effective means of serializing data and can
>> be very compact and, as I recall, extended to covering most data
>> encodings that I ran into but it never seemed trivial to me. Debugging a
>> stream of encoded data can lead to premature blindness and post
>> traumatic drunkenness.
>
> I wrote a vary basic ASN.1 decoder. It didn't seem that hard...
>

yeah, not very hard, and not very notable either...


ASN.1 BER is a moderately compact TLV format.
but, then again, so is Matroska MKV.


it can be beaten out by more specialized formats regarding compactness.
many other designs also need not use a schema, which can be an arbitrary 
limitation for a file-format.

neither allows stream resynchronization AFAIK (basically, like what is 
commonly done in things like MPEG streams).


a more compact format allowing for resynchronization and not requiring a 
schema can be created by borrowing much of its design from 
data-compression formats.

but, sadly, some people seem to think of data-compression techniques as 
bordering on magic...


a sad thing here is that many protocol designers recently have taken to 
a strategy of being like:
we will take a very inefficient representation (like textual XML), and 
feed it through Deflate, and it will be "compact".

it works, sort of, but is far from optimal in many areas (if a person 
knows what is going on internally), and why, for example, Deflate isn't 
actually very good for streaming data (its design is much better suited 
for what it was originally designed for: file compression).

but, people are lazy, and XML+Deflate is often an "easy" solution.
(and, presumably for them, EXI was created...).


but, I guess in the minds of many people, designing a protocol (or 
file-format) based directly on an entropy-coded bit-stream likely seems 
like pure evil.

then again, even in the design of my own protocol, there were a few 
things I would probably do differently now (mostly small things, like 
related to how the escape markers work, ...).


but, I did what was in my case (more or less) the path of least effort.
like, why did I choose a list-based high-level representation rather 
than XML?...

mostly because lists are (in my case) considerably higher-performance, 
and also easier to work with (composing and processing XML data is 
considerably more of a pain than doing ad-hoc "Lisp in C" stuff).

I originally considered using XML, but the awkwardness and relative 
inefficiency of using a DOM-like API for composing/processing messages 
was discouraging, so I opted for using lists instead (even if, sadly, 
they are some sort of "arcane" technology).


granted though, this system is not based on any sort of "data binding", 
as in-general, I am not really that big into data-binding (data-binding 
has its own sets of drawbacks).

so, basically, explicit code walks the scene-graph and generates the 
messages, and explicit code on the other end processes the messages...

(the lists still exist in both single and multiplayer games, but the 
actual compression and sockets are usually only used for multiplayer games).


or such...

[toc] | [prev] | [next] | [standalone]

#29407

On 12-12-16 11:17 AM, BGB wrote:
> On 12/16/2012 5:49 AM, Nick Keighley wrote:
>> On Dec 14, 7:58 pm, Greg Martin <g...@softsprocket.com> wrote:

>>>
>>> At one point I did a lot of work with ASN.1, which is a specification
>>> for defining protocols but is implemented using a variety of encoding
>>> schemes. It actually is an effective means of serializing data and can
>>> be very compact and, as I recall, extended to covering most data
>>> encodings that I ran into but it never seemed trivial to me. Debugging a
>>> stream of encoded data can lead to premature blindness and post
>>> traumatic drunkenness.
>>
>> I wrote a vary basic ASN.1 decoder. It didn't seem that hard...
>>
>
> yeah, not very hard, and not very notable either...
>
>
> ASN.1 BER is a moderately compact TLV format.
> but, then again, so is Matroska MKV.
>
>
> it can be beaten out by more specialized formats regarding compactness.
> many other designs also need not use a schema, which can be an arbitrary
> limitation for a file-format.
>

I found that to be the real issue with ASN.1. The schema's I was working 
to were complex and deeply layered and, sometimes, poorly documented.

> neither allows stream resynchronization AFAIK (basically, like what is
> commonly done in things like MPEG streams).
>
>
> a more compact format allowing for resynchronization and not requiring a
> schema can be created by borrowing much of its design from
> data-compression formats.
>
> but, sadly, some people seem to think of data-compression techniques as
> bordering on magic...
>
>
> a sad thing here is that many protocol designers recently have taken to
> a strategy of being like:
> we will take a very inefficient representation (like textual XML), and
> feed it through Deflate, and it will be "compact".
>

While I think that XML is too verbose, JSON is easy to understand, debug 
and work with. Viewing XML as anything other then markup leads to 
nastiness like SOAP.

I prefer to see the compression decoupled from the protocol. It makes 
debugging and decoding simpler when a stream can be captured and read as 
text. YMMV.

[toc] | [prev] | [next] | [standalone]

#29416

On 12/17/2012 11:07 AM, Greg Martin wrote:
> On 12-12-16 11:17 AM, BGB wrote:
>> On 12/16/2012 5:49 AM, Nick Keighley wrote:
>>> On Dec 14, 7:58 pm, Greg Martin <g...@softsprocket.com> wrote:
>
>>>>
>>>> At one point I did a lot of work with ASN.1, which is a specification
>>>> for defining protocols but is implemented using a variety of encoding
>>>> schemes. It actually is an effective means of serializing data and can
>>>> be very compact and, as I recall, extended to covering most data
>>>> encodings that I ran into but it never seemed trivial to me.
>>>> Debugging a
>>>> stream of encoded data can lead to premature blindness and post
>>>> traumatic drunkenness.
>>>
>>> I wrote a vary basic ASN.1 decoder. It didn't seem that hard...
>>>
>>
>> yeah, not very hard, and not very notable either...
>>
>>
>> ASN.1 BER is a moderately compact TLV format.
>> but, then again, so is Matroska MKV.
>>
>>
>> it can be beaten out by more specialized formats regarding compactness.
>> many other designs also need not use a schema, which can be an arbitrary
>> limitation for a file-format.
>>
>
> I found that to be the real issue with ASN.1. The schema's I was working
> to were complex and deeply layered and, sometimes, poorly documented.
>

yeah, pretty much...


>> neither allows stream resynchronization AFAIK (basically, like what is
>> commonly done in things like MPEG streams).
>>
>>
>> a more compact format allowing for resynchronization and not requiring a
>> schema can be created by borrowing much of its design from
>> data-compression formats.
>>
>> but, sadly, some people seem to think of data-compression techniques as
>> bordering on magic...
>>
>>
>> a sad thing here is that many protocol designers recently have taken to
>> a strategy of being like:
>> we will take a very inefficient representation (like textual XML), and
>> feed it through Deflate, and it will be "compact".
>>
>
> While I think that XML is too verbose, JSON is easy to understand, debug
> and work with. Viewing XML as anything other then markup leads to
> nastiness like SOAP.
>

XML is fairly commonly used as a data-storage format though.
hence, things like SOAP are hardly uncommon.

not really that SOAP is "actually good"... (people just use it for 
whatever reason).


> I prefer to see the compression decoupled from the protocol. It makes
> debugging and decoding simpler when a stream can be captured and read as
> text. YMMV.
>

the problem is that this "decoupling" often does considerable damage to 
the compression, such as making the compressed data take around 3x to 4x 
more space...

granted, this works in many use cases, "well, all the users have fast 
internet connections...", but still isn't necessarily ideal (nor 
necessarily applicable to domains where it "actually matters").

for example, many "web" protocols just wouldn't work for many types of 
online gaming... the latency and performance would be unusably bad...


I had argued about it before: it wasn't that the other person thought 
SOAP could handle online gaming tasks, but that the person thought that 
the sorts of things commonly done by online gaming were impossible over 
the internet... (it can be done, just people can't design their 
protocols to work like SOAP).


it is roughly along the same lines of implementing a video codec by 
simply taking raw PPM forms of each video frame and running them through 
GZip.
it could technically work, but the bit-rate would suck...

(people could then despair as it requires around 100Mbps or so for to 
watch low-resolution web videos...).


(then I just had a thought of if someone thought it a good idea to make 
something like an XML-based analogue of PPM):
<xppm xmlns="...">
   <head>
      <image width="352" height="288" bits="8"/>
   </head>
   <body>
      <pixel red="134" green="220" blue="76" alpha="255"/>
      ...
   </body>
</xppm>

( then web-people jump on it, because they then-to can have a 
cameraphone quality photographic image require upwards of 200MB, and 
maybe deflate to somewhere around 40MB... )


a problem are with text+Deflate is that it can't really recognize or 
efficiently handle most of the data-types being transmitted (seeing them 
as raw text), and often a partial or full state-flush is involved for 
sending each message. this hurts because some things have to be sent for 
every message (such as the Huffman tables), and (in the full-flush 
case), any context from prior messages will be forgotten when sending 
the next message (all this may be unavoidable, say, if using UDP, but is 
a waste with TCP).


in my case, the contents of the message stream can be dumped textually, 
but mostly via printing out the S-Expressions. granted, yes, this isn't 
exactly "what goes over the wire".

the protocol then is mostly just a specialized compressed binary 
serialization of the data (dynamically-typed / Lisp-style lists).

it could be better, but could also be worse...


side note:
http://en.wikipedia.org/wiki/S-expression

http://en.wikipedia.org/wiki/Lisp_%28programming_language%29#Conses_and_lists


even lists were a compromise:
I was lazy, so I used lists as an intermediate step.

an alternative would have been having the code for generating deltas 
write into the bitstream directly, more like in Quake3 and Doom3, but 
this would be ugly IMO (and, not really necessary, as dailup is dead...).

so, I chose instead to generate lists as the intermediate step, and then 
encode these lists.

so, yeah, there is at least a layer of abstraction between the 
"high-level protocol" and the serialization (the serialization layer 
doesn't really know/care what it is sending, just that it is lists).

technically, it can send other types of data as well, so long as they 
are wrapped up in lists (and represented with the appropriate data-types).


or such...

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | comp.lang.c

csiph-web