Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #29240

Re: Serialization library, request for feedback

Show all headers | View raw

On 12/14/2012 9:22 AM, Ben Bacarisse wrote:
> Rui Maciel <rui.maciel@gmail.com> writes:
>
>> Ben Bacarisse wrote:
>>
>>> I don't think JSON is a good fit for the OP's task.  There are lots of C
>>> data structures that have no obvious JSON representation at all.
>>
>> What data structures do you have in mind?
>
> My ring_buffer example is one.  In fact, any structure where pointers are
> "shared" would seem to be a bad fit with JSON.
>
> Maybe "no obvious JSON representation" is too strong because you can
> probably always map pointers so some sort of index or string label, but
> it's not a natural fit.
>

yeah, these are general problems with data serialization, namely that 
there is no good way to deal with "everything" within any reasonable 
level of complexity.

it is easier then to find a reasonable subset of things that can be 
serialized via a given mechanism, and use this.


more careful design of a serialization interface would be needed, for 
example, to be able to deal with cyclic data structures, and more 
in-turn to deal with preserving the identity of serialized objects (1).

1: if obja==objb, will this be true or false after the data is saved, 
and re-loaded?...


some options will not preserve any identity, so if the same object is 
reached via multiple paths, each may become separate objects when the 
same image is read-in later.

other options may in-turn create false identity, where pretty much any 
object which "looks the same" will have the same identity, regardless of 
whether or not it was the same object prior to serialization.

identity becomes a particularly ugly issue if it needs to be dealt with 
across the same object being saved across multiple images, or if network 
gets involved (currently, I just don't deal with this problem). (this 
part gets much more important if dealing with trying to serialize things 
like program-state, say, where a "running program image" may be saved to 
disk, and reloaded later, and lots of little "subtle details" will need 
to be preserved, or else things will often go "critically wrong" when 
the program state gets reloaded later on).


or, at least, short of much simpler but very ugly solutions, like doing 
a raw memory-dump of the program's heap (I have done this before, but it 
"doesn't really scale well"), but more practical solutions to general 
program heap serialization are overly complicated, and it is easier just 
to deal with smaller scale serialization and reloading (such as 
"serialize this particular data structure").

a basic idea for how the heap-dump serialization works is:
imagine, if you will, that on startup the application uses a 
memory-mapped file as the main VM heap, and uses special magic to 
preserve any "roots" (generally global variables). on exiting and 
reloading, any objects in this heap can be preserved, with the "roots" 
automatically restored to their prior values.

yeah, long ago, one of my prior VMs actually did this, although the heap 
wasn't a single giant file, but rather a bunch of several-MB "heap chunks".

I later generally abandoned this strategy, as it had some particularly 
nasty technical issues (every exit/reload cycle leaked memory that could 
not be reclaimed, nasty "bit rot" problems, structure-layout problems, ...).

it is IMO better to actually just try to "serialize the contents of the 
data", rather than just "stupidly dump the memory to the HDD and try to 
reload it later".



but, in general, dealing with these sorts of issues requires more 
careful attention to detail, and sadly, this isn't really something JSON 
does very well.

not that this makes JSON useless though.


more generally though, I just use plain files for data storage, as they 
just work so-much better and more reliably.


one possible (partial) solution here for cyclic structures/... though is 
an "extended and non-standard" JSON variant, say, where everything is 
indexed.

#1 = { name: "foo", link: #2 };
#2 = { name: "bar", link: #3 };
#3 = { name: "baz", link: #4, first: #1 };
...
{ name: "someobject", listHead: #1 }

but, dealing effectively with a more comprehensive type-system would 
require further somewhat extending the syntax, say, adding some 
mechanism to identify up-front the type of object being serialized, ...


this is partly what my "somewhat hacked-up S-Expressions" do, even if 
they are a little out of place (long ago, I had a Scheme implementation, 
and even now, vestiges of it still remain, floating around in my 
codebase...).


or such...

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 13:46 -0800
  Re: Serialization library, request for feedback Keith Thompson <kst-u@mib.org> - 2012-12-13 13:57 -0800
    Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 14:30 -0800
      Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-14 19:38 +1300
        Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 03:47 -0600
          Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-14 23:23 +1300
            Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 11:29 -0600
          Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 03:51 -0800
            Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 12:02 -0600
        Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 12:18 +0000
          Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 14:27 +0000
            Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 15:22 +0000
              Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 13:00 -0600
                Re: Serialization library, request for feedback Greg Martin <greg@softsprocket.com> - 2012-12-14 11:58 -0800
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 15:08 -0600
                Re: Serialization library, request for feedback William Ahern <william@wilbur.25thandClement.com> - 2012-12-14 20:07 -0800
                Re: Serialization library, request for feedback Nick Keighley <nick_keighley_nospam@hotmail.com> - 2012-12-16 03:49 -0800
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-16 13:17 -0600
                Re: Serialization library, request for feedback Greg Martin <greg@softsprocket.com> - 2012-12-17 09:07 -0800
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-17 13:25 -0600
              Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 09:23 +1300
                Re: Serialization library, request for feedback James Kuyper <jameskuyper@verizon.net> - 2012-12-14 15:31 -0500
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 15:26 -0600
                Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 12:25 +1300
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 18:41 -0600
                Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 14:56 +1300
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 23:36 -0600
                Re: Serialization library, request for feedback Ian Collins <ian-news@hotmail.com> - 2012-12-15 20:13 +1300
                Re: Serialization library, request for feedback Shao Miller <sha0.miller@gmail.com> - 2012-12-14 21:42 -0500
                Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 23:47 -0600
              Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 21:12 +0000
                Re: Serialization library, request for feedback Johann Klammer <klammerj@NOSPAM.a1.net> - 2012-12-15 05:06 +0100
    Re: Serialization library, request for feedback Willem <willem@turtle.stack.nl> - 2012-12-14 22:20 +0000
  Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-13 23:11 +0000
    Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-13 15:57 -0800
      Re: Serialization library, request for feedback Ben Bacarisse <ben.usenet@bsb.me.uk> - 2012-12-14 02:53 +0000
        Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 04:25 -0800
      Re: Serialization library, request for feedback Bart van Ingen Schenau <bart@ingen.ddns.info.invalid> - 2012-12-14 08:04 +0000
  Re: Serialization library, request for feedback Rui Maciel <rui.maciel@gmail.com> - 2012-12-14 14:25 +0000
    Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 13:58 -0600
      Re: Serialization library, request for feedback "BartC" <bc@freeuk.com> - 2012-12-14 20:44 +0000
        Re: Serialization library, request for feedback BGB <cr88192@hotmail.com> - 2012-12-14 16:58 -0600
        Re: Serialization library, request for feedback Ulf Åström <ulf.astrom@gmail.com> - 2012-12-14 17:39 -0800
    Re: Serialization library, request for feedback Malcolm McLean <malcolm.mclean5@btinternet.com> - 2012-12-14 14:19 -0800

csiph-web