Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #54993 > unrolled thread

Moving JSON around.

Back to article view | Back to comp.lang.forth

Contents

  Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-18 23:37 -0800
    Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-19 10:07 +0200
      Re: Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-19 00:35 -0800
        Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-19 12:12 +0200
          Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-19 13:17 +0200
            Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-19 13:33 +0200
              Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-19 16:25 +0200
                Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-20 11:57 +0200
                  Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-20 02:03 -0800
                    Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-20 12:56 +0200
                      Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-21 16:18 -0800
                        Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-22 06:39 +0200
                  Re: Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-24 22:36 -0800
                    Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-25 01:50 -0800
                    Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-25 20:02 +0200
                      Re: Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-25 23:13 -0800
                        Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-26 10:13 +0200
                        Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-26 13:01 +0200
                  Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-26 16:05 +0200
    Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-19 01:05 -0800
      Re: Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-19 01:17 -0800
        Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-19 01:38 -0800
          Re: Moving JSON around. franck.bensusan@gmail.com - 2017-02-19 16:08 -0800
            Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-19 17:16 -0800
            Re: Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-19 17:52 -0800
              Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-20 07:26 +0200
              Re: Moving JSON around. franck.bensusan@gmail.com - 2017-02-20 03:01 -0800

Page 1 of 2  [1] 2  Next page →

#54993 — Moving JSON around.

Hello clf,

A lot of my recent work is with JavaScript on the frontend and, on the
backend, various scripting languages. Mostly Perl. Mostly the backend
does very little work:

  1. PUT a JSON object to a server
  2. receive JSON object from server, in reply
  3. clean the object up a bit, maybe aggregate it with other fetched
     data, and write it to a file
  4. (later:) a webpage's javascript fetches the file

Why not do this with Forth, I wondered.

The Forth Foundation Library includes jis.fs, JSON Input Stream. Give it a
string and it'll produce JSON elements from the string, one at a time. For
example, given {a:1,b:2} jis will yield

  jis.start-object
  s" a" jis.name
  1 jis.number
  s" b" jis.name
  2 jis.number
  jis.end-object
  jis.done

Right away this is not as convenient, but it makes sense: FFL is
leaving it to you to repackage the data into your own data structures
if you like.

So how well does it perform?

  include ffl/jis.fs
  jis-new constant jis
  
  : slurp ( c-addr u -- c-addr u )
    r/o open-file throw dup file-size throw abort" too big"
    dup allocate throw locals| a n fid |
    a n fid read-file throw
    n <> abort" failed to read entire file"
    fid close-file throw  a n ;
  
  s" dat/xxx.json" slurp jis jis-set-string
  
  : x ( -- )
    begin 
      jis jis-read if depth 0 ?do drop loop
      else bye then
    again ;
   x

On a few lines of hand-written JSON (dat/small.json), the performance
isn't worth mentioning. How about on 181MB of JSON (dat/big.json),
from https://github.com/zemirco/sf-city-lots-json ?

Results:

  gforth: float stack overflow
  iforth: hardware error in a float operator
      sf: less useful error message
     vfx: it's been 170 minutes, running at 100% CPU, and it's still
          going

Meanwhile:

  $ time perl -MJSON -e 'undef $/; my $x = decode_json(<>); print ref($x)' < dat/big.json 
  HASH
  real	0m5.344s
  user	0m4.735s
  sys	0m0.590s

This is an unusually severe loss for Forth.


Although there's clearly not been a lot of effort into a JSON library
for Forth, there has been some effort into JSON libraries for C.

One such library is 'json-c'. A minimal use of it:

  library libjson-c.so.2
  function: json_tokener_parse ( a -- a )
  function: json_object_to_json_string ( obj -- z )
  
  : zslurp ( c-addr u -- z )
    r/o open-file throw dup file-size throw abort" too big"
    dup 1+ allocate throw locals| a n fid |
    a n fid read-file throw
    n <> abort" failed to read entire file"
    fid close-file throw
    0 a n + 1+ c!
    a ;
  
  s" dat/big.json" zslurp json_tokener_parse json_object_to_json_string zcount type bye

...is pretty similar to Perl's performance, when it's also asked to
reproduce this object:

  $ /usr/bin/time perl -MJSON -e 'undef $/; my $x = decode_json(<>); print encode_json($x)' < dat/big.json > /dev/null
  15.55user 0.65system 0:16.24elapsed 99%CPU (0avgtext+0avgdata 1163624maxresident)k
  0inputs+0outputs (0major+336279minor)pagefaults 0swaps
  
  $ /usr/bin/time sf json3.fs > /dev/null
  21.39user 0.90system 0:22.35elapsed 99%CPU (0avgtext+0avgdata 1853440maxresident)k
  0inputs+16outputs (0major+462943minor)pagefaults 0swaps

The library's not that fun to use. Here's an early attempt:

  \ SWOOP object definitions omitted--they're just copies of the data
  \ structures in json-c/linkhash.h

  : .object ( json -- )
    ." {" dup json_object_get_object USING json-object json.head @ 
    begin dup while ( json entry )
      2dup USING json-entry entry.key @ json_object_object_get
      json_object_get_string zcount .\" \"" cr type .\" \": "
      ( json entry )
      2dup USING json-entry entry.value @ json_object_object_get
      json_object_get_string zcount type ." |"
      ( json entry )
      USING json-entry entry.next @
    repeat 2drop ;

Which given { "hello": "there", "hiya": "world" } , outputs:

  {"there": |"world": |

It's not formatted correctly because I still haven't figured out where
the object keys are hiding. You'd think the key would be under ->key
and the corresponding value under ->value, but that's not the case.

So I think I'll keep using Perl for now. Before writing much more of a
JSON library (wrapper) for Forth, I'd like to shop around and see how
the C libraries compare.

Cheers,
Julian

[toc] | [next] | [standalone]

#54994

On 02/19/2017 09:37, Julian Fondren wrote:
> Hello clf,
> 
> A lot of my recent work is with JavaScript on the frontend and, on the
> backend, various scripting languages. Mostly Perl. Mostly the backend
> does very little work:
> 
>   1. PUT a JSON object to a server
>   2. receive JSON object from server, in reply
>   3. clean the object up a bit, maybe aggregate it with other fetched
>      data, and write it to a file
>   4. (later:) a webpage's javascript fetches the file
> 
> Why not do this with Forth, I wondered.

8th has native JSON support, so doing this is almost trivial.  In fact,
I just wrote a CGI script to handle logging using JSON, for a client's
application.

[toc] | [prev] | [next] | [standalone]

#54995

On Sunday, February 19, 2017 at 2:07:46 AM UTC-6, Ron Aaron wrote:
> On 02/19/2017 09:37, Julian Fondren wrote:
> > Hello clf,
> > 
> > A lot of my recent work is with JavaScript on the frontend and, on the
> > backend, various scripting languages. Mostly Perl. Mostly the backend
> > does very little work:
> > 
> >   1. PUT a JSON object to a server
> >   2. receive JSON object from server, in reply
> >   3. clean the object up a bit, maybe aggregate it with other fetched
> >      data, and write it to a file
> >   4. (later:) a webpage's javascript fetches the file
> > 
> > Why not do this with Forth, I wondered.
> 
> 8th has native JSON support, so doing this is almost trivial.  In fact,
> I just wrote a CGI script to handle logging using JSON, for a client's
> application.

Aye, I saw your post about POSTing a JSON request to a URL. That's
much like what I've been doing.

How does 8th compare to Perl on parsing and dumping that huge JSON file?

[toc] | [prev] | [next] | [standalone]

#54999

On 19/02/2017 10:35, Julian Fondren wrote:
> On Sunday, February 19, 2017 at 2:07:46 AM UTC-6, Ron Aaron wrote:
>> On 02/19/2017 09:37, Julian Fondren wrote:
>>> Hello clf,
>>>
>>> A lot of my recent work is with JavaScript on the frontend and, on the
>>> backend, various scripting languages. Mostly Perl. Mostly the backend
>>> does very little work:
>>>
>>>   1. PUT a JSON object to a server
>>>   2. receive JSON object from server, in reply
>>>   3. clean the object up a bit, maybe aggregate it with other fetched
>>>      data, and write it to a file
>>>   4. (later:) a webpage's javascript fetches the file
>>>
>>> Why not do this with Forth, I wondered.
>>
>> 8th has native JSON support, so doing this is almost trivial.  In fact,
>> I just wrote a CGI script to handle logging using JSON, for a client's
>> application.
> 
> Aye, I saw your post about POSTing a JSON request to a URL. That's
> much like what I've been doing.
> 
> How does 8th compare to Perl on parsing and dumping that huge JSON file?

That depends :)

The naive "just do it" approach works, but it takes a whopping 143.6
seconds.  The reason for that is that 8th is made to use less memory to
begin with, so it keeps allocating more pools for data to be stored.

The smart "I know I'll use lots of data" approach takes 3.78 seconds.
What's a good Perl script to compare with?

[toc] | [prev] | [next] | [standalone]

#55003

On 02/19/2017 12:12, Ron Aaron wrote:

> What's a good Perl script to compare with?

OK, so on second glance I noticed your perl script.  But for me, it
doesn't seem to ever finish.  11 minutes and counting... perl 5.18.2,
not an ancient version.

[toc] | [prev] | [next] | [standalone]

#55004

More complete answer:

Using this perl code:

  use JSON::Parse 'parse_json';
  undef $/;
  my $x = parse_json(<>);
  print ref($x);
  exit(0);

time reported user 2.90, system 0.38, max resident 1196704

Using this 8th code: (the stuff at the beginning is to set the sizes of
the pools needed):

  ns:n 7900000 pool-size
  ns:s 5200000 pool-size
  ns:a 3100000 pool-size
  ns:m 620000 pool-size

  "citylots.json" f:slurp json>
  bye

time reported user 2.23, system 0.32, max resident 2088996

Interestingly, this is on my slower Core i5 Linux system; it's faster at
this than the Core i7 MacBook Pro ...

[toc] | [prev] | [next] | [standalone]

#55008

So I reran the tests on the same machine (this time the Mac -- ignore my
previous comment about it being slower, it has a slower CPU than my
Linux box...)

Anyway, the times (best of 5 runs) I recorded for parsing that huge JSON
file were:

        time (s)
8th:     4.446
perl:    5.050
ruby:    6.106
python: 10.327

[toc] | [prev] | [next] | [standalone]

#55017

On 19/02/2017 16:25, Ron Aaron wrote:
> So I reran the tests on the same machine (this time the Mac -- ignore my
> previous comment about it being slower, it has a slower CPU than my
> Linux box...)
> 
> Anyway, the times (best of 5 runs) I recorded for parsing that huge JSON
> file were:
> 
>         time (s)
> 8th:     4.446
> perl:    5.050
> ruby:    6.106
> python: 10.327

One more:
  node:    2.449

Not surprising, particularly.

[toc] | [prev] | [next] | [standalone]

#55018

Ron Aaron <rambamist@gmail.com> writes:
> One more:
>   node:    2.449
> Not surprising, particularly.

C++ rapidjson (gcc 6.3, gcc -O3, e3-1230v3): 0.536 sec

Process size: 273MB after parsing from file stream (i.e.
not reading the file into memory first).

[toc] | [prev] | [next] | [standalone]

#55019

On 02/20/2017 12:03, Paul Rubin wrote:
> Ron Aaron <rambamist@gmail.com> writes:
>> One more:
>>   node:    2.449
>> Not surprising, particularly.
> 
> C++ rapidjson (gcc 6.3, gcc -O3, e3-1230v3): 0.536 sec
> 
> Process size: 273MB after parsing from file stream (i.e.
> not reading the file into memory first).

That's interesting.  I've had a very bad time using rapidjson in a
customer's project, as it unexpectedly lets go of things, and throws
asserts which seem incorrect.  I'm switching (as we speak) to a
different library.

[toc] | [prev] | [next] | [standalone]

#55040

Ron Aaron <rambamist@gmail.com> writes:
>> C++ rapidjson (gcc 6.3, gcc -O3, e3-1230v3): 0.536 sec
> That's interesting.  I've had a very bad time using rapidjson in a
> customer's project, as it unexpectedly lets go of things, and throws
> asserts which seem incorrect.  I'm switching (as we speak) to a
> different library.

Thanks, that's good to know.  I'd never used rapidjson before and it
worked ok for that test but idk any more about it.  Can I ask what
you're switching to?  C++ doesn't seem like the greatest environment for
munging tons of json data anyway, of course.

[toc] | [prev] | [next] | [standalone]

#55042

On 22/02/2017 2:18, Paul Rubin wrote:
> Ron Aaron <rambamist@gmail.com> writes:
>> I'm switching (as we speak) to a
>> different library.
> 
> Thanks, that's good to know.  I'd never used rapidjson before and it
> worked ok for that test but idk any more about it.  Can I ask what
> you're switching to?  C++ doesn't seem like the greatest environment for
> munging tons of json data anyway, of course.

I found "JSON for modern C++" here : https://github.com/nlohmann/json

It is easier to work with than rapidjson, but it throws an exception if
a key isn't present... so I had to make a wrapper for it :/

[toc] | [prev] | [next] | [standalone]

#55089

On Monday, February 20, 2017 at 3:57:45 AM UTC-6, Ron Aaron wrote:
> On 19/02/2017 16:25, Ron Aaron wrote:
> > So I reran the tests on the same machine (this time the Mac -- ignore my
> > previous comment about it being slower, it has a slower CPU than my
> > Linux box...)
> > 
> > Anyway, the times (best of 5 runs) I recorded for parsing that huge JSON
> > file were:
> > 
> >         time (s)
> > 8th:     4.446
> > perl:    5.050
> > ruby:    6.106
> > python: 10.327
> 
> One more:
>   node:    2.449
> 
> Not surprising, particularly.

This time, just parsing a flat array of 24 million (+1) numbers and
printing the length of the array:

node     3.189s
perl     3.995s
json.fs  13.916s

Early, early code. Since there was talk about streaming parsers, I
made it one - it's an FSM that considers one character at a time:

  : decode-json ( c-addr u -- )
    json-reset-state
    bounds ?do i c@ decode-json-char loop ;

One interesting thing that came out of this is that, although JSON has
heterogeneous containers, you can still notice that you're looking at a
homogeneous container and produce a specialized representation with a
different tag. Let's say that you push container elements onto a stack
and then actually build the container once its complete. Type
transitions for the container might look like:

  | processed | container type |
  |-----------+----------------|
  | [         | empty array    |
  | 1         | int array      |
  | 2         | int array      |
  | "oh no"   | mixed array    |
  | 3         | mixed array    |
  | ]         | mixed array    |

With a stack for nested containers, of course.

In the similar manner you might maintain a type for the entire tree.
Although the only benefit of whole-tree typing that comes to mind is
working with 2d arrays of numbers, I think this *kind* of thinking is
part of how doing such things in Forth might result in an actual win
vs. other languages, rather than parity or (at the moment) defeat.

[toc] | [prev] | [next] | [standalone]

#55094

Julian Fondren <julian.fondren@gmail.com> writes:
> different tag. Let's say that you push container elements onto a stack

If you mean the normal Forth stacks, I don't think that's a usable
approach for large Json documents.  Some of the packages you tested
apparently tried that, and got stack overflows.

> In the similar manner you might maintain a type for the entire tree.
> Although the only benefit of whole-tree typing that comes to mind is
> working with 2d arrays of numbers

There's a scheme (no pun intended) out there for supplying XML-like
schemas to Json documents.  I think I saw something about it on
rapidjson.org.  That would allow assigning types to the whole tree,
individual records in the tree, etc.

[toc] | [prev] | [next] | [standalone]

#55101

On 25/02/2017 8:36, Julian Fondren wrote:
> On Monday, February 20, 2017 at 3:57:45 AM UTC-6, Ron Aaron wrote:
>> On 19/02/2017 16:25, Ron Aaron wrote:
>>> So I reran the tests on the same machine (this time the Mac -- ignore my
>>> previous comment about it being slower, it has a slower CPU than my
>>> Linux box...)
>>>
>>> Anyway, the times (best of 5 runs) I recorded for parsing that huge JSON
>>> file were:
>>>
>>>         time (s)
>>> 8th:     4.446
>>> perl:    5.050
>>> ruby:    6.106
>>> python: 10.327
>>
>> One more:
>>   node:    2.449
>>
>> Not surprising, particularly.
> 
> This time, just parsing a flat array of 24 million (+1) numbers and
> printing the length of the array:
> 
> node     3.189s
> perl     3.995s
> json.fs  13.916s
> 
> Early, early code. Since there was talk about streaming parsers, I
> made it one - it's an FSM that considers one character at a time:

Are you storing the data or just counting?  What's your perl or node code?

> One interesting thing that came out of this is that, although JSON has
> heterogeneous containers,

8th's containers are also heterogeneous.

[toc] | [prev] | [next] | [standalone]

#55107

On Saturday, February 25, 2017 at 12:02:32 PM UTC-6, Ron Aaron wrote:
> On 25/02/2017 8:36, Julian Fondren wrote:
> > 
> > This time, just parsing a flat array of 24 million (+1) numbers and
> > printing the length of the array:
> > 
> > node     3.189s
> > perl     3.995s
> > json.fs  13.916s
> > 
> > Early, early code. Since there was talk about streaming parsers, I
> > made it one - it's an FSM that considers one character at a time:
> 
> Are you storing the data or just counting?  What's your perl or node code?

  time node -e 'console.log(JSON.parse(require("fs").readFileSync("bigarray.json", "utf8")).length)'

  time perl -MJSON -e 'undef $/; print scalar(@{decode_json(<>)})' < bigarray.json 

The Forth equivalent to that is presently:

 s" bigarray.json" slurp decode-json objects stack> cell+ cr ?

I don't know of any way, in Perl or node, to count the data *without* storing
it. All three of these are building an object and then checking on its size.

If I modify the Forth to count the contents of the array without storing them or
building the array, the runtime goes down to 2s. This was a useful test: it's
very clear that storing data is the slowest part by far. I'd assumed the parser
would be -- it's using CASE ENDCASE after all :)

2s is also about how much time it takes to build the 24 million (+1) array after
the parser reaches the end of it - when I don't build an intarray instead.

140ms though is all that's needed by a version that throws an error if an object
has more than 10000 elements (this time considering only the cost of slurping
the file and parsing it). Maybe that would be useful in something like a web
application firewall, or like one of those servers that rejects bad SQL before
asking MySQL to handle. A situation where you have legacy or third-party code
that blows up on malicious JSON and it's too hard to fix the software itself.

[toc] | [prev] | [next] | [standalone]

#55108

On 02/26/2017 09:13, Julian Fondren wrote:

> I don't know of any way, in Perl or node, to count the data *without* storing
> it. All three of these are building an object and then checking on its size.

OK.  My test w/ 8th took about 3.4 seconds (vs node 1.7 on the same data).

> 2s is also about how much time it takes to build the 24 million (+1) array after
> the parser reaches the end of it - when I don't build an intarray instead.

Right.  In 8th, the arrays auto-expand.  But they do so in fixed
"quanta", so I guess that's the real problem... hold on, let me test
that theory...

Nope.  That reallocation is not the bottleneck for 8th.  Instead, it's
the number parsing (according to 'perf').  Really need to make that
faster...

[toc] | [prev] | [next] | [standalone]

#55112

I will just offer you a word of thanks for pointing out an area for 8th
to improve.

The (upcoming) version has a (significantly) faster parser :)

[toc] | [prev] | [next] | [standalone]

#55116

On 20/02/2017 11:57, Ron Aaron wrote:
> 
> 
> On 19/02/2017 16:25, Ron Aaron wrote:
>> So I reran the tests on the same machine (this time the Mac -- ignore my
>> previous comment about it being slower, it has a slower CPU than my
>> Linux box...)
>>
>> Anyway, the times (best of 5 runs) I recorded for parsing that huge JSON
>> file were:
>>
>>         time (s)
>> 8th:     4.446
>> perl:    5.050
>> ruby:    6.106
>> python: 10.327
> 
> One more:
>   node:    2.449
> 
> Not surprising, particularly.

After optimizations to 8th's parser, the new, improved time for 8th is
3.388.  Much better ...

[toc] | [prev] | [next] | [standalone]

#54996

Julian Fondren <julian.fondren@gmail.com> writes:
>   $ time perl -MJSON ...
>   real	0m5.344s
>   user	0m4.735s

Same file takes around 6 seconds with Python's json module on a Xeon
E3-1230v3, and the Python image then uses around 1.3GB of ram.  I was
surprised it parsed that fast.

But, slurping a document that size all at once is almost surely a bad
idea.  I've never seen a streaming JSON parser (like a SAX XML parser)
but a quick web search shows they exist.  That would be a natural
approach to use in Forth.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.forth

csiph-web