Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #54993 > unrolled thread

Moving JSON around.

Back to article view | Back to comp.lang.forth

Contents

  Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-18 23:37 -0800
    Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-19 10:07 +0200
      Re: Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-19 00:35 -0800
        Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-19 12:12 +0200
          Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-19 13:17 +0200
            Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-19 13:33 +0200
              Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-19 16:25 +0200
                Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-20 11:57 +0200
                  Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-20 02:03 -0800
                    Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-20 12:56 +0200
                      Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-21 16:18 -0800
                        Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-22 06:39 +0200
                  Re: Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-24 22:36 -0800
                    Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-25 01:50 -0800
                    Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-25 20:02 +0200
                      Re: Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-25 23:13 -0800
                        Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-26 10:13 +0200
                        Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-26 13:01 +0200
                  Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-26 16:05 +0200
    Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-19 01:05 -0800
      Re: Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-19 01:17 -0800
        Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-19 01:38 -0800
          Re: Moving JSON around. franck.bensusan@gmail.com - 2017-02-19 16:08 -0800
            Re: Moving JSON around. Paul Rubin <no.email@nospam.invalid> - 2017-02-19 17:16 -0800
            Re: Moving JSON around. Julian Fondren <julian.fondren@gmail.com> - 2017-02-19 17:52 -0800
              Re: Moving JSON around. Ron Aaron <rambamist@gmail.com> - 2017-02-20 07:26 +0200
              Re: Moving JSON around. franck.bensusan@gmail.com - 2017-02-20 03:01 -0800

Page 2 of 2 — ← Prev page 1 [2]

#54997

On Sunday, February 19, 2017 at 3:05:45 AM UTC-6, Paul Rubin wrote:
> Julian Fondren <julian.fondren@gmail.com> writes:
> >   $ time perl -MJSON ...
> >   real      0m5.344s
> >   user      0m4.735s
> 
> Same file takes around 6 seconds with Python's json module on a Xeon
> E3-1230v3, and the Python image then uses around 1.3GB of ram.  I was
> surprised it parsed that fast.
> 
> But, slurping a document that size all at once is almost surely a bad
> idea.  I've never seen a streaming JSON parser (like a SAX XML parser)
> but a quick web search shows they exist.  That would be a natural
> approach to use in Forth.

The thing about JSON, in my experience, is that it's all destined to
get read by JavaScript running in a webpage. So it can't really ever
be much of a threat to server memory -- if it were that big, the
browser would be even less pleased by it. Most of the "clean the JSON
up a bit" step is stripping out what the browser isn't interested in,
to speed things up.

Although I'm sure that gigabytes of JSON exist, somewhere.

[toc] | [prev] | [next] | [standalone]

#54998

Julian Fondren <julian.fondren@gmail.com> writes:
> The thing about JSON, in my experience, is that it's all destined to
> get read by JavaScript running in a webpage. 

I don't find that at all--it's a very widely used format now, showing up
in all kinds of places, like REST APIs that are used by many sorts of
clients.  I'm constantly pulling json data from places and generally
munching it with python scripts, though the docs are usually much
smaller.  Databases like MongoDB use Json as their communication format,
and a more compact binary version internally.

> So it can't really ever be much of a threat to server memory -- if it
> were that big, the browser would be even less pleased by it.

Heh, most of my servers are 128MB VPS's and browsers are monstrous
compared to that.  It's like the 1990s joke that computers kept getting
faster and faster, so Microsoft kept making MS Word slower so you'd have
to upgrade your Wintel hardware.  Finally computers got so fast that
Microsoft could no longer bog them down with Word while keeping a
straight face.  So now we have browsers eating more and more resources
instead.  People are running whole OS's in browsers now.  Like if you go
to Archive.org, you can run various classic computer games through JS
emulation of the x86 machine code in the old MSDOS and 68k Macintosh
binaries.

> Although I'm sure that gigabytes of JSON exist, somewhere.

I mess with terabyte XML documents sometimes, using a streaming parser.
I guess those could be jsonized.  The file you linked to is around
180MB.  Its main object is a list of something like 200k smaller
dictionaries.  I'd probably write them each out as a separate document
of 1MB or so, rather than 1 giant document.

http://rapidjson.org/ has a streaming json parser in C++ that claims to
be super fast.  I might give it a try.

I use Expat (a fast streaming C parser) for XML processing but it still
takes some time on very large documents, so maybe converting them to
JSON (or a binary format) isn't crazy.

I wonder if a Forth 2000x recognizer could handle JSON.  { } [ ] " and ,
would all be separate words that pushed items to and popped them from
a parsing stack, calling user functions at each event.

[toc] | [prev] | [next] | [standalone]

#55012

Le dimanche 19 février 2017 10:38:16 UTC+1, Paul Rubin a écrit :
> I wonder if a Forth 2000x recognizer could handle JSON.  { } [ ] " and ,
> would all be separate words that pushed items to and popped them from
> a parsing stack, calling user functions at each event.

Hello,

Json objects (and arrays) could be handled directly by the outer 
interpreter with the data stack, but with changes of some words behavior : 

1) { and } words should be dedicated to Json objects and not locals.
{ creates a json frame on the control stack.
} creates a json object with Pairs on the data stack.
Locals could be handled by ( and ) and commentaries only handled by \

2) : checks if it is the beginning of a definition or just a json 
separator. If Json separator, do nothing.

3) [ and ] words are dedicated to arrays definition.
Entering/ending runtime mode is done by other words 
(<% and %> for instance).

4) , word is dedicated to array and Json separator.
For Json, it creates a Pair with the two objects on the stack.

5) true, false and null are a defined as constants.

6) The outer interpreter is hooked to handle some words without a need 
for a following space (or using a recogniser ? ) : { } [ ] and ,

7) Dot is used for Floats.

With those changes, the outer interpreter can parse Json objects and arrays 
as any other code and EVALUATE can evaluate strings containing those objects.

Is it worth it to make those changes and break the behavior of those
Forth words ? Can it be a possible evolution of the standard ? 
Or is it possible to do this with recognisers ? I don't know.

Oforth implements this behavoir, so its outer interpreter handles 
arrays and json without the need of a specific parser.

For instance : 
{ "aaa": 12, "bbb" : 13.2, "ccc" : [ 1, 2, 3 ], "ddd" : "eee" } .s
[1] (Json) {"aaa" : 12, "bbb" : 13.2, "ccc" : [1, 2, 3], "ddd" : "eee" }
ok
asString .s
[1] (String) {"aaa" : 12, "bbb" : 13.2, "ccc" : [1, 2, 3], "ddd" : "eee" }
ok
evaluate .s
[1] (Json) {"aaa" : 12, "bbb" : 13.2, "ccc" : [1, 2, 3], "ddd" : "eee" }
ok

: testCompile -- aJson 
   { "aaa": 12, "bbb" : 13.2, "ccc" : [ 1, 2, 3 ], "ddd" : "eee" } 
;

testCompile .s
[1] (Json) {"aaa" : 12, "bbb" : 13.2, "ccc" : [1, 2, 3], "ddd" : "eee" }


For the big JSON provided in this post, the outer interpreter failed because 
all elements are kept on the data stack before creating a Json (or an array).
Json (or array) are created only when ] or } is evaluated ie when
the number of items is available.

Into this json, a least one json (or array) had much more than 1000 
elements and this overflows the data stack.

I updated { and [ to create arrays and jsons at the beginning and
add each element when , or } or ] is evaluated.
 
This is much less performant as the number of elements is unknown at the
beginning and arrays will have be reallocated if needed (and here, it 
is needed a lot). But it works with any number of items.

Using Oforth version under development : 

: slurp	  	\ s -- sb   
   File new File.READ over open
   dup size swap read
;

: test
  "citylots.json" slurp evaluate ;

On my laptop (Core I7), it takes about 24 seconds for the outer interpreter to 
evaluate this string.

But I am not sure that handling declared arrays/jsons with more than 1000 
items worth the reduction of performances. So I don't think I will keep this 
update.

Franck

[toc] | [prev] | [next] | [standalone]

#55013

franck.bensusan@gmail.com writes:
> 1) { and } words should be dedicated to Json objects and not locals. ...

Yes the idea is you would temperorarily switch the outer interpreter to
use a JSON dictionary and recognizer, similar to how CODE words change
the dictionary:

    JSON 
       {"foo":[3,2,"bar"],....}
    END-JSON

or something like that.  END-JSON could be a do-nothing since the
dictionary would have to stay in place until the closing } .

[toc] | [prev] | [next] | [standalone]

#55014

On Sunday, February 19, 2017 at 8:25:52 AM UTC-6, Ron Aaron wrote:
> So I reran the tests on the same machine (this time the Mac -- ignore my
> previous comment about it being slower, it has a slower CPU than my
> Linux box...)
> 
> Anyway, the times (best of 5 runs) I recorded for parsing that huge JSON
> file were:
> 
>         time (s)
> 8th:     4.446
> perl:    5.050
> ruby:    6.106
> python: 10.327

Cool. With the perl/ruby/python progression like that, I wonder if the
differences aren't due to the time it takes to work with the
languages' respective data structures through their 'FFI'.

On Sunday, February 19, 2017 at 6:08:01 PM UTC-6, franck....@gmail.com wrote:
> 
> Is it worth it to make those changes and break the behavior of those
> Forth words ? Can it be a possible evolution of the standard ? 
> Or is it possible to do this with recognisers ? I don't know.

Or all of this behavior could be toggled on for the construction of a
single JSON object a world like JSON: 

My first inlination is to have such words parse ahead to see how they
should behave but a toggle would be less troublesome.

> 
> For the big JSON provided in this post, the outer interpreter failed because 
> all elements are kept on the data stack before creating a Json (or an array).
> Json (or array) are created only when ] or } is evaluated ie when
> the number of items is available.
> 
> Into this json, a least one json (or array) had much more than 1000 
> elements and this overflows the data stack.
> 
> I updated { and [ to create arrays and jsons at the beginning and
> add each element when , or } or ] is evaluated.
>  
> This is much less performant as the number of elements is unknown at the
> beginning and arrays will have be reallocated if needed (and here, it 
> is needed a lot). But it works with any number of items.
> 

For the kind of specific application that I've discussed, where a
program is fetching JSON that it knows about from specific services
and processing it and then passing it along for eventual display in a
browser, the programmer can know (or learn after a phonecall of the
middle of the night) that structures of at most some size are
required. So other options are

1. a way to tell Oforth to have stacks of at least some depth

2. a way to tell Oforth to prepare its JSON structures with a
temporary memory space of some size. Rather than reallocate, Oforth
can put everything first in this space and then allocate exactly the
size needed by the object after it's complete, and copy it out.

3. Or some special behavior for when JSON (and only JSON--using your
evaluator would permit someone to send you Oforth code to execute,
no?) is parsed from a string.

> 
> But I am not sure that handling declared arrays/jsons with more than 1000 
> items worth the reduction of performances. So I don't think I will keep this 
> update.
> 
> Franck

I'm not dealing with hundreds of megabytes of JSON either, but I'm
definitely dealing very frequently with arrays of more than a 1000
items.


But 8th and Oforth both have JSON literals, huh. This reminds me of
the era where all of the Forth-like languages had quotations.

[toc] | [prev] | [next] | [standalone]

#55015

On 20/02/2017 3:52, Julian Fondren wrote:
> On Sunday, February 19, 2017 at 8:25:52 AM UTC-6, Ron Aaron wrote:
>> So I reran the tests on the same machine (this time the Mac -- ignore my
>> previous comment about it being slower, it has a slower CPU than my
>> Linux box...)
>>
>> Anyway, the times (best of 5 runs) I recorded for parsing that huge JSON
>> file were:
>>
>>         time (s)
>> 8th:     4.446
>> perl:    5.050
>> ruby:    6.106
>> python: 10.327
> 
> Cool. With the perl/ruby/python progression like that, I wonder if the
> differences aren't due to the time it takes to work with the
> languages' respective data structures through their 'FFI'.

I don't know.  8th has a native JSON parser as part of the
outer-interpreter, so I'm guessing that may help.

> I'm not dealing with hundreds of megabytes of JSON either, but I'm
> definitely dealing very frequently with arrays of more than a 1000
> items.

Yeah, I don't use huge JSON myself, preferring to use a SQL database (or
some other file-based structure) for such large amounts of data.  It is
nice to know 8th can handle it if needed, though.

And it may be required for persistence, e.g. if one had a large array
and wished to save it as a JSON array (to send on the wire, or store for
later or something).  So it's not as silly an example as it first seems.

> But 8th and Oforth both have JSON literals, huh. This reminds me of
> the era where all of the Forth-like languages had quotations.

8th has always had "JSON literals" (not exactly), because I wanted a
simple and standard way to exchange and declare data.  Since JSON is so
widely used, and is a readable format, I decided to use it.  It has the
advantage as well of being easy to parse.

Why "not exactly"?  Because the JSON is parsed into native 8th data
types (array, map etc.).  So in the source code you write JSON, but at
interpret time that gets converted into a map or whatever.

[toc] | [prev] | [next] | [standalone]

#55020

Le lundi 20 février 2017 02:52:44 UTC+1, Julian Fondren a écrit :
> On Sunday, February 19, 2017 at 8:25:52 AM UTC-6, Ron Aaron wrote:
> > So I reran the tests on the same machine (this time the Mac -- ignore my
> > previous comment about it being slower, it has a slower CPU than my
> > Linux box...)
> > 
> > Anyway, the times (best of 5 runs) I recorded for parsing that huge JSON
> > file were:
> > 
> >         time (s)
> > 8th:     4.446
> > perl:    5.050
> > ruby:    6.106
> > python: 10.327
> 
> Cool. With the perl/ruby/python progression like that, I wonder if the
> differences aren't due to the time it takes to work with the
> languages' respective data structures through their 'FFI'.
> 
> On Sunday, February 19, 2017 at 6:08:01 PM UTC-6, franck....@gmail.com wrote:
> > 
> > Is it worth it to make those changes and break the behavior of those
> > Forth words ? Can it be a possible evolution of the standard ? 
> > Or is it possible to do this with recognisers ? I don't know.
> 
> Or all of this behavior could be toggled on for the construction of a
> single JSON object a world like JSON: 
> 
> My first inlination is to have such words parse ahead to see how they
> should behave but a toggle would be less troublesome.

Probably, yes. This was the most difficult choice to make for oforth 
design because it breaks some core words behavior.

Most word differences between oforth and forth come from here : 
handling arrays and json directly by the outer interpreter.

Oforth words and numbers are fully compatible with the json syntax.

For the toggle, this should be possible, but I don't know if this does 
not mean, in the end, writing a json parser (handling floats, chars, ...).

btw, I forgot ' that should handle characters (Oforth tick is # ).


> > 
> > For the big JSON provided in this post, the outer interpreter failed because 
> > all elements are kept on the data stack before creating a Json (or an array).
> > Json (or array) are created only when ] or } is evaluated ie when
> > the number of items is available.
> > 
> > Into this json, a least one json (or array) had much more than 1000 
> > elements and this overflows the data stack.
> > 
> > I updated { and [ to create arrays and jsons at the beginning and
> > add each element when , or } or ] is evaluated.
> >  
> > This is much less performant as the number of elements is unknown at the
> > beginning and arrays will have be reallocated if needed (and here, it 
> > is needed a lot). But it works with any number of items.
> > 
> 
> For the kind of specific application that I've discussed, where a
> program is fetching JSON that it knows about from specific services
> and processing it and then passing it along for eventual display in a
> browser, the programmer can know (or learn after a phonecall of the
> middle of the night) that structures of at most some size are
> required. So other options are
> 
> 1. a way to tell Oforth to have stacks of at least some depth

This is available with --S command line option, but you have to know,
at start-up, that you will deal with big arrays or jsons. The data stack
does not realloacte dynamically after Oforth is started.

btw, I tried -S5000 and -S10000 and this amount of stack is not sufficient 
to handle this json. So there is probably, somewhere into those 180 Mo, 
an array or json with more than 10000 items.

And, above -S10000, I found a bug in Oforth with a stack allocated 
bigger than 64Ko :).


> 2. a way to tell Oforth to prepare its JSON structures with a
> temporary memory space of some size. Rather than reallocate, Oforth
> can put everything first in this space and then allocate exactly the
> size needed by the object after it's complete, and copy it out.

To handle this kind of json, the temporary area would be huge. Also, this
would be declared at startup so you have to know that you will handle huge
json before launching. I will think about it.


> 3. Or some special behavior for when JSON (and only JSON--using your
> evaluator would permit someone to send you Oforth code to execute,
> no?) is parsed from a string.

There are 2 words declared for a string : 
- evaluate, that is used by the outer interpreter and handle code too.
- perform, that restrict the use of words evaluated into the string.

With #perform, executing Oforth code other that { } [ ] , ... is not
permitted and will raise an exception.

perform allows the word # (the tick in Ofort), so you can send a json like : 

{ "function" : #rot , "params" : [ 1 , 2, 3 ] }

This will not execute code, but, if the receiver knows #rot, this will refer 
to its own word. Otherwise, an exception is raised.

#perform for strings was available in previous Oforth versions, but 
not in the last one (during the phase where the outer interpreter is beeing
rewritten in Oforth). It will be available again in the next version.


> > 
> > But I am not sure that handling declared arrays/jsons with more than 1000 
> > items worth the reduction of performances. So I don't think I will keep this 
> > update.
> > 
> > Franck
> 
> I'm not dealing with hundreds of megabytes of JSON either, but I'm
> definitely dealing very frequently with arrays of more than a 1000
> items.
> 

You mean a declared Array with more than 1000 items like in this json ? 
Of course you can create an array with more than 1000 items in Oforth, 
but not using the [ , ... , ] syntax. For instance : 

#sqrt 1000000 seq map

If you mean an explicitely declared array (using [ , ... , ] syntax) with 
more than 1000 items, then perhap's I should keep this update (or find a way 
to handle this kind of object with a temporay area like you suggested).

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.forth

csiph-web