Path: csiph.com!eternal-september.org!feeder.eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail From: Paul Rubin Newsgroups: comp.lang.forth Subject: Re: Moving JSON around. Date: Sun, 19 Feb 2017 01:38:15 -0800 Organization: A noiseless patient Spider Lines: 43 Message-ID: <877f4mmsfs.fsf@nightsong.com> References: <9520efbe-f7e6-46b9-a052-7c0b99b0a352@googlegroups.com> <87bmtymty0.fsf@nightsong.com> <7b0d720e-787a-4e81-93df-b386c9e7f487@googlegroups.com> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: mx02.eternal-september.org; posting-host="438961605d72ac5d3c11ac134f5a0e06"; logging-data="17755"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19H9c5cgvwYVS7ATVMSCMoG" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3 (gnu/linux) Cancel-Lock: sha1:KXHcb8bAoz6bX5EbAncgQdOZqvI= sha1:XFa2T/ACsMQ2l1kLwYGB2RddrtA= Xref: csiph.com comp.lang.forth:54998 Julian Fondren writes: > The thing about JSON, in my experience, is that it's all destined to > get read by JavaScript running in a webpage. I don't find that at all--it's a very widely used format now, showing up in all kinds of places, like REST APIs that are used by many sorts of clients. I'm constantly pulling json data from places and generally munching it with python scripts, though the docs are usually much smaller. Databases like MongoDB use Json as their communication format, and a more compact binary version internally. > So it can't really ever be much of a threat to server memory -- if it > were that big, the browser would be even less pleased by it. Heh, most of my servers are 128MB VPS's and browsers are monstrous compared to that. It's like the 1990s joke that computers kept getting faster and faster, so Microsoft kept making MS Word slower so you'd have to upgrade your Wintel hardware. Finally computers got so fast that Microsoft could no longer bog them down with Word while keeping a straight face. So now we have browsers eating more and more resources instead. People are running whole OS's in browsers now. Like if you go to Archive.org, you can run various classic computer games through JS emulation of the x86 machine code in the old MSDOS and 68k Macintosh binaries. > Although I'm sure that gigabytes of JSON exist, somewhere. I mess with terabyte XML documents sometimes, using a streaming parser. I guess those could be jsonized. The file you linked to is around 180MB. Its main object is a list of something like 200k smaller dictionaries. I'd probably write them each out as a separate document of 1MB or so, rather than 1 giant document. http://rapidjson.org/ has a streaming json parser in C++ that claims to be super fast. I might give it a try. I use Expat (a fast streaming C parser) for XML processing but it still takes some time on very large documents, so maybe converting them to JSON (or a binary format) isn't crazy. I wonder if a Forth 2000x recognizer could handle JSON. { } [ ] " and , would all be separate words that pushed items to and popped them from a parsing stack, calling user functions at each event.