Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #143872 > unrolled thread

Static JSON parser for embedded

Back to article view | Back to comp.lang.c

Contents

  Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 09:20 +0200
    Re: Static JSON parser for embedded Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-05-17 08:48 +0000
      Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 11:06 +0200
        Re: Static JSON parser for embedded Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-05-17 11:22 +0000
          Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 04:40 -0700
            Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 14:10 +0200
              Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 05:18 -0700
                Re: Static JSON parser for embedded Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-05-18 18:18 +0000
          Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 14:10 +0200
            Re: Static JSON parser for embedded Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-05-18 18:22 +0000
    Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 11:39 +0100
      Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 13:19 +0200
        Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 05:09 -0700
          Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 05:15 -0700
        Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 15:06 +0100
          Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 16:37 +0200
            Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 17:56 +0100
              Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 10:21 -0700
                Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 18:43 +0100
                  Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 10:58 -0700
                  Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 11:16 -0700
                    Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 19:43 +0100
                      Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 11:57 -0700
                        Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 20:11 +0100
                          Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 12:22 -0700
                      Re: Static JSON parser for embedded Ben Bacarisse <ben.usenet@bsb.me.uk> - 2019-05-17 21:33 +0100
                  Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-18 00:11 +0200
              Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 22:33 +0100
                Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-18 00:17 +0200
                  Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-18 01:25 +0100
              Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 23:43 +0200
    Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 04:38 -0700
    Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 07:38 -0700
      Re: Static JSON parser for embedded Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2019-05-17 08:05 -0700
        Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 08:15 -0700
        Re: Static JSON parser for embedded Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2019-05-17 11:53 -0400
    Re: Static JSON parser for embedded Ian Collins <ian-news@hotmail.com> - 2019-05-18 15:33 +1200
      Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-18 11:57 +0100
    Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-05-18 17:14 +1000
      Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-18 16:03 +0200
        Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-05-19 09:12 +1000
          Re: Static JSON parser for embedded Niklas Holsti <niklas.holsti@tidorum.invalid> - 2019-05-19 17:13 +0300
            Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-05-20 09:25 +1000
          Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-06 11:12 +0200
            Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-07 09:11 +1000
              Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-07 08:59 +0200
                Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-12 09:41 +1000
      Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-05 16:44 +0200
        Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-06 09:27 +1000
          Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-06 09:50 +0200
            Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-07 09:08 +1000
        Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-06 09:48 +1000
          Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-06 09:54 +0200

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#143896

On Friday, May 17, 2019 at 1:43:47 PM UTC-4, Bart wrote:
> On 17/05/2019 18:21, Rick C. Hodgin wrote:
> > Might make a good comp.lang.c coding challenge.
> 
> Maybe, but the whole thing is rather open-ended.

Let's define it.

    1)  The user provides a structure.  Format must be per line:
            field_name, type[, optional]
    2)  Must be able to receive input data like this:
            field_name = data
    3)  Must generate output JSON.
    4)  Must be able to parse JSON input into:
            field_name = data

As a baseline.  Edits / revisions are welcome.

> The OP added a few extra bits in as well, like /adding/ optional flag 
> members to the struct, with the implication that the struct definition 
> now additionally has to be part of the generated code. (Itself full of 
> problems, if the struct has dependencies of a 1000s of lines of C 
> headers; you don't want to drag in all that.)

I think if it produced an internal structure that could be
queried by member, it wouldn't need to be in a JSON string
format or field_name = data format.

As things are needed, the API call to the "get_type()" and
"get_value()" functions could be given, which return a void*
to the indicated type, which is discriminated against by an
input int* variable that is populated with one of the #define
values (one for each type).

I welcome to proposals / revisions.  Might be a fun coding
challenge.

-- 
Rick C. Hodgin

[toc] | [prev] | [next] | [standalone]

#143897

On 17/05/2019 19:16, Rick C. Hodgin wrote:
> On Friday, May 17, 2019 at 1:43:47 PM UTC-4, Bart wrote:
>> On 17/05/2019 18:21, Rick C. Hodgin wrote:
>>> Might make a good comp.lang.c coding challenge.
>>
>> Maybe, but the whole thing is rather open-ended.
> 
> Let's define it.
> 
>      1)  The user provides a structure.  Format must be per line:
>              field_name, type[, optional]

This part is not clear. I think he prefers this structure to be defined 
as normal C code. If that's too hard, then an alternative can be used.

But I don't think what you have is sufficient:

What are the possibilities for 'type'? The OP's example included 
'char[32]' and 'int[8]' and 'unsigned int'. Just how elaborate can these 
get? And what size is 'int' anyway?

How do you deal with nested structs?

>      2)  Must be able to receive input data like this:
>              field_name = data

I don't remember this was part of the requirement. Is this C code?


>      3)  Must generate output JSON.

I don't remember this part either! (If needed, there are likely to be 
existing solutions.)

>      4)  Must be able to parse JSON input into:
>              field_name = data

That's not so simple either. You can't do this with a char[32] type or 
int[8]. And that still requires 'data' to be captured in a suitable 
format, which may be unpacked (so char[3] taking 3 bytes may be 
represented by a list of 3 ints taking 12 bytes).

Additionally, you can't expect the date to be present in the same order 
as in the struct. This is what makes a hard-coded parser very much 
harder. You can't do:

     readinteger("id", S->id, 4);
     readstring("name", &S->name,32);
     readarray("array", &S->array,8,4);

You have to tentatively read a field tag, and try and match it with one 
of the fields of the struct. It has to be more table-driven, but in that 
case, it is more the data that is hard-coded rather than the parsing code.

[toc] | [prev] | [next] | [standalone]

#143898

On Friday, May 17, 2019 at 2:43:42 PM UTC-4, Bart wrote:
> On 17/05/2019 19:16, Rick C. Hodgin wrote:
> > On Friday, May 17, 2019 at 1:43:47 PM UTC-4, Bart wrote:
> >> On 17/05/2019 18:21, Rick C. Hodgin wrote:
> >>> Might make a good comp.lang.c coding challenge.
> >>
> >> Maybe, but the whole thing is rather open-ended.
> > 
> > Let's define it.
> > 
> >      1)  The user provides a structure.  Format must be per line:
> >              field_name, type[, optional]
> 
> This part is not clear. I think he prefers this structure to be defined 
> as normal C code. If that's too hard, then an alternative can be used.
> 
> But I don't think what you have is sufficient:
> 
> What are the possibilities for 'type'? The OP's example included 
> 'char[32]' and 'int[8]' and 'unsigned int'. Just how elaborate can these 
> get? And what size is 'int' anyway?

In JSON, you're always dealing with text versions of the data,
so you translate into text.  char[32] can contain "rick" or
"bart" only.  It doesn't need to contain the full 32, but if
it is given "Bartholmew Montgomery Scott, Esquire" (36 chars),
then it truncates it to 32.

> How do you deal with nested structs?

type = JSON.

> >      2)  Must be able to receive input data like this:
> >              field_name = data
> 
> I don't remember this was part of the requirement. Is this C code?

No.  This would just be a way to receive input to encode into
the JSON string.

> >      3)  Must generate output JSON.
> 
> I don't remember this part either! (If needed, there are likely to be 
> existing solutions.)

His goals were to be interoperative with other systems, so it
must be able to take local data, build JSON, and be able to
take remote JSON, and build local data.

> >      4)  Must be able to parse JSON input into:
> >              field_name = data
> 
> That's not so simple either. You can't do this with a char[32] type or 
> int[8].

    char = "rick"
    array = [1, 2, 3, 4, 5, 6, 7, 8]

> And that still requires 'data' to be captured in a suitable 
> format, which may be unpacked (so char[3] taking 3 bytes may be 
> represented by a list of 3 ints taking 12 bytes).

It's always in text form in JSON.  sprintf(output, "%d", value)
for each and you convert int values to text.

> Additionally, you can't expect the date to be present in the same order 
> as in the struct. This is what makes a hard-coded parser very much 
> harder. You can't do:
> 
>      readinteger("id", S->id, 4);
>      readstring("name", &S->name,32);
>      readarray("array", &S->array,8,4);
> 
> You have to tentatively read a field tag, and try and match it with one 
> of the fields of the struct. It has to be more table-driven, but in that 
> case, it is more the data that is hard-coded rather than the parsing code.

The requirements of JSON are all that's required on the inside.
Does JSON have a required order?  I think it's more by protocol
or use cases that it would need to be in a particular order.  At
the same level, each item should be made visible to the caller.

I would generically parse an input structure defining the needs
of the JSON output, but be able to receive input in any order.
Only when the output is required to be in that order would it
then make it be that way.

I would probably also set a "populated" flag to indicate if one
of the fields is populated or not, and if not do not even include
it in the output ... unless it's flagged required, or possibly if
it's not flagged optional.

I think it would get us close.  But, we just need to look at the
needs of JSON, and then interpolate between C's needs, and JSON's
needs.

-- 
Rick C. Hodgin

[toc] | [prev] | [next] | [standalone]

#143899

On 17/05/2019 19:57, Rick C. Hodgin wrote:
> On Friday, May 17, 2019 at 2:43:42 PM UTC-4, Bart wrote:
>> On 17/05/2019 19:16, Rick C. Hodgin wrote:
>>> On Friday, May 17, 2019 at 1:43:47 PM UTC-4, Bart wrote:
>>>> On 17/05/2019 18:21, Rick C. Hodgin wrote:
>>>>> Might make a good comp.lang.c coding challenge.
>>>>
>>>> Maybe, but the whole thing is rather open-ended.
>>>
>>> Let's define it.
>>>
>>>       1)  The user provides a structure.  Format must be per line:
>>>               field_name, type[, optional]
>>
>> This part is not clear. I think he prefers this structure to be defined
>> as normal C code. If that's too hard, then an alternative can be used.
>>
>> But I don't think what you have is sufficient:
>>
>> What are the possibilities for 'type'? The OP's example included
>> 'char[32]' and 'int[8]' and 'unsigned int'. Just how elaborate can these
>> get? And what size is 'int' anyway?
> 
> In JSON, you're always dealing with text versions of the data,
> so you translate into text

/TO/ text; why? We're trying to parse existing text!


   char[32] can contain "rick" or
> "bart" only.  It doesn't need to contain the full 32, but if
> it is given "Bartholmew Montgomery Scott, Esquire" (36 chars),
> then it truncates it to 32.

That doesn't answer the question. How complicated can these types get? 
Can there be arrays of arrays of structs containing further arrays? The 
answer determines the syntax you might have to devise to represent the 
layout of the C struct.

> 
>> How do you deal with nested structs?
> 
> type = JSON.

And the layout of that struct is?


>> That's not so simple either. You can't do this with a char[32] type or
>> int[8].
> 
>      char = "rick"
>      array = [1, 2, 3, 4, 5, 6, 7, 8]

You've lost me. We are still talking about doing this in C, yes? Then 
that is not valid C syntax. Neither is it clear what the LHS is supposed 
to represent here.


>> And that still requires 'data' to be captured in a suitable
>> format, which may be unpacked (so char[3] taking 3 bytes may be
>> represented by a list of 3 ints taking 12 bytes).
> 
> It's always in text form in JSON.  sprintf(output, "%d", value)
> for each and you convert int values to text.

I don't understand. This is generating text from binary, but the 
requirement is to convert JSON text /to/ binary. Specifically into the 
fields of a C struct instance.

I think we must be talking at cross-purposes.

I'm fairly sure the OP wants to be able to convert JSON text to C binary 
data (look at the subject line).

There is no C to JSON, except that another requirement is a tool to 
automatically convert a C struct definition into a parser that can do 
that conversion (JSON to binary).

[toc] | [prev] | [next] | [standalone]

#143900

On Friday, May 17, 2019 at 3:11:57 PM UTC-4, Bart wrote:
> I don't understand. This is generating text from binary, but the 
> requirement is to convert JSON text /to/ binary. Specifically into the 
> fields of a C struct instance.

I view that aspect of the translation into another layer, because
translating what's coming in from a JSON string, or from data with
commensurate JSON attributes, is just translation into a parsable
form.  What comes in and be ranged into auto-types (an incoming
integer that will fit into a 32-bit signed integer range can be
stored as a 32-bit signed integer, those which are bigger into
something bigger.  For text, store it as a variable-length char
array, determined by what data was parsed on the incoming string.

To then take that data, which is in known minimum-require forms,
can be translated into a C structure form, and can generate the
C source code to process it.

This is basically what SOAP does, by the way.  You take an incoming
WSDL file and parse it into all of the requisite data structs for
your target language.

> I think we must be talking at cross-purposes.

It's just a different take on how it is processed, and in what
forms.  And, my goals in this part of the thread are more for
the coding contest, and not so much to directly address the
needs of the OP.

> I'm fairly sure the OP wants to be able to convert JSON text to C binary 
> data (look at the subject line).
> 
> There is no C to JSON, except that another requirement is a tool to 
> automatically convert a C struct definition into a parser that can do 
> that conversion (JSON to binary).

He mentioned in one of his posts that he would be communicating
with existing systems, which I assumed was two-way.  Maybe it's
not.  Maybe the OP's app is only receiving input.

-- 
Rick C. Hodgin

[toc] | [prev] | [next] | [standalone]

#143901

Bart <bc@freeuk.com> writes:

> On 17/05/2019 19:16, Rick C. Hodgin wrote:
>> On Friday, May 17, 2019 at 1:43:47 PM UTC-4, Bart wrote:
>>> On 17/05/2019 18:21, Rick C. Hodgin wrote:
>>>> Might make a good comp.lang.c coding challenge.
>>>
>>> Maybe, but the whole thing is rather open-ended.
>>
>> Let's define it.
>>
>>      1)  The user provides a structure.  Format must be per line:
>>              field_name, type[, optional]
>
> This part is not clear. I think he prefers this structure to be
> defined as normal C code. If that's too hard, then an alternative can
> be used.
>
> But I don't think what you have is sufficient:
>
> What are the possibilities for 'type'?

My favourite is functions (or function pointers in C).  The OP
presumably has no need for these since JSON can't represent them.  They
make sense (and are indeed very useful) in remote procedure calls.

> The OP's example included
> 'char[32]' and 'int[8]' and 'unsigned int'. Just how elaborate can
> these get? And what size is 'int' anyway?

I'd be surprised if the size really matters.  You know what type is
needed to represent a number in the "input" (it's a double precision
IEEE floating point number) and the generated code will simply convert
that to whatever numeric type the corresponding struct member has.  The
specification should say what happens in anomalous situations like a
negative source and an unsigned target or a fractional source and an
integral target and so on, but the "size" of int is really a matter of
whether the input should be checked for out of range values.  That may
not even be part of the requirements.

> How do you deal with nested structs?

I'm getting a strong sense of déjà vu here.  I did this (twice, in fact)
in the early 80s for a remote procedure call mechanism, first in C and
then again in Common Lisp (way more fun!).  There is a huge literature
about "argument marshalling" for RPC and there are almost certainly
JSON-based RPC mechanisms out there.  That might help in a search for
existing code.

>>      2)  Must be able to receive input data like this:
>>              field_name = data
>
> I don't remember this was part of the requirement. Is this C code?
>
>>      3)  Must generate output JSON.
>
> I don't remember this part either!

Nor I (to both of the above).

> You have to tentatively read a field tag, and try and match it with
> one of the fields of the struct. It has to be more table-driven, but
> in that case, it is more the data that is hard-coded rather than the
> parsing code.

There's going to a spectrum from a solution where bespoke C functions
get generated for every case, to one where a generic parser uses a
table of auxiliary data to assign value to struct members.

-- 
Ben.

[toc] | [prev] | [next] | [standalone]

#143904

Il 17/05/2019 19:43, Bart ha scritto:
> On 17/05/2019 18:21, Rick C. Hodgin wrote:
>> On Friday, May 17, 2019 at 12:56:33 PM UTC-4, Bart wrote:
>>> I think I'll have a go at doing this, although I don't know if I'll come
>>> up with anything that will be of any use to you.
>>>
>>> But some questions about the spec:
>>>
>>> * Tags in JSON can be in any order?
>>>
>>> * Unknown tags to be ignored?
>>>
>>> * Missing tags to be ignored? (Allowing both of these I think allows any
>>> JSON object to match any struct)
>>>
>>> * No char* fields? (These might need heap memory)
>>>
>>> * Float fields needed?
>>>
>>> * Will arrays fields be only 1D, or can be 2D and 3D?
>>
>> Might make a good comp.lang.c coding challenge.
> 
> Maybe, but the whole thing is rather open-ended. Even forgetting about 
> converting a C struct to C parsing code, just the parsing part, even for 
> a struct format that is hard-coded, is full of choices that need to be 
> made.
> 
> The specifications are not clear enough (the above is just part of it).
> 
> Having data come in in any order makes it entirely different: reading an 
> expected string or number is easier than reading an known value of half 
> a dozen type categories. What about underflow in array elements? Too 
> many array elements? 

You define a maximum number of elements of an array in the C struct 
definition. If there are less elements in the JSON message, we can use a 
new meta-member size. If there are more elements, the extra elements are 
simply ignored.

struct {
   int array[MAX_ARRAY_SIZE];
   size_t array_size;

   int array2[MAX_ARRAY2_SIZE];
   size_t array2_size;
} myStruct;


> Mixed types within an array? (All are possible in 
> JSON data.)

No, I wouldn't support this.


> How to distinguish between 'char[32]' which can be initialised with a 
> string, and one which is an array of integers, or allow either?

If we decide to define the C struct with a newer language (not pure C), 
we can define the exact type of the member (string or array of chars).

Another possibility is to use char[] for strings and (unsigned) int for 
integers (no short or long).


> It is rather a can of worms. (I'm doing it in a soft language first to 
> see what is possible.)
> 
> The OP added a few extra bits in as well, like /adding/ optional flag 
> members to the struct, with the implication that the struct definition 
> now additionally has to be part of the generated code. 

Considering it is very difficult to read the definition of a C struct in 
a pure C code (think of comments, preprocessor macros...), I think we 
start with a new struct definition language (.cj). The output of the 
generator will be .c and .h file, latter with C struct definition.

> (Itself full of 
> problems, if the struct has dependencies of a 1000s of lines of C 
> headers; you don't want to drag in all that.)
> 
> (The way it's coming out, the struct definition doesn't need to be 
> visible in the generated parser; it just needs tables of names, types, 
> offsets and sizes.)

Take a look at jsmn[1]. The parser generates an array of tokens. Each 
token has a type, a name, the offset in the message and the size of the 
token.

However, after parsing, it's difficult to search for a precise tag and 
retrive its value to save in the struct.


Another nice feature that some JSON parsers have and that can be very 
useful in restricted environments, is the possibility to parse a message 
incrementally.
Suppose you have a 4kB JSON message stored in an external memory module. 
The processor can't read directly from the external memory that isn't on 
a different bus (SPI, I2C).

The application could read part of the message, maybe 256 bytes, and 
feed the parser for each part, until the end.


[1] https://github.com/zserge/jsmn

[toc] | [prev] | [next] | [standalone]

#143902

On 17/05/2019 17:56, Bart wrote:
> On 17/05/2019 15:37, pozz wrote:

>> Consider a big JSON file stored on an external I2C or SPI memory (not 
>> on the same bus of the processor). The application could read 
>> 128-bytes parts and feed the parser, saving a lot of memory.

> I think I'll have a go at doing this, although I don't know if I'll come 
> up with anything that will be of any use to you.

I feel that doing a fully working solution in C (to convert C code 
declaring a struct, to C source that can parse that struct) would be too 
much work to complete. But I will continue to play with it.

At least I now have my own JSON parser 
(https://github.com/sal55/qx/blob/master/readjson.b, not in C, which 
converts a json file to a 'dict' dynamic data structure)

The next step would have been to link the recursive scanning of that 
input file, with a description** of the C struct layout, which could 
then be filled in as it scanned.

(** A table where each entry describes one field of the struct, and 
contains: field name; generic type (integer, string, array, further 
struct), size, offset, array/string length, and element size for the 
array. Not sure how nested structs would be handled.)

The routine for scanning JSON would be given a pointer to this table, 
and a pointer to a receiving struct. This would implement the 'dynamic' 
version of a struct parser, mentioned in an earlier post, rather than a 
dedicated one. The struct definition itself does not need to be visible 
to such a parser; only the tables are needed.

The next step after /that/ would have been to implement it all in C 
(ugh..), and then the next task would have been to hack a C compiler to 
be able to generate the tables needed. (And then that compiler - not in 
C - would need to be translated to C, something I haven't tried recently.)

So some ideas anyway if you (the OP) are thinking of a DIY solution.

[toc] | [prev] | [next] | [standalone]

#143905

Il 17/05/2019 23:33, Bart ha scritto:
> On 17/05/2019 17:56, Bart wrote:
>> On 17/05/2019 15:37, pozz wrote:
> 
>>> Consider a big JSON file stored on an external I2C or SPI memory (not 
>>> on the same bus of the processor). The application could read 
>>> 128-bytes parts and feed the parser, saving a lot of memory.
> 
>> I think I'll have a go at doing this, although I don't know if I'll 
>> come up with anything that will be of any use to you.
> 
> I feel that doing a fully working solution in C (to convert C code 
> declaring a struct, to C source that can parse that struct) would be too 
> much work to complete. But I will continue to play with it.
> 
> At least I now have my own JSON parser 
> (https://github.com/sal55/qx/blob/master/readjson.b, not in C, which 
> converts a json file to a 'dict' dynamic data structure)

Are you sure starting from JSON file is a good choice? I was thinking to 
start from C struct definition (in pure C code or something else).

Otherwise you should have a JSON file with populated keys and values.


> The next step would have been to link the recursive scanning of that 
> input file, with a description** of the C struct layout, which could 
> then be filled in as it scanned.
> 
> (** A table where each entry describes one field of the struct, and 
> contains: field name; generic type (integer, string, array, further 
> struct), size, offset, array/string length, and element size for the 
> array. Not sure how nested structs would be handled.)
> 
> The routine for scanning JSON would be given a pointer to this table, 
> and a pointer to a receiving struct. This would implement the 'dynamic' 
> version of a struct parser, mentioned in an earlier post, rather than a 
> dedicated one. The struct definition itself does not need to be visible 
> to such a parser; only the tables are needed.
> 
> The next step after /that/ would have been to implement it all in C 
> (ugh..), and then the next task would have been to hack a C compiler to 
> be able to generate the tables needed. (And then that compiler - not in 
> C - would need to be translated to C, something I haven't tried recently.)
> 
> So some ideas anyway if you (the OP) are thinking of a DIY solution.

I'm sorry, it's very difficult for me.

[toc] | [prev] | [next] | [standalone]

#143906

On 17/05/2019 23:17, pozz wrote:
> Il 17/05/2019 23:33, Bart ha scritto:
>> On 17/05/2019 17:56, Bart wrote:
>>> On 17/05/2019 15:37, pozz wrote:
>>
>>>> Consider a big JSON file stored on an external I2C or SPI memory 
>>>> (not on the same bus of the processor). The application could read 
>>>> 128-bytes parts and feed the parser, saving a lot of memory.
>>
>>> I think I'll have a go at doing this, although I don't know if I'll 
>>> come up with anything that will be of any use to you.
>>
>> I feel that doing a fully working solution in C (to convert C code 
>> declaring a struct, to C source that can parse that struct) would be 
>> too much work to complete. But I will continue to play with it.
>>
>> At least I now have my own JSON parser 
>> (https://github.com/sal55/qx/blob/master/readjson.b, not in C, which 
>> converts a json file to a 'dict' dynamic data structure)
> 
> Are you sure starting from JSON file is a good choice? I was thinking to 
> start from C struct definition (in pure C code or something else).
> 
> Otherwise you should have a JSON file with populated keys and values.

Well, that seemed to work. I modified my test reader so that it would 
fill in the values of a struct:

   https://github.com/sal55/qx/blob/master/parsejson.b

(Dynamic language, not C, not Python.)

Here I've manually hard-coded the table for your example struct (lines 
64 to 67). The struct definition does exist, or a version of the C one 
(lines 19 to 23), but here it's just for testing - to more easily set up 
a dummy destination of the right size, and to easily print the result. 
Otherwise it's not needed.

The struct printed from this program using that table, as populated from 
the JSON example in your OP:

   (12345, John, (-3,7,8,0,0,0,0,0))

I won't take this further as there are lots of other complications (like 
nested arrays, nested structs, optional fields and extra flag fields); 
all sorts of other details I don't know about and which would come up, 
and I don't know the overall context (how even does it decide which 
struct a bit of incoming JSON will describe).

But it shows that a 'dynamic' parser - hard-coded program, but tables 
somehow generated from C structs, could work, even in C. (A C version 
would just have taken longer to try out.)


>> So some ideas anyway if you (the OP) are thinking of a DIY solution.
> 
> I'm sorry, it's very difficult for me.

Maybe you should make it into a contest like someone suggested. People 
here quite like writing code. But the task needs to be smaller and more 
well-defined.

[toc] | [prev] | [next] | [standalone]

#143903

Il 17/05/2019 18:56, Bart ha scritto:
> On 17/05/2019 15:37, pozz wrote:
>> Il 17/05/2019 16:06, Bart ha scritto:
> 
>> Too difficult. It would be better to describe the C struct in a 
>> different simpler language and let the tool to generates the C struct 
>> definition and the parser code.
>>
>>
>>> The output can be a specific parser function, but that also would 
>>> rely on a more general library that provides, among other things, the 
>>> tokenising for JSON text data.
>>
>> Yes, of course.
>>
>>
>>> It'd make quite an interesting project actually, although my 
>>> experience (based on XML) suggests that a deceptively simple format 
>>> like JSON isn't quite so simple in practice..
>>
>> I agree. It isn't so simple in practice.
>>
>> Another good thing is incremental parsing. Many time the JSON string 
>> isn't completely in memory, but it is read in parts. It would be nice 
>> to call repeatedly parsing function with new parts until the JSON 
>> string ends.
>>
>> Consider a big JSON file stored on an external I2C or SPI memory (not 
>> on the same bus of the processor). The application could read 
>> 128-bytes parts and feed the parser, saving a lot of memory.
> 
> I think I'll have a go at doing this, although I don't know if I'll come 
> up with anything that will be of any use to you.

Thank you, maybe we can arrange a community github project and anyone 
can help.


> But some questions about the spec:
> 
> * Tags in JSON can be in any order?

Yes, i tags are keys. JSON standard defines a grammar, but it doesn't 
define how to define a schema of a certain message.

   { "name": "Bart", "language": "C" }

and

   { "language": "C", "name": "Bart" }

are perfectly identical.


> * Unknown tags to be ignored?

I think it's the better choice. Imagine that the generator (another 
application) will add some data in the future. We don't want to break 
old receivers.


> * Missing tags to be ignored? (Allowing both of these I think allows any 
> JSON object to match any struct)

Do you mean tags that are present in the C struct and not present in the 
JSON text?

I think we have two possibilities.
The tag can be mandatory or optional. If the key is mandatory, the 
parser should exit with an error (JSON message doesn't match the C struct).
If the tag is optional, we can add a bool "meta-member" in the final C 
struct that shows if the tag was present in the message.

struct {
   char name[32];
   bool has_language;
   char language[32];
} myStruct;


> * No char* fields? (These might need heap memory)

Do you mean a generic binary stream? I don't think.


> * Float fields needed?

I'm not interested in float fields in my current project. Maybe you can 
"disable" float feature with #define.


> * Will arrays fields be only 1D, or can be 2D and 3D?

{ "coord": [ [1, 2], [3, 4], [5, 6] ] }

This is a 2D array and it is perfectly legal in JSON grammar.


Consider that JSON specifications are too much generic. You can think of 
esoteric perfectly legal JSON messages that can't be directly 
deserialized in a C struct.

For example, duplicated tags:

{ "name": "Bart", "name": "John" }


Or non homogeneous arrays.

{
   "array": [
     "string", 1, false, {
       "tag1": "value1", "tag2": "value2"
     }
   ]
}

I think we can ignore those possibilities.

[toc] | [prev] | [next] | [standalone]

#143878

On Friday, May 17, 2019 at 4:20:26 AM UTC-3, pozz wrote:
> I'd like to use JSON documents in my embedded project. I don't need full 
> parsing capabilities, because JSON could be very complex in general.
> 
> However I will use simple C structs that are equivalent to JSON 
> messages, for example:
> 
> 
> struct myStruct {
>    unsigned int id;
>    char name[32];
>    int array[8];
> };
> 
> {
>    "id": 12345,
>    "name": "John",
>    "array": [ -3, 7, 8 ]
> }
> 
> 
> I'm searching a static converter from JSON to C struct, some sort of 
> preprocessor that generates C code from C struct definition.
> 
> The parse function prototype could be:
> 
>    int json_parse(const char *s, struct myStruct *dst);
> 
> if json_parse() returns 0, the result is in dst and I can access all the 
> members of the struct in a simple way in C code.
> 
> The parsing code should ignore JSON objects not defined in the struct 
> (for future use), however parsing well known members. For example, the 
> function should work well with this JSON string:
> 
> {
>    "id": 12345,
>    "name": "John",
>    "nickname": "mynick",
>    "array": [ -3, 7, 8 ]
> }
> 
> Another feature could be to mark a member as mandatory or optional, 
> maybe through comments:
> 
> struct myStruct {
>    unsigned int id;
>    char name[32];
>    char nickname[32];   //@ json_optional
>    int array[8];
> };
> 
> In this case the generated code could define the real struct as:
> 
> struct myStruct {
>    unsigned int id;
>    char name[32];
>    char nickname[32];   //@ json_optional
>    bool is_nickname_present;
>    int array[8];
> };
> 
> Do you know of something code generator?

I have the same need and I can share what I have so far
to implement the runtime of this generator.

 - Stream (support utf8 files or string)
 - Scanner that emits JSON tokens.

Do you want to parse a fixed JSON? For instance, do you always
have the json with the same layout and order of your struct? 

I have this exact need as well, but I also have situation
where the user can edit the json (like configuration) and
then the order can be modified and some items missing.

For this case (where the order can be modified) I need to
search for the property name.
For this task, I created the 'C string switch generator'
http://thradams.com/switchgenerator.html
and then I don't need to use a dictionary structure.

Basically I have all we need for the runtime, or to 
implement this by hand.

The generator can be created in JS or C and run online
just like 'C string switch generator'

[toc] | [prev] | [next] | [standalone]

#143888

On Friday, May 17, 2019 at 3:20:26 AM UTC-4, pozz wrote:
> I'd like to use JSON documents in my embedded project. I don't need full 
> parsing capabilities, because JSON could be very complex in general.

If you have an embedded project, choose another structure.  One
which is directly traversable directly with pointers.

> However I will use simple C structs that are equivalent to JSON 
> messages, for example:
> 
> 
> struct myStruct {
>    unsigned int id;
>    char name[32];
>    int array[8];
> };
> 
> {
>    "id": 12345,
>    "name": "John",
>    "array": [ -3, 7, 8 ]
> }
> 

Instead, encode as:

    Offset, Length    Content
      0       5       [4:tag offset][1:len]
      5       5       [4:data offset][1:type=int]
     10       5       [4:tag offset][1:len]
     15       6       [4:data offset][1:type=char*][1:len]
     21       5       [4:tag offset][1:len]
     26       6       [4:data offset][1:type=int array][1:elements=8]
     32       4       [4:terminator = 0]

     36               "id"
     38               [integer 12345]
     42               "name"
     46               "John"
     52               "array"
     57               [integers -3, 7, 8, 0, 0, 0, 0, 0]
     // Total = 89 bytes

It would allow you to traverse the header top-down, adjusting only
based on whatever options there are for the 1:type field on every
alternate row.  Everything else is directly computed by offset into
the data.

It can be encoded as mime64 for transport, and decoded for use,
so it's all text, able to go over a web server or whatever.

An algorithm would encode it.  An algorithm would decode it.  And,
it would be very fast, low-overhead, small code base, and concise.

-- 
Rick C. Hodgin

[toc] | [prev] | [next] | [standalone]

#143889

On Friday, 17 May 2019 15:38:45 UTC+1, Rick C. Hodgin  wrote:
> On Friday, May 17, 2019 at 3:20:26 AM UTC-4, pozz wrote:
> > I'd like to use JSON documents in my embedded project. I don't need full 
> > parsing capabilities, because JSON could be very complex in general.
> 
> If you have an embedded project, choose another structure.  One
> which is directly traversable directly with pointers.
> 
> > However I will use simple C structs that are equivalent to JSON 
> > messages, for example:
> > 
> > 
> > struct myStruct {
> >    unsigned int id;
> >    char name[32];
> >    int array[8];
> > };
> > 
> > {
> >    "id": 12345,
> >    "name": "John",
> >    "array": [ -3, 7, 8 ]
> > }
> > 
> 
> Instead, encode as:
> 
>     Offset, Length    Content
>       0       5       [4:tag offset][1:len]
>       5       5       [4:data offset][1:type=int]
>      10       5       [4:tag offset][1:len]
>      15       6       [4:data offset][1:type=char*][1:len]
>      21       5       [4:tag offset][1:len]
>      26       6       [4:data offset][1:type=int array][1:elements=8]
>      32       4       [4:terminator = 0]
> 
>      36               "id"
>      38               [integer 12345]
>      42               "name"
>      46               "John"
>      52               "array"
>      57               [integers -3, 7, 8, 0, 0, 0, 0, 0]
>      // Total = 89 bytes
> 
> It would allow you to traverse the header top-down, adjusting only
> based on whatever options there are for the 1:type field on every
> alternate row.  Everything else is directly computed by offset into
> the data.
> 
> It can be encoded as mime64 for transport, and decoded for use,
> so it's all text, able to go over a web server or whatever.
> 
> An algorithm would encode it.  An algorithm would decode it.  And,
> it would be very fast, low-overhead, small code base, and concise.
> 
Instead of using C structs with names in C struct member name scope
matching the JSON "name" fields, it might be easier to generate something
like the following

struct json_element
{
   char *name; // can be null if the field is part of an array
   struct json_element *next; // next field
   struct json_element *child; // for arrays and objects
   char *string_ value;  //  only valid if the type matches. Could use 
   double real_value;   // a union to save a few bytes 
   int type; //NUMERICAL, STRING, ARRAY, OBJECT
};

Now of course you need to write a parser to parse the parsed JSON.
  
For many applications, you have a flat structure of strings and scalars,
and you know in advance what the field names will be for valid data.
So the access functions are then quite easy to write.

[toc] | [prev] | [next] | [standalone]

#143890

On Friday, May 17, 2019 at 11:05:55 AM UTC-4, Malcolm McLean wrote:
> Instead of using C structs with names in C struct member name scope
> matching the JSON "name" fields, it might be easier to generate something
> like the following
> 
> struct json_element
> {
>    char *name; // can be null if the field is part of an array
>    struct json_element *next; // next field
>    struct json_element *child; // for arrays and objects
>    char *string_ value;  //  only valid if the type matches. Could use 
>    double real_value;   // a union to save a few bytes 
>    int type; //NUMERICAL, STRING, ARRAY, OBJECT
> };
> 
> Now of course you need to write a parser to parse the parsed JSON.
>   
> For many applications, you have a flat structure of strings and scalars,
> and you know in advance what the field names will be for valid data.
> So the access functions are then quite easy to write.

My thinking is for an embedded app, that you don't have to stick
with generic interoperative standards between remote systems, but
rather are free to enter proprietary domains of efficiency and need.

However, I read later on in the thread that he is looking to inter-
act with other machines that use JSON, so he has a constraint that
makes my solution non-viable in this case.

I think there should be a standard similar to what I posted above,
one which uses offsets from the start of the data, and one which
allows direct traversal, and to use that ability in lieu of JSON.
Having all that extra parsing is just inviting disaster and using
additional power that's not necessary.

That format above could be converted to a JSON format for physical
examination by a person, but computers should not have to deal with
text.

Just my opinion / philosophy on the subject.

Note also, that's why I created BXML (Binary XML), which allows for
tags to have an equal sign after them to indicate how long their
data is, so you don't have to fully parse, but can partially parse
and otherwise traverse:

    <xml name:4=Rick age:2=49 occupation:5=human/>

In this example, you parse <, xml, name, and based on the :4,
you directly copy Rick, and ignore the space, parse age:2, and
directly copy the 49, and ignore the space, parse occupation:5
and direction copy human, and encounter / and parse >.  It does
allow for faster parsing, as well as the embedding of large
amounts of data in an XML structure, such as sound data, graphics
or video data, such as:

    <song title:14=Old, Old Story mp3:398321=.../>

The mp3 data of 398,321 bytes is directly encoded within.  No
time to parse.  No ridiculous CDATA sections, etc.

The idea I posted above for pozz takes that a step further to
allow more direct parsing.

-- 
Rick C. Hodgin

[toc] | [prev] | [next] | [standalone]

#143891

Malcolm McLean wrote:

> On Friday, 17 May 2019 15:38:45 UTC+1, Rick C. Hodgin  wrote:
>> On Friday, May 17, 2019 at 3:20:26 AM UTC-4, pozz wrote:
>> > I'd like to use JSON documents in my embedded project. I don't need
>> > full parsing capabilities, because JSON could be very complex in
>> > general.
>> 
>> If you have an embedded project, choose another structure.  One
>> which is directly traversable directly with pointers.
>> 
>> > However I will use simple C structs that are equivalent to JSON
>> > messages, for example:
[snip]
>> An algorithm would encode it.  An algorithm would decode it.  And,
>> it would be very fast, low-overhead, small code base, and concise.
>> 
> Instead of using C structs with names in C struct member name scope
> matching the JSON "name" fields, it might be easier to generate something
> like the following
> 
> struct json_element
> {
>    char *name; // can be null if the field is part of an array
>    struct json_element *next; // next field
>    struct json_element *child; // for arrays and objects
>    char *string_ value;  //  only valid if the type matches. Could use
>    double real_value;   // a union to save a few bytes
>    int type; //NUMERICAL, STRING, ARRAY, OBJECT
> };
> 
> Now of course you need to write a parser to parse the parsed JSON.

Perhaps a place to start would be with an existing (simple) json parser.
A quick google search brought up
  https://gist.github.com/justjkk/436828/
which is a sample json parser written in Lex and Yacc (compilable to C 
source), licenced with an MIT open licence.

[snip]


-- 
Lew Pitcher
"In Skills, We Trust"

[toc] | [prev] | [next] | [standalone]

#143907

On 17/05/2019 19:20, pozz wrote:
> I'd like to use JSON documents in my embedded project. I don't need full
> parsing capabilities, because JSON could be very complex in general.
> 
> However I will use simple C structs that are equivalent to JSON
> messages, for example:
> 
> 
> struct myStruct {
>     unsigned int id;
>     char name[32];
>     int array[8];
> };
> 
> {
>     "id": 12345,
>     "name": "John",
>     "array": [ -3, 7, 8 ]
> }
> 


I can't see anywhere in your post where you specify the size of a string 
field.  Maybe

{
    "id": 12345,
    "name": {"string": 32, "value": "John"},
    "array": [ -3, 7, 8 ]
}

?

If you are interested in JSON style meta-languages, have a look at CTF 
(Used for Linux trace data amongst other things).

-- 
Ian.

[toc] | [prev] | [next] | [standalone]

#143909

On 18/05/2019 04:33, Ian Collins wrote:
> On 17/05/2019 19:20, pozz wrote:
>> I'd like to use JSON documents in my embedded project. I don't need full
>> parsing capabilities, because JSON could be very complex in general.
>>
>> However I will use simple C structs that are equivalent to JSON
>> messages, for example:
>>
>>
>> struct myStruct {
>>     unsigned int id;
>>     char name[32];
>>     int array[8];
>> };
>>
>> {
>>     "id": 12345,
>>     "name": "John",
>>     "array": [ -3, 7, 8 ]
>> }
>>
> 
> 
> I can't see anywhere in your post where you specify the size of a string 
> field.

It's specified in the C struct (assuming char[32] denotes a string), and 
here it specifies an upper limit, to include a terminator.


> Maybe
> 
> {
>     "id": 12345,
>     "name": {"string": 32, "value": "John"},
>     "array": [ -3, 7, 8 ]
> }

It doesn't really belong in the JSON data, which is more type and 
language independent (there is no dimension for 'array' either). What 
purpose would be serve? What does it mean? The data is a string with a 
length that is the length of the string.

The same JSON item could be read by a multitudinous set of programs and 
languages all with a different destination size, even within the same 
program, or maybe it doesn't matter.

But now a simple string is a complex object, with an ambiguous meaning 
(is this one string item, or a nested struct with an int field called 
"string", and a string field called "value"?).

[toc] | [prev] | [next] | [standalone]

#143908

On 17/5/19 5:20 pm, pozz wrote:
> I'd like to use JSON documents in my embedded project. I don't need full 
> parsing capabilities, because JSON could be very complex in general.
> 
> Do you know of something code generator?

Ok. Latecomer, but I read most of the thread. Introduction: I've been 
writing C since 1979 and have a million lines of my code in production 
in mission-critical software on tens of millions of enterprise 
computers. Although these days my C is mostly C++, I use C++ mostly for 
the features that make it "a better C".

Neither JSON nor C has a very strong type system. Neither is a good 
specification language. You should choose a specification language that 
does a good job of specifying your intent, and a compiler for it that 
emits C that handles one or more packing syntaxes such as JSON (if you 
must!)

The European Space Agency has followed the lead of the ISO networking 
standards and built a compiler for ASN.1, which emits C code targeted at 
static-memory embedded systems. The downside: I don't think it has a 
JSON packing rule (the ASN.1 name for a transfer syntax). Other ASN.1 
compilers do support JSON, but I don't know whether they target 
static-memory or embedded systems.

The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.

It supports BER (Basic encoding rules, a compact & fast binary format), 
PER (Packed Encoding Rules, a bit-packed binary format that will be 
slower but tighter), XER (XML Encoding Rules, you can imagine!), and 
ACN, where you get to define the encoding yourself.

If you wish to serve the community as well as yourself, write JER, JSON 
Encoding Rules and submit it.

A quick search finds a proposed standard: 
<https://www.obj-sys.com/docs/JSONEncodingRules.pdf> and a tool that 
analyses a JSON fragment and produces a proposed ASN.1 definition for 
the shape of the fragment <https://asn1.io/json2asn/>

So there is existing work, and no doubt community support for more.
You could do a lot worse than start with ASN1CC.

Clifford Heath.

[toc] | [prev] | [next] | [standalone]

#143910

Il 18/05/2019 09:14, Clifford Heath ha scritto:
> On 17/5/19 5:20 pm, pozz wrote:
>> I'd like to use JSON documents in my embedded project. I don't need 
>> full parsing capabilities, because JSON could be very complex in general.
>>
>> Do you know of something code generator?
> 
> Ok. Latecomer, but I read most of the thread. Introduction: I've been 
> writing C since 1979 and have a million lines of my code in production 
> in mission-critical software on tens of millions of enterprise 
> computers. Although these days my C is mostly C++, I use C++ mostly for 
> the features that make it "a better C".
> 
> Neither JSON nor C has a very strong type system. Neither is a good 
> specification language. You should choose a specification language that 
> does a good job of specifying your intent, and a compiler for it that 
> emits C that handles one or more packing syntaxes such as JSON (if you 
> must!)
> 
> The European Space Agency has followed the lead of the ISO networking 
> standards and built a compiler for ASN.1, which emits C code targeted at 
> static-memory embedded systems. The downside: I don't think it has a 
> JSON packing rule (the ASN.1 name for a transfer syntax). Other ASN.1 
> compilers do support JSON, but I don't know whether they target 
> static-memory or embedded systems.
> 
> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
> 
> It supports BER (Basic encoding rules, a compact & fast binary format), 
> PER (Packed Encoding Rules, a bit-packed binary format that will be 
> slower but tighter), XER (XML Encoding Rules, you can imagine!), and 
> ACN, where you get to define the encoding yourself.
> 
> If you wish to serve the community as well as yourself, write JER, JSON 
> Encoding Rules and submit it.
> 
> A quick search finds a proposed standard: 
> <https://www.obj-sys.com/docs/JSONEncodingRules.pdf> and a tool that 
> analyses a JSON fragment and produces a proposed ASN.1 definition for 
> the shape of the fragment <https://asn1.io/json2asn/>
> 
> So there is existing work, and no doubt community support for more.
> You could do a lot worse than start with ASN1CC.

ASN.1 is interested. I will look at it more deeply.

Just a simple question. Are optional fields possible in a message?

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.lang.c

csiph-web