Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.c > #143872 > unrolled thread

Static JSON parser for embedded

Back to article view | Back to comp.lang.c

Contents

  Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 09:20 +0200
    Re: Static JSON parser for embedded Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-05-17 08:48 +0000
      Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 11:06 +0200
        Re: Static JSON parser for embedded Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-05-17 11:22 +0000
          Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 04:40 -0700
            Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 14:10 +0200
              Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 05:18 -0700
                Re: Static JSON parser for embedded Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-05-18 18:18 +0000
          Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 14:10 +0200
            Re: Static JSON parser for embedded Jorgen Grahn <grahn+nntp@snipabacken.se> - 2019-05-18 18:22 +0000
    Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 11:39 +0100
      Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 13:19 +0200
        Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 05:09 -0700
          Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 05:15 -0700
        Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 15:06 +0100
          Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 16:37 +0200
            Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 17:56 +0100
              Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 10:21 -0700
                Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 18:43 +0100
                  Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 10:58 -0700
                  Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 11:16 -0700
                    Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 19:43 +0100
                      Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 11:57 -0700
                        Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 20:11 +0100
                          Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 12:22 -0700
                      Re: Static JSON parser for embedded Ben Bacarisse <ben.usenet@bsb.me.uk> - 2019-05-17 21:33 +0100
                  Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-18 00:11 +0200
              Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-17 22:33 +0100
                Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-18 00:17 +0200
                  Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-18 01:25 +0100
              Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-17 23:43 +0200
    Re: Static JSON parser for embedded Thiago Adams <thiago.adams@gmail.com> - 2019-05-17 04:38 -0700
    Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 07:38 -0700
      Re: Static JSON parser for embedded Malcolm McLean <malcolm.arthur.mclean@gmail.com> - 2019-05-17 08:05 -0700
        Re: Static JSON parser for embedded "Rick C. Hodgin" <rick.c.hodgin@gmail.com> - 2019-05-17 08:15 -0700
        Re: Static JSON parser for embedded Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2019-05-17 11:53 -0400
    Re: Static JSON parser for embedded Ian Collins <ian-news@hotmail.com> - 2019-05-18 15:33 +1200
      Re: Static JSON parser for embedded Bart <bc@freeuk.com> - 2019-05-18 11:57 +0100
    Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-05-18 17:14 +1000
      Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-05-18 16:03 +0200
        Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-05-19 09:12 +1000
          Re: Static JSON parser for embedded Niklas Holsti <niklas.holsti@tidorum.invalid> - 2019-05-19 17:13 +0300
            Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-05-20 09:25 +1000
          Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-06 11:12 +0200
            Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-07 09:11 +1000
              Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-07 08:59 +0200
                Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-12 09:41 +1000
      Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-05 16:44 +0200
        Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-06 09:27 +1000
          Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-06 09:50 +0200
            Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-07 09:08 +1000
        Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-06 09:48 +1000
          Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-06 09:54 +0200

Page 3 of 3 — ← Prev page 1 2 [3]

#143916

On 19/5/19 12:03 am, pozz wrote:
> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>> On 17/5/19 5:20 pm, pozz wrote:
>>> I'd like to use JSON documents in my embedded project. I don't need 
>>> full parsing capabilities, because JSON could be very complex in 
>>> general.
>>>
>>> Do you know of something code generator?
>>
>> Ok. Latecomer, but I read most of the thread. Introduction: I've been 
>> writing C since 1979 and have a million lines of my code in production 
>> in mission-critical software on tens of millions of enterprise 
>> computers. Although these days my C is mostly C++, I use C++ mostly 
>> for the features that make it "a better C".
>>
>> Neither JSON nor C has a very strong type system. Neither is a good 
>> specification language. You should choose a specification language 
>> that does a good job of specifying your intent, and a compiler for it 
>> that emits C that handles one or more packing syntaxes such as JSON 
>> (if you must!)
>>
>> The European Space Agency has followed the lead of the ISO networking 
>> standards and built a compiler for ASN.1, which emits C code targeted 
>> at static-memory embedded systems. The downside: I don't think it has 
>> a JSON packing rule (the ASN.1 name for a transfer syntax). Other 
>> ASN.1 compilers do support JSON, but I don't know whether they target 
>> static-memory or embedded systems.
>>
>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>
>> It supports BER (Basic encoding rules, a compact & fast binary 
>> format), PER (Packed Encoding Rules, a bit-packed binary format that 
>> will be slower but tighter), XER (XML Encoding Rules, you can 
>> imagine!), and ACN, where you get to define the encoding yourself.
>>
>> If you wish to serve the community as well as yourself, write JER, 
>> JSON Encoding Rules and submit it.
>>
>> A quick search finds a proposed standard: 
>> <https://www.obj-sys.com/docs/JSONEncodingRules.pdf> and a tool that 
>> analyses a JSON fragment and produces a proposed ASN.1 definition for 
>> the shape of the fragment <https://asn1.io/json2asn/>
>>
>> So there is existing work, and no doubt community support for more.
>> You could do a lot worse than start with ASN1CC.
> 
> ASN.1 is interested. I will look at it more deeply.
> 
> Just a simple question. Are optional fields possible in a message?

Yes. And pretty-much everything else you might need in a static type 
system; sets of alternates with a discriminator (discriminated unions), 
arbitrary nesting of arrays of structures of arrays of structures, 
strings of many and various character sets and encodings, enumerations, 
numbers limited by range not storage size, etc.

Not all ASN.1 implementations are equal. This one is targeted at systems 
that use no dynamic memory allocation, which is truly an excellent thing 
- but you probably can't do JSON like that. It's used for the majority 
of inter-module communications on ESA space missions, so it's pretty 
well tested and reliable too.

The point is that the ASN.1 definitions are focussed on defining the 
values that can be meaningfully transferred, not the possible values of 
a set of representations. It's not fully powerful, but it's a lot better 
than most programming languages.

I plan to emit ASN.1 from my Constellation Query Language tools at some 
stage this year, see <http://dataconstellation.com/ActiveFacts/>. CQL 
has a complete first-order logic constraint language, so it is even more 
powerful than ASN.1; and every feature (definition, constraint) is also 
expressible in controlled natural language and *compilable* from that 
language.

Clifford Heath.

[toc] | [prev] | [next] | [standalone]

#143918

On 19-05-19 02:12 , Clifford Heath wrote:
> On 19/5/19 12:03 am, pozz wrote:
>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>> ...
>>> The European Space Agency has followed the lead of the ISO networking
>>> standards and built a compiler for ASN.1, which emits C code targeted
>>> at static-memory embedded systems.

(By the way, it emits Ada code too.)

  [snip]

>>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>>
>  [snip]
 >
> This one is [snip] used for the majority
> of inter-module communications on ESA space missions,

Are you sure about that usage? I have been implementing on-board SW for 
ESA missions since 1995 or so, and I have never come across a system 
using ASN.1 or the ASN1SCC tool. There may be such, and there may be 
more in the future, but I doubt they can be in the majority now.

Perhaps you are confusing the ASN1SCC tool with the specific 
packet-based protocol (specific grammar, in other words) that is used by 
the majority of ESA missions now: the Packet Utilization Standard (PUS)? 
As I understand it, the ASN1SCC tool was implemented fairly recently to 
help with future SW implementations of PUS and similar packet-structure 
standards.

Of course this point is not very important for the OP, but it would 
interest me greatly if you can point to some ESA missions that use ASN1SCC.

(More off-topic: several years ago I heard a very negative opinion about 
the ASN.1 metalanguage from a senior colleague who was an expert in 
formal grammars and meta-compilers. He felt that the committee that 
defined ASN.1 had done so on an "amateur" level without properly using 
the metalanguage technology already well known in the computer science 
domain.)

(Even more off topic: IMO the intrusion into ESA of ITU working methods 
for standards has generally resulted in a drastic decrease in the 
usability of the ESA/ECSS standard documents (although not in the 
standards themselves). The volume of text and the number of identified 
requirements has increased hugely for purely formal reasons, without a 
corresponding increase in the scope of the standards. This applies in 
particular to the PUS, when one compares the original standard (PUS A) 
with the current one (PUS C).)

-- 
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
       .      @       .

[toc] | [prev] | [next] | [standalone]

#143924

On 20/5/19 12:13 am, Niklas Holsti wrote:
> On 19-05-19 02:12 , Clifford Heath wrote:
>> On 19/5/19 12:03 am, pozz wrote:
>>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>>> ...
>>>> The European Space Agency has followed the lead of the ISO networking
>>>> standards and built a compiler for ASN.1, which emits C code targeted
>>>> at static-memory embedded systems.
> (By the way, it emits Ada code too.)

Yes, but I didn't think c.l.c would care to know that :P

>   [snip]
> 
>>>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>>>
>>  [snip]
>  >
>> This one is [snip] used for the majority
>> of inter-module communications on ESA space missions,
> 
> Are you sure about that usage? > Perhaps you are confusing the ASN1SCC tool with the specific
> packet-based protocol (specific grammar, in other words) that is used by 
> the majority of ESA missions now: the Packet Utilization Standard (PUS)? 

You're right, I'm referring to the PUS standard. ASN1CC is one of the 
approaches to implementing it. Of course there are still many competing 
technologies in play. My contact is Serge Valera, who's been managing 
telecommand & telemetry (monitoring and control) standards ICDs 
(Interface Control Documents) and the verification of the 
implementations of the ICDs for over 30 years. He is the main author of 
the "ECSS-E-ST-70-41C - Telemetry and telecommand packet utilization" 
which defines the overarching standard for definition of these 
protocols. The bulk of this standard was machine-generated from a data 
model created as an Object Role Model (ORM) in the NORMA tool - about 
550 pages of text out of the total 650-ish in the document. That ORM 
model was created by Serge Valera.

https://ecss.nl/standard/ecss-e-st-70-41c-space-engineering-telemetry-and-telecommand-packet-utilization-15-april-2016/

Serge is also the main guy behind this, which is also ORM-inspired:

https://ecss.nl/hbstms/ecss-e-tm-10-23a-space-system-data-repository/

> Of course this point is not very important for the OP, but it would 
> interest me greatly if you can point to some ESA missions that use ASN1SCC.

I would need to contact Serge for that information. I don't have many 
details on specific missions.

> (More off-topic: several years ago I heard a very negative opinion about 
> the ASN.1 metalanguage from a senior colleague who was an expert in 
> formal grammars and meta-compilers. He felt that the committee that 
> defined ASN.1 had done so on an "amateur" level without properly using 
> the metalanguage technology already well known in the computer science 
> domain.)

The "right" way to do this is with a fact-based language like ORM (of 
which my CQL is a plain-text implementation; ORM is graphical but we 
share the underlying logic). ASN.1 is showing its age, but is solid and 
the tools are good for implementors; ORM tools are still rather thin on 
the ground. I'm curious to know what your colleague would recommend instead?

> (Even more off topic: IMO the intrusion into ESA of ITU working methods 
> for standards has generally resulted in a drastic decrease in the 
> usability of the ESA/ECSS standard documents (although not in the 
> standards themselves). The volume of text and the number of identified 
> requirements has increased hugely for purely formal reasons, without a 
> corresponding increase in the scope of the standards. This applies in 
> particular to the PUS, when one compares the original standard (PUS A) 
> with the current one (PUS C).)

Interesting point. The ESA process requires carefully-constructed 
English, and that is very verbose compared to the ORM model that 
generated PUS (which I have, BTW). Of course it would be much better if 
the underlying ORM model could be standardised, but again, the tool 
maturity is not there yet. That's why I'm working to bring CQL up to the 
required standard. Feel free to contact me privately 
(@dataconstellation.com) if you have further input to offer.

Clifford Heath, cjh@...

[toc] | [prev] | [next] | [standalone]

#144522

Il 19/05/2019 01:12, Clifford Heath ha scritto:
> On 19/5/19 12:03 am, pozz wrote:
>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>> On 17/5/19 5:20 pm, pozz wrote:
>>>> I'd like to use JSON documents in my embedded project. I don't need 
>>>> full parsing capabilities, because JSON could be very complex in 
>>>> general.
>>>>
>>>> Do you know of something code generator?
>>>
>>> Ok. Latecomer, but I read most of the thread. Introduction: I've been 
>>> writing C since 1979 and have a million lines of my code in 
>>> production in mission-critical software on tens of millions of 
>>> enterprise computers. Although these days my C is mostly C++, I use 
>>> C++ mostly for the features that make it "a better C".
>>>
>>> Neither JSON nor C has a very strong type system. Neither is a good 
>>> specification language. You should choose a specification language 
>>> that does a good job of specifying your intent, and a compiler for it 
>>> that emits C that handles one or more packing syntaxes such as JSON 
>>> (if you must!)
>>>
>>> The European Space Agency has followed the lead of the ISO networking 
>>> standards and built a compiler for ASN.1, which emits C code targeted 
>>> at static-memory embedded systems. The downside: I don't think it has 
>>> a JSON packing rule (the ASN.1 name for a transfer syntax). Other 
>>> ASN.1 compilers do support JSON, but I don't know whether they target 
>>> static-memory or embedded systems.
>>>
>>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>>
>>> It supports BER (Basic encoding rules, a compact & fast binary 
>>> format), PER (Packed Encoding Rules, a bit-packed binary format that 
>>> will be slower but tighter), XER (XML Encoding Rules, you can 
>>> imagine!), and ACN, where you get to define the encoding yourself.
>>>
>>> If you wish to serve the community as well as yourself, write JER, 
>>> JSON Encoding Rules and submit it.
>>>
>>> A quick search finds a proposed standard: 
>>> <https://www.obj-sys.com/docs/JSONEncodingRules.pdf> and a tool that 
>>> analyses a JSON fragment and produces a proposed ASN.1 definition for 
>>> the shape of the fragment <https://asn1.io/json2asn/>
>>>
>>> So there is existing work, and no doubt community support for more.
>>> You could do a lot worse than start with ASN1CC.
>>
>> ASN.1 is interested. I will look at it more deeply.
>>
>> Just a simple question. Are optional fields possible in a message?
> 
> Yes. And pretty-much everything else you might need in a static type 
> system; sets of alternates with a discriminator (discriminated unions), 
> arbitrary nesting of arrays of structures of arrays of structures, 
> strings of many and various character sets and encodings, enumerations, 
> numbers limited by range not storage size, etc.
> 
> Not all ASN.1 implementations are equal. This one is targeted at systems 
> that use no dynamic memory allocation, which is truly an excellent thing 
> - but you probably can't do JSON like that. It's used for the majority 
> of inter-module communications on ESA space missions, so it's pretty 
> well tested and reliable too.
> 
> The point is that the ASN.1 definitions are focussed on defining the 
> values that can be meaningfully transferred, not the possible values of 
> a set of representations. It's not fully powerful, but it's a lot better 
> than most programming languages.
> 
> I plan to emit ASN.1 from my Constellation Query Language tools at some 
> stage this year, see <http://dataconstellation.com/ActiveFacts/>. CQL 
> has a complete first-order logic constraint language, so it is even more 
> powerful than ASN.1; and every feature (definition, constraint) is also 
> expressible in controlled natural language and *compilable* from that 
> language.

Incredibly, it seems there isn't any ASN.1 PER codec for 
javascript/typescript (web-oriented) world. This limits the use to BER 
only :-(

[toc] | [prev] | [next] | [standalone]

#144552

On 6/6/19 7:12 pm, pozz wrote:
> Il 19/05/2019 01:12, Clifford Heath ha scritto:
>> On 19/5/19 12:03 am, pozz wrote:
>>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>>> On 17/5/19 5:20 pm, pozz wrote:
>>>>> I'd like to use JSON documents in my embedded project. I don't need 
>>>>> full parsing capabilities, because JSON could be very complex in 
>>>>> general.
>>>>>
>>>>> Do you know of something code generator?
>>>>
>>>> Ok. Latecomer, but I read most of the thread. Introduction: I've 
>>>> been writing C since 1979 and have a million lines of my code in 
>>>> production in mission-critical software on tens of millions of 
>>>> enterprise computers. Although these days my C is mostly C++, I use 
>>>> C++ mostly for the features that make it "a better C".
>>>>
>>>> Neither JSON nor C has a very strong type system. Neither is a good 
>>>> specification language. You should choose a specification language 
>>>> that does a good job of specifying your intent, and a compiler for 
>>>> it that emits C that handles one or more packing syntaxes such as 
>>>> JSON (if you must!)
>>>>
>>>> The European Space Agency has followed the lead of the ISO 
>>>> networking standards and built a compiler for ASN.1, which emits C 
>>>> code targeted at static-memory embedded systems. The downside: I 
>>>> don't think it has a JSON packing rule (the ASN.1 name for a 
>>>> transfer syntax). Other ASN.1 compilers do support JSON, but I don't 
>>>> know whether they target static-memory or embedded systems.
>>>>
>>>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>>>
>>>> It supports BER (Basic encoding rules, a compact & fast binary 
>>>> format), PER (Packed Encoding Rules, a bit-packed binary format that 
>>>> will be slower but tighter), XER (XML Encoding Rules, you can 
>>>> imagine!), and ACN, where you get to define the encoding yourself.
>>>>
>>>> If you wish to serve the community as well as yourself, write JER, 
>>>> JSON Encoding Rules and submit it.
>>>>
>>>> A quick search finds a proposed standard: 
>>>> <https://www.obj-sys.com/docs/JSONEncodingRules.pdf> and a tool that 
>>>> analyses a JSON fragment and produces a proposed ASN.1 definition 
>>>> for the shape of the fragment <https://asn1.io/json2asn/>
>>>>
>>>> So there is existing work, and no doubt community support for more.
>>>> You could do a lot worse than start with ASN1CC.
>>>
>>> ASN.1 is interested. I will look at it more deeply.
>>>
>>> Just a simple question. Are optional fields possible in a message?
>>
>> Yes. And pretty-much everything else you might need in a static type 
>> system; sets of alternates with a discriminator (discriminated 
>> unions), arbitrary nesting of arrays of structures of arrays of 
>> structures, strings of many and various character sets and encodings, 
>> enumerations, numbers limited by range not storage size, etc.
>>
>> Not all ASN.1 implementations are equal. This one is targeted at 
>> systems that use no dynamic memory allocation, which is truly an 
>> excellent thing - but you probably can't do JSON like that. It's used 
>> for the majority of inter-module communications on ESA space missions, 
>> so it's pretty well tested and reliable too.
>>
>> The point is that the ASN.1 definitions are focussed on defining the 
>> values that can be meaningfully transferred, not the possible values 
>> of a set of representations. It's not fully powerful, but it's a lot 
>> better than most programming languages.
>>
>> I plan to emit ASN.1 from my Constellation Query Language tools at 
>> some stage this year, see <http://dataconstellation.com/ActiveFacts/>. 
>> CQL has a complete first-order logic constraint language, so it is 
>> even more powerful than ASN.1; and every feature (definition, 
>> constraint) is also expressible in controlled natural language and 
>> *compilable* from that language.
> 
> Incredibly, it seems there isn't any ASN.1 PER codec for 
> javascript/typescript (web-oriented) world. This limits the use to BER 
> only :-(

ASN1CC has support for two languages, so I assume that it is set up for 
multi-language support. If that has been done well (and I haven't 
checked but I'd be surprised if it hasn't) it should be a fairly 
straight-forward matter to add support for a third language. Probably 
easier than implementing a fixed JS/TS processor for your desired 
message schema. I daresay you'd make many people grateful for that.

Clifford Heath.

[toc] | [prev] | [next] | [standalone]

#144567

Il 07/06/2019 01:11, Clifford Heath ha scritto:
> On 6/6/19 7:12 pm, pozz wrote:
>> Il 19/05/2019 01:12, Clifford Heath ha scritto:
>>> On 19/5/19 12:03 am, pozz wrote:
>>>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>>>> On 17/5/19 5:20 pm, pozz wrote:
>>>>>> I'd like to use JSON documents in my embedded project. I don't 
>>>>>> need full parsing capabilities, because JSON could be very complex 
>>>>>> in general.
>>>>>>
>>>>>> Do you know of something code generator?
>>>>>
>>>>> Ok. Latecomer, but I read most of the thread. Introduction: I've 
>>>>> been writing C since 1979 and have a million lines of my code in 
>>>>> production in mission-critical software on tens of millions of 
>>>>> enterprise computers. Although these days my C is mostly C++, I use 
>>>>> C++ mostly for the features that make it "a better C".
>>>>>
>>>>> Neither JSON nor C has a very strong type system. Neither is a good 
>>>>> specification language. You should choose a specification language 
>>>>> that does a good job of specifying your intent, and a compiler for 
>>>>> it that emits C that handles one or more packing syntaxes such as 
>>>>> JSON (if you must!)
>>>>>
>>>>> The European Space Agency has followed the lead of the ISO 
>>>>> networking standards and built a compiler for ASN.1, which emits C 
>>>>> code targeted at static-memory embedded systems. The downside: I 
>>>>> don't think it has a JSON packing rule (the ASN.1 name for a 
>>>>> transfer syntax). Other ASN.1 compilers do support JSON, but I 
>>>>> don't know whether they target static-memory or embedded systems.
>>>>>
>>>>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>>>>
>>>>> It supports BER (Basic encoding rules, a compact & fast binary 
>>>>> format), PER (Packed Encoding Rules, a bit-packed binary format 
>>>>> that will be slower but tighter), XER (XML Encoding Rules, you can 
>>>>> imagine!), and ACN, where you get to define the encoding yourself.
>>>>>
>>>>> If you wish to serve the community as well as yourself, write JER, 
>>>>> JSON Encoding Rules and submit it.
>>>>>
>>>>> A quick search finds a proposed standard: 
>>>>> <https://www.obj-sys.com/docs/JSONEncodingRules.pdf> and a tool 
>>>>> that analyses a JSON fragment and produces a proposed ASN.1 
>>>>> definition for the shape of the fragment <https://asn1.io/json2asn/>
>>>>>
>>>>> So there is existing work, and no doubt community support for more.
>>>>> You could do a lot worse than start with ASN1CC.
>>>>
>>>> ASN.1 is interested. I will look at it more deeply.
>>>>
>>>> Just a simple question. Are optional fields possible in a message?
>>>
>>> Yes. And pretty-much everything else you might need in a static type 
>>> system; sets of alternates with a discriminator (discriminated 
>>> unions), arbitrary nesting of arrays of structures of arrays of 
>>> structures, strings of many and various character sets and encodings, 
>>> enumerations, numbers limited by range not storage size, etc.
>>>
>>> Not all ASN.1 implementations are equal. This one is targeted at 
>>> systems that use no dynamic memory allocation, which is truly an 
>>> excellent thing - but you probably can't do JSON like that. It's used 
>>> for the majority of inter-module communications on ESA space 
>>> missions, so it's pretty well tested and reliable too.
>>>
>>> The point is that the ASN.1 definitions are focussed on defining the 
>>> values that can be meaningfully transferred, not the possible values 
>>> of a set of representations. It's not fully powerful, but it's a lot 
>>> better than most programming languages.
>>>
>>> I plan to emit ASN.1 from my Constellation Query Language tools at 
>>> some stage this year, see 
>>> <http://dataconstellation.com/ActiveFacts/>. CQL has a complete 
>>> first-order logic constraint language, so it is even more powerful 
>>> than ASN.1; and every feature (definition, constraint) is also 
>>> expressible in controlled natural language and *compilable* from that 
>>> language.
>>
>> Incredibly, it seems there isn't any ASN.1 PER codec for 
>> javascript/typescript (web-oriented) world. This limits the use to BER 
>> only :-(
> 
> ASN1CC has support for two languages, so I assume that it is set up for 
> multi-language support. If that has been done well (and I haven't 
> checked but I'd be surprised if it hasn't) it should be a fairly 
> straight-forward matter to add support for a third language. Probably 
> easier than implementing a fixed JS/TS processor for your desired 
> message schema. I daresay you'd make many people grateful for that.

Yes, but I haven't the know-how and time to add a third language to 
asn1scc. I need to find an off-the-shelf solution.

It seems there isn't a *proprietary* ASN1 PER Javascript implementation 
too (I found only JAVA support).

[toc] | [prev] | [next] | [standalone]

#144755

On 7/6/19 4:59 pm, pozz wrote:
> Il 07/06/2019 01:11, Clifford Heath ha scritto:
>> On 6/6/19 7:12 pm, pozz wrote:
>>> Il 19/05/2019 01:12, Clifford Heath ha scritto:
>>>> On 19/5/19 12:03 am, pozz wrote:
>>> Incredibly, it seems there isn't any ASN.1 PER codec for 
>>> javascript/typescript (web-oriented) world. This limits the use to 
>>> BER only :-(
>>
>> ASN1CC has support for two languages, so I assume that it is set up 
>> for multi-language support. If that has been done well (and I haven't 
>> checked but I'd be surprised if it hasn't) it should be a fairly 
>> straight-forward matter to add support for a third language. Probably 
>> easier than implementing a fixed JS/TS processor for your desired 
>> message schema. I daresay you'd make many people grateful for that.
> 
> Yes, but I haven't the know-how and time to add a third language to 
> asn1scc. I need to find an off-the-shelf solution.
> 
> It seems there isn't a *proprietary* ASN1 PER Javascript implementation 
> too (I found only JAVA support).

I have the know-how. Contact me if it's worth enough to you.

Clifford Heath.

[toc] | [prev] | [next] | [standalone]

#144481

Il 18/05/2019 09:14, Clifford Heath ha scritto:
> On 17/5/19 5:20 pm, pozz wrote:
>> I'd like to use JSON documents in my embedded project. I don't need 
>> full parsing capabilities, because JSON could be very complex in general.
>>
>> Do you know of something code generator?
> 
> Ok. Latecomer, but I read most of the thread. Introduction: I've been 
> writing C since 1979 and have a million lines of my code in production 
> in mission-critical software on tens of millions of enterprise 
> computers. Although these days my C is mostly C++, I use C++ mostly for 
> the features that make it "a better C".
> 
> Neither JSON nor C has a very strong type system. Neither is a good 
> specification language. You should choose a specification language that 
> does a good job of specifying your intent, and a compiler for it that 
> emits C that handles one or more packing syntaxes such as JSON (if you 
> must!)
> 
> The European Space Agency has followed the lead of the ISO networking 
> standards and built a compiler for ASN.1, which emits C code targeted at 
> static-memory embedded systems. The downside: I don't think it has a 
> JSON packing rule (the ASN.1 name for a transfer syntax). Other ASN.1 
> compilers do support JSON, but I don't know whether they target 
> static-memory or embedded systems.
> 
> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
> 
> It supports BER (Basic encoding rules, a compact & fast binary format), 
> PER (Packed Encoding Rules, a bit-packed binary format that will be 
> slower but tighter), XER (XML Encoding Rules, you can imagine!), and 
> ACN, where you get to define the encoding yourself.
> 
> If you wish to serve the community as well as yourself, write JER, JSON 
> Encoding Rules and submit it.
> 
> A quick search finds a proposed standard: 
> <https://www.obj-sys.com/docs/JSONEncodingRules.pdf> and a tool that 
> analyses a JSON fragment and produces a proposed ASN.1 definition for 
> the shape of the fragment <https://asn1.io/json2asn/>
> 
> So there is existing work, and no doubt community support for more.
> You could do a lot worse than start with ASN1CC.

I eventually take a look at asn1scc and found a big limitation: it 
doesn't support extensibility feature of ASN.1.

Do you really use fixed messages? Don't you ever change a message 
definition adding some new variables in a future release?

[toc] | [prev] | [next] | [standalone]

#144507

On 6/6/19 12:44 am, pozz wrote:
> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>> So there is existing work, and no doubt community support for more.
>> You could do a lot worse than start with ASN1CC.
> 
> I eventually take a look at asn1scc and found a big limitation: it 
> doesn't support extensibility feature of ASN.1.
> 
> Do you really use fixed messages? Don't you ever change a message 
> definition adding some new variables in a future release?

The problem of extensibility affects more than just your ability to 
lexically decode the message. The eXtensibility in XML is purely 
lexical, it doesn't help at all with semantics.

When new content is added to a message, there is no certainty that the 
previous items' meanings have not changed. Simply ignoring the new 
content is no guarantee of correct behaviour. What if the extension is 
critical to the logic required by the application, and your code doesn't 
even know that?

I think it's better to manage protocol versioning explicitly. Put a 
protocol version number in the handshake and take it from there.

However, some purposes for which ASN.1 is used (like X.509 public key 
messages) define structures which have a variable-length element that 
contains the nested encoding of another ASN.1 object. The outer envelope 
says what it is, so the correct decoding can be applied. In the case of 
X.509 certificate extensions, the wrapper also says whether an 
understanding of the embedded content is critical or not. So there's one 
example of how to handle it.

The same kind of approach is taken in TIFF - the acronym actually means 
Tagged Image File Format.

There's a whole theory around versioning and interoperability and a 
seven-level model for it. See 
<https://en.wikipedia.org/wiki/Conceptual_interoperability>. I know 
Andraes Tolk, the co-creator of this model.

Clifford Heath.

[toc] | [prev] | [next] | [standalone]

#144519

Il 06/06/2019 01:27, Clifford Heath ha scritto:
> On 6/6/19 12:44 am, pozz wrote:
>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>> So there is existing work, and no doubt community support for more.
>>> You could do a lot worse than start with ASN1CC.
>>
>> I eventually take a look at asn1scc and found a big limitation: it 
>> doesn't support extensibility feature of ASN.1.
>>
>> Do you really use fixed messages? Don't you ever change a message 
>> definition adding some new variables in a future release?
> 
> The problem of extensibility affects more than just your ability to 
> lexically decode the message. The eXtensibility in XML is purely 
> lexical, it doesn't help at all with semantics.
> 
> When new content is added to a message, there is no certainty that the 
> previous items' meanings have not changed. Simply ignoring the new 
> content is no guarantee of correct behaviour. What if the extension is 
> critical to the logic required by the application, and your code doesn't 
> even know that?

It's the author of definitions that should decide if an extension breaks 
old parsers. In my experience, many times I added some additional data 
to a message (proprietary encoded) in a second time, without breaking 
older receivers.

Just an example. I have a central unit that implements a logic and an 
external HMI. They communicates by a serial RS485 connection. During 
first development stage, I defined all the messages and encoded with 
custom rules:

- this message has 1 byte, than 2 byte, then a string null-terminated...
- this message has 3 bytes, ...

You got the point. When a new feature is added in the control unit, I 
can add related data to the end of the message. The receiver decodes the 
data it knows, discarding the rest of the message. So v1 receivers can 
continue decoding v2 messages. Of course, new feature isn't represented 
on the HMI (until someone decides to update the HMI too), but v1 HMIs 
and v2 controls can coexist.


> I think it's better to manage protocol versioning explicitly. Put a 
> protocol version number in the handshake and take it from there.

It's a possibile solution, yes.


> However, some purposes for which ASN.1 is used (like X.509 public key 
> messages) define structures which have a variable-length element that 
> contains the nested encoding of another ASN.1 object. The outer envelope 
> says what it is, so the correct decoding can be applied. In the case of 
> X.509 certificate extensions, the wrapper also says whether an 
> understanding of the embedded content is critical or not. So there's one 
> example of how to handle it.

Please, could you produce an example? I can't follow you.

> 
> The same kind of approach is taken in TIFF - the acronym actually means 
> Tagged Image File Format.
> 
> There's a whole theory around versioning and interoperability and a 
> seven-level model for it. See 
> <https://en.wikipedia.org/wiki/Conceptual_interoperability>. I know 
> Andraes Tolk, the co-creator of this model.
> 
> Clifford Heath.

[toc] | [prev] | [next] | [standalone]

#144551

On 6/6/19 5:50 pm, pozz wrote:
> Il 06/06/2019 01:27, Clifford Heath ha scritto:
>> On 6/6/19 12:44 am, pozz wrote:
>>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>>> So there is existing work, and no doubt community support for more.
>>>> You could do a lot worse than start with ASN1CC.
>>>
>>> I eventually take a look at asn1scc and found a big limitation: it 
>>> doesn't support extensibility feature of ASN.1.
>>>
>>> Do you really use fixed messages? Don't you ever change a message 
>>> definition adding some new variables in a future release?
>>
>> The problem of extensibility affects more than just your ability to 
>> lexically decode the message. The eXtensibility in XML is purely 
>> lexical, it doesn't help at all with semantics.
>>
>> When new content is added to a message, there is no certainty that the 
>> previous items' meanings have not changed. Simply ignoring the new 
>> content is no guarantee of correct behaviour. What if the extension is 
>> critical to the logic required by the application, and your code 
>> doesn't even know that?
> 
> It's the author of definitions that should decide if an extension breaks 
> old parsers.

Again, you are confusing between parsing (syntax) and purpose 
(semantics). It's easy to define formats that allow extensions without 
breaking parsing, but that gives you no confidence that there is not a 
semantic error in doing that. JSON, XML, YAML, etc, all support 
extensible parsing.

XML, when used with XSD (XML schemas) allows validation of the correct 
structure of the base document, and also of one or more sets of 
extension schemas that are merged in to the base document. This is still 
syntax, but it means that anything added to the base document which is 
not allowed in that base schema can be detected and rejected. So it 
gives you a way to detect unexpected modifications to the base syntax, 
and also support extensions with similar validation.

I believe that JSONSchema does the same, but I haven't used it and 
wouldn't choose to.

I have created a language called ADL (Aspect Definition Language) with 
implementations so far only in C# and Ruby, which has some very nice 
characteristics in addition to these - though it's still not suitable 
for resource-constrained embedded environments. 
<https://github.com/cjheath/adl>

> You got the point.

Yes, I got the point. Email header fields are related and similar: you 
can add them and transport them through nodes that don't recognise them.

>> I think it's better to manage protocol versioning explicitly. Put a 
>> protocol version number in the handshake and take it from there.
> 
> It's a possibile solution, yes.
> 
> 
>> However, some purposes for which ASN.1 is used (like X.509 public key 
>> messages) define structures which have a variable-length element that 
>> contains the nested encoding of another ASN.1 object. The outer 
>> envelope says what it is, so the correct decoding can be applied. In 
>> the case of X.509 certificate extensions, the wrapper also says 
>> whether an understanding of the embedded content is critical or not. 
>> So there's one example of how to handle it.
> 
> Please, could you produce an example? I can't follow you.

Look at the way X.509 certificate extensions are handled. There's a text 
dump visible here: 
<https://access.redhat.com/documentation/en-US/Red_Hat_Certificate_System/8.0/html/Admin_Guide/Standard_X.509_v3_Certificate_Extensions.html>.

The key thing is that the outer layer (the certificate itself) is fully 
typed, but contains a section for extensions. Every extension has an 
identifier, a critical flag, and then a variable blob of data. The first 
one there is the "Certificate Usage" which is a bit-mask of services 
that may rely on this certificate. But basically the "blob" can be 
skipped by a parser that doesn't recognise the identifier, and decoded 
only by a parser that does recognise it.

Clifford Heath.

[toc] | [prev] | [next] | [standalone]

#144508

On 6/6/19 12:44 am, pozz wrote:
>> So there is existing work, and no doubt community support for more.
>> You could do a lot worse than start with ASN1CC.
> 
> I eventually take a look at asn1scc and found a big limitation: it 
> doesn't support extensibility feature of ASN.1.

One more comment. You mention (in other messages) that JSON is difficult 
without using dynamic memory. ASN1CC is designed to avoid using dynamic 
memory (for the same reason; embedded system constraints), which is why 
it cannot implement arbitrary extensions. The best it could do is to 
skip unknown extensions. But in PER, items are not tagged or 
length-delimited, and skipping an extension requires at least a 
knowledge of how big it is.

Clifford Heath.

[toc] | [prev] | [next] | [standalone]

#144520

Il 06/06/2019 01:48, Clifford Heath ha scritto:
> On 6/6/19 12:44 am, pozz wrote:
>>> So there is existing work, and no doubt community support for more.
>>> You could do a lot worse than start with ASN1CC.
>>
>> I eventually take a look at asn1scc and found a big limitation: it 
>> doesn't support extensibility feature of ASN.1.
> 
> One more comment. You mention (in other messages) that JSON is difficult 
> without using dynamic memory. ASN1CC is designed to avoid using dynamic 
> memory (for the same reason; embedded system constraints), which is why 
> it cannot implement arbitrary extensions. The best it could do is to 
> skip unknown extensions. 

Yes, many times this behaviour is good, instead of breaking at all the 
system.

> But in PER, items are not tagged or 
> length-delimited, and skipping an extension requires at least a 
> knowledge of how big it is.

Ok, PER can't be used to skip extended variables. However I can use BER. 
Why do asn1scc refuses to compile definitions file with extensions when 
BER is used?

[toc] | [prev] | [standalone]

Page 3 of 3 — ← Prev page 1 2 [3]

Back to top | Article view | comp.lang.c

csiph-web