Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #28642 > unrolled thread

Re: Static JSON parser for embedded

Back to article view | Back to comp.arch.embedded

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-05-19 09:12 +1000
    Re: Static JSON parser for embedded Niklas Holsti <niklas.holsti@tidorum.invalid> - 2019-05-19 17:13 +0300
      Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-05-20 09:25 +1000
    Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-06 11:12 +0200
      Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-07 09:11 +1000
        Re: Static JSON parser for embedded pozz <pozzugno@gmail.com> - 2019-06-07 08:59 +0200
          Re: Static JSON parser for embedded Clifford Heath <no.spam@please.net> - 2019-06-12 09:41 +1000

#28642 — Re: Static JSON parser for embedded

On 19/5/19 12:03 am, pozz wrote:
> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>> On 17/5/19 5:20 pm, pozz wrote:
>>> I'd like to use JSON documents in my embedded project. I don't need 
>>> full parsing capabilities, because JSON could be very complex in 
>>> general.
>>>
>>> Do you know of something code generator?
>>
>> Ok. Latecomer, but I read most of the thread. Introduction: I've been 
>> writing C since 1979 and have a million lines of my code in production 
>> in mission-critical software on tens of millions of enterprise 
>> computers. Although these days my C is mostly C++, I use C++ mostly 
>> for the features that make it "a better C".
>>
>> Neither JSON nor C has a very strong type system. Neither is a good 
>> specification language. You should choose a specification language 
>> that does a good job of specifying your intent, and a compiler for it 
>> that emits C that handles one or more packing syntaxes such as JSON 
>> (if you must!)
>>
>> The European Space Agency has followed the lead of the ISO networking 
>> standards and built a compiler for ASN.1, which emits C code targeted 
>> at static-memory embedded systems. The downside: I don't think it has 
>> a JSON packing rule (the ASN.1 name for a transfer syntax). Other 
>> ASN.1 compilers do support JSON, but I don't know whether they target 
>> static-memory or embedded systems.
>>
>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>
>> It supports BER (Basic encoding rules, a compact & fast binary 
>> format), PER (Packed Encoding Rules, a bit-packed binary format that 
>> will be slower but tighter), XER (XML Encoding Rules, you can 
>> imagine!), and ACN, where you get to define the encoding yourself.
>>
>> If you wish to serve the community as well as yourself, write JER, 
>> JSON Encoding Rules and submit it.
>>
>> A quick search finds a proposed standard: 
>> <https://www.obj-sys.com/docs/JSONEncodingRules.pdf> and a tool that 
>> analyses a JSON fragment and produces a proposed ASN.1 definition for 
>> the shape of the fragment <https://asn1.io/json2asn/>
>>
>> So there is existing work, and no doubt community support for more.
>> You could do a lot worse than start with ASN1CC.
> 
> ASN.1 is interested. I will look at it more deeply.
> 
> Just a simple question. Are optional fields possible in a message?

Yes. And pretty-much everything else you might need in a static type 
system; sets of alternates with a discriminator (discriminated unions), 
arbitrary nesting of arrays of structures of arrays of structures, 
strings of many and various character sets and encodings, enumerations, 
numbers limited by range not storage size, etc.

Not all ASN.1 implementations are equal. This one is targeted at systems 
that use no dynamic memory allocation, which is truly an excellent thing 
- but you probably can't do JSON like that. It's used for the majority 
of inter-module communications on ESA space missions, so it's pretty 
well tested and reliable too.

The point is that the ASN.1 definitions are focussed on defining the 
values that can be meaningfully transferred, not the possible values of 
a set of representations. It's not fully powerful, but it's a lot better 
than most programming languages.

I plan to emit ASN.1 from my Constellation Query Language tools at some 
stage this year, see <http://dataconstellation.com/ActiveFacts/>. CQL 
has a complete first-order logic constraint language, so it is even more 
powerful than ASN.1; and every feature (definition, constraint) is also 
expressible in controlled natural language and *compilable* from that 
language.

Clifford Heath.

[toc] | [next] | [standalone]

#28643

On 19-05-19 02:12 , Clifford Heath wrote:
> On 19/5/19 12:03 am, pozz wrote:
>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>> ...
>>> The European Space Agency has followed the lead of the ISO networking
>>> standards and built a compiler for ASN.1, which emits C code targeted
>>> at static-memory embedded systems.

(By the way, it emits Ada code too.)

  [snip]

>>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>>
>  [snip]
 >
> This one is [snip] used for the majority
> of inter-module communications on ESA space missions,

Are you sure about that usage? I have been implementing on-board SW for 
ESA missions since 1995 or so, and I have never come across a system 
using ASN.1 or the ASN1SCC tool. There may be such, and there may be 
more in the future, but I doubt they can be in the majority now.

Perhaps you are confusing the ASN1SCC tool with the specific 
packet-based protocol (specific grammar, in other words) that is used by 
the majority of ESA missions now: the Packet Utilization Standard (PUS)? 
As I understand it, the ASN1SCC tool was implemented fairly recently to 
help with future SW implementations of PUS and similar packet-structure 
standards.

Of course this point is not very important for the OP, but it would 
interest me greatly if you can point to some ESA missions that use ASN1SCC.

(More off-topic: several years ago I heard a very negative opinion about 
the ASN.1 metalanguage from a senior colleague who was an expert in 
formal grammars and meta-compilers. He felt that the committee that 
defined ASN.1 had done so on an "amateur" level without properly using 
the metalanguage technology already well known in the computer science 
domain.)

(Even more off topic: IMO the intrusion into ESA of ITU working methods 
for standards has generally resulted in a drastic decrease in the 
usability of the ESA/ECSS standard documents (although not in the 
standards themselves). The volume of text and the number of identified 
requirements has increased hugely for purely formal reasons, without a 
corresponding increase in the scope of the standards. This applies in 
particular to the PUS, when one compares the original standard (PUS A) 
with the current one (PUS C).)

-- 
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
       .      @       .

[toc] | [prev] | [next] | [standalone]

#28644

On 20/5/19 12:13 am, Niklas Holsti wrote:
> On 19-05-19 02:12 , Clifford Heath wrote:
>> On 19/5/19 12:03 am, pozz wrote:
>>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>>> ...
>>>> The European Space Agency has followed the lead of the ISO networking
>>>> standards and built a compiler for ASN.1, which emits C code targeted
>>>> at static-memory embedded systems.
> (By the way, it emits Ada code too.)

Yes, but I didn't think c.l.c would care to know that :P

>   [snip]
> 
>>>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>>>
>>  [snip]
>  >
>> This one is [snip] used for the majority
>> of inter-module communications on ESA space missions,
> 
> Are you sure about that usage? > Perhaps you are confusing the ASN1SCC tool with the specific
> packet-based protocol (specific grammar, in other words) that is used by 
> the majority of ESA missions now: the Packet Utilization Standard (PUS)? 

You're right, I'm referring to the PUS standard. ASN1CC is one of the 
approaches to implementing it. Of course there are still many competing 
technologies in play. My contact is Serge Valera, who's been managing 
telecommand & telemetry (monitoring and control) standards ICDs 
(Interface Control Documents) and the verification of the 
implementations of the ICDs for over 30 years. He is the main author of 
the "ECSS-E-ST-70-41C - Telemetry and telecommand packet utilization" 
which defines the overarching standard for definition of these 
protocols. The bulk of this standard was machine-generated from a data 
model created as an Object Role Model (ORM) in the NORMA tool - about 
550 pages of text out of the total 650-ish in the document. That ORM 
model was created by Serge Valera.

https://ecss.nl/standard/ecss-e-st-70-41c-space-engineering-telemetry-and-telecommand-packet-utilization-15-april-2016/

Serge is also the main guy behind this, which is also ORM-inspired:

https://ecss.nl/hbstms/ecss-e-tm-10-23a-space-system-data-repository/

> Of course this point is not very important for the OP, but it would 
> interest me greatly if you can point to some ESA missions that use ASN1SCC.

I would need to contact Serge for that information. I don't have many 
details on specific missions.

> (More off-topic: several years ago I heard a very negative opinion about 
> the ASN.1 metalanguage from a senior colleague who was an expert in 
> formal grammars and meta-compilers. He felt that the committee that 
> defined ASN.1 had done so on an "amateur" level without properly using 
> the metalanguage technology already well known in the computer science 
> domain.)

The "right" way to do this is with a fact-based language like ORM (of 
which my CQL is a plain-text implementation; ORM is graphical but we 
share the underlying logic). ASN.1 is showing its age, but is solid and 
the tools are good for implementors; ORM tools are still rather thin on 
the ground. I'm curious to know what your colleague would recommend instead?

> (Even more off topic: IMO the intrusion into ESA of ITU working methods 
> for standards has generally resulted in a drastic decrease in the 
> usability of the ESA/ECSS standard documents (although not in the 
> standards themselves). The volume of text and the number of identified 
> requirements has increased hugely for purely formal reasons, without a 
> corresponding increase in the scope of the standards. This applies in 
> particular to the PUS, when one compares the original standard (PUS A) 
> with the current one (PUS C).)

Interesting point. The ESA process requires carefully-constructed 
English, and that is very verbose compared to the ORM model that 
generated PUS (which I have, BTW). Of course it would be much better if 
the underlying ORM model could be standardised, but again, the tool 
maturity is not there yet. That's why I'm working to bring CQL up to the 
required standard. Feel free to contact me privately 
(@dataconstellation.com) if you have further input to offer.

Clifford Heath, cjh@...

[toc] | [prev] | [next] | [standalone]

#28657

Il 19/05/2019 01:12, Clifford Heath ha scritto:
> On 19/5/19 12:03 am, pozz wrote:
>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>> On 17/5/19 5:20 pm, pozz wrote:
>>>> I'd like to use JSON documents in my embedded project. I don't need 
>>>> full parsing capabilities, because JSON could be very complex in 
>>>> general.
>>>>
>>>> Do you know of something code generator?
>>>
>>> Ok. Latecomer, but I read most of the thread. Introduction: I've been 
>>> writing C since 1979 and have a million lines of my code in 
>>> production in mission-critical software on tens of millions of 
>>> enterprise computers. Although these days my C is mostly C++, I use 
>>> C++ mostly for the features that make it "a better C".
>>>
>>> Neither JSON nor C has a very strong type system. Neither is a good 
>>> specification language. You should choose a specification language 
>>> that does a good job of specifying your intent, and a compiler for it 
>>> that emits C that handles one or more packing syntaxes such as JSON 
>>> (if you must!)
>>>
>>> The European Space Agency has followed the lead of the ISO networking 
>>> standards and built a compiler for ASN.1, which emits C code targeted 
>>> at static-memory embedded systems. The downside: I don't think it has 
>>> a JSON packing rule (the ASN.1 name for a transfer syntax). Other 
>>> ASN.1 compilers do support JSON, but I don't know whether they target 
>>> static-memory or embedded systems.
>>>
>>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>>
>>> It supports BER (Basic encoding rules, a compact & fast binary 
>>> format), PER (Packed Encoding Rules, a bit-packed binary format that 
>>> will be slower but tighter), XER (XML Encoding Rules, you can 
>>> imagine!), and ACN, where you get to define the encoding yourself.
>>>
>>> If you wish to serve the community as well as yourself, write JER, 
>>> JSON Encoding Rules and submit it.
>>>
>>> A quick search finds a proposed standard: 
>>> <https://www.obj-sys.com/docs/JSONEncodingRules.pdf> and a tool that 
>>> analyses a JSON fragment and produces a proposed ASN.1 definition for 
>>> the shape of the fragment <https://asn1.io/json2asn/>
>>>
>>> So there is existing work, and no doubt community support for more.
>>> You could do a lot worse than start with ASN1CC.
>>
>> ASN.1 is interested. I will look at it more deeply.
>>
>> Just a simple question. Are optional fields possible in a message?
> 
> Yes. And pretty-much everything else you might need in a static type 
> system; sets of alternates with a discriminator (discriminated unions), 
> arbitrary nesting of arrays of structures of arrays of structures, 
> strings of many and various character sets and encodings, enumerations, 
> numbers limited by range not storage size, etc.
> 
> Not all ASN.1 implementations are equal. This one is targeted at systems 
> that use no dynamic memory allocation, which is truly an excellent thing 
> - but you probably can't do JSON like that. It's used for the majority 
> of inter-module communications on ESA space missions, so it's pretty 
> well tested and reliable too.
> 
> The point is that the ASN.1 definitions are focussed on defining the 
> values that can be meaningfully transferred, not the possible values of 
> a set of representations. It's not fully powerful, but it's a lot better 
> than most programming languages.
> 
> I plan to emit ASN.1 from my Constellation Query Language tools at some 
> stage this year, see <http://dataconstellation.com/ActiveFacts/>. CQL 
> has a complete first-order logic constraint language, so it is even more 
> powerful than ASN.1; and every feature (definition, constraint) is also 
> expressible in controlled natural language and *compilable* from that 
> language.

Incredibly, it seems there isn't any ASN.1 PER codec for 
javascript/typescript (web-oriented) world. This limits the use to BER 
only :-(

[toc] | [prev] | [next] | [standalone]

#28659

On 6/6/19 7:12 pm, pozz wrote:
> Il 19/05/2019 01:12, Clifford Heath ha scritto:
>> On 19/5/19 12:03 am, pozz wrote:
>>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>>> On 17/5/19 5:20 pm, pozz wrote:
>>>>> I'd like to use JSON documents in my embedded project. I don't need 
>>>>> full parsing capabilities, because JSON could be very complex in 
>>>>> general.
>>>>>
>>>>> Do you know of something code generator?
>>>>
>>>> Ok. Latecomer, but I read most of the thread. Introduction: I've 
>>>> been writing C since 1979 and have a million lines of my code in 
>>>> production in mission-critical software on tens of millions of 
>>>> enterprise computers. Although these days my C is mostly C++, I use 
>>>> C++ mostly for the features that make it "a better C".
>>>>
>>>> Neither JSON nor C has a very strong type system. Neither is a good 
>>>> specification language. You should choose a specification language 
>>>> that does a good job of specifying your intent, and a compiler for 
>>>> it that emits C that handles one or more packing syntaxes such as 
>>>> JSON (if you must!)
>>>>
>>>> The European Space Agency has followed the lead of the ISO 
>>>> networking standards and built a compiler for ASN.1, which emits C 
>>>> code targeted at static-memory embedded systems. The downside: I 
>>>> don't think it has a JSON packing rule (the ASN.1 name for a 
>>>> transfer syntax). Other ASN.1 compilers do support JSON, but I don't 
>>>> know whether they target static-memory or embedded systems.
>>>>
>>>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>>>
>>>> It supports BER (Basic encoding rules, a compact & fast binary 
>>>> format), PER (Packed Encoding Rules, a bit-packed binary format that 
>>>> will be slower but tighter), XER (XML Encoding Rules, you can 
>>>> imagine!), and ACN, where you get to define the encoding yourself.
>>>>
>>>> If you wish to serve the community as well as yourself, write JER, 
>>>> JSON Encoding Rules and submit it.
>>>>
>>>> A quick search finds a proposed standard: 
>>>> <https://www.obj-sys.com/docs/JSONEncodingRules.pdf> and a tool that 
>>>> analyses a JSON fragment and produces a proposed ASN.1 definition 
>>>> for the shape of the fragment <https://asn1.io/json2asn/>
>>>>
>>>> So there is existing work, and no doubt community support for more.
>>>> You could do a lot worse than start with ASN1CC.
>>>
>>> ASN.1 is interested. I will look at it more deeply.
>>>
>>> Just a simple question. Are optional fields possible in a message?
>>
>> Yes. And pretty-much everything else you might need in a static type 
>> system; sets of alternates with a discriminator (discriminated 
>> unions), arbitrary nesting of arrays of structures of arrays of 
>> structures, strings of many and various character sets and encodings, 
>> enumerations, numbers limited by range not storage size, etc.
>>
>> Not all ASN.1 implementations are equal. This one is targeted at 
>> systems that use no dynamic memory allocation, which is truly an 
>> excellent thing - but you probably can't do JSON like that. It's used 
>> for the majority of inter-module communications on ESA space missions, 
>> so it's pretty well tested and reliable too.
>>
>> The point is that the ASN.1 definitions are focussed on defining the 
>> values that can be meaningfully transferred, not the possible values 
>> of a set of representations. It's not fully powerful, but it's a lot 
>> better than most programming languages.
>>
>> I plan to emit ASN.1 from my Constellation Query Language tools at 
>> some stage this year, see <http://dataconstellation.com/ActiveFacts/>. 
>> CQL has a complete first-order logic constraint language, so it is 
>> even more powerful than ASN.1; and every feature (definition, 
>> constraint) is also expressible in controlled natural language and 
>> *compilable* from that language.
> 
> Incredibly, it seems there isn't any ASN.1 PER codec for 
> javascript/typescript (web-oriented) world. This limits the use to BER 
> only :-(

ASN1CC has support for two languages, so I assume that it is set up for 
multi-language support. If that has been done well (and I haven't 
checked but I'd be surprised if it hasn't) it should be a fairly 
straight-forward matter to add support for a third language. Probably 
easier than implementing a fixed JS/TS processor for your desired 
message schema. I daresay you'd make many people grateful for that.

Clifford Heath.

[toc] | [prev] | [next] | [standalone]

#28660

Il 07/06/2019 01:11, Clifford Heath ha scritto:
> On 6/6/19 7:12 pm, pozz wrote:
>> Il 19/05/2019 01:12, Clifford Heath ha scritto:
>>> On 19/5/19 12:03 am, pozz wrote:
>>>> Il 18/05/2019 09:14, Clifford Heath ha scritto:
>>>>> On 17/5/19 5:20 pm, pozz wrote:
>>>>>> I'd like to use JSON documents in my embedded project. I don't 
>>>>>> need full parsing capabilities, because JSON could be very complex 
>>>>>> in general.
>>>>>>
>>>>>> Do you know of something code generator?
>>>>>
>>>>> Ok. Latecomer, but I read most of the thread. Introduction: I've 
>>>>> been writing C since 1979 and have a million lines of my code in 
>>>>> production in mission-critical software on tens of millions of 
>>>>> enterprise computers. Although these days my C is mostly C++, I use 
>>>>> C++ mostly for the features that make it "a better C".
>>>>>
>>>>> Neither JSON nor C has a very strong type system. Neither is a good 
>>>>> specification language. You should choose a specification language 
>>>>> that does a good job of specifying your intent, and a compiler for 
>>>>> it that emits C that handles one or more packing syntaxes such as 
>>>>> JSON (if you must!)
>>>>>
>>>>> The European Space Agency has followed the lead of the ISO 
>>>>> networking standards and built a compiler for ASN.1, which emits C 
>>>>> code targeted at static-memory embedded systems. The downside: I 
>>>>> don't think it has a JSON packing rule (the ASN.1 name for a 
>>>>> transfer syntax). Other ASN.1 compilers do support JSON, but I 
>>>>> don't know whether they target static-memory or embedded systems.
>>>>>
>>>>> The ESA-funded ASN1CC is here: <https://github.com/ttsiodras/asn1scc>.
>>>>>
>>>>> It supports BER (Basic encoding rules, a compact & fast binary 
>>>>> format), PER (Packed Encoding Rules, a bit-packed binary format 
>>>>> that will be slower but tighter), XER (XML Encoding Rules, you can 
>>>>> imagine!), and ACN, where you get to define the encoding yourself.
>>>>>
>>>>> If you wish to serve the community as well as yourself, write JER, 
>>>>> JSON Encoding Rules and submit it.
>>>>>
>>>>> A quick search finds a proposed standard: 
>>>>> <https://www.obj-sys.com/docs/JSONEncodingRules.pdf> and a tool 
>>>>> that analyses a JSON fragment and produces a proposed ASN.1 
>>>>> definition for the shape of the fragment <https://asn1.io/json2asn/>
>>>>>
>>>>> So there is existing work, and no doubt community support for more.
>>>>> You could do a lot worse than start with ASN1CC.
>>>>
>>>> ASN.1 is interested. I will look at it more deeply.
>>>>
>>>> Just a simple question. Are optional fields possible in a message?
>>>
>>> Yes. And pretty-much everything else you might need in a static type 
>>> system; sets of alternates with a discriminator (discriminated 
>>> unions), arbitrary nesting of arrays of structures of arrays of 
>>> structures, strings of many and various character sets and encodings, 
>>> enumerations, numbers limited by range not storage size, etc.
>>>
>>> Not all ASN.1 implementations are equal. This one is targeted at 
>>> systems that use no dynamic memory allocation, which is truly an 
>>> excellent thing - but you probably can't do JSON like that. It's used 
>>> for the majority of inter-module communications on ESA space 
>>> missions, so it's pretty well tested and reliable too.
>>>
>>> The point is that the ASN.1 definitions are focussed on defining the 
>>> values that can be meaningfully transferred, not the possible values 
>>> of a set of representations. It's not fully powerful, but it's a lot 
>>> better than most programming languages.
>>>
>>> I plan to emit ASN.1 from my Constellation Query Language tools at 
>>> some stage this year, see 
>>> <http://dataconstellation.com/ActiveFacts/>. CQL has a complete 
>>> first-order logic constraint language, so it is even more powerful 
>>> than ASN.1; and every feature (definition, constraint) is also 
>>> expressible in controlled natural language and *compilable* from that 
>>> language.
>>
>> Incredibly, it seems there isn't any ASN.1 PER codec for 
>> javascript/typescript (web-oriented) world. This limits the use to BER 
>> only :-(
> 
> ASN1CC has support for two languages, so I assume that it is set up for 
> multi-language support. If that has been done well (and I haven't 
> checked but I'd be surprised if it hasn't) it should be a fairly 
> straight-forward matter to add support for a third language. Probably 
> easier than implementing a fixed JS/TS processor for your desired 
> message schema. I daresay you'd make many people grateful for that.

Yes, but I haven't the know-how and time to add a third language to 
asn1scc. I need to find an off-the-shelf solution.

It seems there isn't a *proprietary* ASN1 PER Javascript implementation 
too (I found only JAVA support).

[toc] | [prev] | [next] | [standalone]

#28674

On 7/6/19 4:59 pm, pozz wrote:
> Il 07/06/2019 01:11, Clifford Heath ha scritto:
>> On 6/6/19 7:12 pm, pozz wrote:
>>> Il 19/05/2019 01:12, Clifford Heath ha scritto:
>>>> On 19/5/19 12:03 am, pozz wrote:
>>> Incredibly, it seems there isn't any ASN.1 PER codec for 
>>> javascript/typescript (web-oriented) world. This limits the use to 
>>> BER only :-(
>>
>> ASN1CC has support for two languages, so I assume that it is set up 
>> for multi-language support. If that has been done well (and I haven't 
>> checked but I'd be surprised if it hasn't) it should be a fairly 
>> straight-forward matter to add support for a third language. Probably 
>> easier than implementing a fixed JS/TS processor for your desired 
>> message schema. I daresay you'd make many people grateful for that.
> 
> Yes, but I haven't the know-how and time to add a third language to 
> asn1scc. I need to find an off-the-shelf solution.
> 
> It seems there isn't a *proprietary* ASN1 PER Javascript implementation 
> too (I found only JAVA support).

I have the know-how. Contact me if it's worth enough to you.

Clifford Heath.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.arch.embedded

csiph-web