Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #25214 > unrolled thread

Do you use serialization formats for communication?

Back to article view | Back to comp.arch.embedded

Contents

  Do you use serialization formats for communication? pozz <pozzugno@gmail.com> - 2016-10-20 00:22 +0200
    Re: Do you use serialization formats for communication? Dimiter_Popoff <dp@tgi-sci.com> - 2016-10-20 01:30 +0300
      Re: Do you use serialization formats for communication? Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> - 2016-10-19 23:23 +0000
    Re: Do you use serialization formats for communication? Don Y <blockedofcourse@foo.invalid> - 2016-10-19 16:35 -0700
    Re: Do you use serialization formats for communication? David Brown <david.brown@hesbynett.no> - 2016-10-20 09:40 +0200
      Re: Do you use serialization formats for communication? pozz <pozzugno@gmail.com> - 2016-10-20 13:52 +0200
        Re: Do you use serialization formats for communication? Tauno Voipio <tauno.voipio@notused.fi.invalid> - 2016-10-20 15:58 +0300
        Re: Do you use serialization formats for communication? David Brown <david.brown@hesbynett.no> - 2016-10-20 15:45 +0200
          Re: Do you use serialization formats for communication? pozz <pozzugno@gmail.com> - 2016-10-20 18:06 +0200
            Re: Do you use serialization formats for communication? David Brown <david.brown@hesbynett.no> - 2016-10-20 23:39 +0200
              Re: Do you use serialization formats for communication? George Neuner <gneuner2@comcast.net> - 2016-10-20 21:01 -0400
                Re: Do you use serialization formats for communication? David Brown <david.brown@hesbynett.no> - 2016-10-21 08:56 +0200
                  Re: Do you use serialization formats for communication? George Neuner <gneuner2@comcast.net> - 2016-10-23 07:05 -0400
                    Re: Do you use serialization formats for communication? David Brown <david.brown@hesbynett.no> - 2016-10-23 18:23 +0200
              Re: Do you use serialization formats for communication? pozz <pozzugno@gmail.com> - 2016-10-21 13:04 +0200
              Re: Do you use serialization formats for communication? pozz <pozzugno@gmail.com> - 2016-10-23 16:07 +0200
                Re: Do you use serialization formats for communication? David Brown <david.brown@hesbynett.no> - 2016-10-23 18:33 +0200
      Re: Do you use serialization formats for communication? kalvin.news@gmail.com - 2016-11-02 02:14 -0700
        Re: Do you use serialization formats for communication? David Brown <david.brown@hesbynett.no> - 2016-11-02 11:17 +0100
    Re: Do you use serialization formats for communication? kalvin.news@gmail.com - 2016-10-20 03:30 -0700
      Re: Do you use serialization formats for communication? pozz <pozzugno@gmail.com> - 2016-10-20 23:18 +0200
    Re: Do you use serialization formats for communication? Chris <xxx.syseng.yyy@gfsys.co.uk> - 2016-11-06 14:49 +0000
      Re: Do you use serialization formats for communication? Chris <xxx.syseng.yyy@gfsys.co.uk> - 2016-11-06 14:58 +0000
    Re: Do you use serialization formats for communication? Vincent vB <embedded@spam.com> - 2016-11-11 09:37 +0100
      Re: Do you use serialization formats for communication? Chris <xxx.syseng.yyy@gfsys.co.uk> - 2016-11-11 12:25 +0000
        Re: Do you use serialization formats for communication? Tim Wescott <seemywebsite@myfooter.really> - 2016-11-11 13:14 -0600
          Re: Do you use serialization formats for communication? Chris <xxx.syseng.yyy@gfsys.co.uk> - 2016-11-11 22:00 +0000
            Re: Do you use serialization formats for communication? Tim Wescott <seemywebsite@myfooter.really> - 2016-11-11 17:51 -0600
              Re: Do you use serialization formats for communication? Tom Gardner <spamjunk@blueyonder.co.uk> - 2016-11-12 00:07 +0000
        Re: Do you use serialization formats for communication? Vincent vB <embedded@spam.com> - 2016-11-29 15:54 +0100

Page 2 of 2 — ← Prev page 1 [2]

#25228

Il 20/10/2016 12:30, kalvin.news@gmail.com ha scritto:
 > [...]
> The FRAME-PAYLOAD is a sequence on PAYLOAD-ITEMs:
>
> PAYLOAD-ITEM:
> ITEM-ID
> ITEM-LENGTH
> ITEM-PAYLOAD.
>
> The ITEM-ID tells what the data item this is, the ITEM-LENGTH tells
> the length of the item payload and the ITEM-PAYLOAD contains the byte
> information for the item.
>
> Pretty simple message format and easy to parse. If the parser
> doesn't  recognize the item id, it knows how many bytes to skip for the next
 > item.

Yes, of course this *is* a "self-made" serialization format. There are 
many other that are similar, but introduce many other advantages.

For example, consider MessagePack[1] (I don't work for MessagePack).

You can encode many types of data. Your serialization format can be 
encoded as a MessagePack map: a sequence of key-value pairs (a 
dictionary in Python).

As in your format, MessagePack encodes the key (your ITEM-ID), the value 
type (similar to your ITEM-LENGTH) and the value (your ITEM-PAYLOAD). 
The value type automatically specify the length and the object type 
(integer, boolean, an array...).

For example, the dictionary { 3: 5 } (only one item, 3 as ID and 5 as 
value) is encoded as three bytes { 0x81, 0x03, 0x05 }.
If the value is much higher, for example { 3: 1000000 }, the encoded 
stream is 7-bytes long { 0x81, 0x03, 0xCE, 0x00, 0x0F, 0x42, 0x40 }.
The decoder automatically understand what is the key (type, length and 
value) and the value (type, length and value).

If you want, you can use a string for the keys (all keys or some of 
them). Or you can encode an array instead of a map/dictionary.


[1]http://msgpack.org/

[toc] | [prev] | [next] | [standalone]

#25406

On 10/19/16 22:22, pozz wrote:
> I often have the need to exchange some data between two or more MCUs. I
> usually use I2C or UART as physical layers.
>
> Normally I design a simple protocol between the MCUs: one framing
> mechanism (Start Of Frame, End Of Frame), one integrity check mechanism
> (CRC), and so on.
>
> The payload is statically defined between the two MCUs:
> - first byte is the version
> - second byte is the voltage monitoring level
> - third and fourt bytes are some flags
> - ... and so on
>
> As you can understand, both MCUs *must* know and agree about that
> protocol format. However during the lifetime of the product, I need to
> add some functionality or fix some bugs and those activites can lead to
> a review of the protocol format (maybe i need two bytes for the voltage
> level). Sometime, the two MCUs have a different version with a different
> protocol format implementation. In order to avoid protocol
> incompatibility, they all knows about the protocol formats used before,
> so they can adapt the parsing function to the real current protocol format.
> As you can understand, it could be a trouble.
>
> So I'm thinking to use a "self-descriptive" serializer protocol format,
> such as Protobuf, Message Pack, BSON and so on.
>
> Do you use one serialization format? Which one?
>
> Of course, it should be simple to implement (in transmission/encoding
> and reception/decoding) in a small embedded MCU in C language, without
> dynamic memory support.

I'm not sure you need a complex format here, if you think of the
problem in two layers, keeping the protocol layer transparent to data.

A simple frame format could be:

Start of frame byte
Data length, N
Data, N bytes
Checksum or crc
End frame byte

You then write a simple state machine to verify checksum, extract
the data and its length. Pass that to the protocol layer, which
knows where to where to look in the data for it's revision level.
To maintain compatability, any new parameters are tagged on to the
end of the existing data and the data length increased to suit.

Either that, or negotiation between ends to agree capabilities, but
that's much more complex and you should be able to avoid it...

Regards,

Chris

[toc] | [prev] | [next] | [standalone]

#25408

On 11/06/16 14:49, Chris wrote:

>

> the data and its length. Pass that to the protocol layer, which
                                             ^^^^^^^^

Sorry, typo, should have been "decode" layer.

[toc] | [prev] | [next] | [standalone]

#25474

Hi Pozz,

On 20-10-2016 at 0:22, pozz wrote:

> I often have the need to exchange some data between two or more MCUs. I

> usually use I2C or UART as physical layers.

>
> So I'm thinking to use a "self-descriptive" serializer protocol format,
> such as Protobuf, Message Pack, BSON and so on.
>

We use protobuf for our communication (nanopb). We're very happy with it.


> Do you use one serialization format? Which one?
>
> Of course, it should be simple to implement (in transmission/encoding
> and reception/decoding) in a small embedded MCU in C language, without
> dynamic memory support.

We don't use dynamic memory with nanopb. However, we do have some 
'scratch space', which is used as a sort of stack for unknown length 
data, but it is cleared after each time the data is send.

Vincent

[toc] | [prev] | [next] | [standalone]

#25475

On 11/11/16 08:37, Vincent vB wrote:

>
> We don't use dynamic memory with nanopb. However, we do have some
> 'scratch space', which is used as a sort of stack for unknown length
> data, but it is cleared after each time the data is send.
>
> Vincent

Hi,

If you know that you will never need more than a certain buffer
size, then it can be statically allocated at startup. Where that
isn't known for sure, include instrumentation to check for a
high water mark, then run the code worst case in a test harness
to find out what it's actually using.

Chris

[toc] | [prev] | [next] | [standalone]

#25476

On Fri, 11 Nov 2016 12:25:14 +0000, Chris wrote:

> On 11/11/16 08:37, Vincent vB wrote:
> 
> 
>> We don't use dynamic memory with nanopb. However, we do have some
>> 'scratch space', which is used as a sort of stack for unknown length
>> data, but it is cleared after each time the data is send.
>>
>> Vincent
> 
> Hi,
> 
> If you know that you will never need more than a certain buffer size,
> then it can be statically allocated at startup. Where that isn't known
> for sure, include instrumentation to check for a high water mark, then
> run the code worst case in a test harness to find out what it's actually
> using.
> 
> Chris

Yick.  And then get blind-sided when reality hoses you.

You either need a protocol that, end to end, guarantees some maximum 
buffer size, or you need a system that's tolerant to communications 
occasionally breaking down.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

I'm looking for work -- see my website!

[toc] | [prev] | [next] | [standalone]

#25477

On 11/11/16 19:14, Tim Wescott wrote:

>
> Yick.  And then get blind-sided when reality hoses you.

Sounds pretty meaningless to me :-).

>
> You either need a protocol that, end to end, guarantees some maximum
> buffer size, or you need a system that's tolerant to communications
> occasionally breaking down.
>

Buffer sizes for many embedded systems are known, but worst case values
are not always predictable, so you can end up allocating far more
memory than is needed, which isn't helpful with limited memory
systems. If you are building thousands of an item, you don't choose a
micro with more ram than you need, since ram size can be one of the
major price differentiators. Fine if you are running on PC hardware,
Linux, whatever with Mbytes of ram, but most embedded systems don't
have such luxury.

Typical recent example was a system that received commands via rs423
at 9600 with circular buffer fifos in and out. Did some quick sums and
allocated bigger queues than we needed, with water marks a couple of
points, then drove the system hard to see what was actually being
used. Turned out that we needed far less than we thought.

Many embedded systems need that sort of fine tuning to optimise resource
usage against cost. It's also good engineering practice from an
efficiency point of view...

Chris

[toc] | [prev] | [next] | [standalone]

#25479

On Fri, 11 Nov 2016 22:00:32 +0000, Chris wrote:

> On 11/11/16 19:14, Tim Wescott wrote:
> 
> 
>> Yick.  And then get blind-sided when reality hoses you.
> 
> Sounds pretty meaningless to me :-).
> 
> 
>> You either need a protocol that, end to end, guarantees some maximum
>> buffer size, or you need a system that's tolerant to communications
>> occasionally breaking down.
>>
>>
> Buffer sizes for many embedded systems are known, but worst case values
> are not always predictable, so you can end up allocating far more memory
> than is needed, which isn't helpful with limited memory systems. If you
> are building thousands of an item, you don't choose a micro with more
> ram than you need, since ram size can be one of the major price
> differentiators. Fine if you are running on PC hardware, Linux, whatever
> with Mbytes of ram, but most embedded systems don't have such luxury.
> 
> Typical recent example was a system that received commands via rs423 at
> 9600 with circular buffer fifos in and out. Did some quick sums and
> allocated bigger queues than we needed, with water marks a couple of
> points, then drove the system hard to see what was actually being used.
> Turned out that we needed far less than we thought.
> 
> Many embedded systems need that sort of fine tuning to optimise resource
> usage against cost. It's also good engineering practice from an
> efficiency point of view...

Yes, if you're going into things blind it's good engineering practice to 
do things by measurement.

But my point is that if you can design the protocol, or if you have good 
specifications on it, it's _far better_ engineering practice to break out 
your pencil and paper and do the proof from first principles.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

I'm looking for work -- see my website!

[toc] | [prev] | [next] | [standalone]

#25480

On 11/11/16 23:51, Tim Wescott wrote:
> But my point is that if you can design the protocol, or if you have good
> specifications on it, it's _far better_ engineering practice to break out
> your pencil and paper and do the proof from first principles.

Oh, how old fashioned. Surely you realise that it is possible
and desirable to test quality into a product :(

[toc] | [prev] | [next] | [standalone]

#25517

On 11-11-2016 at 13:25, Chris wrote:
> On 11/11/16 08:37, Vincent vB wrote:
>
>>
>> We don't use dynamic memory with nanopb. However, we do have some
>> 'scratch space', which is used as a sort of stack for unknown length
>> data, but it is cleared after each time the data is send.
>>
>> Vincent
>
> Hi,
>
> If you know that you will never need more than a certain buffer
> size, then it can be statically allocated at startup. Where that
> isn't known for sure, include instrumentation to check for a
> high water mark, then run the code worst case in a test harness
> to find out what it's actually using.
>
> Chris

Hi Chris,

We know exactly how much data is written at most. The scratch space is 
dimensioned for this case. Otherwise you'd need to test it.

Vincent

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.arch.embedded

csiph-web