Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #24803 > unrolled thread

What is your strategy to save non volatile data?

Back to article view | Back to comp.arch.embedded

Contents

  What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-11 14:48 +0200
    Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 06:27 -0700
    Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-11 16:12 +0200
      Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 07:53 -0700
        Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 08:27 -0700
          Re: What is your strategy to save non volatile data? bitrex <bitrex@de.lete.earthlink.net> - 2016-08-11 12:11 -0400
            Re: What is your strategy to save non volatile data? bitrex <bitrex@de.lete.earthlink.net> - 2016-08-11 12:12 -0400
              Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 11:35 -0700
                Re: What is your strategy to save non volatile data? bitrex <bitrex@de.lete.earthlink.net> - 2016-08-11 20:59 -0400
            Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 11:33 -0700
          Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-12 00:21 +0200
            Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 15:40 -0700
        Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-11 23:07 +0200
          Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-12 00:17 +0200
            Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 15:30 -0700
              Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-12 09:02 +0200
                Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-12 09:29 -0700
              Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 11:54 +0200
                Re: What is your strategy to save non volatile data? John Devereux <john@devereux.me.uk> - 2016-08-12 13:14 +0100
                  Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 15:22 +0200
                Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-12 09:40 -0700
                  Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 22:40 +0200
                    Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-13 09:44 -0700
                      Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-14 23:01 +0200
            Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 09:52 +0200
              Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-12 16:30 +0200
                Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 17:09 +0200
          Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 15:36 -0700
          Re: What is your strategy to save non volatile data? Paul <paul@pcserviceselectronics.co.uk> - 2016-08-12 03:21 +0100
      Re: What is your strategy to save non volatile data? bitrex <bitrex@de.lete.earthlink.net> - 2016-08-11 12:19 -0400
        Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-11 23:49 +0200
          Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 15:37 -0700
      Re: What is your strategy to save non volatile data? John Devereux <john@devereux.me.uk> - 2016-08-11 20:50 +0100
      Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-12 00:16 +0200
        Re: What is your strategy to save non volatile data? Paul <paul@pcserviceselectronics.co.uk> - 2016-08-12 03:34 +0100
        Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 11:59 +0200
    Re: What is your strategy to save non volatile data? Paul Rubin <no.email@nospam.invalid> - 2016-08-11 11:29 -0700
    Re: What is your strategy to save non volatile data? Les Cargill <lcargill99@comcast.com> - 2016-08-12 12:15 -0500
    Re: What is your strategy to save non volatile data? Joe Chisolm <jchisolm6@earthlink.net> - 2016-08-12 12:10 -0500
      Re: What is your strategy to save non volatile data? Paul <paul@pcserviceselectronics.co.uk> - 2016-08-12 22:00 +0100
    Re: What is your strategy to save non volatile data? Tim Wescott <tim@seemywebsite.com> - 2016-08-13 12:06 -0500
      Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-13 23:42 +0200
        Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-13 15:55 -0700
        Re: What is your strategy to save non volatile data? Paul <paul@pcserviceselectronics.co.uk> - 2016-08-14 00:08 +0100
        Re: What is your strategy to save non volatile data? Tim Wescott <tim@seemywebsite.com> - 2016-08-13 19:06 -0500
          Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-22 09:46 +0200

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#24840

On 8/12/2016 2:54 AM, David Brown wrote:
> On 12/08/16 00:30, Don Y wrote:
>> On 8/11/2016 3:17 PM, pozz wrote:
>>
>> [much elided]
>>
>>> "Everything in one structure" means you have a global (as the scope in C)
>>> struct definition that is accessed by every piece of code in the
>>> project, even
>>> if they are very different section of the project.
>>
>> No.  It only has to be *effectively* global.
>>
>> E.g., you can have a global "update_parameter()" that takes a
>> "key" (name for a parameter -- this could be a literal string
>> *or* just a small integer) and a parameter value (or pointer to
>> a parameter value).  Update_parameter() then moves the "value"
>> into the referenced portion of the "parameters to be saved struct".
>
> First, /don't/ use strings for that sort of thing unless you have no
> other choice.  And don't use arbitrary integers.  Use an enum, that is
> defined in one place.  That way you get it right.  Your code refers to
> the parameters using sensible names, rather than numbers, you don't get
> overlap or gaps (unless you want them, of course), and you can easily
> change things when you need to.  And when one part of your program
> writes parameter "backgroundColour" and another part reads parameter
> "backgroundColor", you get a compile-time error rather than weird
> behaviour at run-time.

You throw an error at run-time -- because you *still* have to do
run-time checks on parameters and their values!

And, it would be silly to type out the name of the key in each
invocation.  Instead, you'd use:

#define PARAM_BACKGROUND_COLOR ("bacKgRouNdCoLOR")
#define PARAM_FOREGROUND_COLOR ("4GrouNDcoLOUr")
etc.

How is this any different from:

#define PARAM_BACKGROUND_COLOR (1)
#define PARAM_FOREGROUND_COLOR (2)
etc.?

Enums, IMO, are a bad idea -- it's far too easy to slip a new one
"in the middle" and effectively bodge the keys associated with all
those that follow.  I.e., a software revision changes:

{PARAM_BACKGROUND_COLOR, PARAM_FOREGROUND_COLOR, ...}
to
{PARAM_BACKGROUND_COLOR, PARAM_BORDER_COLOR, PARAM_FOREGROUND_COLOR, ...}

and now any "legacy" configuration files are effectively unreadable.

(How do you refer to "old_PARAM_FOREGROUND_COLOR" in the code that
attempts to patch-up a legacy configuration dataset in light of
the *new* "PARAM_FOREGROUND_COLOR"?  I.e., you end up having to create
another enumeration for the legacy representation -- using DIFFERENT
names!)

By contrast, if you manually enumerate them, you are keenly aware of
the original (name,key) mapping.  Add new names to the end of the
list (who cares if "COLOR_X" and "COLOR_Y" have adjacent keys?!).
And, you can simply put a comment after any keys that have been obsoleted
over time:

#define PARAM_VARIABLE_X (...)   // obsolete as per rev x.y.z

Strings have value in that you can *see* them in logs:
    log( "Updating parameter %s to value %s", param, value )
instead of:
    log( "updating parameter %d to value %d", param, value )
or (ick):
    log( "Updating parameter %s to value %s",
         param_to_string(param), value_to_string(value) )

Integers (and enums) suffer in poorly typed languages:
    update_parameter(PARAM_BACKGROUND_COLOR, VALUE_BROCOLLI);

I.e., there's no way to ensure only the values valid for
PARAM_BACKGROUND_COLOR are applied to PARAM_BACKGROUND_COLOR!
The above function call may succeed (depends on whether
VALUE_BROCOLLI happens to coincide with a value for a
PARAM_BACKGROUND_COLOR) -- or not.  I.e., you still have to
rely on run-time checks; you can't do it all at compile time.

In my current project, parameters are just fields in an RDBMS table.
So, the RDBMS *enforces* "correct" (in the sense of "valid") values.
Can't set Background_Color to "Blue" unless "Blue" is a valid choice
for *that* field.  And, surely can't set it to "Brocolli"!  :>

And, as the RDBMS is charged with maintaining the integrity of this
data, there's no need to *check* it when reading it from the RDBMS
(it was checked "on the way in"!)

As everything is SQL, I *see* "Blue" not some arbitrary
*encoding* of PARAM_BACKGROUND_COLOR_BLUE (which might differ
from PARAM_FOREGROUND_COLOR_BLUE -- even though they are the
same color!)

> But passing the data back and forth through functions like
> "update_parameter" and "get_parameter" is an alternative to accessing
> the struct directly, and can be easier to manage.  It is certainly
> convenient if you want to trigger automatic saving of parameters.

It creates a private namespace for *just* those parameters.
And, also puts a "gatekeeper" in place; limiting "who" can
access and alter the parameters stored therein (e.g., password)
as well as *when* they can be altered.

Even if you don't have any run-time authentication mechanisms,
its easy to stub update-parameter() and have it announce each time
it is invoked (during development) than to have to set a "watchpoint"
on individual parameter struct members.

>> In this way, the definition of the struct can remain opaque.
>> The same sort of "accessor" can be used to retrieve those
>> parameters from the struct -- without exposing the struct
>> or the manner in which the individual parameters are "encoded" (!)
>> in the struct.
>
> Yes.
>
>> E.g., you can pack a bunch of (potentially unrelated) parameters
>> into a bitfield within that struct.  Data coming *into* it gets
>> massaged to manipulate the appropriate fields.  Likewise for data
>> coming *out*.
>
> That sounds like a complication that is unlikely to be worth the effort,
> but it's possible.  In many cases its enough to simply say "all
> parameters are 32 bits - it's up to the app if this is 4 characters, a
> float, an integer, etc.".

Depends on resources available *and* how parameters are accessed.
Surely easier to fetch a *set* of related small fields in one
access than it is to have to fetch several individual fields
(each consisting of lots of 0 bits!)

It also hides the representation from the caller.  Should I store
the time-of-last-update as a time_t?  Or, as some sort of string?
Or, as something that really only makes sense to the routine that
does the updates?

>> Finally, because this represents a monitor, of sorts, you can ensure
>> atomic operations on that "data to be preserved" -- up to and including
>> an action that automatically saves the struct to persistent store
>> when power is failing (the rest of the time, "updates" happen to
>> the "in-RAM" copy of the struct so you don't incur the costs of
>> updating FLASH more often than necessary)
>
> Also true - it's always important to make sure your data is consistent
> at all times.

[toc] | [prev] | [next] | [standalone]

#24845

On 12/08/16 18:40, Don Y wrote:
> On 8/12/2016 2:54 AM, David Brown wrote:
>> On 12/08/16 00:30, Don Y wrote:
>>> On 8/11/2016 3:17 PM, pozz wrote:
>>>
>>> [much elided]
>>>
>>>> "Everything in one structure" means you have a global (as the scope
>>>> in C)
>>>> struct definition that is accessed by every piece of code in the
>>>> project, even
>>>> if they are very different section of the project.
>>>
>>> No.  It only has to be *effectively* global.
>>>
>>> E.g., you can have a global "update_parameter()" that takes a
>>> "key" (name for a parameter -- this could be a literal string
>>> *or* just a small integer) and a parameter value (or pointer to
>>> a parameter value).  Update_parameter() then moves the "value"
>>> into the referenced portion of the "parameters to be saved struct".
>>
>> First, /don't/ use strings for that sort of thing unless you have no
>> other choice.  And don't use arbitrary integers.  Use an enum, that is
>> defined in one place.  That way you get it right.  Your code refers to
>> the parameters using sensible names, rather than numbers, you don't get
>> overlap or gaps (unless you want them, of course), and you can easily
>> change things when you need to.  And when one part of your program
>> writes parameter "backgroundColour" and another part reads parameter
>> "backgroundColor", you get a compile-time error rather than weird
>> behaviour at run-time.
>
> You throw an error at run-time -- because you *still* have to do
> run-time checks on parameters and their values!
>

The difference here is that with checks on parameter values, you can 
(normally) just set them to safe default values if they occur, and it is 
something that typically would never happen.

But if you have a spelling mistake in your texts, your program is likely 
to silently ignore storage of changes in parameters.

> And, it would be silly to type out the name of the key in each
> invocation.  Instead, you'd use:
>
> #define PARAM_BACKGROUND_COLOR ("bacKgRouNdCoLOR")
> #define PARAM_FOREGROUND_COLOR ("4GrouNDcoLOUr")
> etc.
>
> How is this any different from:
>
> #define PARAM_BACKGROUND_COLOR (1)
> #define PARAM_FOREGROUND_COLOR (2)
> etc.?

It is different, because it is - IMHO - daft.  Working with strings like 
this is vastly less efficient, and more error prone.   And if you've got 
a good place to write out these #defines, then you are better off using 
an enum there with numerical values:

enum {
	paramBackgroundColour,
	paramForegroundColour,
	etc.
}

It's far more efficient, easier on the eye, easier to get IDE help in 
automatic completion, and easier for the compiler to check (such as 
compiler warnings if you use switch on an enum but forget some cases). 
And if you are using C++ rather than C, you can use type-safe strong 
enums, as well as have better scoping and namespace usage.


>
> Enums, IMO, are a bad idea -- it's far too easy to slip a new one
> "in the middle" and effectively bodge the keys associated with all
> those that follow.  I.e., a software revision changes:
>
> {PARAM_BACKGROUND_COLOR, PARAM_FOREGROUND_COLOR, ...}
> to
> {PARAM_BACKGROUND_COLOR, PARAM_BORDER_COLOR, PARAM_FOREGROUND_COLOR, ...}
>
> and now any "legacy" configuration files are effectively unreadable.

If that is a concern (and it sometimes /is/ a concern), give the enum 
explicit values:

enum {
	paramBackgroundColour = 1,
	paramForegroundColour = 2,
	etc.
}


>
> (How do you refer to "old_PARAM_FOREGROUND_COLOR" in the code that
> attempts to patch-up a legacy configuration dataset in light of
> the *new* "PARAM_FOREGROUND_COLOR"?  I.e., you end up having to create
> another enumeration for the legacy representation -- using DIFFERENT
> names!)

And how is that worse than using different names with #define'd values?

>
> By contrast, if you manually enumerate them, you are keenly aware of
> the original (name,key) mapping.  Add new names to the end of the
> list (who cares if "COLOR_X" and "COLOR_Y" have adjacent keys?!).
> And, you can simply put a comment after any keys that have been obsoleted
> over time:
>
> #define PARAM_VARIABLE_X (...)   // obsolete as per rev x.y.z
>
> Strings have value in that you can *see* them in logs:
>     log( "Updating parameter %s to value %s", param, value )
> instead of:
>     log( "updating parameter %d to value %d", param, value )
> or (ick):
>     log( "Updating parameter %s to value %s",
>          param_to_string(param), value_to_string(value) )
>
> Integers (and enums) suffer in poorly typed languages:
>     update_parameter(PARAM_BACKGROUND_COLOR, VALUE_BROCOLLI);
>
> I.e., there's no way to ensure only the values valid for
> PARAM_BACKGROUND_COLOR are applied to PARAM_BACKGROUND_COLOR!
> The above function call may succeed (depends on whether
> VALUE_BROCOLLI happens to coincide with a value for a
> PARAM_BACKGROUND_COLOR) -- or not.  I.e., you still have to
> rely on run-time checks; you can't do it all at compile time.
>
> In my current project, parameters are just fields in an RDBMS table.
> So, the RDBMS *enforces* "correct" (in the sense of "valid") values.
> Can't set Background_Color to "Blue" unless "Blue" is a valid choice
> for *that* field.  And, surely can't set it to "Brocolli"!  :>
>
> And, as the RDBMS is charged with maintaining the integrity of this
> data, there's no need to *check* it when reading it from the RDBMS
> (it was checked "on the way in"!)
>
> As everything is SQL, I *see* "Blue" not some arbitrary
> *encoding* of PARAM_BACKGROUND_COLOR_BLUE (which might differ
> from PARAM_FOREGROUND_COLOR_BLUE -- even though they are the
> same color!)

I don't disagree that storing things as strings can be useful sometimes. 
  But we are not talking about big, wasteful PC software - we are 
talking about a small number of parameters in a small embedded program. 
  SQL is too big by orders of magnitude for such a system.  It would a 
different matter if we were talking about an embedded linux system, for 
example - then you just write your code in Python, use a JSON string to 
hold a dict of the parameters, and you've got the system working in an 
hour of developer time.  Different tools for different tasks.

>
>> But passing the data back and forth through functions like
>> "update_parameter" and "get_parameter" is an alternative to accessing
>> the struct directly, and can be easier to manage.  It is certainly
>> convenient if you want to trigger automatic saving of parameters.
>
> It creates a private namespace for *just* those parameters.
> And, also puts a "gatekeeper" in place; limiting "who" can
> access and alter the parameters stored therein (e.g., password)
> as well as *when* they can be altered.
>
> Even if you don't have any run-time authentication mechanisms,
> its easy to stub update-parameter() and have it announce each time
> it is invoked (during development) than to have to set a "watchpoint"
> on individual parameter struct members.
>
>>> In this way, the definition of the struct can remain opaque.
>>> The same sort of "accessor" can be used to retrieve those
>>> parameters from the struct -- without exposing the struct
>>> or the manner in which the individual parameters are "encoded" (!)
>>> in the struct.
>>
>> Yes.
>>
>>> E.g., you can pack a bunch of (potentially unrelated) parameters
>>> into a bitfield within that struct.  Data coming *into* it gets
>>> massaged to manipulate the appropriate fields.  Likewise for data
>>> coming *out*.
>>
>> That sounds like a complication that is unlikely to be worth the effort,
>> but it's possible.  In many cases its enough to simply say "all
>> parameters are 32 bits - it's up to the app if this is 4 characters, a
>> float, an integer, etc.".
>
> Depends on resources available *and* how parameters are accessed.
> Surely easier to fetch a *set* of related small fields in one
> access than it is to have to fetch several individual fields
> (each consisting of lots of 0 bits!)

If you have the resources to use string keys for the parameters, you 
have the resources for using individual data items for your bits.

And if you have a number of related bits to track together, then use a 
flags parameter - have the application module keep the flags in a uint 
and pass them all together.

>
> It also hides the representation from the caller.  Should I store
> the time-of-last-update as a time_t?  Or, as some sort of string?
> Or, as something that really only makes sense to the routine that
> does the updates?
>
>>> Finally, because this represents a monitor, of sorts, you can ensure
>>> atomic operations on that "data to be preserved" -- up to and including
>>> an action that automatically saves the struct to persistent store
>>> when power is failing (the rest of the time, "updates" happen to
>>> the "in-RAM" copy of the struct so you don't incur the costs of
>>> updating FLASH more often than necessary)
>>
>> Also true - it's always important to make sure your data is consistent
>> at all times.
>

[toc] | [prev] | [next] | [standalone]

#24849

On 8/12/2016 1:40 PM, David Brown wrote:
> On 12/08/16 18:40, Don Y wrote:
>> On 8/12/2016 2:54 AM, David Brown wrote:
>>> On 12/08/16 00:30, Don Y wrote:
>>>> On 8/11/2016 3:17 PM, pozz wrote:
>>>>
>>>> [much elided]
>>>>
>>>>> "Everything in one structure" means you have a global (as the scope
>>>>> in C)
>>>>> struct definition that is accessed by every piece of code in the
>>>>> project, even
>>>>> if they are very different section of the project.
>>>>
>>>> No.  It only has to be *effectively* global.
>>>>
>>>> E.g., you can have a global "update_parameter()" that takes a
>>>> "key" (name for a parameter -- this could be a literal string
>>>> *or* just a small integer) and a parameter value (or pointer to
>>>> a parameter value).  Update_parameter() then moves the "value"
>>>> into the referenced portion of the "parameters to be saved struct".
>>>
>>> First, /don't/ use strings for that sort of thing unless you have no
>>> other choice.  And don't use arbitrary integers.  Use an enum, that is
>>> defined in one place.  That way you get it right.  Your code refers to
>>> the parameters using sensible names, rather than numbers, you don't get
>>> overlap or gaps (unless you want them, of course), and you can easily
>>> change things when you need to.  And when one part of your program
>>> writes parameter "backgroundColour" and another part reads parameter
>>> "backgroundColor", you get a compile-time error rather than weird
>>> behaviour at run-time.
>>
>> You throw an error at run-time -- because you *still* have to do
>> run-time checks on parameters and their values!
>
> The difference here is that with checks on parameter values, you can (normally)
> just set them to safe default values if they occur, and it is something that
> typically would never happen.
>
> But if you have a spelling mistake in your texts, your program is likely to
> silently ignore storage of changes in parameters.

Why would your program silently ignore them?  Don't you write ROBUST code?

If you tried to set parameter 47 to 982 -- but there were only 42 supported
parameters, would your code "silently ignore" this?  Or, if parameter 47
*did* exist but expected to be set to a string, would you ignore the
attempt to set it to a numeric?

>> And, it would be silly to type out the name of the key in each
>> invocation.  Instead, you'd use:
>>
>> #define PARAM_BACKGROUND_COLOR ("bacKgRouNdCoLOR")
>> #define PARAM_FOREGROUND_COLOR ("4GrouNDcoLOUr")
>> etc.
>>
>> How is this any different from:
>>
>> #define PARAM_BACKGROUND_COLOR (1)
>> #define PARAM_FOREGROUND_COLOR (2)
>> etc.?
>
> It is different, because it is - IMHO - daft.  Working with strings like this
> is vastly less efficient, and more error prone.   And if you've got a good

So, if you opted to use an int as a tag/key, you'd insist on using values
like 1, 2, 3, 4, ... instead of "BACK", "FORE", "BORD", for your tags?
(i.e., each of these are 32b integers)

And, force yourself to maintain a cheat sheet mapping *arbitrary* integers
to these interpretations -- just because your grade school teacher taught
you to start counting from '1'?

Using strings is "daft" when you don't have the resources (memory/MIPS)
to do so.  Forcing yourself to use enums/integers when you *do* have
the resources is "daft"!

(why isn't the windows registry just one big set of binary tuples?  why
aren't UN*X configuration files encoded in some magical, highly space and
time efficient manner -- with a suitable "decoder" to allow for the
*infrequent* human use?  why do databases give names to fields instead
of just 'field1', 'field2', etc.?)

> place to write out these #defines, then you are better off using an enum there
> with numerical values:
>
> enum {
>      paramBackgroundColour,
>      paramForegroundColour,
>      etc.
> }
>
> It's far more efficient, easier on the eye, easier to get IDE help in automatic
> completion, and easier for the compiler to check (such as compiler warnings if
> you use switch on an enum but forget some cases). And if you are using C++
> rather than C, you can use type-safe strong enums, as well as have better
> scoping and namespace usage.

And, all it ADDS is the requirement that you insert this extra step
(and keep it consistently, everywhere you use it).

I just got notified of an "error 27".  What's that mean?  What's
the symbolic label associated with that numeric value?  *Then*,
what does that symbolic label actually *mean*?

[Isn't this the same argument applied to "error codes"?]

>> Enums, IMO, are a bad idea -- it's far too easy to slip a new one
>> "in the middle" and effectively bodge the keys associated with all
>> those that follow.  I.e., a software revision changes:
>>
>> {PARAM_BACKGROUND_COLOR, PARAM_FOREGROUND_COLOR, ...}
>> to
>> {PARAM_BACKGROUND_COLOR, PARAM_BORDER_COLOR, PARAM_FOREGROUND_COLOR, ...}
>>
>> and now any "legacy" configuration files are effectively unreadable.
>
> If that is a concern (and it sometimes /is/ a concern), give the enum explicit
> values:
>
> enum {
>      paramBackgroundColour = 1,
>      paramForegroundColour = 2,
>      etc.
> }

This is then another maintenance issue:  ensuring that paramBorderColor gets
MANUALLY assigned the correct value regardless of where it appears in
the list of enumerations:
    enum {
       paramBackgroundColour = 1,
       paramForegroundColour = 2,
       paramBorderColor = 27,
       paramShadowColor = 3,   // return to the value that would have been used
                               // had not paramBorderColor been introduced

or, alternatively:
    enum {
       paramBackgroundColour = 1,
       paramForegroundColour = 2,
       paramShadowColor,   // 3
       ...   // manually skip over 24 labels (assuming none of them explicitly
             // alter their default values) to ensure we're at "27" when the
             // next label (border color) is encountered
       paramBorderColor,   // 27

>> (How do you refer to "old_PARAM_FOREGROUND_COLOR" in the code that
>> attempts to patch-up a legacy configuration dataset in light of
>> the *new* "PARAM_FOREGROUND_COLOR"?  I.e., you end up having to create
>> another enumeration for the legacy representation -- using DIFFERENT
>> names!)
>
> And how is that worse than using different names with #define'd values?

Because there is no "old_" value!  The "old" and "new" are exactly the same!
You just add new values to the end and obsolete values "in the middle".

#define PARAM_BACKGROUND_COLOR (3)
#define PARAM_FOREGROUND_COLOR (4)
#define PARAM_BORDER_COLOR (5)
#define PARAM_SHADOW_COLOR (6)

becomes:

#define PARAM_BACKGROUND_COLOR (3)
#define PARAM_FOREGROUND_COLOR (4)
#define PARAM_BORDER_COLOR (5)         // no longer used
#define PARAM_SHADOW_COLOR (6)

when the border color parameter is obsoleted.  So, if you encounter
a "configuration struct" that references parameter 5, you know that
it is referencing the obsolete border color parameter -- not the
*new* "highlight color" that took its place in the list of numerical
parameter identifier values.

>> By contrast, if you manually enumerate them, you are keenly aware of
>> the original (name,key) mapping.  Add new names to the end of the
>> list (who cares if "COLOR_X" and "COLOR_Y" have adjacent keys?!).
>> And, you can simply put a comment after any keys that have been obsoleted
>> over time:
>>
>> #define PARAM_VARIABLE_X (...)   // obsolete as per rev x.y.z
>>
>> Strings have value in that you can *see* them in logs:
>>     log( "Updating parameter %s to value %s", param, value )
>> instead of:
>>     log( "updating parameter %d to value %d", param, value )
>> or (ick):
>>     log( "Updating parameter %s to value %s",
>>          param_to_string(param), value_to_string(value) )
>>
>> Integers (and enums) suffer in poorly typed languages:
>>     update_parameter(PARAM_BACKGROUND_COLOR, VALUE_BROCOLLI);
>>
>> I.e., there's no way to ensure only the values valid for
>> PARAM_BACKGROUND_COLOR are applied to PARAM_BACKGROUND_COLOR!
>> The above function call may succeed (depends on whether
>> VALUE_BROCOLLI happens to coincide with a value for a
>> PARAM_BACKGROUND_COLOR) -- or not.  I.e., you still have to
>> rely on run-time checks; you can't do it all at compile time.
>>
>> In my current project, parameters are just fields in an RDBMS table.
>> So, the RDBMS *enforces* "correct" (in the sense of "valid") values.
>> Can't set Background_Color to "Blue" unless "Blue" is a valid choice
>> for *that* field.  And, surely can't set it to "Brocolli"!  :>
>>
>> And, as the RDBMS is charged with maintaining the integrity of this
>> data, there's no need to *check* it when reading it from the RDBMS
>> (it was checked "on the way in"!)
>>
>> As everything is SQL, I *see* "Blue" not some arbitrary
>> *encoding* of PARAM_BACKGROUND_COLOR_BLUE (which might differ
>> from PARAM_FOREGROUND_COLOR_BLUE -- even though they are the
>> same color!)
>
> I don't disagree that storing things as strings can be useful sometimes.  But
> we are not talking about big, wasteful PC software - we are talking about a
> small number of parameters in a small embedded program.  SQL is too big by

Who said it was a "small embedded program"?  Did I miss that, somewhere
upthread??  Just because its "embedded" doesn't implicitly make it "small".

How many Linux-based appliances still use fstab(5), ethers(5), inetd.conf(5),
etc.?  Surely they could replace those with "compiled" versions and elide
all of that *text* from their configurations!  After all, they already
*have* run-time code to parse those configuration files and configure
the system accordingly; why not just skip the "text processing" and store
a bunch of magic integers in a file, someplace?

> orders of magnitude for such a system.  It would a different matter if we were
> talking about an embedded linux system, for example - then you just write your
> code in Python, use a JSON string to hold a dict of the parameters, and you've
> got the system working in an hour of developer time.  Different tools for
> different tasks.

Exactly.  If I tried to replace the *thousands* of "configuration
parameters" in my current system with enums/small integers, I (and
anyone following after me) would spend all out time consulting cheat
sheets to try to remember which parameter controls which set of options.

And, of course, document all of that nicely as it evolves.

Even the little /pro bono/ job I'm working on this month has enough
"settings" to make that a daunting undertaking.  Much easier to
just say:
    result = write("/dev/display/refresh", "60")
than to sort out which ioctl(2) sets the display refresh rate to 60Hz.
Or:
    result = write("/dev/eia1/ctl", "odd")
instead of trying to remember if the manifest constant for parity
is PARODD or !PAREVEN.

Likewise, the "result" need not be some cryptic value that the
developer needs to look up and then translate into a *familiar*
form (so all developers use similar reporting messages):
    if (result != nil)
       print("Error: " result)

[This also seems to be a trend in CONFIGURING devices and apps in
other OS's as well:  "mixerctl outputs.headphones=+5" ]

If I had to provide a set of manuals enumerating all sorts of "error codes",
no one would ever attempt to alter the system from its default state.
"How do I change the color of the background?  *Why* didn't my
attempt to change the background to "vilet" [sic] fail?"

>>>> E.g., you can pack a bunch of (potentially unrelated) parameters
>>>> into a bitfield within that struct.  Data coming *into* it gets
>>>> massaged to manipulate the appropriate fields.  Likewise for data
>>>> coming *out*.
>>>
>>> That sounds like a complication that is unlikely to be worth the effort,
>>> but it's possible.  In many cases its enough to simply say "all
>>> parameters are 32 bits - it's up to the app if this is 4 characters, a
>>> float, an integer, etc.".
>>
>> Depends on resources available *and* how parameters are accessed.
>> Surely easier to fetch a *set* of related small fields in one
>> access than it is to have to fetch several individual fields
>> (each consisting of lots of 0 bits!)
>
> If you have the resources to use string keys for the parameters, you have the
> resources for using individual data items for your bits.

You're assuming all of my comments apply to  single implementation.
I'm presenting options that have/can be used for a variety of scenarios.
Many are inconsistent with each other (but the OP hadn't indicated which
SPECIFIC capabilities/characteristics his system involves)

> And if you have a number of related bits to track together, then use a flags
> parameter - have the application module keep the flags in a uint and pass them
> all together.
>
>> It also hides the representation from the caller.  Should I store
>> the time-of-last-update as a time_t?  Or, as some sort of string?
>> Or, as something that really only makes sense to the routine that
>> does the updates?
>>
>>>> Finally, because this represents a monitor, of sorts, you can ensure
>>>> atomic operations on that "data to be preserved" -- up to and including
>>>> an action that automatically saves the struct to persistent store
>>>> when power is failing (the rest of the time, "updates" happen to
>>>> the "in-RAM" copy of the struct so you don't incur the costs of
>>>> updating FLASH more often than necessary)
>>>
>>> Also true - it's always important to make sure your data is consistent
>>> at all times.

[toc] | [prev] | [next] | [standalone]

#24856

On 13/08/16 18:44, Don Y wrote:
> On 8/12/2016 1:40 PM, David Brown wrote:
>> On 12/08/16 18:40, Don Y wrote:
>>> On 8/12/2016 2:54 AM, David Brown wrote:
>>>> On 12/08/16 00:30, Don Y wrote:
>>>>> On 8/11/2016 3:17 PM, pozz wrote:
>>>>>
>>>>> [much elided]
>>>>>
>>>>>> "Everything in one structure" means you have a global (as the scope
>>>>>> in C)
>>>>>> struct definition that is accessed by every piece of code in the
>>>>>> project, even
>>>>>> if they are very different section of the project.
>>>>>
>>>>> No.  It only has to be *effectively* global.
>>>>>
>>>>> E.g., you can have a global "update_parameter()" that takes a
>>>>> "key" (name for a parameter -- this could be a literal string
>>>>> *or* just a small integer) and a parameter value (or pointer to
>>>>> a parameter value).  Update_parameter() then moves the "value"
>>>>> into the referenced portion of the "parameters to be saved struct".
>>>>
>>>> First, /don't/ use strings for that sort of thing unless you have no
>>>> other choice.  And don't use arbitrary integers.  Use an enum, that is
>>>> defined in one place.  That way you get it right.  Your code refers to
>>>> the parameters using sensible names, rather than numbers, you don't get
>>>> overlap or gaps (unless you want them, of course), and you can easily
>>>> change things when you need to.  And when one part of your program
>>>> writes parameter "backgroundColour" and another part reads parameter
>>>> "backgroundColor", you get a compile-time error rather than weird
>>>> behaviour at run-time.
>>>
>>> You throw an error at run-time -- because you *still* have to do
>>> run-time checks on parameters and their values!
>>
>> The difference here is that with checks on parameter values, you can
>> (normally)
>> just set them to safe default values if they occur, and it is
>> something that
>> typically would never happen.
>>
>> But if you have a spelling mistake in your texts, your program is
>> likely to
>> silently ignore storage of changes in parameters.
>
> Why would your program silently ignore them?  Don't you write ROBUST code?
>

If you write code that does run-time checks on strings, then you open 
yourself to the risk that spelling mistakes or other mismatches can 
cause problems that are only discovered at run time.  That means you 
need extra layers of complications - extra checks in the code to ensure 
that there are no such problems, and extra test suites in order to check 
this checking code.  Sure, it can be made robust - but at the cost of 
significant effort.

On the other hand, if such checks are made by the compiler (such as by 
using enums), no test code is needed - the compiler will spot the problem.

> If you tried to set parameter 47 to 982 -- but there were only 42 supported
> parameters, would your code "silently ignore" this?  Or, if parameter 47
> *did* exist but expected to be set to a string, would you ignore the
> attempt to set it to a numeric?

If your parameters were made using a C++ strong enum, then you could not 
write code that tried to set parameter 47 when there are only 42 
supported parameters.  If you are just using numbers (or normal C 
enums), and are using gcc (or clang), you can do something like this:

// settings.h
extern void __attribute__((error("Bad parameter number"))) 
badParameterNumber(void);

extern void doSetParameterUint32(int paramNo, uint32_t newValue);
static inline setParameterUint32(int paramNo, uint32_t newValue)
{
	if (__builtin_constant_p(paramNo)) {
		if ((paramNo < 0) || (paramNo > 41)) {
			badParameterNumber();
		}
	}
	doSetParameterUint32(paramNo, newValue);
}

Then in the very common case of using a fixed number (preferably from 
your enum, or at least from #define's), you will get compile-time 
checking of valid numbers.

The more work you can get from the compiler, the less work you have to 
do yourself as a developer, and the less work your target cpu has to do 
at run-time.


>
>>> And, it would be silly to type out the name of the key in each
>>> invocation.  Instead, you'd use:
>>>
>>> #define PARAM_BACKGROUND_COLOR ("bacKgRouNdCoLOR")
>>> #define PARAM_FOREGROUND_COLOR ("4GrouNDcoLOUr")
>>> etc.
>>>
>>> How is this any different from:
>>>
>>> #define PARAM_BACKGROUND_COLOR (1)
>>> #define PARAM_FOREGROUND_COLOR (2)
>>> etc.?
>>
>> It is different, because it is - IMHO - daft.  Working with strings
>> like this
>> is vastly less efficient, and more error prone.   And if you've got a
>> good
>
> So, if you opted to use an int as a tag/key, you'd insist on using values
> like 1, 2, 3, 4, ... instead of "BACK", "FORE", "BORD", for your tags?
> (i.e., each of these are 32b integers)
>
> And, force yourself to maintain a cheat sheet mapping *arbitrary* integers
> to these interpretations -- just because your grade school teacher taught
> you to start counting from '1'?

The "cheat sheet" is an enum.

>
> Using strings is "daft" when you don't have the resources (memory/MIPS)
> to do so.  Forcing yourself to use enums/integers when you *do* have
> the resources is "daft"!

Doing work yourself that can be done by the compiler is "daft".

Of course there are times when strings would be the most convenient 
types of tag.  It can reduce coupling between modules, and that may be 
of overriding importance.  But generally, it is best to keep things 
simple, and to get as much compile-time checking and help as possible.

>
> (why isn't the windows registry just one big set of binary tuples?

The windows registry is well-known to be a complete mess, hugely 
inefficient, unstable and insecure.  And many of its keys are just 
meaningless numbers (there is no handy "enum" to translate them).  It is 
/not/ a good example for your argument!

> why
> aren't UN*X configuration files encoded in some magical, highly space and
> time efficient manner -- with a suitable "decoder" to allow for the
> *infrequent* human use?  why do databases give names to fields instead
> of just 'field1', 'field2', etc.?)
>

Using an enum with integer keys for your parameters is great in a small 
program - it is less great for a big program, and it is a bad choice if 
you expect humans to manually alter the settings.


>> place to write out these #defines, then you are better off using an
>> enum there
>> with numerical values:
>>
>> enum {
>>      paramBackgroundColour,
>>      paramForegroundColour,
>>      etc.
>> }
>>
>> It's far more efficient, easier on the eye, easier to get IDE help in
>> automatic
>> completion, and easier for the compiler to check (such as compiler
>> warnings if
>> you use switch on an enum but forget some cases). And if you are using
>> C++
>> rather than C, you can use type-safe strong enums, as well as have better
>> scoping and namespace usage.
>
> And, all it ADDS is the requirement that you insert this extra step
> (and keep it consistently, everywhere you use it).
>
> I just got notified of an "error 27".  What's that mean?  What's
> the symbolic label associated with that numeric value?  *Then*,
> what does that symbolic label actually *mean*?
>
> [Isn't this the same argument applied to "error codes"?]
>
>>> Enums, IMO, are a bad idea -- it's far too easy to slip a new one
>>> "in the middle" and effectively bodge the keys associated with all
>>> those that follow.  I.e., a software revision changes:
>>>
>>> {PARAM_BACKGROUND_COLOR, PARAM_FOREGROUND_COLOR, ...}
>>> to
>>> {PARAM_BACKGROUND_COLOR, PARAM_BORDER_COLOR, PARAM_FOREGROUND_COLOR,
>>> ...}
>>>
>>> and now any "legacy" configuration files are effectively unreadable.
>>
>> If that is a concern (and it sometimes /is/ a concern), give the enum
>> explicit
>> values:
>>
>> enum {
>>      paramBackgroundColour = 1,
>>      paramForegroundColour = 2,
>>      etc.
>> }
>
> This is then another maintenance issue:  ensuring that paramBorderColor
> gets
> MANUALLY assigned the correct value regardless of where it appears in
> the list of enumerations:
>     enum {
>        paramBackgroundColour = 1,
>        paramForegroundColour = 2,
>        paramBorderColor = 27,
>        paramShadowColor = 3,   // return to the value that would have
> been used
>                                // had not paramBorderColor been introduced
>
> or, alternatively:
>     enum {
>        paramBackgroundColour = 1,
>        paramForegroundColour = 2,
>        paramShadowColor,   // 3
>        ...   // manually skip over 24 labels (assuming none of them
> explicitly
>              // alter their default values) to ensure we're at "27" when
> the
>              // next label (border color) is encountered
>        paramBorderColor,   // 27
>

And how is that different from using a bunch of #define's with specific 
numbers or strings?

>>> (How do you refer to "old_PARAM_FOREGROUND_COLOR" in the code that
>>> attempts to patch-up a legacy configuration dataset in light of
>>> the *new* "PARAM_FOREGROUND_COLOR"?  I.e., you end up having to create
>>> another enumeration for the legacy representation -- using DIFFERENT
>>> names!)
>>
>> And how is that worse than using different names with #define'd values?
>
> Because there is no "old_" value!  The "old" and "new" are exactly the
> same!
> You just add new values to the end and obsolete values "in the middle".
>
> #define PARAM_BACKGROUND_COLOR (3)
> #define PARAM_FOREGROUND_COLOR (4)
> #define PARAM_BORDER_COLOR (5)
> #define PARAM_SHADOW_COLOR (6)
>
> becomes:
>
> #define PARAM_BACKGROUND_COLOR (3)
> #define PARAM_FOREGROUND_COLOR (4)
> #define PARAM_BORDER_COLOR (5)         // no longer used
> #define PARAM_SHADOW_COLOR (6)
>
> when the border color parameter is obsoleted.  So, if you encounter
> a "configuration struct" that references parameter 5, you know that
> it is referencing the obsolete border color parameter -- not the
> *new* "highlight color" that took its place in the list of numerical
> parameter identifier values.

Again, how is that different from doing /exactly/ the same thing with an 
enum?

>
>>> By contrast, if you manually enumerate them, you are keenly aware of
>>> the original (name,key) mapping.  Add new names to the end of the
>>> list (who cares if "COLOR_X" and "COLOR_Y" have adjacent keys?!).
>>> And, you can simply put a comment after any keys that have been
>>> obsoleted
>>> over time:
>>>
>>> #define PARAM_VARIABLE_X (...)   // obsolete as per rev x.y.z
>>>
>>> Strings have value in that you can *see* them in logs:
>>>     log( "Updating parameter %s to value %s", param, value )
>>> instead of:
>>>     log( "updating parameter %d to value %d", param, value )
>>> or (ick):
>>>     log( "Updating parameter %s to value %s",
>>>          param_to_string(param), value_to_string(value) )
>>>
>>> Integers (and enums) suffer in poorly typed languages:
>>>     update_parameter(PARAM_BACKGROUND_COLOR, VALUE_BROCOLLI);
>>>
>>> I.e., there's no way to ensure only the values valid for
>>> PARAM_BACKGROUND_COLOR are applied to PARAM_BACKGROUND_COLOR!
>>> The above function call may succeed (depends on whether
>>> VALUE_BROCOLLI happens to coincide with a value for a
>>> PARAM_BACKGROUND_COLOR) -- or not.  I.e., you still have to
>>> rely on run-time checks; you can't do it all at compile time.
>>>
>>> In my current project, parameters are just fields in an RDBMS table.
>>> So, the RDBMS *enforces* "correct" (in the sense of "valid") values.
>>> Can't set Background_Color to "Blue" unless "Blue" is a valid choice
>>> for *that* field.  And, surely can't set it to "Brocolli"!  :>
>>>
>>> And, as the RDBMS is charged with maintaining the integrity of this
>>> data, there's no need to *check* it when reading it from the RDBMS
>>> (it was checked "on the way in"!)
>>>
>>> As everything is SQL, I *see* "Blue" not some arbitrary
>>> *encoding* of PARAM_BACKGROUND_COLOR_BLUE (which might differ
>>> from PARAM_FOREGROUND_COLOR_BLUE -- even though they are the
>>> same color!)
>>
>> I don't disagree that storing things as strings can be useful
>> sometimes.  But
>> we are not talking about big, wasteful PC software - we are talking
>> about a
>> small number of parameters in a small embedded program.  SQL is too
>> big by
>
> Who said it was a "small embedded program"?  Did I miss that, somewhere
> upthread??  Just because its "embedded" doesn't implicitly make it "small".

I don't think the OP has said so explicitly, but if you look at the code 
he has given it is far more in line with a small embedded program than 
an embedded Linux application.

>
> How many Linux-based appliances still use fstab(5), ethers(5),
> inetd.conf(5),
> etc.?  Surely they could replace those with "compiled" versions and elide
> all of that *text* from their configurations!  After all, they already
> *have* run-time code to parse those configuration files and configure
> the system accordingly; why not just skip the "text processing" and store
> a bunch of magic integers in a file, someplace?
>
>> orders of magnitude for such a system.  It would a different matter if
>> we were
>> talking about an embedded linux system, for example - then you just
>> write your
>> code in Python, use a JSON string to hold a dict of the parameters,
>> and you've
>> got the system working in an hour of developer time.  Different tools for
>> different tasks.
>
> Exactly.  If I tried to replace the *thousands* of "configuration
> parameters" in my current system with enums/small integers, I (and
> anyone following after me) would spend all out time consulting cheat
> sheets to try to remember which parameter controls which set of options.
>
> And, of course, document all of that nicely as it evolves.
>
> Even the little /pro bono/ job I'm working on this month has enough
> "settings" to make that a daunting undertaking.  Much easier to
> just say:
>     result = write("/dev/display/refresh", "60")
> than to sort out which ioctl(2) sets the display refresh rate to 60Hz.
> Or:
>     result = write("/dev/eia1/ctl", "odd")
> instead of trying to remember if the manifest constant for parity
> is PARODD or !PAREVEN.
>
> Likewise, the "result" need not be some cryptic value that the
> developer needs to look up and then translate into a *familiar*
> form (so all developers use similar reporting messages):
>     if (result != nil)
>        print("Error: " result)
>
> [This also seems to be a trend in CONFIGURING devices and apps in
> other OS's as well:  "mixerctl outputs.headphones=+5" ]
>
> If I had to provide a set of manuals enumerating all sorts of "error
> codes",
> no one would ever attempt to alter the system from its default state.
> "How do I change the color of the background?  *Why* didn't my
> attempt to change the background to "vilet" [sic] fail?"
>
>>>>> E.g., you can pack a bunch of (potentially unrelated) parameters
>>>>> into a bitfield within that struct.  Data coming *into* it gets
>>>>> massaged to manipulate the appropriate fields.  Likewise for data
>>>>> coming *out*.
>>>>
>>>> That sounds like a complication that is unlikely to be worth the
>>>> effort,
>>>> but it's possible.  In many cases its enough to simply say "all
>>>> parameters are 32 bits - it's up to the app if this is 4 characters, a
>>>> float, an integer, etc.".
>>>
>>> Depends on resources available *and* how parameters are accessed.
>>> Surely easier to fetch a *set* of related small fields in one
>>> access than it is to have to fetch several individual fields
>>> (each consisting of lots of 0 bits!)
>>
>> If you have the resources to use string keys for the parameters, you
>> have the
>> resources for using individual data items for your bits.
>
> You're assuming all of my comments apply to  single implementation.
> I'm presenting options that have/can be used for a variety of scenarios.
> Many are inconsistent with each other (but the OP hadn't indicated which
> SPECIFIC capabilities/characteristics his system involves)
>
>> And if you have a number of related bits to track together, then use a
>> flags
>> parameter - have the application module keep the flags in a uint and
>> pass them
>> all together.
>>
>>> It also hides the representation from the caller.  Should I store
>>> the time-of-last-update as a time_t?  Or, as some sort of string?
>>> Or, as something that really only makes sense to the routine that
>>> does the updates?
>>>
>>>>> Finally, because this represents a monitor, of sorts, you can ensure
>>>>> atomic operations on that "data to be preserved" -- up to and
>>>>> including
>>>>> an action that automatically saves the struct to persistent store
>>>>> when power is failing (the rest of the time, "updates" happen to
>>>>> the "in-RAM" copy of the struct so you don't incur the costs of
>>>>> updating FLASH more often than necessary)
>>>>
>>>> Also true - it's always important to make sure your data is consistent
>>>> at all times.
>

[toc] | [prev] | [next] | [standalone]

#24829

On 12/08/16 00:17, pozz wrote:
> Il 11/08/2016 23:07, David Brown ha scritto:
>> On 11/08/16 16:53, Don Y wrote:
>>> On 8/11/2016 7:12 AM, David Brown wrote:
>>>
>>>> But for the common case of storing a small set of parameters, I usually
>>>> work with a struct definition.  Along with the parameters in
> question, I
>>>> include a "magic number" for identification, a "parameter structure
>>>> version number" (very useful for upgrading the parameter set if a later
>>>> version of the firmware uses more parameters), an update serial number,
>>>> a CRC check field, and some space reserved for future additional
>>>> parameters.
>>>
>>> To be clear, you are wrapping *all* of your parameters into a single
>>> "parameters_to_be_saved" struct.
>>>
>>> This can work if they are "configuration parameters" -- data that are
>>> consulted when setting up the application, restarting it, etc.
>>
>> Yes, exactly.  I find this to be quite a common situation - but we all
>> agree that different systems have different requirements and therefore
>> different "best" solutions.
>>
>>>
>>> This is different than trying to "save application/process state"
>>> across power outages.  In that case, the "state" may be scattered
>>> through many variables dispersed in memory.  And, more difficult to
>>> ensure catching a "consistent" snapshot.
>>>
>>> I.e., if *this* variable is changed, are there any others whose state
>>> must be reconciled *as* it is being preserved?
>>
>> Usually in such cases, it is still easy enough to have everything in one
>> structure.  At its simplest, all those different state variables
>> scattered around the program can be fields of the one state struct.
> 
> "Everything in one structure" means you have a global (as the scope in
> C) struct definition that is accessed by every piece of code in the
> project, even if they are very different section of the project.
> 

Yes, that is correct.  It is still modular, structured and organised -
just organised in a different way.  It is a simple and clear method, but
works best if the code is being written by a single person rather than a
team working on the different parts of the program.

> In the past I used this approach with a global struct container:
>   /* In one .h file */
>   struct settings_s {
>     uint8_t backlight_level;
>     enum DisplayTheme theme;
>     uint16_t psu_output_voltage_calibration;
>     char password[7];
>     ...
>   };
>   extern struct settings_s settings;
> 
>   /* In one .c file */
>   struct settings_s settings;
> 
>   /* In display module */
>   backligh_set(settings.backlight_level);
> 
> Every module that manages one or more persistent parameters should know
> the presence of a global settings variable with an exposed type
> definition (struct settings_s).
> I think this is a bad approach that increase the coupling between
> modules instead of decreasing it as it should be.
> 

Coupling like this has disadvantages, but it is not necessarily a bad
thing.  And usually you are going to end up with some sort of coupling
anyway - the display module and the settings module have to exchange
information here somehow.  They either do so directly, or they do so via
a third module that co-ordinates the process.  Direct coupling can make
it easier to follow what is happening, but is most suited to smaller
programs than larger ones.

> 
>>> [It's unclear what the OP is addressing]
>>
>> Indeed.
> 
> Mainly I'm interested in saving configuration parameters/settings (color
> theme, calibration data, backlight level, and so on).
> They are parameters that are changed by the user not so frequent.
> 

[toc] | [prev] | [next] | [standalone]

#24837

Il 12/08/2016 09:52, David Brown ha scritto:
> On 12/08/16 00:17, pozz wrote:
>> Il 11/08/2016 23:07, David Brown ha scritto:
>>> On 11/08/16 16:53, Don Y wrote:
>>>> On 8/11/2016 7:12 AM, David Brown wrote:
>>>>
>>>>> But for the common case of storing a small set of parameters, I usually
>>>>> work with a struct definition.  Along with the parameters in
>> question, I
>>>>> include a "magic number" for identification, a "parameter structure
>>>>> version number" (very useful for upgrading the parameter set if a later
>>>>> version of the firmware uses more parameters), an update serial number,
>>>>> a CRC check field, and some space reserved for future additional
>>>>> parameters.
>>>>
>>>> To be clear, you are wrapping *all* of your parameters into a single
>>>> "parameters_to_be_saved" struct.
>>>>
>>>> This can work if they are "configuration parameters" -- data that are
>>>> consulted when setting up the application, restarting it, etc.
>>>
>>> Yes, exactly.  I find this to be quite a common situation - but we all
>>> agree that different systems have different requirements and therefore
>>> different "best" solutions.
>>>
>>>>
>>>> This is different than trying to "save application/process state"
>>>> across power outages.  In that case, the "state" may be scattered
>>>> through many variables dispersed in memory.  And, more difficult to
>>>> ensure catching a "consistent" snapshot.
>>>>
>>>> I.e., if *this* variable is changed, are there any others whose state
>>>> must be reconciled *as* it is being preserved?
>>>
>>> Usually in such cases, it is still easy enough to have everything in one
>>> structure.  At its simplest, all those different state variables
>>> scattered around the program can be fields of the one state struct.
>>
>> "Everything in one structure" means you have a global (as the scope in
>> C) struct definition that is accessed by every piece of code in the
>> project, even if they are very different section of the project.
>>
>
> Yes, that is correct.  It is still modular, structured and organised -
> just organised in a different way.  It is a simple and clear method, but
> works best if the code is being written by a single person rather than a
> team working on the different parts of the program.
>
>> In the past I used this approach with a global struct container:
>>   /* In one .h file */
>>   struct settings_s {
>>     uint8_t backlight_level;
>>     enum DisplayTheme theme;
>>     uint16_t psu_output_voltage_calibration;
>>     char password[7];
>>     ...
>>   };
>>   extern struct settings_s settings;
>>
>>   /* In one .c file */
>>   struct settings_s settings;
>>
>>   /* In display module */
>>   backligh_set(settings.backlight_level);
>>
>> Every module that manages one or more persistent parameters should know
>> the presence of a global settings variable with an exposed type
>> definition (struct settings_s).
>> I think this is a bad approach that increase the coupling between
>> modules instead of decreasing it as it should be.
>>
>
> Coupling like this has disadvantages, but it is not necessarily a bad
> thing.  And usually you are going to end up with some sort of coupling
> anyway - the display module and the settings module have to exchange
> information here somehow.  They either do so directly, or they do so via
> a third module that co-ordinates the process.  Direct coupling can make
> it easier to follow what is happening, but is most suited to smaller
> programs than larger ones.

In your approach all the modules should know exactly which parameters 
(among theirs) must be considered persistent and which aren't.
During initialization, the persistent parameters must be get, from a 
global struct that includes persistent parameters from other modules too 
or from a wrapper get_persistent(PERSISTENT_PARAMETER_ID, ...).

The global struct is simple and effective, but there is a strong 
coupling between module and the global struct that could change in a 
different project where I'd like to reuse the same module.

The wrapper function hide the global struct, but introduces another 
problem. What is the return value of get_persistent()? uint32_t could be 
a solution for the worst case... however I could have uint64_t 
parameters or big structs.

Another problem with this approach is every module *must* know all the 
parameters that are classified persistent *for that project*, i.e. this 
must be loaded from the global struct/wrapper, that are not persistent 
(they could have a fixed default value set during initialization or 
changed at run-time through a function).
What happens if you change the classification of persistent parameters 
during developing (it's better to save also the last channel selected by 
the user so I can restore it at next start-up)?  Or the next project 
needs to consider persistent a parameter that wasn't considered 
persistent in the current project?
In both cases you need to change some things in the module.

I'm thinking at another approach.  The module starts with default 
parameters (persistent and not persistent) at init() function.
The "nvconfig" (non-volatile configuration) module loads all the 
parameters from the non-volatile memory (most probably it will be a 
simple raw C struct) and put them in a *local* container (C struct).

Now nvconfig_restore() function will be responsible to set each 
parameter by calling relevant set() functions of relevant module.
Most probably, you already have a set() function to change one parameter.

When the user changes a persistent (for that project) parameter, a 
nvconfig_set_<param_name>() is called.  This funcion is a wrapper for 
the relevant set() function of relevant module.  If the new value is 
accepted, it is copied in the local nvconfig C struct.  And, if needed, 
the nvconfig_save() can be automatically called... otherwise it can be 
called by the user when he click on the "OK" button.

Some piece of code is better than a million words :-)

/* nvconfig.c */
#include "display.h"
#include "tuner.h"
#include "psu.h"
#include "audio.h"

*static* struct {
   uint16_t magic;
   uint16_t sn;
   struct {
     uint16_t backlight_level;
     uint16_t contrast_level;
     Color background_color;
   } display;
   uint16_t psu_calibration_value;
   struct {
     uint8_t last_selected_channel;
     enum TUNER_MODE mode;  // analog, digital
   } tuner;
   struct {
     uint16_t volume_level;
     uint16_t balance;
   } audio;
   uint16_t crc16;
} nvconfig;

static void nvconfig_set_default();

void nvconfig_load(void) {
   ... restore nvconfig data from non-volatile memory...
   if (nvconfig.magic != MY_MAGIC_NUMBER) {
     nvconfig_set_default();
   }
   if (nvconfig.crc16 != crc16(&nvconfig, sizeof(nvconfig))) {
     nvconfig_set_default();
   }
}

void nvconfig_restore(void) {
   display_backlight_set(nvconfig.display.backlight_level);
   display_contrast_set(nvconfig.display.contrast_level);

   psu_calibrate(nvconfig.psu_calibration_value);

   tuner_channel_set(nvconfig.tuner.last_selected_channel);
   tuner_mode_set(nvconfig.tuner.mode);

   audio_volume_set(nvconfig.audio.volume_level);
   audio_balance_set(nvconfig.audio.balance);
}

void nvconfig_volume_set(uint16_t new_value) {
   if (audio_volume_set(new_value) == OK) {
     nvconfig.audio.volume_level = new_value;
   }
   nvconfig_save();  // If necessary
}


With this approach, the modules aren't interesting in knowing which 
parameters are persistent... they doesn't know the existence of the 
"nvconfig" module and non-volatile memory at all.



>>>> [It's unclear what the OP is addressing]
>>>
>>> Indeed.
>>
>> Mainly I'm interested in saving configuration parameters/settings (color
>> theme, calibration data, backlight level, and so on).
>> They are parameters that are changed by the user not so frequent.
>>
>

[toc] | [prev] | [next] | [standalone]

#24838

On 12/08/16 16:30, pozz wrote:

> In your approach all the modules should know exactly which parameters
> (among theirs) must be considered persistent and which aren't.

True.  But /some/ module must know that!  (We are considering a
relatively simple and static system here.  It would be different if you
had hundreds of parameters that are mostly the default values, and you
only wanted to store those few that are non-default.)

> During initialization, the persistent parameters must be get, from a
> global struct that includes persistent parameters from other modules too
> or from a wrapper get_persistent(PERSISTENT_PARAMETER_ID, ...).
> 

Yes.

> The global struct is simple and effective, but there is a strong
> coupling between module and the global struct that could change in a
> different project where I'd like to reuse the same module.

True.

It is possible to use multiple include files, or smart pre-processing in
your build system, in order to minimise or eliminate this.  For example,
your re-usable module could be packages as:

woozle.h	# Normal interface file
woozle.c	# Normal implementation file
woozle_params.inc	# Struct include file:
	struct {
		uint32_t numberOfWoozles;
		float woozleScale;
	} woozle;

Then your global settings struct in settings.h looks like this:

typedef struct {
	uint16_t crc;
	uint16_t magicNumber;
	...
#include "woozle_params.inc"
#include "wotsit_params.inc"
	...
} settings_t;

And your woozle.c code can access it as
"currentSettings.woozle.numberOfWoozles".

A smart build process could even generate a single "all_params.inc" file
from all the "*_params.inc" files to reduce that coupling too.


> 
> The wrapper function hide the global struct, but introduces another
> problem. What is the return value of get_persistent()? uint32_t could be
> a solution for the worst case... however I could have uint64_t
> parameters or big structs.

That's one of the reasons I'm not too keen on wrapper functions like
this.  But it shouldn't be too hard to have a few versions returning
different type, or you could use a union.  A bigger problem is making
sure that you are consistent about the type used for each parameter
(preferably in a way that is checked by the compiler as much as possible).

> 
> Another problem with this approach is every module *must* know all the
> parameters that are classified persistent *for that project*, i.e. this
> must be loaded from the global struct/wrapper, that are not persistent
> (they could have a fixed default value set during initialization or
> changed at run-time through a function).

The rest of the code can /see/ the other modules' parameters - but it
can happily ignore them.

> What happens if you change the classification of persistent parameters
> during developing (it's better to save also the last channel selected by
> the user so I can restore it at next start-up)?  Or the next project
> needs to consider persistent a parameter that wasn't considered
> persistent in the current project?
> In both cases you need to change some things in the module.
> 
> I'm thinking at another approach.  The module starts with default
> parameters (persistent and not persistent) at init() function.
> The "nvconfig" (non-volatile configuration) module loads all the
> parameters from the non-volatile memory (most probably it will be a
> simple raw C struct) and put them in a *local* container (C struct).
> 
> Now nvconfig_restore() function will be responsible to set each
> parameter by calling relevant set() functions of relevant module.
> Most probably, you already have a set() function to change one parameter.

This means even more coupling between the nvconfig module and the other
modules.  But certainly it can be a workable method.

> 
> When the user changes a persistent (for that project) parameter, a
> nvconfig_set_<param_name>() is called.  This funcion is a wrapper for
> the relevant set() function of relevant module.  If the new value is
> accepted, it is copied in the local nvconfig C struct.  And, if needed,
> the nvconfig_save() can be automatically called... otherwise it can be
> called by the user when he click on the "OK" button.
> 
> Some piece of code is better than a million words :-)
> 
> /* nvconfig.c */
> #include "display.h"
> #include "tuner.h"
> #include "psu.h"
> #include "audio.h"
> 
> *static* struct {
>   uint16_t magic;
>   uint16_t sn;
>   struct {
>     uint16_t backlight_level;
>     uint16_t contrast_level;
>     Color background_color;
>   } display;
>   uint16_t psu_calibration_value;
>   struct {
>     uint8_t last_selected_channel;
>     enum TUNER_MODE mode;  // analog, digital
>   } tuner;
>   struct {
>     uint16_t volume_level;
>     uint16_t balance;
>   } audio;
>   uint16_t crc16;
> } nvconfig;
> 
> static void nvconfig_set_default();
> 
> void nvconfig_load(void) {
>   ... restore nvconfig data from non-volatile memory...
>   if (nvconfig.magic != MY_MAGIC_NUMBER) {
>     nvconfig_set_default();
>   }
>   if (nvconfig.crc16 != crc16(&nvconfig, sizeof(nvconfig))) {
>     nvconfig_set_default();
>   }
> }

Minor niggle - you might want that second "if" to be an "else if".
There's no point in setting the defaults twice, or checking the crc
after you have restored the defaults.

> 
> void nvconfig_restore(void) {
>   display_backlight_set(nvconfig.display.backlight_level);
>   display_contrast_set(nvconfig.display.contrast_level);
> 
>   psu_calibrate(nvconfig.psu_calibration_value);
> 
>   tuner_channel_set(nvconfig.tuner.last_selected_channel);
>   tuner_mode_set(nvconfig.tuner.mode);
> 
>   audio_volume_set(nvconfig.audio.volume_level);
>   audio_balance_set(nvconfig.audio.balance);
> }
> 
> void nvconfig_volume_set(uint16_t new_value) {
>   if (audio_volume_set(new_value) == OK) {
>     nvconfig.audio.volume_level = new_value;
>   }
>   nvconfig_save();  // If necessary
> }
> 
> 
> With this approach, the modules aren't interesting in knowing which
> parameters are persistent... they doesn't know the existence of the
> "nvconfig" module and non-volatile memory at all.
> 

Yes.  It's another way to arrange things, and may be a better choice for
some uses.  If this suits you for your project, I see nothing wrong with it.

> 
> 
>>>>> [It's unclear what the OP is addressing]
>>>>
>>>> Indeed.
>>>
>>> Mainly I'm interested in saving configuration parameters/settings (color
>>> theme, calibration data, backlight level, and so on).
>>> They are parameters that are changed by the user not so frequent.
>>>
>>
> 

[toc] | [prev] | [next] | [standalone]

#24821

On 8/11/2016 2:07 PM, David Brown wrote:
>> I've used a circular buffer into which "tagged" parameters are placed.
>> On restart, I "play back" the buffer to restore the latest versions of
>> the parameters encountered, there.  This lets me mix snapshots and
>> incremental updates in the same mechanism.
>
> I've done something like that too (that's sort of what I meant be a log
> structure - though I didn't describe it in detail).  It worked well for a flash
> based system - once one sector is getting a bit full, I ran a garbage
> collection pass by running through the buffer copying the latest parameters
> into a new sector, and then erased the old buffer with the outdated values.

It has the advantage that it also serves as a debugging tool.

I employ such "black boxes" (named for the fabled flight recorders on
aircraft) in most subsystems as cheap (fast, low resource) mechanisms
for keeping track of what is happening in a subsystem.  In a /post mortem/,
I can peruse their contents to see what was happening in each module just
prior to the "end".

For "settings" that need to be preserved as part of the normal operation
of a device (i.e., more like "state" than "configuration data"), there
can be frequent writes to these BB's -- every interaction with the user!
As such, they can reveal a lot of what else was happening in the system.

(If you just have a "configuration mode", then the amount of traffic is
probably too infrequent to be of much use in a /post mortem/ -- unless,
of course, you've a bug in your UI!)

[toc] | [prev] | [next] | [standalone]

#24826

In article <noipdo$nnv$1@dont-email.me>, david.brown@hesbynett.no 
says...
> 
> On 11/08/16 16:53, Don Y wrote:
> > On 8/11/2016 7:12 AM, David Brown wrote:
> >
> >> But for the common case of storing a small set of parameters, I usually
> >> work with a struct definition.  Along with the parameters in question, I
> >> include a "magic number" for identification, a "parameter structure
> >> version number" (very useful for upgrading the parameter set if a later
> >> version of the firmware uses more parameters), an update serial number,
> >> a CRC check field, and some space reserved for future additional
> >> parameters.
> >
> > To be clear, you are wrapping *all* of your parameters into a single
> > "parameters_to_be_saved" struct.
> >
> > This can work if they are "configuration parameters" -- data that are
> > consulted when setting up the application, restarting it, etc.
> 
> Yes, exactly.  I find this to be quite a common situation - but we all 
> agree that different systems have different requirements and therefore 
> different "best" solutions.

Whatever method you use on boot-up load a default set into RAM before 
checking if any blocks/pages/memory areas contain a VALID set of 
parameters or states or save sets. Often easier to do reads or checksums
of blocks then later do a block copy. Failing that temp copy to RAM 
buffer check its validity and do memcpy to where it is needed.

Valid includes not a higher version than current firmware, always add
new features to END of struct. Seen that forgotten too many times.

Be prepared to erase invalid sets at that stage. However that erase is 
done even if single location erase.

Work with more than one set, get the latest (or latest per user set)

> > This is different than trying to "save application/process state"
> > across power outages.  In that case, the "state" may be scattered
> > through many variables dispersed in memory.  And, more difficult to
> > ensure catching a "consistent" snapshot.
> >
> > I.e., if *this* variable is changed, are there any others whose state
> > must be reconciled *as* it is being preserved?
> 
> Usually in such cases, it is still easy enough to have everything in one 
> structure.  At its simplest, all those different state variables 
> scattered around the program can be fields of the one state struct.

Worst case two or three structures or structure tree

	Basic configuratio pair min
	Save state pair min
	Any other type of data set pair min

Generally I run same principles on all data sets only load those that
are valid to supercede defaults

> >
> > [It's unclear what the OP is addressing]
> 
> Indeed.

We rarely do
 
-- 
Paul Carpenter          | paul@pcserviceselectronics.co.uk
<http://www.pcserviceselectronics.co.uk/>    PC Services
<http://www.pcserviceselectronics.co.uk/pi/>  Raspberry Pi Add-ons
<http://www.pcserviceselectronics.co.uk/fonts/> Timing Diagram Font
<http://www.badweb.org.uk/> For those web sites you hate

[toc] | [prev] | [next] | [standalone]

#24810

On 08/11/2016 10:12 AM, David Brown wrote:
> On 11/08/16 14:48, pozz wrote:
>> Many times I work on embedded projects where it is required to save some
>> data in a non volatile way.  Usualy I use external (to the main
>> microcontroller) serial/parallel EEPROM or internal Flash memory (with a
>> software layer for EEPROM emulation).
>>
>> However my goal isn't to discuss the hw alternatives, but the data
>> format and strategy.
>>
>> I know there are many serialization data format (XML, JSON, Protocol
>> Buffer, SQLite), but I don't think they are valid solutions for
>> medium/low end microcontrollers.

Why not? There are implementations of the FAT32 filesystem for SD cards 
and XML libraries for platforms like the Arduino that are really compact.

SD cards are cheap; I think I'd consider using them as an alternative to 
external EEPROM. There's also the advantage that you can just stick them 
in a PC and edit the data, if you want.

[toc] | [prev] | [next] | [standalone]

#24816

Il 11/08/2016 18:19, bitrex ha scritto:
 > On 08/11/2016 10:12 AM, David Brown wrote:
 >> On 11/08/16 14:48, pozz wrote:
 >>> Many times I work on embedded projects where it is required to save 
some
 >>> data in a non volatile way.  Usualy I use external (to the main
 >>> microcontroller) serial/parallel EEPROM or internal Flash memory 
(with a
 >>> software layer for EEPROM emulation).
 >>>
 >>> However my goal isn't to discuss the hw alternatives, but the data
 >>> format and strategy.
 >>>
 >>> I know there are many serialization data format (XML, JSON, Protocol
 >>> Buffer, SQLite), but I don't think they are valid solutions for
 >>> medium/low end microcontrollers.
 >
 > Why not? There are implementations of the FAT32 filesystem for SD cards
 > and XML libraries for platforms like the Arduino that are really compact.

Even if you're right (I don't know how simple to integrate and compact
are those libraries for microcontrollers, such as Cortex-M0+), the price
is the first reason to avoid this.  Connector and SD card are much more
expensive against a simple serial EEPROM or the internal Flash memory
that you already have for free.

Another reason could be additional space on the board.

Anyway, could you link those compact libraries for FAT32 and XML?


 > SD cards are cheap; I think I'd consider using them as an alternative to
 > external EEPROM. There's also the advantage that you can just stick them
 > in a PC and edit the data, if you want.
 >

[toc] | [prev] | [next] | [standalone]

#24822

On 8/11/2016 2:49 PM, pozz wrote:
> Il 11/08/2016 18:19, bitrex ha scritto:
>  > On 08/11/2016 10:12 AM, David Brown wrote:
>  >> On 11/08/16 14:48, pozz wrote:
>  >>> Many times I work on embedded projects where it is required to save some
>  >>> data in a non volatile way.  Usualy I use external (to the main
>  >>> microcontroller) serial/parallel EEPROM or internal Flash memory (with a
>  >>> software layer for EEPROM emulation).
>  >>>
>  >>> However my goal isn't to discuss the hw alternatives, but the data
>  >>> format and strategy.
>  >>>
>  >>> I know there are many serialization data format (XML, JSON, Protocol
>  >>> Buffer, SQLite), but I don't think they are valid solutions for
>  >>> medium/low end microcontrollers.
>  >
>  > Why not? There are implementations of the FAT32 filesystem for SD cards
>  > and XML libraries for platforms like the Arduino that are really compact.
>
> Even if you're right (I don't know how simple to integrate and compact
> are those libraries for microcontrollers, such as Cortex-M0+), the price
> is the first reason to avoid this.  Connector and SD card are much more
> expensive against a simple serial EEPROM or the internal Flash memory
> that you already have for free.
>
> Another reason could be additional space on the board.

Another reason could be the issue of now having to deal with a
memory card that can be arbitrarily removed DURING a write!

[toc] | [prev] | [next] | [standalone]

#24814

David Brown <david.brown@hesbynett.no> writes:

> On 11/08/16 14:48, pozz wrote:
>> Many times I work on embedded projects where it is required to save some
>> data in a non volatile way.  Usualy I use external (to the main
>> microcontroller) serial/parallel EEPROM or internal Flash memory (with a
>> software layer for EEPROM emulation).
>> 
>> However my goal isn't to discuss the hw alternatives, but the data
>> format and strategy.
>> 
>> I know there are many serialization data format (XML, JSON, Protocol
>> Buffer, SQLite), but I don't think they are valid solutions for
>> medium/low end microcontrollers.  They are complex to implement and they
>> aren't really necessary.
>> Data are written and read from the same platform, so the endianess and
>> byte order are not a problem.
>> 
>> IMHO directly saving a C structure to the non-volatile memory is the
>> best method.  When you will load bytes from the non volatile memory, you
>> will have magically your C structure in RAM filled with the saved values.
>> 
>> Is it the same strategy that you use?
>
> As Don says, there is no single best method.  The answer to almost every
> question in embedded development is "it depends".
>
> But for the common case of storing a small set of parameters, I usually
> work with a struct definition.  Along with the parameters in question, I
> include a "magic number" for identification, a "parameter structure
> version number" (very useful for upgrading the parameter set if a later
> version of the firmware uses more parameters), an update serial number,
> a CRC check field, and some space reserved for future additional
> parameters.

Yay, that's *exactly* what I have ended up with.

Binary copying of structs seems to be frowned on but it saves a lot of
work (and code). AFAICS it's fine since it is defined by the ARM EABI
(in my case). And I am hardly going to be unsoldering the ARM and
replacing it with a MIPS.

> Typically I have a statically allocated struct in ram, and two copies in
> eeprom or flash.  On startup, I check both non-volatile copies to see
> which is the latest version (highest serial number) that has a valid
> CRC, and copy that into RAM.  If the parameter structure version number
> does not match the current firmware requirements, I upgrade the
> parameters.

You've copied my stuff haven't you.

> When I need to save the parameters, I use the oldest of the two NV sets.
>  First, I erase it (for eeprom, that's just clearing the magic number).
>  Then I copy in most of the data and CRC, leaving the magic number for
>  last.

> I have had occasion to use a log structured parameter "file system" in
> flash, but usually such complications are unnecessary.

It may be needed if you want to store a counter rather than some
occasionally altered settings. That's when I did it and it was a PITA.


>And certainly
> SQLite, or JSON, etc., are normally overkill.

-- 

John Devereux

[toc] | [prev] | [next] | [standalone]

#24817

Il 11/08/2016 16:12, David Brown ha scritto:
 > On 11/08/16 14:48, pozz wrote:
 >> Many times I work on embedded projects where it is required to save some
 >> data in a non volatile way.  Usualy I use external (to the main
 >> microcontroller) serial/parallel EEPROM or internal Flash memory (with a
 >> software layer for EEPROM emulation).
 >>
 >> However my goal isn't to discuss the hw alternatives, but the data
 >> format and strategy.
 >>
 >> I know there are many serialization data format (XML, JSON, Protocol
 >> Buffer, SQLite), but I don't think they are valid solutions for
 >> medium/low end microcontrollers.  They are complex to implement and they
 >> aren't really necessary.
 >> Data are written and read from the same platform, so the endianess and
 >> byte order are not a problem.
 >>
 >> IMHO directly saving a C structure to the non-volatile memory is the
 >> best method.  When you will load bytes from the non volatile memory, you
 >> will have magically your C structure in RAM filled with the saved 
values.
 >>
 >> Is it the same strategy that you use?
 >
 > As Don says, there is no single best method.  The answer to almost every
 > question in embedded development is "it depends".

Of course, yes.  However if we define and restrict the requirements, the 
number of good strategies is reduced.


 > But for the common case of storing a small set of parameters,

Indeed my intent is that situation.


 > I usually
 > work with a struct definition.  Along with the parameters in question, I
 > include a "magic number" for identification, a "parameter structure
 > version number" (very useful for upgrading the parameter set if a later
 > version of the firmware uses more parameters), an update serial number,
 > a CRC check field, and some space reserved for future additional 
parameters.
 >
 > Typically I have a statically allocated struct in ram, and two copies in
 > eeprom or flash.  On startup, I check both non-volatile copies to see
 > which is the latest version (highest serial number) that has a valid
 > CRC, and copy that into RAM.  If the parameter structure version number
 > does not match the current firmware requirements, I upgrade the 
parameters.
 >
 > When I need to save the parameters, I use the oldest of the two NV sets.
 >  First, I erase it (for eeprom, that's just clearing the magic number).
 >  Then I copy in most of the data and CRC, leaving the magic number 
for last.

Interesting approach.

I couldn't understand what is "magic number" for identification purpose. 
  CRC can be used to check if the saved struct is valid or not, so the 
"magic number" is useless for this goal.

Maybe you have more than one struct saved to the non volatile memory, 
and the "magic number" identifies the type of the struct?
In this case, I think the physical address where you save the struct 
automatically identifies the struct itsel.
Do you have a different "magic number", "serial number", CRC for each 
saved struct?

Or do you have a dynamic list of structs that could be saved at 
different places of non volatile memory (non fixed address)?  In this 
case, you have a file list somewhere, similar to a FAT table.  Maybe you 
have implemented a minimal filesystem where a file is a struct.


 > I have had occasion to use a log structured parameter "file system" in
 > flash, but usually such complications are unnecessary.  And certainly
 > SQLite, or JSON, etc., are normally overkill.
 >

[toc] | [prev] | [next] | [standalone]

#24827

In article <noitgh$8fa$1@dont-email.me>, pozzugno@gmail.com says...
> 
> Il 11/08/2016 16:12, David Brown ha scritto:
>  > I usually
>  > work with a struct definition.  Along with the parameters in question, I
>  > include a "magic number" for identification, a "parameter structure
>  > version number" (very useful for upgrading the parameter set if a later
>  > version of the firmware uses more parameters), an update serial number,
>  > a CRC check field, and some space reserved for future additional 
> parameters.
>  >
>  > Typically I have a statically allocated struct in ram, and two copies in
>  > eeprom or flash.  On startup, I check both non-volatile copies to see
>  > which is the latest version (highest serial number) that has a valid
>  > CRC, and copy that into RAM.  If the parameter structure version number
>  > does not match the current firmware requirements, I upgrade the 
> parameters.
>  >
>  > When I need to save the parameters, I use the oldest of the two NV sets.
>  >  First, I erase it (for eeprom, that's just clearing the magic number).
>  >  Then I copy in most of the data and CRC, leaving the magic number 
> for last.
> 
> Interesting approach.
> 
> I couldn't understand what is "magic number" for identification purpose. 
>   CRC can be used to check if the saved struct is valid or not, so the 
> "magic number" is useless for this goal.

You really want to be sure that you have no other problems like hardware 
fault that addresses wrong area of memory that happens to have a 
location that appears to give a valid CRC, watch everything fail in a 
misleading way. Simply could be firmware change and someone has got
the save and read address (even internal ones of external devices)
wrong due to part of upgrade being different size 'EEPROM'.

I have seen some systems fail if the external storage has been fully
erased and give wrong configuration for many other reasons.
 
> Maybe you have more than one struct saved to the non volatile memory, 
> and the "magic number" identifies the type of the struct?
> In this case, I think the physical address where you save the struct 
> automatically identifies the struct itsel.
> Do you have a different "magic number", "serial number", CRC for each 
> saved struct?

You could use magic numbers that way but I tend to use additional 
members for expected type and a diff magic number for double check, to 
avoid different sizes of each struct and misleading CRC appearing 
'valid'.

Assumption that you have the right data because you think you are only
accessing one area of memory especially if external (or even SD Card)
can lead to all sorts of confusion.

-- 
Paul Carpenter          | paul@pcserviceselectronics.co.uk
<http://www.pcserviceselectronics.co.uk/>    PC Services
<http://www.pcserviceselectronics.co.uk/pi/>  Raspberry Pi Add-ons
<http://www.pcserviceselectronics.co.uk/fonts/> Timing Diagram Font
<http://www.badweb.org.uk/> For those web sites you hate

[toc] | [prev] | [next] | [standalone]

#24831

On 12/08/16 00:16, pozz wrote:
> Il 11/08/2016 16:12, David Brown ha scritto:

<snip>

>> I usually
>> work with a struct definition.  Along with the parameters in question, I
>> include a "magic number" for identification, a "parameter structure
>> version number" (very useful for upgrading the parameter set if a later
>> version of the firmware uses more parameters), an update serial number,
>> a CRC check field, and some space reserved for future additional
> parameters.
>>
>> Typically I have a statically allocated struct in ram, and two copies in
>> eeprom or flash.  On startup, I check both non-volatile copies to see
>> which is the latest version (highest serial number) that has a valid
>> CRC, and copy that into RAM.  If the parameter structure version number
>> does not match the current firmware requirements, I upgrade the
> parameters.
>>
>> When I need to save the parameters, I use the oldest of the two NV sets.
>>  First, I erase it (for eeprom, that's just clearing the magic number).
>>  Then I copy in most of the data and CRC, leaving the magic number for
> last.
> 
> Interesting approach.
> 
> I couldn't understand what is "magic number" for identification purpose.
>  CRC can be used to check if the saved struct is valid or not, so the
> "magic number" is useless for this goal.
> 
> Maybe you have more than one struct saved to the non volatile memory,
> and the "magic number" identifies the type of the struct?
> In this case, I think the physical address where you save the struct
> automatically identifies the struct itsel.
> Do you have a different "magic number", "serial number", CRC for each
> saved struct?

The "magic number" is to identify that the type of record.  Perhaps the
system will have more than one possible set of data, or the same board
is used for two completely different applications with different
parameter sets.

> 
> Or do you have a dynamic list of structs that could be saved at
> different places of non volatile memory (non fixed address)?  In this
> case, you have a file list somewhere, similar to a FAT table.  Maybe you
> have implemented a minimal filesystem where a file is a struct.
> 
> 
>> I have had occasion to use a log structured parameter "file system" in
>> flash, but usually such complications are unnecessary.  And certainly
>> SQLite, or JSON, etc., are normally overkill.
>>
> 

[toc] | [prev] | [next] | [standalone]

#24811

pozz <pozzugno@gmail.com> writes:
> IMHO directly saving a C structure to the non-volatile memory is the
> best method.  When you will load bytes from the non volatile memory,
> you will have magically your C structure

Two problems come to mind:

1) if the C structure has pointers in it, then on reload everything must
end up at the same address as before.  That constrains you a lot and can
be a source of bugs and can get in your way if you want to change the
software then reload the parameters.  So you are probably better off
serializing the data somehow--a simple binary format is fine, json etc.
are overkill.

2) As David Brown indicated, you can get in trouble if the program
crashes (loses power, etc) during the save operation.  You need some way
to deal with that, such as by writing out two copies as he describes.
Having a version number in the data as he mentions is also often
worthwhile.

[toc] | [prev] | [next] | [standalone]

#24842

pozz wrote:
> Many times I work on embedded projects where it is required to save some
> data in a non volatile way.  Usualy I use external (to the main
> microcontroller) serial/parallel EEPROM or internal Flash memory (with a
> software layer for EEPROM emulation).
>
> However my goal isn't to discuss the hw alternatives, but the data
> format and strategy.
>
> I know there are many serialization data format (XML, JSON, Protocol
> Buffer, SQLite), but I don't think they are valid solutions for
> medium/low end microcontrollers.  They are complex to implement and they
> aren't really necessary.


They call these "serialization formats".

The simplest XML solution I've found is ezXML. It's sort of fiddly.
JSON is another choice. IMO, JSON is a little easier to look at.

This being said, just dotted names in a text file is equivalent to
each of these. Line of Class.instance.42.attribute.thing="86.42"\n works 
just as well.

> Data are written and read from the same platform, so the endianess and
> byte order are not a problem.
>

Text formats make this problem go away.

> IMHO directly saving a C structure to the non-volatile memory is the
> best method.

Unless the shape of the struct changes. Having to maintain and feed a 
version ID to cover all possible struct shapes is a pain.

> When you will load bytes from the non volatile memory, you
> will have magically your C structure in RAM filled with the saved values.
>
> Is it the same strategy that you use?

I have, but there's some advantage to mapping the struct to a text file 
of attribute-value pairs through a table of names. How much trouble
this is depends on how many attribute value pairs you're storing.

-- 
Les Cargill

[toc] | [prev] | [next] | [standalone]

#24843

On Thu, 11 Aug 2016 14:48:57 +0200, pozz wrote:

> Many times I work on embedded projects where it is required to save some
> data in a non volatile way.  Usualy I use external (to the main
> microcontroller) serial/parallel EEPROM or internal Flash memory (with a
> software layer for EEPROM emulation).
> 
> However my goal isn't to discuss the hw alternatives, but the data
> format and strategy.
> 
> I know there are many serialization data format (XML, JSON, Protocol
> Buffer, SQLite), but I don't think they are valid solutions for
> medium/low end microcontrollers.  They are complex to implement and they
> aren't really necessary.
> Data are written and read from the same platform, so the endianess and
> byte order are not a problem.
> 
> IMHO directly saving a C structure to the non-volatile memory is the
> best method.  When you will load bytes from the non volatile memory, you
> will have magically your C structure in RAM filled with the saved
> values.
> 
> Is it the same strategy that you use?

I think JSON, XML, etc are over kill for data that will never be
exposed to the outside world.  If you are going to have an editable
config file, then yes.

Writing structs directly to storage can introduce upgrade/downgrade
issues.  But depending on the stability of the structs and how you
manage introducing new data sets into the struct it can work.  Saving
the entire config data set works well with flash where you are block
erasing, block writing anyway.

I have always used either on chip or external byte writable EEPROM.
Address zero is always a version id of the data in the eeprom. I decouple
the FW rev from the eeprom rev.  All new items are added at the end.
FW rev 1 reads 0x00 to say 0x50 with offsets known.  FW rev 2
reads 0x00-0x50 for existing items, and 0x51... for new items.  If I need
to downgrade to FW rev 1 the EEPROM items 0x51 and beyond are simply
ignored.

Another simple way is a tag-length-value list.  You have a search
routine that reads through the eeprom looking for the specific tag.

With either of these approaches  you have to deal with the issue of
config item "X" is a u_int8 and now needs to be a u_int32.  How do you
manage upgrade path fw ver 1 -> fw ver 2 (that needs u_int32) and then
downgrade back to fw ver 1 because of some fatal flaw in ver 2 that will
take you a month to fix and QA.  You have to roll back the fw so the
units will be operational until you can fix ver 2.

-- 
Chisolm
Republic of Texas

[toc] | [prev] | [next] | [standalone]

#24846

In article <gaydnb_QsOlinzPKnZ2dnUU7-eGdnZ2d@earthlink.com>, jchisolm6
@earthlink.net says...
> 
> On Thu, 11 Aug 2016 14:48:57 +0200, pozz wrote:
> 
> > Many times I work on embedded projects where it is required to save some
> > data in a non volatile way.  Usualy I use external (to the main
> > microcontroller) serial/parallel EEPROM or internal Flash memory (with a
> > software layer for EEPROM emulation).
> > 
> > However my goal isn't to discuss the hw alternatives, but the data
> > format and strategy.
> > 
> > I know there are many serialization data format (XML, JSON, Protocol
> > Buffer, SQLite), but I don't think they are valid solutions for
> > medium/low end microcontrollers.  They are complex to implement and they
> > aren't really necessary.
> > Data are written and read from the same platform, so the endianess and
> > byte order are not a problem.
> > 
> > IMHO directly saving a C structure to the non-volatile memory is the
> > best method.  When you will load bytes from the non volatile memory, you
> > will have magically your C structure in RAM filled with the saved
> > values.
> > 
> > Is it the same strategy that you use?
> 
> I think JSON, XML, etc are over kill for data that will never be
> exposed to the outside world.  If you are going to have an editable
> config file, then yes.
> 
> Writing structs directly to storage can introduce upgrade/downgrade
> issues.  But depending on the stability of the structs and how you
> manage introducing new data sets into the struct it can work.  Saving
> the entire config data set works well with flash where you are block
> erasing, block writing anyway.
> 
> I have always used either on chip or external byte writable EEPROM.
> Address zero is always a version id of the data in the eeprom. I decouple
> the FW rev from the eeprom rev.  All new items are added at the end.
> FW rev 1 reads 0x00 to say 0x50 with offsets known.  FW rev 2
> reads 0x00-0x50 for existing items, and 0x51... for new items.  If I need
> to downgrade to FW rev 1 the EEPROM items 0x51 and beyond are simply
> ignored.

I try to AVOID using struct control parameters (version, magic number, 
type, byte count, CRC/checksum) to ever have 0, 0xFF, -1 as legal values
to avoid various erased block values being confused as valid values.

Personally rather than having to have tables of which version has how 
many bytes, I store byte count as well to compare against software 
version defaults size to work out which count to useeither they are same 
or defaults size is smaller so used (downgrade) or external is smaller
(upgrade). This works if you are concientious in your

> Another simple way is a tag-length-value list.  You have a search
> routine that reads through the eeprom looking for the specific tag.
> 
> With either of these approaches  you have to deal with the issue of
> config item "X" is a u_int8 and now needs to be a u_int32.  How do you
> manage upgrade path fw ver 1 -> fw ver 2 (that needs u_int32) and then
> downgrade back to fw ver 1 because of some fatal flaw in ver 2 that will
> take you a month to fix and QA.  You have to roll back the fw so the
> units will be operational until you can fix ver 2.

If changing the size of a parameter, I put the NEW type on end and leave 
old one at last used setting. For your example I would not want new 
version to use same position in struct, for exact reasons above. That 
sort of change is going to require code change anyway and really should 
be noted more than just the change of a declaration. The operations on
that parameter 

If you have to do special conversions then do that based on knowing
the different versions between defaults (firmware version) and external
version. As per your example upgrade changes type so the default would 
either need conversion or using new default. This should be part of 
your update design. I try to work on using new defaults for extra value 
as the norm as update usually means recalibration or similar task on 
first use.


-- 
Paul Carpenter          | paul@pcserviceselectronics.co.uk
<http://www.pcserviceselectronics.co.uk/>    PC Services
<http://www.pcserviceselectronics.co.uk/pi/>  Raspberry Pi Add-ons
<http://www.pcserviceselectronics.co.uk/fonts/> Timing Diagram Font
<http://www.badweb.org.uk/> For those web sites you hate

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | comp.arch.embedded

csiph-web