Re: What is your strategy to save non volatile data?

From	Don Y <blockedofcourse@foo.invalid>
Newsgroups	comp.arch.embedded
Subject	Re: What is your strategy to save non volatile data?
Date	2016-08-13 09:44 -0700
Organization	A noiseless patient Spider
Message-ID	<noniof$m47$1@dont-email.me> (permalink)
References	(4 earlier) <noithu$8fa$2@dont-email.me> <noiuaf$au4$1@dont-email.me> <nok6df$krv$1@dont-email.me> <noku5v$510$1@dont-email.me> <nolc7l$lpn$1@dont-email.me>

Show all headers | View raw

On 8/12/2016 1:40 PM, David Brown wrote:
> On 12/08/16 18:40, Don Y wrote:
>> On 8/12/2016 2:54 AM, David Brown wrote:
>>> On 12/08/16 00:30, Don Y wrote:
>>>> On 8/11/2016 3:17 PM, pozz wrote:
>>>>
>>>> [much elided]
>>>>
>>>>> "Everything in one structure" means you have a global (as the scope
>>>>> in C)
>>>>> struct definition that is accessed by every piece of code in the
>>>>> project, even
>>>>> if they are very different section of the project.
>>>>
>>>> No.  It only has to be *effectively* global.
>>>>
>>>> E.g., you can have a global "update_parameter()" that takes a
>>>> "key" (name for a parameter -- this could be a literal string
>>>> *or* just a small integer) and a parameter value (or pointer to
>>>> a parameter value).  Update_parameter() then moves the "value"
>>>> into the referenced portion of the "parameters to be saved struct".
>>>
>>> First, /don't/ use strings for that sort of thing unless you have no
>>> other choice.  And don't use arbitrary integers.  Use an enum, that is
>>> defined in one place.  That way you get it right.  Your code refers to
>>> the parameters using sensible names, rather than numbers, you don't get
>>> overlap or gaps (unless you want them, of course), and you can easily
>>> change things when you need to.  And when one part of your program
>>> writes parameter "backgroundColour" and another part reads parameter
>>> "backgroundColor", you get a compile-time error rather than weird
>>> behaviour at run-time.
>>
>> You throw an error at run-time -- because you *still* have to do
>> run-time checks on parameters and their values!
>
> The difference here is that with checks on parameter values, you can (normally)
> just set them to safe default values if they occur, and it is something that
> typically would never happen.
>
> But if you have a spelling mistake in your texts, your program is likely to
> silently ignore storage of changes in parameters.

Why would your program silently ignore them?  Don't you write ROBUST code?

If you tried to set parameter 47 to 982 -- but there were only 42 supported
parameters, would your code "silently ignore" this?  Or, if parameter 47
*did* exist but expected to be set to a string, would you ignore the
attempt to set it to a numeric?

>> And, it would be silly to type out the name of the key in each
>> invocation.  Instead, you'd use:
>>
>> #define PARAM_BACKGROUND_COLOR ("bacKgRouNdCoLOR")
>> #define PARAM_FOREGROUND_COLOR ("4GrouNDcoLOUr")
>> etc.
>>
>> How is this any different from:
>>
>> #define PARAM_BACKGROUND_COLOR (1)
>> #define PARAM_FOREGROUND_COLOR (2)
>> etc.?
>
> It is different, because it is - IMHO - daft.  Working with strings like this
> is vastly less efficient, and more error prone.   And if you've got a good

So, if you opted to use an int as a tag/key, you'd insist on using values
like 1, 2, 3, 4, ... instead of "BACK", "FORE", "BORD", for your tags?
(i.e., each of these are 32b integers)

And, force yourself to maintain a cheat sheet mapping *arbitrary* integers
to these interpretations -- just because your grade school teacher taught
you to start counting from '1'?

Using strings is "daft" when you don't have the resources (memory/MIPS)
to do so.  Forcing yourself to use enums/integers when you *do* have
the resources is "daft"!

(why isn't the windows registry just one big set of binary tuples?  why
aren't UN*X configuration files encoded in some magical, highly space and
time efficient manner -- with a suitable "decoder" to allow for the
*infrequent* human use?  why do databases give names to fields instead
of just 'field1', 'field2', etc.?)

> place to write out these #defines, then you are better off using an enum there
> with numerical values:
>
> enum {
>      paramBackgroundColour,
>      paramForegroundColour,
>      etc.
> }
>
> It's far more efficient, easier on the eye, easier to get IDE help in automatic
> completion, and easier for the compiler to check (such as compiler warnings if
> you use switch on an enum but forget some cases). And if you are using C++
> rather than C, you can use type-safe strong enums, as well as have better
> scoping and namespace usage.

And, all it ADDS is the requirement that you insert this extra step
(and keep it consistently, everywhere you use it).

I just got notified of an "error 27".  What's that mean?  What's
the symbolic label associated with that numeric value?  *Then*,
what does that symbolic label actually *mean*?

[Isn't this the same argument applied to "error codes"?]

>> Enums, IMO, are a bad idea -- it's far too easy to slip a new one
>> "in the middle" and effectively bodge the keys associated with all
>> those that follow.  I.e., a software revision changes:
>>
>> {PARAM_BACKGROUND_COLOR, PARAM_FOREGROUND_COLOR, ...}
>> to
>> {PARAM_BACKGROUND_COLOR, PARAM_BORDER_COLOR, PARAM_FOREGROUND_COLOR, ...}
>>
>> and now any "legacy" configuration files are effectively unreadable.
>
> If that is a concern (and it sometimes /is/ a concern), give the enum explicit
> values:
>
> enum {
>      paramBackgroundColour = 1,
>      paramForegroundColour = 2,
>      etc.
> }

This is then another maintenance issue:  ensuring that paramBorderColor gets
MANUALLY assigned the correct value regardless of where it appears in
the list of enumerations:
    enum {
       paramBackgroundColour = 1,
       paramForegroundColour = 2,
       paramBorderColor = 27,
       paramShadowColor = 3,   // return to the value that would have been used
                               // had not paramBorderColor been introduced

or, alternatively:
    enum {
       paramBackgroundColour = 1,
       paramForegroundColour = 2,
       paramShadowColor,   // 3
       ...   // manually skip over 24 labels (assuming none of them explicitly
             // alter their default values) to ensure we're at "27" when the
             // next label (border color) is encountered
       paramBorderColor,   // 27

>> (How do you refer to "old_PARAM_FOREGROUND_COLOR" in the code that
>> attempts to patch-up a legacy configuration dataset in light of
>> the *new* "PARAM_FOREGROUND_COLOR"?  I.e., you end up having to create
>> another enumeration for the legacy representation -- using DIFFERENT
>> names!)
>
> And how is that worse than using different names with #define'd values?

Because there is no "old_" value!  The "old" and "new" are exactly the same!
You just add new values to the end and obsolete values "in the middle".

#define PARAM_BACKGROUND_COLOR (3)
#define PARAM_FOREGROUND_COLOR (4)
#define PARAM_BORDER_COLOR (5)
#define PARAM_SHADOW_COLOR (6)

becomes:

#define PARAM_BACKGROUND_COLOR (3)
#define PARAM_FOREGROUND_COLOR (4)
#define PARAM_BORDER_COLOR (5)         // no longer used
#define PARAM_SHADOW_COLOR (6)

when the border color parameter is obsoleted.  So, if you encounter
a "configuration struct" that references parameter 5, you know that
it is referencing the obsolete border color parameter -- not the
*new* "highlight color" that took its place in the list of numerical
parameter identifier values.

>> By contrast, if you manually enumerate them, you are keenly aware of
>> the original (name,key) mapping.  Add new names to the end of the
>> list (who cares if "COLOR_X" and "COLOR_Y" have adjacent keys?!).
>> And, you can simply put a comment after any keys that have been obsoleted
>> over time:
>>
>> #define PARAM_VARIABLE_X (...)   // obsolete as per rev x.y.z
>>
>> Strings have value in that you can *see* them in logs:
>>     log( "Updating parameter %s to value %s", param, value )
>> instead of:
>>     log( "updating parameter %d to value %d", param, value )
>> or (ick):
>>     log( "Updating parameter %s to value %s",
>>          param_to_string(param), value_to_string(value) )
>>
>> Integers (and enums) suffer in poorly typed languages:
>>     update_parameter(PARAM_BACKGROUND_COLOR, VALUE_BROCOLLI);
>>
>> I.e., there's no way to ensure only the values valid for
>> PARAM_BACKGROUND_COLOR are applied to PARAM_BACKGROUND_COLOR!
>> The above function call may succeed (depends on whether
>> VALUE_BROCOLLI happens to coincide with a value for a
>> PARAM_BACKGROUND_COLOR) -- or not.  I.e., you still have to
>> rely on run-time checks; you can't do it all at compile time.
>>
>> In my current project, parameters are just fields in an RDBMS table.
>> So, the RDBMS *enforces* "correct" (in the sense of "valid") values.
>> Can't set Background_Color to "Blue" unless "Blue" is a valid choice
>> for *that* field.  And, surely can't set it to "Brocolli"!  :>
>>
>> And, as the RDBMS is charged with maintaining the integrity of this
>> data, there's no need to *check* it when reading it from the RDBMS
>> (it was checked "on the way in"!)
>>
>> As everything is SQL, I *see* "Blue" not some arbitrary
>> *encoding* of PARAM_BACKGROUND_COLOR_BLUE (which might differ
>> from PARAM_FOREGROUND_COLOR_BLUE -- even though they are the
>> same color!)
>
> I don't disagree that storing things as strings can be useful sometimes.  But
> we are not talking about big, wasteful PC software - we are talking about a
> small number of parameters in a small embedded program.  SQL is too big by

Who said it was a "small embedded program"?  Did I miss that, somewhere
upthread??  Just because its "embedded" doesn't implicitly make it "small".

How many Linux-based appliances still use fstab(5), ethers(5), inetd.conf(5),
etc.?  Surely they could replace those with "compiled" versions and elide
all of that *text* from their configurations!  After all, they already
*have* run-time code to parse those configuration files and configure
the system accordingly; why not just skip the "text processing" and store
a bunch of magic integers in a file, someplace?

> orders of magnitude for such a system.  It would a different matter if we were
> talking about an embedded linux system, for example - then you just write your
> code in Python, use a JSON string to hold a dict of the parameters, and you've
> got the system working in an hour of developer time.  Different tools for
> different tasks.

Exactly.  If I tried to replace the *thousands* of "configuration
parameters" in my current system with enums/small integers, I (and
anyone following after me) would spend all out time consulting cheat
sheets to try to remember which parameter controls which set of options.

And, of course, document all of that nicely as it evolves.

Even the little /pro bono/ job I'm working on this month has enough
"settings" to make that a daunting undertaking.  Much easier to
just say:
    result = write("/dev/display/refresh", "60")
than to sort out which ioctl(2) sets the display refresh rate to 60Hz.
Or:
    result = write("/dev/eia1/ctl", "odd")
instead of trying to remember if the manifest constant for parity
is PARODD or !PAREVEN.

Likewise, the "result" need not be some cryptic value that the
developer needs to look up and then translate into a *familiar*
form (so all developers use similar reporting messages):
    if (result != nil)
       print("Error: " result)

[This also seems to be a trend in CONFIGURING devices and apps in
other OS's as well:  "mixerctl outputs.headphones=+5" ]

If I had to provide a set of manuals enumerating all sorts of "error codes",
no one would ever attempt to alter the system from its default state.
"How do I change the color of the background?  *Why* didn't my
attempt to change the background to "vilet" [sic] fail?"

>>>> E.g., you can pack a bunch of (potentially unrelated) parameters
>>>> into a bitfield within that struct.  Data coming *into* it gets
>>>> massaged to manipulate the appropriate fields.  Likewise for data
>>>> coming *out*.
>>>
>>> That sounds like a complication that is unlikely to be worth the effort,
>>> but it's possible.  In many cases its enough to simply say "all
>>> parameters are 32 bits - it's up to the app if this is 4 characters, a
>>> float, an integer, etc.".
>>
>> Depends on resources available *and* how parameters are accessed.
>> Surely easier to fetch a *set* of related small fields in one
>> access than it is to have to fetch several individual fields
>> (each consisting of lots of 0 bits!)
>
> If you have the resources to use string keys for the parameters, you have the
> resources for using individual data items for your bits.

You're assuming all of my comments apply to  single implementation.
I'm presenting options that have/can be used for a variety of scenarios.
Many are inconsistent with each other (but the OP hadn't indicated which
SPECIFIC capabilities/characteristics his system involves)

> And if you have a number of related bits to track together, then use a flags
> parameter - have the application module keep the flags in a uint and pass them
> all together.
>
>> It also hides the representation from the caller.  Should I store
>> the time-of-last-update as a time_t?  Or, as some sort of string?
>> Or, as something that really only makes sense to the routine that
>> does the updates?
>>
>>>> Finally, because this represents a monitor, of sorts, you can ensure
>>>> atomic operations on that "data to be preserved" -- up to and including
>>>> an action that automatically saves the struct to persistent store
>>>> when power is failing (the rest of the time, "updates" happen to
>>>> the "in-RAM" copy of the struct so you don't incur the costs of
>>>> updating FLASH more often than necessary)
>>>
>>> Also true - it's always important to make sure your data is consistent
>>> at all times.

Thread

What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-11 14:48 +0200
  Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 06:27 -0700
  Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-11 16:12 +0200
    Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 07:53 -0700
      Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 08:27 -0700
        Re: What is your strategy to save non volatile data? bitrex <bitrex@de.lete.earthlink.net> - 2016-08-11 12:11 -0400
          Re: What is your strategy to save non volatile data? bitrex <bitrex@de.lete.earthlink.net> - 2016-08-11 12:12 -0400
            Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 11:35 -0700
              Re: What is your strategy to save non volatile data? bitrex <bitrex@de.lete.earthlink.net> - 2016-08-11 20:59 -0400
          Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 11:33 -0700
        Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-12 00:21 +0200
          Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 15:40 -0700
      Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-11 23:07 +0200
        Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-12 00:17 +0200
          Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 15:30 -0700
            Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-12 09:02 +0200
              Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-12 09:29 -0700
            Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 11:54 +0200
              Re: What is your strategy to save non volatile data? John Devereux <john@devereux.me.uk> - 2016-08-12 13:14 +0100
                Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 15:22 +0200
              Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-12 09:40 -0700
                Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 22:40 +0200
                Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-13 09:44 -0700
                Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-14 23:01 +0200
          Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 09:52 +0200
            Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-12 16:30 +0200
              Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 17:09 +0200
        Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 15:36 -0700
        Re: What is your strategy to save non volatile data? Paul <paul@pcserviceselectronics.co.uk> - 2016-08-12 03:21 +0100
    Re: What is your strategy to save non volatile data? bitrex <bitrex@de.lete.earthlink.net> - 2016-08-11 12:19 -0400
      Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-11 23:49 +0200
        Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-11 15:37 -0700
    Re: What is your strategy to save non volatile data? John Devereux <john@devereux.me.uk> - 2016-08-11 20:50 +0100
    Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-12 00:16 +0200
      Re: What is your strategy to save non volatile data? Paul <paul@pcserviceselectronics.co.uk> - 2016-08-12 03:34 +0100
      Re: What is your strategy to save non volatile data? David Brown <david.brown@hesbynett.no> - 2016-08-12 11:59 +0200
  Re: What is your strategy to save non volatile data? Paul Rubin <no.email@nospam.invalid> - 2016-08-11 11:29 -0700
  Re: What is your strategy to save non volatile data? Les Cargill <lcargill99@comcast.com> - 2016-08-12 12:15 -0500
  Re: What is your strategy to save non volatile data? Joe Chisolm <jchisolm6@earthlink.net> - 2016-08-12 12:10 -0500
    Re: What is your strategy to save non volatile data? Paul <paul@pcserviceselectronics.co.uk> - 2016-08-12 22:00 +0100
  Re: What is your strategy to save non volatile data? Tim Wescott <tim@seemywebsite.com> - 2016-08-13 12:06 -0500
    Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-13 23:42 +0200
      Re: What is your strategy to save non volatile data? Don Y <blockedofcourse@foo.invalid> - 2016-08-13 15:55 -0700
      Re: What is your strategy to save non volatile data? Paul <paul@pcserviceselectronics.co.uk> - 2016-08-14 00:08 +0100
      Re: What is your strategy to save non volatile data? Tim Wescott <tim@seemywebsite.com> - 2016-08-13 19:06 -0500
        Re: What is your strategy to save non volatile data? pozz <pozzugno@gmail.com> - 2016-08-22 09:46 +0200

csiph-web