Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #27825

Re: Serial EEPROM or Serial Flash?

Show all headers | View raw

Il 16/06/2018 15:20, Richard Damon ha scritto:
> On 6/16/18 2:26 AM, pozz wrote:
>> Il 16/06/2018 00:25, Richard Damon ha scritto:
>>> On 6/15/18 4:54 PM, pozz wrote:
>>>> Il 15/06/2018 19:11, Richard Damon ha scritto:
>>>>> I typically also do something like this. The data structure is a union
>>>>> of the basic data structure with a preamble that includes (in very
>> fixed
>>>>> locations) a data structure version, checksum/crc, and if a versioning
>>>>> store a timestamp/data generation number. A 'Magic Number' isn't often
>>>>> needed unless it is a removable media as it will either be or not the
>>>>> expected data, nothing else could be there (if the unit might have
>>>>> different sorts of programs, then a piece of the data version would
>> be a
>>>>> program ID.)
>>>>>
>>>>> Often I will have TWO copies of the data packet.
>>>>
>>>> TWO copies in EEPROM or in RAM?
>>>
>>> As I described, normally both.
>>
>> So you are so lucky that you have abundant RAM.
> 
> If you can't spare the space for the two buffers, then you become forced
> to 'lock' the system (or at least the parameter table) for the entire
> time of the flash write, otherwise you end up with corrupted (CRC error)
> data blocks. 

Why?  You have a single datablock in RAM that must be transfered to the 
serial memory when changes occur.  You start an activity (a task) that 
performs the copy in background.
If new changes occur during writing, the task can be restarted without 
problems, even on the same memory area.  I hope the changes aren't so 
frequent so finally you will have a valid datablock (with correct CRC) 
written in memory, without blocking the main application.

Why do you need an additional datablock to avoid blocking task?


> It also makes writing the format uprev code much easier.
> You comment about how to update 'in place' becomes a REAL question which
> needs to be answered, and that answer will generally require detailed
> understanding of how the elements are physically stored and making sure
> that you do things in the right order (which may require limiting the
> optimizations the compiler can do for that code, as you are likely to do
> actions the Standard defines as Undefined Behavior).

Yes, solving the problem of converting an old layout in a new completely 
different layout is very complex without an additional structure.  The 
in-place conversion would be very difficult and depends on the changes.

I think you can avoid all this stuff, as I said, changing the datablock 
layout with care (adding parameters at the end, without changing the 
first/old layout).


>>>>> Often organized something like this:
>>>>>
>>>>> struct SystemParameters {
>>>>>         // List of data parameters that need to be saved
>>>>> } parms;
>>>>>
>>>>> union {
>>>>>        uint8_t    raw_data[DATA_SIZE];
>>>>>        struct {
>>>>>            struct StandardHeader header;
>>>>>            struct SystemParameters parms;
>>>>>
>>>>> } flash_parms;
>>>>>
>>>>> /* Check that sizeof(flash_parms) == DATA_SIZE, and that DATA_SIZE is
>>>>> a multiple of the flash sector size */
>>>>
>>>> Do you need raw_data[] array only to set a well-known size to
>> flash_parms?
>>>> Do you alloc an entire (or a multiple of) flash sector in RAM?  Even if
>>>> your params in flash take only 1/5 or 1/2 or 1/10 of a sector?
>>>>
>>>
>>> One purpose of the raw_data array is to fix the size of the flash data.
>>
>> Sincerely I can't understand why you need a fixed data size in RAM.
>> Suppose release 1.0 needs 2kB, but I'm so smart to provide a double
>> space for new params in future releases. So I decide to use 4kB blocks
>> *on the serial memory*.  Maybe this big space will never be used in all
>> the future releases.
>>
>> Why the hell I should load all 4kB datablock in RAM at startup?  IMHO it
>> is sufficient to load only 2kB.
>>
>> The only reason I see is to simplify the checksum calculation.
>> CRC must be calculated over the entire fixed-size 4kB block, otherwise
>> how to know the datablock size if it could change from one release to
>> the other?
>> However you can calc the checksum of 4kB block by reading small blocks
>> multiple times (for example, 16 sections of 256 bytes).
>> After CRC validation passed, you can read in RAM *only* 2kB of data that
>> the application 1.0 needs.
>>
>> I know, maybe this process is slower, but it happens only at startup.
>> During writing of 4kB datablock, the checksum can be calculated on-the-fly.
>>
> A big reason for having a copy in RAM being an exact copy of the flash
> memory is design isolation. It allows me to define a checksum routine to
> compute the value of an arbitrary buffer, and flash routines that
> transfer arbitrary data without it needing full understanding of the
> parameter storage.

Yes, your earn some cents on implementation of some functions.


> As to the use of extra memory, If I anticipate that I am going to
> possibly need a 4k parameter block in the future, but don't have the ram
> memory now to read in the 4k block, most assuredly in the future when I
> do need that bigger parameter block, I am not going to have more ram on
> THIS processor. Unless the flash memory is a removable media (which
> presents a totally different set of issues) there is no need to reserve
> the extra space for this processor.

When you reserve 4k block for future uses, you don't know if it will be 
REALLY used in the future.  You are right when you say "no space now, no 
space in the future", but are we sure we can know if we have space or not?

I don't know if my approach is correct, I usually don't use heap memory, 
so I have only static data and stack.  It's very difficult to calculate 
how much space you need for stack to avoid stack overflow in ANY 
condition.  So I end up to reserve all the free RAM space to stack, 
where the free space is total RAM minus static allocation.

So I try being parsimonius when allocate global static variables, mostly 
if they are big.
Reserving some space for extra not used memory will sacrify stack space.

Maybe the stack space is sufficient even with extra padding in RAM, who 
knows.  At first I try to do all my best to avoid useless big static 
allocation in RAM.

At the contrary, what happens if you notice that your stack is critical 
when padding is in RAM?  Do you prefer to reduce the datablock *padding* 
size or to recode some functions to avoid having the padding at all in 
the RAM?


I think there is another approach to keep checksum functions simple and 
avoid having padding space in RAM.
The problem here is the space where CRC is calculated.  At startup we 
don't know the size of data, because it could belong to an old or new 
layout, so we can't calculate CRC on the right area.  We should know the 
data layout version before calculating CRC, but version code is *in* the 
CRC area.

We could think to pull the layout code outside the CRC area and protect 
it by duplicating.

typedef struct {
   uint8_t layout_version;
   uint8_t layout_version_check;
   uint16_t crc;
   /* The CRC is calculated starting from the following byte */
   uint16_t counter;
   union {
     nvmdatav1_t datav1;
     nvmdatav2_t datav2;
}

At startup the layout version is read and checked.  Then CRC is read and 
checked against the correct size that depends on the layout version. 
The CRC checksum is simple.

if (datablock.layout_version == datablock.layout_version_check) {
   size_t s = datablock.layout_version == 1 ?
              sizeof(nvmdatav1_t) :
              sizeof(nvmdatav2_t);
   if (datablock.crc ==
         calc_crc(&((const unsigned char *)&datablock)[5]),
              s) {
     if (datablock.counter > prev_counter) {
       /* New valid datablock found */
     }
   }
}

No padding at all (in RAM and in serial memory), simple functions.

Of course padding must be taken into account during writing to separate 
the blocks in memory for redundancy and wear-leveling.


>>> Another purpose is to provide an easy way to access to pre-'zero' the
>> data
>>> Another purpose is to provide an easy way to compute the CRC/Checksum.
>>>
>>> Yes, the last two could be done via type casting via void* parameters.
>>
>> Yes, indeed.
>>
>>
>>>>> And a request to save the data will fill flash_parms with a 'NULL'
>>>>> fill (0 or all ones, or whatever I want unused flash locations to be),
>>>>
>>>> Is it really necessary?  What happens if you don't care about unused
>>>> bytes in flash_parms?  I think... nothing.
>>>>
>>>>
>>>
>>> Setting them to what an erased byte is can sometimes save you power and
>>> time for programming. Having them be a know value can also make it
>>> easier to read the contents, and
>>>
>>>>> then
>>>>> copy the parms into flash_parms.parms, then fill in the header, and
>> then
>>>>> program the flash. Only the part that copys parms to flash_parms.parms
>>>>> needs to be protected from other operation in the system updating the
>>>>> main parms structure.
>>>>>
>>>>> The flash_parms structure can be further modified to be a union of the
>>>>> current data structure of the SystemParameters structure and any
>>>>> previous 'incompatible' version of it. (A version from just prior to
>>>>> adding elements at the end doesn't need to be listed). This way at
>>>>> startup I can read the values into that same structure, and if it was
>>>>> old, parse to old version in flash_parms to get the new version in
>>>>> parms, defaulting new parameters as needed.
>>>>
>>>> It seems difficult, if the changes from one version to the other are
>>>> "important".
>>>>
>>>> struct {
>>>>      uint16_t foo;
>>>>      uint16_t bar;
>>>>      uint16_t dummy;
>>>> } SystemParameters;
>>>>
>>> should be struct SystemParameters { or typedef struct { you are defining
>>> a type here, not a varaiable.
>>
>> Yes, you're right.
>>
>>
>>>> struct {
>>>>      uint8_t foo;
>>>>      uint8_t bar;
>>>>      uint8_t dummy;
>>>> } SystemParametersOld;
>>>
>>> ditto.
>>
>> Of course.
>>
>>
>>>> union {
>>>>       uint8_t    raw_data[DATA_SIZE];
>>>>       struct {
>>>>           struct StandardHeader header;
>>>>           struct SystemParameters parms;
>>>>           struct SystemParametersOld parms_old;
>>>> } flash_parms;
>>>>
>>> should be:
>>>
>>> union {
>>>       uint8_t raw_data[DATA_SIZE];
>>>       struct {
>>>          struct StandardHeader header;
>>>          union {
>>>             struct SystemParameters parms_new;
>>>             struct SystemParametersOld parms_old;
>>>          } parms;
>>>       } data;
>>> } flash_parms;
>>
>> Again you're right.
>>
>>
>>> I actually don't use labels like Old but like Rev1, Rev2 etc.
>>
>> Yes, it was only for the example here.
>>
>>
>>> For some compilers you can use an 'anonymous union' an omit the parms
>>> name here and in the code.
>>>>
>>>> for (uint8_t sector = 0; sector < SECTORS_NUM; sector++) {
>>>>      uint16_t memory_address = OFFSET + sector * SECTOR_SIZE;
>>>>      sermem_read(&flash_params, memory_address, sizeof(flash_params));
>>>>      uint16_t chk_read = flash_parms.header.crc;
>>>>      flash_parms.header.crc = 0xFFFF;
>>>>      if (checksum_calc(&flash_parms, sizeof(flash_parms)) == chk_read) {
>>>>        /* Datablock is valid */
>>>>        if (flash_parms.header.counter > prev_counter) {
>>>>          /* Datablock is newer */
>>>>          prev_counter = flash_parms.header.counter;
>>>>          if (flash_parms.header.version == FLASH_PARMS_OLD) {
>>>>            /* How to convert old version in new version in place? */
>>>>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>>          } else {
>>>>            /* Ok, new version. No convertion is needed. */
>>>>          }
>>>>        }
>>>>      }
>>>> }
>>>>
>>>> As I wrote in the comment, how to convert old version to new version,
>>>> using the same union in RAM?  Do I need another union?
>>>
>>> You do NOT convert 'in place', but in transfering from the flash_parms
>>> block to the main usage block. Only the flash parm save/load routine has
>>> access to flash_parms, and they are used just for save and load. If you
>>> are tight on memory, it could be allocated on the heap only for load
>>> and/or save and freed after, but you then you do need to be sure the
>>> memory IS available when you need it. maybe something else turns off for
>>> a bit when you need to do it.
>>>
>>> sample code:
>>>
>>> parms.foo = flash_parms.data.parms.parms_old.foo;
>>> parms.bar = flash_parms.data.parms.parms_old.bar;
>>> parms.dummy = flash_parms.data.parms.parms_old.dummy;
>>>
>>> and for the new code, you do
>>>
>>> parms = flash_parms.data.parms.parms_new;
>>>
>>> you can do a single line and let the compiler move it all.
>>
>> Now I understand your process.  You have an image in RAM of datablock
>> saved in serial memory *and* you have another "operative" datablock in
>> RAM (maybe identical, maybe differently organized) that is actually used
>> by the code.
>>
>> I don't like to waste so much space in valuable RAM memory, maybe
>> because I started with very limited 8-bit MCUs where the RAM was very
>> small.
>>
>> One good compromise could be to change datablock layout with great care,
>> only:
>> * adding variables at the end
>> * nullifying no-more used parameters by simply changing their name with
>> "_deprecated" (or "_old") suffix (in order to be sure the code will
>> never use them again).
>> Immediately after an upgrade, the "_old" params will be converted in new
>> params, if needed (for example if they change from uint8_t in uint16_t).
> 
> Yes, for VERY small processors this doesn't work. But programs on very
> small processors tend to have small parameter blocks and flashes with
> small sectors (at the very least you are going to probably need at least
> one memory block reserved the size of a flash sector, and the
> flash_parms structure can fit in that.
> 
> And yes, most of the time when you revise your needs in the parameter
> block, you add new items at the end, and if you stop using an item, it
> gets renamed with an distinctive suffix. When possible I try to maintain
> it having a reasonable representation of its value, so if you 'downrev'
> the code, it gets something reasonable. I try to use a semantic
> versioning method for my flash version numbers, so that code can decide
> if the parameter values are usable. If the major revision number is
> bigger than what it knows, it will refuse to use the parameter table
> (and either throw an error or revert to defaults). If the major revision
> matches, and the minor is greater than or equal to what it has, it just
> uses the parameters as is. If the major is equal and the minor is lower,
> after copying the parameters, it adds appropriate defaults (maybe
> computed from other items) to any items that have been added since the
> indicated revision. If the major revision is lower, then the code needs
> a full conversion routine as shown above (or throw the error/go to
> defaults for too old like too new).

Good suggestion.


> The changing of the type of a parameter is one of the cases where you
> are forced to bump the major revision, or you create new parameters in
> with the new type, and the code does its best to give the old, now
> obsolete, values as reasonable a value as possible in case of a down rev.

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-14 12:20 +0200
  Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-14 12:49 +0200
    Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 09:38 +0200
      Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 00:54 -0700
        Re: Serial EEPROM or Serial Flash? Reinhardt Behm <rbehm@hushmail.com> - 2018-06-15 16:26 +0800
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 01:31 -0700
          Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 10:50 +0200
            Re: Serial EEPROM or Serial Flash? Reinhardt Behm <rbehm@hushmail.com> - 2018-06-15 17:03 +0800
              Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 14:59 +0200
                Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 15:51 +0200
        Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 10:47 +0200
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 02:20 -0700
            Re: Serial EEPROM or Serial Flash? Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2018-06-15 13:41 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 15:01 +0200
      Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 11:25 +0200
        Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 15:10 +0200
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 06:51 -0700
          Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 16:21 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 17:20 +0200
              Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-15 13:11 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 22:54 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-15 18:25 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 08:26 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-16 09:20 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 19:38 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-16 16:41 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 23:05 +0200
              Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-17 18:41 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 20:12 +0200
              Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-17 19:27 +0200
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-19 17:40 +0200
              Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-17 17:05 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-19 17:44 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-19 23:14 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-20 11:03 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-20 08:03 -0400
  Re: Serial EEPROM or Serial Flash? Michael Karas <mkaras@carousel-design.com> - 2018-06-23 09:43 -0700

csiph-web