Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.arch.embedded > #27820

Re: Serial EEPROM or Serial Flash?

Show all headers | View raw

Il 15/06/2018 19:11, Richard Damon ha scritto:
 > On 6/15/18 11:20 AM, pozz wrote:
 >> Il 15/06/2018 16:21, David Brown ha scritto:
 >>> On 15/06/18 15:10, pozz wrote:
 >>>> Il 15/06/2018 11:25, David Brown ha scritto:
 >>>>> On 15/06/18 09:38, pozz wrote:
 >>>>>> Il 14/06/2018 12:49, David Brown ha scritto:
 >>>>>>> On 14/06/18 12:20, pozz wrote:
 >>>>>>>> I need to save some data on a non-volatile memory. They are some
 >>>>>>>> parameters that the user could change infrequently, for example 10
 >>>>>>>> times
 >>>>>>>> per day at a maximum.  In the range 100-2kB.
 >>>>>>>>
 >>> <snip>
 >>>>>>>
 >>>>>>> Multiple copies (at least 2) are key, along with timestamps or
 >>>>>>> counters
 >>>>>>
 >>>>>> Two should be sufficient to prevent corruption caused by 
interruption
 >>>>>> during the writing of a block of data.
 >>>>>> Maybe three (or more) are needed to face memory *physical* 
corruption
 >>>>>> (maybe for many writings).
 >>>>>> I think I can ignore this event in my actual project.
 >>>>>>
 >>>>>
 >>>>> You use more for wear levelling.  Often your flash chip is /way/ 
bigger
 >>>>> than you need - perhaps by a factor of 1000 simply because that's 
what
 >>>>> you have on stock, or that's the cheapest device in the package you
 >>>>> want.  Spread your writes over 1000 blocks instead of 2, and you have
 >>>>> 500 times the endurance.  Use a good checksum and accept that 
sometimes
 >>>>> a block will be worn out and move onto the next one, and you have
 >>>>> perhaps a million times the endurance (because most blocks last a lot
 >>>>> longer than the guaranteed minimum).
 >>>>
 >>>> So the writing process should be:
 >>>>
 >>>> 1. write the data at block i+1 (where i is the block of the current
 >>>>     data in RAM)
 >>>> 2. read back the block i+1 and check if checksum is ok
 >>>> 3. if ok, writing process is finished
 >>>> 4. if not, go to block i+2 and start again from 1.
 >>>>
 >>>
 >>> Yes.
 >>>
 >>> Then comes step 5 (for flash with separate erasing) :
 >>>
 >>> 5. If you have written block x, check if block x+1 (modulo the size of
 >>> the device) is erased.  If not, then erase it ready for the next write.
 >>>
 >>> Note that it does not matter if the erase block size is bigger than the
 >>> program block size - if it is, then your "erase block x+1" command will
 >>> cover the next few program blocks.
 >>>
 >>>>>>> and checksums.
 >>>>>>
 >>>>>> This is interesting.  Why do I need a checksum?  My approach is 
to use
 >>>>>> only a magic number plus a counter... and two memory areas.
 >>>>>> At first startup magic number isn't found on any areas, so the 
device
 >>>>>> starts with default and write data on Area1 (magic/0, where 0 is the
 >>>>>> counter).
 >>>>>
 >>>>> You use checksums to ensure that you haven't had a power-out or
 >>>>> reset in
 >>>>> the middle of writing,
 >>>>
 >>>> Only for this thing, you can write the counter as the last byte. 
If the
 >>>> writing is interrupted in the middle, counter hasn't written yet, 
so the
 >>>> block is not valid (because considered too old or empty).
 >>>
 >>> Nope.  You can't rely on that, unless you are absolutely sure that you
 >>> have a simple, old-fashioned E²PROM device.  Many devices are /not/ 
like
 >>> that, even if they provide an interface that matches it logically.
 >>>
 >>> A common structure for a modern device is to have 32-byte pages as a
 >>> compromise between density, cost, and flexibility.  (Bigger pages are
 >>> more efficient in device area and cost.)  When you send a command to
 >>> write a byte, the device reads the old 32-byte page into ram, 
erases the
 >>> old page, updates the ram with the new data, the writes the whole 32
 >>> byte page back in.
 >>>
 >>> The write process is done by a loop that writes all the data, reads it
 >>> back at a low voltage to see if it has stuck, and writes again as 
needed
 >>> until the data looks good.  Then it writes again a few times for safety
 >>> - either a fixed number, or a percentage of the number of writes taken.
 >>>
 >>> So it is /entirely/ possible for an interrupted write to give you a
 >>> valid counter, but invalid data.  It is also entirely possible to get
 >>> some bits of the counter as valid while others are still erased (giving
 >>> ones on most devices).
 >>>
 >>> And that is just for simple devices that don't do any fancy wear
 >>> levelling, packing, garbage collection, etc.
 >>>
 >>>
 >>>>
 >>>>> and that the flash has not worn out.
 >>>>>
 >>>>>>
 >>>>>> When the configuration is changed, Area2 is written with magic/1,
 >>>>>> being
 >>>>>> careful to save magic/1 only at the end of area writing.
 >>>>>>
 >>>>>> At startup magics and counters from both area are loaded, and 
one area
 >>>>>> is chosen (magic should be valid and counter should be the maximum).
 >>>>>>
 >>>>>> I think this approach works, even when the area writing is 
interrupted
 >>>>>> at the middle.
 >>>>>>
 >>>>>> Why do I need checksum?  The only thing that comes in mind is to
 >>>>>> prevent
 >>>>>> writing errors: for example, I want to write 0x00 but the value 
really
 >>>>>> written is 0x01, maybe for a noise on the serial bus.
 >>>>>>
 >>>>>> To solve this situation, I need checksum... but also I need to 
re-read
 >>>>>> and re-calculate the checksum at *every* area writing... and start
 >>>>>> a new
 >>>>>> writing if something was wrong.
 >>>>>>
 >>>>>> Do you have a better strategy?
 >>>>>
 >>>>> You calculate the checksum for a block before writing it, and you 
check
 >>>>> it when reading it.  Simple.
 >>>>
 >>>> Do you calc the checksum of all the data block in RAM, including 
padding
 >>>> bytes?
 >>>
 >>> Yes, of course.  The trick is not to have unknown padding bytes.  I 
make
 >>> a point of /never/ having compiler-generated padding in my structs.
 >>>
 >>> So you have something like this:
 >>>
 >>> #define sizeOfRawBlock 32
 >>> #define noOfRawBlocks 4
 >>> #define magicNumber 0x9185be91
 >>> #define dataStructVersionExpected 0x0001
 >>>
 >>> typedef union {
 >>>      uint8_t raw8[sizeOfRawBlock * noOfRawBlocks];
 >>>      uint16_t raw16[sizeOfRawBlock * noOfRawBlocks / 2];
 >>>      struct {
 >>>          uint32_t magic;
 >>>          uint16_t dataStructVersion;
 >>>          uint16_t crc;
 >>>          uint32_t count;
 >>>
 >>>          // real data
 >>>      }
 >>> } nvmData_t;
 >>>
 >>> static_assert(sizeof(nvmData_t) == (sizeofRawBlock * noOfRawBlocks),
 >>>       "Check size of nvnData!");
 >>
 >> Why raw8[]?
 >>
 >> I think you can avoid raw16[] too. If you have the function:
 >>
 >>     uint16_t calc_crc(const void *data, size_t size);
 >>
 >> you can simply call:
 >>
 >>     nvmData.crc =
 >>       calc_crc( ((unsigned char *)&nvmData) + 8, sizeof(nvmData) );
 >>
 >>
 >
 > I typically also do something like this. The data structure is a union
 > of the basic data structure with a preamble that includes (in very fixed
 > locations) a data structure version, checksum/crc, and if a versioning
 > store a timestamp/data generation number. A 'Magic Number' isn't often
 > needed unless it is a removable media as it will either be or not the
 > expected data, nothing else could be there (if the unit might have
 > different sorts of programs, then a piece of the data version would be a
 > program ID.)
 >
 > Often I will have TWO copies of the data packet.

TWO copies in EEPROM or in RAM?


 > Often organized something like this:
 >
 > struct SystemParameters {
 >     // List of data parameters that need to be saved
 > } parms;
 >
 > union {
 >    uint8_t	raw_data[DATA_SIZE];
 >    struct {
 >        struct StandardHeader header;
 >        struct SystemParameters parms;
 >
 > } flash_parms;
 >
 > /* Check that sizeof(flash_parms) == DATA_SIZE, and that DATA_SIZE is
 > a multiple of the flash sector size */

Do you need raw_data[] array only to set a well-known size to flash_parms?
Do you alloc an entire (or a multiple of) flash sector in RAM?  Even if 
your params in flash take only 1/5 or 1/2 or 1/10 of a sector?


 > And a request to save the data will fill flash_parms with a 'NULL'
 > fill (0 or all ones, or whatever I want unused flash locations to be),

Is it really necessary?  What happens if you don't care about unused
bytes in flash_parms?  I think... nothing.


 > then
 > copy the parms into flash_parms.parms, then fill in the header, and then
 > program the flash. Only the part that copys parms to flash_parms.parms
 > needs to be protected from other operation in the system updating the
 > main parms structure.
 >
 > The flash_parms structure can be further modified to be a union of the
 > current data structure of the SystemParameters structure and any
 > previous 'incompatible' version of it. (A version from just prior to
 > adding elements at the end doesn't need to be listed). This way at
 > startup I can read the values into that same structure, and if it was
 > old, parse to old version in flash_parms to get the new version in
 > parms, defaulting new parameters as needed.

It seems difficult, if the changes from one version to the other are 
"important".

struct {
   uint16_t foo;
   uint16_t bar;
   uint16_t dummy;
} SystemParameters;

struct {
   uint8_t foo;
   uint8_t bar;
   uint8_t dummy;
} SystemParametersOld;

union {
    uint8_t	raw_data[DATA_SIZE];
    struct {
        struct StandardHeader header;
        struct SystemParameters parms;
        struct SystemParametersOld parms_old;
} flash_parms;


for (uint8_t sector = 0; sector < SECTORS_NUM; sector++) {
   uint16_t memory_address = OFFSET + sector * SECTOR_SIZE;
   sermem_read(&flash_params, memory_address, sizeof(flash_params));
   uint16_t chk_read = flash_parms.header.crc;
   flash_parms.header.crc = 0xFFFF;
   if (checksum_calc(&flash_parms, sizeof(flash_parms)) == chk_read) {
     /* Datablock is valid */
     if (flash_parms.header.counter > prev_counter) {
       /* Datablock is newer */
       prev_counter = flash_parms.header.counter;
       if (flash_parms.header.version == FLASH_PARMS_OLD) {
         /* How to convert old version in new version in place? */
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       } else {
         /* Ok, new version. No convertion is needed. */
       }
     }
   }
}

As I wrote in the comment, how to convert old version to new version, 
using the same union in RAM?  Do I need another union?

 > Checksums / CRC are always over the whole data array (skipping over,
 > or using a fixed value, the value of it stored in the Header).
 >
 >
 >>> When you want to store the data, run your crc calculation over the
 >>> raw16[] from index 4 upwards, and set it in the crc field.  Set the
 >>> magic field with a different fixed value per program.  Use the
 >>> dataStructVersion to track versions of the structure of the data - so
 >>> that updated firmware can recognise old data and quietly update it.
 >>
 >> This is interesting.  Suppose you have a new struct version and you are
 >> reading an old version from the memory.  How do you read it?  Do you put
 >> it completely in RAM and after copy it in the final struct making the
 >> changes from one version to the other?  In this case you need a double
 >> space in RAM, for old and new version.
 >>
 >> Or you declare a "unsigned char reserved_for_future[1024]" big array in
 >> the struct, so the CRC is calculated over the full area?
 >>
 >>
 > (see above)
 >
 >>> Set the count to a new number each time.
 >>>
 >>> Leave plenty of room to add new fields to your nvm structure.
 >>>
 >>>
 >>>>
 >>>>
 >>>>>>>> This means I need to copy&paste an entire block everytime, even
 >>>>>>>> for a
 >>>>>>>> single byte change.  And this is similar to Flash approach, 
where I
 >>>>>>>> *need* a sector erase before changing a single byte.
 >>>>>>>
 >>>>>>> Yes.
 >>>>>>>
 >>>>>>> Of course, it is possible to have multiple smaller blocks 
rather than
 >>>>>>> just one big one.
 >>>>>>
 >>>>>> This is good to reduce writing time of a block of data.
 >>>>>> In my case I have 2kB of data, around 62 32-bytes EEPROM pages,
 >>>>>> that is
 >>>>>> 5*62=300ms.  I don't like to block a task (in a cooperative OS) 
for so
 >>>>>> long time.
 >>>>>
 >>>>> Your writing should be done in the background.  The background writer
 >>>>> task is blocked - the system is not.
 >>>>
 >>>> Are you thinking of a preemption OS?  I'm using a cooperative OS.
 >>>>
 >>>
 >>> It doesn't matter.  In a pre-emptive OS, your background writer is:
 >>                             ^^^^^^^^^^^
 >> You mean cooperative.
 >>
 > You can use cooperative or pre-emptive. If cooperative then your flash
 > write routine should yield (like the example seems to show) while it is
 > waiting for the flash to do the actual operations. (If pre-emptive you
 > just need to block for a bit and let the pre-emption do its thing, or
 > make the flash routine a background level operation so a spin wait
 > doesn't impact other operations)
 >
 >>>
 >>> static enum { idle, writing, checking, erasing } writerState;
 >>>
 >>> void doWriter(void) {
 >>>      switch (writerState) {
 >>>          case idle :
 >>>              if (writeTriggered) {
 >>>                  writeTriggered = false;
 >>>                  startWriting();
 >>>                  writerState = writing;
 >>>              }
 >>>              break;
 >>>          case writing :
 >>>              bool stillWorking = pollNVMdevice();
 >>>              if (stillWorking) return;
 >>>              startChecking();
 >>>              writerState = checking;
 >>>              break;
 >>>          case checking :
 >>>              bool stillWorking = pollNVMdevice();
 >>>              if (stillWorking) return;
 >>>              startErasingNextBlock();
 >>>              writerState = erasing;
 >>>              break;
 >>>          case erasing :
 >>>              bool stillWorking = pollNVMdevice();
 >>>              if (stillWorking) return;
 >>>              writerState = idle;
 >>>              break;
 >>>      }
 >>> }
 >>>
 >>> There - half your program is done :-)
 >>>
 >>> Cooperative multitasking does not mean busy waiting for long-running
 >>> tasks.  It means using state machines, interrupt-driven SPI/I²C 
drivers,
 >>> polling, etc., so that you can mix fast and slow tasks.
 >>
 >> Yes, ok.
 >>
 >

Back to comp.arch.embedded | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-14 12:20 +0200
  Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-14 12:49 +0200
    Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 09:38 +0200
      Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 00:54 -0700
        Re: Serial EEPROM or Serial Flash? Reinhardt Behm <rbehm@hushmail.com> - 2018-06-15 16:26 +0800
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 01:31 -0700
          Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 10:50 +0200
            Re: Serial EEPROM or Serial Flash? Reinhardt Behm <rbehm@hushmail.com> - 2018-06-15 17:03 +0800
              Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 14:59 +0200
                Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 15:51 +0200
        Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 10:47 +0200
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 02:20 -0700
            Re: Serial EEPROM or Serial Flash? Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2018-06-15 13:41 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 15:01 +0200
      Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 11:25 +0200
        Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 15:10 +0200
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 06:51 -0700
          Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 16:21 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 17:20 +0200
              Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-15 13:11 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 22:54 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-15 18:25 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 08:26 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-16 09:20 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 19:38 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-16 16:41 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 23:05 +0200
              Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-17 18:41 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 20:12 +0200
              Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-17 19:27 +0200
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-19 17:40 +0200
              Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-17 17:05 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-19 17:44 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-19 23:14 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-20 11:03 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-20 08:03 -0400
  Re: Serial EEPROM or Serial Flash? Michael Karas <mkaras@carousel-design.com> - 2018-06-23 09:43 -0700

csiph-web