Re: Serial EEPROM or Serial Flash?

From	David Brown <david.brown@hesbynett.no>
Newsgroups	comp.arch.embedded
Subject	Re: Serial EEPROM or Serial Flash?
Date	2018-06-17 18:41 +0200
Organization	A noiseless patient Spider
Message-ID	<pg62v7$hq5$1@dont-email.me> (permalink)
References	(2 earlier) <pfvqei$9t7$1@dont-email.me> <pg00mh$cmj$1@dont-email.me> <pg0ds2$sal$1@dont-email.me> <pg0i0n$nlr$1@dont-email.me> <pg0lfs$es1$1@dont-email.me>

Show all headers | View raw

On 15/06/18 17:20, pozz wrote:
> Il 15/06/2018 16:21, David Brown ha scritto:
>> On 15/06/18 15:10, pozz wrote:
>>> Il 15/06/2018 11:25, David Brown ha scritto:
>>>> On 15/06/18 09:38, pozz wrote:
>>>>> Il 14/06/2018 12:49, David Brown ha scritto:
>>>>>> On 14/06/18 12:20, pozz wrote:
>>>>>>> I need to save some data on a non-volatile memory. They are some
>>>>>>> parameters that the user could change infrequently, for example 10
>>>>>>> times
>>>>>>> per day at a maximum.  In the range 100-2kB.
>>>>>>>
>> <snip>
>>>>>>
>>>>>> Multiple copies (at least 2) are key, along with timestamps or 
>>>>>> counters
>>>>>
>>>>> Two should be sufficient to prevent corruption caused by interruption
>>>>> during the writing of a block of data.
>>>>> Maybe three (or more) are needed to face memory *physical* corruption
>>>>> (maybe for many writings).
>>>>> I think I can ignore this event in my actual project.
>>>>>
>>>>
>>>> You use more for wear levelling.  Often your flash chip is /way/ bigger
>>>> than you need - perhaps by a factor of 1000 simply because that's what
>>>> you have on stock, or that's the cheapest device in the package you
>>>> want.  Spread your writes over 1000 blocks instead of 2, and you have
>>>> 500 times the endurance.  Use a good checksum and accept that sometimes
>>>> a block will be worn out and move onto the next one, and you have
>>>> perhaps a million times the endurance (because most blocks last a lot
>>>> longer than the guaranteed minimum).
>>>
>>> So the writing process should be:
>>>
>>> 1. write the data at block i+1 (where i is the block of the current
>>>    data in RAM)
>>> 2. read back the block i+1 and check if checksum is ok
>>> 3. if ok, writing process is finished
>>> 4. if not, go to block i+2 and start again from 1.
>>>
>>
>> Yes.
>>
>> Then comes step 5 (for flash with separate erasing) :
>>
>> 5. If you have written block x, check if block x+1 (modulo the size of
>> the device) is erased.  If not, then erase it ready for the next write.
>>
>> Note that it does not matter if the erase block size is bigger than the
>> program block size - if it is, then your "erase block x+1" command will
>> cover the next few program blocks.
>>
>>>>>> and checksums.
>>>>>
>>>>> This is interesting.  Why do I need a checksum?  My approach is to use
>>>>> only a magic number plus a counter... and two memory areas.
>>>>> At first startup magic number isn't found on any areas, so the device
>>>>> starts with default and write data on Area1 (magic/0, where 0 is the
>>>>> counter).
>>>>
>>>> You use checksums to ensure that you haven't had a power-out or 
>>>> reset in
>>>> the middle of writing,
>>>
>>> Only for this thing, you can write the counter as the last byte. If the
>>> writing is interrupted in the middle, counter hasn't written yet, so the
>>> block is not valid (because considered too old or empty).
>>
>> Nope.  You can't rely on that, unless you are absolutely sure that you
>> have a simple, old-fashioned E²PROM device.  Many devices are /not/ like
>> that, even if they provide an interface that matches it logically.
>>
>> A common structure for a modern device is to have 32-byte pages as a
>> compromise between density, cost, and flexibility.  (Bigger pages are
>> more efficient in device area and cost.)  When you send a command to
>> write a byte, the device reads the old 32-byte page into ram, erases the
>> old page, updates the ram with the new data, the writes the whole 32
>> byte page back in.
>>
>> The write process is done by a loop that writes all the data, reads it
>> back at a low voltage to see if it has stuck, and writes again as needed
>> until the data looks good.  Then it writes again a few times for safety
>> - either a fixed number, or a percentage of the number of writes taken.
>>
>> So it is /entirely/ possible for an interrupted write to give you a
>> valid counter, but invalid data.  It is also entirely possible to get
>> some bits of the counter as valid while others are still erased (giving
>> ones on most devices).
>>
>> And that is just for simple devices that don't do any fancy wear
>> levelling, packing, garbage collection, etc.
>>
>>
>>>
>>>> and that the flash has not worn out.
>>>>
>>>>>
>>>>> When the configuration is changed, Area2 is written with magic/1, 
>>>>> being
>>>>> careful to save magic/1 only at the end of area writing.
>>>>>
>>>>> At startup magics and counters from both area are loaded, and one area
>>>>> is chosen (magic should be valid and counter should be the maximum).
>>>>>
>>>>> I think this approach works, even when the area writing is interrupted
>>>>> at the middle.
>>>>>
>>>>> Why do I need checksum?  The only thing that comes in mind is to 
>>>>> prevent
>>>>> writing errors: for example, I want to write 0x00 but the value really
>>>>> written is 0x01, maybe for a noise on the serial bus.
>>>>>
>>>>> To solve this situation, I need checksum... but also I need to re-read
>>>>> and re-calculate the checksum at *every* area writing... and start 
>>>>> a new
>>>>> writing if something was wrong.
>>>>>
>>>>> Do you have a better strategy?
>>>>
>>>> You calculate the checksum for a block before writing it, and you check
>>>> it when reading it.  Simple.
>>>
>>> Do you calc the checksum of all the data block in RAM, including padding
>>> bytes?
>>
>> Yes, of course.  The trick is not to have unknown padding bytes.  I make
>> a point of /never/ having compiler-generated padding in my structs.
>>
>> So you have something like this:
>>
>> #define sizeOfRawBlock 32
>> #define noOfRawBlocks 4
>> #define magicNumber 0x9185be91
>> #define dataStructVersionExpected 0x0001
>>
>> typedef union {
>>     uint8_t raw8[sizeOfRawBlock * noOfRawBlocks];
>>     uint16_t raw16[sizeOfRawBlock * noOfRawBlocks / 2];
>>     struct {
>>         uint32_t magic;
>>         uint16_t dataStructVersion;
>>         uint16_t crc;
>>         uint32_t count;
>>
>>         // real data
>>     }
>> } nvmData_t;
>>
>> static_assert(sizeof(nvmData_t) == (sizeofRawBlock * noOfRawBlocks),
>>      "Check size of nvnData!");
> 
> Why raw8[]?
> 
> I think you can avoid raw16[] too. If you have the function:
> 
>     uint16_t calc_crc(const void *data, size_t size);
> 
> you can simply call:
> 
>     nvmData.crc =
>       calc_crc( ((unsigned char *)&nvmData) + 8, sizeof(nvmData) );
> 

No, you can't necessarily do that.  C has rules about what pointers can 
alias what data in what ways - so the compiler knows the data inside the 
struct is independent of a pointer to an uint16_t (such as you might use 
in the crc function), unless they are part of a union.  In many cases, 
the compiler can't actually make use of such information for 
optimisation - so code messing about with pointers of different types 
will work.  But sometimes it won't, especially with higher optimisation 
choices and link-time optimisation.  It is much better to get in the 
habit of doing things correctly - if you want to access the data as 
16-bit or 32-bit blocks (for checksums, for copying data, for passing in 
bulk to external memory, etc.) then use a union here.  It might not be 
necessary, depending on the rest of your code (and by C's alias rules, 
the 8-bit raw is not needed), but I'm showing the principle here.

> 
>> When you want to store the data, run your crc calculation over the
>> raw16[] from index 4 upwards, and set it in the crc field.  Set the
>> magic field with a different fixed value per program.  Use the
>> dataStructVersion to track versions of the structure of the data - so
>> that updated firmware can recognise old data and quietly update it. 
> 
> This is interesting.  Suppose you have a new struct version and you are 
> reading an old version from the memory.  How do you read it?  Do you put 
> it completely in RAM and after copy it in the final struct making the 
> changes from one version to the other?  In this case you need a double 
> space in RAM, for old and new version.
> 

That depends on the application.  I'm not doing /all/ your work for you :-)

> Or you declare a "unsigned char reserved_for_future[1024]" big array in 
> the struct, so the CRC is calculated over the full area?
> 

Keeping some extra space for the future is a good idea.  How much you 
use is a matter of taste.

> 
>> Set the count to a new number each time.
>>
>> Leave plenty of room to add new fields to your nvm structure.
>>
>>
>>>
>>>
>>>>>>> This means I need to copy&paste an entire block everytime, even 
>>>>>>> for a
>>>>>>> single byte change.  And this is similar to Flash approach, where I
>>>>>>> *need* a sector erase before changing a single byte.
>>>>>>
>>>>>> Yes.
>>>>>>
>>>>>> Of course, it is possible to have multiple smaller blocks rather than
>>>>>> just one big one.
>>>>>
>>>>> This is good to reduce writing time of a block of data.
>>>>> In my case I have 2kB of data, around 62 32-bytes EEPROM pages, 
>>>>> that is
>>>>> 5*62=300ms.  I don't like to block a task (in a cooperative OS) for so
>>>>> long time.
>>>>
>>>> Your writing should be done in the background.  The background writer
>>>> task is blocked - the system is not.
>>>
>>> Are you thinking of a preemption OS?  I'm using a cooperative OS.
>>>
>>
>> It doesn't matter.  In a pre-emptive OS, your background writer is:
>                             ^^^^^^^^^^^
> You mean cooperative.

Yes.

> 
>>
>> static enum { idle, writing, checking, erasing } writerState;
>>
>> void doWriter(void) {
>>     switch (writerState) {
>>         case idle :
>>             if (writeTriggered) {
>>                 writeTriggered = false;
>>                 startWriting();
>>                 writerState = writing;
>>             }
>>             break;
>>         case writing :
>>             bool stillWorking = pollNVMdevice();
>>             if (stillWorking) return;
>>             startChecking();
>>             writerState = checking;
>>             break;
>>         case checking :
>>             bool stillWorking = pollNVMdevice();
>>             if (stillWorking) return;
>>             startErasingNextBlock();
>>             writerState = erasing;
>>             break;
>>         case erasing :
>>             bool stillWorking = pollNVMdevice();
>>             if (stillWorking) return;
>>             writerState = idle;
>>             break;
>>     }
>> }
>>
>> There - half your program is done :-)
>>
>> Cooperative multitasking does not mean busy waiting for long-running
>> tasks.  It means using state machines, interrupt-driven SPI/I²C drivers,
>> polling, etc., so that you can mix fast and slow tasks.
> 
> Yes, ok.
>

Thread

Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-14 12:20 +0200
  Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-14 12:49 +0200
    Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 09:38 +0200
      Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 00:54 -0700
        Re: Serial EEPROM or Serial Flash? Reinhardt Behm <rbehm@hushmail.com> - 2018-06-15 16:26 +0800
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 01:31 -0700
          Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 10:50 +0200
            Re: Serial EEPROM or Serial Flash? Reinhardt Behm <rbehm@hushmail.com> - 2018-06-15 17:03 +0800
              Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 14:59 +0200
                Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 15:51 +0200
        Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 10:47 +0200
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 02:20 -0700
            Re: Serial EEPROM or Serial Flash? Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2018-06-15 13:41 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 15:01 +0200
      Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 11:25 +0200
        Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 15:10 +0200
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 06:51 -0700
          Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 16:21 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 17:20 +0200
              Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-15 13:11 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 22:54 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-15 18:25 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 08:26 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-16 09:20 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 19:38 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-16 16:41 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 23:05 +0200
              Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-17 18:41 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 20:12 +0200
              Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-17 19:27 +0200
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-19 17:40 +0200
              Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-17 17:05 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-19 17:44 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-19 23:14 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-20 11:03 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-20 08:03 -0400
  Re: Serial EEPROM or Serial Flash? Michael Karas <mkaras@carousel-design.com> - 2018-06-23 09:43 -0700

csiph-web