Re: Serial EEPROM or Serial Flash?

From	David Brown <david.brown@hesbynett.no>
Newsgroups	comp.arch.embedded
Subject	Re: Serial EEPROM or Serial Flash?
Date	2018-06-15 16:21 +0200
Organization	A noiseless patient Spider
Message-ID	<pg0i0n$nlr$1@dont-email.me> (permalink)
References	<pftfgp$l3j$1@dont-email.me> <pfth8a$vpp$1@dont-email.me> <pfvqei$9t7$1@dont-email.me> <pg00mh$cmj$1@dont-email.me> <pg0ds2$sal$1@dont-email.me>

Show all headers | View raw

On 15/06/18 15:10, pozz wrote:
> Il 15/06/2018 11:25, David Brown ha scritto:
>> On 15/06/18 09:38, pozz wrote:
>>> Il 14/06/2018 12:49, David Brown ha scritto:
>>>> On 14/06/18 12:20, pozz wrote:
>>>>> I need to save some data on a non-volatile memory. They are some
>>>>> parameters that the user could change infrequently, for example 10
>>>>> times
>>>>> per day at a maximum.  In the range 100-2kB.
>>>>>
<snip>
>>>>
>>>> Multiple copies (at least 2) are key, along with timestamps or counters
>>>
>>> Two should be sufficient to prevent corruption caused by interruption
>>> during the writing of a block of data.
>>> Maybe three (or more) are needed to face memory *physical* corruption
>>> (maybe for many writings).
>>> I think I can ignore this event in my actual project.
>>>
>>
>> You use more for wear levelling.  Often your flash chip is /way/ bigger
>> than you need - perhaps by a factor of 1000 simply because that's what
>> you have on stock, or that's the cheapest device in the package you
>> want.  Spread your writes over 1000 blocks instead of 2, and you have
>> 500 times the endurance.  Use a good checksum and accept that sometimes
>> a block will be worn out and move onto the next one, and you have
>> perhaps a million times the endurance (because most blocks last a lot
>> longer than the guaranteed minimum).
> 
> So the writing process should be:
> 
> 1. write the data at block i+1 (where i is the block of the current
>   data in RAM)
> 2. read back the block i+1 and check if checksum is ok
> 3. if ok, writing process is finished
> 4. if not, go to block i+2 and start again from 1.
> 

Yes.

Then comes step 5 (for flash with separate erasing) :

5. If you have written block x, check if block x+1 (modulo the size of
the device) is erased.  If not, then erase it ready for the next write.

Note that it does not matter if the erase block size is bigger than the
program block size - if it is, then your "erase block x+1" command will
cover the next few program blocks.

>>>> and checksums.
>>>
>>> This is interesting.  Why do I need a checksum?  My approach is to use
>>> only a magic number plus a counter... and two memory areas.
>>> At first startup magic number isn't found on any areas, so the device
>>> starts with default and write data on Area1 (magic/0, where 0 is the
>>> counter).
>>
>> You use checksums to ensure that you haven't had a power-out or reset in
>> the middle of writing, 
> 
> Only for this thing, you can write the counter as the last byte. If the
> writing is interrupted in the middle, counter hasn't written yet, so the
> block is not valid (because considered too old or empty).

Nope.  You can't rely on that, unless you are absolutely sure that you
have a simple, old-fashioned E²PROM device.  Many devices are /not/ like
that, even if they provide an interface that matches it logically.

A common structure for a modern device is to have 32-byte pages as a
compromise between density, cost, and flexibility.  (Bigger pages are
more efficient in device area and cost.)  When you send a command to
write a byte, the device reads the old 32-byte page into ram, erases the
old page, updates the ram with the new data, the writes the whole 32
byte page back in.

The write process is done by a loop that writes all the data, reads it
back at a low voltage to see if it has stuck, and writes again as needed
until the data looks good.  Then it writes again a few times for safety
- either a fixed number, or a percentage of the number of writes taken.

So it is /entirely/ possible for an interrupted write to give you a
valid counter, but invalid data.  It is also entirely possible to get
some bits of the counter as valid while others are still erased (giving
ones on most devices).

And that is just for simple devices that don't do any fancy wear
levelling, packing, garbage collection, etc.


> 
>> and that the flash has not worn out.
>>
>>>
>>> When the configuration is changed, Area2 is written with magic/1, being
>>> careful to save magic/1 only at the end of area writing.
>>>
>>> At startup magics and counters from both area are loaded, and one area
>>> is chosen (magic should be valid and counter should be the maximum).
>>>
>>> I think this approach works, even when the area writing is interrupted
>>> at the middle.
>>>
>>> Why do I need checksum?  The only thing that comes in mind is to prevent
>>> writing errors: for example, I want to write 0x00 but the value really
>>> written is 0x01, maybe for a noise on the serial bus.
>>>
>>> To solve this situation, I need checksum... but also I need to re-read
>>> and re-calculate the checksum at *every* area writing... and start a new
>>> writing if something was wrong.
>>>
>>> Do you have a better strategy?
>>
>> You calculate the checksum for a block before writing it, and you check
>> it when reading it.  Simple.
> 
> Do you calc the checksum of all the data block in RAM, including padding
> bytes?

Yes, of course.  The trick is not to have unknown padding bytes.  I make
a point of /never/ having compiler-generated padding in my structs.

So you have something like this:

#define sizeOfRawBlock 32
#define noOfRawBlocks 4
#define magicNumber 0x9185be91
#define dataStructVersionExpected 0x0001

typedef union {
	uint8_t raw8[sizeOfRawBlock * noOfRawBlocks];
	uint16_t raw16[sizeOfRawBlock * noOfRawBlocks / 2];
	struct {
		uint32_t magic;
		uint16_t dataStructVersion;
		uint16_t crc;
		uint32_t count;

		// real data
	}
} nvmData_t;

static_assert(sizeof(nvmData_t) == (sizeofRawBlock * noOfRawBlocks),
	 "Check size of nvnData!");


When you want to store the data, run your crc calculation over the
raw16[] from index 4 upwards, and set it in the crc field.  Set the
magic field with a different fixed value per program.  Use the
dataStructVersion to track versions of the structure of the data - so
that updated firmware can recognise old data and quietly update it.  Set
the count to a new number each time.

Leave plenty of room to add new fields to your nvm structure.


> 
> 
>>>>> This means I need to copy&paste an entire block everytime, even for a
>>>>> single byte change.  And this is similar to Flash approach, where I
>>>>> *need* a sector erase before changing a single byte.
>>>>
>>>> Yes.
>>>>
>>>> Of course, it is possible to have multiple smaller blocks rather than
>>>> just one big one.
>>>
>>> This is good to reduce writing time of a block of data.
>>> In my case I have 2kB of data, around 62 32-bytes EEPROM pages, that is
>>> 5*62=300ms.  I don't like to block a task (in a cooperative OS) for so
>>> long time.
>>
>> Your writing should be done in the background.  The background writer
>> task is blocked - the system is not.
> 
> Are you thinking of a preemption OS?  I'm using a cooperative OS.
> 

It doesn't matter.  In a pre-emptive OS, your background writer is:

static enum { idle, writing, checking, erasing } writerState;

void doWriter(void) {
	switch (writerState) {
		case idle :
			if (writeTriggered) {
				writeTriggered = false;
				startWriting();
				writerState = writing;
			}
			break;
		case writing :
			bool stillWorking = pollNVMdevice();
			if (stillWorking) return;
			startChecking();
			writerState = checking;
			break;
		case checking :
			bool stillWorking = pollNVMdevice();
			if (stillWorking) return;
			startErasingNextBlock();
			writerState = erasing;
			break;
		case erasing :
			bool stillWorking = pollNVMdevice();
			if (stillWorking) return;
			writerState = idle;
			break;
	}
}

There - half your program is done :-)

Cooperative multitasking does not mean busy waiting for long-running
tasks.  It means using state machines, interrupt-driven SPI/I²C drivers,
polling, etc., so that you can mix fast and slow tasks.

Thread

Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-14 12:20 +0200
  Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-14 12:49 +0200
    Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 09:38 +0200
      Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 00:54 -0700
        Re: Serial EEPROM or Serial Flash? Reinhardt Behm <rbehm@hushmail.com> - 2018-06-15 16:26 +0800
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 01:31 -0700
          Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 10:50 +0200
            Re: Serial EEPROM or Serial Flash? Reinhardt Behm <rbehm@hushmail.com> - 2018-06-15 17:03 +0800
              Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 14:59 +0200
                Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 15:51 +0200
        Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 10:47 +0200
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 02:20 -0700
            Re: Serial EEPROM or Serial Flash? Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> - 2018-06-15 13:41 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 15:01 +0200
      Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 11:25 +0200
        Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 15:10 +0200
          Re: Serial EEPROM or Serial Flash? Jack <jack4747@gmail.com> - 2018-06-15 06:51 -0700
          Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-15 16:21 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 17:20 +0200
              Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-15 13:11 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-15 22:54 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-15 18:25 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 08:26 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-16 09:20 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 19:38 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-16 16:41 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 23:05 +0200
              Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-17 18:41 +0200
            Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-16 20:12 +0200
              Re: Serial EEPROM or Serial Flash? David Brown <david.brown@hesbynett.no> - 2018-06-17 19:27 +0200
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-19 17:40 +0200
              Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-17 17:05 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-19 17:44 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-19 23:14 -0400
                Re: Serial EEPROM or Serial Flash? pozz <pozzugno@gmail.com> - 2018-06-20 11:03 +0200
                Re: Serial EEPROM or Serial Flash? Richard Damon <Richard@Damon-Family.org> - 2018-06-20 08:03 -0400
  Re: Serial EEPROM or Serial Flash? Michael Karas <mkaras@carousel-design.com> - 2018-06-23 09:43 -0700

csiph-web