Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1637041 > unrolled thread

Re: Race to power off harming SATA SSDs

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Race to power off harming SATA SSDs Pavel Machek <pavel@ucw.cz> - 2017-05-08 00:00 +0200
    Re: Race to power off harming SATA SSDs David Woodhouse <dwmw2@infradead.org> - 2017-05-08 09:30 +0200
      Re: Race to power off harming SATA SSDs Ricard Wanderlof <ricard.wanderlof@axis.com> - 2017-05-08 09:40 +0200
        Re: Race to power off harming SATA SSDs David Woodhouse <dwmw2@infradead.org> - 2017-05-08 10:20 +0200
          Re: Race to power off harming SATA SSDs Ricard Wanderlof <ricard.wanderlof@axis.com> - 2017-05-08 10:40 +0200
            Re: Race to power off harming SATA SSDs David Woodhouse <dwmw2@infradead.org> - 2017-05-08 11:00 +0200
              Re: Race to power off harming SATA SSDs Ricard Wanderlof <ricard.wanderlof@axis.com> - 2017-05-08 11:10 +0200
                Re: Race to power off harming SATA SSDs Hans de Goede <hdegoede@redhat.com> - 2017-05-08 11:10 +0200
                  Re: Race to power off harming SATA SSDs David Woodhouse <dwmw2@infradead.org> - 2017-05-08 12:20 +0200
                    Re: Race to power off harming SATA SSDs Boris Brezillon <boris.brezillon@free-electrons.com> - 2017-05-08 14:00 +0200
                      Re: Race to power off harming SATA SSDs David Woodhouse <dwmw2@infradead.org> - 2017-05-08 17:50 +0200
                        Re: Race to power off harming SATA SSDs Pavel Machek <pavel@ucw.cz> - 2017-05-08 23:40 +0200
                      Re: Race to power off harming SATA SSDs Pavel Machek <pavel@ucw.cz> - 2017-05-08 18:50 +0200
                        Re: Race to power off harming SATA SSDs Tejun Heo <tj@kernel.org> - 2017-05-08 19:50 +0200
                          Re: Race to power off harming SATA SSDs Pavel Machek <pavel@ucw.cz> - 2017-05-08 21:00 +0200
                            Re: Race to power off harming SATA SSDs Tejun Heo <tj@kernel.org> - 2017-05-08 21:10 +0200
                        RE: Race to power off harming SATA SSDs Atlant Schmidt <aschmidt@dekaresearch.com> - 2017-05-08 20:50 +0200
                Re: Race to power off harming SATA SSDs David Woodhouse <dwmw2@infradead.org> - 2017-05-08 12:20 +0200
      Re: Race to power off harming SATA SSDs Pavel Machek <pavel@ucw.cz> - 2017-05-08 11:30 +0200
        Re: Race to power off harming SATA SSDs David Woodhouse <dwmw2@infradead.org> - 2017-05-08 11:40 +0200
          Re: Race to power off harming SATA SSDs Pavel Machek <pavel@ucw.cz> - 2017-05-08 13:00 +0200
            Re: Race to power off harming SATA SSDs Richard Weinberger <richard.weinberger@gmail.com> - 2017-05-08 13:10 +0200
              Re: Race to power off harming SATA SSDs Boris Brezillon <boris.brezillon@free-electrons.com> - 2017-05-08 14:00 +0200
              Re: Race to power off harming SATA SSDs Boris Brezillon <boris.brezillon@free-electrons.com> - 2017-05-08 14:00 +0200
                Re: Race to power off harming SATA SSDs Richard Weinberger <richard@nod.at> - 2017-05-08 14:20 +0200
            Re: Race to power off harming SATA SSDs David Woodhouse <dwmw2@infradead.org> - 2017-05-08 13:20 +0200
              Re: Race to power off harming SATA SSDs Pavel Machek <pavel@ucw.cz> - 2017-05-08 14:40 +0200
        Re: Race to power off harming SATA SSDs Richard Weinberger <richard.weinberger@gmail.com> - 2017-05-08 12:00 +0200

Page 1 of 2  [1] 2  Next page →

#1637041 — Re: Race to power off harming SATA SSDs

Hi!

> > However, *IN PRACTICE*, SATA STANDBY IMMEDIATE command completion
> > [often?] only indicates that the device is now switching to the target
> > power management state, not that it has reached the target state.  Any
> > further device status inquires would return that it is in STANDBY mode,
> > even if it is still entering that state.
> > 
> > The kernel then continues the shutdown path while the SSD is still
> > preparing itself to be powered off, and it becomes a race.  When the
> > kernel + firmware wins, platform power is cut before the SSD has
> > finished (i.e. the SSD is subject to an unclean power-off).
> 
> At that point, the device is fully flushed and in terms of data
> integrity should be fine with losing power at any point anyway.

Actually, no, that is not how it works.

"Fully flushed" is one thing, surviving power loss is
different. Explanation below.

> > NOTE: unclean SSD power-offs are dangerous and may brick the device in
> > the worst case, or otherwise harm it (reduce longevity, damage flash
> > blocks).  It is also not impossible to get data corruption.
> 
> I get that the incrementing counters might not be pretty but I'm a bit
> skeptical about this being an actual issue.  Because if that were
> true, the device would be bricking itself from any sort of power
> losses be that an actual power loss, battery rundown or hard power off
> after crash.

And that's exactly what users see. If you do enough power fails on a
SSD, you usually brick it, some die sooner than others. There was some
test results published, some are here
http://lkcl.net/reports/ssd_analysis.html, I believe I seen some
others too.

It is very hard for a NAND to work reliably in face of power
failures. In fact, not even Linux MTD + UBIFS works well in that
regards. See
http://www.linux-mtd.infradead.org/faq/ubi.html. (Unfortunately, its
down now?!). If we can't get it right, do you believe SSD manufactures
do?

[Issue is, if you powerdown during erase, you get "weakly erased"
page, which will contain expected 0xff's, but you'll get bitflips
there quickly. Similar issue exists for writes. It is solveable in
software, just hard and slow... and we don't do it.]
									
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [next] | [standalone]

#1637265

[Multipart message — attachments visible in raw view] — view raw

On Sun, 2017-05-07 at 22:40 +0200, Pavel Machek wrote:
> > > NOTE: unclean SSD power-offs are dangerous and may brick the device in
> > > the worst case, or otherwise harm it (reduce longevity, damage flash
> > > blocks).  It is also not impossible to get data corruption.
>
> > I get that the incrementing counters might not be pretty but I'm a bit
> > skeptical about this being an actual issue.  Because if that were
> > true, the device would be bricking itself from any sort of power
> > losses be that an actual power loss, battery rundown or hard power off
> > after crash.
>
> And that's exactly what users see. If you do enough power fails on a
> SSD, you usually brick it, some die sooner than others. There was some
> test results published, some are here
> http://lkcl.net/reports/ssd_analysis.html, I believe I seen some
> others too.
> 
> It is very hard for a NAND to work reliably in face of power
> failures. In fact, not even Linux MTD + UBIFS works well in that
> regards. See
> http://www.linux-mtd.infradead.org/faq/ubi.html. (Unfortunately, its
> down now?!). If we can't get it right, do you believe SSD manufactures
> do?
> 
> [Issue is, if you powerdown during erase, you get "weakly erased"
> page, which will contain expected 0xff's, but you'll get bitflips
> there quickly. Similar issue exists for writes. It is solveable in
> software, just hard and slow... and we don't do it.]

It's not that hard. We certainly do it in JFFS2. I was fairly sure that
it was also part of the design considerations for UBI — it really ought
to be right there too. I'm less sure about UBIFS but I would have
expected it to be OK.

SSDs however are often crap; power fail those at your peril. And of
course there's nothing you can do when they do fail, whereas we accept
patches for things which are implemented in Linux.

[toc] | [prev] | [next] | [standalone]

#1637275

On Mon, 8 May 2017, David Woodhouse wrote:

> > [Issue is, if you powerdown during erase, you get "weakly erased"
> > page, which will contain expected 0xff's, but you'll get bitflips
> > there quickly. Similar issue exists for writes. It is solveable in
> > software, just hard and slow... and we don't do it.]
> 
> It's not that hard. We certainly do it in JFFS2. I was fairly sure that
> it was also part of the design considerations for UBI ? it really ought
> to be right there too. I'm less sure about UBIFS but I would have
> expected it to be OK.

I've got a problem with the underlying mechanism. How long does it take to 
erase a NAND block? A couple of milliseconds. That means that for an erase 
to be "weak" du to a power fail, the host CPU must issue an erase command, 
and then the power to the NAND must drop within those milliseconds. 
However, in most systems there will be a power monitor which will 
essentially reset the CPU as soon as the power starts dropping. So in 
practice, by the time the voltage is too low to successfully supply the 
NAND chip, the CPU has already been reset, hence, no reset command will 
have been given by the time NAND runs out of steam.

Sure, with switchmode power supplies, we don't have those large capacitors 
in the power supply which can keep the power going for a second or more, 
but still, I would think that the power wouldn't die fast enough for this 
to be an issue.

But I could very well be wrong and I haven't had experience with that many 
NAND flash systems. But then please tell me where the above reasoning is 
flawed.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

[toc] | [prev] | [next] | [standalone]

#1637301

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2017-05-08 at 09:38 +0200, Ricard Wanderlof wrote:
> On Mon, 8 May 2017, David Woodhouse wrote:
> 
> > 
> > > 
> > > [Issue is, if you powerdown during erase, you get "weakly erased"
> > > page, which will contain expected 0xff's, but you'll get bitflips
> > > there quickly. Similar issue exists for writes. It is solveable in
> > > software, just hard and slow... and we don't do it.]
> > It's not that hard. We certainly do it in JFFS2. I was fairly sure that
> > it was also part of the design considerations for UBI ? it really ought
> > to be right there too. I'm less sure about UBIFS but I would have
> > expected it to be OK.
> I've got a problem with the underlying mechanism. How long does it take to 
> erase a NAND block? A couple of milliseconds. That means that for an erase 
> to be "weak" du to a power fail, the host CPU must issue an erase command, 
> and then the power to the NAND must drop within those milliseconds. 
> However, in most systems there will be a power monitor which will 
> essentially reset the CPU as soon as the power starts dropping. So in 
> practice, by the time the voltage is too low to successfully supply the 
> NAND chip, the CPU has already been reset, hence, no reset command will 
> have been given by the time NAND runs out of steam.
> 
> Sure, with switchmode power supplies, we don't have those large capacitors 
> in the power supply which can keep the power going for a second or more, 
> but still, I would think that the power wouldn't die fast enough for this 
> to be an issue.
> 
> But I could very well be wrong and I haven't had experience with that many 
> NAND flash systems. But then please tell me where the above reasoning is 
> flawed.

Our empirical testing trumps your "can never happen" theory :)

[toc] | [prev] | [next] | [standalone]

#1637307

On Mon, 8 May 2017, David Woodhouse wrote:

> > I've got a problem with the underlying mechanism. How long does it take to 
> > erase a NAND block? A couple of milliseconds. That means that for an erase 
> > to be "weak" du to a power fail, the host CPU must issue an erase command, 
> > and then the power to the NAND must drop within those milliseconds. 
> > However, in most systems there will be a power monitor which will 
> > essentially reset the CPU as soon as the power starts dropping. So in 
> > practice, by the time the voltage is too low to successfully supply the 
> > NAND chip, the CPU has already been reset, hence, no reset command will 
> > have been given by the time NAND runs out of steam.
> > 
> > Sure, with switchmode power supplies, we don't have those large capacitors 
> > in the power supply which can keep the power going for a second or more, 
> > but still, I would think that the power wouldn't die fast enough for this 
> > to be an issue.
> > 
> Our empirical testing trumps your "can never happen" theory :)

I'm sure it does. But what is the explanation then? Has anyone analyzed 
what is going on using an oscilloscope to verify relationship between 
erase command and supply voltage drop?

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

[toc] | [prev] | [next] | [standalone]

#1637315

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2017-05-08 at 10:36 +0200, Ricard Wanderlof wrote:
> On Mon, 8 May 2017, David Woodhouse wrote:
> > Our empirical testing trumps your "can never happen" theory :)
>
> I'm sure it does. But what is the explanation then? Has anyone analyzed 
> what is going on using an oscilloscope to verify relationship between 
> erase command and supply voltage drop?

Not that I'm aware of. Once we have reached the "it does happen and we
have to cope" there was not a lot of point in working out *why* it
happened.

In fact, the only examples I *personally* remember were on NOR flash,
which takes longer to erase. So it's vaguely possible that it doesn't
happen on NAND. But really, it's not something we should be depending
on and the software mechanisms have to remain in place.

[toc] | [prev] | [next] | [standalone]

#1637317

On Mon, 8 May 2017, David Woodhouse wrote:

> > On Mon, 8 May 2017, David Woodhouse wrote:
> > > Our empirical testing trumps your "can never happen" theory :)
> >
> > I'm sure it does. But what is the explanation then? Has anyone analyzed 
> > what is going on using an oscilloscope to verify relationship between 
> > erase command and supply voltage drop?
> 
> Not that I'm aware of. Once we have reached the "it does happen and we
> have to cope" there was not a lot of point in working out *why* it
> happened.
> 
> In fact, the only examples I *personally* remember were on NOR flash,
> which takes longer to erase. So it's vaguely possible that it doesn't
> happen on NAND. But really, it's not something we should be depending
> on and the software mechanisms have to remain in place.

My point is really that say that the problem is in fact not that the erase 
is cut short due to the power fail, but that the software issues a second 
command before the first erase command has completed, for instance, or 
some other situation. Then we'd have a concrete situation which we can 
resolve (i.e., fix the bug), rather than assuming that it's the hardware's 
fault and implement various software workarounds.

On the other hand, making the software resilient to erase problems 
essentially makes the system more robust in any case, so it's not a bad 
thing of course.

It's just that I've seen this "we're software guys, and it must be the 
hardware's fault" (and vice versa) enough times to cause a small warning 
bell to off here.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

[toc] | [prev] | [next] | [standalone]

#1637318

Hi,

On 08-05-17 11:06, Ricard Wanderlof wrote:
> 
> On Mon, 8 May 2017, David Woodhouse wrote:
> 
>>> On Mon, 8 May 2017, David Woodhouse wrote:
>>>> Our empirical testing trumps your "can never happen" theory :)
>>>
>>> I'm sure it does. But what is the explanation then? Has anyone analyzed
>>> what is going on using an oscilloscope to verify relationship between
>>> erase command and supply voltage drop?
>>
>> Not that I'm aware of. Once we have reached the "it does happen and we
>> have to cope" there was not a lot of point in working out *why* it
>> happened.
>>
>> In fact, the only examples I *personally* remember were on NOR flash,
>> which takes longer to erase. So it's vaguely possible that it doesn't
>> happen on NAND. But really, it's not something we should be depending
>> on and the software mechanisms have to remain in place.
> 
> My point is really that say that the problem is in fact not that the erase
> is cut short due to the power fail, but that the software issues a second
> command before the first erase command has completed, for instance, or
> some other situation. Then we'd have a concrete situation which we can
> resolve (i.e., fix the bug), rather than assuming that it's the hardware's
> fault and implement various software workarounds.

You're forgetting that the SSD itself (this thread is about SSDs) also has
a major software component which is doing housekeeping all the time, so even
if the main CPU gets reset the SSD's controller may still happily be erasing
blocks.

Regards,

Hans

[toc] | [prev] | [next] | [standalone]

#1637346

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2017-05-08 at 11:09 +0200, Hans de Goede wrote:
> You're forgetting that the SSD itself (this thread is about SSDs) also has
> a major software component which is doing housekeeping all the time, so even
> if the main CPU gets reset the SSD's controller may still happily be erasing
> blocks.

We're not really talking about SSDs at all any more; we're talking
about real flash with real maintainable software.

[toc] | [prev] | [next] | [standalone]

#1637407

On Mon, 08 May 2017 11:13:10 +0100
David Woodhouse <dwmw2@infradead.org> wrote:

> On Mon, 2017-05-08 at 11:09 +0200, Hans de Goede wrote:
> > You're forgetting that the SSD itself (this thread is about SSDs) also has
> > a major software component which is doing housekeeping all the time, so even
> > if the main CPU gets reset the SSD's controller may still happily be erasing
> > blocks.  
> 
> We're not really talking about SSDs at all any more; we're talking
> about real flash with real maintainable software.

It's probably a good sign that this new discussion should take place in
a different thread :-).

[toc] | [prev] | [next] | [standalone]

#1637530

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2017-05-08 at 13:50 +0200, Boris Brezillon wrote:
> On Mon, 08 May 2017 11:13:10 +0100
> David Woodhouse <dwmw2@infradead.org> wrote:
> 
> > 
> > On Mon, 2017-05-08 at 11:09 +0200, Hans de Goede wrote:
> > > 
> > > You're forgetting that the SSD itself (this thread is about SSDs) also has
> > > a major software component which is doing housekeeping all the time, so even
> > > if the main CPU gets reset the SSD's controller may still happily be erasing
> > > blocks.  
> > We're not really talking about SSDs at all any more; we're talking
> > about real flash with real maintainable software.
>
> It's probably a good sign that this new discussion should take place in
> a different thread :-).

Well, maybe. But it was a silly thread in the first place. SATA SSDs
aren't *expected* to be reliable.

[toc] | [prev] | [next] | [standalone]

#1637740

On Mon 2017-05-08 16:40:11, David Woodhouse wrote:
> On Mon, 2017-05-08 at 13:50 +0200, Boris Brezillon wrote:
> > On Mon, 08 May 2017 11:13:10 +0100
> > David Woodhouse <dwmw2@infradead.org> wrote:
> > 
> > > 
> > > On Mon, 2017-05-08 at 11:09 +0200, Hans de Goede wrote:
> > > > 
> > > > You're forgetting that the SSD itself (this thread is about SSDs) also has
> > > > a major software component which is doing housekeeping all the time, so even
> > > > if the main CPU gets reset the SSD's controller may still happily be erasing
> > > > blocks.  
> > > We're not really talking about SSDs at all any more; we're talking
> > > about real flash with real maintainable software.
> >
> > It's probably a good sign that this new discussion should take place in
> > a different thread :-).
> 
> Well, maybe. But it was a silly thread in the first place. SATA SSDs
> aren't *expected* to be reliable.

Citation needed?

I'm pretty sure SATA SSDs are expected to be reliable, up to maximum
amount of gigabytes written (specified by manufacturer), as long as
you don't cut power without warning.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1637578

[Multipart message — attachments visible in raw view] — view raw

On Mon 2017-05-08 13:50:05, Boris Brezillon wrote:
> On Mon, 08 May 2017 11:13:10 +0100
> David Woodhouse <dwmw2@infradead.org> wrote:
> 
> > On Mon, 2017-05-08 at 11:09 +0200, Hans de Goede wrote:
> > > You're forgetting that the SSD itself (this thread is about SSDs) also has
> > > a major software component which is doing housekeeping all the time, so even
> > > if the main CPU gets reset the SSD's controller may still happily be erasing
> > > blocks.  
> > 
> > We're not really talking about SSDs at all any more; we're talking
> > about real flash with real maintainable software.
> 
> It's probably a good sign that this new discussion should take place in
> a different thread :-).

Well, you are right.. and I'm responsible.

What I was trying to point out was that storage people try to treat
SSDs as HDDs... and SSDs are very different. Harddrives mostly survive
powerfails (with emergency parking), while it is very, very difficult
to make SSD survive random powerfail, and we have to make sure we
always powerdown SSDs "cleanly".

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1637618

Hello,

On Mon, May 08, 2017 at 06:43:22PM +0200, Pavel Machek wrote:
> What I was trying to point out was that storage people try to treat
> SSDs as HDDs... and SSDs are very different. Harddrives mostly survive
> powerfails (with emergency parking), while it is very, very difficult
> to make SSD survive random powerfail, and we have to make sure we
> always powerdown SSDs "cleanly".

We do.

The issue raised is that some SSDs still increment the unexpected
power loss count even after clean shutdown sequence and that the
kernel should wait for some secs before powering off.

We can do that for select devices but I want something more than "this
SMART counter is getting incremented" before doing that.

Thanks.

-- 
tejun

[toc] | [prev] | [next] | [standalone]

#1637643

[Multipart message — attachments visible in raw view] — view raw

On Mon 2017-05-08 13:43:03, Tejun Heo wrote:
> Hello,
> 
> On Mon, May 08, 2017 at 06:43:22PM +0200, Pavel Machek wrote:
> > What I was trying to point out was that storage people try to treat
> > SSDs as HDDs... and SSDs are very different. Harddrives mostly survive
> > powerfails (with emergency parking), while it is very, very difficult
> > to make SSD survive random powerfail, and we have to make sure we
> > always powerdown SSDs "cleanly".
> 
> We do.
> 
> The issue raised is that some SSDs still increment the unexpected
> power loss count even after clean shutdown sequence and that the
> kernel should wait for some secs before powering off.
> 
> We can do that for select devices but I want something more than "this
> SMART counter is getting incremented" before doing that.

Well... the SMART counter tells us that the device was not shut down
correctly. Do we have reason to believe that it is _not_ telling us
truth? It is more than one device.

SSDs die when you power them without warning:
http://lkcl.net/reports/ssd_analysis.html

What kind of data would you like to see? "I have been using linux and
my SSD died"? We have had such reports. "I have killed 10 SSDs in a
week then I added one second delay, and this SSD survived 6 months"?


									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1637651

Hello,

On Mon, May 08, 2017 at 08:56:15PM +0200, Pavel Machek wrote:
> Well... the SMART counter tells us that the device was not shut down
> correctly. Do we have reason to believe that it is _not_ telling us
> truth? It is more than one device.

It also finished power off command successfully.

> SSDs die when you power them without warning:
> http://lkcl.net/reports/ssd_analysis.html
> 
> What kind of data would you like to see? "I have been using linux and
> my SSD died"? We have had such reports. "I have killed 10 SSDs in a
> week then I added one second delay, and this SSD survived 6 months"?

Repeating shutdown cycles and showing that the device actually is in
trouble would be great.  It doesn't have to reach full-on device
failure.  Showing some sign of corruption would be enough - increase
in CRC failure counts, bad block counts (a lot of devices report
remaining reserve or lifetime in one way or the other) and so on.
Right now, it might as well be just the SMART counter being funky.

Thanks.

-- 
tejun

[toc] | [prev] | [next] | [standalone]

#1637637

> Well, you are right.. and I'm responsible.
>
> What I was trying to point out was that storage people try to treat SSDs as HDDs...
> and SSDs are very different. Harddrives mostly survive powerfails (with emergency
> parking), while it is very, very difficult to make SSD survive random powerfail,
> and we have to make sure we always powerdown SSDs "cleanly".

  It all depends on the class of SSD that we're discussing.
  "Enterprise class" SSDs will often use either ultracapacitors
  or batteries to allow them to successfully complete all of
  the necessary operations upon a power cut.

This e-mail and the information, including any attachments it contains, are intended to be a confidential communication only to the person or entity to whom it is addressed and may contain information that is privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender and destroy the original message. Thank you. Please consider the environment before printing this email.

[toc] | [prev] | [next] | [standalone]

#1637349

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2017-05-08 at 11:06 +0200, Ricard Wanderlof wrote:
> 
> My point is really that say that the problem is in fact not that the erase 
> is cut short due to the power fail, but that the software issues a second 
> command before the first erase command has completed, for instance, or 
> some other situation. Then we'd have a concrete situation which we can 
> resolve (i.e., fix the bug), rather than assuming that it's the hardware's 
> fault and implement various software workarounds.

On NOR flash we have *definitely* seen it during powerfail testing.

A block looks like it's all 0xFF when you read it back on mount, but if
you read it repeatedly, you may see bit flips because it wasn't
completely erased. And even if you read it ten times and 'trust' that
it's properly erased, it could start to show those bit flips when you
start to program it.

It was very repeatable, and that's when we implemented the 'clean
markers' written after a successful erase, rather than trusting a block
that "looks empty".

[toc] | [prev] | [next] | [standalone]

#1637326

On Mon 2017-05-08 08:21:34, David Woodhouse wrote:
> On Sun, 2017-05-07 at 22:40 +0200, Pavel Machek wrote:
> > > > NOTE: unclean SSD power-offs are dangerous and may brick the device in
> > > > the worst case, or otherwise harm it (reduce longevity, damage flash
> > > > blocks).  It is also not impossible to get data corruption.
> >
> > > I get that the incrementing counters might not be pretty but I'm a bit
> > > skeptical about this being an actual issue.  Because if that were
> > > true, the device would be bricking itself from any sort of power
> > > losses be that an actual power loss, battery rundown or hard power off
> > > after crash.
> >
> > And that's exactly what users see. If you do enough power fails on a
> > SSD, you usually brick it, some die sooner than others. There was some
> > test results published, some are here
> > http://lkcl.net/reports/ssd_analysis.html, I believe I seen some
> > others too.
> > 
> > It is very hard for a NAND to work reliably in face of power
> > failures. In fact, not even Linux MTD + UBIFS works well in that
> > regards. See
> > http://www.linux-mtd.infradead.org/faq/ubi.html. (Unfortunately, its
> > down now?!). If we can't get it right, do you believe SSD manufactures
> > do?
> > 
> > [Issue is, if you powerdown during erase, you get "weakly erased"
> > page, which will contain expected 0xff's, but you'll get bitflips
> > there quickly. Similar issue exists for writes. It is solveable in
> > software, just hard and slow... and we don't do it.]
> 
> It's not that hard. We certainly do it in JFFS2. I was fairly sure that
> it was also part of the design considerations for UBI — it really ought
> to be right there too. I'm less sure about UBIFS but I would have
> expected it to be OK.

Are you sure you have it right in JFFS2? Do you journal block erases?
Apparently, that was pretty much non-issue on older flashes.

https://web-beta.archive.org/web/20160923094716/http://www.linux-mtd.infradead.org:80/doc/ubifs.html#L_unstable_bits

> SSDs however are often crap; power fail those at your peril. And of
> course there's nothing you can do when they do fail, whereas we accept
> patches for things which are implemented in Linux.

Agreed. If the SSD indiciates unexpected powerdown, it is a problem
and we need to fix it.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[toc] | [prev] | [next] | [standalone]

#1637332

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2017-05-08 at 11:28 +0200, Pavel Machek wrote:
> 
> Are you sure you have it right in JFFS2? Do you journal block erases?
> Apparently, that was pretty much non-issue on older flashes.

It isn't necessary in JFFS2. It is a *purely* log-structured file
system (which is why it doesn't scale well past the 1GiB or so that we
made it handle for OLPC).

So we don't erase a block until all its contents are obsolete. And if
we fail to complete the erase... well the contents are either going to
fail a CRC check, or... still be obsoleted by later entries elsewhere.

And even if it *looks* like an erase has completed and the block is all
0xFF, we erase it again and write a 'clean marker' to it to indicate
that the erase was completed successfully. Because otherwise it can't
be trusted.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web