Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #201992 > unrolled thread

Slow writes to disk

Back to article view | Back to linux.debian.user

Contents

  Slow writes to disk Stefan Monnier <monnier@iro.umontreal.ca> - 2018-11-02 13:40 +0100
    Re: Slow writes to disk Michael Stone <mstone@debian.org> - 2018-11-02 13:50 +0100
      Re: Slow writes to disk Stefan Monnier <monnier@iro.umontreal.ca> - 2018-11-02 20:00 +0100
    Re: Slow writes to disk Martin <keine-eile@online.de> - 2018-11-02 15:20 +0100
      Re: Slow writes to disk Curt <curty@free.fr> - 2018-11-02 16:30 +0100
    Re: Slow writes to disk David Christensen <dpchrist@holgerdanske.com> - 2018-11-03 03:40 +0100
      Re: Slow writes to disk Michael Stone <mstone@debian.org> - 2018-11-03 13:00 +0100
        Re: Slow writes to disk David Christensen <dpchrist@holgerdanske.com> - 2018-11-03 20:30 +0100
          Re: Slow writes to disk Michael Stone <mstone@debian.org> - 2018-11-03 21:40 +0100
            Re: Slow writes to disk David Christensen <dpchrist@holgerdanske.com> - 2018-11-04 03:00 +0100
              Re: Slow writes to disk Michael Stone <mstone@debian.org> - 2018-11-04 13:30 +0100
                Re: Slow writes to disk David Christensen <dpchrist@holgerdanske.com> - 2018-11-04 23:10 +0100
    Re: Slow writes to disk songbird <songbird@anthive.com> - 2018-11-03 18:30 +0100

#201992 — Slow writes to disk

For a while now I noticed that aptitude is very slow on one of my
machine (Thinkpad T61) running Debian testing.  At first I thought it
was because its disk (a fairly old 120GB SSD) was suffering from some
kind of problem, so I replaced it with an almost new 240GB Samsung 840.

It seemed to bet better at first, but maybe it was just an impression.
In any case, now it's definitely very slow.  Digging more into it,
I found out that part of the problem seems to be very slow writes to
the disk.  I can reproduce tests where `dd`ing a 40MB file proceeds at
the ridiculous speed of about 500KB/s (tho sometimes the speed is much
higher).

I've been using SSD drives on most of my machines for many years now,
and it's the first time I notice such a slow performance.  `smartctl`
shows that the drive an not registered any errors at all and has a "wear
leveling" of 18.

I figured maybe the drive is having trouble for lack of free space which
leads to high load on the "GC", so I did a `fstrim` on / and /home
(both on LVM), which took a bit of time and returned without any
message, which I took to be a positive sign that it managed to do what
it's supposed to.  But it made no difference to the performance.

Any idea?


        Stefan

[toc] | [next] | [standalone]

#201993

On Fri, Nov 02, 2018 at 08:31:50AM -0400, Stefan Monnier wrote:
>It seemed to bet better at first, but maybe it was just an impression.
>In any case, now it's definitely very slow.  Digging more into it,
>I found out that part of the problem seems to be very slow writes to
>the disk.  I can reproduce tests where `dd`ing a 40MB file proceeds at
>the ridiculous speed of about 500KB/s (tho sometimes the speed is much
>higher).

Using what command line?

[toc] | [prev] | [next] | [standalone]

#202000

>>It seemed to bet better at first, but maybe it was just an impression.
>>In any case, now it's definitely very slow.  Digging more into it,
>>I found out that part of the problem seems to be very slow writes to
>>the disk.  I can reproduce tests where `dd`ing a 40MB file proceeds at
>>the ridiculous speed of about 500KB/s (tho sometimes the speed is much
>>higher).
> Using what command line?

Various, tried from/to different partitions, with different
block sizes.  First saw this same issue while `strace`ing "aptitude
update", and also reproduced it with `cp`.


        Stefan

[toc] | [prev] | [next] | [standalone]

#201998

How does your partition alignment look for that disk?
Or, in other words, what does 'fdisk -l' tell you about this one?

Am 02.11.18 um 13:31 schrieb Stefan Monnier:
> For a while now I noticed that aptitude is very slow on one of my
> machine (Thinkpad T61) running Debian testing.  At first I thought it
> was because its disk (a fairly old 120GB SSD) was suffering from some
> kind of problem, so I replaced it with an almost new 240GB Samsung 840.
> 
> It seemed to bet better at first, but maybe it was just an impression.
> In any case, now it's definitely very slow.  Digging more into it,
> I found out that part of the problem seems to be very slow writes to
> the disk.  I can reproduce tests where `dd`ing a 40MB file proceeds at
> the ridiculous speed of about 500KB/s (tho sometimes the speed is much
> higher).
> 
> I've been using SSD drives on most of my machines for many years now,
> and it's the first time I notice such a slow performance.  `smartctl`
> shows that the drive an not registered any errors at all and has a "wear
> leveling" of 18.
> 
> I figured maybe the drive is having trouble for lack of free space which
> leads to high load on the "GC", so I did a `fstrim` on / and /home
> (both on LVM), which took a bit of time and returned without any
> message, which I took to be a positive sign that it managed to do what
> it's supposed to.  But it made no difference to the performance.
> 
> Any idea?
> 
> 
>         Stefan
> 

[toc] | [prev] | [next] | [standalone]

#201999

On 2018-11-02, Martin <keine-eile@online.de> wrote:
> How does your partition alignment look for that disk?
> Or, in other words, what does 'fdisk -l' tell you about this one?
>

The enduring mystery is how he managed to squeeze so little into so
much.

-- 
When you have fever you are heavy and light, you are small and swollen, you
climb endlessly a ladder which turns like a wheel. 
--Jean Rhys, Voyage in the Dark

[toc] | [prev] | [next] | [standalone]

#202005

On 11/2/18 5:31 AM, Stefan Monnier wrote:
> For a while now I noticed that aptitude is very slow on one of my
> machine (Thinkpad T61) running Debian testing.  At first I thought it
> was because its disk (a fairly old 120GB SSD) was suffering from some
> kind of problem, so I replaced it with an almost new 240GB Samsung 840.
> 
> It seemed to bet better at first, but maybe it was just an impression.
> In any case, now it's definitely very slow.  Digging more into it,
> I found out that part of the problem seems to be very slow writes to
> the disk.  I can reproduce tests where `dd`ing a 40MB file proceeds at
> the ridiculous speed of about 500KB/s (tho sometimes the speed is much
> higher).
> 
> I've been using SSD drives on most of my machines for many years now,
> and it's the first time I notice such a slow performance.  `smartctl`
> shows that the drive an not registered any errors at all and has a "wear
> leveling" of 18.
> 
> I figured maybe the drive is having trouble for lack of free space which
> leads to high load on the "GC", so I did a `fstrim` on / and /home
> (both on LVM), which took a bit of time and returned without any
> message, which I took to be a positive sign that it managed to do what
> it's supposed to.  But it made no difference to the performance.
> 
> Any idea?

1.  Backup your data and configuration settings.

2.  Run the following commands as root and paste your console session 
(prompt, commands, output) into a reply:

	cat /etc/debian_version

	uname -a

	lsblk

	parted -l

	mount

	df	

For the /dev/sdX that corresponds to the "240 GB Samsung 840" (replace 
sdX with the correct letter):

	smartctl -i /dev/sdX

3.  Download and run the manufacturer's diagnostic utility (Windows may 
be required):

	https://www.samsung.com/us/support/owners/product/840-evo-sata-iii

If the tool reports the drive is bad, RMA or recycle it.

4.  Most likely, the manufacturer diagnostic will say the drive is good. 
  This means the the problem is software -- e.g. bugs, misconfiguration, 
incompatible packages, cruft, gremlins, etc..  Find a disk benchmarking 
tool.  I use dbench:

	https://linux.die.net/man/1/dbench

Install and run it.  Post the summary at the end.  For example:

	dbench -D /mnt/sda4 10 | tee dbench-i72600s-debian-7-amd64-btrfs.out
	...
	 Operation      Count    AvgLat    MaxLat
	 ----------------------------------------
	 NTCreateX    23414985     0.018    21.010
	 Close        17200117     0.001     2.229
	 Rename        991488     0.116    21.193
	 Unlink       4728380     0.093    21.652
	 Deltree          600     5.621    22.280
	 Mkdir            300     0.003     0.024
	 Qpathinfo    21223225     0.009     6.276
	 Qfileinfo    3719536     0.001     2.636
	 Qfsinfo      3891488     0.005     2.656
	 Sfileinfo    1907346     0.006     8.431
	 Find         8205416     0.034    12.780
	 WriteX       11675714     0.018    20.792
	 ReadX        36703998     0.004     8.348
	 LockX          76244     0.003     0.302
	 UnlockX        76244     0.002     0.276
	 Flush        1641062     2.368    51.611

	Throughput 1225.6 MB/sec  10 clients  10 procs  max_latency=51.617 ms

Look in dmesg and /var/log/* for errors, warnings, or other clues.  Post 
anything that looks useful.


David

[toc] | [prev] | [next] | [standalone]

#202020

On Fri, Nov 02, 2018 at 07:27:41PM -0700, David Christensen wrote:
>1.  Backup your data and configuration settings.

never a bad idea

>3.  Download and run the manufacturer's diagnostic utility (Windows
>may be required):

this is basically going to be the equivalent of smartctl -H, no need to
try to figure out how to run windows.

>4.  Most likely, the manufacturer diagnostic will say the drive is
>good.  This means the the problem is software -- e.g. bugs,
>misconfiguration, incompatible packages, cruft, gremlins, etc..  Find
>a disk benchmarking tool.  I use dbench:

Running a full blown disk benchmark isn't going to add anything at this 
point.

On Fri, Nov 02, 2018 at 02:56:06PM -0400, Stefan Monnier wrote:
>> Using what command line?
>
>Various, tried from/to different partitions, with different
>block sizes.

Well, it would be helpful to actually see one next time someone says 
"with what command line". :) What is the output if you run the following 
from a suitably large partition?
dd if=/dev/zero of=testfile.bin bs=64k count=300000 conv=fsync
dd if=testfile.bin of=/dev/null bs=64k 
remember to delete testfile.bin when you're done. That creates an 18GiB 
file; if you have more than 16GiB RAM you should increase the count.

[toc] | [prev] | [next] | [standalone]

#202046

On 11/3/18 4:58 AM, Michael Stone wrote:
> On Fri, Nov 02, 2018 at 07:27:41PM -0700, David Christensen wrote:
>> 3.  Download and run the manufacturer's diagnostic utility (Windows
>> may be required):
> 
> this is basically going to be the equivalent of smartctl -H, 

As the tools are proprietary, figuring out what they actually do would 
be difficult.  I must presume they are at least as good as smartctl.


If anyone encounters a drive the passes one but fails they other, please 
document and post.


> no need to try to figure out how to run windows.

Bootable CD/USB diagnostics are convenient for general maintenance.


OS-based tools can provide additional features and optimizations for 
that specific OS.


>> 4.  Most likely, the manufacturer diagnostic will say the drive is
>> good.  This means the the problem is software -- e.g. bugs,
>> misconfiguration, incompatible packages, cruft, gremlins, etc..  Find
>> a disk benchmarking tool.  I use dbench:
> 
> Running a full blown disk benchmark isn't going to add anything at this 
> point.

A good benchmark will stress the drive, possibly causing messages to 
dmesg and/or /var/log/* that would be helpful for trouble shooting the 
"slow writes to disk".


David

[toc] | [prev] | [next] | [standalone]

#202051

On Sat, Nov 03, 2018 at 12:20:34PM -0700, David Christensen wrote:
>On 11/3/18 4:58 AM, Michael Stone wrote:
>>On Fri, Nov 02, 2018 at 07:27:41PM -0700, David Christensen wrote:
>>>3.  Download and run the manufacturer's diagnostic utility (Windows
>>>may be required):
>>
>>this is basically going to be the equivalent of smartctl -H,
>
>As the tools are proprietary, figuring out what they actually do would 
>be difficult.  I must presume they are at least as good as smartctl.

They read the smart data which everyone has standardized on. The only 
real reason to use the proprietary tool is if they'll only honor the 
warranty with the proprietary tool. Since consumer drives have close to 
no warranty these days, it's a moot point. (If there is an actual 
warranty then jump through whatever hoops are needed. The smart log via 
smartctl should show you whether there's a reason to do so.)

>>no need to try to figure out how to run windows.
[...]
>OS-based tools can provide additional features and optimizations for 
>that specific OS.

Sure. But since this is a debian mailing list it seems silly to worry 
about windows. There are other lists for that.

>A good benchmark will stress the drive, possibly causing messages to 
>dmesg and/or /var/log/* that would be helpful for trouble shooting the 
>"slow writes to disk".

It seems unnecessary for someone to install and run tangentally related 
programs because "possibly" they might generate messages. The odds of 
that are low enough for it to just be busy work.

[toc] | [prev] | [next] | [standalone]

#202070

On 11/3/18 1:35 PM, Michael Stone wrote:
> On Sat, Nov 03, 2018 at 12:20:34PM -0700, David Christensen wrote:
>> On 11/3/18 4:58 AM, Michael Stone wrote:
>>> On Fri, Nov 02, 2018 at 07:27:41PM -0700, David Christensen wrote:
>>>> 3.  Download and run the manufacturer's diagnostic utility (Windows
>>>> may be required):
...> Sure. But since this is a debian mailing list it seems silly to worry
> about windows. There are other lists for that.

I was warning the OP that the tool for his drive may require Windows.


>> A good benchmark will stress the drive, possibly causing messages to 
>> dmesg and/or /var/log/* that would be helpful for trouble shooting the 
>> "slow writes to disk".
> 
> It seems unnecessary for someone to install and run tangentally related 
> programs because "possibly" they might generate messages. The odds of 
> that are low enough for it to just be busy work.

When trouble shooting, it is useful to have a means to elicit the 
failure mode on demand.  dbench is especially helpful for intermittent 
failures, as it does a lot of reads and writes per second.


David

[toc] | [prev] | [next] | [standalone]

#202089

On Sat, Nov 03, 2018 at 06:49:59PM -0700, David Christensen wrote:
>When trouble shooting, it is useful to have a means to elicit the 
>failure mode on demand.

Sure. That's just not it.

[toc] | [prev] | [next] | [standalone]

#202112

On 11/4/18 4:21 AM, Michael Stone wrote:
> On Sat, Nov 03, 2018 at 06:49:59PM -0700, David Christensen wrote:
>> When trouble shooting, it is useful to have a means to elicit the 
>> failure mode on demand.
> 
> Sure. That's just not it.

You are deleting too much context in your replies.


If dbench(1) is not the tool for eliciting drive failures, then what is?


David

[toc] | [prev] | [next] | [standalone]

#202033

Stefan Monnier wrote:
> For a while now I noticed that aptitude is very slow on one of my
> machine (Thinkpad T61) running Debian testing.  At first I thought it
> was because its disk (a fairly old 120GB SSD) was suffering from some
> kind of problem, so I replaced it with an almost new 240GB Samsung 840.

  what happens if you change the devices around (switch them)?


  songbird

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web