Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #238858 > unrolled thread

nvme SSD and poor performance

Back to article view | Back to linux.debian.user

Contents

  nvme SSD and poor performance Pierre Willaime <pierre.willaime@univ-lorraine.fr> - 2021-08-17 12:10 +0200
    Re: nvme SSD and poor performance Dan Ritter <dsr@randomstring.org> - 2021-08-17 14:10 +0200
      Re: nvme SSD and poor performance Pierre Willaime <pierre.willaime@univ-lorraine.fr> - 2021-08-17 15:20 +0200
        Re: nvme SSD and poor performance Dan Ritter <dsr@randomstring.org> - 2021-08-17 16:10 +0200
        Re: nvme SSD and poor performance David Christensen <dpchrist@holgerdanske.com> - 2021-08-18 00:30 +0200
          Re: nvme SSD and poor performance Pierre Willaime <pierre.willaime@univ-lorraine.fr> - 2021-08-20 11:00 +0200
            Re: nvme SSD and poor performance Christian Britz <cbritz@t-online.de> - 2021-08-20 17:50 +0200
              Re: nvme SSD and poor performance Tixy <tixy@yxit.co.uk> - 2021-08-20 19:40 +0200
            Re: nvme SSD and poor performance Linux-Fan <Ma_Sys.ma@web.de> - 2021-08-20 19:20 +0200
            Re: nvme SSD and poor performance "Alexander V. Makartsev" <avbetev@gmail.com> - 2021-08-20 19:50 +0200
              Re: nvme SSD and poor performance Pierre Willaime <pierre.willaime@univ-lorraine.fr> - 2021-08-23 10:50 +0200
                Re: nvme SSD and poor performance "Alexander V. Makartsev" <avbetev@gmail.com> - 2021-08-23 14:10 +0200
                  Re: nvme SSD and poor performance piorunz <piorunz@gmx.com> - 2021-08-23 14:20 +0200
            Re: nvme SSD and poor performance David Christensen <dpchrist@holgerdanske.com> - 2021-08-21 04:10 +0200
    Re: nvme SSD and poor performance Linux-Fan <Ma_Sys.ma@web.de> - 2021-08-17 15:40 +0200
      Re: nvme SSD and poor performance Marco Möller <talby@debianlists.mobilxpress.net> - 2021-08-17 16:10 +0200
        Re: nvme SSD and poor performance piorunz <piorunz@gmx.com> - 2021-08-20 18:30 +0200
          Re: nvme SSD and poor performance Marco Möller <talby@debianlists.mobilxpress.net> - 2021-08-20 19:20 +0200
            Re: nvme SSD and poor performance piorunz <piorunz@gmx.com> - 2021-08-20 20:10 +0200
      Re: nvme SSD and poor performance Christian Britz <cbritz@t-online.de> - 2021-08-17 17:30 +0200
        Re: nvme SSD and poor performance Linux-Fan <Ma_Sys.ma@web.de> - 2021-08-17 17:40 +0200
    Re: nvme SSD and poor performance Stefan Monnier <monnier@iro.umontreal.ca> - 2021-08-17 17:00 +0200
    Re: nvme SSD and poor performance Jochen Spieker <ml@well-adjusted.de> - 2021-08-18 22:00 +0200
      Re: nvme SSD and poor performance Anssi Saari <as@sci.fi> - 2021-08-19 08:10 +0200
        Re: nvme SSD and poor performance Sven Hartge <sven@svenhartge.de> - 2021-08-21 15:00 +0200

Page 1 of 2  [1] 2  Next page →

#238858 — nvme SSD and poor performance

Hi,

I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable 
(bulleye now).

For a long time, I suffer poor I/O performances which slow down a lot of 
tasks (apt upgrade when unpacking for example).

I am now trying to fix this issue.

Using fstrim seems to restore speed. There are always many GiB which are 
reduced :

	#  fstrim -v /
	/ : 236,7 GiB (254122389504 octets) réduits

then, directly after :

	#  fstrim -v /
	/ : 0 B (0 octets) réduits

but few minutes later, there are already 1.2 Gib to trim again :

	#  fstrim -v /
	/ : 1,2 GiB (1235369984 octets) réduits


/Is it a good idea to trim, if yes how (and how often)?/

Some people use fstrim as a cron job, some other add "discard" option to 
the /etc/fstab / line. I do not know what is the best if any. I also 
read triming frequently could reduce the ssd life.

	

I also noticed many I/O access from jbd2 and kworker such as :

	# iotop -bktoqqq -d .5
	11:11:16     364 be/3 root        0.00 K/s    7.69 K/s  0.00 % 23.64 % 
[jbd2/nvme0n1p2-]
	11:11:16       8 be/4 root        0.00 K/s    0.00 K/s  0.00 % 25.52 % 
[kworker/u32:0-flush-259:0]

The percentage given by iotop (time the thread/process spent while 
swapping in and while waiting on I/O) is often high.

I do not know what to do for kworker and if it is a normal behavior. For 
jdb2, I have read it is filesystem (ext4 here) journal.

I added the "noatime" option to /etc/fstab / line but it does not seem 
to reduce the number of access.

Regards,
Pierre


P-S: If triming it is needed for ssd, why debian do not trim by default?

[toc] | [next] | [standalone]

#238865

Pierre Willaime wrote: 
> Hi,
> 
> I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
> (bulleye now).
> 
> For a long time, I suffer poor I/O performances which slow down a lot of
> tasks (apt upgrade when unpacking for example).
> 
> I am now trying to fix this issue.

The first question is, how slow is this storage?


Here is a good article on using fio:
https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/


> Using fstrim seems to restore speed. There are always many GiB which are
> reduced :
> 
> 	#  fstrim -v /
> 	/ : 236,7 GiB (254122389504 octets) réduits
> 
> then, directly after :
> 
> 	#  fstrim -v /
> 	/ : 0 B (0 octets) réduits

That's normal. 

> but few minutes later, there are already 1.2 Gib to trim again :
> 
> 	#  fstrim -v /
> 	/ : 1,2 GiB (1235369984 octets) réduits

That's high but not impossible.

> /Is it a good idea to trim, if yes how (and how often)?/

I trim in a cron job once per week on high activity systems and
once per month on lower activity systems.


> I also noticed many I/O access from jbd2 and kworker such as :
> 
> 	# iotop -bktoqqq -d .5
> 	11:11:16     364 be/3 root        0.00 K/s    7.69 K/s  0.00 % 23.64 %
> [jbd2/nvme0n1p2-]
> 	11:11:16       8 be/4 root        0.00 K/s    0.00 K/s  0.00 % 25.52 %
> [kworker/u32:0-flush-259:0]
> 
> The percentage given by iotop (time the thread/process spent while swapping
> in and while waiting on I/O) is often high.

These are the kernel processes responsible for writing to disk. 

> I added the "noatime" option to /etc/fstab / line but it does not seem to
> reduce the number of access.

It does, but not much more than the Debian default.

Test your drive's speed, then come back with the results.

-dsr-

[toc] | [prev] | [next] | [standalone]

#238871

Le 17/08/2021 à 14:02, Dan Ritter a écrit :
> The first question is, how slow is this storage?
> 
> 
> Here is a good article on using fio:
> https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/

Thanks for the help.

Here are the output of fio tests.


Single 4KiB random write process test:

WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), 
io=12.0GiB (12.9GB), run=62271-62271msec

https://pastebin.com/5Cyg9Xvt


16 parallel 64KiB random write processes (two different results, further 
tests are closer to the second than the first):

WRITE: bw=523MiB/s (548MB/s), 31.8MiB/s-33.0MiB/s (33.4MB/s-35.6MB/s), 
io=35.5GiB (38.1GB), run=63568-69533msec

WRITE: bw=201MiB/s (211MB/s), 11.9MiB/s-14.8MiB/s (12.5MB/s-15.5MB/s), 
io=14.3GiB (15.3GB), run=60871-72618msec


https://pastebin.com/XVpPpqsC
https://pastebin.com/HEx8VvhS


Single 1MiB random write process:

   WRITE: bw=270MiB/s (283MB/s), 270MiB/s-270MiB/s (283MB/s-283MB/s), 
io=16.0GiB (17.2GB), run=60722-60722msec

https://pastebin.com/skk6mi7M

[toc] | [prev] | [next] | [standalone]

#238876

Pierre Willaime wrote: 
> Le 17/08/2021 à 14:02, Dan Ritter a écrit :
> > The first question is, how slow is this storage?
> > 
> > 
> > Here is a good article on using fio:
> > https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/
> 
> Thanks for the help.
> 
> Here are the output of fio tests.
> 
> Single 4KiB random write process test:
> 
> WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s),
> io=12.0GiB (12.9GB), run=62271-62271msec
> 
> 16 parallel 64KiB random write processes (two different results, further
> tests are closer to the second than the first):
> 
> WRITE: bw=523MiB/s (548MB/s), 31.8MiB/s-33.0MiB/s (33.4MB/s-35.6MB/s),
> io=35.5GiB (38.1GB), run=63568-69533msec
> 
> WRITE: bw=201MiB/s (211MB/s), 11.9MiB/s-14.8MiB/s (12.5MB/s-15.5MB/s),
> io=14.3GiB (15.3GB), run=60871-72618msec
> 
> Single 1MiB random write process:
> 
>   WRITE: bw=270MiB/s (283MB/s), 270MiB/s-270MiB/s (283MB/s-283MB/s),
> io=16.0GiB (17.2GB), run=60722-60722msec


These numbers look reasonable for a low-end SSD that
manufacturers throw into their laptops -- which is what your
drive is, as far as I can tell. I can't find specs, but I
suspect it is a QLC set with a small DRAM cache.

I don't think you have a significant performance problem, but
you are definitely feeling some pain -- so can you tell us more
about what feels slow? Does it happen during the ordinary course
of the day?


-dsr-

[toc] | [prev] | [next] | [standalone]

#238920

On 8/17/21 2:54 AM, Pierre Willaime wrote:
 > Hi,
 >
 > I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
 > (bulleye now).
 >
 > For a long time, I suffer poor I/O performances which slow down a lot of
 > tasks (apt upgrade when unpacking for example).
 >
 > I am now trying to fix this issue.
 >
 > Using fstrim seems to restore speed. There are always many GiB which are
 > reduced :
 >
 >      #  fstrim -v /
 >      / : 236,7 GiB (254122389504 octets) réduits
 >
 > then, directly after :
 >
 >      #  fstrim -v /
 >      / : 0 B (0 octets) réduits
 >
 > but few minutes later, there are already 1.2 Gib to trim again :
 >
 >      #  fstrim -v /
 >      / : 1,2 GiB (1235369984 octets) réduits
 >
 >
 > /Is it a good idea to trim, if yes how (and how often)?/
 >
 > Some people use fstrim as a cron job, some other add "discard" option to
 > the /etc/fstab / line. I do not know what is the best if any. I also
 > read triming frequently could reduce the ssd life.
 >
 >
 >
 > I also noticed many I/O access from jbd2 and kworker such as :
 >
 >      # iotop -bktoqqq -d .5
 >      11:11:16     364 be/3 root        0.00 K/s    7.69 K/s  0.00 %
 > 23.64 % [jbd2/nvme0n1p2-]
 >      11:11:16       8 be/4 root        0.00 K/s    0.00 K/s  0.00 %
 > 25.52 % [kworker/u32:0-flush-259:0]
 >
 > The percentage given by iotop (time the thread/process spent while
 > swapping in and while waiting on I/O) is often high.
 >
 > I do not know what to do for kworker and if it is a normal behavior. For
 > jdb2, I have read it is filesystem (ext4 here) journal.
 >
 > I added the "noatime" option to /etc/fstab / line but it does not seem
 > to reduce the number of access.
 >
 > Regards,
 > Pierre
 >
 >
 > P-S: If triming it is needed for ssd, why debian do not trim by default?



On 8/17/21 6:14 AM, Pierre Willaime wrote:
> Le 17/08/2021 à 14:02, Dan Ritter a écrit :
>> The first question is, how slow is this storage?
>>
>>
>> Here is a good article on using fio:
>> https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/ 
>>
> 
> Thanks for the help.
> 
> Here are the output of fio tests.
> 
> 
> Single 4KiB random write process test:
> 
> WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), 
> io=12.0GiB (12.9GB), run=62271-62271msec
> 
> https://pastebin.com/5Cyg9Xvt
> 
> 
> 16 parallel 64KiB random write processes (two different results, further 
> tests are closer to the second than the first):
> 
> WRITE: bw=523MiB/s (548MB/s), 31.8MiB/s-33.0MiB/s (33.4MB/s-35.6MB/s), 
> io=35.5GiB (38.1GB), run=63568-69533msec
> 
> WRITE: bw=201MiB/s (211MB/s), 11.9MiB/s-14.8MiB/s (12.5MB/s-15.5MB/s), 
> io=14.3GiB (15.3GB), run=60871-72618msec
> 
> 
> https://pastebin.com/XVpPpqsC
> https://pastebin.com/HEx8VvhS
> 
> 
> Single 1MiB random write process:
> 
>    WRITE: bw=270MiB/s (283MB/s), 270MiB/s-270MiB/s (283MB/s-283MB/s), 
> io=16.0GiB (17.2GB), run=60722-60722msec
> 
> https://pastebin.com/skk6mi7M
> 


Thank you for posting your fio(1) runs on pastebin -- it is far easier 
to comment on real data.  :-)


It would help if you told us:

1.  Make and model of computer (or motherboard and chassis, if DIY).

2.  Make and model of CPU.

3.  Quantity, make, and model of memory modules, and how your memory 
slots are populated.

4.  NVMe drive partitioning, formatting, space usage, etc..


STFW "CAZ-82512-Q11 NVMe LITEON 512GB", that looks like a decent desktop 
NVMe drive.


Looking at your fio(1) runs:

1. It appears that you held 6 parameters constant:

	--name=random-write
	--ioengine=posixaio
	--rw=randwrite
	--runtime=60
	--time_based
	--end_fsync=1

2.  It appears that you varied 4 parameters:

	run	bs	size	numjobs	iodepth
	1	4k	4g	1	1
	2	64k	256m	16	16
	3	64k	256m	16	16
	4	1m	16g	1	1

3.  Runs #2 and #3 had the same parameters, but had very different 
operation and results.  Why did run #3 layout 16 IO files, but run #2 
did not?  I suspect that experimental prodedure and/or system loading 
were not the same during the two runs (?).

4.  The best way to eliminate system loading variances is to create your 
own Debian USB live stick.  At the "Choose software to install" screen, 
choose "SSH server" and "standard system utilities".  Once the system is 
running, install the tools you want.  Use script(1) to capture console 
sessions.

5.  Pick one set of benchmark tool parameters and do several runs.  You 
want to gain enough skill running the tool so that the runs have little 
variance.

6.  In addition to running the benchmark tool the same way every time, 
you need to do all the other steps the same way every time -- rebooting, 
clearing kernel buffers, erasing files, trimming SSD's, enabling/ 
disabling processes, whatever.

7.  Document your experimental procedure such that you and others can 
duplicate your work and compare results.

8.  For each set of runs, throw out the top third of results, throw out 
the bottom third of results, and use the middle third of results to make 
decisions.

9.  Pick one parameter to sweep, keep all other parameters fixed, and do 
a set of runs for each combination.  I would do --bs=64k, --size=20%, 
--numjobs equal to the number of cores in your CPU, and sweep --iodepth 
with a binary exponential progression -- e.g. 1, 2, 4, 8, 16, etc.. 
Once I found the sweet spot for --iodepth, I would fix --iodepth at that 
value, and sweep --bs from 4k to 1m.


David


References:

[1] 
https://www.harddrivebenchmark.net/hdd.php?hdd=CX2-8B512-Q11%20NVMe%20LITEON%20512GB

[2] 
https://www.notebookcheck.net/Lite-on-CX2-8B512-Q11-256-GB-Benchmarked.458645.0.html

[3] https://linux.die.net/man/1/fio

[4] https://linux.die.net/man/8/fstrim

[5] https://linux.die.net/man/1/script

[6] https://linux.die.net/man/1/df

[toc] | [prev] | [next] | [standalone]

#239079

Thanks all.

I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).

But I do not think my issue is trim related after all. I have always a 
lot of I/O activities from jdb2 even just after booting and even when 
the computer is doing nothing for hours.

Here is an extended  log of iotop where you can see jdb2 anormal 
activities: https://pastebin.com/eyGcGdUz

I cannot (yet) find what process is generating this activities. I tried 
to kill a lot of jobs seing in atop output with no results.

> I don't think you have a significant performance problem, but
> you are definitely feeling some pain -- so can you tell us more
> about what feels slow? Does it happen during the ordinary course
> of the day?

Program are slow to start. Sometimes there is a delay when I type 
(letters are displayed few second after typing). Apt unpack take forewer 
(5 hours to unpack packages when upgrading to bulleye).

The computer is a recent Dell precision desktop with i9-9900 as CPU, an 
NVIDIA GP107GL [Quadro P400] (and the GPU integrated to the CPU). The 
nvme SSD is supposed to be a decent one. This desktop is yet a lot 
slower than my (more basic) laptop.

Complete system info: https://pastebin.com/zaGVEpae




Le 18/08/2021 à 00:24, David Christensen a écrit :
> On 8/17/21 2:54 AM, Pierre Willaime wrote:
>  > Hi,
>  >
>  > I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
>  > (bulleye now).
>  >
>  > For a long time, I suffer poor I/O performances which slow down a lot of
>  > tasks (apt upgrade when unpacking for example).
>  >
>  > I am now trying to fix this issue.
>  >
>  > Using fstrim seems to restore speed. There are always many GiB which are
>  > reduced :
>  >
>  >      #  fstrim -v /
>  >      / : 236,7 GiB (254122389504 octets) réduits
>  >
>  > then, directly after :
>  >
>  >      #  fstrim -v /
>  >      / : 0 B (0 octets) réduits
>  >
>  > but few minutes later, there are already 1.2 Gib to trim again :
>  >
>  >      #  fstrim -v /
>  >      / : 1,2 GiB (1235369984 octets) réduits
>  >
>  >
>  > /Is it a good idea to trim, if yes how (and how often)?/
>  >
>  > Some people use fstrim as a cron job, some other add "discard" option to
>  > the /etc/fstab / line. I do not know what is the best if any. I also
>  > read triming frequently could reduce the ssd life.
>  >
>  >
>  >
>  > I also noticed many I/O access from jbd2 and kworker such as :
>  >
>  >      # iotop -bktoqqq -d .5
>  >      11:11:16     364 be/3 root        0.00 K/s    7.69 K/s  0.00 %
>  > 23.64 % [jbd2/nvme0n1p2-]
>  >      11:11:16       8 be/4 root        0.00 K/s    0.00 K/s  0.00 %
>  > 25.52 % [kworker/u32:0-flush-259:0]
>  >
>  > The percentage given by iotop (time the thread/process spent while
>  > swapping in and while waiting on I/O) is often high.
>  >
>  > I do not know what to do for kworker and if it is a normal behavior. For
>  > jdb2, I have read it is filesystem (ext4 here) journal.
>  >
>  > I added the "noatime" option to /etc/fstab / line but it does not seem
>  > to reduce the number of access.
>  >
>  > Regards,
>  > Pierre
>  >
>  >
>  > P-S: If triming it is needed for ssd, why debian do not trim by default?
> 
> 
> 
> On 8/17/21 6:14 AM, Pierre Willaime wrote:
>> Le 17/08/2021 à 14:02, Dan Ritter a écrit :
>>> The first question is, how slow is this storage?
>>>
>>>
>>> Here is a good article on using fio:
>>> https://arstechnica.com/gadgets/2020/02/how-fast-are-your-disks-find-out-the-open-source-way-with-fio/ 
>>>
>>
>> Thanks for the help.
>>
>> Here are the output of fio tests.
>>
>>
>> Single 4KiB random write process test:
>>
>> WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), 
>> io=12.0GiB (12.9GB), run=62271-62271msec
>>
>> https://pastebin.com/5Cyg9Xvt
>>
>>
>> 16 parallel 64KiB random write processes (two different results, 
>> further tests are closer to the second than the first):
>>
>> WRITE: bw=523MiB/s (548MB/s), 31.8MiB/s-33.0MiB/s (33.4MB/s-35.6MB/s), 
>> io=35.5GiB (38.1GB), run=63568-69533msec
>>
>> WRITE: bw=201MiB/s (211MB/s), 11.9MiB/s-14.8MiB/s (12.5MB/s-15.5MB/s), 
>> io=14.3GiB (15.3GB), run=60871-72618msec
>>
>>
>> https://pastebin.com/XVpPpqsC
>> https://pastebin.com/HEx8VvhS
>>
>>
>> Single 1MiB random write process:
>>
>>    WRITE: bw=270MiB/s (283MB/s), 270MiB/s-270MiB/s (283MB/s-283MB/s), 
>> io=16.0GiB (17.2GB), run=60722-60722msec
>>
>> https://pastebin.com/skk6mi7M
>>
> 
> 
> Thank you for posting your fio(1) runs on pastebin -- it is far easier 
> to comment on real data.  :-)
> 
> 
> It would help if you told us:
> 
> 1.  Make and model of computer (or motherboard and chassis, if DIY).
> 
> 2.  Make and model of CPU.
> 
> 3.  Quantity, make, and model of memory modules, and how your memory 
> slots are populated.
> 
> 4.  NVMe drive partitioning, formatting, space usage, etc..
> 
> 
> STFW "CAZ-82512-Q11 NVMe LITEON 512GB", that looks like a decent desktop 
> NVMe drive.
> 
> 
> Looking at your fio(1) runs:
> 
> 1. It appears that you held 6 parameters constant:
> 
>      --name=random-write
>      --ioengine=posixaio
>      --rw=randwrite
>      --runtime=60
>      --time_based
>      --end_fsync=1
> 
> 2.  It appears that you varied 4 parameters:
> 
>      run    bs    size    numjobs    iodepth
>      1    4k    4g    1    1
>      2    64k    256m    16    16
>      3    64k    256m    16    16
>      4    1m    16g    1    1
> 
> 3.  Runs #2 and #3 had the same parameters, but had very different 
> operation and results.  Why did run #3 layout 16 IO files, but run #2 
> did not?  I suspect that experimental prodedure and/or system loading 
> were not the same during the two runs (?).
> 
> 4.  The best way to eliminate system loading variances is to create your 
> own Debian USB live stick.  At the "Choose software to install" screen, 
> choose "SSH server" and "standard system utilities".  Once the system is 
> running, install the tools you want.  Use script(1) to capture console 
> sessions.
> 
> 5.  Pick one set of benchmark tool parameters and do several runs.  You 
> want to gain enough skill running the tool so that the runs have little 
> variance.
> 
> 6.  In addition to running the benchmark tool the same way every time, 
> you need to do all the other steps the same way every time -- rebooting, 
> clearing kernel buffers, erasing files, trimming SSD's, enabling/ 
> disabling processes, whatever.
> 
> 7.  Document your experimental procedure such that you and others can 
> duplicate your work and compare results.
> 
> 8.  For each set of runs, throw out the top third of results, throw out 
> the bottom third of results, and use the middle third of results to make 
> decisions.
> 
> 9.  Pick one parameter to sweep, keep all other parameters fixed, and do 
> a set of runs for each combination.  I would do --bs=64k, --size=20%, 
> --numjobs equal to the number of cores in your CPU, and sweep --iodepth 
> with a binary exponential progression -- e.g. 1, 2, 4, 8, 16, etc.. Once 
> I found the sweet spot for --iodepth, I would fix --iodepth at that 
> value, and sweep --bs from 4k to 1m.
> 
> 
> David
> 
> 
> References:
> 
> [1] 
> https://www.harddrivebenchmark.net/hdd.php?hdd=CX2-8B512-Q11%20NVMe%20LITEON%20512GB 
> 
> 
> [2] 
> https://www.notebookcheck.net/Lite-on-CX2-8B512-Q11-256-GB-Benchmarked.458645.0.html 
> 
> 
> [3] https://linux.die.net/man/1/fio
> 
> [4] https://linux.die.net/man/8/fstrim
> 
> [5] https://linux.die.net/man/1/script
> 
> [6] https://linux.die.net/man/1/df
> 

-- 
Pierre Willaime - CNRS
Archives Henri Poincaré - Nancy

[toc] | [prev] | [next] | [standalone]

#239090

Am 20.08.21 um 10:50 schrieb Pierre Willaime:
>
> I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).
>
Interesting. I am almost 100% certain that it was enabled for me by 
default on bullseye. Maybe that behaviour changed during the release 
process.

Best Regards,
Christian

[toc] | [prev] | [next] | [standalone]

#239096

On Fri, 2021-08-20 at 17:48 +0200, Christian Britz wrote:
> 
> Am 20.08.21 um 10:50 schrieb Pierre Willaime:
> > 
> > I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).
> > 
> Interesting. I am almost 100% certain that it was enabled for me by 
> default on bullseye. Maybe that behaviour changed during the release 
> process.

It's enabled on my PC which was a fresh install in December using a
Bullseye release candidate installer.

-- 
Tixy

[toc] | [prev] | [next] | [standalone]

#239093

[Multipart message — attachments visible in raw view] — view raw

Pierre Willaime writes:

> Thanks all.
>
> I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).

You're welcome :)

> But I do not think my issue is trim related after all. I have always a lot  
> of I/O activities from jdb2 even just after booting and even when the  
> computer is doing nothing for hours.
>
> Here is an extended  log of iotop where you can see jdb2 anormal activities:  
> https://pastebin.com/eyGcGdUz

According to that, a lot of firefox-esr and dpkg and some thunderbird  
processes are active. Is there a high intensity of I/O operations when all  
Firefox, Thunderbird instances and system upgrades are closed?

When testing with iotop here, options `-d 10 -P` seemed to help getting a  
steadier and less cluttered view. Still, filtering your iotop output for  
Firefox, Thunderbird and DPKG respectively seems to be quite revealing:

| $ grep firefox-esr eyGcGdUz | grep -E '[0-9]{4,}.[0-9]{2} K/s'
| 10:38:51    3363 be/4 pierre      0.00 K/s 1811.89 K/s  0.00 % 17.64 % firefox-esr [mozStorage #3]
| 10:39:58    5117 be/4 pierre      0.00 K/s 1112.59 K/s  0.00 %  0.37 % firefox-esr [IndexedDB #14]
| 10:41:55    3363 be/4 pierre      0.00 K/s 6823.06 K/s  0.00 %  0.00 % firefox-esr [mozStorage #3]
| 10:41:55    3305 be/4 pierre   1469.88 K/s    0.00 K/s  0.00 % 60.57 % firefox-esr [QuotaManager IO]
| 10:41:55    3363 be/4 pierre   6869.74 K/s 6684.07 K/s  0.00 % 31.96 % firefox-esr [mozStorage #3]
| 10:41:56    6752 be/4 pierre   2517.19 K/s    0.00 K/s  0.00 % 99.99 % firefox-esr [Indexed~Mnt #13]
| 10:41:56    6755 be/4 pierre   31114.18 K/s    0.00 K/s  0.00 % 99.58 % firefox-esr [Indexed~Mnt #16]
| 10:41:56    3363 be/4 pierre   9153.40 K/s    0.00 K/s  0.00 % 87.06 % firefox-esr [mozStorage #3]
| 10:41:57    6755 be/4 pierre   249206.18 K/s    0.00 K/s  0.00 % 59.01 % firefox-esr [Indexed~Mnt #16]
| 10:41:57    6755 be/4 pierre   251353.11 K/s    0.00 K/s  0.00 % 66.02 % firefox-esr [Indexed~Mnt #16]
| 10:41:58    6755 be/4 pierre   273621.58 K/s    0.00 K/s  0.00 % 59.51 % firefox-esr [Indexed~Mnt #16]
| 10:41:58    6755 be/4 pierre   51639.70 K/s    0.00 K/s  0.00 % 94.90 % firefox-esr [Indexed~Mnt #16]
| 10:41:59    6755 be/4 pierre   113869.64 K/s    0.00 K/s  0.00 % 79.03 % firefox-esr [Indexed~Mnt #16]
| 10:41:59    6755 be/4 pierre   259549.09 K/s    0.00 K/s  0.00 % 56.99 % firefox-esr [Indexed~Mnt #16]
| 10:44:41    3265 be/4 pierre   1196.21 K/s    0.00 K/s  0.00 % 20.89 % firefox-esr
| 10:44:41    3289 be/4 pierre   3813.36 K/s  935.22 K/s  0.00 %  4.59 % firefox-esr [Cache2 I/O]
| 10:44:53    3363 be/4 pierre      0.00 K/s 1176.90 K/s  0.00 %  0.00 % firefox-esr [mozStorage #3]
| 10:49:28    3363 be/4 pierre      0.00 K/s 1403.16 K/s  0.00 %  0.43 % firefox-esr [mozStorage #3]

So there are incredible amounts of data being read by Firefox (Gigabytes in  
a few minutes)? Does this load reflect in atop or iotop's summarizing lines  
at the begin of the respective screens?

| $ grep thunderbird eyGcGdUz | grep -E '[0-9]{4,}.[0-9]{2} K/s'
| 10:38:43    2846 be/4 pierre      0.00 K/s 1360.19 K/s  0.00 % 15.51 % thunderbird [mozStorage #1]
| 10:39:49    2873 be/4 pierre      0.00 K/s 4753.74 K/s  0.00 %  0.00 % thunderbird [mozStorage #6]
| 10:39:49    2875 be/4 pierre      0.00 K/s 19217.56 K/s  0.00 %  0.00 % thunderbird [mozStorage #7]
| 10:39:50    2883 be/4 pierre      0.00 K/s 18014.56 K/s  0.00 % 29.39 % thunderbird [mozStorage #8]
| 10:39:50    2883 be/4 pierre      0.00 K/s 3305.94 K/s  0.00 % 27.28 % thunderbird [mozStorage #8]
| 10:39:51    2883 be/4 pierre      0.00 K/s 61950.19 K/s  0.00 % 63.11 % thunderbird [mozStorage #8]
| 10:39:51    2883 be/4 pierre      0.00 K/s 41572.77 K/s  0.00 % 27.19 % thunderbird [mozStorage #8]
| 10:39:52    2883 be/4 pierre      0.00 K/s 20961.20 K/s  0.00 % 65.02 % thunderbird [mozStorage #8]
| 10:39:52    2883 be/4 pierre      0.00 K/s 43345.16 K/s  0.00 %  0.19 % thunderbird [mozStorage #8]
| 10:42:27    2846 be/4 pierre      0.00 K/s 1189.63 K/s  0.00 %  0.45 % thunderbird [mozStorage #1]
| 10:42:33    2846 be/4 pierre      0.00 K/s 1058.52 K/s  0.00 %  0.31 % thunderbird [mozStorage #1]
| 10:47:27    2846 be/4 pierre      0.00 K/s 2113.53 K/s  0.00 %  0.66 % thunderbird [mozStorage #1]

Thunderbird seems to write a lot here. This would average at ~18 MiB/s of writing  
and hence explain why the SSD is loaded continuously. Again: Does it match the  
data reported by atop? [I am not experienced in reading iotop output, hence  
I might interpret the data wrongly].

By comparison, dpkg looks rather harmless:

| $ grep dpkg eyGcGdUz | grep -E '[0-9]{4,}.[0-9]{2} K/s'
| 10:38:25    4506 be/4 root        0.00 K/s 4553.67 K/s  0.00 %  0.26 % dpkg --status-fd 23 --no-triggers --unpack --auto-deconfigure --force-remove-protected --recursive /tmp/apt-dpkg-install-E69bfZ
| 10:38:33    4506 be/4 root        7.73 K/s 4173.77 K/s  0.00 %  1.52 % dpkg --status-fd 23 --no-triggers --unpack --auto-deconfigure --force-remove-protected --recursive /tmp/apt-dpkg-install-E69bfZ
| 10:38:49    4506 be/4 root        7.74 K/s 3770.01 K/s  0.00 %  1.41 % dpkg --status-fd 23 --no-triggers --unpack --auto-deconfigure --force-remove-protected --recursive /tmp/apt-dpkg-install-E69bfZ
| 10:39:01    4506 be/4 root        0.00 K/s 5529.57 K/s  0.00 %  5.85 % [dpkg]
| 10:39:29    4664 be/4 root        0.00 K/s 5480.00 K/s  0.00 %  3.43 % dpkg --status-fd 23 --configure --pending

This would explain that you are seeing slow progress...

[...]

HTH
Linux-Fan

öö

[toc] | [prev] | [next] | [standalone]

#239097

[Multipart message — attachments visible in raw view] — view raw

On 20.08.2021 13:50, Pierre Willaime wrote:
>
> Program are slow to start. Sometimes there is a delay when I type 
> (letters are displayed few second after typing). Apt unpack take 
> forewer (5 hours to unpack packages when upgrading to bulleye).
>
That looks abnormal to me. Have you tried to update SSD firmware¹ and 
BIOS for your Dell PC?
LiteOn Storage branch was bought by KIOXIA (Toshiba), so I can't find 
any official information about technical specifications of SSD you have, 
but just by looking at its photos, I assume it is an average one, with 
external cache and TLC NAND chips.
The performance of a NVMe looks like it working at slower x2 speed, 
instead of utilizing full x4 lanes. But it could be also its normal 
working state and delays\slowdowns could come from parts unrelated to 
NVMe, like SATA HDDs.
A "smartctl" output for all storage devices could be useful.

> The computer is a recent Dell precision desktop with i9-9900 as CPU, 
> an NVIDIA GP107GL [Quadro P400] (and the GPU integrated to the CPU). 
> The nvme SSD is supposed to be a decent one. This desktop is yet a lot 
> slower than my (more basic) laptop.
>
> Complete system info: https://pastebin.com/zaGVEpae
>


[1] 
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=0n4c4

-- 
With kindest regards, Alexander.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀

[toc] | [prev] | [next] | [standalone]

#239239

Le 20/08/2021 à 19:45, Alexander V. Makartsev a écrit :

> That looks abnormal to me. Have you tried to update SSD firmware¹ and 
> BIOS for your Dell PC?
> 
> [1] https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=0n4c4
Thanks all again for your precious answers.

I will try first to update Bios and SSD firmwares.

For the ssd, instructions (see link below) are for windows only.

Is there a way to update dell ssd firmware without taking the ssd out 
and plug it in into a windows machine?

[toc] | [prev] | [next] | [standalone]

#239244

[Multipart message — attachments visible in raw view] — view raw

On 23.08.2021 13:46, Pierre Willaime wrote:
> Le 20/08/2021 à 19:45, Alexander V. Makartsev a écrit :
>
>> That looks abnormal to me. Have you tried to update SSD firmware¹ and 
>> BIOS for your Dell PC?
>>
>> [1] 
>> https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=0n4c4
> Thanks all again for your precious answers.
>
> I will try first to update Bios and SSD firmwares.
>
> For the ssd, instructions (see link below) are for windows only.
>
> Is there a way to update dell ssd firmware without taking the ssd out 
> and plug it in into a windows machine?
>
In this particular case, I don't thinks so, because it looks like SSD 
firmware and flasher are fused together into a single PE file and I 
don't think WINE will be able to handle it properly.
There is a possibility that extracted "23201112_WIN.EXE" file will work 
under WinPE / WinRE environments booted from a USB flash drive, but I 
wouldn't count on it.

If you don't have a NVMe-capable Windows PC at hand, you can download an 
evaluation version of Windows 10 from Microsoft¹ and install it on 
separate storage device, just to update SSD firmware.
Fill the form with gibberish. Microsoft Product Key or Activation are 
not required. Evaluation copy will work for 90 days.


[1] 
https://www.microsoft.com/en-us/evalcenter/evaluate-windows-10-enterprise

-- 
With kindest regards, Alexander.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀

[toc] | [prev] | [next] | [standalone]

#239245

On 23/08/2021 13:09, Alexander V. Makartsev wrote:
> If you don't have a NVMe-capable Windows PC at hand, you can download an
> evaluation version of Windows 10 from Microsoft¹ and install it on
> separate storage device, just to update SSD firmware.
> Fill the form with gibberish. Microsoft Product Key or Activation are
> not required. Evaluation copy will work for 90 days.

It can be even easier:

Download Windows 10 ISO from here
https://www.microsoft.com/en-gb/software-download/windows10iso

put on USB stick, do 10 minutes install, do you thing and nuke it. No
need to register anything. Do it while internet is disconnected, if will
install even faster.

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀

[toc] | [prev] | [next] | [standalone]

#239106

On 8/17/21 2:54 AM, Pierre Willaime wrote:
 > I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable
 > (bulleye now).
 >
 > For a long time, I suffer poor I/O performances which slow down a lot of
 > tasks (apt upgrade when unpacking for example).


On 8/20/21 1:50 AM, Pierre Willaime wrote:
> Thanks all.
> 
> I activated `# systemctl enable fstrim.timer` (thanks Linux-Fan).
> 
> But I do not think my issue is trim related after all. 


I agree.


> I have always a 
> lot of I/O activities from jdb2 even just after booting and even when 
> the computer is doing nothing for hours.
> 
> Here is an extended  log of iotop where you can see jdb2 anormal 
> activities: https://pastebin.com/eyGcGdUz


Was your command something like this?

# iotop -o -k -t -q -q -q > iotop.out


> I cannot (yet) find what process is generating this activities. I tried 
> to kill a lot of jobs seing in atop output with no results.


Analyzing the first ten minutes worth of data with an improvised Perl 
script:

index of field 'read_bw'
     1256840.19 firefox-esr
       77926.07 apt
         316.08 perl
          22.74 dpkg
          15.27 [kworker/6:0-events]
index of field 'write_bw'
      220512.79 thunderbird
       29613.87 firefox-esr
       23221.20 dpkg
       15211.66 [jbd2/nvme0n1p2-]
        5529.57 [dpkg]
        4148.09 systemd-journald
        1699.13 perl
         533.28 mandb
         507.15 apt
         145.61 rsyslogd
         131.59 atop
         115.77 syncthing
          46.35 xfce4-terminal
          38.60 smartd
          15.48 Xorg
          15.44 NetworkManager
           7.64 bash
index of field 'swap_percent'
index of field 'io_wait_percent'
       12427.58 [jbd2/nvme0n1p2-]
        1334.15 firefox-esr
         568.91 dpkg
         385.31 thunderbird
         293.57 mandb
          99.99 syncthing
          64.82 [kworker/13:3-events_freezable_power_]
          63.86 smartd
          55.12 [kworker/u32:3+flush-259:0]
          38.64 [kworker/u32:2-flush-259:0]
          25.27 [kworker/u32:3-events_unbound]
          23.13 [kworker/4:0-events_freezable_power_]
          22.68 [kworker/u32:2-events_unbound]
          21.55 apt
          12.51 [kworker/u32:1-ext4-rsv-conversion]
           9.87 [kworker/u32:2-ext4-rsv-conversion]
           9.63 [kworker/9:1-events]
           8.90 [kworker/u32:1-flush-259:0]
           8.58 perl
           8.11 [kworker/9:1-events_freezable_power_]
           5.85 [dpkg]
           4.33 [kworker/u32:3-ext4-rsv-conversion]
           4.26 NetworkManager
           3.57 [kworker/13:3-mm_percpu_wq]
           2.85 [kworker/9:1-mm_percpu_wq]
           2.71 [kworker/4:0-mm_percpu_wq]
           0.40 [kworker/13:3-events] [kworker/4:0-events]
           0.38 [kworker/6:1-events]
           0.36 [kworker/9:1-rcu_gp]
           0.30 [kworker/u32:3-flush-259:0]
           0.26 [kworker/6:0-events_freezable_power_]
           0.16 systemd-journald
           0.03 [kworker/6:0-events]


I appears:

- firefox-esr used the most read bandwidth -- 1256840.19 K/s total

- thunderbird used the most write bandwdith -- 220512.79 K/s total

- No processes were swapping.

- jbd2/nvme0n1p2- waited the longest for I/O -- 12427.58 % total


Both apt(8) and dpkg(1) were also running and using a small amount of 
I/O.  While I may leave Firefox and Thunderbird running when installing 
a package or two, I shut them down for updates and upgrades.  Was the 
iotop data collected during a long-running upgrade?


AIUI the jbd2/nvme0n1p2 I/O corresponds to the bottom half of the kernel 
(e.g. device driver interface, DDI) in response to I/O via the top half 
of the kernel (e.g. application programming interface, API).  The way to 
reduce jdb2 I/O is to reduce application I/O.


 > On 8/17/21 7:07 AM, Dan Ritter wrote:
 >> I don't think you have a significant performance problem, but
 >> you are definitely feeling some pain -- so can you tell us more
 >> about what feels slow? Does it happen during the ordinary course
 >> of the day?

> Program are slow to start. Sometimes there is a delay when I type 
> (letters are displayed few second after typing). Apt unpack take forewer 
> (5 hours to unpack packages when upgrading to bulleye).
> 
> The computer is a recent Dell precision desktop with i9-9900 as CPU, an 
> NVIDIA GP107GL [Quadro P400] (and the GPU integrated to the CPU). The 
> nvme SSD is supposed to be a decent one. This desktop is yet a lot 
> slower than my (more basic) laptop.
> 
> Complete system info: https://pastebin.com/zaGVEpae


That's a good workstation.  :-)


Firefox and Thunderbird are habitual trouble-makers on my Debian 
desktops.  I run Xfce with the CPU Graph panel applet.  If I leave 
Firefox or Thunderbird open long enough, eventually I will see a core 
pegged at 100% and the CPU fan will spin up.  Both applications keep 
working in this state; but a swirling toilet bowl mouse pointer in 
Thunderbird is a danger sign -- I have lost e-mail messages when moving 
a message produced that symptom.  The only cure is to close the 
offending program(s) and implement additional measures to duplicate 
incoming and outgoing e-mail.  My conclusion is that both Firefox and 
Thunderbird have multi-threaded programming bugs.  The next time either 
malfunctions, I may capture some iotop(8) data and analyze it.


I have also noticed that Linux is not very good at sharing I/O between 
processes and/or drives.  If one process is hammering I/O, any other 
processes that attempts I/O will stutter.  I have seen this both when 
two processes access the same drive and when two processes access 
different drives.


The last consideration is that Debian 11.0 has just been released.  I 
have encountered many problems with Debian *.0 releases over the years, 
and typically wait several months and a couple of minor version upgrades 
before doing a fresh install and migrating services and data.  (The 
Debian 10 nouveau driver did not like the NVIDIA Optimus graphics in my 
Dell Latitude E6520 laptop until recently -- Debian 10.9?)


I suggest that you do a fresh install of Debian 10.10 and evaluate that. 
  If you still see problems, try Debian 9.13.


David

[toc] | [prev] | [next] | [standalone]

#238875

[Multipart message — attachments visible in raw view] — view raw

Pierre Willaime writes:

> I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable  
> (bulleye now).
>
> For a long time, I suffer poor I/O performances which slow down a lot of  
> tasks (apt upgrade when unpacking for example).
>
> I am now trying to fix this issue.
>
> Using fstrim seems to restore speed. There are always many GiB which are  
> reduced :

[...]

> but few minutes later, there are already 1.2 Gib to trim again :
>
> 	#  fstrim -v /
> 	/ : 1,2 GiB (1235369984 octets) réduits
>
> Is it a good idea to trim, if yes how (and how often)?
>
> Some people use fstrim as a cron job, some other add "discard" option to the  
> /etc/fstab / line. I do not know what is the best if any. I also read  
> triming frequently could reduce the ssd life.

I do `fstrim` once per week by a minimalistic and custom script as cron job:
https://github.com/m7a/lp-ssd-optimization

There is no need for custom scripts anymore, you can nowdays enable the  
timer from `util-linux` without hassle:

	# systemctl enable fstrim.timer

This will perform the trim once per week by default.

When the use of SSDs increased people tried out the `discard` options and  
found them to have strange performance characteristics, potential negative  
effects on SSD life and in some cases even data corruption (?) Back then it  
was recommended to use the periodic `fstrim` instead. I do not know if any  
of the issues with discard are still there today.

> I also noticed many I/O access from jbd2 and kworker such as :
>
> 	# iotop -bktoqqq -d .5
> 	11:11:16     364 be/3 root        0.00 K/s    7.69 K/s  0.00 % 23.64  
> % [jbd2/nvme0n1p2-]
> 	11:11:16       8 be/4 root        0.00 K/s    0.00 K/s  0.00 % 25.52  
> % [kworker/u32:0-flush-259:0]

[...]

> I do not know what to do for kworker and if it is a normal behavior. For  
> jdb2, I have read it is filesystem (ext4 here) journal.

I highly recommend you to find out what exactly is causing the high number  
of I/O operations. Usually there is a userspace process responsible (or a  
RAID resync operation) for all the I/O which is then processed by the kernel  
threads you see in iotop.

I usually look at `atop -a 4` (package `atop`) for half a minute or so to  
find out what processes are active on the system. It is possible that  
something is amiss and causing an exceedingly high I/O load leading to the  
performance degradation you observe.

[...]

> P-S: If triming it is needed for ssd, why debian do not trim by default?

Detecting reliably if the current system has SSDs that would benefit from  
trimming AND that the user has not taken their own measures is difficult. I  
guess this might be the reason for there not being an automatism, but you  
can enable the systemd timer suggested above with a single command.

HTH
Linux-Fan

öö

[toc] | [prev] | [next] | [standalone]

#238877

On 17.08.21 15:30, Linux-Fan wrote:
> Pierre Willaime writes:
> 
>> I have a nvme SSD (CAZ-82512-Q11 NVMe LITEON 512GB) on debian stable 
>> (bulleye now).
>>
>> For a long time, I suffer poor I/O performances which slow down a lot 
>> of tasks (apt upgrade when unpacking for example).
>>
>> I am now trying to fix this issue.
>>
>> Using fstrim seems to restore speed. There are always many GiB which 
>> are reduced :
> 
> [...]
> 
>> but few minutes later, there are already 1.2 Gib to trim again :
>>
>>     #  fstrim -v /
>>     / : 1,2 GiB (1235369984 octets) réduits
>>
>> Is it a good idea to trim, if yes how (and how often)?
>>
>> Some people use fstrim as a cron job, some other add "discard" option 
>> to the /etc/fstab / line. I do not know what is the best if any. I 
>> also read triming frequently could reduce the ssd life.
> 
> I do `fstrim` once per week by a minimalistic and custom script as cron 
> job:
> https://github.com/m7a/lp-ssd-optimization
> 
> There is no need for custom scripts anymore, you can nowdays enable the 
> timer from `util-linux` without hassle:
> 
>      # systemctl enable fstrim.timer
> 
> This will perform the trim once per week by default.
> 
> When the use of SSDs increased people tried out the `discard` options 
> and found them to have strange performance characteristics, potential 
> negative effects on SSD life and in some cases even data corruption (?) 
> Back then it was recommended to use the periodic `fstrim` instead. I do 
> not know if any of the issues with discard are still there today.
> 
>> I also noticed many I/O access from jbd2 and kworker such as :
>>
>>     # iotop -bktoqqq -d .5
>>     11:11:16     364 be/3 root        0.00 K/s    7.69 K/s  0.00 % 
>> 23.64 % [jbd2/nvme0n1p2-]
>>     11:11:16       8 be/4 root        0.00 K/s    0.00 K/s  0.00 % 
>> 25.52 % [kworker/u32:0-flush-259:0]
> 
> [...]
> 
>> I do not know what to do for kworker and if it is a normal behavior. 
>> For jdb2, I have read it is filesystem (ext4 here) journal.
> 
> I highly recommend you to find out what exactly is causing the high 
> number of I/O operations. Usually there is a userspace process 
> responsible (or a RAID resync operation) for all the I/O which is then 
> processed by the kernel threads you see in iotop.
> 
> I usually look at `atop -a 4` (package `atop`) for half a minute or so 
> to find out what processes are active on the system. It is possible that 
> something is amiss and causing an exceedingly high I/O load leading to 
> the performance degradation you observe.
> 
> [...]
> 
>> P-S: If triming it is needed for ssd, why debian do not trim by default?
> 
> Detecting reliably if the current system has SSDs that would benefit 
> from trimming AND that the user has not taken their own measures is 
> difficult. I guess this might be the reason for there not being an 
> automatism, but you can enable the systemd timer suggested above with a 
> single command.
> 
> HTH
> Linux-Fan
> 
> öö


I have no experience with SSD, but running my Debian Desktop from a USB 
Memory Stick since years, please allow me to share information which 
supports the suggestion of Linux-Fan to also investigate if there is 
extraordinary I/O taking place and maybe could be avoided:
In the past I found extreme(!) I/O to be produced by Firefox, when it is 
writing its cache, and when it is writing its session restore 
information. These writes to my observation occure all the time, kind of 
nonstop! I could get it very satisfactorily reduced by applying a tool 
called "Profile Sync Daemon" (psd) from package "profile-sync-daemon". 
While it is packaged for Debian, as a starting point to study its 
documentation I suggest to better look it up in the Arch Linux Wiki.

Good Luck!
Marco

[toc] | [prev] | [next] | [standalone]

#239091

On 17/08/2021 15:03, Marco Möller wrote:
> I have no experience with SSD, but running my Debian Desktop from a USB
> Memory Stick since years, please allow me to share information which
> supports the suggestion of Linux-Fan to also investigate if there is
> extraordinary I/O taking place and maybe could be avoided:
> In the past I found extreme(!) I/O to be produced by Firefox, when it is
> writing its cache, and when it is writing its session restore
> information. These writes to my observation occure all the time, kind of
> nonstop! I could get it very satisfactorily reduced by applying a tool
> called "Profile Sync Daemon" (psd) from package "profile-sync-daemon".
> While it is packaged for Debian, as a starting point to study its
> documentation I suggest to better look it up in the Arch Linux Wiki.

Firefox setting which determines how often write session data to disk is:
browser.sessionstore.interval

Default setting is value of 15000 = 15 seconds, not that bad. But still
I changed that to 10 minutes (value of 600000). If I lose an open tab
once in a year when Firefox crashes, so be it. To be honest, I haven't
seen Fx crash in a year, or more.

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀

[toc] | [prev] | [next] | [standalone]

#239094

On 20.08.21 18:22, piorunz wrote:
> On 17/08/2021 15:03, Marco Möller wrote:
>> I have no experience with SSD, but running my Debian Desktop from a USB
>> Memory Stick since years, please allow me to share information which
>> supports the suggestion of Linux-Fan to also investigate if there is
>> extraordinary I/O taking place and maybe could be avoided:
>> In the past I found extreme(!) I/O to be produced by Firefox, when it is
>> writing its cache, and when it is writing its session restore
>> information. These writes to my observation occure all the time, kind of
>> nonstop! I could get it very satisfactorily reduced by applying a tool
>> called "Profile Sync Daemon" (psd) from package "profile-sync-daemon".
>> While it is packaged for Debian, as a starting point to study its
>> documentation I suggest to better look it up in the Arch Linux Wiki.
> 
> Firefox setting which determines how often write session data to disk is:
> browser.sessionstore.interval
> 
> Default setting is value of 15000 = 15 seconds, not that bad. But still
> I changed that to 10 minutes (value of 600000). If I lose an open tab
> once in a year when Firefox crashes, so be it. To be honest, I haven't
> seen Fx crash in a year, or more.
> 
> -- 
> 
> With kindest regards, piorunz.
> 
> ⢀⣴⠾⠻⢶⣦⠀
> ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
> ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
> ⠈⠳⣄⠀⠀⠀⠀
> 

browser.sessionstore.interval
Storing the session is not the only data written to the disk. If it 
would only be this data, then indeed setting a higher interval would be 
sufficient. But there is much more going on. Especially the cache seems 
to be a cause for the extreme high I/O of Firefox. That's why I finally 
decided to go for the psd tool, and this tool made it for me. Well, I 
assume it has good reasons that someone programmed psd, probably the 
author also noticed that with Firefox parameters alone it often is not 
enough to get the I/O to the HDD caused by Firefox significantly reduced?

I am curious to see what's producing the high jdb2 traffic on the system 
of Pierre, and if this can be stopped and if this finally can be 
attributed to have been the cause for the trouble.

Unfortunately I did not make good notes on how I searched for what was 
causing the traffic on my system, I only remember to have monitored my 
system with iotop and inotifywait.

---
Always stay in good spirits!
Marco

[toc] | [prev] | [next] | [standalone]

#239098

On 20/08/2021 18:11, Marco Möller wrote:
> browser.sessionstore.interval
> Storing the session is not the only data written to the disk. If it
> would only be this data, then indeed setting a higher interval would be
> sufficient. But there is much more going on. Especially the cache seems
> to be a cause for the extreme high I/O of Firefox. That's why I finally
> decided to go for the psd tool, and this tool made it for me.

That's why you can tune disk cache in Firefox, disable it, make is
smaller and so on. By definition disk cache for a browser is a hot cache
where everything lands and it's being taken from again instead of being
downloaded from internet. On my PC I have /home on RAID10 made of 4
spinning drives, so all programs can cache and thrash on the storage all
day. If you, or anyone else in this topic, have /home on SSD and/or
Firefox/TB data churn is a concern, you could disable Fx disk cache
altogether.
I am glad that solution you have chosen, that psd tool (I don't know it)
work for you.

--

With kindest regards, piorunz.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀⠀⠀⠀

[toc] | [prev] | [next] | [standalone]

#238885

On 17.08.21 at 15:30 Linux-Fan wrote:
> Pierre Willaime writes:
>
>> P-S: If triming it is needed for ssd, why debian do not trim by default?
>
> Detecting reliably if the current system has SSDs that would benefit 
> from trimming AND that the user has not taken their own measures is 
> difficult. I guess this might be the reason for there not being an 
> automatism, but you can enable the systemd timer suggested above with 
> a single command.

I am pretty sure that I have never played with fstrim.timer and this is 
the output of "systemctl status fstrim.timer" on my bullseye system:

● fstrim.timer - Discard unused blocks once a week
      Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor 
preset: enabled)
      Active: active (waiting) since Tue 2021-08-17 14:10:17 CEST; 3h 
5min ago
     Trigger: Mon 2021-08-23 01:01:39 CEST; 5 days left
    Triggers: ● fstrim.service
        Docs: man:fstrim

So it seems this weekly schedule is enabled by default on bullseye.
BTW, if my system is not online at that time, will it be triggered on 
next boot?

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web