Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.hardware > #3242 > unrolled thread

Speed ups for a disk IO bound machine

Back to article view | Back to comp.os.linux.hardware

Contents

  Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-07 22:41 +0000
    Re: Speed ups for a disk IO bound machine Roger Blake <rogblake@iname.invalid> - 2016-12-08 16:44 +0000
      Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-10 11:24 +0000
        Re: Speed ups for a disk IO bound machine Roger Blake <rogblake@iname.invalid> - 2016-12-11 01:34 +0000
          Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 22:07 +0000
    Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-10 12:43 +0100
      Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-10 20:44 +0000
        Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-10 23:05 +0100
          Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 11:54 +0000
            Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-11 14:10 +0100
              Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-11 22:46 +0000
                Re: Speed ups for a disk IO bound machine Piergiorgio Sartor <piergiorgio.sartor.this.should.not.be.used@nexgo.REMOVETHIS.de> - 2016-12-12 19:22 +0100
              Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 10:17 +0100
                Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-14 20:03 +0000
                  Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-15 12:55 +0100
                    Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-15 23:18 +0000
                      Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-16 09:34 +0100
          Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 09:57 +0100
    Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-12 09:11 +0100
      Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-12 23:08 +0000
        Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-13 09:29 +0100
          Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-14 23:16 +0000
            Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-15 13:15 +0100
              Re: Speed ups for a disk IO bound machine Peter Chant <pete@petezilla.co.uk> - 2016-12-15 22:50 +0000
                Re: Speed ups for a disk IO bound machine David Brown <david.brown@hesbynett.no> - 2016-12-16 09:39 +0100

Page 2 of 2 — ← Prev page 1 [2]

#3256

On 13/12/16 00:08, Peter Chant wrote:
> On 12/12/2016 08:11 AM, David Brown wrote:
> 
>> My first thought is to ask what the machine is being used for.  It is
>> not normal to have apache/php/mariadb and firefox/browsing on the same
>> machine.  Usually you have either a server (and therefore no desktop
>> processes), or a desktop (and therefore no server processes of
>> particular relevance).  So what are you doing with the system?  What is
>> it that is using the I/O ?  Is the machine really too slow, and what do
>> you /feel/ is slow on it?
>>
> 
> Home desktop / server.  I have mariadb on it and the easierst way for a
> front end for some stuff I am doing seemed to be a lamp stack.

Do you mean you are running server an active webserver on the same
system you are using for browsing, development, games, email, etc.?
That is a poor setup, for efficiency, reliability, and security.  Of
course, economics can be a factor - but if you can afford to be playing
around with SSDs and multiple disks, then you should also consider if
you should have a separate machine for the server.  A small Intel NUC
with a single disk is likely to be good enough for your LAMP stack and
mediatomb server, leaving your desktop free to be a desktop.

> 
> Also have mediatomb on it and serve files to mpd on a raspberry pi.
> Mediatomb seems to churn the disk sometimes.  Not installed right now.
> 
> 
>> My second thought is that often the best way to improve I/O performance
>> is to add more ram.  Is that a possibility with your hardware?
>>
> 
> I've got 8GB and about 40% acts as a disk cache.  I could try hunting
> down 16GB on ebay but I paid full price a year or so ago for the 8GB.
> Looking at kinfocentre right now I have 19% of my memory free, 35% in
> disk cache and 44% application data.  So would adding more ram do
> anything but add to free memory?
> 

Yes, adding memory will make a /serious/ difference when you are trying
to work as a server and a desktop - /if/ you really are having
performance issues with I/O.  But to be honest, I don't think you /are/
having I/O performance issues - I suspect you just think you are.  If
you have a lot of free memory (and 19% is quite a lot, unless you have
just stopped a large process) then you are not actually doing a lot of
I/O, because disk data is cached in ram whenever there is /any/ free ram.

With more ram, writes go faster because they stay in memory for longer
and don't get flushed to disk as often.  Reads go faster because it is
much more likely that the data is already in memory.  You also have the
option of putting /tmp on tmpfs to speed up processes that use a lot of
temporary files.

But again, I would strongly suggest you try to identify what /feels/
slow, and consider how you use the machine.  What are you doing in
parallel?  What sort of serving are you actually doing, and is it
running in parallel with desktop usage?  Why do you think your I/O is slow?

[toc] | [prev] | [next] | [standalone]

#3260

On 12/13/2016 08:29 AM, David Brown wrote:
> On 13/12/16 00:08, Peter Chant wrote:
>> On 12/12/2016 08:11 AM, David Brown wrote:
>>
>>> My first thought is to ask what the machine is being used for.  It is
>>> not normal to have apache/php/mariadb and firefox/browsing on the same
>>> machine.  Usually you have either a server (and therefore no desktop
>>> processes), or a desktop (and therefore no server processes of
>>> particular relevance).  So what are you doing with the system?  What is
>>> it that is using the I/O ?  Is the machine really too slow, and what do
>>> you /feel/ is slow on it?
>>>
>>
>> Home desktop / server.  I have mariadb on it and the easierst way for a
>> front end for some stuff I am doing seemed to be a lamp stack.
> 
> Do you mean you are running server an active webserver on the same
> system you are using for browsing, development, games, email, etc.?
> That is a poor setup, for efficiency, reliability, and security.  Of
> course, economics can be a factor - but if you can afford to be playing
> around with SSDs and multiple disks, then you should also consider if
> you should have a separate machine for the server.  A small Intel NUC
> with a single disk is likely to be good enough for your LAMP stack and
> mediatomb server, leaving your desktop free to be a desktop.
> 

I think my lamp stack is somewhat atypical.  I'm storing numerical data
in it and doing some calcs on that.  Might be good for storage but calcs
in python and php are not optimal.  However, this is partly historic and
partly convenience and rework would be a major pita.

However, if I get slowdowns on this fairly elderly machine and a six
core cpu and 8GB of ram then with albeit a newer generation CPU I don't
see a NUC being much faster, though I admit I've never got my ands on
one.  Plus there is not room for the two hdds plus the ssd OS disk.
Using this machine as the server and the nuc as the desktop would make
more sense.

I did think about this in the past, or getting a nice laptop / docking
station combination and a server setup.

The system is fairly responsive right now but I am not hitting the HDDs
right now using thunderbird as I type this.  This is with duperemove
hitting the HDDs hard and I'd not consider doing anything else that hit
the HDDs.  Incidentally application data is now 72% of physical memory
and disk cache 24-25% with the remainder few % free.


>>
>> Also have mediatomb on it and serve files to mpd on a raspberry pi.
>> Mediatomb seems to churn the disk sometimes.  Not installed right now.
>>
>>
>>> My second thought is that often the best way to improve I/O performance
>>> is to add more ram.  Is that a possibility with your hardware?
>>>
>>
>> I've got 8GB and about 40% acts as a disk cache.  I could try hunting
>> down 16GB on ebay but I paid full price a year or so ago for the 8GB.
>> Looking at kinfocentre right now I have 19% of my memory free, 35% in
>> disk cache and 44% application data.  So would adding more ram do
>> anything but add to free memory?
>>
> 
> Yes, adding memory will make a /serious/ difference when you are trying
> to work as a server and a desktop - /if/ you really are having
> performance issues with I/O.  But to be honest, I don't think you /are/
> having I/O performance issues - I suspect you just think you are.  If
> you have a lot of free memory (and 19% is quite a lot, unless you have
> just stopped a large process) then you are not actually doing a lot of
> I/O, because disk data is cached in ram whenever there is /any/ free ram.
> 

Given that there is little free now the numbers I quoted earlier might
not have been representative.   I did not see a noticeable difference
between 4 & 8 GB therefore I'd not considered more ram.  However, if it
is really likely to make a big difference and with 16GB of DDR2 going
for between £15 and £45 on ebay then some research is warranted.

> With more ram, writes go faster because they stay in memory for longer
> and don't get flushed to disk as often.  Reads go faster because it is
> much more likely that the data is already in memory.  You also have the
> option of putting /tmp on tmpfs to speed up processes that use a lot of
> temporary files.

I've put /tmp on tmpfs before.  I have /dev/shm on tmpfs at the moment
as part of slackware's default config.  Generally I've abandoned this
when compiling packages filled up /tmp and I ran out of tmp space.
Generally a failure to clean up /tmp, but some packages are large and
have a lot of dependencies.

> 
> But again, I would strongly suggest you try to identify what /feels/
> slow, and consider how you use the machine.  What are you doing in
> parallel?  What sort of serving are you actually doing, and is it
> running in parallel with desktop usage?  Why do you think your I/O is slow?
> 

I used the term 'IO bound' as I've seen the HDDs hit 80-90% for long
periods yet CPU usage has been relatively low.  So to me IO was the
limiting factor.  Going out and spending lots of cash (not cache!) on
the latest i7 + motherboard and memory therefore I assume would not
improve the user experience whereas speeding up the existing IO would,
if possible.

The lamp load above is likely excessive but I have seen slowdowns before
with this machine.  Sometimes btrfs seems to build up a backlog of stuff
to do (btrfs cleaner, transactions etc) for a while after doing
something disk intensive.  But I've noticed this less lately.  Btrfs has
not got a reputation for being slow although odd and specific cases do
show up on the mailing list from time to time.  I'm not planning on
swapping file systems unless to another with subvolumes and probably
snapshots as subvolumes have let me organise things in a much more
logical and efficient manner since I have started using them.

I have a nagging feeling that something just is not right.  However, I
need to benchmark.  I also have had a cheap two interface SATAIII card.
If there is something odd with the disk interface (can't see what) maybe
that will shake it out.  It should allow the SSD to function to its
potential anyway, so it is not a bad idea.

Unfortunately the slightly higher range 4 port SATA III PCIe x2 cards
seem limited right now, I'd have to go up quite a notch in price to
eight port / SAS cards and I'm starting to through reasonable sums of
money at an elderly mobo / processor / ram combination with no assured
outcome.  However, cheap improvements and especially improvements with
existing kit are definitely work pursuing.

[toc] | [prev] | [next] | [standalone]

#3262

On 15/12/16 00:16, Peter Chant wrote:
> On 12/13/2016 08:29 AM, David Brown wrote:
>> On 13/12/16 00:08, Peter Chant wrote:
>>> On 12/12/2016 08:11 AM, David Brown wrote:
>>>
>>>> My first thought is to ask what the machine is being used for.  It is
>>>> not normal to have apache/php/mariadb and firefox/browsing on the same
>>>> machine.  Usually you have either a server (and therefore no desktop
>>>> processes), or a desktop (and therefore no server processes of
>>>> particular relevance).  So what are you doing with the system?  What is
>>>> it that is using the I/O ?  Is the machine really too slow, and what do
>>>> you /feel/ is slow on it?
>>>>
>>>
>>> Home desktop / server.  I have mariadb on it and the easierst way for a
>>> front end for some stuff I am doing seemed to be a lamp stack.
>>
>> Do you mean you are running server an active webserver on the same
>> system you are using for browsing, development, games, email, etc.?
>> That is a poor setup, for efficiency, reliability, and security.  Of
>> course, economics can be a factor - but if you can afford to be playing
>> around with SSDs and multiple disks, then you should also consider if
>> you should have a separate machine for the server.  A small Intel NUC
>> with a single disk is likely to be good enough for your LAMP stack and
>> mediatomb server, leaving your desktop free to be a desktop.
>>
> 
> I think my lamp stack is somewhat atypical.  I'm storing numerical data
> in it and doing some calcs on that.  Might be good for storage but calcs
> in python and php are not optimal.  However, this is partly historic and
> partly convenience and rework would be a major pita.

For Python, you can look at numpy and scipy for serious calculations -
if you can work with your data as homogeneous arrays then numpy will do
the calculations in fast C libraries rather than interpreted Python.
Also look at pypy or psyco as ways to speed up Python code.  You may
also find that if you have heavy Python pages you are better using
Twisted as a webserver rather than Apache so that you work entirely
within the one Python process rather than starting and stopping
processes all the time.

> 
> However, if I get slowdowns on this fairly elderly machine and a six
> core cpu and 8GB of ram then with albeit a newer generation CPU I don't
> see a NUC being much faster, though I admit I've never got my ands on
> one.  Plus there is not room for the two hdds plus the ssd OS disk.

A NUC will give you as good I/O disk throughput (or better - I think you
only had SATA-2 on your machine?).  The processor may or may not be
better - there are far too many NUC variants to keep track of!

But the point is to separate the different types of usage.

> Using this machine as the server and the nuc as the desktop would make
> more sense.

Maybe that's the way to do it.

> 
> I did think about this in the past, or getting a nice laptop / docking
> station combination and a server setup.
> 
> The system is fairly responsive right now but I am not hitting the HDDs
> right now using thunderbird as I type this.  This is with duperemove
> hitting the HDDs hard and I'd not consider doing anything else that hit
> the HDDs.  Incidentally application data is now 72% of physical memory
> and disk cache 24-25% with the remainder few % free.
> 
> 
>>>
>>> Also have mediatomb on it and serve files to mpd on a raspberry pi.
>>> Mediatomb seems to churn the disk sometimes.  Not installed right now.
>>>
>>>
>>>> My second thought is that often the best way to improve I/O performance
>>>> is to add more ram.  Is that a possibility with your hardware?
>>>>
>>>
>>> I've got 8GB and about 40% acts as a disk cache.  I could try hunting
>>> down 16GB on ebay but I paid full price a year or so ago for the 8GB.
>>> Looking at kinfocentre right now I have 19% of my memory free, 35% in
>>> disk cache and 44% application data.  So would adding more ram do
>>> anything but add to free memory?
>>>
>>
>> Yes, adding memory will make a /serious/ difference when you are trying
>> to work as a server and a desktop - /if/ you really are having
>> performance issues with I/O.  But to be honest, I don't think you /are/
>> having I/O performance issues - I suspect you just think you are.  If
>> you have a lot of free memory (and 19% is quite a lot, unless you have
>> just stopped a large process) then you are not actually doing a lot of
>> I/O, because disk data is cached in ram whenever there is /any/ free ram.
>>
> 
> Given that there is little free now the numbers I quoted earlier might
> not have been representative.   I did not see a noticeable difference
> between 4 & 8 GB therefore I'd not considered more ram.  However, if it
> is really likely to make a big difference and with 16GB of DDR2 going
> for between £15 and £45 on ebay then some research is warranted.

I have seen extra ram make an impressive difference to speed.  Not long
ago a fellow developer here thought he needed a new graphics card at
about £500 because his current £300 one was too slow for the 3D
rendering he was doing.  But £30 more ram doubled the speed of the
system, while a new graphics card would have made little difference.

> 
>> With more ram, writes go faster because they stay in memory for longer
>> and don't get flushed to disk as often.  Reads go faster because it is
>> much more likely that the data is already in memory.  You also have the
>> option of putting /tmp on tmpfs to speed up processes that use a lot of
>> temporary files.
> 
> I've put /tmp on tmpfs before.  I have /dev/shm on tmpfs at the moment
> as part of slackware's default config.  Generally I've abandoned this
> when compiling packages filled up /tmp and I ran out of tmp space.
> Generally a failure to clean up /tmp, but some packages are large and
> have a lot of dependencies.

/dev/shm is always on tmpfs (in modern systems, anyway).  Processes use
it specifically as a convenient way to have a block of memory shared
between them - the create a file on the tmpfs while remains in memory,
then mmap it to access the memory directly.

It is often more efficient to have /tmp on tmpfs and let it spill out
into swap, than to have the /tmp directly on the disk.

> 
>>
>> But again, I would strongly suggest you try to identify what /feels/
>> slow, and consider how you use the machine.  What are you doing in
>> parallel?  What sort of serving are you actually doing, and is it
>> running in parallel with desktop usage?  Why do you think your I/O is slow?
>>
> 
> I used the term 'IO bound' as I've seen the HDDs hit 80-90% for long
> periods yet CPU usage has been relatively low.  So to me IO was the
> limiting factor.  Going out and spending lots of cash (not cache!) on
> the latest i7 + motherboard and memory therefore I assume would not
> improve the user experience whereas speeding up the existing IO would,
> if possible.
> 
> The lamp load above is likely excessive but I have seen slowdowns before
> with this machine.  Sometimes btrfs seems to build up a backlog of stuff
> to do (btrfs cleaner, transactions etc) for a while after doing
> something disk intensive.  But I've noticed this less lately.  Btrfs has
> not got a reputation for being slow although odd and specific cases do
> show up on the mailing list from time to time.  I'm not planning on
> swapping file systems unless to another with subvolumes and probably
> snapshots as subvolumes have let me organise things in a much more
> logical and efficient manner since I have started using them.

Agreed.  Btrfs is not perfect, but I find it the best choice for my
usage.  Cheap snapshots are really nice!

> 
> I have a nagging feeling that something just is not right.  However, I
> need to benchmark.  I also have had a cheap two interface SATAIII card.
> If there is something odd with the disk interface (can't see what) maybe
> that will shake it out.  It should allow the SSD to function to its
> potential anyway, so it is not a bad idea.

hdparm can give you a simple test of the buffer bandwidth, and smartctrl
can be useful to list the features of the interface and device to check
for obvious missing points.

A newer cpu and motherboard may not seem useful from the viewpoint of
processor power, but they will have better throughput on the I/O and
faster native SATA.

> 
> Unfortunately the slightly higher range 4 port SATA III PCIe x2 cards
> seem limited right now, I'd have to go up quite a notch in price to
> eight port / SAS cards and I'm starting to through reasonable sums of
> money at an elderly mobo / processor / ram combination with no assured
> outcome.  However, cheap improvements and especially improvements with
> existing kit are definitely work pursuing.
> 

I'd avoid SAS - but that is because I have been bitten by it.  There is
not much that SAS does better than SATA (queued trim commands is one
thing), and the disks cost twice as much.  I prefer just to get twice as
many SATA disks for the same money.  I had one server that had SAS
because the salesman convinced me that the extra reliability of SAS
drivers over SATA drivers was worth the cost.  That machine broke when
the SAS controller card died, leaving me with a useless server and a
disk that I could not read because everything else was SATA.  Sometimes
you learn by doing!

[toc] | [prev] | [next] | [standalone]

#3263

On 12/15/2016 12:15 PM, David Brown wrote:

>>
>> I think my lamp stack is somewhat atypical.  I'm storing numerical data
>> in it and doing some calcs on that.  Might be good for storage but calcs
>> in python and php are not optimal.  However, this is partly historic and
>> partly convenience and rework would be a major pita.
> 
> For Python, you can look at numpy and scipy for serious calculations -
> if you can work with your data as homogeneous arrays then numpy will do
> the calculations in fast C libraries rather than interpreted Python.
> Also look at pypy or psyco as ways to speed up Python code.  You may
> also find that if you have heavy Python pages you are better using
> Twisted as a webserver rather than Apache so that you work entirely
> within the one Python process rather than starting and stopping
> processes all the time.
> 

At this stage the rewrite is probally not worthwhile.  I'm not sure this
bit is necessarily the bottleneck anyway.  Investigation required.


>>
>> Given that there is little free now the numbers I quoted earlier might
>> not have been representative.   I did not see a noticeable difference
>> between 4 & 8 GB therefore I'd not considered more ram.  However, if it
>> is really likely to make a big difference and with 16GB of DDR2 going
>> for between £15 and £45 on ebay then some research is warranted.
> 
> I have seen extra ram make an impressive difference to speed.  Not long
> ago a fellow developer here thought he needed a new graphics card at
> about £500 because his current £300 one was too slow for the 3D
> rendering he was doing.  But £30 more ram doubled the speed of the
> system, while a new graphics card would have made little difference.
> 

The lack of noticable difference put me off.  But a little careful
ebaying might bump me to 16 for a reasonable price.


> It is often more efficient to have /tmp on tmpfs and let it spill out
> into swap, than to have the /tmp directly on the disk.
> 

That is good to know for the odd occasion.  Hmm.  I've just noticed a
tiny swap hit with 7% free physical memory as I write this.  First ever,
that I have noticed!


> A newer cpu and motherboard may not seem useful from the viewpoint of
> processor power, but they will have better throughput on the I/O and
> faster native SATA.
> 

I've a cheap as chips PCIe SATA 3 controller waiting to go in.  I'm
curious as to weather it would make a difference to the HDDs, it should
not.  Anyway, it should help the SSD or SSDs.

[toc] | [prev] | [next] | [standalone]

#3266

On 15/12/16 23:50, Peter Chant wrote:
> On 12/15/2016 12:15 PM, David Brown wrote:
> 
>>>
>>> I think my lamp stack is somewhat atypical.  I'm storing numerical data
>>> in it and doing some calcs on that.  Might be good for storage but calcs
>>> in python and php are not optimal.  However, this is partly historic and
>>> partly convenience and rework would be a major pita.
>>
>> For Python, you can look at numpy and scipy for serious calculations -
>> if you can work with your data as homogeneous arrays then numpy will do
>> the calculations in fast C libraries rather than interpreted Python.
>> Also look at pypy or psyco as ways to speed up Python code.  You may
>> also find that if you have heavy Python pages you are better using
>> Twisted as a webserver rather than Apache so that you work entirely
>> within the one Python process rather than starting and stopping
>> processes all the time.
>>
> 
> At this stage the rewrite is probally not worthwhile.  I'm not sure this
> bit is necessarily the bottleneck anyway.  Investigation required.

If you are using 32-bit Python 2, psyco can be a great solution.
Install it, and then add this to your Python script:

	import psyco
	psyco.full()

That's all you need, and psyco will do JIT compilation of functions that
are run often.

As a project, psyco is now "dead" - PyPy is the successor for newer
Python systems.  But it can be a very easy way to speed up some kinds of
code quite significantly.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.os.linux.hardware

csiph-web