Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #12235 > unrolled thread

O.T. optimising file placement

Back to article view | Back to comp.lang.java.programmer

Contents

  O.T. optimising file placement Roedy Green <see_website@mindprod.com.invalid> - 2012-02-21 09:59 -0800
    Re: O.T. optimising file placement Daniel Pitts <newsgroup.nospam@virtualinfinity.net> - 2012-02-21 18:05 -0800
    Re: O.T. optimising file placement Jeff Higgins <jeff@invalid.invalid> - 2012-02-21 21:41 -0500
      Re: O.T. optimising file placement Jeff Higgins <jeff@invalid.invalid> - 2012-02-21 23:23 -0500
        Re: O.T. optimising file placement Roedy Green <see_website@mindprod.com.invalid> - 2012-02-22 11:15 -0800
          Re: O.T. optimising file placement Jeff Higgins <jeff@invalid.invalid> - 2012-02-22 16:00 -0500
            Re: O.T. optimising file placement Jeff Higgins <jeff@invalid.invalid> - 2012-02-22 16:03 -0500
              Re: O.T. optimising file placement Gene Wirchenko <genew@ocis.net> - 2012-02-22 14:52 -0800
                Re: O.T. optimising file placement Jeff Higgins <jeff@invalid.invalid> - 2012-02-22 18:24 -0500
                  Re: O.T. optimising file placement Daniel Pitts <newsgroup.nospam@virtualinfinity.net> - 2012-02-22 17:10 -0800
                  Re: O.T. optimising file placement Gene Wirchenko <genew@ocis.net> - 2012-02-22 20:02 -0800
          Re: O.T. optimising file placement Lew <noone@lewscanon.com> - 2012-02-22 14:27 -0800
            Re: O.T. optimising file placement Martin Gregorie <martin@address-in-sig.invalid> - 2012-02-23 01:31 +0000
              Re: O.T. optimising file placement Arne Vajhøj <arne@vajhoej.dk> - 2012-02-22 21:40 -0500
                Re: O.T. optimising file placement Martin Gregorie <martin@address-in-sig.invalid> - 2012-02-23 23:16 +0000
                  Re: O.T. optimising file placement Patricia Shanahan <pats@acm.org> - 2012-02-23 16:22 -0800
                  Re: O.T. optimising file placement Patricia Shanahan <pats@acm.org> - 2012-02-23 21:45 -0800
                    Re: O.T. optimising file placement Martin Gregorie <martin@address-in-sig.invalid> - 2012-02-24 20:21 +0000
                  Re: O.T. optimising file placement Arne Vajhøj <arne@vajhoej.dk> - 2012-02-24 20:31 -0500
                    Re: O.T. optimising file placement Martin Gregorie <martin@address-in-sig.invalid> - 2012-02-25 13:15 +0000
              Re: O.T. optimising file placement Lew <noone@lewscanon.com> - 2012-02-23 16:39 -0800
                Re: O.T. optimising file placement Martin Gregorie <martin@address-in-sig.invalid> - 2012-02-24 01:49 +0000
                  Re: O.T. optimising file placement Lew <noone@lewscanon.com> - 2012-02-24 10:50 -0800
                    Re: O.T. optimising file placement Martin Gregorie <martin@address-in-sig.invalid> - 2012-02-24 23:59 +0000
                Re: O.T. optimising file placement Arne Vajhøj <arne@vajhoej.dk> - 2012-02-24 20:25 -0500
      Re: O.T. optimising file placement Roedy Green <see_website@mindprod.com.invalid> - 2012-02-22 10:54 -0800

Page 2 of 2 — ← Prev page 1 [2]

#12289

On 02/22/2012 05:31 PM, Martin Gregorie wrote:
> On Wed, 22 Feb 2012 14:27:17 -0800, Lew wrote:
>
>> Modern hard drives, pretty much all of them, have a buffer and
>> microprocessor as part of the hardware. We're not going to get any
>> "Marthaing" as you describe it (wherever the heck /that/ term came from)
>> because what they're already doing is already so effective.
>>
>> What they mostly do is collect read and write requests and combine them
>> in elevator-seek order, along with full-track readahead. This optimizes
>> disk access for single sweeps of the drive heads.
>>
> Agreed, and a mainframe OS I was using in the early '70s (ICL's George 3)
> was doing it back then and very effective it is too for speeding up disk
> access. Back in the day it pushed the speed of the 2800 rpm, 60 MB
> washing-machine sized disk drives up from around 8 accesses/sec to
> something like 20-30 per sec.
>
> However, its ineffective unless there are many active processes
> simultaneously requesting disk i/o. If all the requests come from one

Spurious point, and not entirely true. It's effective if many requests come in 
between I/O activity windows regardless of how many processes are involved.

> single threaded process then it can't optimize head movement because
> there's never more than one pending request at a time. I know this is

Except for when there is. Also, consumer OSes like Windows are always running 
multiple processes, most of them in the background like, oh, the Windows 
indexer which is a big consumer of I/O services.

> reduction ad absurdam, but it does make the point that a small active
> process population is unlikely to be optimised as well as a large one.

It's a spurious point because who needs to optimize a slow, single-process 
scenario?

> This is relevant today for allmost all single-user workstations
> regardless of whether they are running Windows, Linux or OS X. Since the

You mean irrelevant.

First of all, single-user does not mean single-process. All these systems are 
multiple-process with multiple processes at all times.

Second, if the system is so light on demand that you don't benefit from 
elevator seeking and on-disk cache, why do you care about moving tracks or any 
other optimization? You don't compare race cars to beetles by saying that the 
performance enhancements in the race car don't matter to street driving.

> majority of applications run on these machines are single threaded, about

Show us some data.  Please don't throw around terms like "the majorit of 
applications [that] run on these machines" unless you have data.

This point is highly disputable. I've looked at "top" and the Windows System 
Manager and seen many poly-threaded or poly-process processes running. I've 
never run the numbers to see which are in the majority, nor quite sure what I 
would define as "majority" in that circumstance.

Do you weight them, say by giving virtual memory a very high weight?  Oh, 
yeah, remember virtual memory?  Now there's a multi-threaded, high-priority 
consumer of disk.

In any event, enough processes in consumer or, as you call it, "single-user" 
stations are concurrent and disk-hungry to make it plausible that elevator 
seeking and track buffers will benefit even such systems.

> the only time you have more than one process accessing the disk is when
> the user is hammering away at a task, be it wordprocessing, spread-sheet,
> browser or IDE and the mail reader, sitting in the background, finds some
> mail waiting.

I think we can all agree by now that that is not true.

>> The on-drive buffer
>> also holds enough data for most reads and writes, overtaking any
>> advantage that any (perforce extremely slow) physical re-ordering of the
>> tracks could accomplish.
>>
> Yep, the on-drive buffer will almost always be capable of holding several
> physical tracks and, in addition, on a *NIX system anyway, all RAM not
> occupied by running processes and their data will contain disk buffers.

-- 
Lew
Honi soit qui mal y pense.
http://upload.wikimedia.org/wikipedia/commons/c/cf/Friz.jpg

[toc] | [prev] | [next] | [standalone]

#12290

On Thu, 23 Feb 2012 16:39:42 -0800, Lew wrote:

> 
> I think we can all agree by now that that is not true.
>
Unlike some, I take a good deal of interest in what my machines are up 
to, so I was quoting what I see using top on my Linux systems. During 
normal operation there is very little activity on my laptop except from 
the programs I'm actively using unless, as you say, logwatch/smartd/
rkhunter/updatedb get run by atd, but on a reasonably quick machine they 
don't run for long.

Of course, the house server is a different case, since it has several 
24/7 services on it, but again its only heavy, continuous disk activity 
is overnight when it runs backups/logwatch/smartd/updatedb. Apart from 
that requests that wake up Postfix/Spamassassin/Apache/or ftpd/sshd are 
pretty sporadic and the disk LED flashes are best described as 
intermittent.

The longest continuously busy time on either machine is during backups 
and even there there precious little contention since rsync or tar+gzip 
since the only stuff being written to the disk its reading from are 
backup logs. Same applies to software update sessions. To the best of my 
knowledge (and watching top) none of yum, rpm, tar, gzip or rsync are 
multi-threaded: rsync is probably using poll() based async i/o but from 
top and observed behaviour none of the others seem to do that. In fact 
the only long-running programs on my systems that I know to be multi-
threaded are Apache, Postgres, SA and Postfix. 


-- 
martin@   | Martin Gregorie
gregorie. | Essex, UK
org       |

[toc] | [prev] | [next] | [standalone]

#12293

Martin Gregorie wrote:
> Unlike some, I take a good deal of interest in what my machines are up
> to, so I was quoting what I see using top on my Linux systems. During
> normal operation there is very little activity on my laptop except from
> the programs I'm actively using unless, as you say, logwatch/smartd/
> rkhunter/updatedb get run by atd, but on a reasonably quick machine they
> don't run for long.
>
> Of course, the house server is a different case, since it has several
> 24/7 services on it, but again its only heavy, continuous disk activity
> is overnight when it runs backups/logwatch/smartd/updatedb. Apart from
> that requests that wake up Postfix/Spamassassin/Apache/or ftpd/sshd are
> pretty sporadic and the disk LED flashes are best described as
> intermittent.

Sounds like disk optimizations would help that system.

> The longest continuously busy time on either machine is during backups
> and even there there precious little contention since rsync or tar+gzip
> since the only stuff being written to the disk its reading from are
> backup logs. Same applies to software update sessions. To the best of my
> knowledge (and watching top) none of yum, rpm, tar, gzip or rsync are
> multi-threaded: rsync is probably using poll() based async i/o but from
> top and observed behaviour none of the others seem to do that. In fact
> the only long-running programs on my systems that I know to be multi-
> threaded are Apache, Postgres, SA and Postfix.

Now /that/ is objective evidence.

In your particular case you have no need of optimization of your disk 
processes. You don't mention it but by omission I will grant you that virtual 
memory on your system does not seriously contend for disk either. But a 
typical consumer scenario is to listen to a stream while surfing the web on 
Windows with several chat windows open, causing multiple disk IO ops on a 
constant basis of themselves and also putting pressure on virtual memory. Even 
such a single-user system can benefit from elevator seeking and on-disk buffers.

Consider also that burstiness of demand does not argue against the need for 
optimization, really. During bursts the optimization helps, and a user might 
complain if their disks got weird once an hour.

Regardless, if you don't need optimization why worry? It's like the Pope 
comparing brands of condoms.

Again, we don't excoriate the value of optimizations by citing examples where 
optimization isn't needed. We evaluate optimizations by how useful they are 
when they are needed.

-- 
Lew
Honi soit qui mal y pense.
http://upload.wikimedia.org/wikipedia/commons/c/cf/Friz.jpg

[toc] | [prev] | [next] | [standalone]

#12302

On Fri, 24 Feb 2012 10:50:39 -0800, Lew wrote:

> Martin Gregorie wrote:
>> Unlike some, I take a good deal of interest in what my machines are up
>> to, so I was quoting what I see using top on my Linux systems. During
>> normal operation there is very little activity on my laptop except from
>> the programs I'm actively using unless, as you say, logwatch/smartd/
>> rkhunter/updatedb get run by atd, but on a reasonably quick machine
>> they don't run for long.
>>
>> Of course, the house server is a different case, since it has several
>> 24/7 services on it, but again its only heavy, continuous disk activity
>> is overnight when it runs backups/logwatch/smartd/updatedb. Apart from
>> that requests that wake up Postfix/Spamassassin/Apache/or ftpd/sshd are
>> pretty sporadic and the disk LED flashes are best described as
>> intermittent.
> 
> Sounds like disk optimizations would help that system.
>
Probably not - they are all cron jobs and hence get run sequentially.
 
>> The longest continuously busy time on either machine is during backups
>> and even there there precious little contention since rsync or tar+gzip
>> since the only stuff being written to the disk its reading from are
>> backup logs. Same applies to software update sessions. To the best of
>> my knowledge (and watching top) none of yum, rpm, tar, gzip or rsync
>> are multi-threaded: rsync is probably using poll() based async i/o but
>> from top and observed behaviour none of the others seem to do that. In
>> fact the only long-running programs on my systems that I know to be
>> multi- threaded are Apache, Postgres, SA and Postfix.
> 
> In your particular case you have no need of optimization of your disk
> processes. You don't mention it but by omission I will grant you that
> virtual memory on your system does not seriously contend for disk
> either.
>
Well spotted. My type of load almost never swaps. That was the case with 
the old 512 MB RAM box and is double true with its replacement (4 GB 
RAM), but that still doesn't stop me setting swap space at twice RAM.

In fact the only program I have that does use gobs on RAM is a JavaMail + 
Postgres app and I'm not sure if its a problem due to JavaMail's queueing 
or if I've got overly long lived Object instances. Tracking this down in 
on my to-do list. All I know at present is that the same program using 
the same JVM uses gobs more RAM on the new machine (which is 6 times 
faster as well as having 8x more RAM), so it might simply be a case of 
persuading the GC to run more often. 

> But a typical consumer scenario is to listen to a stream while
> surfing the web on Windows with several chat windows open, causing
> multiple disk IO ops on a constant basis of themselves and also putting
> pressure on virtual memory. Even such a single-user system can benefit
> from elevator seeking and on-disk buffers.
>
I'm not saying head movement optimisation is a bad thing, just that it 
can be difficult to get enough queued requests for it to work without a 
large population of active processes that all do a lot of disk accesses.

You may well be right about the typical consumer setup: I lack any 
experience that: all I understand is the pattern that my own use pattern 
generates. However, I would point out that streamed music or video may 
never touch the disk (though of course a torrent will). The amount of 
disk i/o due to chat/IM/Twitter/web browsers may be less that we'd expect 
because its (a) very bursty and (b) disk i/o time is vastly outweighed by 
human reading and typing time.
  
> Consider also that burstiness of demand does not argue against the need
> for optimization, really. During bursts the optimization helps, and a
> user might complain if their disks got weird once an hour.
>
Sure, but the user's activity scan and resulting interaction with one 
program at a time, which may well be single threaded, for a few minutes 
before switching to another. This tends to produce widely separated 
bursts of i/o from one or two processes.
 
> Regardless, if you don't need optimization why worry? It's like the Pope
> comparing brands of condoms.
> 
Like it! 

> Again, we don't excoriate the value of optimizations by citing examples
> where optimization isn't needed. We evaluate optimizations by how useful
> they are when they are needed.
>
I wasn't intending to do that, having seen just how well head scheduling 
works. I merely intended to point out that there are corner cases where 
such algorithms don't help - but are not a hindrance either.


-- 
martin@   | Martin Gregorie
gregorie. | Essex, UK
org       |

[toc] | [prev] | [next] | [standalone]

#12307

On 2/23/2012 7:39 PM, Lew wrote:
> On 02/22/2012 05:31 PM, Martin Gregorie wrote:
>> This is relevant today for allmost all single-user workstations
>> regardless of whether they are running Windows, Linux or OS X. Since the
>
> You mean irrelevant.
>
> First of all, single-user does not mean single-process. All these
> systems are multiple-process with multiple processes at all times.

And multithreaded.

My Win7 has right now 105 processes with 1858 threads!

>> majority of applications run on these machines are single threaded, about
>
> Show us some data. Please don't throw around terms like "the majorit of
> applications [that] run on these machines" unless you have data.

Only 6 out of 105 is singlethreaded here.

Arne

[toc] | [prev] | [next] | [standalone]

#12249

On Tue, 21 Feb 2012 21:41:56 -0500, Jeff Higgins
<jeff@invalid.invalid> wrote, quoted or indirectly quoted someone who
said :

><http://en.wikipedia.org/wiki/ReadyBoost>

"Windows 7 allows up to eight devices for a maximum of 256 GB of
additional memory,[5] with up to 32 GB on a single storage device.[6]"

It seems like yesterday I tested it in Vista where it had  a limit of
4 Gig. 
-- 
Roedy Green Canadian Mind Products
http://mindprod.com
One of the most useful comments you can put in a program is 
"If you change this, remember to change ?XXX? too".
 

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.java.programmer

csiph-web