Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #38708

Re: Massiv creating and copying of files

Show all headers | View raw

Am 17.01.19 um 15:19 schrieb Sandro Leinzinger:
> I wrote a program which does this:
> 
> 1. Create a temp folder
> 2. Requesting a file from a web server and copy to temp folder
> 3. Generating some files in the temp folder from 1.
> 4. Generating a .iso-file from the temp-folder content with mkisofs
> 5. Zipping the whole content (3GB) of temp
> 
> All this operations are happening on the same harddisk array. The porblem is, that the array is getting slower and slower and works on 100% even with only 5 Threads executing this task.
> 
> Can some one tell me some optimisation in this task? Maybe create them in ram and then copy the zip to the hdd? But this would also bring the disks to 100% of load?

There is no Java dependence in your question.

Traditional single hard disks slow down roughly by a factor of 10 on 
random access. The latter is likely with concurrent operations.
Disk arrays perform a bit better if well configured as long as the 
number of concurrent requests is not significantly larger than the 
number of physical disks.

So if you want to speed up things you have exactly three options:
1. reduce the amount of I/O.
2. do preferably linear reads and writes to avoid the slowdown.
3. Use an SSD. SSDs do not show the slowdown on concurrent I/O.

The first one can be accomplished by writing less temp files. E.g. you 
can switch to stream processing. Currently you are writing all data 
roughly 4 times. (Step 2 to 5)
You can eliminate at least step 4 by using a pipe to pass the result of 
mkisofs on the fly to the compression task.
You may further eliminate I/O by passing the result of the Web requests 
directly to your generation code from step 3, if this is possible.

To reduce concurrency of I/O ensure that all buffers used for disk I/O 
are at least in the order of 5MB. When reading or writing 5MB at once 
the slowdown of HDDs typically becomes neglectable. This applies to 
buffered Java InputStreams or OutputStreams as well as to your external 
applications that read ans write data.


Marcel

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

Massiv creating and copying of files Sandro Leinzinger <leinzinger.sandro@googlemail.com> - 2019-01-17 06:19 -0800
  Re: Massiv creating and copying of files Arne Vajhøj <arne@vajhoej.dk> - 2019-01-17 09:30 -0500
  Re: Massiv creating and copying of files Eric Sosman <esosman@comcast-dot-net.invalid> - 2019-01-17 11:30 -0500
    Re: Massiv creating and copying of files Sandro Leinzinger <leinzinger.sandro@googlemail.com> - 2019-01-17 11:11 -0800
      Re: Massiv creating and copying of files Eric Douglas <e.d.programmer@gmail.com> - 2019-01-17 11:24 -0800
        Re: Massiv creating and copying of files Eric Sosman <esosman@comcast-dot-net.invalid> - 2019-01-17 17:16 -0500
          Re: Massiv creating and copying of files Martin Gregorie <martin@mydomain.invalid> - 2019-01-17 23:33 +0000
  Re: Massiv creating and copying of files Marcel Mueller <news.5.maazl@spamgourmet.org> - 2019-01-18 07:57 +0100

csiph-web