Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #38702

Re: Massiv creating and copying of files

Show all headers | View raw

On 1/17/2019 9:19 AM, Sandro Leinzinger wrote:
> I wrote a program which does this:
> 
> 1. Create a temp folder
> 2. Requesting a file from a web server and copy to temp folder
> 3. Generating some files in the temp folder from 1.
> 4. Generating a .iso-file from the temp-folder content with mkisofs
> 5. Zipping the whole content (3GB) of temp
> 
> All this operations are happening on the same harddisk array. The
> porblem is, that the array is getting slower and slower and works on
> 100% even with only 5 Threads executing this task.
> 
> Can some one tell me some optimisation in this task? Maybe create
> them in ram and then copy the zip to the hdd? But this would also
> bring the disks to 100% of load?
I don't think there is anything Java specific in this.

It seems to involve:
* writing 3 GB of files to disk
* create ISO which will be another 3 GB
* create ZIP which will probably be another 1.0-1.5 GB

Depending on the IO system that may take some time.

And it sounds like all files eventually need to be on disk,
so no real option for avoiding any writes.

So back to basic optimization:
* ensure that your code read and write really large
   chunks (not a raw FileXxxxStream but wrapped in a
   BufferedXxxxStream in Java)
* ensure that there is plenty of space on disk (less than
   half full may be best) to avoid fragmentation
* ensure that the OS has sufficient memory for IO
   buffers

But there will be a hard limit on what the optimization can
achieve. If the IO system take T time to write 7.5 GB, then
that is it.

Arne

Back to comp.lang.java.programmer | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Massiv creating and copying of files Sandro Leinzinger <leinzinger.sandro@googlemail.com> - 2019-01-17 06:19 -0800
  Re: Massiv creating and copying of files Arne Vajhøj <arne@vajhoej.dk> - 2019-01-17 09:30 -0500
  Re: Massiv creating and copying of files Eric Sosman <esosman@comcast-dot-net.invalid> - 2019-01-17 11:30 -0500
    Re: Massiv creating and copying of files Sandro Leinzinger <leinzinger.sandro@googlemail.com> - 2019-01-17 11:11 -0800
      Re: Massiv creating and copying of files Eric Douglas <e.d.programmer@gmail.com> - 2019-01-17 11:24 -0800
        Re: Massiv creating and copying of files Eric Sosman <esosman@comcast-dot-net.invalid> - 2019-01-17 17:16 -0500
          Re: Massiv creating and copying of files Martin Gregorie <martin@mydomain.invalid> - 2019-01-17 23:33 +0000
  Re: Massiv creating and copying of files Marcel Mueller <news.5.maazl@spamgourmet.org> - 2019-01-18 07:57 +0100

csiph-web