Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.programming > #3096 > unrolled thread

Parallel archiver version 1.2 is here...

Back to article view | Back to comp.programming

Contents

  Parallel archiver version 1.2 is here... aminer <aminer@toto.ca> - 2013-02-24 14:15 -0800
    Re: Parallel archiver version 1.2 is here... aminer <aminer@toto.ca> - 2013-02-24 17:15 -0800
    Re: Parallel archiver version 1.2 is here... aminer <aminer@toto.ca> - 2013-02-24 17:50 -0800
    Re: Parallel archiver version 1.2 is here... aminer <aminer@toto.ca> - 2013-02-24 18:26 -0800
    Re: Parallel archiver version 1.2 is here... aminer <aminer@toto.ca> - 2013-02-24 18:33 -0800
    Re: Parallel archiver version 1.2 is here... aminer <aminer@toto.ca> - 2013-02-24 18:46 -0800
    Re: Parallel archiver version 1.2 is here... aminer <aminer@toto.ca> - 2013-02-25 15:07 -0800

#3096 — Parallel archiver version 1.2 is here...

Hello,

Parallel archiver version 1.2 is here, i have added Parallel LZO
and i have added support for memory and file streams, and
you can now use it as a hashtable from the hardisk, and
i have cleaned more the source code, and i have stress
tested it more and it is now a more stable version , you can be more 
confident with it now...


Author: Amine Moulay Ramdane


Description: Parallel archiver using my Parallel Zlib and Parallel LZO..

Supported features:

- Opens and creates archives using my ParrallelZlib

- Compiles into exe - no dll/ocx required.

- 64 bit supports - lets you create archive files over 4 GB

- Now my Parallel Zlib gives 5% better performance than Pigz.

- Supports memory and file streams

- You can use it as a hashtable from the hardisk

- Fault tolerant to power failures etc..

- Supports Parallel AES encryption.

- Parallel compression and parallel decompression are extremely fast

- It supports both compression and decompression rate indicator

- You can test the integrity of your archive

- Easy object programming interface

- Full source codes available.

- Platform: Win32 ,Win64 (will be ported soon to Linux and Mac OS

Will support LZMA , Bzip compressions in the near future

Please look at test_pzlib.pas and tesT_plzo.pas demos inside the zip 
file, compile and execute it.. -

When you want to delete files inside the archive you have to call the 
DeleteFiles() method , the DeleteFiles() method will not delete the 
files, it will mark the files as deleted , when you want to delete 
completly the files , you have to call the DeletedItems() method to see 
how many files are marked deleted and after that you use the Clean() 
method to delete completly the files from the archive. I have 
implemented it like that, cause it's better in my opinion..

Other than that, you have to call the LoadIndex() just after you create 
your TPZlibArchiver object with the constructor, it's mandatory, and by 
calling the LoadIndex() method , my Parallel archiver will be fault 
tolerant to power failures etc.

Y
And my parallel archiver uses a hashtable to store the file names and 
there corresponding file positions so that you can direct access to 
files inside the archive when decompressing, and deleting etc. so it's 
very fast.

Also, i have included the FindFile component, it's very easy to use, 
here is how to use it:
ff:=TFindFile.create(nil); // the constructor
ff.recurse:=true; // recurse on directories
ff.directory:='c:\zip'; // your directory
ff.Attributes:=[ffArchive];
ff.filter:='*.*'; // your files..
ff.execute; // and execute

After that you pass the ff.files (that is a TStrings) to the AddFiles() 
method.

That's very easy.

Pealse look at the test.pas demo inside the zip file to see how to use 
my Parallel archiver.

And please don't use directly the ParalleZlib.pas that i have included 
inside the Parallel archiver zip file, cause i have modified it to work 
correclty with my Parallel archiver.

If you want to use my ParallelZlib library just download it from my 
website, or download my other Parallel compression library.

In the near future i will add LZMA and BZip compressions and 
decompressions, and i will port it to Linux and Mac OSX(x86).
You can now use my Parallel archiver as a hashtable from the hardisk 
with 0(1) access, you can for example
stream your database row with my ParallelVarFiler into a memory stream 
or into a string, and store it with my Parallel archiver into an 
archive, and after that your can access your rows into the hardisk as a 
hashtable with O(1) access, you can use it like that as a database if 
you you have id keys that you want to map to database rows, that will be 
a good idea to
use my Parallel archiver as a hashtable. I will add the Distributed 
Reader-Writer Mutex 1.13 that
i have implemented to my Parallel archiver soon, so that you can 
parallelize the read access.
Question:

What's your newest ideas behind your parallel archiver ?

Answer:

Of course my Parallel Archiver does support Parallel commpression etc. 
but my newest ideas behind my Parallel Archiver are this:

I have played with Winzip and 7Zip , but if you want to give some files 
to extract or to test there integrity, they both (Winzip and 7Zip) will 
use sequential access and that's bad i think, so i have decided to 
implement a O(1) access that is very fast for extraction and and for 
testing the integrity etc. into my Parallel Archiver and for that i have 
used an in-memory hashtable that maintains the files names and there 
correponding file positions , and my second idea is that my Parallel 
Archiver is fault tolerant to for exemple power failures and also if 
your hardisk is full and you get file corruption etc. so my Parallel 
Archiver is fault tolerant to this kind of problems , 7Zip and Winzip i 
think are not fault tolerant to those kind of problems, and of course 
you will also soon use my Parallel archiver as a hashtable from the 
hardisk.
I have just played with 7Zip , and i have compressed 3 files into the 
archive and after than i have opened the archive with an editor and i 
have deleted some bytes and i have saved the file and after that when i 
have tried to open the archive, 7zip responded that the file is 
corrupted, so 7Zip is not fault tolerant, i think that with WinZip it's 
the same, but i have done the same test with  my Parallel archiver, and 
it's recovering from the file damage, so it's fault tolerant to this 
kind of damages, such as power failures and when also the disk is full 
and you get a file corruption etc. I have implemented this kind of 
fault tolerancy
into my Parallel archiver.

Hope you will enjoy my Parallel archiver, and i will soon extend it more.


You can download my Parallel archiver version 1.2 from:

http://pages.videotron.com/aminer/


Here is the public methods that i have implemented:
Constructor Create(file1:string,size:integer;nbrprocs:integer);
- Creates a new PZArchiver ready to use, size is the hashtable size for 
the index(Key file names and the corresponding file position ,and file1 
is the file archive, nbrprocs is the number of cores you have specify to 
run ParallelZlib in parallel.

Destructor Destroy;
- Destroys the PZArchiver object and cleans up.

function AddFiles;
- Add the TStringList content to the archive.

function AddStream;
-Add the stream to the archive.

function DeleteFiles;
- Delete the TStringList content from the archive.

function ExtractFiles;
- Extract the TStringList content from the archive.

function ExtractAll;
- Extract all the files from the archive.

function Extract;
-Extract the file to the stream.

function Test;
- Test the integrity of the files inside the archive.

function GetInfo;
- Get the file info that is returned in a TPZArchiver record.

function ClearFile;
- Deletes all contents of the archive.

function Clean:boolean
- Clean the marked deleted items from the file.

function DeletedItems:integer
- Return the number of items marked deleted.

function LoadIndex:boolean
- Load the the fils names key and there correponding file positions 
values from the file passed to the constructor, into the hashtable.

function Exists(Name : String) : Boolean;
- Returns True if a file Name exists

procedure GetKeys(Strings : Tstrings);
- Fills up a TStrings descendant with all the file names.

function Count : Integer;
- Returns the number of files inside the archive.


PUBLIC PROPERTIES:

Items : Variant
- Get the value (indexed).
Indicator : boolean
- To show the compression and decompression indicator.
CompressionLevel;
- Set and read the compression level.


Language: FPC Pascal v2.2.0+ / Delphi 7+: http://www.freepascal.org/

Operating Systems: Win32 and Win64 (will be ported soon to Linux and Mac 
(x86)).

And inside defines.inc you can use the following defines:

{$DEFINE CPU32} and {$DEFINE Win32} for 32 bits systems
{$DEFINE CPU64} and {$DEFINE Win64} for 64 bits systems

Required FPC switches: -O3 -Sd -dFPC -dWin32 -dFreePascal

-Sd for delphi mode....

Required Delphi switches: -DMSWINDOWS -$H+ -DDelphi




Thank you,
Amine Moulay Ramdane.

[toc] | [next] | [standalone]

#3097

Hello,

I have thought more about my Parallel archiver, and i have
come the conclusion that i must add an overloaded test()
method, so now you can pass to the test() method two TStringList , the 
first StringList contains the files to test, and the
second TStringList will contains the returned files
that have failed the integrity test, if the integrity test
of one or more files have failed the returned boolean value
of the test() method will be equal to false.

And i have added an AddStream(), to add a memory or file stream, the 
first parameter to pass to AddStream() is the archive
file name where the stream will be saved, and the second parameter is 
the memory or file stream.

And i have added an Extract() method , its first parameter
is the archive file name , and its second parameter is the memory
or file stream where the archive file will be copied.

And now you can use my Parallel archiver as a hashtable
from the harddisk also, with O(1) very fast access.

Also i have added Parallel LZO compression so that
you can compress very fast bigger files such us terabytes files.

I think that my Parallel archiver is now more professional
and more stable , and it's very fast. Enjoy it.


Next step is to add LZMA and Bzip compressions and to port
it to Linux(x86) and MacOSX(x86) and to add self extract for
windows 32 bit and 64 bit.


That's all.

Thank you,
Amine Moulay Ramdane.

[toc] | [prev] | [next] | [standalone]

#3098

Hello,


Parallel archiver is  now at 5000 lines of code, it took me
3 days to code it and to get it to a stable version, it is
now more professional and very fast, and with a wonderful and very easy 
programming interface. I will soon extend it more, and
we will perhaps implement with it a graphical Parallel archiver that 
works on Windows, Linux(x86) , Mac OSX(x86). Enjoy it.



Thank you,
Amine Moulay Ramdane.

[toc] | [prev] | [next] | [standalone]

#3099

Hello again,

I didn't implement an  Update() method, if you want to update
a file inside the archive, you have to use the Exists() method
to see if the file already exist and if it already exist you have to
mark the file deleted  with DeleteFiles() or delete it completly
after that with the Clean() method, and after that you have to  add the 
new file to the archive, this is how you can
do an Update(), so no need for an Update() method i think.



Thank you,
Amine Moulay Ramdane.

[toc] | [prev] | [next] | [standalone]

#3100

Hello,

Sorry, i have to be smarter than that, i think i have to implement
an Update() method using the methods that i have showed you
inside a critical section , so that you can use it in a multithreaded 
way, the critical section is mandatory, so i will
add the update() method soon.


Thank you,
Amine Moulay Ramdane

[toc] | [prev] | [next] | [standalone]

#3101

Hello,

The Update() method is good to have, so that you can have
all the methods to use my Parallel archiver as a hashtable.


So tu use it as a hashtable you have: the DeleteFiles() (with
O(1) access)  and Clean() methods to delete the Data from the archive, 
and the Exists() method with O(1) access to see if a file exist inside 
the archive, and the AddStream() to add the Data
to the archive and the Extract() and ExtractFiles() with O(1) access to 
extract the Data from the archive, so you can use it like that as a 
hashtable with O(1) very fast acces, that's a good idea to use
my Parallel archiver as a hashtable from the harddisk.


So i will add soon the Update() method.


Thank you,
Amine Moulay Ramdane.

[toc] | [prev] | [next] | [standalone]

#3108

Hello,

But be warned, you can not pass a directory such us:

c:\amine\..\

the '..' must be at the beginning of directory path, like:

'..\' or '..\..\' etc. and also you can pass '.\' that means the current 
directory.



Thank you,
Amine Moulay Ramdane.

[toc] | [prev] | [standalone]

Back to top | Article view | comp.programming

csiph-web