Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.sys.apple2.programmer > #1183 > unrolled thread

Archiver Project for the 8 Bit Apple II - Request for Comments

Back to article view | Back to comp.sys.apple2.programmer

Contents

  Archiver Project for the 8 Bit Apple II - Request for Comments "Bill Buckels" <bbuckels@mts.net> - 2014-03-06 14:54 -0600
    Archiver Project for the 8 Bit Apple II - Request for Comments Antoine Vignau <antoine.vignau@laposte.net> - 2014-03-06 23:30 -0800
      Re: Archiver Project for the 8 Bit Apple II - Request for Comments "Bill Buckels" <bbuckels@mts.net> - 2014-03-07 07:20 -0600
        Re: Archiver Project for the 8 Bit Apple II - Request for Comments gids.rs@sasktel.net - 2014-03-07 09:35 -0800
          Re: Archiver Project for the 8 Bit Apple II - Request for Comments "Bill Buckels" <bbuckels@mts.net> - 2014-03-07 17:31 -0600
            Re: Archiver Project for the 8 Bit Apple II - Request for Comments gids.rs@sasktel.net - 2014-03-07 16:58 -0800
    Re: Archiver Project for the 8 Bit Apple II - Request for Comments Jurai <Jurai@WhereTheHellAmI.???> - 2014-03-08 09:10 -0700
      Re: Archiver Project for the 8 Bit Apple II - Request for Comments "Bill Buckels" <bbuckels@mts.net> - 2014-03-08 20:32 -0600
        Re: Archiver Project for the 8 Bit Apple II - Request for Comments Dagen Brock <DagenBrock@gmail.com> - 2014-03-22 22:27 -0700
          Re: Archiver Project for the 8 Bit Apple II - Request for Comments "Bill Buckels" <bbuckels@mts.net> - 2014-03-23 04:28 -0500

#1183 — Archiver Project for the 8 Bit Apple II - Request for Comments

Dear All:)

I am in the begiining of the process of writing an archiver that will run in 
Windows, Unix-like systems, and anywhere that Ansi C can be used, and a 
second archiver which run on the Apple II as an Aztec C65 SYS program. These 
programs will be called HUF2 and will create archives called "HUF Balls" 
with the extension of .HUF. The Win32 gcc MinGW executable version will be 
called Huf2.exe and the MS-DOS version will be called Huf216.exe, and the 
MicroSoft C Win32 version will be Huf232.exe. I don't care what the other 
versions will be called... likely Huf2 would be best though. The Aztec C65 
Prodos version willl be called Huf2.SYS and the DOS 3.3 version will be 
Huf2.BIN. The Aztec C65 Shell version for ProDOS will be called Huf2. If I 
produce an Aztec C65 DOS 3.3 shell version it will also be called Huf2.

Distribution will be on disk images for the Apple II, and the whole works 
including docs and source code will be distributed in a ZIP file (not a HUF2 
file) and distributed from the Apple Oldies Website.

Here is the idea. Years ago I took one of Yoshi's many sample archivers 
which used Adaptive Huffman encoding and created yet another MS-DOS archiver 
Called "ARBUCKLE".
Compression was pretty good, but I did not provide support for 
subirectories. I kept it simple. The file extension was ARB. I never 
distributed this program because I did not think that the world needed yet 
another archiver. I wrote it for my own enjoyment. I even provided a 
self-extracting archive scheme.

For the Apple II version of my new archiver, I want to keep it simple. I 
want to be able to use auxiliary memory on the Apple II as working space. 
And I want to be able to append (add) or update files in an existing 
archive, and to delete files from an existing archive.

I want this available in 2 versions:

1. DOS 3.3 FileTypes
2. PRODOS FileTypes

I want to use CiderPress File Attribute Preservation tags to store the files 
and provide auto-translation between filetypes in ProDOS or DOS 3.3 
including name length translation.

Anyone who has ever written an archiver of non-trivial nature will 
immediately understand that on today's computers, memory, disk space, and 
processing speed are not an issue. Anyone that understands the Apple II also 
understands that all 3 of memory, disk space, and processing speed are in 
fact issues.

Some assembly language will be required to get the kind of performance out 
of Aztec C65 that I need to make this work.

The Ansi Version of this archiver has no such contraints, but will require 
file attribute preservation tags to be used for specific file type names 
when archiving from the host system, be it Windows, or the Mac, or Linux, 
etc.

I will also use CiderPress logic when file attribute preservation tags are 
not used; they will be stored as a ProDOS NON Type with applicable 
translation when dearchiving on the Apple II.

In this case my many Apple II Utilities which change FileTypes in ProDOS and 
DOS 3.3 can be used, as well as text translation utilities which I have 
written of late.

Nothing will persuade me to support directories.

I think that about covers what I am going to do, except to say that I 
envision this utility being used for 2 specific purposes by some:

1. To pack files together in a HUF Ball to save Apple II disk space and 
increase transfer speed for upload or downloads between Apple II and other 
computers.

2. For other purposes that I haven't thought of.

Some others will ignore it, or learn from it, or crticize it.

So for the first part of my design phase, which is being inspired by an 
email from Rob, I will leave this open for comments. As always I will 
follow-through with the program, so this is not something that I am wishing 
for. I just want to hear what the guys have to say about it.

I'll lay-out exactly everything in the documentation the way I usually do 
and the code will be fully commented and then some.

I also wrote an image converter for MS-DOS back in those days that provided 
compression aproximately equal to GIF. So if you are looking for some point 
of reference for compression ratio, that's about it.

In my opinion the ZIP format isn't well suited to my project nor LZW. So 
I'll leave that Troll remark out there to see what get's said about that 
too.

I do want to thank Rob for the idea. I also want to scold Rob for the idea; 
I swore almost 30 years ago that I would do nothing further with archivers 
when I wrote ARBUCKLE. I also swore I would do nothing about the image 
format that I developed around Adaptive Huffman. But it just so happens that 
his suggestion seems to have trumped my resolution, which is hardly 
surprising considering that my new agenda is retro-development which 30 
years ago I never imagined. It is better than beer, and better for my health 
as well, so I may as well thank Rob twice.

Besides, he is a fellow countryman, and deserves a reward for surviving in 
the same snowbelt that I live in. We're tougher than a bunch of southerners, 
and better looking too.

All the Best,

Bill

[toc] | [next] | [standalone]

#1185

Good luck, Bill.
will you support Prodos extended files (those with a resource fork)?

av

[toc] | [prev] | [next] | [standalone]

#1187

"Antoine Vignau" <antoine.vignau@laposte.net> wrote:
>will you support Prodos extended files (those with a resource fork)?
Hi Antoine,

If I do an Orca C Version,  it may; one way at least. But I can't think of a 
single use case where files coming from the Apple //e and ProDOS 8 or DOS 
3.3 would have resource forks, nor where files going to the Apple //e would 
need them. So doing so would be scope creep at this time.

This is not an installer nor a conversion utility. Nor is it meant to 
replace NuLib:

http://www.nulib.com/

or something else Andy has done:

http://www.fadden.com/dl-apple2/index.htm

And like the years it took to develop Convert3200, it would take years of 
dedicated development to build a competitive product to NuLib. The 
programmer would need to be a gifted expert; not your average Apple II 
enthisiast.

The primary goal behind the resurrection of Aztec C was always to simulate 
the perspective of the MS-DOS programmer of the 1980's building programs for 
MS-DOS, CP/M, the Commodore 64 and the Apple //e. Native mode compilers like 
Aztec C68000 for the Amiga are interesting for those whose history is on 
those platforms, and for purists who wish to experience native mode 
development rather than cross-development, or for me in particular as a 
collector.

The primary target user for Aztec C is the Windows User of today, but with 
the ready availability of MS-DOS emulators like DOSBox (and DOSemu on 
Ubuntu) users of other less popular desktops can also have access to the 
Aztec C MS-DOS cross-development retro-experience.

As a work in progress to continue to pursue this primary goal of targeting 
the Windows retro-computing enthusiast who specifically wants to work with 
the Apple //e, I continue to develop and provide 8 bit 6502 demo and example 
programs and code for ProDOS 8 and DOS 3.3 that the average enthusiast can 
easily read, build, and expand on. And documentation. The converters and 
utilities that I write are fundamentally targeted at this living museum 
concept, and obviously they also can be useful on other platforms, or in 
their own right.

Aztec C65 was no longer being developed when the GS was introduced. The 
decline and extinction of Aztec C (and many other C compilers of the 1980's) 
overlapped the release of the Apple IIgs (and the VOC). My work with the 
Super Hi-Res Demos and utilities is in support of that part of history, and 
of course the current history of the Carte-Blanche and the Kegs32 emulator.

I regard each utility and project, program, or demo that I write in Aztec C 
to be what that programmer could have done relatively easily then with the 
tools we Windows users have now, but I shape the scope of this somewhat 
realistically... I don't think resource forks would have been something that 
an MS-DOS programmer working in the Aztec C65 cross-compiler would have 
attempted. Not if he had any brains to put it bluntly.

By the time the GS came along, and ByteWorks Orca Development environment 
became available, the MS-DOS Programmer could not have possibly competently 
developed anything really robust for the GS in Aztec C65; in my view the GS 
supporting ProDOS 8 and DOS 3.3 is like Windows XP supporting MS-DOS 
applications... arguably both systems are emulators for legacy OS's.

A word about Orca C; this is a "next-generation" compiler, and really a 
modern compiler more than a retro-compiler. But was never available in a 
cross-development version. By that time, or shortly thereafter, MS-DOS was 
also dead, 16-bit Windows began to dominate as the World's Most Popular 
Desktop with 90% of Market Share by the mid-90's. A guy was either a Windows 
developer or an MS-DOS developer. Cross-Development shortly reached a very 
specialized level... not for the average developer. Pakages like XVT Design 
and ProGraph appeared. It was a full-time job these GUI's!

The short answer is that memory is extremely tight on the Apple //e so 
additional code not needed for proper archiving of 8 bit ProDOS files and 
DOS 3.3 files will not be included in this project.

Go back to the primary specific purpose of this archiver itself:

1. To pack files together in a HUF Ball to save Apple II disk space and 
increase transfer speed for upload or downloads between Apple II and other 
computers.

The archiver code is derived from the C source for lzhuf compression, by 
Haruyasu Yoshizaki

LZHUF, the algorithm of Haruyasu Yoshizaki's archiver LHarc, replaced 
LZARI's adaptive arithmetic coding with adaptive Huffman.  LZHUF encodes the 
most significant 6 bits of the position in its 4096-byte buffer by table 
lookup.  More recent, and hence more probable, positions are coded in less 
bits.  On the other hand, the remaining 6 bits are sent verbatim.  Because 
Huffman coding encodes each letter into a fixed number of bits, table lookup 
can be easily implemented.

Though theoretically Huffman cannot exceed arithmetic compression, the 
difference is very slight, and LZHUF is fairly fast.

References
  [1] J. Ziv and A. Lempel, IEEE Trans. IT-23, 337-343 (1977).
  [2] J. A. Storer and T. G. Szymanski, J. ACM, 29, 928-951
      (1982).
  [3] T. C. Bell, IEEE Trans. COM-34, 1176-1182 (1986).
  [4] J. Ziv and A. Lempel, IEEE Trans. IT-24, 530-536 (1978).
  [5] T. A. Welch, Computer, 17, No.6, 8-19 (1984).
  [6] J. A. Storer, Data Compression: Methods and Theory
      (Computer Science Press, 1988).
  [7] D. A. Huffman, Proc IRE 40, 1098-1101 (1952).
  [8] R. Sedgewick, Algorithms, 2nd ed. (Addison-Wesley, 1988).
  [9] R. G. Gallager, IEEE Trans. IT-24, 668-674 (1978).
 [10] I. E. Witten, R. M. Neal, and J. G. Cleary, Commun. ACM
      30, 520-540 (1987).

What I have to start with is a fully functional MS-DOS archiver based on 
Yoshi's archiver code noted above with a small memory footprint. Since I did 
the original code I can port it with certainty.

The work buffers will fit into auxiliary memory. The trick will be to page 
these in and out as needed.

I may have to split the Apple II Sys Program into UnHuf2.sys and Huf2.sys to 
get the code to fit with with the buffers. It is tight. The proof of concept 
will compress and decompress one file only, and will be written for the 
Apple II only. It is easy to port from the Apple II after porting from 
MS-DOS to the Apple II. It is also easy to expand functionality after 
getting the basic archiving routines in place.

Simply finishing this, documenting it, and distributing it will be enough 
for me to handle in my spare time, and so for that reason alone, as well as 
the other reasons noted above, GS resource forks will not be supported in 
this project's scope, nor will creating Microsoft Windows File Associations 
or anything of a GUI nature.

There are 3 tiers in this development:

1. The System Layer
2. The Data Layer
3. The UI Layer

The system layer is ProDOS 8 or DOS 3.3 or Win32 or MS-DOS, or other systems 
supported by gcc.

The data layer is not object oriented. It is data only and does not support 
links like resource forks or unix linked files either. The operating system 
takes care of that, and if not it ain't there, nor should behaviour beyond 
what Ansi C provides be supported, because it is non-portable. Data is Data 
and it is binary, period.

File attribute preservation tags in systems that support long filenames are 
portable. My utility CiderTag does support tagging files with File Attribute 
Presevation Tags, so can be used on the Windows or Unix-like command line to 
prepare files for archiving, but this is beyond the scope of this archiver.

There is no GUI in the UI. Two-tiered applications that link to a GUI 
through an API are non-portable. This is a command line utility or 
equivalent only. If the ProDOS 8 or DOS 3.3 versions have a UI, it will be 
because there is no equivalent command-line for these systems.

Those are some of my thoughts. Please take me to task on any of this if you 
wish... especially you Mr. T.

All the Best,

Bill

[toc] | [prev] | [next] | [standalone]

#1188

On Friday, 7 March 2014 07:20:10 UTC-6, Bill Buckels  wrote:
> "Antoine Vignau" wrote:
> 
> >will you support Prodos extended files (those with a resource fork)?

> Hi Antoine,

> If I do an Orca C Version,  it may; one way at least. But I can't think of a 
> single use case where files coming from the Apple //e and ProDOS 8 or DOS 
> 3.3 would have resource forks, nor where files going to the Apple //e would 
> need them. So doing so would be scope creep at this time.


I have one.  There are a lot of $50 TextEdit text files that are forked.  Some are not.  But to preserve them if they are moved to a disk that is mainly used on a IIe/IIc, it would be great to have these compressed.

I have written a text file reader that can display text from forked files from an 8-bit machine.  The text should be without formatting for now.

My idea would be to keep the text file compressed and use the reader to uncompress the text file to extended RAM to view it.  This will make it unnecessary to keep an uncompressed version of the text file on the same disk.

Rob

[toc] | [prev] | [next] | [standalone]

#1189

<gids.rs@sasktel.net> wrote:
>There are a lot of $50 TextEdit text files that are forked.  Some are not. 
>But to preserve them if they are moved to a disk that is mainly used on a 
>IIe/IIc, it would be great to have these compressed.

So support for resource forks via a HUF2.cfg file which will contain Type, 
Auxtype in the [fork] section. I don't see TextEdit files here:

http://www.kreativekorp.com/miscpages/a2info/filetypes.shtml

>I have written a text file reader that can display text from forked files 
>from an 8-bit machine.  The text should be without formatting for now.

>My idea would be to keep the text file compressed and use the reader to 
>uncompress the text file to extended RAM to view it.  This will make it 
>unnecessary to keep an uncompressed version of the text file on the same 
>disk.

So after I get the Adaptive Huffman routines working on the Apple II I'll 
send you the code including aztec C's assembly pass for the compress and 
uncompress routines for your review.

Bill 

[toc] | [prev] | [next] | [standalone]

#1190

On Friday, 7 March 2014 17:31:20 UTC-6, Bill Buckels  wrote:

> >There are a lot of $50 TextEdit text files that are forked.  Some are not. 
> >But to preserve them if they are moved to a disk that is mainly used on a 
> >IIe/IIc, it would be great to have these compressed.

> So support for resource forks via a HUF2.cfg file which will contain Type, 
> Auxtype in the [fork] section. I don't see TextEdit files here:



I get mixed up with the Mac version of a good text editor.

Teach text it is.

[toc] | [prev] | [next] | [standalone]

#1191

On Thu, 6 Mar 2014 14:54:37 -0600, "Bill Buckels" <bbuckels@mts.net>
wrote: 

I have to correct you on 1 point: The only 2 things better than beer
are sex, and MORE beer! :)
					--Jurai


<SNIP>
>years ago I never imagined. It is better than beer, and better for my health 
>as well, so I may as well thank Rob twice.
<SNIP>

[toc] | [prev] | [next] | [standalone]

#1192

"Jurai" <Jurai@WhereTheHellAmI.???> wrote:
>I have to correct you on 1 point: The only 2 things better than beer are 
>sex, and MORE beer! :)

Jurai, I stand corrected. I was under the impression that pot and sex were 
both better than beer, and both together were better than either alone...

Bill

[toc] | [prev] | [next] | [standalone]

#1196

On Saturday, March 8, 2014 8:32:53 PM UTC-6, Bill Buckels wrote:

> Jurai, I stand corrected. I was under the impression that pot and sex were 
> both better than beer, and both together were better than either alone...


As with all things Bill Buckels, that escalated quickly.  ;)

[toc] | [prev] | [next] | [standalone]

#1197

"Dagen Brock" <DagenBrock@gmail.com> wrote:
>As with all things Bill Buckels, that escalated quickly.  ;)

Indeed!

However, the proposed archiver is still in the beer fridge. I can't seem to 
quit having sex with SHR conversion algorithms. Even the smoke of the Aztec 
C65 website is being neglected.

Bill

[toc] | [prev] | [standalone]

Back to top | Article view | comp.sys.apple2.programmer

csiph-web