Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > alt.sys.pdp10 > #9969 > unrolled thread

boxing all integers (was Re: Resources to learn common lisp?)

Back to article view | Back to alt.sys.pdp10

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  boxing all integers (was Re: Resources to learn common lisp?) Kragen Javier Sitaker <kragen@canonical.org> - 2026-08-29 17:52 -0300
    Re: boxing all integers (was Re: Resources to learn common lisp?) Lars Brinkhoff <lars.spam@nocrew.org> - 2026-08-30 06:43 +0000
      Re: reprints of old AI memos (was Re: boxing all integers)f Kragen Javier Sitaker <kragen@canonical.org> - 2026-08-30 13:11 -0300
        Re: reprints of old AI memos (was Re: boxing all integers)f Lars Brinkhoff <lars.spam@nocrew.org> - 2026-08-31 05:14 +0000
          Re: reprints of old AI memos (was Re: boxing all integers) Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-08-31 06:54 +0000
          Re: reprints of old AI memos (was Re: boxing all integers)f Kragen Javier Sitaker <kragen@canonical.org> - 2026-08-31 13:43 -0300
            Re: reprints of old AI memos (was Re: boxing all integers)f Kragen Javier Sitaker <kragen@canonical.org> - 2026-08-31 13:56 -0300
              Re: reprints of old AI memos (was Re: boxing all integers)f Paul Rubin <no.email@nospam.invalid> - 2026-08-31 13:34 -0700
                Re: reprints of old AI memos (was Re: boxing all integers)f Rich Alderson <news@alderson.users.panix.com> - 2026-08-31 18:12 -0400
                  Re: reprints of old AI memos (was Re: boxing all integers)f scott@slp53.sl.home (Scott Lurndal) - 2026-09-01 15:12 +0000
              Re: reprints of old AI memos (was Re: boxing all integers)f Lars Brinkhoff <lars.spam@nocrew.org> - 2026-09-01 05:59 +0000
                Re: reprints of old AI memos (was Re: boxing all integers)f Kragen Javier Sitaker <kragen@canonical.org> - 2026-09-02 15:23 -0300
            Re: reprints of old AI memos (was Re: boxing all integers) Lars Brinkhoff <lars.spam@nocrew.org> - 2026-09-01 05:45 +0000
              Re: reprints of old AI memos (was Re: boxing all integers) Lars Brinkhoff <lars.spam@nocrew.org> - 2026-09-01 07:07 +0000
              Re: reprints of old AI memos (was Re: boxing all integers) Rich Alderson <news@alderson.users.panix.com> - 2026-09-01 17:36 -0400
                the history of the Hershey fonts (was Re: reprints of old AI memos) Kragen Javier Sitaker <kragen@canonical.org> - 2026-09-02 15:55 -0300
                  Re: the history of the Hershey fonts (was Re: reprints of old AI memos) Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-09-02 21:31 +0000
                    Re: the history of the Hershey fonts (was Re: reprints of old AI memos) Kragen Javier Sitaker <kragen@canonical.org> - 2026-09-03 09:59 -0300
              Re: reprints of old AI memos (was Re: boxing all integers) Kragen Javier Sitaker <kragen@canonical.org> - 2026-09-02 16:35 -0300
                Re: reprints of old AI memos (was Re: boxing all integers) Lawrence D’Oliveiro <ldo@nz.invalid> - 2026-09-02 21:50 +0000
                  PDF file compactness (was Re: reprints of old AI memos) Kragen Javier Sitaker <kragen@canonical.org> - 2026-09-03 12:06 -0300
                Re: reprints of old AI memos (was Re: boxing all integers) Lars Brinkhoff <lars.spam@nocrew.org> - 2026-09-03 05:42 +0000
                  Re: reprints of old AI memos (was Re: boxing all integers) Kragen Javier Sitaker <kragen@canonical.org> - 2026-09-03 12:50 -0300
              origin of XGP fonts (was Re: reprints of old AI memos) Kragen Javier Sitaker <kragen@canonical.org> - 2026-09-28 10:44 -0300
                Re: origin of XGP fonts (was Re: reprints of old AI memos) Peter Flass <Peter@Iron-Spring.com> - 2026-09-28 07:33 -0700
                  Re: origin of XGP fonts (was Re: reprints of old AI memos) Johann 'Myrkraverk' Oskarsson <johann@myrkraverk.invalid> - 2026-09-28 22:53 +0800
        Re: reprints of old AI memos (was Re: boxing all integers)f Lars Brinkhoff <lars.spam@nocrew.org> - 2026-09-06 08:40 +0000

Page 2 of 2 — ← Prev page 1 [2]

#9994 — PDF file compactness (was Re: reprints of old AI memos)

Lawrence D’Oliveiro <ldo@nz.invalid> writes:
> On Wed, 02 Sep 2026 16:35:26 -0300, Kragen Javier Sitaker wrote:
>> Contrary to popular belief, PDF is a relatively compact file format.
>
> Only if you apply compression to it.

This is ambiguous; you might be intending to say, “only if
you use the PDF format’s compression features” or, “only if
you compress the PDF file with an additional compressor
after creating it.”

I disagree with both of these interpretations.

For text, PDF can be a relatively compact file format even
if you do neither of these.  There’s some file-format
overhead of about a kilobyte — you need a header, a
catalog, a page tree, an xrefs table, and trailer, even for
a one-page document — and then you need to specify the font
and the coordinates where your text starts.  After that, I
think the per-line overhead is about 4 bytes per line.  I
haven’t tested this content-stream example, but if it’s
missing something, it’s not missing much:

    BT
    /F0 12 Tf  50 706 Td
    (For text, PDF is a relatively compact file format even if) '
    (you do neither of these.  There's file format overhead of) '
    (about a kilobyte - you need a page tree and xrefs table even) '
    (for a one-page document - and then you need to specify the) '
    (font and the coordinates where your text starts. After) '
    (that, the per-line overhead is a few bytes per line. I) '
    (haven't tested this content-stream example, but if it's) '
    (missing something, it's not missing much:) '
    ET

The `'` PDF content-stream operator is just `T* TJ`, if
you’re familiar with those.  There’s also a `"` shortcut
operator that lets you set a different letter spacing and
word spacing for each line:

    1 1.4 (for a one-page document - and then you need to specify the) "

That costs you about 11 bytes per line instead of 4.

So, a simply formatted PDF file without using any kind of
compression is about 5% bigger than a plain ASCII text file,
plus about one kilobyte.

Perhaps you don't consider 5% overhead to be “relatively
compact”, but I do.

Also, of course, PDF *does* support Deflate compression for
content streams; and, since PDF 1.5, it also supports it for
xrefs and object streams, so a PDF file can easily be half
the size of a plain ASCII text file, down to a minimal size
of a few hundred bytes.  Amusingly, I’ve even seen PDF files
that apply Paeth compression to the xrefs table.

Now, in practice, PDF files are often not this compact.  A
much more typical example of a PDF content-stream (from the
Derctuo PDF: <http://canonical.org/~kragen/derctuo/>) looks
like this (slightly reformatted):

    1 0 0 1 0 0 cm  BT /F1 12 Tf 14.4 TL ET
    0 0 0 rg
    0 0 0 rg
    BT 1 0 0 1 6 747.6 Tm .533333 0 0 rg
    /F2+0 24 Tf 28.8 TL (Derctuo) Tj  T* ET
    0 0 0 rg
    BT 1 0 0 1 6 714.84 Tm /F2+0 12 Tf 14.4 TL (    ) Tj  T* ET
    0 0 0 rg
    BT 1 0 0 1 6 700.44 Tm /F3+0 12 Tf 14.4 TL ( ) Tj
    /F4+0 12 Tf 14.4 TL (\200\201\201\200) Tj  T* ET
    0 0 0 rg
    BT 1 0 0 1 6 686.04 Tm /F3+0 12 Tf 14.4 TL ( ) Tj
    (Kragen ) Tj (Javier ) Tj (Sitaker) Tj  T* ET
    0 0 0 rg
    BT 1 0 0 1 6 671.64 Tm /F3+0 12 Tf 14.4 TL ( ) Tj
    (Buenos ) Tj (Aires) Tj  T* ET
    0 0 0 rg
    BT 1 0 0 1 6 657.24 Tm /F3+0 12 Tf 14.4 TL ( ) Tj
    (December, ) Tj (02020) Tj  T* ET
    0 0 0 rg
    BT 1 0 0 1 6 642.84 Tm /F3+0 12 Tf 14.4 TL ( ) Tj
    (Public ) Tj (domain ) Tj (work) Tj  T* ET
    0 0 0 rg
    BT 1 0 0 1 6 628.44 Tm /F3+0 12 Tf 14.4 TL ( ) Tj
    /F4+0 12 Tf 14.4 TL (\200\201\201\200) Tj  T* ET
    0 0 0 rg
    0 0 0 rg
    BT 1 0 0 1 6 599.64 Tm /F2+0 12 Tf 14.4 TL ( ) Tj  T* ET
    0 0 0 rg
    BT 1 0 0 1 6 581.28 Tm /F2+0 12 Tf 14.4 TL (    ) Tj
    (Derctuo ) Tj (is ) Tj (a ) Tj (book ) Tj (of ) Tj
    (notes ) Tj (on ) Tj (various ) Tj (topics, )
    Tj (mostly ) Tj (science ) Tj (and ) Tj  T* ET

It’s relatively straightforward to uncompress
content-streams like this from PDF files in Python:

    zlib.decompress(base64.a85decode(a8.removesuffix(b'~>'))
    ).decode('utf-8')

This is obviously inefficient in many different ways:

- Reportlab decided to Ascii85Decode the compressed data for
  no real reason, even though I specified pageCompression=True.
- There’s no need to Tj each word separately.  You could Tj
  the whole line of text.  I think this was my fault; I
  hacked together this PDF renderer in a week for a deadline.
- It’s unnecessary to set the transformation matrix (cm) to
  the identity matrix.  That’s the default.
- Similarly, setting the RGB color to black for each line
  (and twice for the first line) is unnecessary.  Black is
  the default.  I think this is ReportLab’s fault.
- In the one case where the color is set to a non-default
  color, it’s unnecessary to specify that color to six
  significant figures.
- Displaying runs of spaces is generally unnecessary.
- Changing fonts to display runs of spaces is extra
  unnecessary.
- Changing fonts twice per line is unnecessary.  Most of
  this text is in a single font.
- Chanting to the same font again is also unnecessary.
- Creating a new text object for every line (BT ET) is
  unnecessary and also counterproductive for copy-and-paste.

Despite all this, the 47 lines of text on the page are 8202
bytes of uncompressed content-stream; FlateDecode encoded
and Ascii85Decode encoded, they pack down to only 2471
bytes, plus 149 bytes of per-stream overhead (also mostly
unnecessary), plus 309 bytes of the Page object containing
the content stream (also mostly unnecessary), for a total of
3K per page, which is slightly more compact than plain ASCII
text.  (This doesn’t count the hyperlinks on the page,
though.)

It’s easy to see how small inefficiencies like these can
pile up when people (like me) who don’t really understand
what they’re doing get things to more or less work, and then
stop.  And that seems to be how most PDF files are built.
Most of them are even worse than the Derctuo PDF.

Derctuo is far from exemplary, but the PDF is 986 pages and
5.91 megabytes, roughly 5.9KiB per page; it divides up as
follows:

- bytes 569 to 2.23e6: intermixed page objects and link objects
- bytes 2.23e6 to 2.50e6: embedded fonts, covering ASCII and
  a bunch of Unicode for things like math and Greek, in
  eight display styles
- bytes 2.50e6 to 2.62e6: more document structure, including
  outline and page tree
- bytes 2.62e6 to 5.74e6: page content streams
- bytes 5.74e6 to 5.91e6: xrefs and trailer

Due to the inept content-stream structure I demonstrated
above, if the page content streams were uncompressed, they
would be about 3.3× as large, going from about 3.12
megabytes to 10.3 megabytes.  This would inflate the Derctuo
PDF from 5.9 megabytes to 13.1 megabytes, which works out to
about 13KiB per page.

This is about three times bigger than plain ASCII text, but
that’s only because of how badly I screwed the pooch in
building the content streams.

I was mostly using Edward Tufte’s “ET Book” TrueType version
of Bembo, falling back to DejaVu Serif fonts for non-ASCII
characters, and using Latin Modern Mono Light Condensed (a
modified Computer Modern Typewriter) for typewriter text,
falling back to FreeMono and DejaVu Sans Mono fonts.

Embedding eight typefaces thus cost me 270K.  If you want a
PDF document to be much under 100K, you more or less have to
restrict yourself to the 14 core PDF fonts instead of
embedding your own, or hope that the fonts you want to use
happen to be installed on the reader's system (prohibited in
PDF/A and, I believe, PDF 2.0).

Nearly half of the bytes in the Derctuo PDF are hyperlinks,
which mostly look like this (I’ve elided the CRs ReportLab
inserted before LFs):

    % 'Annot.NUMBER2006': class LinkAnnotation 
    2343 0 obj
    << /Border [ 0
     0
     .1 ]
     /C [ .6
     .6
     1 ]
     /Contents (notes/lithium-fuel.html)
     /Dest [ 2559 0 R
     /XYZ
     null
     null
     null ]
     /Rect [ 4.8
     595.44
     33.62578
     609.84 ]
     /Subtype /Link
     /Type /Annot >>
    endobj

2559 0 obj is the /Page object for page 371, where the note
on lithium fuel begins.  I think there are a lot of
opportunities for optimization here, including unnecessary
whitespace, and I don’t think the PDF spec *requires* an
/Annot to be a top-level object (I think you can embed it
inside the /Page object), but honestly most of those would
go away if you just used a PDF 1.5 deflated object stream.

Kragen

[toc] | [prev] | [next] | [standalone]

#9992 — Re: reprints of old AI memos (was Re: boxing all integers)

Kragen Javier Sitaker wrote:
> Without detracting from your achievement, I think it may be possible to
> build on it to get a 20× smaller file that is more readable, more
> aesthetically appealing, easier to quote from and search for text in,
> and more accessible to blind users.

Oh, certainly.  My goal was to make all the original PDP-10 and PDP-11
code run, and have it print something close to the original XGP.
Bypassing and essentially reimplementing the rendering done in the
PDP-11 would make it possible to output text instead of graphics.

> I poked around the repo a bit, but couldn't find which code you meant
> that might be rasterizing a metaball model.

Look in files called xgp, meatball, and print.  Perhaps what you're
after is in the function downsample.

> I see.  Does that mean that the “XGP file” has glyph indices but not
> (x, y) positions in it, because the PDP-11 has to produce some of
> those (x, y) positions from the font metrics?

Yes, to the first part.  Regarding what the PDP-11 does, probably.

> A smaller PDF file would either have to use Courier

I shudder at the thought.

> Small size reduces the cost of archival preservation, making it more
> likely to happen.

I think this is a strained argument to make.  My guess is no archivist
will care if a document is 1K or 1M in size.  Be that as it may, you
don't have to convince anyone, just go ahead and do it if you want to.
I'm sorry to say much of your message looks like LLM output.

[toc] | [prev] | [next] | [standalone]

#9995 — Re: reprints of old AI memos (was Re: boxing all integers)

Lars Brinkhoff <lars.spam@nocrew.org> writes:
> Kragen Javier Sitaker wrote:
>> Small size reduces the cost of archival preservation, making it more
>> likely to happen.
>
> I think this is a strained argument to make.  My guess is no archivist
> will care if a document is 1K or 1M in size.  Be that as it may, you
> don't have to convince anyone, just go ahead and do it if you want to.
> I’m sorry to say much of your message looks like LLM output.

I didn’t use any LLMs in writing the message, and I am myself fully
human; I just tried to organize it to be as clear as I could, to the
point of dividing it with headings, and I tend to use a wider range of
punctuation than most people, including em dashes.  Also, I write pretty
fast, and I don’t make a lot of spelling errors.

Unfortunately, nowadays, people sometimes mistake me for an LLM, because
they write very fast and they’ve been extensively trained on writing
like mine!  If you read my post carefully, though, you’ll see that it
makes sense and contains original thoughts, while LLMs generally don’t.

With respect to size, at some point any archival effort runs into space
limits.  You might not care if *one* document is 1K or 1M, or in this
case 100K or 1.5M; but the Internet Archive cares if a trillion
documents are 1.5 exabytes or 100 petabytes, Bitsavers cares if a
billion documents are 1.5 petabytes or 100 terabytes, garden-variety
data hoarders care if ten million documents are 15 terabytes or just 1
terabyte, and I personally care whether ten thousand documents are a
gigabyte or 15 gigabytes.  (I’m tempted to put that into a bulleted list
for clarity, but I’ll resist the temptation.)

Kragen

[toc] | [prev] | [next] | [standalone]

#10011 — origin of XGP fonts (was Re: reprints of old AI memos)

Lars Brinkhoff <lars.spam@nocrew.org> writes:
> Kragen Javier Sitaker wrote:
>> I wonder where they [the XGP fonts] came from?
>
> I read a story somwhere, but I forgot the details and where I found it.
> It was something like getting a font catalog from a major foundry and
> digitizing it by hand.
>
> People also made their own; the AST file format is just a text file with
> pixel data, and on ITS there's FED (font "editor") to display a font on
> the 340 or a Knight TV.  I think SAIL had something else, certaily so
> when TeX and Metafont were developed.
>
> "The Last Whole XGP Font Catalog" (play on "Last Whole Earth Catalog")
> has a lot to say about the XGP.

I was reading Butler Lampson’s 01988 retrospective on the Alto today
<https://dl.acm.org/doi/abs/10.1145/61975.66921> and he answered this
question:

: The first imager, written by Peter Deutsch, drove the Xerox Graphics
: Printer.  This machine is slow (five pages/minute) and low resolution
: (200 dots/inch); it is also asynchronous, so the imager can take as
: long as it likes to generate each line of the raster. Only crude
: imaging software was developed for it, together with a few fonts.
: Raster fonts were unheard of at the time, except in 5 × 7 and 7 × 9
: resolution for terminals, or at very high resolution for expensive
: photo-typesetters. The XGP fonts were developed entirely manually,
: using an editor that allows the operator to turn individual dots in
: the roughly 20 × 20 matrix on and off. They had to be new designs,
: since it is impractical to faithfully copy a printer’s font at such
: low resolution. These XGP fonts were later widely used in
: universities.


Kragen

[toc] | [prev] | [next] | [standalone]

#10012 — Re: origin of XGP fonts (was Re: reprints of old AI memos)

On 9/28/26 06:44, Kragen Javier Sitaker wrote:
> Lars Brinkhoff <lars.spam@nocrew.org> writes:
>> Kragen Javier Sitaker wrote:
>>> I wonder where they [the XGP fonts] came from?
>>
>> I read a story somwhere, but I forgot the details and where I found it.
>> It was something like getting a font catalog from a major foundry and
>> digitizing it by hand.
>>
>> People also made their own; the AST file format is just a text file with
>> pixel data, and on ITS there's FED (font "editor") to display a font on
>> the 340 or a Knight TV.  I think SAIL had something else, certaily so
>> when TeX and Metafont were developed.
>>
>> "The Last Whole XGP Font Catalog" (play on "Last Whole Earth Catalog")
>> has a lot to say about the XGP.
> 
> I was reading Butler Lampson’s 01988 retrospective on the Alto today
> <https://dl.acm.org/doi/abs/10.1145/61975.66921> and he answered this
> question:
> 
> : The first imager, written by Peter Deutsch, drove the Xerox Graphics
> : Printer.  This machine is slow (five pages/minute) and low resolution
> : (200 dots/inch); it is also asynchronous, so the imager can take as
> : long as it likes to generate each line of the raster. Only crude
> : imaging software was developed for it, together with a few fonts.
> : Raster fonts were unheard of at the time, except in 5 × 7 and 7 × 9
> : resolution for terminals, or at very high resolution for expensive
> : photo-typesetters. The XGP fonts were developed entirely manually,
> : using an editor that allows the operator to turn individual dots in
> : the roughly 20 × 20 matrix on and off. They had to be new designs,
> : since it is impractical to faithfully copy a printer’s font at such
> : low resolution. These XGP fonts were later widely used in
> : universities.
> 
> 

The SAIL fonts were, at one point, available from one of the PC 
shareware sites. I imagine they are still kicking around somewhere. I 
played with them at one time, I think the major obstacle is that they 
were (IIRC) 200dpi.

[toc] | [prev] | [next] | [standalone]

#10013 — Re: origin of XGP fonts (was Re: reprints of old AI memos)

On 9/28/2026 10:33 PM, Peter Flass wrote:
> On 9/28/26 06:44, Kragen Javier Sitaker wrote:
>> Lars Brinkhoff <lars.spam@nocrew.org> writes:
>>> Kragen Javier Sitaker wrote:
>>>> I wonder where they [the XGP fonts] came from?
>>>
>>> I read a story somwhere, but I forgot the details and where I found it.
>>> It was something like getting a font catalog from a major foundry and
>>> digitizing it by hand.
>>>
>>> People also made their own; the AST file format is just a text file with
>>> pixel data, and on ITS there's FED (font "editor") to display a font on
>>> the 340 or a Knight TV.  I think SAIL had something else, certaily so
>>> when TeX and Metafont were developed.
>>>
>>> "The Last Whole XGP Font Catalog" (play on "Last Whole Earth Catalog")
>>> has a lot to say about the XGP.
>>
>> I was reading Butler Lampson’s 01988 retrospective on the Alto today
>> <https://dl.acm.org/doi/abs/10.1145/61975.66921> and he answered this
>> question:
>>
>> : The first imager, written by Peter Deutsch, drove the Xerox Graphics
>> : Printer.  This machine is slow (five pages/minute) and low resolution
>> : (200 dots/inch); it is also asynchronous, so the imager can take as
>> : long as it likes to generate each line of the raster. Only crude
>> : imaging software was developed for it, together with a few fonts.
>> : Raster fonts were unheard of at the time, except in 5 × 7 and 7 × 9
>> : resolution for terminals, or at very high resolution for expensive
>> : photo-typesetters. The XGP fonts were developed entirely manually,
>> : using an editor that allows the operator to turn individual dots in
>> : the roughly 20 × 20 matrix on and off. They had to be new designs,
>> : since it is impractical to faithfully copy a printer’s font at such
>> : low resolution. These XGP fonts were later widely used in
>> : universities.
>>
>>
> 
> The SAIL fonts were, at one point, available from one of the PC 
> shareware sites. I imagine they are still kicking around somewhere. I 
> played with them at one time, I think the major obstacle is that they 
> were (IIRC) 200dpi.

A slight tangent on the discussion of pixel fonts.  I seem to recall
reading the Microsoft's .FON specification -- I don't know if it's
still available on their website, but no matter -- and it supported
coloured fonts.  I don't recall if each individual pixel was colour-
able, or just the character, or the whole set.

It seems nobody ever used this, not even Microsoft.

Other than video games, I don't know anyone using two or more colours
per character in a pixel font.


Does alt.folklore.computers know better?
-- 
Johann | email: invalid -> com | http://www.myrkraverk.com/blog/
I'm not from the Internet, I just work there. | via XS News
https://bsky.app/profile/myrkraverk.bsky.social | for ( ;; ) _:;
Federated at https://fed.brid.gy/bsky/myrkraverk.bsky.social

[toc] | [prev] | [next] | [standalone]

#9996 — Re: reprints of old AI memos (was Re: boxing all integers)f

This new program by Rupert Lane takes a XGP file from SAIL and renders
it as PNG or PDF.  I don't know whether the input file format is the
same as at MIT.

https://github.com/timereshared/xgptopdf

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | alt.sys.pdp10

csiph-web