Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #229738 > unrolled thread

ps2pdf: why differences in different instances using unchanged PS source file

Back to article view | Back to linux.debian.user

Contents

  ps2pdf: why differences in different instances using unchanged PS  source file Tom Browder <tom.browder@gmail.com> - 2020-12-13 17:50 +0100
    Re: ps2pdf: why differences in different instances using unchanged  PS source file The Wanderer <wanderer@fastmail.fm> - 2020-12-13 18:10 +0100
      Re: ps2pdf: why differences in different instances using unchanged  PS source file The Wanderer <wanderer@fastmail.fm> - 2020-12-13 18:30 +0100
      Re: ps2pdf: why differences in different instances using unchanged PS  source file Tom Browder <tom.browder@gmail.com> - 2020-12-13 22:20 +0100
        Re: ps2pdf: why differences in different instances using unchanged PS  source file David Warring <david.warring@gmail.com> - 2020-12-14 23:30 +0100
          Re: ps2pdf: why differences in different instances using unchanged PS  source file Tom Browder <tom.browder@gmail.com> - 2020-12-14 23:40 +0100
            Re: ps2pdf: why differences in different instances using unchanged PS  source file Tom Browder <tom.browder@gmail.com> - 2020-12-15 00:50 +0100
          Re: ps2pdf: why differences in different instances using unchanged PS  source file Tom Browder <tom.browder@gmail.com> - 2020-12-15 00:00 +0100
          Re: ps2pdf: why differences in different instances using unchanged  PS source file David Wright <deblis@lionunicorn.co.uk> - 2020-12-17 02:30 +0100

#229738 — ps2pdf: why differences in different instances using unchanged PS source file

I have been using ps2pdf for many years on a PS source file for a
personalized calendar for my wife. This year I have been cleaning up
my old Perl generating code (in preparation for converting it to Raku
[https://raku.org]) and noticed I am getting a different pdf output
for each run, even when the PS output source file is unchanged!

I have looked at the detailed options for ps2pdf and don't see
anything that stands out to me when using the default settings.

I attempted a comparison of two different pdf outputs by running "od
-c file.pdf" on the two outputs but cannot identify anything familiar
to me about the differences.

My command that runs the actual process is this:

    $ ps2pdf  -dAutoRotatePages=/None  file.ps  file.pdf

I am running on Debian Buster with current updates.

Any suggestions to produce constant output pdf for constant ps input
are appreciated.

Blessings,

-Tom

[toc] | [next] | [standalone]

#229739 — Re: ps2pdf: why differences in different instances using unchanged PS source file

[Multipart message — attachments visible in raw view] — view raw

On 2020-12-13 at 11:40, Tom Browder wrote:

> I have been using ps2pdf for many years on a PS source file for a
> personalized calendar for my wife. This year I have been cleaning up
> my old Perl generating code (in preparation for converting it to Raku
> [https://raku.org]) and noticed I am getting a different pdf output
> for each run, even when the PS output source file is unchanged!
> 
> I have looked at the detailed options for ps2pdf and don't see
> anything that stands out to me when using the default settings.
> 
> I attempted a comparison of two different pdf outputs by running "od
> -c file.pdf" on the two outputs but cannot identify anything familiar
> to me about the differences.

What observation leads you to notice that the files are different?

> My command that runs the actual process is this:
> 
>     $ ps2pdf  -dAutoRotatePages=/None  file.ps  file.pdf
> 
> I am running on Debian Buster with current updates.
> 
> Any suggestions to produce constant output pdf for constant ps input
> are appreciated.

I ran this command on a random .ps file from a package I have installed,
to two output files.

cmp verified that the files were different.


Running vbindiff on the files shows that the and second differences are
in "ModifyDate" anbd "CreateDate" tags; each file appears to contain
multiple timestamps, and since the files were not generated at the same
second, those timestamps are going to be different.

The third difference is a UUID; that's intended to be globally unique,
and I'm guessing they intentionally generate a new one per PDF so that
they can be told apart.

The fourth difference is a CreationDate label, again with a timestamp.

The fifth and final difference, right before the end of the file, is an
unintelligible string of hex digits. My first suspicion is that it's a
hash of the rest of the file, to make validity checking possible.

So it doesn't look like the differences are meaningful, but they do
exist.


ps2pdf appears to be part of ghostscript. Neither its man page nor the
one for 'gs' (which it references) seem to contain any options for
setting these timestamps, preventing their creation, or doing similarly
with that UUID.


The Debian reproducible-builds project has a page on eliminating this
type of "multiple runs on the same input don't produce identical output"
issue more generally[1], which links to a page on doing so with PDFs
generated by Ghostscript specifically[2].

I'm not enough of an expert to follow that much further, but at a
glance, it looks like the solution is to patch Ghostscript; the patch
doesn't seem to have been accepted upstream, and it's not clear whether
or not it's (going to be) distributed in the Debian package.

If you're not OK with handling that on your own end, then you may be out
of luck with this.

[1] https://wiki.debian.org/ReproducibleBuilds/Howto
[2] https://wiki.debian.org/ReproducibleBuilds/PdfGeneratedByGhostscript

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#229740 — Re: ps2pdf: why differences in different instances using unchanged PS source file

[Multipart message — attachments visible in raw view] — view raw

On 2020-12-13 at 12:03, The Wanderer wrote:

> ps2pdf appears to be part of ghostscript. Neither its man page nor the
> one for 'gs' (which it references) seem to contain any options for
> setting these timestamps, preventing their creation, or doing similarly
> with that UUID.
> 
> 
> The Debian reproducible-builds project has a page on eliminating this
> type of "multiple runs on the same input don't produce identical output"
> issue more generally[1], which links to a page on doing so with PDFs
> generated by Ghostscript specifically[2].
> 
> I'm not enough of an expert to follow that much further, but at a
> glance, it looks like the solution is to patch Ghostscript; the patch
> doesn't seem to have been accepted upstream, and it's not clear whether
> or not it's (going to be) distributed in the Debian package.
> 
> If you're not OK with handling that on your own end, then you may be out
> of luck with this.
> 
> [1] https://wiki.debian.org/ReproducibleBuilds/Howto
> [2] https://wiki.debian.org/ReproducibleBuilds/PdfGeneratedByGhostscript

On closer examination of that latter link, apparently it's possible to
bypass the timestamps issue by using a command called 'faketime' (from a
package of the same name), to tell the program that the system-clock
time is set to something other than what it actually is.

The UUID is generated based on time, but apparently it's done in a way
that isn't affected by faketime, and that is what needs to be patched.

So you can probably eliminate three of the five differences by just
prepending 'faketime ' to your ps2pdf command. The other two will still
probably need patching of ghostscript to fix, however, and the link to
the patch from the above page seems to be dead.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [next] | [standalone]

#229742

On Sun, Dec 13, 2020 at 11:03 The Wanderer <wanderer@fastmail.fm> wrote:
> On 2020-12-13 at 11:40, Tom Browder wrote:
> > personalized calendar for my wife. This year I have been cleaning up
> > my old Perl generating code (in preparation for converting it to Raku
> > [https://raku.org]) and noticed I am getting a different pdf output
> > for each run, even when the PS output source file is unchanged!
...
> What observation leads you to notice that the files are different?

When I have both files under git revision control and committed, a
rerun shows no change in the PS but a change in the PDF.

I read the rest of this post and your additional post. Thank you for
your forensic information. As long as I know what has changed, thanks
to you, I am content for the moment.
I have a Raku online friend (in CC above) who is an expert on PDF and
has built voluminous tools with his PDF modules in Raku. I think he
will either know about the problem or possibly know how to fix it
since he deals with the binary output and modifying it after the fact.

Have a very Happy New Year!

Cheers,

- Tom

P.S. I wasn't aware of vbindiff (or maybe I forgot).

[toc] | [prev] | [next] | [standalone]

#229755

[Multipart message — attachments visible in raw view] — view raw

Hi Tom,
Yes quite right. As The Wanderer points out the CreationDate and ModDate
differ, as do the uuid in the Catalog Metadatas and the trailer ID.

I've found that deleting the optional dates from the Info dictionary, the
Metadata entry from the Catalog and resetting the ID does seem to be enough
to make output PDFs identical. In the simple case at least.

This gist https://gist.github.com/dwarring/1e4e056d84d6fe125262bba1da1f58fb
does
this using the Raku PDF module. Usage is:

p2s2pdf-strip.raku in.pdf [out.pdf]     # post-process ps2pdf output

I've used the lower level PDF::Reader interface, which doesn't often get
used directly. But in this case, we also need to bypass Raku PDF also
attempting to update the same fields. The PDF also needs to be rewritten,
rather than being incrementally updated.

Hope that works in this case.

Regards
David Warring

On Mon, Dec 14, 2020 at 10:14 AM Tom Browder <tom.browder@gmail.com> wrote:

> On Sun, Dec 13, 2020 at 11:03 The Wanderer <wanderer@fastmail.fm> wrote:
> > On 2020-12-13 at 11:40, Tom Browder wrote:
> > > personalized calendar for my wife. This year I have been cleaning up
> > > my old Perl generating code (in preparation for converting it to Raku
> > > [https://raku.org]) and noticed I am getting a different pdf output
> > > for each run, even when the PS output source file is unchanged!
> ...
> > What observation leads you to notice that the files are different?
>
> When I have both files under git revision control and committed, a
> rerun shows no change in the PS but a change in the PDF.
>
> I read the rest of this post and your additional post. Thank you for
> your forensic information. As long as I know what has changed, thanks
> to you, I am content for the moment.
> I have a Raku online friend (in CC above) who is an expert on PDF and
> has built voluminous tools with his PDF modules in Raku. I think he
> will either know about the problem or possibly know how to fix it
> since he deals with the binary output and modifying it after the fact.
>
> Have a very Happy New Year!
>
> Cheers,
>
> - Tom
>
> P.S. I wasn't aware of vbindiff (or maybe I forgot).
>

[toc] | [prev] | [next] | [standalone]

#229756

[Multipart message — attachments visible in raw view] — view raw

On Mon, Dec 14, 2020 at 16:06 David Warring <david.warring@gmail.com> wrote:

> Hi Tom,
> Yes quite right. As The Wanderer points out the CreationDate and ModDate
> differ, as do the uuid in the Catalog Metadatas and the trailer ID.
>
...

>
David, thanks! I knew you could do some fancy PDF hackery. I look forward
to trying it soon and will report back here.

Cheers!

-Tom

[toc] | [prev] | [next] | [standalone]

#229758

On Mon, Dec 14, 2020 at 4:36 PM Tom Browder <tom.browder@gmail.com> wrote:
> On Mon, Dec 14, 2020 at 16:06 David Warring <david.warring@gmail.com> wrote:
>> Hi Tom,
>> Yes quite right. As The Wanderer points out the CreationDate and ModDate differ, as do the uuid in the Catalog Metadatas and the trailer ID.
...
> David, thanks! I knew you could do some fancy PDF hackery. I look forward to trying it soon and will report back here.

It works like a charm!

I ran the script on each pdf, your program did its magic on each in
place, and a diff showed no difference!

Thank you so much, David! Another reason for people to pick up Raku programming.

Cheers and Merry Christmas to all of you.

-Tom

>
> Cheers!
>
> -Tom

[toc] | [prev] | [next] | [standalone]

#229757

[Multipart message — attachments visible in raw view] — view raw

On Mon, Dec 14, 2020 at 16:06 David Warring <david.warring@gmail.com> wrote:

> This gist
> https://gist.github.com/dwarring/1e4e056d84d6fe125262bba1da1f58fb does
> this using the Raku PDF module. Usage is:
>
> p2s2pdf-strip.raku in.pdf [out.pdf]     # post-process ps2pdf output
>

I can't read the fancy chars being input. Are they significant?

-Tom

[toc] | [prev] | [next] | [standalone]

#229800 — Re: ps2pdf: why differences in different instances using unchanged PS source file

On Tue 15 Dec 2020 at 11:06:33 (+1300), David Warring wrote:
> On Mon, Dec 14, 2020 at 10:14 AM Tom Browder <tom.browder@gmail.com> wrote:
> > On Sun, Dec 13, 2020 at 11:03 The Wanderer <wanderer@fastmail.fm> wrote:
> > > On 2020-12-13 at 11:40, Tom Browder wrote:
> > > > personalized calendar for my wife. This year I have been cleaning up
> > > > my old Perl generating code (in preparation for converting it to Raku
> > > > [https://raku.org]) and noticed I am getting a different pdf output
> > > > for each run, even when the PS output source file is unchanged!
> > ...
> > > What observation leads you to notice that the files are different?
> >
> > When I have both files under git revision control and committed, a
> > rerun shows no change in the PS but a change in the PDF.
> >
> > I read the rest of this post and your additional post. Thank you for
> > your forensic information. As long as I know what has changed, thanks
> > to you, I am content for the moment.
> > I have a Raku online friend (in CC above) who is an expert on PDF and
> > has built voluminous tools with his PDF modules in Raku. I think he
> > will either know about the problem or possibly know how to fix it
> > since he deals with the binary output and modifying it after the fact.
> >

> Yes quite right. As The Wanderer points out the CreationDate and ModDate
> differ, as do the uuid in the Catalog Metadatas and the trailer ID.
> 
> I've found that deleting the optional dates from the Info dictionary, the
> Metadata entry from the Catalog and resetting the ID does seem to be enough
> to make output PDFs identical. In the simple case at least.
> 
> This gist https://gist.github.com/dwarring/1e4e056d84d6fe125262bba1da1f58fb
> does
> this using the Raku PDF module. Usage is:
> 
> p2s2pdf-strip.raku in.pdf [out.pdf]     # post-process ps2pdf output
> 
> I've used the lower level PDF::Reader interface, which doesn't often get
> used directly. But in this case, we also need to bypass Raku PDF also
> attempting to update the same fields. The PDF also needs to be rewritten,
> rather than being incrementally updated.

Another way of tackling this is to use faketime when creating the PDF
file, which deals with the times and uuids. The only difference left is
the pair of ID strings (each 32 hex chars) at the end, which I assume
are random (when generated by ps2pdf).

One way of dealing with this is to run   cmp -bl a.pdf b.pdf   and
subtract the integer at the beginning of the first line of output from
that in the last line. The difference should not exceed 65 (2×32+2-1).

If writing a script to perform this check, one might parameterise this
number, as I did notice that convert (from ImageMagick) must take a
different approach to generating its IDs. The strings are 64 chars
long, but identical from run to run. So although just using faketime
can produce a perfect match in this case, it's a warning that the IDs
may have different properties when generated by other conversion tools.

Cheers,
David.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web