Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1130145 > unrolled thread

Bug#1026976: Upcoming test suite regression due to changes in file/libmagic

Back to article view | Back to linux.debian.bugs.dist

Contents

  Bug#1026976: Upcoming test suite regression due to changes in file/libmagic Christoph Biedl <debian.axhn@manchmal.in-ulm.de> - 2022-12-25 14:30 +0100
    Bug#1026976: Upcoming test suite regression due to changes in file/libmagic FC Stegerman <flx@obfusk.net> - 2022-12-25 22:30 +0100
    Bug#1026976: Upcoming test suite regression due to changes in file/libmagic "Chris Lamb" <lamby@debian.org> - 2022-12-28 11:20 +0100
      Bug#1026976: Upcoming test suite regression due to changes in file/libmagic Christoph Biedl <debian.axhn@manchmal.in-ulm.de> - 2022-12-29 16:50 +0100
        Bug#1026976: Upcoming test suite regression due to changes in file/libmagic FC Stegerman <flx@obfusk.net> - 2022-12-31 01:30 +0100
          Bug#1026976: Upcoming test suite regression due to changes in file/libmagic Christos Zoulas <christos@zoulas.com> - 2022-12-31 21:40 +0100
    Bug#1026976: Upcoming test suite regression due to changes in file/libmagic Christoph Biedl <debian.axhn@manchmal.in-ulm.de> - 2023-01-01 18:20 +0100
      Bug#1026976: Upcoming test suite regression due to changes in file/libmagic FC Stegerman <flx@obfusk.net> - 2023-01-01 22:30 +0100

#1130145 — Bug#1026976: Upcoming test suite regression due to changes in file/libmagic

[Multipart message — attachments visible in raw view] — view raw

Source: strip-nondeterminism
Version: 1.13.0-2
Severity: important

Hello,

possibly you've seen the similar story in diffoscope already: The last
upload of file/libmagic (1:5.43-3, currently in experimental) broke also
the strip-nondeterminism test suite:

======================================================================
    #   Failed test 'Normalizer found for /tmp/IhdQGJbeMu/pyzip'
    #   at t/fixtures.t line 83.
    #          got: undef
    #     expected: anything else

    #   Failed test 'Test output /tmp/IhdQGJbeMu/pyzip matched expected t/fixtures/pyzip/pyzip.out'
    #   at t/fixtures.t line 89.
    # Looks like you failed 2 tests of 9.

#   Failed test 't/fixtures/pyzip/pyzip.in'
#   at t/fixtures.t line 101.
strip-nondeterminism: /tmp/IZ0B8oD7YF/encrypted.zip: ignoring encrypted zip file
# Looks like you failed 1 test of 22.
t/fixtures.t ..·
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/22 subtests·
======================================================================

As I understand it, this is result of how t/fixtures/pyzip/pyzip.in is
described by file(1):

- a /usr/bin/python3 script executable (binary data)
+ Zip archive, with extra data prepended

Now that looks a bit delicate ... if you think this is something that
should be handled in file/libmagic, let me know.

In case you're courious, the change was:

<https://github.com/file/file/commit/a92246be4a8ceff26f0d4fdaa0390fc110146d7d>:

commit a92246be4a8ceff26f0d4fdaa0390fc110146d7d
Author: Christos Zoulas <christos@zoulas.com>
Date:   Sun Oct 2 12:34:00 2022 +0000

    Recognize ZIP archives with prepended data by end-of-central-directory record
    (Michal Gorny)

Regards,

    Christoph

[toc] | [next] | [standalone]

#1130192

* Christoph Biedl <debian.axhn@manchmal.in-ulm.de> [2022-12-25 14:20]:
> As I understand it, this is result of how t/fixtures/pyzip/pyzip.in is
> described by file(1):
> 
> - a /usr/bin/python3 script executable (binary data)
> + Zip archive, with extra data prepended
> 
> Now that looks a bit delicate ... if you think this is something that
> should be handled in file/libmagic, let me know.

[Disclaimer: this is just my personal opinion, I'm an RB contributor
but not involved in maintaining strip-nondeterminism.]

IMO that seems like a useful change in and of itself, but perhaps not
something that should take precedence over a #! line.

Perhaps something like

  a /usr/bin/python3 script executable (Zip archive)

would be ideal in this case, but I imagine that might not be trivial
to implement.

Thanks!

- FC

[toc] | [prev] | [next] | [standalone]

#1130485

Hi Christoph,

Thanks again for the "new version of file in experimental" bugs;
very much appreciated.

> - a /usr/bin/python3 script executable (binary data)
> + Zip archive, with extra data prepended
>
> Now that looks a bit delicate ... if you think this is something that
> should be handled in file/libmagic, let me know.

Naturally, as one of the maintainers of strip-nondeterminism, I am
inclined to believe that this is a minor regression in file. :)

However, this "extra data prepended" doesn't fit well under the rubric
of "data". Yes, this "#!/usr/bin/python3\n" shebang is definitely
"data" of a kind, but a shebang isn't really data in that way given
the special-treatment afforded to it by UNIX systems. Even if this
noun was replaced by the more general "bytes", the magic nature of the
shebang would still remain… as would the desire to discriminate
between pyzip files and other ZIP files with prepended data.

Could another — different — string be emitted in the case that these
prepended bytes are a shebang? We could potentially look for the file
starting with #! and for that to take precedence over this new case.


Regards,

-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby@debian.org 🍥 chris-lamb.co.uk
       `-

[toc] | [prev] | [next] | [standalone]

#1130663

[Multipart message — attachments visible in raw view] — view raw

Chris Lamb wrote...

> Thanks again for the "new version of file in experimental" bugs; very
> much appreciated.

You're welcome. Given file's volatility, I guess this will not be the
last conversation in that regard ...

> > - a /usr/bin/python3 script executable (binary data)
> > + Zip archive, with extra data prepended
> >
> > Now that looks a bit delicate ... if you think this is something that
> > should be handled in file/libmagic, let me know.
>
> Naturally, as one of the maintainers of strip-nondeterminism, I am
> inclined to believe that this is a minor regression in file. :)

This indeed was the rare where I think a change in file(1), while done
with best intentions, caused more harm than benefit.

> However, this "extra data prepended" doesn't fit well under the rubric
> of "data". Yes, this "#!/usr/bin/python3\n" shebang is definitely
> "data" of a kind, but a shebang isn't really data in that way given
> the special-treatment afforded to it by UNIX systems. Even if this
> noun was replaced by the more general "bytes", the magic nature of the
> shebang would still remain… as would the desire to discriminate
> between pyzip files and other ZIP files with prepended data.
>
> Could another — different — string be emitted in the case that these
> prepended bytes are a shebang? We could potentially look for the file
> starting with #! and for that to take precedence over this new case.

After some more thinking: I suggest you do nothing for the time being
while I take this to upstream. If all else fails, I'll revert the change
for the next upload. By the way, that will be 1:5.44-1, but the changes
to 1:5.43-3 are minimal.

    Christoph

[toc] | [prev] | [next] | [standalone]

#1130879

[Multipart message — attachments visible in raw view] — view raw

* Christoph Biedl <debian.axhn@manchmal.in-ulm.de> [2022-12-29 16:39]:
> Chris Lamb wrote...
> > However, this "extra data prepended" doesn't fit well under the rubric
> > of "data". Yes, this "#!/usr/bin/python3\n" shebang is definitely
> > "data" of a kind, but a shebang isn't really data in that way given
> > the special-treatment afforded to it by UNIX systems. Even if this
> > noun was replaced by the more general "bytes", the magic nature of the
> > shebang would still remain… as would the desire to discriminate
> > between pyzip files and other ZIP files with prepended data.
> >
> > Could another — different — string be emitted in the case that these
> > prepended bytes are a shebang? We could potentially look for the file
> > starting with #! and for that to take precedence over this new case.
> 
> After some more thinking: I suggest you do nothing for the time being
> while I take this to upstream. If all else fails, I'll revert the change
> for the next upload. By the way, that will be 1:5.44-1, but the changes
> to 1:5.43-3 are minimal.

FWIW I've attached a patch that makes it discriminate between ZIP
files with prepended data with or without a shebang:

  $ file -b zipfile-with-data-prepended
  Zip archive, with extra data prepended
  $ file -b pyzip
  a /usr/bin/python3 script executable (Zip archive)

I'm not sure that completely solves the issue, because there could be
other (executable) file formats affected, not just pyzip and other
formats using a shebang.  So perhaps reverting the change completely
is better after all.  We probably need upstream to figure that out.

- FC

[toc] | [prev] | [next] | [standalone]

#1130986

[Multipart message — attachments visible in raw view] — view raw

Committed, thanks!

christos

> On Dec 30, 2022, at 7:17 PM, FC Stegerman <flx@obfusk.net> wrote:
> 
> * Christoph Biedl <debian.axhn@manchmal.in-ulm.de> [2022-12-29 16:39]:
>> Chris Lamb wrote...
>>> However, this "extra data prepended" doesn't fit well under the rubric
>>> of "data". Yes, this "#!/usr/bin/python3\n" shebang is definitely
>>> "data" of a kind, but a shebang isn't really data in that way given
>>> the special-treatment afforded to it by UNIX systems. Even if this
>>> noun was replaced by the more general "bytes", the magic nature of the
>>> shebang would still remain… as would the desire to discriminate
>>> between pyzip files and other ZIP files with prepended data.
>>> 
>>> Could another — different — string be emitted in the case that these
>>> prepended bytes are a shebang? We could potentially look for the file
>>> starting with #! and for that to take precedence over this new case.
>> 
>> After some more thinking: I suggest you do nothing for the time being
>> while I take this to upstream. If all else fails, I'll revert the change
>> for the next upload. By the way, that will be 1:5.44-1, but the changes
>> to 1:5.43-3 are minimal.
> 
> FWIW I've attached a patch that makes it discriminate between ZIP
> files with prepended data with or without a shebang:
> 
>  $ file -b zipfile-with-data-prepended
>  Zip archive, with extra data prepended
>  $ file -b pyzip
>  a /usr/bin/python3 script executable (Zip archive)
> 
> I'm not sure that completely solves the issue, because there could be
> other (executable) file formats affected, not just pyzip and other
> formats using a shebang.  So perhaps reverting the change completely
> is better after all.  We probably need upstream to figure that out.
> 
> - FC
> <pyzip.patch>_______________________________________________
> Reproducible-builds mailing list
> Reproducible-builds@alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds

[toc] | [prev] | [next] | [standalone]

#1131138

[Multipart message — attachments visible in raw view] — view raw

Control: reassign 1026976 file 1:5.43-2
Control: tags 1026976 pending

Christoph Biedl wrote...

> As I understand it, this is result of how t/fixtures/pyzip/pyzip.in is
> described by file(1):
> 
> - a /usr/bin/python3 script executable (binary data)
> + Zip archive, with extra data prepended

Thanks to FC Stegerman, this has been fixed upstream. So this bug now
belongs into the domain of the file package, and will be fixed in the
next upload.

    Christoph

[toc] | [prev] | [next] | [standalone]

#1131172

[Multipart message — attachments visible in raw view] — view raw

* Christoph Biedl <debian.axhn@manchmal.in-ulm.de> [2023-01-01 18:17]:
> > As I understand it, this is result of how t/fixtures/pyzip/pyzip.in is
> > described by file(1):
> > 
> > - a /usr/bin/python3 script executable (binary data)
> > + Zip archive, with extra data prepended
> 
> Thanks to FC Stegerman, this has been fixed upstream. So this bug now
> belongs into the domain of the file package, and will be fixed in the
> next upload.

Unfortunately, that's not completely correct.  Whilst IMO my patch for
file makes its output "more correct", it's still different from the
output that strip-nondeterminism currently expects:

- a /usr/bin/python3 script executable (binary data)
+ a /usr/bin/python3 script executable (Zip archive)

I've attached a patch for matching both old and new output and opened
a merge request [1] on salsa as well.

- FC

[1] https://salsa.debian.org/reproducible-builds/strip-nondeterminism/-/merge_requests/15

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web