Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.bugs.dist > #1102232 > unrolled thread

Bug#1010570: binaries in source without related source

Back to article view | Back to linux.debian.bugs.dist

Contents

  Bug#1010570: binaries in source without related source Antoine Beaupre <anarcat@debian.org> - 2022-05-04 17:20 +0200
    Bug#1010570: binaries in source without related source Tino Mettler <tino.mettler@tikei.de> - 2022-05-04 18:00 +0200
      Bug#1010570: binaries in source without related source Antoine Beaupré <anarcat@orangeseeds.org> - 2022-05-04 19:10 +0200
        Bug#1010570: binaries in source without related source Tino Mettler <tino.mettler@tikei.de> - 2022-05-04 19:40 +0200
          Bug#1010570: binaries in source without related source Tino Mettler <tino.mettler@tikei.de> - 2022-05-04 20:10 +0200
            Bug#1010570: binaries in source without related source Antoine Beaupré <anarcat@orangeseeds.org> - 2022-05-04 20:30 +0200
              Bug#1010570: binaries in source without related source Tino Mettler <tino.mettler@tikei.de> - 2022-05-04 21:20 +0200
                Bug#1010570: binaries in source without related source Antoine Beaupré <anarcat@debian.org> - 2022-05-04 21:30 +0200
                  Bug#1010570: binaries in source without related source Tino Mettler <tino.mettler@tikei.de> - 2022-05-04 22:20 +0200
                    Bug#1010570: binaries in source without related source Tino Mettler <tino.mettler@tikei.de> - 2022-05-04 23:10 +0200
                      Bug#1010570: binaries in source without related source Tino Mettler <tino.mettler@tikei.de> - 2022-05-05 10:40 +0200
                        Bug#1010570: binaries in source without related source Tino Mettler <tino.mettler@tikei.de> - 2022-05-05 10:40 +0200
                          Bug#1010570: binaries in source without related source Antoine Beaupré <anarcat@debian.org> - 2022-05-05 15:30 +0200
          Bug#1010570: binaries in source without related source Antoine Beaupré <anarcat@orangeseeds.org> - 2022-05-04 20:10 +0200
    Bug#1010570: binaries in source without related source Tino Mettler <tino.mettler@tikei.de> - 2022-05-05 22:00 +0200
      Bug#1010570: binaries in source without related source Antoine Beaupré <anarcat@orangeseeds.org> - 2022-05-05 22:20 +0200
    Bug#1010570: binaries in source without related source Tino Mettler <tino.mettler@tikei.de> - 2022-05-06 11:00 +0200
      Bug#1010570: binaries in source without related source Antoine Beaupré <anarcat@orangeseeds.org> - 2022-05-06 15:10 +0200

#1102232 — Bug#1010570: binaries in source without related source

Package: rapid-photo-downloader
Version: 0.9.26-2
Severity: normal

I'm not sure about this, but it seems to me that `upgrade.py` and
`qrc_resources.py` ship binary data as source code.

This is normally a red flag in Debian packages, because source code
should typically be text file that are humanly understandable, or at
least usable with free software tools.

Basing myself on the excellent investigation done by Tino Mettler:

https://salsa.debian.org/debian/rapid-photo-downloader/-/merge_requests/2#note_307363

... i t seems like the `upgrade.py` file is fairly innocuous: it's
actually an encoded ZIP file that has .mo files generated from the
provided .po files. I'm not sure that needs to be removed, as there is
probably an obvious source for those.

The other file, `qrc_resources.py`, is more problematic. It bundles
binary data like images and those don't seem to have an associated
source in the source code. It's unclear if that file could be
redistributed as is, as it's clearly not modifiable, and would
possibly be a license violation.

It seems like the source images for that file are missing from the
upstream source as well, crucially. Normally, we should be able to
recompile that file from source, but because those images are missing,
it's not possible.

I'm not an expert on those issues as I used to be, so I'm not sure
about all this, but it seemed important to flag this as an issue on
the package.

-- System Information:
Debian Release: 11.3
  APT prefers stable-security
  APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'stable'), (1, 'unstable'), (1, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.10.0-13-amd64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_FIRMWARE_WORKAROUND
Locale: LANG=fr_CA.UTF-8, LC_CTYPE=fr_CA.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages rapid-photo-downloader depends on:
ii  gir1.2-gexiv2-0.10                     0.12.1-1
ii  gir1.2-glib-2.0                        1.66.1-1+b1
ii  gir1.2-gstreamer-1.0                   1.18.4-2.1
ii  gir1.2-gudev-1.0                       234-1
ii  gir1.2-notify-0.7                      0.7.9-3
ii  gir1.2-udisks-2.0                      2.9.2-2+deb11u1
ii  gstreamer1.0-libav                     1.18.4-3
ii  gstreamer1.0-plugins-good              1.18.4-2
ii  libgphoto2-6                           2.5.27-1
ii  libimage-exiftool-perl                 12.16+dfsg-2
ii  libmediainfo0v5                        20.09+dfsg-2
ii  libqt5svg5                             5.15.2-3
ii  python3                                3.9.2-3
ii  python3-arrow                          1.2.1-1
ii  python3-babel                          2.8.0+dfsg.1-7
ii  python3-colour                         0.1.5-2
ii  python3-dateutil                       2.8.1-6
ii  python3-easygui                        0.98.1-1
ii  python3-gi                             3.38.0-2
ii  python3-gphoto2                        1.9.0-1+b2
ii  python3-gphoto2cffi [python3-gphoto2]  0.4.3~a1-1.1+b1
ii  python3-psutil                         5.8.0-1
ii  python3-pymediainfo                    5.0.3-1
ii  python3-pyqt5                          5.15.2+dfsg-3
ii  python3-requests                       2.25.1+dfsg-2
ii  python3-sortedcontainers               2.1.0-2
ii  python3-tenacity                       6.2.0-4
ii  python3-tornado                        6.1.0-1+b1
ii  python3-xdg                            0.27-2
ii  python3-zmq                            20.0.0-1+b1
ii  qt5-image-formats-plugins              5.15.2-2

Versions of packages rapid-photo-downloader recommends:
ii  libraw-bin  0.20.2-1

rapid-photo-downloader suggests no packages.

-- debconf-show failed

[toc] | [next] | [standalone]

#1102236

Hi,

the images used to generate qrc_resources.py are present in the
upstream git repository, but not in the release tarballs. Once they are
copied into the source directory of the Debian package,
qrc_resources.py can be regenerated using pyrcc5 (the PyQT resource
compiler).

I opened an issue in the upstream git repository and described the
nature of this bug.  I also asked if it is possible to provide signed
release tarballs with the images included.

Regards,
Tino

[toc] | [prev] | [next] | [standalone]

#1102242

On 2022-05-04 17:41:42, Tino Mettler wrote:
> Hi,
>
> the images used to generate qrc_resources.py are present in the
> upstream git repository, but not in the release tarballs. Once they are
> copied into the source directory of the Debian package,
> qrc_resources.py can be regenerated using pyrcc5 (the PyQT resource
> compiler).
>
> I opened an issue in the upstream git repository and described the
> nature of this bug.  I also asked if it is possible to provide signed
> release tarballs with the images included.

Maybe a solution that wouldn't involve upstream would be to change the
way we generate the tarball, from git instead of the upstream tarball.

After all, if we're going to rebuild thing from scratch ourselves
anyways...

[toc] | [prev] | [next] | [standalone]

#1102244

Am Wed, May 04, 2022 at 13:03:19 -0400 schrieb Antoine Beaupré:

[...]

> Maybe a solution that wouldn't involve upstream would be to change the
> way we generate the tarball, from git instead of the upstream tarball.
> 
> After all, if we're going to rebuild thing from scratch ourselves
> anyways...

I also think that would be the way to go, as the upstream author is
currently fighting with health issues.

https://github.com/damonlynch/rapid-photo-downloader/issues/73#issuecomment-1117546807

Regards,
Tino

[toc] | [prev] | [next] | [standalone]

#1102245

Hi,

the upstream git repository has no release tags, so I guess that we
have to drop uscan support and create the upstream source archive with
a selfmade script.

Regards,
Tino

[toc] | [prev] | [next] | [standalone]

#1102248

On 2022-05-04 20:07:28, Tino Mettler wrote:
> Hi,
>
> the upstream git repository has no release tags, so I guess that we
> have to drop uscan support and create the upstream source archive with
> a selfmade script.

we could lay down release tags ourselves if we can figure out what those
are... 

[toc] | [prev] | [next] | [standalone]

#1102254

> Am 04.05.2022 um 20:17 schrieb Antoine Beaupré <anarcat@orangeseeds.org>:
> 
> On 2022-05-04 20:07:28, Tino Mettler wrote:
>> Hi,
>> 
>> the upstream git repository has no release tags, so I guess that we
>> have to drop uscan support and create the upstream source archive with
>> a selfmade script.
> 
> we could lay down release tags ourselves if we can figure out what those
> are... 

Hi,

that would be an option. I'm currently crafting a script that creates a tarball. Just removing upgrade.py doesn't work, because it is referenced in the translations and that will cause a build errors. Replacing it with an empty file works, and should not affect Debian users.

Regards,
Tino

[toc] | [prev] | [next] | [standalone]

#1102256

On 2022-05-04 21:06:38, Tino Mettler wrote:
>> Am 04.05.2022 um 20:17 schrieb Antoine Beaupré <anarcat@orangeseeds.org>:
>> 
>> On 2022-05-04 20:07:28, Tino Mettler wrote:
>>> Hi,
>>> 
>>> the upstream git repository has no release tags, so I guess that we
>>> have to drop uscan support and create the upstream source archive with
>>> a selfmade script.
>> 
>> we could lay down release tags ourselves if we can figure out what those
>> are... 
>
> Hi,
>
> that would be an option. I'm currently crafting a script that creates a tarball. Just removing upgrade.py doesn't work, because it is referenced in the translations and that will cause a build errors. Replacing it with an empty file works, and should not affect Debian users.

maybe we could just use a +ds tarball then. add the files to redact in
debian/copyright and then uscan will just do the right thing for us, no
need to worry about the git stuff.

-- 
La guerre, c'est le massacre d'hommes qui ne se connaissent pas,
au profit d'hommes qui se connaissent mais ne se massacreront pas.
                        - Paul Valéry

[toc] | [prev] | [next] | [standalone]

#1102261

Am Wed, May 04, 2022 at 15:17:05 -0400 schrieb Antoine Beaupré:

[...]

> maybe we could just use a +ds tarball then. add the files to redact in
> debian/copyright and then uscan will just do the right thing for us, no
> need to worry about the git stuff.

We need the images/ and tips/ folders from the upstream git repository.

This script would work. I already prepared a package using the +dfsg upstream
source generated by the script. The resources are compiles during
build, the result looks good.

#!/bin/sh -ex

git reset --hard $1
VERSION=$(sed '/^-------------/,$d' CHANGES.md | tail -n 1 | cut -d' ' -f1)

echo > upgrade.py
git rm raphodo/qrc_resources.py
git commit -m "Remove source files with embedded binary blobs"

git archive HEAD --prefix=rapid-photo-downloader-$VERSION/ -o ../rapid-photo-downloader_${VERSION}+dfsg.orig.tar.xz

Regards,
Tino

[toc] | [prev] | [next] | [standalone]

#1102267

Am Wed, May 04, 2022 at 22:12:20 +0200 schrieb Tino Mettler:
> Am Wed, May 04, 2022 at 15:17:05 -0400 schrieb Antoine Beaupré:
> 
> [...]
> 
> > maybe we could just use a +ds tarball then. add the files to redact in
> > debian/copyright and then uscan will just do the right thing for us, no
> > need to worry about the git stuff.
> 
> We need the images/ and tips/ folders from the upstream git repository.
> 
> This script would work. I already prepared a package using the +dfsg upstream
> source generated by the script. The resources are compiles during
> build, the result looks good.

I forgot to git add the upgrade.py. Furthermore, Damon mentioned
install.py. And yes, it contains another binary blob (the same ZIP
archive as upgrade.py, I guess).

Here is an improved script:

#!/bin/sh -ex

git reset --hard $1
VERSION=$(sed '/^-------------/,$d' CHANGES.md | tail -n 1 | cut -d' ' -f1)

echo > upgrade.py
echo > install.py
git add upgrade.py install.py
git rm raphodo/qrc_resources.py
git commit -m "Remove source files with embedded binary blobs"

git archive HEAD --prefix=rapid-photo-downloader-$VERSION/ -o ../rapid-photo-downloader_${VERSION}+dfsg.orig.tar.xz

Regards,
Tino

[toc] | [prev] | [next] | [standalone]

#1102294

Hi,

I pushed a branch that uses the dfsg source generated by the above
script.

It contains one additional change to generate the qrc_resources.py
file.

https://salsa.debian.org/scorpi-guest/rapid-photo-downloader/-/commits/debian/unstable-0.9.33+dfsg

[toc] | [prev] | [next] | [standalone]

#1102295

Hi,

regarding the comments in the code, the binary blob in install.py and
upgrade.py can be generated during build time using tasks/generate.py. 
However, both files are not part of the binary package so I see no
point in doing that. So I think replacing them with empty files in
the dfsg source tarball is okay.

Regards,
Tino

[toc] | [prev] | [next] | [standalone]

#1102322

On 2022-05-05 10:37:41, Tino Mettler wrote:
> Hi,
>
> regarding the comments in the code, the binary blob in install.py and
> upgrade.py can be generated during build time using tasks/generate.py. 
> However, both files are not part of the binary package so I see no
> point in doing that. So I think replacing them with empty files in
> the dfsg source tarball is okay.

Agreed.

-- 
Music gives a soul to the universe, wings to the mind, flight to the
imagination and life to everything
                         - Plato

[toc] | [prev] | [next] | [standalone]

#1102246

On 2022-05-04 19:29:24, Tino Mettler wrote:
> Am Wed, May 04, 2022 at 13:03:19 -0400 schrieb Antoine Beaupré:
>
> [...]
>
>> Maybe a solution that wouldn't involve upstream would be to change the
>> way we generate the tarball, from git instead of the upstream tarball.
>> 
>> After all, if we're going to rebuild thing from scratch ourselves
>> anyways...
>
> I also think that would be the way to go, as the upstream author is
> currently fighting with health issues.
>
> https://github.com/damonlynch/rapid-photo-downloader/issues/73#issuecomment-1117546807

Oh dear yeah, screw that. Let's not bother that poor upstream
already. :)

Let's make a new tarball with the upstream git repo instead, can you try
that?

Thanks!

a.

-- 
Any sufficiently advanced technology is indistinguishable from magic.
                        - Arthur C. Clarke

[toc] | [prev] | [next] | [standalone]

#1102370

Hi Antoine,

I'm currently thinking about missing bits for a 0.9.33 release.  I
didn't switch a package to dfsg source before, so I'm new to that.

It looks like it should be documented in debian/copyright, e.g. why the
source tarball was repacked and what is the reason.

There should be a changelog entry, which also refers to this bug.

Furthermore, I guess that the procedure to create the upstream tarball
should be described, by providing the script that I used. This could be
done in a README.Packaging.

Please let me know if I am missing something.

Regards,
Tino

[toc] | [prev] | [next] | [standalone]

#1102373

On 2022-05-05 21:51:16, Tino Mettler wrote:
> Hi Antoine,
>
> I'm currently thinking about missing bits for a 0.9.33 release.  I
> didn't switch a package to dfsg source before, so I'm new to that.
>
> It looks like it should be documented in debian/copyright, e.g. why the
> source tarball was repacked and what is the reason.
>
> There should be a changelog entry, which also refers to this bug.

That all sounds about right to me!

> Furthermore, I guess that the procedure to create the upstream tarball
> should be described, by providing the script that I used. This could be
> done in a README.Packaging.

Typically, the script lives in debian/rules, as a target
("get-orig-source"). This has actually been deprecated by Debian policy
4.1.4, interestingly:

https://www.debian.org/doc/debian-policy/upgrading-checklist.html#version-4-1-4

... but is still mentioned in the developer's reference:

https://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.en.html#repackaged-upstream-source

In general, I think that means *maybe* we could try harder to get uscan
to do the right thing here, but I understand if it seems impossible. :)

The documentation about that stuff can also live in README.source:

https://www.debian.org/doc/debian-policy/ch-source.html#source-package-handling-debian-readme-source

> Please let me know if I am missing something.

I think that's about right, thanks for the hard work!

-- 
We will create a civilization of the Mind in Cyberspace. May it be more
humane and fair than the world your governments have made before.
                        - John Perry Barlow, 1996
                        A Declaration of Independence of Cyberspace

[toc] | [prev] | [next] | [standalone]

#1102420

Hi Antoine,

I just discussed this a bit on #debian-mentors in IRC.

Conclusion:

- autogenerated files in the upstream tarball might be okay, if they are
  not used for the binary packages (upgrade.py, install.py) or if they
  are regenerated during build time (qrc_resources.py)

- the details about this should be documented in debian/copyright

- the missing images to generate the qrc_resources.py file can be put
  into debian/missing-sources

- we can discuss this issue again with the upstream author once he has
  recovered

- it is unlikely that there will be a new upstream release
  before this

When we follow this, we don't need to repack the upstream source, just
add the images to debian/missing-sources and document all that in
debian/copyright.

What do you thing about this?

Regards,
Tino

[toc] | [prev] | [next] | [standalone]

#1102435

On 2022-05-06 10:54:56, Tino Mettler wrote:
> Hi Antoine,
>
> I just discussed this a bit on #debian-mentors in IRC.
>
> Conclusion:
>
> - autogenerated files in the upstream tarball might be okay, if they are
>   not used for the binary packages (upgrade.py, install.py) or if they
>   are regenerated during build time (qrc_resources.py)
>
> - the details about this should be documented in debian/copyright
>
> - the missing images to generate the qrc_resources.py file can be put
>   into debian/missing-sources
>
> - we can discuss this issue again with the upstream author once he has
>   recovered
>
> - it is unlikely that there will be a new upstream release
>   before this
>
> When we follow this, we don't need to repack the upstream source, just
> add the images to debian/missing-sources and document all that in
> debian/copyright.
>
> What do you thing about this?

That makes total sense to me.

-- 
If quantum mechanics hasn't profoundly shocked you, you haven't
understood it yet.
                       - Niels Bohr

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.bugs.dist

csiph-web