Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #16540 > unrolled thread

blhc failing due to non-verbose build

Back to article view | Back to linux.debian.maint.python

Contents

  blhc failing due to non-verbose build Soren Stoutner <soren@debian.org> - 2024-12-06 03:10 +0100
    Re: blhc failing due to non-verbose build Bo YU <tsu.yubo@gmail.com> - 2024-12-06 04:30 +0100
    Re: use of waf in pyinstaller (was: blhc) Simon McVittie <smcv@debian.org> - 2024-12-06 11:30 +0100
      Re: use of waf in pyinstaller (was: blhc) Soren Stoutner <soren@debian.org> - 2024-12-07 18:40 +0100
        Re: use of waf in pyinstaller (was: blhc) Scott Kitterman <debian@kitterman.com> - 2024-12-07 19:30 +0100
          Re: use of waf in pyinstaller (was: blhc) Soren Stoutner <soren@debian.org> - 2024-12-07 19:40 +0100

#16540 — blhc failing due to non-verbose build

[Multipart message — attachments visible in raw view] — view raw

I am working on PyInstaller, which is mostly written in Python, but compiles a 
bootloader written in c.  blhc failes because the longs do not contain verbose 
compile flags.

111:NONVERBOSE BUILD: [ 1/21] Compiling src/pyi_utils.c
112:NONVERBOSE BUILD: [ 2/21] Compiling src/pyi_main.c
113:NONVERBOSE BUILD: [ 3/21] Compiling src/pyi_exception_dialog.c
114:NONVERBOSE BUILD: [ 4/21] Compiling src/pyi_apple_events.c
115:NONVERBOSE BUILD: [ 5/21] Compiling src/pyi_splashlib.c
116:NONVERBOSE BUILD: [ 6/21] Compiling src/pyi_global_posix.c
117:NONVERBOSE BUILD: [ 7/21] Compiling src/pyi_multipkg.c

https://salsa.debian.org/python-team/packages/pyinstaller/-/jobs/6701977

The standard tricks for most C/C++ packages don’t work.

# Enable all build hardening flags.
export DEB_BUILD_MAINT_OPTIONS = hardening=+all

# Export the DPKG default build flags.
DPKG_EXPORT_BUILDFLAGS = 1

# Include the default build flags.  <https://wiki.debian.org/Hardening#dpkg-buildflags>
include /usr/share/dpkg/buildflags.mk

# Append the CPPFLAGS to the standard CFLAGS and CXXFLAGS variables, which is 
how CMake likes it.  <https://wiki.debian.org/
Hardening#Notes_for_packages_using_CMake>
CFLAGS += $(CPPFLAGS)
CXXFLAGS += $(CPPFLAGS)

https://salsa.debian.org/python-team/packages/pyinstaller/-/blob/debian/
master/debian/rules?ref_type=heads

Does anyone know how to convince it to populate the logs with the full flags 
used to compile the binaries?

-- 
Soren Stoutner
soren@debian.org

[toc] | [next] | [standalone]

#16542

Hi,

On Fri, Dec 6, 2024 at 10:01 AM Soren Stoutner <soren@debian.org> wrote:
>
...
> # Append the CPPFLAGS to the standard CFLAGS and CXXFLAGS variables, which is
> how CMake likes it.  <https://wiki.debian.org/
> Hardening#Notes_for_packages_using_CMake>
> CFLAGS += $(CPPFLAGS)
> CXXFLAGS += $(CPPFLAGS)
>
> https://salsa.debian.org/python-team/packages/pyinstaller/-/blob/debian/
> master/debian/rules?ref_type=heads
>
> Does anyone know how to convince it to populate the logs with the full flags
> used to compile the binaries?

I would like to suggest you to look at the code:
https://sources.debian.org/src/structure-synth/1.5.0-7.1/debian/rules/#L6
or
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1075694#56

Even if it doesn't work, we can get inspiration on how to append extra
flags during the configure phase.

Hope this helps.

BR,
Bo
>
> --
> Soren Stoutner
> soren@debian.org

[toc] | [prev] | [next] | [standalone]

#16544 — Re: use of waf in pyinstaller (was: blhc)

On Thu, 05 Dec 2024 at 19:00:50 -0700, Soren Stoutner wrote:
> I am working on PyInstaller, which is mostly written in Python, but compiles a 
> bootloader written in c.  blhc failes because the [logs] do not contain verbose 
> compile flags.

You'll need to look at the implementation of the build for the C part, and
then do whatever is most appropriate for that build system.

>From a quick glance at setup.py, it seems to be (a vendored copy of) waf:

        additional_args = os.getenv('PYINSTALLER_BOOTLOADER_WAF_ARGS', '').strip().split()
        cmd = [sys.executable, './waf', 'configure', 'all']
        cmd += additional_args

so hopefully there is something you can add to
PYINSTALLER_BOOTLOADER_WAF_ARGS that would make waf verbose, analogous
to `ninja -v` or Autotools `V=1`?

After that, you'll also need to make sure that the intended build
options are actually used (I don't know whether waf uses CFLAGS, etc. by
default or has to be given them via waf-specific command-line options).
Looking at other packages that use a waf build system and implement build
flags correctly, if any such packages exist, will probably be useful.

We have had problems with waf in the past, both technical and social
(licensing-related and others), so please be careful to ensure that
this package meets Debian's quality standards and doesn't contain any
particularly attractive places to hide malware.

In particular, the recommended way to distribute waf-built code used to
be to vendor a generated script containing a bzip2-compressed tarball,
which is not straightforward to review or patch, and the ftp team does
not consider this to be acceptable in Debian [1]. Is this still the case,
or is PyInstaller redistributing waf as reviewable/patchable files in
something more closely resembling their preferred form for modification?

Has the maintainer of this package (possibly you, I don't know this
package's history) verified that the included copy of waf is something
that we can trust? From the fact that you didn't already know this
package is using waf, I would guess perhaps not?

I can't help wondering whether its upstream would be receptive to the idea
of using Meson with meson-python as a replacement for both setuptools
and waf, which is what I would personally do if I had a Python module
that needed to compile/include C code (and indeed is what I *did* do
in dbus-python).

Good luck!

    smcv

[1] https://wiki.debian.org/UnpackWaf

[toc] | [prev] | [next] | [standalone]

#16552 — Re: use of waf in pyinstaller (was: blhc)

[Multipart message — attachments visible in raw view] — view raw

Simon,

Thank you for taking the time to look at this.

On Friday, December 6, 2024 3:27:55 AM MST Simon McVittie wrote:
> On Thu, 05 Dec 2024 at 19:00:50 -0700, Soren Stoutner wrote:
> > I am working on PyInstaller, which is mostly written in Python, but 
compiles
> > a bootloader written in c.  blhc failes because the [logs] do not contain
> > verbose compile flags.
> 
> You'll need to look at the implementation of the build for the C part, and
> then do whatever is most appropriate for that build system.
> 
> >From a quick glance at setup.py, it seems to be (a vendored copy of) waf:
>         additional_args = os.getenv('PYINSTALLER_BOOTLOADER_WAF_ARGS',
> '').strip().split() cmd = [sys.executable, './waf', 'configure', 'all']
>         cmd += additional_args
> 
> so hopefully there is something you can add to
> PYINSTALLER_BOOTLOADER_WAF_ARGS that would make waf verbose, analogous
> to `ninja -v` or Autotools `V=1`?

This was a very helpful suggestion.  I was able to produce a verbose build by 
adding the following to debian/rules:

# Enable the verbose waf build argument so that blhc can analyze the build 
flags.  waf is the system that builds the bootloader from C code.
export PYINSTALLER_BOOTLOADER_WAF_ARGS = --verbose

> After that, you'll also need to make sure that the intended build
> options are actually used (I don't know whether waf uses CFLAGS, etc. by
> default or has to be given them via waf-specific command-line options).
> Looking at other packages that use a waf build system and implement build
> flags correctly, if any such packages exist, will probably be useful.

The above verbose flag then produces this output in the build logs:

[ 1/21] Compiling src/pyi_utils.c
00:07:52 runner ['/usr/lib/ccache/gcc', '-g', '-O2', '-Werror=implicit-
function-declaration', '-ffile-prefix-map=/builds/python-team/packages/
pyinstaller/debian/output/source_dir=.', '-fstack-protector-strong', '-fstack-
clash-protection', '-Wformat', '-Werror=format-security', '-fcf-protection', 
'-Wdate-time', '-D_FORTIFY_SOURCE=2', '-m64', '-O2', '-Wall', '-Werror', '-
Wno-error=unused-variable', '-Wno-error=unused-function', '-Wno-error=unused-
but-set-variable', '-U_FORTIFY_SOURCE', '-Isrc', '-I../../src', '-Iwindows', 
'-I../../windows', '-Izlib', '-I../../zlib', '-D_REENTRANT', '-D_BSD_SOURCE', 
'-D_DEFAULT_SOURCE', '-D_FORTIFY_SOURCE=2', '-DHAVE_STDBOOL_H=1', '-
DHAVE_UNSETENV=1', '-DHAVE_MKDTEMP=1', '-DHAVE_DIRNAME=1', '-
DHAVE_BASENAME=1', '-DLAUNCH_DEBUG', '-DNDEBUG', '../../src/pyi_utils.c', '-
c', '-o/builds/python-team/packages/pyinstaller/debian/output/source_dir/
bootloader/build/debug/src/pyi_utils.c.1.o', '-Wdate-time', ‘-
D_FORTIFY_SOURCE=2’]

Blhc still reports the above as a NONVERBOSE build because there is a line 
break, so the first line is flagged separate from the second line.  It turns out 
there is an existing blhc bug report for this, which I have added some 
addition information to.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=976175

> We have had problems with waf in the past, both technical and social
> (licensing-related and others), so please be careful to ensure that
> this package meets Debian's quality standards and doesn't contain any
> particularly attractive places to hide malware.
> 
> In particular, the recommended way to distribute waf-built code used to
> be to vendor a generated script containing a bzip2-compressed tarball,
> which is not straightforward to review or patch, and the ftp team does
> not consider this to be acceptable in Debian [1]. Is this still the case,
> or is PyInstaller redistributing waf as reviewable/patchable files in
> something more closely resembling their preferred form for modification?
> 
> Has the maintainer of this package (possibly you, I don't know this
> package's history) verified that the included copy of waf is something
> that we can trust? From the fact that you didn't already know this
> package is using waf, I would guess perhaps not?

I have not had any experience with waf before, and so am not aware of DFSG or 
malware difficulties that other projects have faced.  In the case of 
PyInstaller, most of the waf code is contained in:

https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
master/bootloader/waflib?ref_type=heads

It is written in Python and licensed under the BSD-3-clause.  It is used to 
compile the C code in:

https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
master/bootloader/src?ref_type=heads

Which is licensed under the GPL-2+~with-bootloader-exception, which is the 
main license of the project.  The resulting bootloader (two files) is shipped 
in the binary package in /usr/lib/python3/dist-packages/PyInstaller/
bootloader/Linux-64bit-intel/*.

None of this looks problematic to me.  However, if there are any concerns I 
have missed I would be very interested to hear of them before I submit 
PyInstaller to the NEW queue.

-- 
Soren Stoutner
soren@debian.org

[toc] | [prev] | [next] | [standalone]

#16553 — Re: use of waf in pyinstaller (was: blhc)

On December 7, 2024 5:29:39 PM UTC, Soren Stoutner <soren@debian.org> wrote:
...
>I have not had any experience with waf before, and so am not aware of DFSG or 
>malware difficulties that other projects have faced.  In the case of 
>PyInstaller, most of the waf code is contained in:
>
>https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
>master/bootloader/waflib?ref_type=heads
>
>It is written in Python and licensed under the BSD-3-clause.  It is used to 
>compile the C code in:
>
>https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
>master/bootloader/src?ref_type=heads
>
>Which is licensed under the GPL-2+~with-bootloader-exception, which is the 
>main license of the project.  The resulting bootloader (two files) is shipped 
>in the binary package in /usr/lib/python3/dist-packages/PyInstaller/
>bootloader/Linux-64bit-intel/*.
>
>None of this looks problematic to me.  However, if there are any concerns I 
>have missed I would be very interested to hear of them before I submit 
>PyInstaller to the NEW queue.

Have a look at the waf entry in the FTP Team reject FAQ:

https://ftp-master.debian.org/REJECT-FAQ.html

Scott K

[toc] | [prev] | [next] | [standalone]

#16554 — Re: use of waf in pyinstaller (was: blhc)

[Multipart message — attachments visible in raw view] — view raw

On Saturday, December 7, 2024 11:18:20 AM MST Scott Kitterman wrote:
> On December 7, 2024 5:29:39 PM UTC, Soren Stoutner <soren@debian.org> wrote:
> ...
> 
> >I have not had any experience with waf before, and so am not aware of DFSG 
or
> >malware difficulties that other projects have faced.  In the case of
> >PyInstaller, most of the waf code is contained in:
> >
> >https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
> >master/bootloader/waflib?ref_type=heads
> >
> >It is written in Python and licensed under the BSD-3-clause.  It is used to
> >compile the C code in:
> >
> >https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
> >master/bootloader/src?ref_type=heads
> >
> >Which is licensed under the GPL-2+~with-bootloader-exception, which is the
> >main license of the project.  The resulting bootloader (two files) is 
shipped
> >in the binary package in /usr/lib/python3/dist-packages/PyInstaller/
> >bootloader/Linux-64bit-intel/*.
> >
> >None of this looks problematic to me.  However, if there are any concerns I
> >have missed I would be very interested to hear of them before I submit
> >PyInstaller to the NEW queue.
> 
> Have a look at the waf entry in the FTP Team reject FAQ:
> 
> https://ftp-master.debian.org/REJECT-FAQ.html

"That's a special case of source code missing. Normally packages using waf as 
build system contain a Python script with a compressed tarball embedded as a 
binary blob, where it is not obvious how to get the actual source. As that's 
not considered to be the preferred form of modification, it fails the DFSG. See 
#645190 and https://wiki.debian.org/UnpackWaf for details.”

As I detailed in the previous email, that does not appear to be the case for 
PyInstaller.  There are no binary blobs that I have found (although I would be 
interested in knowing if I have missed them).  I do understand and agree with 
such a concern.  It just doesn’t appear to be how waf is used by PyInstaller.

-- 
Soren Stoutner
soren@debian.org

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web