Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #11533 > unrolled thread

increased build time and disk usage for scipy with pybuild

Back to article view | Back to linux.debian.maint.python

Contents

  increased build time and disk usage for scipy with pybuild Drew Parsons <dparsons@debian.org> - 2019-07-27 16:20 +0200
    Re: increased build time and disk usage for scipy with pybuild Drew Parsons <dparsons@debian.org> - 2019-07-28 15:40 +0200

#11533 — increased build time and disk usage for scipy with pybuild

I've uploaded python-scipy 1.2.2 to unstable.

Previously the build system ran through distutils, but dh now gives an 
error on that, and says pybuild should be used instead.  So I 
reorganised debian/rules to use dh --buildsystem=pybuild.  I also 
reorganised rules to stop the second invocation of dh in the build-arch 
rule.

The build completes successfully.  But it is taking 3 times longer than 
it did before.  It is also using around 75% more disk space than before.

I guess the increase is probably not due to the new 1.2.2 release 
itself, otherwise I'd have expected an increase also in 1.2.1 (in 
experimental) compared to 1.1.0, which didn't happen.  I figure it's 
probably not simply that the buildds are busy catching up after the 
buster release, since I'm not seeing an increase in build time of other 
packages.  pybuild seems a more likely culprit (if not some other aspect 
of debhelper compatibility level 12. Previously debhelper 9 was used).

Is an increase in build resources like this known to happen when pybuild 
is used instead of distutils?


Drew

[toc] | [next] | [standalone]

#11536

On 2019-07-27 22:04, Drew Parsons wrote:
> I've uploaded python-scipy 1.2.2 to unstable.
> 
> Previously the build system ran through distutils, but dh now gives an
> error on that, and says pybuild should be used instead.  So I
> reorganised debian/rules to use dh --buildsystem=pybuild.  I also
> reorganised rules to stop the second invocation of dh in the
> build-arch rule.
> 
> The build completes successfully.  But it is taking 3 times longer
> than it did before.  It is also using around 75% more disk space than
> before.
...
> Is an increase in build resources like this known to happen when
> pybuild is used instead of distutils?


This seems to be why scipy 1.2.2-1 used more build resources:
scipy modules are built in 2 steps:
1) setup.py config_fc --noarch build (with override_dh_auto_configure)
2) setup.py install ... --force --no-compile (with 
override_dh_auto_install)

Previously (building with distutils), both steps built via a 
build/src.linux-x86_64-* builddir, so the --no-compile flag meant object 
files were not compiled twice.

Now with pybuild, the config_fc step is still built in 
build/src.linux-x86_64-*, but the install step is handled via 
build/src.linux-amd64-*.  So object files are compiled twice, first for 
x86_64, and then for amd64.  Likewise arm64 first "configures" builds in 
an aarch64 builddir, and then "installs" via arm64.

Looks like some discrepancy between setup.py config_fc and pybuild's 
setup.py install in handling DEB_*_ARCH_CPU and DEB_*_GNU_CPU.

DEB_*_CPU is not used explicitly by scipy's debian/rules.

Is setup.py config_fc even needed?  Is it a hangover from distutils and 
not needed in a pybuild build?  It's not well documented and not listed 
by "python3 setup.py --help-commands", though is mentioned in 
INSTALL.rst.txt (with a brief reference in scipy/special/setup.py and 
scipy/integrate/setup.py). But debian/rules doesn't use it with 
--fcompiler to specific the fortran compiler.

Drew

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web