Groups > linux.debian.maint.python > #11536

Re: increased build time and disk usage for scipy with pybuild

Path	csiph.com!news.mixmin.net!aioe.org!bofh.it!news.nic.it!robomod
From	Drew Parsons <dparsons@debian.org>
Newsgroups	linux.debian.maint.python
Subject	Re: increased build time and disk usage for scipy with pybuild
Date	Sun, 28 Jul 2019 15:40:01 +0200
Message-ID	<yoRz3-17l-15@gated-at.bofh.it> (permalink)
References	<yovId-4jY-1@gated-at.bofh.it>
X-Mailbox-Line	From debian-python-request@lists.debian.org Sun Jul 28 13:39:14 2019
Old-Return-Path	<dparsons@debian.org>
X-Amavis-Spam-Status	No, score=-7.7 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, LDO_WHITELIST=-5, RCVD_IN_DNSWL_LOW=-0.7] autolearn=ham autolearn_force=no
X-Policyd-Weight	using cached result; rate: -3.5
MIME-Version	1.0
Content-Type	text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding	7bit
Organization	The Debian Project
Reply-To	dparsons@debian.org
Mail-Reply-To	dparsons@debian.org
X-Sender	dparsons@debian.org
User-Agent	Roundcube Webmail/1.3.8
X-Mailing-List	<debian-python@lists.debian.org> archive/latest/17033
List-ID	<debian-python.lists.debian.org>
List-URL	<https://lists.debian.org/debian-python/>
List-Archive	https://lists.debian.org/msgid-search/d6fb2bfdddbdb97ea8866fb78a5f4d20@debian.org
Approved	robomod@news.nic.it
Lines	46
Sender	robomod@news.nic.it
X-Original-Cc	Julian Taylor <jtaylor.debian@googlemail.com>
X-Original-Date	Sun, 28 Jul 2019 21:39:00 +0800
X-Original-Message-ID	<d6fb2bfdddbdb97ea8866fb78a5f4d20@debian.org>
X-Original-References	<00506870eaee7ed3ca38197d1bfc0de7@debian.org>
Xref	csiph.com linux.debian.maint.python:11536

Show key headers only | View raw

On 2019-07-27 22:04, Drew Parsons wrote:
> I've uploaded python-scipy 1.2.2 to unstable.
> 
> Previously the build system ran through distutils, but dh now gives an
> error on that, and says pybuild should be used instead.  So I
> reorganised debian/rules to use dh --buildsystem=pybuild.  I also
> reorganised rules to stop the second invocation of dh in the
> build-arch rule.
> 
> The build completes successfully.  But it is taking 3 times longer
> than it did before.  It is also using around 75% more disk space than
> before.
...
> Is an increase in build resources like this known to happen when
> pybuild is used instead of distutils?

This seems to be why scipy 1.2.2-1 used more build resources:
scipy modules are built in 2 steps:
1) setup.py config_fc --noarch build (with override_dh_auto_configure)
2) setup.py install ... --force --no-compile (with 
override_dh_auto_install)

Previously (building with distutils), both steps built via a 
build/src.linux-x86_64-* builddir, so the --no-compile flag meant object 
files were not compiled twice.

Now with pybuild, the config_fc step is still built in 
build/src.linux-x86_64-*, but the install step is handled via 
build/src.linux-amd64-*.  So object files are compiled twice, first for 
x86_64, and then for amd64.  Likewise arm64 first "configures" builds in 
an aarch64 builddir, and then "installs" via arm64.

Looks like some discrepancy between setup.py config_fc and pybuild's 
setup.py install in handling DEB_*_ARCH_CPU and DEB_*_GNU_CPU.

DEB_*_CPU is not used explicitly by scipy's debian/rules.

Is setup.py config_fc even needed?  Is it a hangover from distutils and 
not needed in a pybuild build?  It's not well documented and not listed 
by "python3 setup.py --help-commands", though is mentioned in 
INSTALL.rst.txt (with a brief reference in scipy/special/setup.py and 
scipy/integrate/setup.py). But debian/rules doesn't use it with 
--fcompiler to specific the fortran compiler.

Drew

Back to linux.debian.maint.python | Previous | Next — Previous in thread | Find similar | Unroll thread

Thread

increased build time and disk usage for scipy with pybuild Drew Parsons <dparsons@debian.org> - 2019-07-27 16:20 +0200
  Re: increased build time and disk usage for scipy with pybuild Drew Parsons <dparsons@debian.org> - 2019-07-28 15:40 +0200

csiph-web