Path: csiph.com!news.mixmin.net!aioe.org!bofh.it!news.nic.it!robomod From: Drew Parsons Newsgroups: linux.debian.maint.python Subject: Re: increased build time and disk usage for scipy with pybuild Date: Sun, 28 Jul 2019 15:40:01 +0200 Message-ID: References: X-Mailbox-Line: From debian-python-request@lists.debian.org Sun Jul 28 13:39:14 2019 Old-Return-Path: X-Amavis-Spam-Status: No, score=-7.7 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, LDO_WHITELIST=-5, RCVD_IN_DNSWL_LOW=-0.7] autolearn=ham autolearn_force=no X-Policyd-Weight: using cached result; rate: -3.5 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Organization: The Debian Project Reply-To: dparsons@debian.org Mail-Reply-To: dparsons@debian.org X-Sender: dparsons@debian.org User-Agent: Roundcube Webmail/1.3.8 X-Mailing-List: archive/latest/17033 List-ID: List-URL: List-Archive: https://lists.debian.org/msgid-search/d6fb2bfdddbdb97ea8866fb78a5f4d20@debian.org Approved: robomod@news.nic.it Lines: 46 Sender: robomod@news.nic.it X-Original-Cc: Julian Taylor X-Original-Date: Sun, 28 Jul 2019 21:39:00 +0800 X-Original-Message-ID: X-Original-References: <00506870eaee7ed3ca38197d1bfc0de7@debian.org> Xref: csiph.com linux.debian.maint.python:11536 On 2019-07-27 22:04, Drew Parsons wrote: > I've uploaded python-scipy 1.2.2 to unstable. > > Previously the build system ran through distutils, but dh now gives an > error on that, and says pybuild should be used instead. So I > reorganised debian/rules to use dh --buildsystem=pybuild. I also > reorganised rules to stop the second invocation of dh in the > build-arch rule. > > The build completes successfully. But it is taking 3 times longer > than it did before. It is also using around 75% more disk space than > before. ... > Is an increase in build resources like this known to happen when > pybuild is used instead of distutils? This seems to be why scipy 1.2.2-1 used more build resources: scipy modules are built in 2 steps: 1) setup.py config_fc --noarch build (with override_dh_auto_configure) 2) setup.py install ... --force --no-compile (with override_dh_auto_install) Previously (building with distutils), both steps built via a build/src.linux-x86_64-* builddir, so the --no-compile flag meant object files were not compiled twice. Now with pybuild, the config_fc step is still built in build/src.linux-x86_64-*, but the install step is handled via build/src.linux-amd64-*. So object files are compiled twice, first for x86_64, and then for amd64. Likewise arm64 first "configures" builds in an aarch64 builddir, and then "installs" via arm64. Looks like some discrepancy between setup.py config_fc and pybuild's setup.py install in handling DEB_*_ARCH_CPU and DEB_*_GNU_CPU. DEB_*_CPU is not used explicitly by scipy's debian/rules. Is setup.py config_fc even needed? Is it a hangover from distutils and not needed in a pybuild build? It's not well documented and not listed by "python3 setup.py --help-commands", though is mentioned in INSTALL.rst.txt (with a brief reference in scipy/special/setup.py and scipy/integrate/setup.py). But debian/rules doesn't use it with --fcompiler to specific the fortran compiler. Drew