Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #11841 > unrolled thread

2Removal: handling circular dependencies / dropping tests before the binary?

Back to article view | Back to linux.debian.maint.python

Contents

  2Removal: handling circular dependencies / dropping tests before the  binary? "Rebecca N. Palmer" <rebecca_palmer@zoho.com> - 2019-09-21 11:40 +0200
    Re: 2Removal: handling circular dependencies / dropping tests before  the binary? Andrey Rahmatullin <wrar@debian.org> - 2019-09-21 12:00 +0200
      Re: 2Removal: handling circular dependencies / dropping tests before  the binary? Sandro Tosi <morph@debian.org> - 2019-09-21 23:10 +0200
    Re: 2Removal: handling circular dependencies "Rebecca N. Palmer" <rebecca_palmer@zoho.com> - 2019-10-24 00:40 +0200
      Re: 2Removal: handling circular dependencies Simon McVittie <smcv@debian.org> - 2019-10-24 10:00 +0200
        Re: 2Removal: handling circular dependencies Andrey Rahmatullin <wrar@debian.org> - 2019-10-24 10:10 +0200
          Re: 2Removal: handling circular dependencies "Rebecca N. Palmer" <rebecca_palmer@zoho.com> - 2019-10-24 19:50 +0200
      Re: 2Removal: handling circular dependencies Sandro Tosi <morph@debian.org> - 2020-03-16 07:30 +0100
        Re: 2Removal: handling circular dependencies "Rebecca N. Palmer" <rebecca_palmer@zoho.com> - 2020-03-16 08:50 +0100

#11841 — 2Removal: handling circular dependencies / dropping tests before the binary?

Some Python 2 packages have circular (build-)dependencies, making it 
impossible to remove them one at a time without breaking anything.

One such cycle is seaborn, sphinx-gallery and matplotlib2 (#940873); I 
have not attempted to systematically check for them.

How should these be dealt with?

--- Cut the cycle by removing one of the dependencies.

This may mean skipping (some of) the tests, as Python build dependencies 
are often actually test dependencies.  There might be cases where it 
isn't reasonably possible at all.

(Related question: if a package with a large test suite is being 
uploaded anyway, should we consider removing the Python 2 tests just to 
save buildd/debci resources?)

--- Prepare all the required packages in salsa / experimental, then 
remove the entire cycle at once.

This is easy if the cycle is small and otherwise ready to remove, but if 
one member has a big dependency tree, the whole cycle might have to stay 
for some time.

(Related question: does it make sense to create no-Python2 salsa 
branches of one's regular modules (particularly any that need nontrivial 
changes) and mark them "please upload when the reverse dependencies have 
been cleared", even if they aren't part of a cycle?)

[toc] | [next] | [standalone]

#11842 — Re: 2Removal: handling circular dependencies / dropping tests before the binary?

[Multipart message — attachments visible in raw view] — view raw

On Sat, Sep 21, 2019 at 10:18:16AM +0100, Rebecca N. Palmer wrote:
> Some Python 2 packages have circular (build-)dependencies, making it
> impossible to remove them one at a time without breaking anything.
Unless I'm missing something, this shouldn't be a problem, just go away
and remove them all and upload them in roughly the same time?

> --- Cut the cycle by removing one of the dependencies.
> 
> This may mean skipping (some of) the tests, as Python build dependencies are
> often actually test dependencies.  There might be cases where it isn't
> reasonably possible at all.
Unless some of those don't have Python 3 packages in the archive yet, why
removing py2 from one package would break building another, if that
another has py2 parts removed too?
Note that having an unbuildable package in sid for several days, until
it's updated to a no-py2 version, is perfectly acceptable.

-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#11845 — Re: 2Removal: handling circular dependencies / dropping tests before the binary?

> Note that having an unbuildable package in sid for several days, until
> it's updated to a no-py2 version, is perfectly acceptable.

please note that may not be fine with every maintainer. dont just drop
packages without checking rdeps and/or if you know that will make
packages uninstallable/unbuildable (unless you check with the people
taking care of such packages first).

btw sphinx-gallery d-b on pyhton-seaborh has been removed.

-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
G+: https://plus.google.com/u/0/+SandroTosi

[toc] | [prev] | [next] | [standalone]

#11931 — Re: 2Removal: handling circular dependencies

[Summary of previous messages: I noted that packages with circular 
dependencies can't be removed one at a time without breakage.  Replies 
were to remove multiple packages at once if necessary, but ask first if 
other maintainers' packages are involved.]

I have now checked what cycles we have:

- 13 small sets (one of 13, others 2-4 each): small enough that "remove 
the whole set at once" is manageable.

- One big tangle (159 packages).  This probably needs breaking up:
--- Some of it involves documentation tools (e.g. sphinx).  These cycles 
can be broken by using the Python 3 version of the tool.
--- Some of it is "A Suggests (or Recommends) A-extension, A-extension 
Depends on A" cycles (e.g. pandas<->statsmodels).  If A-extension is 
otherwise ready to remove but A is not, these can be broken by removing 
the Suggests from A.  (Assuming we're still using "broken Suggests are 
not allowed": this has previously been discussed, I forget where.)

Full listing:

159 pyrex lxml html5lib sphinx mako python-scipy pandas patsy 
statsmodels seaborn sphinx-gallery matplotlib2 mpmath sympy 
texlive-extra nbconvert jupyter-notebook ipywidgets texlive-base dot2tex 
matplotlib numpydoc python-cycler ipykernel python-numpy mayavi2 
python-chaco python-envisage python-enable joblib pyzmq jupyter-client 
python-hypothesis chardet pygments python-pyface python-traitsui 
python-apptools python-traits ipython jupyter-core nbformat 
prompt-toolkit mercurial setuptools-scm python-py pytest-xdist pytest 
python-packaging python-importlib-metadata python-pluggy pyopenssl 
python-urllib3 python-future pyglet python-lz4 fdb sqlalchemy 
sphinxcontrib-websupport requests python-click incremental twisted 
automat python-service-identity python-gevent entrypoints python-flake8 
xcffib cairocffi python-keyring wheel python-pip keyrings.alt execnet 
sphinx-rtd-theme python-prometheus-client pytest-expect pytest-forked 
pytest-runner python-bleach python-mccabe python-atomicwrites 
python-attrs python-babel tap.py dbus-python python-qt4 pyqt5 
python-characteristic apipkg python-cryptography python-dateutil 
freezegun traitlets python-typing openpyxl python-tz python-et-xmlfile 
python-whoosh python-flaky wcwidth pexpect pillow python-docutils pymacs 
ropemacs python-mode enum34 sip4 configobj python-iso8601 pycairo 
pygobject-2 pygobject send2trash pygtk simplejson six unittest2 
python-mock python-psutil contextlib2 python-zipp python-traceback2 
python-debian python-apt python-linecache2 python-funcsigs 
python-concurrent.futures python-pathlib2 pickleshare testpath 
python-genty xlwt more-itertools backports.functools-lru-cache rope 
ropemode colorspacious cython pyyaml cvxopt sphinx-paramlinks 
python-pysqlite2 alabaster python-setupdocs nose mistune terminado 
python-webencodings ipython-genutils pep8 autopep8 pyxdg 
python-nose-exclude pyflakes python-greenlet xapian-bindings

2 rdflib sparql-wrapper-python
2 mgltools-pmv autodocktools
2 salutatoi sat-templates
2 cclib cclib-data
3 pastescript paste pastedeploy
2 sugar sugar-pippy-activity
13 trac-mercurial trac trac-customfieldadmin trac-graphviz 
trac-mastertickets trac-spamfilter trac-wikiprint trac-wysiwyg 
trac-xmlrpc email2trac trac-accountmanager trac-authopenid trac-bitten
4 hachoir-urwid hachoir-core hachoir-parser hachoir-metadata
4 python-pysnmp4 python-pysnmp4-mibs python-pysnmp4-apps pysmi
3 laditools ladish laditools # two 2Removal bugs on the same package
2 rpmlint rpm
2 crossfire crossfire-maps
3 python2.7 python-defaults python-stdlib-extensions

(Based on recent control@bugs.debian.org messages; packages not listed 
are not in a cycle)

[toc] | [prev] | [next] | [standalone]

#11932 — Re: 2Removal: handling circular dependencies

On Wed, 23 Oct 2019 at 23:05:39 +0100, Rebecca N. Palmer wrote:
> - One big tangle (159 packages).  This probably needs breaking up:
> --- Some of it involves documentation tools (e.g. sphinx).  These cycles can
> be broken by using the Python 3 version of the tool.

You've listed dbus-python as being part of this cycle, but since version
1.2.10-1 it only Build-Depends on sphinx-common, python3-sphinx and
python3-sphinx-rtd-theme. Is that enough to remove it from this cycle?

I removed the block when I made that change, but someone('s script?)
seems to have added it back? I can't remove python-dbus and close the
2removal bug yet, because lots of packages still depend on python-dbus.

I also can't see how -sphinx would depend directly or indirectly on
-dbus. Perhaps some of the blocks have been added the wrong way round?

I wonder whether parts of the 159-package cycle are actually smaller
cycles with non-trivial intersection, such as:

     python-sphinx --D--> sphinx-common
          | ^      <--S--
          D S
          v |
     python-alabaster

I'd tend to think of that as two intersecting 2-package cycles, rather
than one larger cycle.

> (Assuming we're still using "broken Suggests are not
> allowed": this has previously been discussed, I forget where.)

I think that's much too strong, particularly during a
transition. Transitions from unstable to testing only look at
(Pre-)Depends and Build-Depends(|-Arch|-Indep), although in general we
also consider broken Recommends to be a serious bug if anyone notices
them (due to Policy §2.2.1 "must not require or recommend a package
outside of main for compilation or execution").

Suggests are specifically not considered by Policy §2.2.1 (packages in
main are allowed to suggest packages that are in contrib, in non-free,
or not in the archive at all) and the pattern of cyclic Depends in one
direction, and Recommends or Suggests in the other, is extremely common.

    smcv

[toc] | [prev] | [next] | [standalone]

#11933 — Re: 2Removal: handling circular dependencies

[Multipart message — attachments visible in raw view] — view raw

On Thu, Oct 24, 2019 at 08:49:41AM +0100, Simon McVittie wrote:
> in general we
> also consider broken Recommends to be a serious bug if anyone notices
> them (due to Policy §2.2.1 "must not require or recommend a package
> outside of main for compilation or execution").
Note that the release policy up to buster explicitly said ""Recommends:"
lines do not count as requirements.", this is not true for
https://release.debian.org/bullseye/rc_policy.txt though.

-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#11934 — Re: 2Removal: handling circular dependencies

Simon McVittie wrote:
> You've listed dbus-python as being part of this cycle, but [it probably shouldn't be]
> Perhaps some of the blocks have been added the wrong way round?

Or perhaps the script is imperfectly guessing which binaries are Python 
2?  e.g. matplotlib is listed even though it's already a Python 3 only 
package.

> I wonder whether parts of the 159-package cycle are actually smaller
> cycles with non-trivial intersection

Very likely: by "tangle" I meant 
https://en.wikipedia.org/wiki/Strongly_connected_component

i.e. not a single cycle, but you still can't remove any subset of them 
(other than all or none) without breaking at least some of the remaining 
ones.

>> (Assuming we're still using "broken Suggests are not
>> allowed": this has previously been discussed, I forget where.)
> 
> I think that's much too strong

If the question is still open for discussion, I'd also prefer letting 
Suggests be temporarily broken over making extra uploads to remove them. 
  (I haven't checked how much this would break up the 159-tangle.)

[toc] | [prev] | [next] | [standalone]

#12365 — Re: 2Removal: handling circular dependencies

Hey Rebecca, do you have an updated list of these cycles?

On Wed, Oct 23, 2019 at 6:39 PM Rebecca N. Palmer
<rebecca_palmer@zoho.com> wrote:
>
> [Summary of previous messages: I noted that packages with circular
> dependencies can't be removed one at a time without breakage.  Replies
> were to remove multiple packages at once if necessary, but ask first if
> other maintainers' packages are involved.]
>
> I have now checked what cycles we have:
>
> - 13 small sets (one of 13, others 2-4 each): small enough that "remove
> the whole set at once" is manageable.
>
> - One big tangle (159 packages).  This probably needs breaking up:
> --- Some of it involves documentation tools (e.g. sphinx).  These cycles
> can be broken by using the Python 3 version of the tool.
> --- Some of it is "A Suggests (or Recommends) A-extension, A-extension
> Depends on A" cycles (e.g. pandas<->statsmodels).  If A-extension is
> otherwise ready to remove but A is not, these can be broken by removing
> the Suggests from A.  (Assuming we're still using "broken Suggests are
> not allowed": this has previously been discussed, I forget where.)
>
> Full listing:
>
> 159 pyrex lxml html5lib sphinx mako python-scipy pandas patsy
> statsmodels seaborn sphinx-gallery matplotlib2 mpmath sympy
> texlive-extra nbconvert jupyter-notebook ipywidgets texlive-base dot2tex
> matplotlib numpydoc python-cycler ipykernel python-numpy mayavi2
> python-chaco python-envisage python-enable joblib pyzmq jupyter-client
> python-hypothesis chardet pygments python-pyface python-traitsui
> python-apptools python-traits ipython jupyter-core nbformat
> prompt-toolkit mercurial setuptools-scm python-py pytest-xdist pytest
> python-packaging python-importlib-metadata python-pluggy pyopenssl
> python-urllib3 python-future pyglet python-lz4 fdb sqlalchemy
> sphinxcontrib-websupport requests python-click incremental twisted
> automat python-service-identity python-gevent entrypoints python-flake8
> xcffib cairocffi python-keyring wheel python-pip keyrings.alt execnet
> sphinx-rtd-theme python-prometheus-client pytest-expect pytest-forked
> pytest-runner python-bleach python-mccabe python-atomicwrites
> python-attrs python-babel tap.py dbus-python python-qt4 pyqt5
> python-characteristic apipkg python-cryptography python-dateutil
> freezegun traitlets python-typing openpyxl python-tz python-et-xmlfile
> python-whoosh python-flaky wcwidth pexpect pillow python-docutils pymacs
> ropemacs python-mode enum34 sip4 configobj python-iso8601 pycairo
> pygobject-2 pygobject send2trash pygtk simplejson six unittest2
> python-mock python-psutil contextlib2 python-zipp python-traceback2
> python-debian python-apt python-linecache2 python-funcsigs
> python-concurrent.futures python-pathlib2 pickleshare testpath
> python-genty xlwt more-itertools backports.functools-lru-cache rope
> ropemode colorspacious cython pyyaml cvxopt sphinx-paramlinks
> python-pysqlite2 alabaster python-setupdocs nose mistune terminado
> python-webencodings ipython-genutils pep8 autopep8 pyxdg
> python-nose-exclude pyflakes python-greenlet xapian-bindings
>
> 2 rdflib sparql-wrapper-python
> 2 mgltools-pmv autodocktools
> 2 salutatoi sat-templates
> 2 cclib cclib-data
> 3 pastescript paste pastedeploy
> 2 sugar sugar-pippy-activity
> 13 trac-mercurial trac trac-customfieldadmin trac-graphviz
> trac-mastertickets trac-spamfilter trac-wikiprint trac-wysiwyg
> trac-xmlrpc email2trac trac-accountmanager trac-authopenid trac-bitten
> 4 hachoir-urwid hachoir-core hachoir-parser hachoir-metadata
> 4 python-pysnmp4 python-pysnmp4-mibs python-pysnmp4-apps pysmi
> 3 laditools ladish laditools # two 2Removal bugs on the same package
> 2 rpmlint rpm
> 2 crossfire crossfire-maps
> 3 python2.7 python-defaults python-stdlib-extensions
>
> (Based on recent control@bugs.debian.org messages; packages not listed
> are not in a cycle)
>


-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

[toc] | [prev] | [next] | [standalone]

#12366 — Re: 2Removal: handling circular dependencies

On 16/03/2020 06:28, Sandro Tosi wrote:
> Hey Rebecca, do you have an updated list of these cycles?

Warning: this is based on open py2removal bugs, *not* actual package 
relationships.

2 crossfire crossfire-maps
2 sugar sugar-pippy-activity
2 rabbyt snowballz
4 python-pysnmp4 pysmi python-pysnmp4-apps python-pysnmp4-mibs
78 mercurial mercurial-extension-utils mercurial-keyring setuptools-scm 
automat twisted incremental pytest chardet beautifulsoup4 lxml html5lib 
soupsieve sphinx alabaster pycairo pygobject-2 pygobject keyrings.alt 
python-keyring dbus-python pygobject pygtk pycurl python-tornado 
python-urllib3 requests python-concurrent.futures pygments 
python-docutils enum34 python-cryptography pyopenssl 
python-service-identity python-hypothesis pyhamcrest python-attrs 
python-packaging python-importlib-metadata python-pluggy python-dateutil 
pymacs python-mode pyrex python-numpy cython pystemmer python-debian 
python-apt entrypoints pexpect pillow pytest-expect python-atomicwrites 
python-babel python-cffi python-flaky python-iso8601 python-py six 
more-itertools python-zipp python-genty python-mock python-pbr 
python-linecache2 python-traceback2 unittest2 contextlib2 
python-funcsigs python-virtualenv python-pathlib2 singledispatch 
python-typing python-tz wcwidth pillow-python2 backports.functools-lru-cache
2 apr apr-util
3 python2.7 python-defaults python-stdlib-extensions

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web