Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > linux.debian.maint.python > #11347 > unrolled thread

python-scipy: autopkgtest fails (Re:bug#919929)

Started byDrew Parsons <dparsons@debian.org>
First post2019-03-05 12:10 +0100
Last post2019-03-16 17:00 +0100
Articles 15 — 3 participants

Back to article view | Back to linux.debian.maint.python


Contents

  python-scipy: autopkgtest fails (Re:bug#919929) Drew Parsons <dparsons@debian.org> - 2019-03-05 12:10 +0100
    Re: python-scipy: autopkgtest fails (Re: bug#919929) Andreas Tille <tille@debian.org> - 2019-03-07 09:10 +0100
      Re: python-scipy: autopkgtest fails (Re: bug#919929) Paul Gevers <elbrus@debian.org> - 2019-03-07 12:10 +0100
        Re: python-scipy: autopkgtest fails (Re: bug#919929) Drew Parsons <dparsons@debian.org> - 2019-03-07 13:20 +0100
          Re: python-scipy: autopkgtest fails (Re: bug#919929) Paul Gevers <elbrus@debian.org> - 2019-03-07 13:50 +0100
            Re: python-scipy: autopkgtest fails (Re: bug#919929) Drew Parsons <dparsons@debian.org> - 2019-03-07 15:00 +0100
              Re: python-scipy: autopkgtest fails (Re: bug#919929) Paul Gevers <elbrus@debian.org> - 2019-03-07 15:10 +0100
                Re: python-scipy: autopkgtest fails (Re: bug#919929) Drew Parsons <dparsons@debian.org> - 2019-03-07 15:30 +0100
            Re: python-scipy: autopkgtest fails (Re: bug#919929) Drew Parsons <dparsons@debian.org> - 2019-03-08 03:10 +0100
              Re: python-scipy: autopkgtest fails (Re: bug#919929) Paul Gevers <elbrus@debian.org> - 2019-03-09 21:00 +0100
                Re: python-scipy: autopkgtest fails (Re: bug#919929) Drew Parsons <dparsons@debian.org> - 2019-03-10 08:50 +0100
                  Re: python-scipy: autopkgtest fails (Re: bug#919929) Drew Parsons <dparsons@debian.org> - 2019-03-11 07:50 +0100
                    unblock request: python-scipy/1.1.0-4 skimage/0.14.2-2: autopkgtest  passes (Re: bug#919929) Drew Parsons <dparsons@debian.org> - 2019-03-16 13:50 +0100
                      Re: unblock request: python-scipy/1.1.0-4 skimage/0.14.2-2:  autopkgtest passes (Re: bug#919929) Paul Gevers <elbrus@debian.org> - 2019-03-16 15:10 +0100
                        Re: unblock request: python-scipy/1.1.0-4 skimage/0.14.2-2:  autopkgtest passes (Re: bug#919929) Drew Parsons <dparsons@debian.org> - 2019-03-16 17:00 +0100

#11347 — python-scipy: autopkgtest fails (Re:bug#919929)

FromDrew Parsons <dparsons@debian.org>
Date2019-03-05 12:10 +0100
Subjectpython-scipy: autopkgtest fails (Re:bug#919929)
Message-ID<xyg7o-6lM-3@gated-at.bofh.it>
python-scipy has recently started failing all debci tests in testing and 
unstable, exacerbating the bug report in Bug#919929 [1].

The failing error is a MemoryError. But understanding the problem is 
hampered by a flood of deprecation warnings, presumably triggered by 
numpy 1.16. scipy 1.2 added instructions to pytest.ini to ignore the 
warnings.

Bug#919929 has not yet been marked RC but I guess it's about to happen.  
I propose in the first instance to apply a patch of the pytest.ini diff 
between scipy 1.1 and 1.2 and see if that clears things up. I'll commit 
the patch now. Should I proceed with upload to unstable?

Drew

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919929

[toc] | [next] | [standalone]


#11350 — Re: python-scipy: autopkgtest fails (Re: bug#919929)

FromAndreas Tille <tille@debian.org>
Date2019-03-07 09:10 +0100
SubjectRe: python-scipy: autopkgtest fails (Re: bug#919929)
Message-ID<xyWgh-7Dy-1@gated-at.bofh.it>
In reply to#11347
On Tue, Mar 05, 2019 at 07:01:54PM +0800, Drew Parsons wrote:
> python-scipy has recently started failing all debci tests in testing and
> unstable, exacerbating the bug report in Bug#919929 [1].
> 
> The failing error is a MemoryError. But understanding the problem is
> hampered by a flood of deprecation warnings, presumably triggered by numpy
> 1.16. scipy 1.2 added instructions to pytest.ini to ignore the warnings.
> 
> Bug#919929 has not yet been marked RC but I guess it's about to happen.  I
> propose in the first instance to apply a patch of the pytest.ini diff
> between scipy 1.1 and 1.2 and see if that clears things up. I'll commit the
> patch now. Should I proceed with upload to unstable?

May be its sensible to coordinate with debian-release (list in CC) and
file a unblock request *before* uploading.  I confirm that I usually
upload simple fixes and ask for unblock afterwards but you intend to do
a version change which does not qualify as simple fix (despite I agree
with you that it is sensible).

Kind regards

       Andreas.
 
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919929

-- 
http://fam-tille.de

[toc] | [prev] | [next] | [standalone]


#11351 — Re: python-scipy: autopkgtest fails (Re: bug#919929)

FromPaul Gevers <elbrus@debian.org>
Date2019-03-07 12:10 +0100
SubjectRe: python-scipy: autopkgtest fails (Re: bug#919929)
Message-ID<xyZ4t-XP-1@gated-at.bofh.it>
In reply to#11350

[Multipart message — attachments visible in raw view] — view raw

Hi Drew,

On 07-03-2019 09:03, Andreas Tille wrote:
> On Tue, Mar 05, 2019 at 07:01:54PM +0800, Drew Parsons wrote:
>> python-scipy has recently started failing all debci tests in testing and
>> unstable, exacerbating the bug report in Bug#919929 [1].
>>
>> The failing error is a MemoryError. But understanding the problem is
>> hampered by a flood of deprecation warnings, presumably triggered by numpy
>> 1.16. scipy 1.2 added instructions to pytest.ini to ignore the warnings.
>>
>> Bug#919929 has not yet been marked RC but I guess it's about to happen.

Can you elaborate why you think that bug should be RC (as that isn't
clear to me from the report itself) and why you haven't marked it as
such if you think it should be?

>> I
>> propose in the first instance to apply a patch of the pytest.ini diff
>> between scipy 1.1 and 1.2 and see if that clears things up. I'll commit the
>> patch now. Should I proceed with upload to unstable?
> 
> May be its sensible to coordinate with debian-release (list in CC) and
> file a unblock request *before* uploading.  I confirm that I usually
> upload simple fixes and ask for unblock afterwards but you intend to do
> a version change which does not qualify as simple fix (despite I agree
> with you that it is sensible).

Slightly depending on the answer above, I'll unblock an upload of
python-scipy with only that change.

Paul

[toc] | [prev] | [next] | [standalone]


#11352 — Re: python-scipy: autopkgtest fails (Re: bug#919929)

FromDrew Parsons <dparsons@debian.org>
Date2019-03-07 13:20 +0100
SubjectRe: python-scipy: autopkgtest fails (Re: bug#919929)
Message-ID<xz0ad-1Bh-1@gated-at.bofh.it>
In reply to#11351
On 2019-03-07 19:00, Paul Gevers wrote:
> Hi Drew,
> 
> On 07-03-2019 09:03, Andreas Tille wrote:
>> On Tue, Mar 05, 2019 at 07:01:54PM +0800, Drew Parsons wrote:
>>> python-scipy has recently started failing all debci tests in testing 
>>> and
>>> unstable, exacerbating the bug report in Bug#919929 [1].
>>> 
>>> The failing error is a MemoryError. But understanding the problem is
>>> hampered by a flood of deprecation warnings, presumably triggered by 
>>> numpy
>>> 1.16. scipy 1.2 added instructions to pytest.ini to ignore the 
>>> warnings.
>>> 
>>> Bug#919929 has not yet been marked RC but I guess it's about to 
>>> happen.
> 
> Can you elaborate why you think that bug should be RC (as that isn't
> clear to me from the report itself) and why you haven't marked it as
> such if you think it should be?

python-scipy is currently failing all debci tests in both unstable and 
testing.

I haven't marked it RC myself since I'm not 100% certain what the usual 
protocol is for marking the severity of debci test failures.  But as I 
understood it, debci test failures is considered RC under the final 
freeze which we're about to enter (but we're not quite in that deep 
freeze yet).


>>> I
>>> propose in the first instance to apply a patch of the pytest.ini diff
>>> between scipy 1.1 and 1.2 and see if that clears things up. I'll 
>>> commit the
>>> patch now. Should I proceed with upload to unstable?
>> 
>> May be its sensible to coordinate with debian-release (list in CC) and
>> file a unblock request *before* uploading.  I confirm that I usually
>> upload simple fixes and ask for unblock afterwards but you intend to 
>> do
>> a version change which does not qualify as simple fix (despite I agree
>> with you that it is sensible).

Hi Andreas, I may not have been clear.  What I mean at this point is to 
upload a small patch for scipy 1.1 to ignore the deprecation warnings 
(scipy 1.2 already does that).

If that doesn't help pass the debci tests then we can consider uploading 
scipy 1.2 instead.  But python-fluids and dipy FTBFS against scipy 1.2 
(a new version of dipy is available which presumeably fixes that)

> Slightly depending on the answer above, I'll unblock an upload of
> python-scipy with only that change.

Certainly please do unblock if the full freeze is already in place. But 
my intention was to first upload python-scipy 1.1 with a small patch, 
not 1.2 just yet.

Drew

[toc] | [prev] | [next] | [standalone]


#11353 — Re: python-scipy: autopkgtest fails (Re: bug#919929)

FromPaul Gevers <elbrus@debian.org>
Date2019-03-07 13:50 +0100
SubjectRe: python-scipy: autopkgtest fails (Re: bug#919929)
Message-ID<xz0Dg-1LB-5@gated-at.bofh.it>
In reply to#11352

[Multipart message — attachments visible in raw view] — view raw

Hi Drew,

On 07-03-2019 13:19, Drew Parsons wrote:
>> Can you elaborate why you think that bug should be RC (as that isn't
>> clear to me from the report itself) and why you haven't marked it as
>> such if you think it should be?
> 
> python-scipy is currently failing all debci tests in both unstable and
> testing.

autopkgtest only, so no FTBFS?

> I haven't marked it RC myself since I'm not 100% certain what the usual
> protocol is for marking the severity of debci test failures.  But as I
> understood it, debci test failures is considered RC under the final
> freeze which we're about to enter (but we're not quite in that deep
> freeze yet).

Some of us want failing autopkgtest to be RC *after* the release of
buster. I am not aware of consensus about that yet. autopkgtest
*regression* in testing is effectively RC since the soft freeze of
12-2-2019. The autopkgtest of python-scipy is already failing in
testing, so it isn't a regression. Hence, it failing is not RC for buster.

> Hi Andreas, I may not have been clear.  What I mean at this point is to
> upload a small patch for scipy 1.1 to ignore the deprecation warnings
> (scipy 1.2 already does that).
> 
> If that doesn't help pass the debci tests then we can consider uploading
> scipy 1.2 instead.  But python-fluids and dipy FTBFS against scipy 1.2
> (a new version of dipy is available which presumeably fixes that)

Please *don't* upload scipy 1.2 to fix only this issue (nor for any
other issue without discussion first), it's for sure not worth it.

>> Slightly depending on the answer above, I'll unblock an upload of
>> python-scipy with only that change.
> 
> Certainly please do unblock if the full freeze is already in place. But
> my intention was to first upload python-scipy 1.1 with a small patch,
> not 1.2 just yet.

If you upload now, your package will not migrate to testing before the
full freeze becomes effective so it would need an unblock. If you want
to fix this issue with the three lines I saw in the bug report, you can
go ahead. However, it is probably worth waiting for a resolution of bug
915738 and combine it with that.

Paul

[toc] | [prev] | [next] | [standalone]


#11354 — Re: python-scipy: autopkgtest fails (Re: bug#919929)

FromDrew Parsons <dparsons@debian.org>
Date2019-03-07 15:00 +0100
SubjectRe: python-scipy: autopkgtest fails (Re: bug#919929)
Message-ID<xz1J0-2pV-1@gated-at.bofh.it>
In reply to#11353
On 2019-03-07 20:46, Paul Gevers wrote:
> Hi Drew,
> 
> On 07-03-2019 13:19, Drew Parsons wrote:
>> 
>> python-scipy is currently failing all debci tests in both unstable and
>> testing.
> 
> autopkgtest only, so no FTBFS?

That's right, scipy builds fine.

> Some of us want failing autopkgtest to be RC *after* the release of
> buster. I am not aware of consensus about that yet. autopkgtest
> *regression* in testing is effectively RC since the soft freeze of
> 12-2-2019. The autopkgtest of python-scipy is already failing in
> testing, so it isn't a regression. Hence, it failing is not RC for 
> buster.

Thanks, that clarifies the appropriate severity.

>> Certainly please do unblock if the full freeze is already in place. 
>> But
>> my intention was to first upload python-scipy 1.1 with a small patch,
>> not 1.2 just yet.
> 
> If you upload now, your package will not migrate to testing before the
> full freeze becomes effective so it would need an unblock. If you want
> to fix this issue with the three lines I saw in the bug report, you can
> go ahead. However, it is probably worth waiting for a resolution of bug
> 915738 and combine it with that.

There hasn't been recent movement on 915738. I'll apply Julian's patch 
and see how we go.

Drew

[toc] | [prev] | [next] | [standalone]


#11355 — Re: python-scipy: autopkgtest fails (Re: bug#919929)

FromPaul Gevers <elbrus@debian.org>
Date2019-03-07 15:10 +0100
SubjectRe: python-scipy: autopkgtest fails (Re: bug#919929)
Message-ID<xz1SG-2Jt-3@gated-at.bofh.it>
In reply to#11354

[Multipart message — attachments visible in raw view] — view raw

Hi Drew,

On 07-03-2019 14:56, Drew Parsons wrote:
> On 2019-03-07 20:46, Paul Gevers wrote:
>> However, it is probably worth waiting for a resolution of bug
>> 915738 and combine it with that.
> 
> There hasn't been recent movement on 915738. I'll apply Julian's patch
> and see how we go.

Huh. There was a comment from doko that the underlying issue is fixed in
gcc-8, no? I think you only need to switch to the default gfortran. On
the other hand, I don't know how feasible it is at this moment to not
release buster with gcc-7. Maybe other release team members can comment
on that?

Paul

[toc] | [prev] | [next] | [standalone]


#11356 — Re: python-scipy: autopkgtest fails (Re: bug#919929)

FromDrew Parsons <dparsons@debian.org>
Date2019-03-07 15:30 +0100
SubjectRe: python-scipy: autopkgtest fails (Re: bug#919929)
Message-ID<xz2c2-2Qf-7@gated-at.bofh.it>
In reply to#11355
On 2019-03-07 22:07, Paul Gevers wrote:
> Hi Drew,
> 
> On 07-03-2019 14:56, Drew Parsons wrote:
>> On 2019-03-07 20:46, Paul Gevers wrote:
>>> However, it is probably worth waiting for a resolution of bug
>>> 915738 and combine it with that.
>> 
>> There hasn't been recent movement on 915738. I'll apply Julian's patch
>> and see how we go.
> 
> Huh. There was a comment from doko that the underlying issue is fixed 
> in
> gcc-8, no? I think you only need to switch to the default gfortran. On
> the other hand, I don't know how feasible it is at this moment to not
> release buster with gcc-7. Maybe other release team members can comment
> on that?

Perhaps it's ok now, as Matthias says.  Testing on zelenka now. I'll do 
the 919929 upload if s390x proves fine.

Drew

[toc] | [prev] | [next] | [standalone]


#11357 — Re: python-scipy: autopkgtest fails (Re: bug#919929)

FromDrew Parsons <dparsons@debian.org>
Date2019-03-08 03:10 +0100
SubjectRe: python-scipy: autopkgtest fails (Re: bug#919929)
Message-ID<xzd7r-1sE-1@gated-at.bofh.it>
In reply to#11353
On 2019-03-07 20:46, Paul Gevers wrote:
> 
> If you upload now, your package will not migrate to testing before the
> full freeze becomes effective so it would need an unblock. If you want
> to fix this issue with the three lines I saw in the bug report, you can
> go ahead. However, it is probably worth waiting for a resolution of bug
> 915738 and combine it with that.
> 

Alas, the deprecation patch (in python-scipy 1.1.0-3) doesn't actually 
prevent emission of the deprecation warnings, so they're still spamming 
the debci log.  On the bright side, s390x is now using gfortran-8 
successfully (#915738).

To remove the deprecation warnings we'd need to deal with them at the 
source. Upstream has patches
https://github.com/scipy/scipy/commit/614847c5fc8d5f8a618980df3c1b93540428ae46
https://github.com/scipy/scipy/commit/e0cfa29e2fbe86f994924c0c7514ff5bbfffd514
and for completeness
https://github.com/scipy/scipy/commit/87e48c3c54d7a85bc6628c88c1de98ac0469b6fa

The deprecation problem (matrix API) appears in many places, but the fix 
is straightfoward: replace np.matrix with matrix from from 
scipy.sparse.sputils

Can you authorise an unblock to apply these 3 upstream patches to 
python-scipy 1.1.0 ?
That won't necessarily fix the debci failure, but it will make it a lot 
easier to see what's actually failing.

Drew

[toc] | [prev] | [next] | [standalone]


#11359 — Re: python-scipy: autopkgtest fails (Re: bug#919929)

FromPaul Gevers <elbrus@debian.org>
Date2019-03-09 21:00 +0100
SubjectRe: python-scipy: autopkgtest fails (Re: bug#919929)
Message-ID<xzQit-M1-5@gated-at.bofh.it>
In reply to#11357

[Multipart message — attachments visible in raw view] — view raw

Hi Drew,

On 08-03-2019 03:08, Drew Parsons wrote:
> On 2019-03-07 20:46, Paul Gevers wrote:
>> If you upload now, your package will not migrate to testing before the
>> full freeze becomes effective so it would need an unblock. If you want
>> to fix this issue with the three lines I saw in the bug report, you can
>> go ahead. However, it is probably worth waiting for a resolution of bug
>> 915738 and combine it with that.
>>
> 
> Alas, the deprecation patch (in python-scipy 1.1.0-3) doesn't actually
> prevent emission of the deprecation warnings, so they're still spamming
> the debci log.

Do you have evidence they did anything at all? If not, please revert
this if we get to a new upload.

> On the bright side, s390x is now using gfortran-8
> successfully (#915738).

Well, at least some success then.

> To remove the deprecation warnings we'd need to deal with them at the
> source. Upstream has patches
> https://github.com/scipy/scipy/commit/614847c5fc8d5f8a618980df3c1b93540428ae46
> 
> https://github.com/scipy/scipy/commit/e0cfa29e2fbe86f994924c0c7514ff5bbfffd514
> 
> and for completeness
> https://github.com/scipy/scipy/commit/87e48c3c54d7a85bc6628c88c1de98ac0469b6fa
> 
> 
> The deprecation problem (matrix API) appears in many places, but the fix
> is straightfoward: replace np.matrix with matrix from from
> scipy.sparse.sputils
> 
> Can you authorise an unblock to apply these 3 upstream patches to
> python-scipy 1.1.0 ?
> That won't necessarily fix the debci failure, but it will make it a lot
> easier to see what's actually failing.

I am slightly unhappy with the second patch, as it seems to be doing
more than you describe above in a few places. This may be correct but
that is difficult to quickly judge.

Also, what is the general documented way that these wrappers can be
used? scipy is sort of taking over the maintenanceship of these
functions in this way if we're not careful.

Paul

[toc] | [prev] | [next] | [standalone]


#11360 — Re: python-scipy: autopkgtest fails (Re: bug#919929)

FromDrew Parsons <dparsons@debian.org>
Date2019-03-10 08:50 +0100
SubjectRe: python-scipy: autopkgtest fails (Re: bug#919929)
Message-ID<xA1nz-7Vg-1@gated-at.bofh.it>
In reply to#11359
On 2019-03-10 03:51, Paul Gevers wrote:
> Hi Drew,
> 
> On 08-03-2019 03:08, Drew Parsons wrote:
>> On 2019-03-07 20:46, Paul Gevers wrote:
>>> If you upload now, your package will not migrate to testing before 
>>> the
>>> full freeze becomes effective so it would need an unblock. If you 
>>> want
>>> to fix this issue with the three lines I saw in the bug report, you 
>>> can
>>> go ahead. However, it is probably worth waiting for a resolution of 
>>> bug
>>> 915738 and combine it with that.
>>> 
>> 
>> Alas, the deprecation patch (in python-scipy 1.1.0-3) doesn't actually
>> prevent emission of the deprecation warnings, so they're still 
>> spamming
>> the debci log.
> 
> Do you have evidence they did anything at all? If not, please revert
> this if we get to a new upload.

It would seem it did not help.  In any case, the upstream patches 
supercede this patch, so it will be removed naturally.

>> To remove the deprecation warnings we'd need to deal with them at the
>> source. Upstream has patches
>> https://github.com/scipy/scipy/commit/614847c5fc8d5f8a618980df3c1b93540428ae46
>> 
>> https://github.com/scipy/scipy/commit/e0cfa29e2fbe86f994924c0c7514ff5bbfffd514
>> 
>> and for completeness
>> https://github.com/scipy/scipy/commit/87e48c3c54d7a85bc6628c88c1de98ac0469b6fa
>> 
>> 
>> The deprecation problem (matrix API) appears in many places, but the 
>> fix
>> is straightfoward: replace np.matrix with matrix from from
>> scipy.sparse.sputils
>> 
>> Can you authorise an unblock to apply these 3 upstream patches to
>> python-scipy 1.1.0 ?
>> That won't necessarily fix the debci failure, but it will make it a 
>> lot
>> easier to see what's actually failing.
> 
> I am slightly unhappy with the second patch, as it seems to be doing
> more than you describe above in a few places. This may be correct but
> that is difficult to quickly judge.

The patches as they are don't apply cleanly to the 1.1.0 source, so I'll 
need to adapt them anyway.  I can retain only the ones relevant to 
updating the matrix API.


> Also, what is the general documented way that these wrappers can be
> used? scipy is sort of taking over the maintenanceship of these
> functions in this way if we're not careful.


It's a good question that the other scipy maintainers might have thought 
more about.  As far as I can tell, the scipy tests affected here involve 
sparse matrices.  The trouble arises from an "inadequacy" in the core 
numpy API, with numpy.matrix only being suitable for dense matrices.  
scipy could be described as "numpy+algorithms", with additional 
algorithms required to handle sparse matrices, provided in 
scipy.sparse.sputils.matrix.

numpy.matrix is documented at 
https://docs.scipy.org/doc/numpy/reference/generated/numpy.matrix.html

The scipy sparse matrix API is at 
https://docs.scipy.org/doc/scipy/reference/sparse.html, but that's 
specifically for scipy.sparse.spmatrix

There is discussion of the distinction between numpy.matrix and 
numpy.ndarray (which is at the heart of the deeprecation warning) at 
https://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html#numpy-matrix-vs-2d-numpy-ndarray

The utility class scipy.sparse.sputils itself is apparently 
undocumented, by which I infer it's intended for internal use only, not 
a public API. I guess it's reasonable for a package to be testing it's 
own internal functions.  Strange thing is, scipy.sparse.sputils.matrix 
is not actually defined in scipy/sparse/sputils.py. Must be inherited or 
defined in some deep pythonfu that I haven't mastered yet.

I'll check that I can adapt those upstream patches to cleanly remove 
these deprecation warnings.

Drew

[toc] | [prev] | [next] | [standalone]


#11361 — Re: python-scipy: autopkgtest fails (Re: bug#919929)

FromDrew Parsons <dparsons@debian.org>
Date2019-03-11 07:50 +0100
SubjectRe: python-scipy: autopkgtest fails (Re: bug#919929)
Message-ID<xAmV3-4Jm-1@gated-at.bofh.it>
In reply to#11360
On 2019-03-10 15:46, Drew Parsons wrote:
> On 2019-03-10 03:51, Paul Gevers wrote:
>> Hi Drew,
>> 
> 
>>> To remove the deprecation warnings we'd need to deal with them at the
>>> source. Upstream has patches
>>> https://github.com/scipy/scipy/commit/614847c5fc8d5f8a618980df3c1b93540428ae46
>>> 
>>> https://github.com/scipy/scipy/commit/e0cfa29e2fbe86f994924c0c7514ff5bbfffd514
>>> 
>>> and for completeness
>>> https://github.com/scipy/scipy/commit/87e48c3c54d7a85bc6628c88c1de98ac0469b6fa
...
>>> Can you authorise an unblock to apply these 3 upstream patches to
>>> python-scipy 1.1.0 ?
>>> That won't necessarily fix the debci failure, but it will make it a 
>>> lot
>>> easier to see what's actually failing.
>> 
>> I am slightly unhappy with the second patch, as it seems to be doing
>> more than you describe above in a few places. This may be correct but
>> that is difficult to quickly judge.

I've adapted the 3 patches and pushed to salsa,
    matrix_API_614847c5.patch
    matrix_API_more_e0cfa29e2.patch
    matrix_API_filter_check_87e48c3c5.patch
https://salsa.debian.org/python-team/modules/python-scipy/tree/master/debian/patches

The other behaviour that you saw in the second patch I think might be 
replacement of "D*diag(v)" with "D@diag(v)". That's matrix 
multiplication, so still relevent to the matrix API patch. @ is not 
available in python2 so I changed it to numpy.matmul, which does the 
same thing.

The numpy.sparse tests pass with this patch, and most of the matrix 
PendingDeprecationWarnings are gone (the upstream patch missed 
integrate/tests/test_ivp.py, but the remaining warnings are few enough 
to not need to worry about).


>> Also, what is the general documented way that these wrappers can be
>> used? scipy is sort of taking over the maintenanceship of these
>> functions in this way if we're not careful.
> 
...
> There is discussion of the distinction between numpy.matrix and
> numpy.ndarray (which is at the heart of the deeprecation warning) at
> https://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html#numpy-matrix-vs-2d-numpy-ndarray
> 
> The utility class scipy.sparse.sputils itself is apparently
> undocumented, by which I infer it's intended for internal use only,
> not a public API. I guess it's reasonable for a package to be testing
> it's own internal functions.  Strange thing is,
> scipy.sparse.sputils.matrix is not actually defined in
> scipy/sparse/sputils.py. Must be inherited or defined in some deep
> pythonfu that I haven't mastered yet.


Actually scipy.sparse.sputils.matrix was defined in these patches.  It 
is a bit odd that upstream is wrapping numpy.matrix just to avoid 
deprecation warnings, rather than following their own advice and using 
numpy.ndarray instead.


With these patches, the sparse matrix tests pass. There remain three 
errors unrelated to sparse matrix:
   spatial/tests/test__plotutils.py::TestPlotting::test_delaunay FAILED   
   [ 76%]
   spatial/tests/test__plotutils.py::TestPlotting::test_voronoi FAILED    
   [ 76%]
   spatial/tests/test__plotutils.py::TestPlotting::test_convex_hull 
FAILED  [ 76%]
   __________________________ TestPlotting.test_delaunay 
__________________________

   self = <scipy.spatial.tests.test__plotutils.TestPlotting object at 
0x7f0f31156eb8>

       def test_delaunay(self):
           # Smoke test
           fig = plt.figure()
           obj = Delaunay(self.points)
           s_before = obj.simplices.copy()
   >       with suppress_warnings as sup:
   E       AttributeError: __enter__

   fig        = <Figure size 640x480 with 0 Axes>
   obj        = <scipy.spatial.qhull.Delaunay object at 0x7f0f31163400>
   s_before   = array([[3, 1, 0],
          [2, 3, 0]], dtype=int32)
   self       = <scipy.spatial.tests.test__plotutils.TestPlotting object 
at 0x7f0f31156eb8>

   
/usr/lib/python3/dist-packages/scipy/spatial/tests/test__plotutils.py:31: 
AttributeError

(likewise for the other 2)
These AttributeErrors are mentioned at 
https://github.com/scipy/scipy/issues/9491. Upstream doesn't seem too 
concerned about them.

Apart from that, I'm happy to upload the sparse matrix patches once the 
s390x update reaches testing.

Drew

[toc] | [prev] | [next] | [standalone]


#11372 — unblock request: python-scipy/1.1.0-4 skimage/0.14.2-2: autopkgtest passes (Re: bug#919929)

FromDrew Parsons <dparsons@debian.org>
Date2019-03-16 13:50 +0100
Subjectunblock request: python-scipy/1.1.0-4 skimage/0.14.2-2: autopkgtest passes (Re: bug#919929)
Message-ID<xCgVb-46g-5@gated-at.bofh.it>
In reply to#11361
On 2019-03-11 14:39, Drew Parsons wrote:
> 
> I've adapted the 3 patches and pushed to salsa,
>    matrix_API_614847c5.patch
>    matrix_API_more_e0cfa29e2.patch
>    matrix_API_filter_check_87e48c3c5.patch
> https://salsa.debian.org/python-team/modules/python-scipy/tree/master/debian/patches
...
> The numpy.sparse tests pass with this patch, and most of the matrix
> PendingDeprecationWarnings are gone (the upstream patch missed
> integrate/tests/test_ivp.py, but the remaining warnings are few enough
> to not need to worry about).

Well, turns out the other warnings worried Aurelien enough to file 
Bug#924396.

Is there enough will to add more scipy patches for the buster release to 
reduce the remaining DeprecationWarnings? (they don't break tests, 
they're just annoying)
Or should we just let it go at this point and let them get cleared in 
future versions?)

...
> 
> With these patches, the sparse matrix tests pass. There remain three
> errors unrelated to sparse matrix:
>   spatial/tests/test__plotutils.py::TestPlotting::test_delaunay FAILED
>     [ 76%]
>   spatial/tests/test__plotutils.py::TestPlotting::test_voronoi FAILED
>     [ 76%]
>   spatial/tests/test__plotutils.py::TestPlotting::test_convex_hull
> FAILED  [ 76%]
,,,
>   >       with suppress_warnings as sup:
>   E       AttributeError: __enter__
> 
> Apart from that, I'm happy to upload the sparse matrix patches once
> the s390x update reaches testing.


Those errors must have been local to me.  scipy 1.1.0-4 now passes debci 
tests cleanly.

That being the case, in the interests of making a stable release that 
passes it own tests, I'd like to request an unblock for 
python-scipy/1.1.0-4 (together with skimage/0.14.2-2)

Drew

[toc] | [prev] | [next] | [standalone]


#11373 — Re: unblock request: python-scipy/1.1.0-4 skimage/0.14.2-2: autopkgtest passes (Re: bug#919929)

FromPaul Gevers <elbrus@debian.org>
Date2019-03-16 15:10 +0100
SubjectRe: unblock request: python-scipy/1.1.0-4 skimage/0.14.2-2: autopkgtest passes (Re: bug#919929)
Message-ID<xCiaB-53T-1@gated-at.bofh.it>
In reply to#11372

[Multipart message — attachments visible in raw view] — view raw

Hi Drew,

On 16-03-2019 13:48, Drew Parsons wrote:
>> The numpy.sparse tests pass with this patch, and most of the matrix
>> PendingDeprecationWarnings are gone (the upstream patch missed
>> integrate/tests/test_ivp.py, but the remaining warnings are few enough
>> to not need to worry about).
> 
> Well, turns out the other warnings worried Aurelien enough to file
> Bug#924396.
> 
> Is there enough will to add more scipy patches for the buster release to
> reduce the remaining DeprecationWarnings? (they don't break tests,
> they're just annoying)
> Or should we just let it go at this point and let them get cleared in
> future versions?)

I'd let it be for now.

> That being the case, in the interests of making a stable release that
> passes it own tests, I'd like to request an unblock for
> python-scipy/1.1.0-4 (together with skimage/0.14.2-2)

skimage was already unblocked. I'll unblock python-scipy as well.

Paul

[toc] | [prev] | [next] | [standalone]


#11374 — Re: unblock request: python-scipy/1.1.0-4 skimage/0.14.2-2: autopkgtest passes (Re: bug#919929)

FromDrew Parsons <dparsons@debian.org>
Date2019-03-16 17:00 +0100
SubjectRe: unblock request: python-scipy/1.1.0-4 skimage/0.14.2-2: autopkgtest passes (Re: bug#919929)
Message-ID<xCjT3-5VQ-1@gated-at.bofh.it>
In reply to#11373
On 2019-03-16 22:07, Paul Gevers wrote:
> On 16-03-2019 13:48, Drew Parsons wrote:
>> 
>> Is there enough will to add more scipy patches for the buster release 
>> to
>> reduce the remaining DeprecationWarnings? (they don't break tests,
>> they're just annoying)
>> Or should we just let it go at this point and let them get cleared in
>> future versions?)
> 
> I'd let it be for now.

No problem, will do (more precisely, not do).


>> That being the case, in the interests of making a stable release that
>> passes it own tests, I'd like to request an unblock for
>> python-scipy/1.1.0-4 (together with skimage/0.14.2-2)
> 
> skimage was already unblocked. I'll unblock python-scipy as well.


Thanks Paul.  Looks like we'll be able to close Bug#919929 once 
python-scipy/1.1.0-4 is settled into testing.

Drew

[toc] | [prev] | [standalone]


Back to top | Article view | linux.debian.maint.python


csiph-web