Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #14769 > unrolled thread

Re: Python 3.11 for bookworm?

Back to article view | Back to linux.debian.maint.python

Contents

  Re: Python 3.11 for bookworm? Matthias Klose <doko@ubuntu.com> - 2022-12-21 19:30 +0100
    Re: Python 3.11 for bookworm? Sandro Tosi <morph@debian.org> - 2022-12-22 01:20 +0100
      Re: Python 3.11 for bookworm? Nicholas D Steeves <sten@debian.org> - 2022-12-22 02:30 +0100
        Re: Python 3.11 for bookworm? "M. Zhou" <lumin@debian.org> - 2022-12-22 06:10 +0100
        Re: Python 3.11 for bookworm? Thomas Goirand <zigo@debian.org> - 2022-12-28 02:00 +0100
          Re: Python 3.11 for bookworm? Andreas Tille <andreas@an3as.eu> - 2022-12-30 10:30 +0100
            Re: Python 3.11 for bookworm? Thomas Goirand <zigo@debian.org> - 2023-01-05 14:00 +0100
              Re: Python 3.11 for bookworm? Andreas Tille <andreas@an3as.eu> - 2023-01-07 09:10 +0100
                Re: Python 3.11 for bookworm? Andrius Merkys <merkys@debian.org> - 2023-01-07 09:30 +0100
                  Re: Python 3.11 for bookworm? Andrius Merkys <merkys@debian.org> - 2023-01-09 16:50 +0100
                Re: Python 3.11 for bookworm? Andreas Tille <andreas@an3as.eu> - 2023-01-16 17:30 +0100
                  Re: Python 3.11 for bookworm? Thomas Goirand <zigo@debian.org> - 2023-01-17 01:10 +0100
                    Re: Python 3.11 for bookworm? Andreas Tille <tille@debian.org> - 2023-01-17 07:50 +0100
              Re: Python 3.11 for bookworm? Étienne Mollier <emollier@emlwks999.eu> - 2023-01-10 22:50 +0100
      Re: Python 3.11 for bookworm? Stefano Rivera <stefanor@debian.org> - 2022-12-22 13:50 +0100
        Re: Python 3.11 for bookworm? Timo Röhling <roehling@debian.org> - 2022-12-22 14:00 +0100
          Re: Python 3.11 for bookworm? Stefano Rivera <stefanor@debian.org> - 2022-12-22 19:50 +0100
            Re: Python 3.11 for bookworm? "M. Zhou" <lumin@debian.org> - 2022-12-23 00:30 +0100
              Re: Python 3.11 for bookworm? "M. Zhou" <lumin@debian.org> - 2022-12-23 00:40 +0100
            Re: Python 3.11 for bookworm? Graham Inggs <ginggs@debian.org> - 2022-12-26 21:30 +0100
    Re: Python 3.11 for bookworm? Graham Inggs <ginggs@debian.org> - 2022-12-26 21:50 +0100

Page 1 of 2  [1] 2  Next page →

#14769 — Re: Python 3.11 for bookworm?

while we have not an 100% agreement to go ahead, I think we should aim for 3.11.

The following steps would be:

  - accept the current python3-defaults into
    testing (adding 3.11 as supported)
  - ask for a transition slot to upload (see #1026825)
    python3-defaults with 3.11 as the default
  - start the no-change NMUs
  - file bug reports.
  - fix issues
  - let 3.11 as default migrate to testing.

If things don't go as planned, we could default back to 3.10.  Mostly that would 
be no-change uploads, however in the case of a 3.11 specific fix that doesn't 
work for 3.10, these fixes would need reverting.  I have no idea who many of 
these we will introduce with this transition.

Also we might want to ask for some freeze exceptions for third party libraries, 
that we can't fix before the feature freeze, unfortunately at this point we 
cannot say which and how many packages would be affected.

Matthias

[toc] | [next] | [standalone]

#14770

thoughts from a concerned maintainer

On Wed, Dec 21, 2022 at 1:24 PM Matthias Klose <doko@ubuntu.com> wrote:
>
> while we have not an 100% agreement to go ahead, I think we should aim for 3.11.
>
> The following steps would be:
>
>   - accept the current python3-defaults into
>     testing (adding 3.11 as supported)
>   - ask for a transition slot to upload (see #1026825)
>     python3-defaults with 3.11 as the default
>   - start the no-change NMUs
>   - file bug reports.
>   - fix issues
>   - let 3.11 as default migrate to testing.
>
> If things don't go as planned, we could default back to 3.10.  Mostly that would
> be no-change uploads, however in the case of a 3.11 specific fix that doesn't
> work for 3.10, these fixes would need reverting.  I have no idea who many of
> these we will introduce with this transition.
>
> Also we might want to ask for some freeze exceptions for third party libraries,
> that we can't fix before the feature freeze, unfortunately at this point we
> cannot say which and how many packages would be affected.

from expressions like

* "If things don't go as planned"
* "no idea who many of these we will introduce with this transition."
* "cannot say which and how many packages would be affected"

it seems this email advocates for a "let's wing it"[1] type of transition.

[1] https://en.wiktionary.org/wiki/wing_it

It appears there has been little work in preparing the work to
introduce python3.11 from its maintainer, instead that works has been
pushed downstream to maintainers.

if we continue with the plan as described above, several python
libraries/applications maintainers will be left with the short end of
the stick and deal with an unknown amount of issues (upstream fixes
not available, not ready and or/ not released, rushed, etc) with less
than a month from the beginning of the transition freeze[2]

[2] https://release.debian.org/bullseye/freeze_policy.html

[2] also highlights at the very beginning "Plan your changes for
bullseye", this change appears as if it was not planned and we should
be skeptical to proceed without further (and in advance) understanding
of the impact it may have on Bullseye.

Regards,
-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

[toc] | [prev] | [next] | [standalone]

#14771

[Multipart message — attachments visible in raw view] — view raw

Sandro Tosi <morph@debian.org> writes:

> thoughts from a concerned maintainer
>

Sandro, thank you for writing this email.

>
> it seems this email advocates for a "let's wing it"[1] type of transition.
>
> [1] https://en.wiktionary.org/wiki/wing_it
>
> It appears there has been little work in preparing the work to
> introduce python3.11 from its maintainer, instead that works has been
> pushed downstream to maintainers.
>
> if we continue with the plan as described above, several python
> libraries/applications maintainers will be left with the short end of
> the stick and deal with an unknown amount of issues (upstream fixes
> not available, not ready and or/ not released, rushed, etc) with less
> than a month from the beginning of the transition freeze[2]
>

Agreed. At a bare minimum, complete data from ratt (Rebuild All The
Things) should be required at this point.

> [2] https://release.debian.org/bullseye/freeze_policy.html
>
> [2] also highlights at the very beginning "Plan your changes for
> bullseye", this change appears as if it was not planned and we should
> be skeptical to proceed without further (and in advance) understanding
> of the impact it may have on Bullseye.
>

100% +1  I'm especially concerned about how a clear plan was not
communicated to other teams--whose work will be broken by the proposed
transition, were an exception to be granted.

Debian is not a paragon of community if it makes late, unannounced
changes that result in a yet-undetermined number of projects being
dropped from bookworm's release.  If Python 3.11 as the only supported
version is a release goal, then the freeze schedule would need to be
modified.

Regards,
Nicholas

[toc] | [prev] | [next] | [standalone]

#14772

There is not yet an accurate estimate of time required to fix the
python packages during the transition -- and I still remember the
transition from python3.9 -> python3.10 took a very long period
that does not seem short enough to be covered by the freeze schedule.

Apart from that, package maintainers have their own plans as well.
I believe that at the current stage, many people have assumed that
the next stable will ship python3.10, and have their packages
finalized for freeze already. Making that transition at the current
stage will push a number of maintainers into a rush of updating
their packages again -- in the worst case, the package upstreams
might be not even ready for python 3.11.

A significant Python performance improvement in 3.11 is good.
But note, when python performance has really become an issue,
people already have mature solutions, e.g. offloading the
performance critical part onto a compiled language.

I think the risk of greatly breaking the release plan
outweighs the gain.

On Wed, 2022-12-21 at 20:21 -0500, Nicholas D Steeves wrote:
> Sandro Tosi <morph@debian.org> writes:
> 
> > thoughts from a concerned maintainer
> > 
> 
> Sandro, thank you for writing this email.
> 
> > 
> > it seems this email advocates for a "let's wing it"[1] type of
> > transition.
> > 
> > [1] https://en.wiktionary.org/wiki/wing_it
> > 
> > It appears there has been little work in preparing the work to
> > introduce python3.11 from its maintainer, instead that works has
> > been
> > pushed downstream to maintainers.
> > 
> > if we continue with the plan as described above, several python
> > libraries/applications maintainers will be left with the short end
> > of
> > the stick and deal with an unknown amount of issues (upstream fixes
> > not available, not ready and or/ not released, rushed, etc) with
> > less
> > than a month from the beginning of the transition freeze[2]
> > 
> 
> Agreed. At a bare minimum, complete data from ratt (Rebuild All The
> Things) should be required at this point.
> 
> > [2] https://release.debian.org/bullseye/freeze_policy.html
> > 
> > [2] also highlights at the very beginning "Plan your changes for
> > bullseye", this change appears as if it was not planned and we
> > should
> > be skeptical to proceed without further (and in advance)
> > understanding
> > of the impact it may have on Bullseye.
> > 
> 
> 100% +1  I'm especially concerned about how a clear plan was not
> communicated to other teams--whose work will be broken by the
> proposed
> transition, were an exception to be granted.
> 
> Debian is not a paragon of community if it makes late, unannounced
> changes that result in a yet-undetermined number of projects being
> dropped from bookworm's release.  If Python 3.11 as the only
> supported
> version is a release goal, then the freeze schedule would need to be
> modified.
> 
> Regards,
> Nicholas

[toc] | [prev] | [next] | [standalone]

#14786

[Multipart message — attachments visible in raw view] — view raw

On 12/22/22 02:21, Nicholas D Steeves wrote:
> 100% +1  I'm especially concerned about how a clear plan was not
> communicated to other teams--whose work will be broken by the proposed
> transition, were an exception to be granted.
> 
> Debian is not a paragon of community if it makes late, unannounced
> changes that result in a yet-undetermined number of projects being
> dropped from bookworm's release.  If Python 3.11 as the only supported
> version is a release goal, then the freeze schedule would need to be
> modified.
> 
> Regards,
> Nicholas

Hi Nicholas,

While I can agree that it may be harsh, and that some packages wont be 
fixed in time, I can't let you say that there was only 1 month given, 
and that all of this comes as a surprise. We've been talking about this 
since last summer.

On 12/22/22 05:48, M. Zhou wrote:
 > A significant Python performance improvement in 3.11 is good.
 > But note, when python performance has really become an issue,
 > people already have mature solutions, e.g. offloading the
 > performance critical part onto a compiled language.

I don't agree with this either.

I'm running large-ish OpenStack swift clusters, where we run up to 18 
physical nodes as proxies (eating 100s of requests per seconds). Even a 
10% gain would be greatly appreciated for us. There's no "C" improvement 
here, it's fully in Python... I tried running all Swift services over 
uwsgi, but it didn't work well (we reverted this type of setup because 
many things broke).

The fact that a new python process can spawn faster is also really a 
good thing (so that's not only the interpreter pure speed that I would 
like to have).

On 12/22/22 05:48, M. Zhou wrote:
 > Apart from that, package maintainers have their own plans as well.
 > I believe that at the current stage, many people have assumed that
 > the next stable will ship python3.10, and have their packages
 > finalized for freeze already. Making that transition at the current
 > stage will push a number of maintainers into a rush of updating
 > their packages again -- in the worst case, the package upstreams
 > might be not even ready for python 3.11.

Well, that's not the first time we are trying to push the latest 
interpreter close to a release. In fact, this happens on nearly each 
freeze. Yes, sometimes, there's no upstream fixes yet and you have to 
write your own. But that's what we do: we maintain packages... and fix 
bugs when we can.

Since last summer, I fixed maybe about 20 to 30 Python 3.11 issues, 
sometimes with, and sometimes without help from upstream. And I have 
about a dozen more to fix in my TODO (see below). I invite everyone to 
try to do the same, so we can reach that goal.

I also very much would appreciate help on these (which all look like 
probably related to py3.11):

#1026524 ironic-inspector: FTBFS: AttributeError: '_thread.RLock' object 
has no attribute 'is_locked'
#1026547 python-pypowervm: FTBFS: AssertionError: Expected 'warn' to be 
called once. Called 2 times.
#1026549 python-pytest-xprocess: FTBFS: tests failed
#1026591 cinder: FTBFS: make[1]: pyversions: No such file or directory
#1026595 python-murano-pkg-check: FTBFS: TypeError: load_all() missing 1 
required positional argument: 'Loader'
#1026610 horizon: FTBFS: tests failed
#1026691 bandit: FTBFS: TypeError: load() missing 1 required positional 
argument: 'Loader'
#1026693 cloudkitty: FTBFS: touch: cannot touch 
'/<<PKGBUILDDIR>>/debian/cloudkitty-doc/usr/share/doc/cloudkitty-doc/html/_static/toggle.js': 
No such file or directory
#1026702 python-cursive: FTBFS: AttributeError: '_RSAPrivateKey' object 
has no attribute 'signer'
#1026705 python-pecan: FTBFS: E AttributeError: 'code' object has no 
attribute 'co_endlinetable'
#1026707 jenkins-job-builder: FTBFS: E: pybuild pybuild:386: test: 
plugin custom failed with: exit code=1: PYTHON=python3.10 stestr run

BTW, thanks a lot to Lucas Nussbaum for doing the archive rebuilt and 
finding these out early!

To sum-up: while I'm not 100% on the side of breaking things that close 
to a release, and would also feel it very painful if one of the above 
bugs isn't fixed in time, I don't feel like you guys are giving good 
point of argumentation, or a solution to improve the process. Doko 
already explained that switching the interpreter (the hard way) is the 
only viable way to find out the remaining bugs. Do you have a better 
solution in mind?

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]

#14789

Am Wed, Dec 28, 2022 at 01:54:13AM +0100 schrieb Thomas Goirand:
> 
> While I can agree that it may be harsh, and that some packages wont be fixed
> in time, I can't let you say that there was only 1 month given, and that all
> of this comes as a surprise. We've been talking about this since last
> summer.

True, I remember the DebConf Python BoF.  My memory tells me that the
plan was to keep 3.10 as default.  Thus Python 3.11 would be really a
surprise.  From a maintainers team with lots of Python packages that
will need heavy work I can't say I'd be happy about a "migrate and see
what might happen afterwards" strategy as it was proposed here.  We have
lots to do even without such an experiment.
 
> Well, that's not the first time we are trying to push the latest interpreter
> close to a release.

The fact that we did so before does not make the idea better.

> I also very much would appreciate help on these (which all look like
> probably related to py3.11):
> 
> #1026524 ironic-inspector: FTBFS: AttributeError: '_thread.RLock' object has
> no attribute 'is_locked'
> #1026547 python-pypowervm: FTBFS: AssertionError: Expected 'warn' to be
> called once. Called 2 times.
> #1026549 python-pytest-xprocess: FTBFS: tests failed
> #1026591 cinder: FTBFS: make[1]: pyversions: No such file or directory
> #1026595 python-murano-pkg-check: FTBFS: TypeError: load_all() missing 1
> required positional argument: 'Loader'
> #1026610 horizon: FTBFS: tests failed
> #1026691 bandit: FTBFS: TypeError: load() missing 1 required positional
> argument: 'Loader'
> #1026693 cloudkitty: FTBFS: touch: cannot touch '/<<PKGBUILDDIR>>/debian/cloudkitty-doc/usr/share/doc/cloudkitty-doc/html/_static/toggle.js':
> No such file or directory
> #1026702 python-cursive: FTBFS: AttributeError: '_RSAPrivateKey' object has
> no attribute 'signer'
> #1026705 python-pecan: FTBFS: E AttributeError: 'code' object has no
> attribute 'co_endlinetable'
> #1026707 jenkins-job-builder: FTBFS: E: pybuild pybuild:386: test: plugin
> custom failed with: exit code=1: PYTHON=python3.10 stestr run

If I would create a list mine whould be way longer.  (The list of
successful fixes we did in the Debian Med team is also quite
impressive.)  We are constantly beaten by removal from testing warnings
even with Python3.11 as an option and sometimes we simply need to remove
that option as a temporary means for bookworm.
 
> To sum-up: while I'm not 100% on the side of breaking things that close to a
> release, and would also feel it very painful if one of the above bugs isn't
> fixed in time, I don't feel like you guys are giving good point of
> argumentation, or a solution to improve the process. Doko already explained
> that switching the interpreter (the hard way) is the only viable way to find
> out the remaining bugs. Do you have a better solution in mind?

No better solution but better timing which means after bookworm release. 

Kind regards
   Andreas.


-- 
http://fam-tille.de

[toc] | [prev] | [next] | [standalone]

#14798

Hi Andreas,

On 12/30/22 10:22, Andreas Tille wrote:
> True, I remember the DebConf Python BoF.  My memory tells me that the
> plan was to keep 3.10 as default.  Thus Python 3.11 would be really a
> surprise.  From a maintainers team with lots of Python packages that
> will need heavy work I can't say I'd be happy about a "migrate and see
> what might happen afterwards" strategy as it was proposed here.  We have
> lots to do even without such an experiment.

This has since already been discussed here: the final decision was to 
"at least try 3.11", which is exactly what we're doing.

>   
>> Well, that's not the first time we are trying to push the latest interpreter
>> close to a release.
> 
> The fact that we did so before does not make the idea better.
> 
>> I also very much would appreciate help on these (which all look like
>> probably related to py3.11):
>>
>> #1026524 ironic-inspector: FTBFS: AttributeError: '_thread.RLock' object has
>> no attribute 'is_locked'
>> #1026547 python-pypowervm: FTBFS: AssertionError: Expected 'warn' to be
>> called once. Called 2 times.
>> #1026549 python-pytest-xprocess: FTBFS: tests failed
>> #1026591 cinder: FTBFS: make[1]: pyversions: No such file or directory
>> #1026595 python-murano-pkg-check: FTBFS: TypeError: load_all() missing 1
>> required positional argument: 'Loader'
>> #1026610 horizon: FTBFS: tests failed
>> #1026691 bandit: FTBFS: TypeError: load() missing 1 required positional
>> argument: 'Loader'
>> #1026693 cloudkitty: FTBFS: touch: cannot touch '/<<PKGBUILDDIR>>/debian/cloudkitty-doc/usr/share/doc/cloudkitty-doc/html/_static/toggle.js':
>> No such file or directory
>> #1026702 python-cursive: FTBFS: AttributeError: '_RSAPrivateKey' object has
>> no attribute 'signer'
>> #1026705 python-pecan: FTBFS: E AttributeError: 'code' object has no
>> attribute 'co_endlinetable'
>> #1026707 jenkins-job-builder: FTBFS: E: pybuild pybuild:386: test: plugin
>> custom failed with: exit code=1: PYTHON=python3.10 stestr run

FYI, I'm down with only 2 major bugs (I don't mind much if the other 3 
RC bugs push the 3 affected packages away from the release, it's just a 
"would be nice" thing ATM):

#1026524 ironic-inspector: FTBFS: AttributeError: '_thread.RLock' object 
has no attribute 'is_locked'
#1026591 cinder: FTBFS: make[1]: pyversions: No such file or directory

The first one also appears in Python 3.10.9, but seems not present in 
Python 3.10.6, as per a discussion with upstream on IRC. It looks like 
the default threading thingy has changed a little bit, leading to 
this... I know already how to fix the actual executed code, but this 
leaves some wrong tests (assert called), so I'm waiting to see if 
upstream can do better than me.

For the Cinder bug, it looks like a test-only issue, which upstream is 
already working on. Worst case: blacklist these 250 failing tests, which 
is better than what it sounds (ie: just like Ironic, the actual 
production code is ok...).

> If I would create a list mine whould be way longer.

Please share it in this list!

> We are constantly beaten by removal from testing warnings
> even with Python3.11 as an option and sometimes we simply need to remove
> that option as a temporary means for bookworm.

Same over here. It's finally looking good for me after 2 months of heavy 
efforts.

> No better solution but better timing which means after bookworm release.

I've read *many* people saying it would be disappointing for them to see 
their package removed because of Python 3.11. Well, please consider that 
it would also be very disappointing to *not* have Python 3.11 for those 
who managed constantly fix issues for it.

The timing was exactly what was discussed during Debconf: it's very 
annoying that this year, upstream Python release was one month late... 
we're only trying to deal with it.

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]

#14802

Hi Thomas,

Am Thu, Jan 05, 2023 at 01:57:43PM +0100 schrieb Thomas Goirand:
> 
> This has since already been discussed here: the final decision was to "at
> least try 3.11", which is exactly what we're doing.

I admit I was not at site and may be I missunderstood what was finally
decided.  From my perspective this "at least try" is that we are
actually trying by having 3.11 as additional supported version which has
triggered quite some work.  We are approaching the Transition and
Toolchain Freeze in 5 days[1].  Given that switching default Python is a
transition I wonder how you can assume that this is the right time to
suggest this.  I would not have been that astonished if you would have
done so say at first December last year.  But now its absolutely clear
that a migration (with the "option" to revert which will cause extra
work) will pour sand into the gears of the release process.
 
> FYI, I'm down with only 2 major bugs (I don't mind much if the other 3 RC
> bugs push the 3 affected packages away from the release, it's just a "would
> be nice" thing ATM):

That's a nice situation for the field you are working in.
 
> > If I would create a list mine whould be way longer.
> 
> Please share it in this list!

   #1023965 [src:pandas] pandas FTBFS with Python 3.11 as supported version
   #1024031 [src:numba] numba FTBFS with Python 3.11 as supported version

These packages have a sufficient amount of rdepends and usually trigger
lots of other autopkgtest failures in other packages which will keep
them out of testing for quite some time.  We could need all helping
hands to fix these ... all noise that will happen afterwards will keep
the relevant teams busy enough.

> > We are constantly beaten by removal from testing warnings
> > even with Python3.11 as an option and sometimes we simply need to remove
> > that option as a temporary means for bookworm.
> 
> Same over here. It's finally looking good for me after 2 months of heavy
> efforts.

You mean you are fixing Python 3.10 manually in some packages diverging
what will be default Python?
 
> > No better solution but better timing which means after bookworm release.
> 
> I've read *many* people saying it would be disappointing for them to see
> their package removed because of Python 3.11. Well, please consider that it
> would also be very disappointing to *not* have Python 3.11 for those who
> managed constantly fix issues for it.

I can understand that we can never satisfy the needs of everybody.  My
main problem is the extremely unfortunate timing that is happening now.
 
> The timing was exactly what was discussed during Debconf: it's very annoying
> that this year, upstream Python release was one month late... we're only
> trying to deal with it.

I do not remember that the scchedule was discussed.

   * Add 3.11 as a supported Python3 version

was done in python3-defaults (3.10.6-2) at Fri, 11 Nov 2022 11:10:31
+0200.  At 12. December Graham suggested on the behalf of the release
team[2] to switch to 3.11.  If we would have acted upon this at that
very time I would have considered this quite dense but the last chance
to consider this in line with the "lets try" attempt discussed at
DebConf.

Bug #1026825: python3.11 as default filed right before (21.12.) a series
of holidays in the region of the world where lots of developers will
typically reduce their activity which is closely followed by the first
freeze step is IMHO something else.  Since I realised that the transition
was started[3] our discussion is a bit useless.  I just want to explain
the motivation why I sounded "astonished" since you said that you do
not understood astonishment since we are following the suggested plan.
 
Kind regards
    Andreas.


[1] https://release.debian.org/testing/freeze_policy.html
[2] https://lists.debian.org/debian-python/2022/12/msg00074.html
[3] https://release.debian.org/transitions/html/python3.11-default.html

-- 
http://fam-tille.de

[toc] | [prev] | [next] | [standalone]

#14803

Hello,

On 2023-01-07 10:05, Andreas Tille wrote:
> Am Thu, Jan 05, 2023 at 01:57:43PM +0100 schrieb Thomas Goirand:
>> Please share it in this list!
> 
>     #1023965 [src:pandas] pandas FTBFS with Python 3.11 as supported version
>     #1024031 [src:numba] numba FTBFS with Python 3.11 as supported version

If I may, I would as well be grateful if someone could give a look at:

#1023972 [src:python-ase] FTBFS with Python 3.11 due to 
pathlib.Path.__enter__() deprecation

I have no idea how to fix this and the upstream is silent.

Best,
Andrius

[toc] | [prev] | [next] | [standalone]

#14810

Hi Simon,

On 2023-01-07 13:24, Simon McVittie wrote:
> On Sat, 07 Jan 2023 at 10:23:19 +0200, Andrius Merkys wrote:
>> If I may, I would as well be grateful if someone could give a look at:
>>
>> #1023972 [src:python-ase] FTBFS with Python 3.11 due to
>> pathlib.Path.__enter__() deprecation
>>
>> I have no idea how to fix this and the upstream is silent.
> My first thought on seeing this was: why were Path objects a context
> manager in the first place? What would that mean?
> 
> Looking at the code in python3.10 and python3.11 pathlib, it seems the
> reason this is deprecated is indeed that it didn't make sense:
> 
>      def __enter__(self):
>          # In previous versions of pathlib, __exit__() marked this path as
>          # closed; subsequent attempts to perform I/O would raise an IOError.
>          # This functionality was never documented, and had the effect of
>          # making Path objects mutable, contrary to PEP 428.
>          # In Python 3.9 __exit__() was made a no-op.
>          # In Python 3.11 __enter__() began emitting DeprecationWarning.
>          # In Python 3.13 __enter__() and __exit__() should be removed.
>          warnings.warn("pathlib.Path.__enter__() is deprecated and scheduled "
>                        "for removal in Python 3.13; Path objects as a context "
>                        "manager is a no-op",
>                        DeprecationWarning, stacklevel=2)
>          return self
> 
>      def __exit__(self, t, v, tb):
>          pass
> 
> So the solution seems to be that if your package contains this:
> 
>      with some_path_object:
>          do_stuff(some_path_object)
> 
> replace it with:
> 
>      do_stuff(some_path_object)
> 
> and if it contains:
> 
>      with Path(...) as my_path:
>          do_stuff(my_path)
> 
> replace with:
> 
>      my_path = Path(...)
>      do_stuff(my_path)
> 
> I hope that should be a relatively straightforward change.

Thanks a lot for the hint, this indeed worked. Failure in __enter__() 
threw me off tracks, but now I recall how it is related to 'with' 
construction.

Best wishes,
Andrius

[toc] | [prev] | [next] | [standalone]

#14827

Am Sat, Jan 07, 2023 at 09:05:21AM +0100 schrieb Andreas Tille:
> Hi Thomas,
> 
> Am Thu, Jan 05, 2023 at 01:57:43PM +0100 schrieb Thomas Goirand:
> > 
> > This has since already been discussed here: the final decision was to "at
> > least try 3.11", which is exactly what we're doing.
> 
> I admit I was not at site and may be I missunderstood what was finally
> decided.  From my perspective this "at least try" is that we are
> actually trying by having 3.11 as additional supported version which has
> triggered quite some work.  We are approaching the Transition and
> Toolchain Freeze in 5 days[1].  Given that switching default Python is a
> transition I wonder how you can assume that this is the right time to
> suggest this.  I would not have been that astonished if you would have
> done so say at first December last year.  But now its absolutely clear
> that a migration (with the "option" to revert which will cause extra
> work) will pour sand into the gears of the release process.

How will we decide whether the "at least try 3.11" is success or fail?
Did we defined anything I might have missed in terms of number of
packages that need to pass or number of packages we shoule not loose?
  
> > FYI, I'm down with only 2 major bugs (I don't mind much if the other 3 RC
> > bugs push the 3 affected packages away from the release, it's just a "would
> > be nice" thing ATM):
> 
> That's a nice situation for the field you are working in.
>  
> > > If I would create a list mine whould be way longer.
> > 
> > Please share it in this list!
> 
>    #1023965 [src:pandas] pandas FTBFS with Python 3.11 as supported version
>    #1024031 [src:numba] numba FTBFS with Python 3.11 as supported version

I'd like to add

  #1027851 [src:pytorch] FTBFS with Python 3.11 as default version

also with lots of rdepends.

> These packages have a sufficient amount of rdepends and usually trigger
> lots of other autopkgtest failures in other packages which will keep
> them out of testing for quite some time.  We could need all helping
> hands to fix these ... all noise that will happen afterwards will keep
> the relevant teams busy enough.

I did not received any response to my "small" list.  What does this
mean for the transition to 3.11 process?

> > > We are constantly beaten by removal from testing warnings
> > > even with Python3.11 as an option and sometimes we simply need to remove
> > > that option as a temporary means for bookworm.
> > 
> > Same over here. It's finally looking good for me after 2 months of heavy
> > efforts.
> 
> You mean you are fixing Python 3.10 manually in some packages diverging
> what will be default Python?

Any answer to this question?

> > > No better solution but better timing which means after bookworm release.
> > 
> > I've read *many* people saying it would be disappointing for them to see
> > their package removed because of Python 3.11. Well, please consider that it
> > would also be very disappointing to *not* have Python 3.11 for those who
> > managed constantly fix issues for it.
> 
> I can understand that we can never satisfy the needs of everybody.  My
> main problem is the extremely unfortunate timing that is happening now.
>  
> > The timing was exactly what was discussed during Debconf: it's very annoying
> > that this year, upstream Python release was one month late... we're only
> > trying to deal with it.
> 
> I do not remember that the scchedule was discussed.
> 
>    * Add 3.11 as a supported Python3 version
> 
> was done in python3-defaults (3.10.6-2) at Fri, 11 Nov 2022 11:10:31
> +0200.  At 12. December Graham suggested on the behalf of the release
> team[2] to switch to 3.11.  If we would have acted upon this at that
> very time I would have considered this quite dense but the last chance
> to consider this in line with the "lets try" attempt discussed at
> DebConf.
> 
> Bug #1026825: python3.11 as default filed right before (21.12.) a series
> of holidays in the region of the world where lots of developers will
> typically reduce their activity which is closely followed by the first
> freeze step is IMHO something else.  Since I realised that the transition
> was started[3] our discussion is a bit useless.  I just want to explain
> the motivation why I sounded "astonished" since you said that you do
> not understood astonishment since we are following the suggested plan.

I keep on thinking that the timing is unfortunate and that it
will spoil the whole release process.

Kind regards
     Andreas.
 
> 
> [1] https://release.debian.org/testing/freeze_policy.html
> [2] https://lists.debian.org/debian-python/2022/12/msg00074.html
> [3] https://release.debian.org/transitions/html/python3.11-default.html
> 
> -- 
> http://fam-tille.de
> 
> 

-- 
http://fam-tille.de

[toc] | [prev] | [next] | [standalone]

#14830

On 1/16/23 17:23, Andreas Tille wrote:
> Am Sat, Jan 07, 2023 at 09:05:21AM +0100 schrieb Andreas Tille:
>>>> If I would create a list mine whould be way longer.
>>>
>>> Please share it in this list!
>>
>>     #1023965 [src:pandas] pandas FTBFS with Python 3.11 as supported version
>>     #1024031 [src:numba] numba FTBFS with Python 3.11 as supported version

I saw the above 2 were fixed.

> I'd like to add
> 
>    #1027851 [src:pytorch] FTBFS with Python 3.11 as default version
> 
> also with lots of rdepends.

So we're back with one single bug. I remember seeing something similar 
in another package ... (scratching my head...). The latest version of 
the upstream code has some modifications to this broken Stream.cpp, have 
you tried to apply them?

Do you have more to share? It's harder to help if you don't ask for it.
IMO, feel free to give a full list of problematic packages in this list, 
so others may help.

> I did not received any response to my "small" list.  What does this
> mean for the transition to 3.11 process?

As much as I know, we're moving toward having Python 3.11 only in 
Bookworm. I'm not the person driving it though, and I am not in the best 
position to make such choice, but I support it (as I would prefer having 
the nice enhancements of Python 3.11 rather than lagging behind). 
Hopefully, I wont regret it and wont find more failures in "my" packages.

>>>> We are constantly beaten by removal from testing warnings
>>>> even with Python3.11 as an option and sometimes we simply need to remove
>>>> that option as a temporary means for bookworm.
>>>
>>> Same over here. It's finally looking good for me after 2 months of heavy
>>> efforts.
>>
>> You mean you are fixing Python 3.10 manually in some packages diverging
>> what will be default Python?
> 
> Any answer to this question?

All of my packages hopefully always test with all available versions, 
and most run autopkgtest. So I was warned early of Py 3.11 failures. 
They are all fixed, as much as I know (no opened RC bug remaining...). 
And no, forcing Python 3.10 is *NOT* an option, one must fix failures in 
Python 3.11.

>> Bug #1026825: python3.11 as default filed right before (21.12.) a series
>> of holidays in the region of the world where lots of developers will
>> typically reduce their activity which is closely followed by the first
>> freeze step is IMHO something else.  Since I realised that the transition
>> was started[3] our discussion is a bit useless.  I just want to explain
>> the motivation why I sounded "astonished" since you said that you do
>> not understood astonishment since we are following the suggested plan.
> 
> I keep on thinking that the timing is unfortunate and that it
> will spoil the whole release process.

I'm sad to read this. Hopefully, this is truth only for some of the 
packages you care, and the vast majority of the packages are fine? I'm 
unfortunately not in a good position to tell (I didn't run any survey of 
broken packages...).

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]

#14832

Am Tue, Jan 17, 2023 at 01:04:50AM +0100 schrieb Thomas Goirand:
> > >     #1023965 [src:pandas] pandas FTBFS with Python 3.11 as supported version
> > >     #1024031 [src:numba] numba FTBFS with Python 3.11 as supported version
> 
> I saw the above 2 were fixed.

Fixed in the sense that there was an upload closing the bug.  If you look
at

   https://tracker.debian.org/pkg/pandas
   https://tracker.debian.org/pkg/numba

you see multiple blockers for a testing migration.  So the problem for
the release persists.  I have confirmations of upstream of several
rdepends of these packages that they do not support 3.11 since these two
packages do not officially support Python 3.11 yet (the Debian packages
are coming with several patches - just one example of an answer from
upstream for python-cogent[1])
 
> > I'd like to add
> > 
> >    #1027851 [src:pytorch] FTBFS with Python 3.11 as default version
> > 
> > also with lots of rdepends.
> 
> So we're back with one single bug. I remember seeing something similar in
> another package ... (scratching my head...). The latest version of the
> upstream code has some modifications to this broken Stream.cpp, have you
> tried to apply them?

No, I have not dealt with torch.  I just know that this package is in
the very competent hands of Mo Zhou who will know what to do.

> Do you have more to share? It's harder to help if you don't ask for it.
> IMO, feel free to give a full list of problematic packages in this list, so
> others may help.

As I said above the packages above are far from testing.  If someone
could lend helping hands to let these packages migrate this would be
really helpful.
 
> > I did not received any response to my "small" list.  What does this
> > mean for the transition to 3.11 process?
> 
> As much as I know, we're moving toward having Python 3.11 only in Bookworm.
> I'm not the person driving it though, and I am not in the best position to
> make such choice, but I support it (as I would prefer having the nice
> enhancements of Python 3.11 rather than lagging behind). Hopefully, I wont
> regret it and wont find more failures in "my" packages.

I understand the advantages of Python 3.11 and I'm not against releasing
Bookworm with it.  I'm against overlong freezing periods which I see as
the consequence of pushing Python 3.11 while sticking to the current
freeze shedule.  If we would decide that Python 3.11 is really important
I would consider shifting the testing period by one or two months.  We
have just seen that every full rebuild of the archive as Lucas recently
did creates quite some new RC bugs that are related to the Python
version bump and I expect more of these at the next rebuild.

> > > You mean you are fixing Python 3.10 manually in some packages diverging
> > > what will be default Python?
> > 
> > Any answer to this question?
> 
> All of my packages hopefully always test with all available versions, and
> most run autopkgtest. So I was warned early of Py 3.11 failures. They are
> all fixed, as much as I know (no opened RC bug remaining...). And no,
> forcing Python 3.10 is *NOT* an option, one must fix failures in Python
> 3.11.

Since you said above that we want to release with 3.11 only this becomes
clear.  Luckily the teams where I'm working in have also quite good
autopkgtest coverage so I'm positive about sensible warnings.
 
> > I keep on thinking that the timing is unfortunate and that it
> > will spoil the whole release process.
> 
> I'm sad to read this. Hopefully, this is truth only for some of the packages
> you care, and the vast majority of the packages are fine? I'm unfortunately
> not in a good position to tell (I didn't run any survey of broken
> packages...).

We are just a couple of people who are fighting hard through scientific
packages with a complex depency tree.  Some of them (like numba) take
ages to build even on amd64 (salsa CI is set to 6h here) and thus take
quite some time to fix.  As I said above I'm not against Python 3.11 per
see.  I'm simply afraid that this decision will delay the release
process and IMHO it makes sense to shift the schedule if we decide that
Python 3.11 is something we want.

Kind regards
   Andreas.

[1] https://github.com/cogent3/cogent3/issues/1263 

-- 
http://fam-tille.de

[toc] | [prev] | [next] | [standalone]

#14811

[Multipart message — attachments visible in raw view] — view raw

Hi,

For information I tagged [#1027851] with help needed for the
python3.11 port of pytorch, following a message from its main
uploader.  To people who might like to have a look at it: please
be careful as pytorch may be quite the resource hog and time
sink if one's time is better spent somewhere else.

#1027851: pytorch FTBFS with Python 3.11 as default version

[#1027851]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1027851

Have a nice day,  :)
-- 
Étienne Mollier <emollier@emlwks999.eu>
Fingerprint:  8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/1, please excuse my verbosity.
On air: A Liquid Landscape - Secret isle

[toc] | [prev] | [next] | [standalone]

#14773

Hi Sandro (2022.12.22_00:13:36_+0000)
> It appears there has been little work in preparing the work to
> introduce python3.11 from its maintainer, instead that works has been
> pushed downstream to maintainers.

That is, I'm afraid, the only realistic approach for handling new Python
versions. It is too much work for one or two people to do. It needs the
help of the team and upstreams to make it happen.

Yes, a maintainer could take all this work on their shoulders, but if we
require them to, I don't think we'll ship even vaguely current Python
versions.

> if we continue with the plan as described above, several python
> libraries/applications maintainers will be left with the short end of
> the stick and deal with an unknown amount of issues (upstream fixes
> not available, not ready and or/ not released, rushed, etc) with less
> than a month from the beginning of the transition freeze[2]

That will almost certainly be the case, yes. So we have a trade-off to
make between shipping a new Python upstream release, that many of our
users would definitely appreciate, and having some libraries / apps miss
the release, that many of our users would probably be affected by.

> [2] also highlights at the very beginning "Plan your changes for
> bullseye", this change appears as if it was not planned and we should
> be skeptical to proceed without further (and in advance) understanding
> of the impact it may have on Bullseye.

We discussed this transition at DebConf 22, and decided to approach it
the way that it has been approached.

Where we currently are in the release, I would lean towards going
through with the transition. So far, it seems to have been roughly as
difficult as previous Python transitions.

There have been rebuilds in Ubuntu that give us some idea of how much
work remains. I think it's tractable, but also will have some package
casualties.

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [prev] | [next] | [standalone]

#14774

[Multipart message — attachments visible in raw view] — view raw

* Stefano Rivera <stefanor@debian.org> [2022-12-22 12:44]:
>There have been rebuilds in Ubuntu that give us some idea of how much
>work remains. I think it's tractable, but also will have some package
>casualties.
I have some spare time right now, and I am happy to help
work on problematic cases, so hopefully nobody will feel left out in
the cold with their favorite packages.


Cheers
Timo

-- 
⢀⣴⠾⠻⢶⣦⠀   ╭────────────────────────────────────────────────────╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling                                       │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄⠀⠀⠀⠀   ╰────────────────────────────────────────────────────╯

[toc] | [prev] | [next] | [standalone]

#14775

Hi Timo (2022.12.22_12:56:20_+0000)
> > There have been rebuilds in Ubuntu that give us some idea of how much
> > work remains. I think it's tractable, but also will have some package
> > casualties.
> I have some spare time right now, and I am happy to help
> work on problematic cases, so hopefully nobody will feel left out in
> the cold with their favorite packages.

Offhand, the one I most expect trouble with is numba. We were reliant on
upstream for the 3.10 transition, and probably will be for 3.11.

Thanks for your help with pony ORM, Timo. I didn't think we'd be able to
port that without upstream, but it did end up being tractable.

I'm expecting to have more time in the upcoming weeks, too.

So, release team, I still think we should go ahead!

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [prev] | [next] | [standalone]

#14776

I'm the regular uploader of python3-llvmlite.

Please give up with numba. Its core dependency llvmlite is not even
ready for llvm != 11, while Sid had already get llvm-11 removed.
I have tried to cherry-pick an upstream fix to bump llvmlite's
llvm dependency to 12/14, but the autopkgtest shows numba would
be vastly broken.

Unless they bump LLVM dependency to a newer version:
https://github.com/numba/llvmlite/issues/897
https://github.com/numba/llvmlite/pull/830
there is zero chance to get numba in stable. I do not want to
bump LLVM by force and leave a broken package in stable.

llvmlite's python 3.11 support is still on the way:
https://github.com/numba/llvmlite/issues/885

One possibility is that we may apply for freeze exception
and wait for the llvmlite v0.40.0 release and see whether
they will bump llvm dependency.


On Thu, 2022-12-22 at 18:46 +0000, Stefano Rivera wrote:
> Hi Timo (2022.12.22_12:56:20_+0000)
> > > There have been rebuilds in Ubuntu that give us some idea of how much
> > > work remains. I think it's tractable, but also will have some package
> > > casualties.
> > I have some spare time right now, and I am happy to help
> > work on problematic cases, so hopefully nobody will feel left out in
> > the cold with their favorite packages.
> 
> Offhand, the one I most expect trouble with is numba. We were reliant on
> upstream for the 3.10 transition, and probably will be for 3.11.
> 
> Thanks for your help with pony ORM, Timo. I didn't think we'd be able to
> port that without upstream, but it did end up being tractable.
> 
> I'm expecting to have more time in the upcoming weeks, too.
> 
> So, release team, I still think we should go ahead!
> 
> SR
> 

[toc] | [prev] | [next] | [standalone]

#14777

It seems I was a little bit out of date. Diane Trout has tried
with an unreleased snapshot which looks good with llvm-14
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024795
Will work on it soon.

On Thu, 2022-12-22 at 18:04 -0500, M. Zhou wrote:
> I'm the regular uploader of python3-llvmlite.
> 
> Please give up with numba. Its core dependency llvmlite is not even
> ready for llvm != 11, while Sid had already get llvm-11 removed.
> I have tried to cherry-pick an upstream fix to bump llvmlite's
> llvm dependency to 12/14, but the autopkgtest shows numba would
> be vastly broken.
> 
> Unless they bump LLVM dependency to a newer version:
> https://github.com/numba/llvmlite/issues/897
> https://github.com/numba/llvmlite/pull/830
> there is zero chance to get numba in stable. I do not want to
> bump LLVM by force and leave a broken package in stable.
> 
> llvmlite's python 3.11 support is still on the way:
> https://github.com/numba/llvmlite/issues/885
> 
> One possibility is that we may apply for freeze exception
> and wait for the llvmlite v0.40.0 release and see whether
> they will bump llvm dependency.
> 
> 
> On Thu, 2022-12-22 at 18:46 +0000, Stefano Rivera wrote:
> > Hi Timo (2022.12.22_12:56:20_+0000)
> > > > There have been rebuilds in Ubuntu that give us some idea of how much
> > > > work remains. I think it's tractable, but also will have some package
> > > > casualties.
> > > I have some spare time right now, and I am happy to help
> > > work on problematic cases, so hopefully nobody will feel left out in
> > > the cold with their favorite packages.
> > 
> > Offhand, the one I most expect trouble with is numba. We were reliant on
> > upstream for the 3.10 transition, and probably will be for 3.11.
> > 
> > Thanks for your help with pony ORM, Timo. I didn't think we'd be able to
> > port that without upstream, but it did end up being tractable.
> > 
> > I'm expecting to have more time in the upcoming weeks, too.
> > 
> > So, release team, I still think we should go ahead!
> > 
> > SR
> > 
> 

[toc] | [prev] | [next] | [standalone]

#14781

Hi Timo, Stefano

On Thu, 22 Dec 2022 at 18:46, Stefano Rivera <stefanor@debian.org> wrote:
>
> Hi Timo (2022.12.22_12:56:20_+0000)
> > > There have been rebuilds in Ubuntu that give us some idea of how much
> > > work remains. I think it's tractable, but also will have some package
> > > casualties.
> > I have some spare time right now, and I am happy to help
> > work on problematic cases, so hopefully nobody will feel left out in
> > the cold with their favorite packages.
>
> Offhand, the one I most expect trouble with is numba. We were reliant on
> upstream for the 3.10 transition, and probably will be for 3.11.
>
> Thanks for your help with pony ORM, Timo. I didn't think we'd be able to
> port that without upstream, but it did end up being tractable.
>
> I'm expecting to have more time in the upcoming weeks, too.
>
> So, release team, I still think we should go ahead!

Thanks for committing your time to this, and for the fixes you've
already uploaded (and to everyone else who has helped).  Please let
the release team know if you need things unstuck.

Regards
Graham

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.maint.python

csiph-web