Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #14628 > unrolled thread

Python 3.11, bytecode and new internals

Back to article view | Back to linux.debian.maint.python

Contents

  Python 3.11, bytecode and new internals Julian Gilbey <julian@d-and-j.net> - 2022-11-21 08:30 +0100
    Re: Python 3.11, bytecode and new internals Louis-Philippe Véronneau <pollo@debian.org> - 2022-11-21 18:10 +0100
      Re: Python 3.11, bytecode and new internals Scott Kitterman <debian@kitterman.com> - 2022-11-21 18:30 +0100
        Re: Python 3.11, bytecode and new internals Scott Kitterman <debian@kitterman.com> - 2022-11-22 15:20 +0100
      Re: Python 3.11, bytecode and new internals Sandro Tosi <morph@debian.org> - 2022-11-21 18:40 +0100
        Re: Python 3.11, bytecode and new internals Thomas Goirand <zigo@debian.org> - 2022-11-22 09:30 +0100
          Re: Python 3.11, bytecode and new internals Julian Gilbey <julian@d-and-j.net> - 2022-11-22 11:00 +0100
            Re: Python 3.11, bytecode and new internals Thomas Goirand <zigo@debian.org> - 2022-11-22 17:10 +0100
              Re: Python 3.11, bytecode and new internals Julian Gilbey <julian@d-and-j.net> - 2022-11-22 18:00 +0100
                Re: Python 3.11, bytecode and new internals Thomas Goirand <zigo@debian.org> - 2022-11-23 00:30 +0100
                  Re: Python 3.11, bytecode and new internals Julian Gilbey <julian@d-and-j.net> - 2022-11-23 08:40 +0100

#14628 — Python 3.11, bytecode and new internals

I've been having a somewhat interesting time with the python3.11-add
transition.  Python 3.11 has made some significant changes to its
bytecode representation, and also changed some of it's internal data
structures related to frames quite significantly.

In my corner of the Python world, several important packages use this
information, including Spyder, IPython and Jupyter (which all
(recursively) depend on python3-parso and/or python3-bytecode), but
these low-level packages are not really ready for 3.11 yet (parso
claims "basic support for Python 3.11 and 3.12", so it's unlikely it
will actually yet work with 3.11, though I haven't tested this, and
bytecode makes no claims to yet be ready for 3.11).

Several packages have tests which now fail on 3.11, preventing
migration to testing (spyder-kernels is one).  In pydevd (which does
not yet support 3.11, because it depends on python3-bytecode), I've
just disabled all the failing tests and put a note in README.Debian,
but this is not a great solution.  I could add X-Python3-Version: <<
3.11 to debian/control, but that is surely not that desirable either.

I'm just flagging this up here, with a question about how we should
proceed.  Certainly we are not ready to make Python 3.11 the default
Python version!!

Best wishes,

   Julian

[toc] | [next] | [standalone]

#14630

On 2022-11-21 02 h 08, Julian Gilbey wrote:
> I'm just flagging this up here, with a question about how we should
> proceed.  Certainly we are not ready to make Python 3.11 the default
> Python version!!

This is a concern I share and I think I've been pretty vocal about it.

I feel the state of python packages for Bookworm with 3.10 was pretty 
good and it seemed reasonable to prioritize stability for our next 
stable release :)

It's very frustrating to work on packaging python libraries and apps for 
a whole release cycle, just to see all that work put in the bin at the 
last minute because upstream doesn't support 3.11...

I've been told the current 3.11 transition was a test, and if it was 
clear too many important things were broken and couldn't be fixed, we 
would roll back and release using 3.10.

-- 
   ⢀⣴⠾⠻⢶⣦⠀
   ⣾⠁⢠⠒⠀⣿⡁  Louis-Philippe Véronneau
   ⢿⡄⠘⠷⠚⠋   pollo@debian.org / veronneau.org
   ⠈⠳⣄

[toc] | [prev] | [next] | [standalone]

#14631

On November 21, 2022 5:02:57 PM UTC, "Louis-Philippe Véronneau" <pollo@debian.org> wrote:
>On 2022-11-21 02 h 08, Julian Gilbey wrote:
>> I'm just flagging this up here, with a question about how we should
>> proceed.  Certainly we are not ready to make Python 3.11 the default
>> Python version!!
>
>This is a concern I share and I think I've been pretty vocal about it.
>
>I feel the state of python packages for Bookworm with 3.10 was pretty good and it seemed reasonable to prioritize stability for our next stable release :)
>
>It's very frustrating to work on packaging python libraries and apps for a whole release cycle, just to see all that work put in the bin at the last minute because upstream doesn't support 3.11...
>
>I've been told the current 3.11 transition was a test, and if it was clear too many important things were broken and couldn't be fixed, we would roll back and release using 3.10.
>


Looks like you can add #1024521 to the list.

Scott K

[toc] | [prev] | [next] | [standalone]

#14636

[Multipart message — attachments visible in raw view] — view raw

On Monday, November 21, 2022 12:25:05 PM EST Scott Kitterman wrote:
> On November 21, 2022 5:02:57 PM UTC, "Louis-Philippe Véronneau" 
<pollo@debian.org> wrote:
> >On 2022-11-21 02 h 08, Julian Gilbey wrote:
> >> I'm just flagging this up here, with a question about how we should
> >> proceed.  Certainly we are not ready to make Python 3.11 the default
> >> Python version!!
> >
> >This is a concern I share and I think I've been pretty vocal about it.
> >
> >I feel the state of python packages for Bookworm with 3.10 was pretty good
> >and it seemed reasonable to prioritize stability for our next stable
> >release :)
> >
> >It's very frustrating to work on packaging python libraries and apps for a
> >whole release cycle, just to see all that work put in the bin at the last
> >minute because upstream doesn't support 3.11...
> >
> >I've been told the current 3.11 transition was a test, and if it was clear
> >too many important things were broken and couldn't be fixed, we would roll
> >back and release using 3.10.
> Looks like you can add #1024521 to the list.

This one is fixed.

Scott K

[toc] | [prev] | [next] | [standalone]

#14632

On Mon, Nov 21, 2022 at 12:03 PM Louis-Philippe Véronneau
<pollo@debian.org> wrote:
>
> On 2022-11-21 02 h 08, Julian Gilbey wrote:
> > I'm just flagging this up here, with a question about how we should
> > proceed.  Certainly we are not ready to make Python 3.11 the default
> > Python version!!
>
> This is a concern I share and I think I've been pretty vocal about it.
>
> I feel the state of python packages for Bookworm with 3.10 was pretty
> good and it seemed reasonable to prioritize stability for our next
> stable release :)
>
> It's very frustrating to work on packaging python libraries and apps for
> a whole release cycle, just to see all that work put in the bin at the
> last minute because upstream doesn't support 3.11...

this, 100 times

> I've been told the current 3.11 transition was a test, and if it was
> clear too many important things were broken and couldn't be fixed, we
> would roll back and release using 3.10.

why are we running a "test" this close to the release? *who* are we
running this test for? who made this decision (i figure RT gave the go
ahead, but still)? is there any searchable source for this claim?

Thanks,
-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

[toc] | [prev] | [next] | [standalone]

#14634

On 11/21/22 18:30, Sandro Tosi wrote:
> On Mon, Nov 21, 2022 at 12:03 PM Louis-Philippe Véronneau
> <pollo@debian.org> wrote:
>>
>> On 2022-11-21 02 h 08, Julian Gilbey wrote:
>>> I'm just flagging this up here, with a question about how we should
>>> proceed.  Certainly we are not ready to make Python 3.11 the default
>>> Python version!!
>>
>> This is a concern I share and I think I've been pretty vocal about it.
>>
>> I feel the state of python packages for Bookworm with 3.10 was pretty
>> good and it seemed reasonable to prioritize stability for our next
>> stable release :)
>>
>> It's very frustrating to work on packaging python libraries and apps for
>> a whole release cycle, just to see all that work put in the bin at the
>> last minute because upstream doesn't support 3.11...
> 
> this, 100 times

I very much don't agree. I think it's going pretty well, and the number 
of breakage isn't high. We just need a little bit of effort to make it 
in good enough shape.

>> I've been told the current 3.11 transition was a test, and if it was
>> clear too many important things were broken and couldn't be fixed, we
>> would roll back and release using 3.10.

Though we're not there yet, as a point were we can say it's a lost battle.

> why are we running a "test" this close to the release?

Let me fix the above sentence for you:
s/release/freeze/

Because Python 3.11 brings many nice feature, one of them being that 
it's 15 to 50% faster, very often 20% faster. Also, releasing the last 
Debian with an already obsolete interpreter version doesn't feel good.

> *who* are we
> running this test for? who made this decision (i figure RT gave the go
> ahead, but still)? is there any searchable source for this claim?

This was discussed during Debconf in Prizren. You are always free to:
- join us during debconf
- join on IRC if you can't go to Debconf
- read the video and voice your concerns on the list

So the decision was made collectively. We also discussed this on IRC, 
and in this very list, if I'm not mistaking (sorry, I will not search 
for it).

Currently, Python 3.11 is only an "available interpreter version", not 
the default. So it's kind of easy to revert (we would "only" need to 
remove the 3.11 interpreter, and rebuild the packages that produce 3.11 
so files). It would be a lot harder to revert having 3.11 as default, 
Mathias said/wrote. So we're good...

I very much thank Stefano for the many fixes he already uploaded.

Now, out of *many* of my packages, only a very few broke. Complicated 
packages like Eventlet for example, just worked. Others had upstream 
patches I applied. And I am in the opinion that we should go ahead and 
make 3.11 the default.

I'd be happy to have the opinion of the rest of the team, especially 
Doko and Stefano.

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]

#14635

On Tue, Nov 22, 2022 at 09:22:05AM +0100, Thomas Goirand wrote:
> > this, 100 times
> 
> I very much don't agree. I think it's going pretty well, and the number of
> breakage isn't high. We just need a little bit of effort to make it in good
> enough shape.
> [...]
> Now, out of *many* of my packages, only a very few broke. Complicated
> packages like Eventlet for example, just worked. Others had upstream patches
> I applied. And I am in the opinion that we should go ahead and make 3.11 the
> default.

If there are people with the expertise to help upstream update
bytecode and parso (and probably several other low-level packages) for
3.11 so that the software that depends on them works with 3.11, then
fine.  (And it is a non-trivial update, AFAICT.)  But until then, I'd
be very reluctant to make 3.11 the default.

I haven't decided what to do with packages which now FTBFS under 3.11
because of this.  Should we just let them fall out of testing (that
certainly includes spyder, and quite possibly ipython as well)?  Or
should we mark them as X-Python3-Version: << 3.11 so they can stay in
testing as long as Python 3.10 is the default?  If we make 3.11 the
default, these packages will likely not be in bookworm, which might
upset our users even more.

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#14637

On 11/22/22 10:59, Julian Gilbey wrote:
> On Tue, Nov 22, 2022 at 09:22:05AM +0100, Thomas Goirand wrote:
>>> this, 100 times
>>
>> I very much don't agree. I think it's going pretty well, and the number of
>> breakage isn't high. We just need a little bit of effort to make it in good
>> enough shape.
>> [...]
>> Now, out of *many* of my packages, only a very few broke. Complicated
>> packages like Eventlet for example, just worked. Others had upstream patches
>> I applied. And I am in the opinion that we should go ahead and make 3.11 the
>> default.
> 
> If there are people with the expertise to help upstream update
> bytecode and parso (and probably several other low-level packages) for
> 3.11 so that the software that depends on them works with 3.11, then
> fine.  (And it is a non-trivial update, AFAICT.)  But until then, I'd
> be very reluctant to make 3.11 the default.

Have you tried this PR?
https://github.com/MatthieuDartiailh/bytecode/pull/107

> I haven't decided what to do with packages which now FTBFS under 3.11
> because of this.  Should we just let them fall out of testing (that
> certainly includes spyder, and quite possibly ipython as well)?
> Or should we mark them as X-Python3-Version: << 3.11 so they can stay in
> testing as long as Python 3.10 is the default?

I don't think this is the way.

> If we make 3.11 the
> default, these packages will likely not be in bookworm, which might
> upset our users even more.

We're not there yet. We have until January to fix, and we haven't 
decided yet if 3.11 will be the default.

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]

#14638

On Tue, Nov 22, 2022 at 05:01:03PM +0100, Thomas Goirand wrote:
> > If there are people with the expertise to help upstream update
> > bytecode and parso (and probably several other low-level packages) for
> > 3.11 so that the software that depends on them works with 3.11, then
> > fine.  (And it is a non-trivial update, AFAICT.)  But until then, I'd
> > be very reluctant to make 3.11 the default.
> 
> Have you tried this PR?
> https://github.com/MatthieuDartiailh/bytecode/pull/107

As you can see by reading it, there is still at least one blocking
point (to use Matthieu's language), and Matthieu is the core bytecode
developer.  Once that is sorted, then pydevd needs to be updated to
use it.  But this is far outside my expertise, and I'm not going to
apply a huge patch that I don't understand and that is known to still
be buggy.  So I'm just going to have to wait.

I don't know the current state of parso; it would be interesting to
see whether parso 0.8.3 successfully works under 3.11, but that is
Piotr Ożarowski's package (not under the Python team); he is active,
but has not yet uploaded this newer version.

> > I haven't decided what to do with packages which now FTBFS under 3.11
> > because of this.  Should we just let them fall out of testing (that
> > certainly includes spyder, and quite possibly ipython as well)?
> > Or should we mark them as X-Python3-Version: << 3.11 so they can stay in
> > testing as long as Python 3.10 is the default?
> 
> I don't think this is the way.

I'm sorry, I don't understand - which is not the way?

> > If we make 3.11 the
> > default, these packages will likely not be in bookworm, which might
> > upset our users even more.
> 
> We're not there yet. We have until January to fix, and we haven't decided
> yet if 3.11 will be the default.

Fair enough.

But I still don't know what to do in the meantime with the spyder
ecosystem besides either wait for upstream to sort bytecode and pydevd
and Piotr (and possibly upstream) to sort parso, or to mark them as
Python 3.10 only.

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#14639

On 11/22/22 17:59, Julian Gilbey wrote:
>>> Or should we mark them as X-Python3-Version: << 3.11 so they can stay in
>>> testing as long as Python 3.10 is the default?
>>
>> I don't think this is the way.
> 
> I'm sorry, I don't understand - which is not the way?

I don't think you should "mark them as X-Python3-Version: << 3.11", 
because either we use 3.10 or 3.11 in Bookworm, I don't think that 
there's a plan for having both interpreters available.

> But I still don't know what to do in the meantime with the spyder
> ecosystem besides either wait for upstream to sort bytecode and pydevd
> and Piotr (and possibly upstream) to sort parso, or to mark them as
> Python 3.10 only.

Well, hopefully for you, you'll get it fixed before next January, or we 
go back to 3.10 only (or both?).

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]

#14640

On Wed, Nov 23, 2022 at 12:21:38AM +0100, Thomas Goirand wrote:
> On 11/22/22 17:59, Julian Gilbey wrote:
> > > > Or should we mark them as X-Python3-Version: << 3.11 so they can stay in
> > > > testing as long as Python 3.10 is the default?
> > > 
> > > I don't think this is the way.
> > 
> > I'm sorry, I don't understand - which is not the way?
> 
> I don't think you should "mark them as X-Python3-Version: << 3.11", because
> either we use 3.10 or 3.11 in Bookworm, I don't think that there's a plan
> for having both interpreters available.

It's more about the present time: if I mark them as << 3.11, then they
will build fine and be able to stay in testing.  If I don't, they'll
be pulled from testing in the next few weeks because of FTBFS bugs.
Obviously, once they do build OK with 3.11, then I can remove that
clause.

Best wishes,

   Julian

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web