Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #15336 > unrolled thread

Bug#1043240: transition: pandas 1.5 -> 2.1

Back to article view | Back to linux.debian.maint.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1043240: transition: pandas 1.5 -> 2.1 "Rebecca N. Palmer" <rebecca_palmer@zoho.com> - 2023-12-10 14:30 +0100
    Bug#1043240: transition: pandas 1.5 -> 2.1 Julian Gilbey <jdg@debian.org> - 2023-12-10 21:30 +0100
      Bug#1043240: pandas 1.5 -> 2.1? "Kingsley G. Morse Jr." <kingsley@loaner.com> - 2023-12-10 22:10 +0100
        Bug#1043240: pandas 1.5 -> 2.1? Drew Parsons <dparsons@debian.org> - 2023-12-11 00:40 +0100
        Bug#1043240: pandas 1.5 -> 2.1? Julian Gilbey <jdg@debian.org> - 2023-12-11 11:10 +0100
    Bug#1043240: transition: pandas 1.5 -> 2.1 Matthias Klose <doko@debian.org> - 2023-12-11 08:20 +0100
      Bug#1043240: transition: pandas 1.5 -> 2.1 Matthias Klose <doko@debian.org> - 2023-12-11 18:40 +0100
    Bug#1043240: transition: pandas 1.5 -> 2.1 "Rebecca N. Palmer" <rebecca_palmer@zoho.com> - 2024-01-21 16:40 +0100
      Bug#1043240: transition: pandas 1.5 -> 2.1 Andreas Tille <andreas@an3as.eu> - 2024-01-21 19:40 +0100
        Bug#1043240: transition: pandas 1.5 -> 2.1 Stelios Moschos <stel.moschos@gmail.com> - 2024-01-21 20:40 +0100
      Bug#1043240: transition: pandas 1.5 -> 2.1 Julian Gilbey <julian@d-and-j.net> - 2024-01-22 13:20 +0100
        Bug#1043240: transition: pandas 1.5 -> 2.1 "Rebecca N. Palmer" <rebecca_palmer@zoho.com> - 2024-01-22 22:00 +0100
          Bug#1043240: transition: pandas 1.5 -> 2.1 Julian Gilbey <julian@d-and-j.net> - 2024-01-23 13:40 +0100
            Bug#1043240: transition: pandas 1.5 -> 2.1 Graham Inggs <ginggs@debian.org> - 2024-01-26 07:50 +0100
              Bug#1043240: transition: pandas 1.5 -> 2.1 Julian Gilbey <julian@d-and-j.net> - 2024-01-26 08:40 +0100

#15336 — Bug#1043240: transition: pandas 1.5 -> 2.1

I'd like to move forward with the pandas 1.5 -> 2.1 transition 
reasonably soon.

Given that pandas 2.x is *not* required for Python 3.12 (but is required 
for Cython 3.0), should we wait for the Python 3.12 transition to be 
done first?

These are broken by pandas 2.x and have a possible (but untested) fix in 
their bug - please test and apply it:
dask(?) dials influxdb-python* python-altair python-feather-format 
python-upsetplot seaborn tqdm*
(* = this package is currently also broken for a non-pandas reason, 
probably Python 3.12, that I don't have a fix for)

These are broken by pandas 2.x and have no known-to-me fix:
augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata 
python-biom-format python-cooler python-nanoget python-skbio python-ulmo 
q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas
Some generic things to try are pandas.util.testing -> pandas.testing, 
.iteritems() -> .items(), and if one exists, a more recent upstream version.

Is this an acceptable amount of breakage or should we continue to wait? 
Bear in mind that if we wait too long, we may be forced into it by some 
transition further up the stack (e.g. a future Python or numpy) that 
breaks pandas 1.x.

Build logs:
https://launchpad.net/~rebecca-palmer/+archive/ubuntu/pandas2p1/+builds?build_text=&build_state=failed
https://launchpad.net/~rebecca-palmer/+archive/ubuntu/pandas2p1n/+builds?build_text=&build_state=failed
(The second is more recent, but includes fewer packages.)

Autopkgtest logs:
https://qa.debian.org/excuses.php?experimental=1&package=pandas
(Because of the Python 3.12 transition, this may currently be wrong 
about what is a regression and what is not.)

[toc] | [next] | [standalone]

#15338

On Sun, Dec 10, 2023 at 01:06:01PM +0000, Rebecca N. Palmer wrote:
> I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably
> soon.
> 
> Given that pandas 2.x is *not* required for Python 3.12 (but is required for
> Cython 3.0), should we wait for the Python 3.12 transition to be done first?

Well, I have seen at least one package that has an RC bug for the
Python 3.12 transition that might be because it's still using an old
version of cython3 :(  So it's a bit of chicken-and-egg - having Cython
3.0 might be very helpful.  But then there is this list of 28 packages
broken by pandas 2.x.  On the other hand, these will need fixing at
some point soon anyway, so I'd be in favour of doing the pandas
transition now, which will allow Cython 3.0 to move into unstable.

Just my 2 cents' worth...

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#15341 — Bug#1043240: pandas 1.5 -> 2.1?

Hi Rebecca, Julian and all science minded pythonistas of debian, great and small!

I like your correspondence about upgrading from
version 1.5 of pandas to 2.1.

It's open, scientific and explores the ideal of
proceeding wisely in a matter of public interest.

My humble thoughts are:

1.) Rebecca: *Why* did you write that you'd like
    to move forward with the pandas 1.5 -> 2.1
    transition? What's your reason?

2.) What may be the advantage of migrating to
    version 3.0 of Cython?

3.) The following one-liner suggests 44 debian
    packages might be affected by the breaks
    Rebecca said would be caused by pandas 2.x:

    $ for s in augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas ; do apt-cache search "$s" ; done | less

4.) The break that worries me the most is
    sklearn-pandas, because it seems to me that
    sklearn is 

        popular and 

        fundamental.

Comment welcome,
Kingsley

On 12/10/2023 20:16, Julian Gilbey wrote:
> On Sun, Dec 10, 2023 at 01:06:01PM +0000, Rebecca N. Palmer wrote:
> > I'd like to move forward with the pandas 1.5 -> 2.1 transition reasonably
> > soon.
> > 
> > Given that pandas 2.x is *not* required for Python 3.12 (but is required for
> > Cython 3.0), should we wait for the Python 3.12 transition to be done first?
> 
> Well, I have seen at least one package that has an RC bug for the
> Python 3.12 transition that might be because it's still using an old
> version of cython3 :(  So it's a bit of chicken-and-egg - having Cython
> 3.0 might be very helpful.  But then there is this list of 28 packages
> broken by pandas 2.x.  On the other hand, these will need fixing at
> some point soon anyway, so I'd be in favour of doing the pandas
> transition now, which will allow Cython 3.0 to move into unstable.
> 
> Just my 2 cents' worth...
> 
> Best wishes,
> 
>    Julian
> 

-- 
Time is the fire in which we all burn.

[toc] | [prev] | [next] | [standalone]

#15342 — Bug#1043240: pandas 1.5 -> 2.1?

On 2023-12-10 21:55, Kingsley G. Morse Jr. wrote:
> Hi Rebecca, Julian and all science minded pythonistas of debian, great
> and small!
> 
> 
> 3.) The following one-liner suggests 44 debian
>     packages might be affected by the breaks
>     Rebecca said would be caused by pandas 2.x:
> 
>     $ for s in augur cnvkit dyda emperor esda mirtop pymatgen pyranges
> python-anndata python-biom-format python-cooler python-nanoget
> python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types
> q2templates sklearn-pandas ; do apt-cache search "$s" ; done | less


I'd say don't worry about pymatgen in any case.  It's got a new version 
to upgrade to, and they make regular releases.

Drew

[toc] | [prev] | [next] | [standalone]

#15346 — Bug#1043240: pandas 1.5 -> 2.1?

Hi Kingsley,

On Sun, Dec 10, 2023 at 12:55:43PM -0800, Kingsley G. Morse Jr. wrote:
> Hi Rebecca, Julian and all science minded pythonistas of debian, great and small!
> 
> I like your correspondence about upgrading from
> version 1.5 of pandas to 2.1.
> 
> It's open, scientific and explores the ideal of
> proceeding wisely in a matter of public interest.
> 
> My humble thoughts are:
> 
> 1.) Rebecca: *Why* did you write that you'd like
>     to move forward with the pandas 1.5 -> 2.1
>     transition? What's your reason?

A thought from me on this: pandas 2.1 has many improvements over
pandas 1.5.  And increasingly, other packages will be requiring these
new features.  So why would one not want to move forward with it?

> 2.) What may be the advantage of migrating to
>     version 3.0 of Cython?

It is compatible with Python 3.12, whereas the current version of
Cython in Debian (0.29.x) is not really.  (For example, it has an
"import imp" in it, and this breaks with Python 3.12, which has
removed this deprecated module.)  As Cython 0.29.x is no longer
maintained upstream, having been superseded by Cython 3.x after many
years of development, our options are to either continue to patch
Cython 0.29.x within Debian to keep it working with Python 3.12 or to
upgrade to Cython 3.x.  As there is also software which now depends on
Cython 3.x to build, the former option seems unappealing.  (At best,
we might wish to keep the cython-legacy package around for building
packages which can't yet use Cython 3.x, but that should be a
short-term thing, not a long-term one.)

> 3.) The following one-liner suggests 44 debian
>     packages might be affected by the breaks
>     Rebecca said would be caused by pandas 2.x:
> 
>     $ for s in augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata python-biom-format python-cooler python-nanoget python-skbio python-ulmo q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas ; do apt-cache search "$s" ; done | less

This does not seem like a particularly helpful one-liner; it picks up
packages such as python3-dyda-pipeline-config which are not in the
original list.  Instead, you perhaps want to count the number of
packages depending on these packages.  But what Rebecca is looking at
(I think) is how many packages would need fixing by the pandas
upgrade.

(But it is probably worse than this: I'm guessing these are only the
packages which fail to build with pandas 2.x or whose autopkgtest
fails with pandas 2.x.  But there may well be other breakage caused by
the upgrade which is not detectable in this way.  That is an issue
which will have to be handled by individual packages as they are
discovered, and the timing of the pandas upgrade is not related to
this problem.)

> 4.) The break that worries me the most is
>     sklearn-pandas, because it seems to me that
>     sklearn is 
> 
>         popular and 
> 
>         fundamental.

It seems that sklearn-pandas is abandoned; there were just two commits
in 2022, and prior to that was May 2021.  There has been no activity
since.  If someone is willing to patch it for Pandas 2.x, great
(perhaps you might help the maintainer to do this?), otherwise it
might have to drop out of Debian.

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#15344

On 10.12.23 14:06, Rebecca N. Palmer wrote:
> I'd like to move forward with the pandas 1.5 -> 2.1 transition 
> reasonably soon.
> 
> Given that pandas 2.x is *not* required for Python 3.12 (but is required 
> for Cython 3.0), should we wait for the Python 3.12 transition to be 
> done first?
> 
> These are broken by pandas 2.x and have a possible (but untested) fix in 
> their bug - please test and apply it:
> dask(?) dials influxdb-python* python-altair python-feather-format 
> python-upsetplot seaborn tqdm*
> (* = this package is currently also broken for a non-pandas reason, 
> probably Python 3.12, that I don't have a fix for)
> 
> These are broken by pandas 2.x and have no known-to-me fix:
> augur cnvkit dyda emperor esda mirtop pymatgen pyranges python-anndata 
> python-biom-format python-cooler python-nanoget python-skbio python-ulmo 
> q2-quality-control q2-demux q2-taxa q2-types q2templates sklearn-pandas
> Some generic things to try are pandas.util.testing -> pandas.testing, 
> .iteritems() -> .items(), and if one exists, a more recent upstream 
> version.
> 
> Is this an acceptable amount of breakage or should we continue to wait? 
> Bear in mind that if we wait too long, we may be forced into it by some 
> transition further up the stack (e.g. a future Python or numpy) that 
> breaks pandas 1.x.

up to the maintainers. But please wait at least until the current pandas 
and numpy migrated to testing, e.g. that the autopkg tests of pandas and 
numpy triggered by python3-defaults pass.

Is there a way to see the binNMUs which are still stuck in unstable, and 
don't migrate?

Matthias

[toc] | [prev] | [next] | [standalone]

#15350

On 11.12.23 08:12, Matthias Klose wrote:
> On 10.12.23 14:06, Rebecca N. Palmer wrote:
>> Is this an acceptable amount of breakage or should we continue to 
>> wait? Bear in mind that if we wait too long, we may be forced into it 
>> by some transition further up the stack (e.g. a future Python or 
>> numpy) that breaks pandas 1.x.
> 
> up to the maintainers. But please wait at least until the current pandas 
> and numpy migrated to testing, e.g. that the autopkg tests of pandas and 
> numpy triggered by python3-defaults pass.

I just nmued pyrle and sorted-nearest, having dependencies on 
cython3-legacy, letting the pyranges autopkg tests fail. Once this 
succeeds, pandas should be able to migrate.

[toc] | [prev] | [next] | [standalone]

#15421

Control: severity 1053943 1053939 1053942 1044053 1044056 serious
Control: severity 1044074 1053946 1044078 1044079 1044077 serious
Control: severity 1044071 1044067 1044068 1044055 1044060 serious
Control: severity 1044072 1044073 1044064 1053945 1044054 serious
Control: severity 1044076 1053940 1044057 1053944 1050144 serious

As previously discussed in this bug, I'd like to move pandas 2.x into 
unstable reasonably soon.  I'm aiming to get it in before the Ubuntu 
24.04 freeze (in about a month), but I am open to disagreement on 
whether this is a good idea.

dask, python-skbio and python-upsetplot have been fixed since the 
previous discussion, but that still leaves the above 25.  6 of these 
have a known-to-me fix (dials influxdb-python python-altair 
python-feather-format seaborn tqdm - see their bugs for details).

Hence, doing this transition now would involve breaking some reverse 
dependencies with no known fix, but given the number of packages 
involved, trying to wait until they're all fixed is rather likely to 
instead end in pandas 1.5 being broken by a new Python/numpy/etc.

[toc] | [prev] | [next] | [standalone]

#15423

Hi Rebecca,

Am Sun, Jan 21, 2024 at 03:29:21PM +0000 schrieb Rebecca N. Palmer:
> 
> Hence, doing this transition now would involve breaking some reverse
> dependencies with no known fix, but given the number of packages involved,
> trying to wait until they're all fixed is rather likely to instead end in
> pandas 1.5 being broken by a new Python/numpy/etc.

Just go for it and lets try to fix issues as soon as possible.

Thanks a lot for all your work on pandas

     Andreas.

-- 
http://fam-tille.de

[toc] | [prev] | [next] | [standalone]

#15424

[Multipart message — attachments visible in raw view] — view raw

Hi, how to remove myself from these lists?

Thank you

On Sun, 21 Jan 2024 at 18:30, Andreas Tille <andreas@an3as.eu> wrote:

> Hi Rebecca,
>
> Am Sun, Jan 21, 2024 at 03:29:21PM +0000 schrieb Rebecca N. Palmer:
> >
> > Hence, doing this transition now would involve breaking some reverse
> > dependencies with no known fix, but given the number of packages
> involved,
> > trying to wait until they're all fixed is rather likely to instead end in
> > pandas 1.5 being broken by a new Python/numpy/etc.
>
> Just go for it and lets try to fix issues as soon as possible.
>
> Thanks a lot for all your work on pandas
>
>      Andreas.
>
> --
> http://fam-tille.de
>
>

[toc] | [prev] | [next] | [standalone]

#15427

On Sun, Jan 21, 2024 at 03:29:21PM +0000, Rebecca N. Palmer wrote:
> Control: severity 1053943 1053939 1053942 1044053 1044056 serious
> Control: severity 1044074 1053946 1044078 1044079 1044077 serious
> Control: severity 1044071 1044067 1044068 1044055 1044060 serious
> Control: severity 1044072 1044073 1044064 1053945 1044054 serious
> Control: severity 1044076 1053940 1044057 1053944 1050144 serious
> 
> As previously discussed in this bug, I'd like to move pandas 2.x into
> unstable reasonably soon.  I'm aiming to get it in before the Ubuntu 24.04
> freeze (in about a month), but I am open to disagreement on whether this is
> a good idea.

Please could we wait until the "Python 3.12 is a supported version"
transition is completed?  Adding another 25 or so RC bugs at this
point will just slow down that transition.  (Unless pandas 1.5 is
preventing the transition, that is.)

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#15428

On 22/01/2024 11:51, Julian Gilbey wrote:
> Please could we wait until the "Python 3.12 is a supported version"
> transition is completed?

How are you defining that?  python3-defaults 3.11.6+ in testing?  (I was 
previously told 3.12-supporting pandas and numpy in testing, which has 
happened.  I don't think any of these 25 packages are on 
https://qa.debian.org/excuses.php?package=python3-defaults , but I 
haven't checked carefully, and at least influxdb-python and tqdm do have 
what I suspect are Python 3.12 related issues.)

> Adding another 25 or so RC bugs at this
> point will just slow down that transition.

What exactly do you want not done until then?   Just not uploading 
pandas 2.x to unstable, or is it also a problem to have these bugs 
marked as RC in the BTS?  (In all 22 cases that are in testing at all, 
the bug is also present in the version in testing, so it being RC 
shouldn't block migration.)

> (Unless pandas 1.5 is
> preventing the transition, that is.)

It isn't.

[toc] | [prev] | [next] | [standalone]

#15430

On Mon, Jan 22, 2024 at 08:50:55PM +0000, Rebecca N. Palmer wrote:
> On 22/01/2024 11:51, Julian Gilbey wrote:
> > Please could we wait until the "Python 3.12 is a supported version"
> > transition is completed?
> 
> How are you defining that?  python3-defaults 3.11.6+ in testing?  (I was
> previously told 3.12-supporting pandas and numpy in testing, which has
> happened.  I don't think any of these 25 packages are on
> https://qa.debian.org/excuses.php?package=python3-defaults , but I haven't
> checked carefully, and at least influxdb-python and tqdm do have what I
> suspect are Python 3.12 related issues.)

https://release.debian.org/transitions/html/python3.12-add.html

We're nearly there (the transition page says it's 99% done), and when
this transition is complete, then python3-defaults 3.11.6+ will be
able to migrate to testing.

I don't fully understand the problem with transitions, but there was a
request to hold back with significant upgrades until a
python3.12-supporting python3-defaults has reached testing.

> > Adding another 25 or so RC bugs at this
> > point will just slow down that transition.
> 
> What exactly do you want not done until then?   Just not uploading pandas
> 2.x to unstable, or is it also a problem to have these bugs marked as RC in
> the BTS?  (In all 22 cases that are in testing at all, the bug is also
> present in the version in testing, so it being RC shouldn't block
> migration.)

Yes - please don't upload it to unstable yet.  Uploading to
experimental is fine.

> > (Unless pandas 1.5 is
> > preventing the transition, that is.)
> 
> It isn't.

Good!

   Julian

[toc] | [prev] | [next] | [standalone]

#15437

Hi

On Tue, 23 Jan 2024 at 14:38, Julian Gilbey <julian@d-and-j.net> wrote:
> We're nearly there (the transition page says it's 99% done), and when
> this transition is complete, then python3-defaults 3.11.6+ will be
> able to migrate to testing.

python3-defaults/3.11.6-1 with Python 3.12 as a supported version is
now in testing [1].

> Yes - please don't upload it to unstable yet.  Uploading to
> experimental is fine.

Uploading to unstable now should be fine, but maybe wait for
pandas/1.5.3+dfsg-12 to migrate first (in about four hours).

Regards
Graham


[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055085#29

[toc] | [prev] | [next] | [standalone]

#15439

On Fri, Jan 26, 2024 at 08:43:03AM +0200, Graham Inggs wrote:
> Hi
> 
> On Tue, 23 Jan 2024 at 14:38, Julian Gilbey <julian@d-and-j.net> wrote:
> > We're nearly there (the transition page says it's 99% done), and when
> > this transition is complete, then python3-defaults 3.11.6+ will be
> > able to migrate to testing.
> 
> python3-defaults/3.11.6-1 with Python 3.12 as a supported version is
> now in testing [1].

Wonderful news!  Congratulations to everyone who helped to make this
happen!

Best wishes,

   Julian

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web