Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #13913 > unrolled thread

Bug#1003567: Please make a separate package for mistune 2.x

Back to article view | Back to linux.debian.maint.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Bug#1003567: Please make a separate package for mistune 2.x Pierre-Elliott Bécue <peb@debian.org> - 2022-02-03 23:40 +0100
    Bug#1003567: Please make a separate package for mistune 2.x Julian Gilbey <julian@d-and-j.net> - 2022-02-04 17:10 +0100
      Bug#1003567: Please make a separate package for mistune 2.x Nilesh Patra <nilesh@tchncs.de> - 2022-02-04 17:30 +0100
        Bug#1003567: Please make a separate package for mistune 2.x Jerome Charaoui <jerome@riseup.net> - 2022-02-04 18:40 +0100
    Bug#1003567: Please make a separate package for mistune 2.x Julian Gilbey <julian@d-and-j.net> - 2022-02-04 17:10 +0100
      Bug#1003567: Please make a separate package for mistune 2.x Nilesh Patra <nilesh@tchncs.de> - 2022-02-04 17:10 +0100
        Bug#1003567: Please make a separate package for mistune 2.x Pierre-Elliott Bécue <peb@debian.org> - 2022-02-04 17:40 +0100
        Bug#1003567: Please make a separate package for mistune 2.x Julian Gilbey <julian@d-and-j.net> - 2022-02-05 20:00 +0100
          Bug#1003567: Please make a separate package for mistune 2.x Nilesh Patra <nilesh@tchncs.de> - 2022-02-05 20:30 +0100
            Bug#1003567: Please make a separate package for mistune 2.x Pierre-Elliott Bécue <peb@debian.org> - 2022-02-05 23:00 +0100
              Bug#1003567: Please make a separate package for mistune 2.x Sandro Tosi <morph@debian.org> - 2022-02-05 23:00 +0100

#13913 — Bug#1003567: Please make a separate package for mistune 2.x

[Multipart message — attachments visible in raw view] — view raw

Hi Michael,

"Michael R. Crusoe" <crusoe@debian.org> wrote on 24/01/2022 at 11:39:23+0100:

> [[PGP Signed Part:No public key for 3C26763F6C67E6E2 created at 2022-01-24T11:39:23+0100 using RSA]]
> On Tue, 11 Jan 2022 22:38:44 +0100 peb@debian.org wrote:
>> Package: python3-schema-salad
>> Severity: wishlist
>>
>> Dear maintainer,
>>
>> Your package python3-schema-salad depends on python3-mistune 0.8.4
>> which is no longer maintained and deprecated in favour of version
>> 2.0.0.
>>
>> As python3-mistune 2.0.0 is not backward compatible, I reverted the
>> upload of it that I did in unstable and python3-mistune 0.8.4 will
>> stay around a bit longer.
>>
>> In the meantime, please try to see if upstream of
>> python3-schema-salad either released a version that is compatible
>> with python3-mistune 2.0.0 or if python3-schema-salad can easily be
>> fixed to support such a version. As soon as you're ready to upload
>> your package please update this bug report and we'll try to
>> coordinate so that I release python3-mistune at a time that is fit
>> for you to also release python3-schema-salad.
>
> Dear Pierre-Elliott Bécue,
>
> Thank you for trying to find a way forward in this mess. I am also the
> upstream maintainer for schema-salad. We have yet to find a way to 
> support mistune 2.x
>
> Here is our attempt so far:
>
> https://github.com/common-workflow-language/schema_salad/pull/496/files?short_path=5c6a130#diff-5c6a1301c6b59b30a040d747d065e861d3dd98bde0e5a4356d92d594e9835986
>
> And an issue I opened with mistune directly about a regression in
> multi-line list handling that goes against multiple markdown specs: 
> https://github.com/lepture/mistune/issues/296
>
> We have not decided if we will switch to another library, wait for a
> fixed mistune 2.x release, or keep with the older version.
>
>> I intend to upload src:mistune 2.0.0 to unstable between March the
>
>> 15th and April the 15th (depending on the progress of its
>> reverse-dependencies). I'll raise the severity of this bug
>> approximately a month before making such an upload.
>
> Since Mistune 2.0.0 regresses its support for standard markdown, I ask
> that a separate package be made for mistune 2.x to give more time for 
> mistune 0.8.x users to migrate to mistune 2.x or another library entirely.
>
> Cheers,

I'm not formally against it, but it's not really standard in my
opinion. It'd lead to maintenance of two packages, probably on some long
term (as it'd relieve the pressure to migrate for maintainers of
reverse-deps of mistune).

Besides, having a source package named "mistune2" while upstream's
package is named "mistune" adds also another layer of complexity.

I'd like to have some python team members' opinion on this, and I am not
sure to be eager to do it as of now.

Cheers!
-- 
PEB

[toc] | [next] | [standalone]

#13917

On Fri, Feb 04, 2022 at 09:27:59PM +0530, Nilesh Patra wrote:
> On 2/4/22 9:18 PM, Julian Gilbey wrote:
> > Basically, the mistune upstream author has completely messed up on
> > this by making what is essentially a completely different package with
> > superficially similar functionality but the same name.
> 
> True.
> > [...]
> > _mistune.py within the Debian package,
> > and have nbconvert do "import nbconvert.filters._mistune as mistune"
> > (see /usr/lib/python3/dist-packages/nbconvert/filters/markdown_mistune.py).
> > That seems like an eminently sensible solution to this problem.
> 
> But that'd lead to a number of mistune's embedded copies in a huge number of packages; since majority of
> the rev-deps (when I last checked) haven't adapted to this new version. When they do,
> and it becomes a overhead to fix each one later.
> Even worse, if we discover a security problem sometime later, then all such packages would be
> effected, and that honestly does not look like a good idea to me.

This is true, though there are only 7 reverse dependencies currently
in testing.

> I somehow do not understand the urgency of uploading this newer version, as the maintainer said:
> 
> | I intend to upload src:mistune 2.0.0 to unstable between March the
> | 15th and April the 15th (depending on the progress of its
> | reverse-dependencies).
> 
> We could simply wait a little more for the dust to settle, IMHO.

That would be a reasonable approach, but how long will it take for the
dust to settle?  With this major change, and no guidance from upstream
on how to migrate, and at least a number of upstream authors happy to
rely on setup.py having "mistune <1.0.0" in the install_requires
field, it might be several months or longer before things are fixed
upstream.  And what do we do when some packages have converted and
some haven't?

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#13920

[Multipart message — attachments visible in raw view] — view raw

On 2/4/22 9:33 PM, Julian Gilbey wrote:
>>> _mistune.py within the Debian package,
>>> and have nbconvert do "import nbconvert.filters._mistune as mistune"
>>> (see /usr/lib/python3/dist-packages/nbconvert/filters/markdown_mistune.py).
>>> That seems like an eminently sensible solution to this problem.
>>
>> But that'd lead to a number of mistune's embedded copies in a huge number of packages; since majority of
>> the rev-deps (when I last checked) haven't adapted to this new version. When they do,
>> and it becomes a overhead to fix each one later.
>> Even worse, if we discover a security problem sometime later, then all such packages would be
>> effected, and that honestly does not look like a good idea to me.
> 
> This is true, though there are only 7 reverse dependencies currently
> in testing.

Yeah, in total the reverse-deps are 8 and there is one different reverse-build-dep (pasted at end of this mail)
But the problem is if these fall through the cracks, and close to the release we notice some
package is not looking nice.

Also, if you suppose that the upstreams are very slow, and we get to the new debian release with these things still
embedded, it'd be a real mess to upload fixes to stable, right...
  
>> I somehow do not understand the urgency of uploading this newer version, as the maintainer said:
>>
>> | I intend to upload src:mistune 2.0.0 to unstable between March the
>> | 15th and April the 15th (depending on the progress of its
>> | reverse-dependencies).
>>
>> We could simply wait a little more for the dust to settle, IMHO.
> 
> That would be a reasonable approach, but how long will it take for the
> dust to settle?  With this major change, and no guidance from upstream
> on how to migrate, and at least a number of upstream authors happy to
> rely on setup.py having "mistune <1.0.0" in the install_requires
> field, it might be several months or longer before things are fixed
> upstream.
I think we could do this transition even a month or two before the soft freeze starts,
which is roughly an year from now (considering general trend timings of starting in feb in release year).
Situation might improve by then, I suppose.

If it does not, we could still go ahead with the older python3-mistune in the archive, I do not see
a huge problem there, except for maybe a few mistune uploads to stable if desired.

> And what do we do when some packages have converted and
> some haven't?

In such a case, we atleast will have a few examples to see how to go about doing the API changes the right way.
We could patch out at our end and send the changes upstream for review provided we are stuck in an unfortunate situation.

Let me know what you'd think about this?

Regards,
Nilesh


Reverse-Depends
* giara [amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x]
* iredis
* lektor
* lookatme
* matrix-mirage [amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x]
* python3-flasgger
* python3-m2r
* python3-schema-salad

Reverse-Testsuite-Triggers
* markdown-it-py

Reverse-Build-Depends
* lektor
* lookatme
* markdown-it-py
* python-flasgger
* python-m2r
* python-schema-salad

[toc] | [prev] | [next] | [standalone]

#13922

Le 2022-02-04 à 11 h 24, Nilesh Patra a écrit :
> On 2/4/22 9:33 PM, Julian Gilbey wrote:
>>>> _mistune.py within the Debian package,
>>>> and have nbconvert do "import nbconvert.filters._mistune as mistune"
>>>> (see 
>>>> /usr/lib/python3/dist-packages/nbconvert/filters/markdown_mistune.py).
>>>> That seems like an eminently sensible solution to this problem.
>>>
>>> But that'd lead to a number of mistune's embedded copies in a huge 
>>> number of packages; since majority of
>>> the rev-deps (when I last checked) haven't adapted to this new 
>>> version. When they do,
>>> and it becomes a overhead to fix each one later.
>>> Even worse, if we discover a security problem sometime later, then 
>>> all such packages would be
>>> effected, and that honestly does not look like a good idea to me.
>>
>> This is true, though there are only 7 reverse dependencies currently
>> in testing.
> 
> Yeah, in total the reverse-deps are 8 and there is one different 
> reverse-build-dep (pasted at end of this mail)
> But the problem is if these fall through the cracks, and close to the 
> release we notice some
> package is not looking nice.
> 
> Also, if you suppose that the upstreams are very slow, and we get to the 
> new debian release with these things still
> embedded, it'd be a real mess to upload fixes to stable, right...
> 
>>> I somehow do not understand the urgency of uploading this newer 
>>> version, as the maintainer said:
>>>
>>> | I intend to upload src:mistune 2.0.0 to unstable between March the
>>> | 15th and April the 15th (depending on the progress of its
>>> | reverse-dependencies).
>>>
>>> We could simply wait a little more for the dust to settle, IMHO.
>>
>> That would be a reasonable approach, but how long will it take for the
>> dust to settle?  With this major change, and no guidance from upstream
>> on how to migrate, and at least a number of upstream authors happy to
>> rely on setup.py having "mistune <1.0.0" in the install_requires
>> field, it might be several months or longer before things are fixed
>> upstream.
> I think we could do this transition even a month or two before the soft 
> freeze starts,
> which is roughly an year from now (considering general trend timings of 
> starting in feb in release year).
> Situation might improve by then, I suppose.
> 
> If it does not, we could still go ahead with the older python3-mistune 
> in the archive, I do not see
> a huge problem there, except for maybe a few mistune uploads to stable 
> if desired.
> 
>> And what do we do when some packages have converted and
>> some haven't?
> 
> In such a case, we atleast will have a few examples to see how to go 
> about doing the API changes the right way.
> We could patch out at our end and send the changes upstream for review 
> provided we are stuck in an unfortunate situation.

FWIW, there's a recent patch and PR for the lektor upstream which add 
mistune 2.x support, so for this particular project I don't think a 
separate package is necessary.

-- Jerome

[toc] | [prev] | [next] | [standalone]

#13918

On Thu, Feb 03, 2022 at 11:34:26PM +0100, Pierre-Elliott Bécue wrote:
> Hi Michael,
> 
> > Since Mistune 2.0.0 regresses its support for standard markdown, I ask
> > that a separate package be made for mistune 2.x to give more time for 
> > mistune 0.8.x users to migrate to mistune 2.x or another library entirely.
> >
> > Cheers,
> 
> I'm not formally against it, but it's not really standard in my
> opinion. It'd lead to maintenance of two packages, probably on some long
> term (as it'd relieve the pressure to migrate for maintainers of
> reverse-deps of mistune).
> 
> Besides, having a source package named "mistune2" while upstream's
> package is named "mistune" adds also another layer of complexity.
> 
> I'd like to have some python team members' opinion on this, and I am not
> sure to be eager to do it as of now.

I agree that this sounds like a terrible idea.  What would the Python
module be called in a separate python3-mistune2 package be called?  If
it were called mistune, then python3-mistune2 would have to conflict
with python3-mistune (or "python3-mistune0.84" or similar), and there
would be little benefit.  If the module itself were renamed to
mistune2, then the Debian mistune package would be incompatible with
any other software requiring mistune.

Basically, the mistune upstream author has completely messed up on
this by making what is essentially a completely different package with
superficially similar functionality but the same name.

What the Debian nbconvert maintainers have done is to vendor mistune
0.8.4: they now include it as _mistune.py within the Debian package,
and have nbconvert do "import nbconvert.filters._mistune as mistune"
(see /usr/lib/python3/dist-packages/nbconvert/filters/markdown_mistune.py).
That seems like an eminently sensible solution to this problem.  Maybe
not ideal, but it will work until the upstream maintainers find a way
to work with mistune 2.0.x.

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#13919

[Multipart message — attachments visible in raw view] — view raw

On 2/4/22 9:18 PM, Julian Gilbey wrote:
> Basically, the mistune upstream author has completely messed up on
> this by making what is essentially a completely different package with
> superficially similar functionality but the same name.

True.
  
> [...]
> _mistune.py within the Debian package,
> and have nbconvert do "import nbconvert.filters._mistune as mistune"
> (see /usr/lib/python3/dist-packages/nbconvert/filters/markdown_mistune.py).
> That seems like an eminently sensible solution to this problem.

But that'd lead to a number of mistune's embedded copies in a huge number of packages; since majority of
the rev-deps (when I last checked) haven't adapted to this new version. When they do,
and it becomes a overhead to fix each one later.
Even worse, if we discover a security problem sometime later, then all such packages would be
effected, and that honestly does not look like a good idea to me.

I somehow do not understand the urgency of uploading this newer version, as the maintainer said:

| I intend to upload src:mistune 2.0.0 to unstable between March the
| 15th and April the 15th (depending on the progress of its
| reverse-dependencies).

We could simply wait a little more for the dust to settle, IMHO.

Regards,
Nilesh

[toc] | [prev] | [next] | [standalone]

#13921

[Multipart message — attachments visible in raw view] — view raw

Le 4 février 2022 16:57:59 GMT+01:00, Nilesh Patra <nilesh@tchncs.de> a écrit :
>On 2/4/22 9:18 PM, Julian Gilbey wrote:
>> Basically, the mistune upstream author has completely messed up on
>> this by making what is essentially a completely different package with
>> superficially similar functionality but the same name.
>
>True.
>  
>> [...]
>> _mistune.py within the Debian package,
>> and have nbconvert do "import nbconvert.filters._mistune as mistune"
>> (see /usr/lib/python3/dist-packages/nbconvert/filters/markdown_mistune.py).
>> That seems like an eminently sensible solution to this problem.
>
>But that'd lead to a number of mistune's embedded copies in a huge number of packages; since majority of
>the rev-deps (when I last checked) haven't adapted to this new version. When they do,
>and it becomes a overhead to fix each one later.
>Even worse, if we discover a security problem sometime later, then all such packages would be
>effected, and that honestly does not look like a good idea to me.
>
>I somehow do not understand the urgency of uploading this newer version, as the maintainer said:
>
>| I intend to upload src:mistune 2.0.0 to unstable between March the
>| 15th and April the 15th (depending on the progress of its
>| reverse-dependencies).
>
>We could simply wait a little more for the dust to settle, IMHO.
>
>Regards,
>Nilesh
>

Because some packages I maintain depend on mistune 2.x and mistune 0.8.4 is, whether we like it or not, obsolete.

I can't really afford to wait for a year to try having mailman3 in testing again...
--
Pierre-Elliott Bécue
From my phone

[toc] | [prev] | [next] | [standalone]

#13923

On Fri, Feb 04, 2022 at 09:27:59PM +0530, Nilesh Patra wrote:
> On 2/4/22 9:18 PM, Julian Gilbey wrote:
> > [...]
> > _mistune.py within the Debian package,
> > and have nbconvert do "import nbconvert.filters._mistune as mistune"
> > (see /usr/lib/python3/dist-packages/nbconvert/filters/markdown_mistune.py).
> > That seems like an eminently sensible solution to this problem.
> 
> But that'd lead to a number of mistune's embedded copies in a huge number of packages; since majority of
> the rev-deps (when I last checked) haven't adapted to this new version. When they do,
> and it becomes a overhead to fix each one later.
> Even worse, if we discover a security problem sometime later, then all such packages would be
> effected, and that honestly does not look like a good idea to me.

I have just had another idea, which might solve all of the problems:
create a new Debian package called mistune0 (or mistune1), which
contains the legacy version of mistune, but with the Python module
called "mistune0" instead of "mistune".  Then it will be
co-installable with mistune 2.x, and the packages which still depend
on mistune 0.8.4 could be patched to say "import mistune0 as mistune"
until they are updated upstream.  This will also avoid having multiple
copies of the legacy code in the archive, which addresses the security
issue, and allow those packages which have migrated to mistune 2.x to
still say "import mistune".

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#13924

[Multipart message — attachments visible in raw view] — view raw

On 2/6/22 12:20 AM, Julian Gilbey wrote:
> On Fri, Feb 04, 2022 at 09:27:59PM +0530, Nilesh Patra wrote:
>> On 2/4/22 9:18 PM, Julian Gilbey wrote:
>>> [...]
>>> _mistune.py within the Debian package,
>>> and have nbconvert do "import nbconvert.filters._mistune as mistune"
>>> (see /usr/lib/python3/dist-packages/nbconvert/filters/markdown_mistune.py).
>>> That seems like an eminently sensible solution to this problem.
>>
>> But that'd lead to a number of mistune's embedded copies in a huge number of packages; since majority of
>> the rev-deps (when I last checked) haven't adapted to this new version. When they do,
>> and it becomes a overhead to fix each one later.
>> Even worse, if we discover a security problem sometime later, then all such packages would be
>> effected, and that honestly does not look like a good idea to me.
> 
> I have just had another idea, which might solve all of the problems:
> create a new Debian package called mistune0 (or mistune1), which
> contains the legacy version of mistune, but with the Python module
> called "mistune0" instead of "mistune".  Then it will be
> co-installable with mistune 2.x, and the packages which still depend
> on mistune 0.8.4 could be patched to say "import mistune0 as mistune"
> until they are updated upstream.  This will also avoid having multiple
> copies of the legacy code in the archive, which addresses the security
> issue, and allow those packages which have migrated to mistune 2.x to
> still say "import mistune".

Ah, looks like we just had to think in the reverse direction from the initial proposal (mistune-{0,1} instead of mistune-{1,2}).
Indeed, that sounds like a much better plan.
I hope PEB agrees with this as well, would like to hear from them :)

Thanks for the discussion, Julian :)

Regards,
Nilesh

[toc] | [prev] | [next] | [standalone]

#13926

[Multipart message — attachments visible in raw view] — view raw

Nilesh Patra <nilesh@tchncs.de> wrote on 05/02/2022 at 20:23:44+0100:

> [[PGP Signed Part:No public key for 00BAE74B343369F1 created at 2022-02-05T20:23:44+0100 using RSA]]
> On 2/6/22 12:20 AM, Julian Gilbey wrote:
>> On Fri, Feb 04, 2022 at 09:27:59PM +0530, Nilesh Patra wrote:
>>> On 2/4/22 9:18 PM, Julian Gilbey wrote:
>>>> [...]
>>>> _mistune.py within the Debian package,
>>>> and have nbconvert do "import nbconvert.filters._mistune as mistune"
>>>> (see /usr/lib/python3/dist-packages/nbconvert/filters/markdown_mistune.py).
>>>> That seems like an eminently sensible solution to this problem.
>>>
>>> But that'd lead to a number of mistune's embedded copies in a huge number of packages; since majority of
>>> the rev-deps (when I last checked) haven't adapted to this new version. When they do,
>>> and it becomes a overhead to fix each one later.
>>> Even worse, if we discover a security problem sometime later, then all such packages would be
>>> effected, and that honestly does not look like a good idea to me.
>> I have just had another idea, which might solve all of the problems:
>> create a new Debian package called mistune0 (or mistune1), which
>> contains the legacy version of mistune, but with the Python module
>> called "mistune0" instead of "mistune".  Then it will be
>> co-installable with mistune 2.x, and the packages which still depend
>> on mistune 0.8.4 could be patched to say "import mistune0 as mistune"
>> until they are updated upstream.  This will also avoid having multiple
>> copies of the legacy code in the archive, which addresses the security
>> issue, and allow those packages which have migrated to mistune 2.x to
>> still say "import mistune".
>
> Ah, looks like we just had to think in the reverse direction from the initial proposal (mistune-{0,1} instead of mistune-{1,2}).
> Indeed, that sounds like a much better plan.
> I hope PEB agrees with this as well, would like to hear from them :)
>
> Thanks for the discussion, Julian :)
>
> Regards,
> Nilesh

I'd like other python team member's opinion on this, and I'm not eager
to maintain that legacy package, as I tend to not want to maintain
obsolete software. Still, I can do the initial work of creating it.

-- 
PEB

[toc] | [prev] | [next] | [standalone]

#13927

> I'd like other python team member's opinion on this, and I'm not eager
> to maintain that legacy package, as I tend to not want to maintain
> obsolete software. Still, I can do the initial work of creating it.

i wouldnt expect much maintenance needed tho: 0.8.4 is essentially
dead upstream, so it will only need to be kept around until its rdeps
are ported to mistune 2.x

the proposal is "okay", it has the downside of requiring the current
rdeps to be updated to point to the new binary package name for the
old mistune, but it leaves the namespace open for mistune 2 to take
over python3-mistune bin pkg

-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web