Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #8959 > unrolled thread

Packaging new version of ZODB (Zope Object Database)

Back to article view | Back to linux.debian.maint.python

Contents

  Packaging new version of ZODB (Zope Object Database) Julien Muchembled <jm@jmuchemb.eu> - 2016-11-01 20:20 +0100
    Re: Packaging new version of ZODB (Zope Object Database) Julien Muchembled <jm@jmuchemb.eu> - 2016-11-02 02:00 +0100
      Re: Packaging new version of ZODB (Zope Object Database) Paul Wise <pabs@debian.org> - 2016-11-02 04:50 +0100
        Re: Packaging new version of ZODB (Zope Object Database) Julien Muchembled <jm@jmuchemb.eu> - 2016-11-03 20:20 +0100
    Re: Packaging new version of ZODB (Zope Object Database) Arnaud Fontaine <arnau@debian.org> - 2016-11-02 03:00 +0100
      Re: Packaging new version of ZODB (Zope Object Database) Barry Warsaw <barry@debian.org> - 2016-11-03 00:00 +0100
        Re: Packaging new version of ZODB (Zope Object Database) Scott Kitterman <debian@kitterman.com> - 2016-11-03 03:20 +0100
        Re: Packaging new version of ZODB (Zope Object Database) Arnaud Fontaine <arnau@debian.org> - 2016-11-03 03:30 +0100
        Re: Packaging new version of ZODB (Zope Object Database) Julien Muchembled <jm@jmuchemb.eu> - 2016-11-03 20:40 +0100
          Re: Packaging new version of ZODB (Zope Object Database) Barry Warsaw <barry@debian.org> - 2016-11-04 15:50 +0100
            Re: Packaging new version of ZODB (Zope Object Database) Scott Kitterman <debian@kitterman.com> - 2016-11-04 16:20 +0100
              Re: Packaging new version of ZODB (Zope Object Database) Julien Muchembled <jm@jmuchemb.eu> - 2016-12-05 20:30 +0100
                gitignore and import of upstream tarballs (was Re: Packaging new  version of ZODB (Zope Object Database)) Julien Muchembled <jm@jmuchemb.eu> - 2016-12-05 21:00 +0100
    DPMT Membership request​ (was Packaging new version of ZODB (Zope Object Database)) Julien Muchembled <jm@jmuchemb.eu> - 2016-11-30 19:30 +0100
      Re: DPMT Membership request​ (was  Packaging new version of ZODB (Zope Object Database)) Piotr Ożarowski <piotr@debian.org> - 2016-12-01 10:20 +0100
    Re: Packaging new version of ZODB (Zope Object Database) Julien Muchembled <jm@jmuchemb.eu> - 2016-12-15 20:20 +0100

#8959 — Packaging new version of ZODB (Zope Object Database)

Hello,

I am co-maintaining the 'zodb' package and I am also an upstream developer.
  http://www.zodb.org/ - a native object database for Python

Debian currently packages branch 3.x of ZODB, which is getting quite old, with only support for Python 2. It's been a while that I am working locally in packaging a newer version, and I have almost finished.

With branch 4.x, ZODB has been splitted into 4 packages (-> means Depends):
 ZEO -> ZODB -> (Btrees, persistent)
The split of persistent was already backported.

And in addition to that:
- new dependencies that are not packaged yet
- some existing dependencies needs to be updated as well

I write to debian-python, because some of the involved packages are not specific to Zope. Actually, I even think that ZODB itself is not specific to Zope, but well, changing section of existing packages can be another topic. I've just requested to join the Python Modules Packaging Team.

So here is the status of my work:

* Generally, I add support for Python 3 everywhere it is missing. I only added PyPy support when it's trivial (e.g. zodbpickle), because that would need work on even more existing packages.

* python-zodb (almost finished)

- changelog entries to be written
- decide an upgrade plan for the split of ZEO (some users may have installed zodb for it): maybe a Recommends: ?
- to give a better overview, here's the new Depends: for Python2:
  python-btrees (>= 4.2.0), python-persistent (>= 4.2.0), python-six, python-transaction (>= 1.5.0), python-zc.lockfile, python-zconfig, python-zodbpickle (>= 0.6.0), python-zope.interface, python:any (<< 2.8), python:any (>= 2.7.5-5~)

* python-persistent (done)

- trivial update to 4.2.1
- I'd like to commit to python-persistent.git (python-modules), or someone does it for me

* python-btrees & python-zodbpickle (done)
https://github.com/zopefoundation/BTrees
https://github.com/zopefoundation/zodbpickle

- new packages
- Debian Python Modules Team

* python-transaction (done)

- update to version 1.6.1
- svn repos is late compared to the packaged version

* python-zeo (not started)
https://github.com/zopefoundation/ZEO

* python-zdaemon (done)

- needs to be updated for ZEO
- new deps: mock, zc.customdoctests

* zc.customdoctests (done)
https://github.com/zopefoundation/zc.customdoctests

- new package
- Debian Python Modules Team

I haven't done any ITP yet.

Julien

[toc] | [next] | [standalone]

#8963

Le 11/01/16 à 20:10, Julien Muchembled a écrit :
> * python-btrees & python-zodbpickle (done)
> https://github.com/zopefoundation/BTrees
> https://github.com/zopefoundation/zodbpickle
> 
> - new packages
> - Debian Python Modules Team
> 
> [...]
> 
> * python-zeo (not started)
> https://github.com/zopefoundation/ZEO
> 
> [...]
> 
> * zc.customdoctests (done)
> https://github.com/zopefoundation/zc.customdoctests
> 
> - new package
> - Debian Python Modules Team
> 
> I haven't done any ITP yet.

Bug#842870: ITP: zodbpickle -- Fork of pickle module, for ZODB
Bug#842874: ITP: python-btrees -- scalable persistent object containers for Python
Bug#842876: ITP: zc.customdoctests -- Use Python doctest with other languages

I'll do ZEO later.

Julien

[toc] | [prev] | [next] | [standalone]

#8967

On Wed, Nov 2, 2016 at 8:51 AM, Julien Muchembled wrote:

> Bug#842870: ITP: zodbpickle -- Fork of pickle module, for ZODB

Please ensure the security team are informed about this fork, via
their embedded-code-copies file:

https://wiki.debian.org/EmbeddedCodeCopies

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

[toc] | [prev] | [next] | [standalone]

#8975

[Multipart message — attachments visible in raw view] — view raw

Le 11/02/16 à 04:41, Paul Wise a écrit :
> On Wed, Nov 2, 2016 at 8:51 AM, Julien Muchembled wrote:
> 
>> Bug#842870: ITP: zodbpickle -- Fork of pickle module, for ZODB
> 
> Please ensure the security team are informed about this fork, via
> their embedded-code-copies file:
> 
> https://wiki.debian.org/EmbeddedCodeCopies
> 
Thanks, I didn't know. I'll do it.

BTW, about the fact that it's a fork, I'm not sure how to write debian/copyright
The upstream license is somewhat a concatenation of:
- the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2 (license of the original code)
- and the Zope Public License (ZPL) Version 2.1 (contributions of the Zope Foundation)

 https://github.com/zopefoundation/zodbpickle/blob/master/LICENSE.txt

As starting point, I have the attached file, which is similar to what I have to python-btrees and zc.customdoctests

Julien

[toc] | [prev] | [next] | [standalone]

#8965

[Multipart message — attachments visible in raw view] — view raw

Hi,

> I write  to debian-python, because  some of the involved  packages are
> not specific to  Zope. Actually, I even think that  ZODB itself is not
> specific to Zope, but well,  changing section of existing packages can
> be another topic.

This has  already been discussed  but all  the packages in  pkg-zope SVN
repository will have to be  moved to python-{modules, apps} repositories
(because there  is almost no activity  on pkg-zope and most  modules are
used independently  of Zope anyway)  and we should use  debian-python ML
for the  same reason,  so yes,  please use  debian-python ML  and commit
everything to python-{modules, apps} repositories.

Also, let  me know  when you  want to  upload as  I can  sponsor without
problem.

Cheers,
-- 
Arnaud Fontaine

[toc] | [prev] | [next] | [standalone]

#8971

[Multipart message — attachments visible in raw view] — view raw

On Nov 02, 2016, at 10:46 AM, Arnaud Fontaine wrote:

>> I write  to debian-python, because  some of the involved  packages are
>> not specific to  Zope. Actually, I even think that  ZODB itself is not
>> specific to Zope, but well,  changing section of existing packages can
>> be another topic.  
>
>This has  already been discussed  but all  the packages in  pkg-zope SVN
>repository will have to be  moved to python-{modules, apps} repositories
>(because there  is almost no activity  on pkg-zope and most  modules are
>used independently  of Zope anyway)  and we should use  debian-python ML
>for the  same reason,  so yes,  please use  debian-python ML  and commit
>everything to python-{modules, apps} repositories.

+1.  I do still touch some of the ztk packages and would dearly love to ditch
svn, but just haven't had the time to think about a proper migration.  Should
we just admit defeat and do on-demand conversions, preserving history if
possible but not worrying about it too much?

And then what about just using gbp and ignoring git-dpm?  The latter still
kind of works but we know it's a dead-end.  Anybody looked at dgit?  Is that
a useful option?

can-of-worms-ly y'rs,
-Barry

[toc] | [prev] | [next] | [standalone]

#8973

On November 2, 2016 6:51:56 PM EDT, Barry Warsaw <barry@debian.org> wrote:
>On Nov 02, 2016, at 10:46 AM, Arnaud Fontaine wrote:
>
>>> I write  to debian-python, because  some of the involved  packages
>are
>>> not specific to  Zope. Actually, I even think that  ZODB itself is
>not
>>> specific to Zope, but well,  changing section of existing packages
>can
>>> be another topic.  
>>
>>This has  already been discussed  but all  the packages in  pkg-zope
>SVN
>>repository will have to be  moved to python-{modules, apps}
>repositories
>>(because there  is almost no activity  on pkg-zope and most  modules
>are
>>used independently  of Zope anyway)  and we should use  debian-python
>ML
>>for the  same reason,  so yes,  please use  debian-python ML  and
>commit
>>everything to python-{modules, apps} repositories.
>
>+1.  I do still touch some of the ztk packages and would dearly love to
>ditch
>svn, but just haven't had the time to think about a proper migration. 
>Should
>we just admit defeat and do on-demand conversions, preserving history
>if
>possible but not worrying about it too much?
>
>And then what about just using gbp and ignoring git-dpm?  The latter
>still
>kind of works but we know it's a dead-end.  Anybody looked at dgit?  Is
>that
>a useful option?

Dgit and git-dpm are orthogonal.  I'd suggest converting the same way we did DPMT for now and then they'll be included in whatever we migrate to next.

Scott K

[toc] | [prev] | [next] | [standalone]

#8974

[Multipart message — attachments visible in raw view] — view raw

Hi,

Barry Warsaw <barry@debian.org> writes:

> +1.  I do still  touch some of the ztk packages  and would dearly love
> to ditch svn,  but just haven't had  the time to think  about a proper
> migration.  Should we just admit  defeat and do on-demand conversions,
> preserving history if possible but not worrying about it too much?

I haven't  had time to  work on  the migration. On-demand  conversion is
fine with me. Is there any problem in preserving history while migrating
From SVN to Git?

Cheers,
-- 
Arnaud Fontaine

[toc] | [prev] | [next] | [standalone]

#8976

Le 11/02/16 à 23:51, Barry Warsaw a écrit :
> On Nov 02, 2016, at 10:46 AM, Arnaud Fontaine wrote:
>> This has  already been discussed  but all  the packages in  pkg-zope SVN
>> repository will have to be  moved to python-{modules, apps} repositories
>> (because there  is almost no activity  on pkg-zope and most  modules are
>> used independently  of Zope anyway)  and we should use  debian-python ML
>> for the  same reason,  so yes,  please use  debian-python ML  and commit
>> everything to python-{modules, apps} repositories.
> 
> +1.  I do still touch some of the ztk packages and would dearly love to ditch
> svn, but just haven't had the time to think about a proper migration.  Should
> we just admit defeat and do on-demand conversions, preserving history if
> possible but not worrying about it too much?
> 
> And then what about just using gbp and ignoring git-dpm?  The latter still
> kind of works but we know it's a dead-end.  Anybody looked at dgit?  Is that
> a useful option?

I'm used to gbp. I don't know git-dpm (or I forgot after seeing I would not like?)

Not sure if all python-modules repositories are like persistent, but for me, mixing Debian work with imported tarballs in the same branch is terrible. When possible, I prefer to fork the upstream repository, otherwise no upstream source at all.

Actually, I already use git-svn+gbp locally for zodb (I have a debian symlink between the upstream and debian repos; that even works with patches), so it's not urgent to convert for me.

Julien

[toc] | [prev] | [next] | [standalone]

#8979

On Nov 03, 2016, at 08:36 PM, Julien Muchembled wrote:

>I'm used to gbp. I don't know git-dpm (or I forgot after seeing I would not
>like?)

git-dpm is usually pretty easy, but it's really only used in a few cases, such
as importing a new upstream, managing the patch stack, and tagging.  We
document the team's use cases pretty well so you don't even really have to
remember much:

https://wiki.debian.org/Python/GitPackaging#New_upstream_release

For a lot of other package management tasks (e.g. building source packages,
cloning, pulling, etc.), gbp works just fine.

That said, we know git-dpm has not been developed in a very long time, and is
for all intents and purposes, abandonware.  It works, so I don't think there's
a huge urgency to get rid of it (obviously, since we haven't ;) but it should
be in our long-term team goals to move away from it.

>Not sure if all python-modules repositories are like persistent, but for me,
>mixing Debian work with imported tarballs in the same branch is
>terrible. When possible, I prefer to fork the upstream repository, otherwise
>no upstream source at all.

You're not alone, but I think that's still a minority opinion in the team.
Our packages are so tightly integrated with PyPI, and source tarballs are such
an ingrained aspect of that service, that a pristine-tar based approach for
team packages still makes sense, IMHO.

Cheers,
-Barry

[toc] | [prev] | [next] | [standalone]

#8980

On Friday, November 04, 2016 10:47:32 AM Barry Warsaw wrote:
> On Nov 03, 2016, at 08:36 PM, Julien Muchembled wrote:
> >I'm used to gbp. I don't know git-dpm (or I forgot after seeing I would not
> >like?)
> 
> git-dpm is usually pretty easy, but it's really only used in a few cases,
> such as importing a new upstream, managing the patch stack, and tagging. 
> We document the team's use cases pretty well so you don't even really have
> to remember much:
> 
> https://wiki.debian.org/Python/GitPackaging#New_upstream_release
> 
> For a lot of other package management tasks (e.g. building source packages,
> cloning, pulling, etc.), gbp works just fine.
> 
> That said, we know git-dpm has not been developed in a very long time, and
> is for all intents and purposes, abandonware.  It works, so I don't think
> there's a huge urgency to get rid of it (obviously, since we haven't ;) but
> it should be in our long-term team goals to move away from it.
> 
> >Not sure if all python-modules repositories are like persistent, but for
> >me, mixing Debian work with imported tarballs in the same branch is
> >terrible. When possible, I prefer to fork the upstream repository,
> >otherwise no upstream source at all.
> 
> You're not alone, but I think that's still a minority opinion in the team.
> Our packages are so tightly integrated with PyPI, and source tarballs are
> such an ingrained aspect of that service, that a pristine-tar based
> approach for team packages still makes sense, IMHO.

You can integrate a new upstream directly from an upstream git repository with 
git-dpm if that's what makes sense for a particular upstream.

Scott K

[toc] | [prev] | [next] | [standalone]

#9069

Le 11/04/16 à 16:15, Scott Kitterman a écrit :
> On Friday, November 04, 2016 10:47:32 AM Barry Warsaw wrote:
>> On Nov 03, 2016, at 08:36 PM, Julien Muchembled wrote:
>>> Not sure if all python-modules repositories are like persistent, but for
>>> me, mixing Debian work with imported tarballs in the same branch is
>>> terrible. When possible, I prefer to fork the upstream repository,
>>> otherwise no upstream source at all.
>>
>> You're not alone, but I think that's still a minority opinion in the team.
>> Our packages are so tightly integrated with PyPI, and source tarballs are
>> such an ingrained aspect of that service, that a pristine-tar based
>> approach for team packages still makes sense, IMHO.
> 
> You can integrate a new upstream directly from an upstream git repository with 
> git-dpm if that's what makes sense for a particular upstream.

That's tempting and I was about to ask how, but I doubt the team would accept. The Zope foundation already releases everything on PyPI and the policy says: « Complete upstream git history should be avoided in the upstream branch. [...] When you must (not want to) deviate, »

So I plan to create only 3 new packages in DPMT:
- python-btrees
- python-zc.customdoctests
- zodbpickle


Packages maintained by the Debian/Ubuntu Zope Team, like zodb, will stay where there are (since I prefer git-svn+source-less), and I'll also create 'zeo' there (because it depends on zodb).


https://wiki.debian.org/Python/GitPackaging:
> Q: Source-full or source-less branches? A: Source-full branches please! There are lots of good reasons for this, including the ability to easily diff between upstream versions, [...]

Such source-full tree is unusable for me. I need git-blame too and auto-generated files are annoying.


Julien

[toc] | [prev] | [next] | [standalone]

#9070 — gitignore and import of upstream tarballs (was Re: Packaging new version of ZODB (Zope Object Database))

[Multipart message — attachments visible in raw view] — view raw

Le 12/05/16 à 20:21, Julien Muchembled a écrit :
> So I plan to create only 3 new packages in DPMT:
> - python-btrees
> - python-zc.customdoctests
> - zodbpickle

I am also preparing an update for python-persistent.

But for all that, I followed https://wiki.debian.org/Python/GitPackaging and there's no indication about gitignore'd files and the *.egg-info directory.

1. Both upstream repositories have a .gitignore file at the root (ignoring *.egg-info among other files) but on PyPI:
- BTrees-4.3.1.tar.gz contains it
- persistent-4.2.2.tar.gz does not contain it

I also have a ~/.gitignore file that ignores *.egg-info/ anyway.

2. The last release of zc.customdoctests is quite old and the tarball contain a egg-info folder that differs slightly to what the new setuptools generates, which is the source of bugs like #825921

Having generated files in Git is not great.


1 thing at a time so just simple questions:
- Should the 'Creating a new package' paragraph on the wiki says 'git add -f .' instead of 'git add .' ?
- It's quite obvious that ~/.gitignore should be ignored, but is it also the case for the one that may be in the tarball ?
- Do we always want *.egg-info directories in Git ?

Meanwhile, the wiki should also recommend to add the following line to debian/source/options:
  extend-diff-ignore="\.egg-info/"

Julien

[toc] | [prev] | [next] | [standalone]

#9052 — DPMT Membership request​ (was Packaging new version of ZODB (Zope Object Database))

Hello,

Le 11/01/16 à 20:10, Julien Muchembled a écrit :
> I am co-maintaining the 'zodb' package and I am also an upstream developer.
>   http://www.zodb.org/ - a native object database for Python
> 
> Debian currently packages branch 3.x of ZODB, which is getting quite old, with only support for Python 2. It's been a while that I am working locally in packaging a newer version, and I have almost finished.
> 
> With branch 4.x, ZODB has been splitted into 4 packages (-> means Depends):
>  ZEO -> ZODB -> (Btrees, persistent)
> The split of persistent was already backported.
> 
> And in addition to that:
> - new dependencies that are not packaged yet
> - some existing dependencies needs to be updated as well
> 
> I write to debian-python, because some of the involved packages are not specific to Zope. Actually, I even think that ZODB itself is not specific to Zope, but well, changing section of existing packages can be another topic. I've just requested to join the Python Modules Packaging Team.

I see that many people request membership on the ML. I did it on the Alioth interface and it looks like my request was lost.

There's already this whole thread explaining what I'm doing.

My Alioth login is jmuchemb-guest.
I have read https://python-modules.alioth.debian.org/policy.html and accept it.

Greetings,
Julien

[toc] | [prev] | [next] | [standalone]

#9053 — Re: DPMT Membership request​ (was Packaging new version of ZODB (Zope Object Database))

[Julien Muchembled, 2016-11-30]
> I see that many people request membership on the ML. I did it on the
> Alioth interface and it looks like my request was lost.

yeah, that's the process described in our policy. I don't know how to
disable Alioth/FusionForge join button (I even tried to set project's
visibility to false for all "Any user logged in" roles... it didn't help)

> There's already this whole thread explaining what I'm doing.
> 
> My Alioth login is jmuchemb-guest.
> I have read https://python-modules.alioth.debian.org/policy.html and accept it.

I added you to the team, welcome :)
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#9093

Updated status of ZODB packaging.

I didn't write it in my previous mail because there was still a lot time before deadlines but now, it's becoming short. I'd like to see ZODB updated for Stretch, at least that's what I had in mind when I started months ago. Having a very old version of ZODB until 2019, without even support for python 3, would be frustrating.

I am actually looking for sponsors who could upload them in time. They all should be in good state. Maybe Arnaud will be available again for the last uploads.

Here is the dependency graph of packages that need(ed) to be updated (I forgot to mention a few of them in my previous package):

zdaemon -> python-zc.customdoctests, zconfig
zodb -> python-persistent, transaction, zodbpickle, python-btrees, zconfig, zc.lockfile (bin-new)
zeo -> zodb, zdaemon
(+ zodb recommending zeo)

Packages would be uploaded in 3 rounds:
(1) python-zc.customdoctests (new), zconfig (bin-new)
(2) python-persistent, transaction, zodbpickle (new), python-btrees (new), zc.lockfile, zdaemon (bin-new)
(3) zodb (bin-new), zeo (new)

(1) is done already.

(2) 3 are on mentors.debian.net
  https://mentors.debian.net/package/python-btrees (RFS #847609)
  https://mentors.debian.net/package/python-persistent
  https://mentors.debian.net/package/zodbpickle (RFS #847608)
  
transaction, zc.lockfile and zdaemon are also finished on my side, but I guess the priority is the new packages.

They all should be in good state.

(3) So that zodb/zeo are uploaded and accepted before the 26th


About VCS, it's too late to think about it now. I'm interested in this topic so I'll be happy to work on this once ZODB is updated. For the moment:
- I followed blindly the policy for the new python-zc.customdoctests/zodbpickle/python-btrees packages
- I stay on svn://anonscm.debian.org/pkg-zope/ for the packages of the Zope team, and for the new 'zeo'.


Regards,
Julien

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web