Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]


Groups > linux.debian.maint.python > #8446 > unrolled thread

Test suite in github but missing from pypi tarballs

Started byEdward Betts <edward@4angle.com>
First post2016-04-21 16:50 +0200
Last post2016-04-23 12:40 +0200
Articles 18 — 11 participants

Back to article view | Back to linux.debian.maint.python


Contents

  Test suite in github but missing from pypi tarballs Edward Betts <edward@4angle.com> - 2016-04-21 16:50 +0200
    Re: Test suite in github but missing from pypi tarballs Thomas Goirand <zigo@debian.org> - 2016-04-21 17:00 +0200
      Re: Test suite in github but missing from pypi tarballs Matthias Klose <doko@debian.org> - 2016-04-21 17:30 +0200
        Re: Test suite in github but missing from pypi tarballs Thomas Goirand <zigo@debian.org> - 2016-04-21 19:10 +0200
      Re: Test suite in github but missing from pypi tarballs Barry Warsaw <barry@debian.org> - 2016-04-21 19:40 +0200
    Re: Test suite in github but missing from pypi tarballs Tristan Seligmann <mithrandi@mithrandi.net> - 2016-04-21 17:20 +0200
      Re: Test suite in github but missing from pypi tarballs Fred Drake <fred@fdrake.net> - 2016-04-21 17:30 +0200
        Re: Test suite in github but missing from pypi tarballs Ian Cordasco <graffatcolmingov@gmail.com> - 2016-04-21 18:10 +0200
          Re: Test suite in github but missing from pypi tarballs Fred Drake <fred@fdrake.net> - 2016-04-21 18:20 +0200
            Source distribution is for building from source (was: Test suite in github but missing from pypi tarballs) Ben Finney <ben+debian@benfinney.id.au> - 2016-04-22 01:10 +0200
        Re: Test suite in github but missing from pypi tarballs Jeremy Stanley <fungi@yuggoth.org> - 2016-04-21 19:20 +0200
          Re: Test suite in github but missing from pypi tarballs Fred Drake <fdrake@gmail.com> - 2016-04-21 20:10 +0200
      Re: Test suite in github but missing from pypi tarballs Barry Warsaw <barry@debian.org> - 2016-04-21 19:40 +0200
        Re: Test suite in github but missing from pypi tarballs Tristan Seligmann <mithrandi@mithrandi.net> - 2016-04-22 07:20 +0200
      Re: Test suite in github but missing from pypi tarballs Brian May <bam@debian.org> - 2016-04-22 12:10 +0200
      Re: Test suite in github but missing from pypi tarballs Brian May <bam@debian.org> - 2016-04-22 12:10 +0200
    Re: Test suite in github but missing from pypi tarballs Edward Betts <edward@4angle.com> - 2016-04-22 12:50 +0200
      Re: Test suite in github but missing from pypi tarballs Edward Betts <edward@4angle.com> - 2016-04-23 12:40 +0200

#8446 — Test suite in github but missing from pypi tarballs

FromEdward Betts <edward@4angle.com>
Date2016-04-21 16:50 +0200
SubjectTest suite in github but missing from pypi tarballs
Message-ID<rqo5A-2dR-33@gated-at.bofh.it>
Recently I've come across some Python libraries that have a test suite in
their github repo but don't include it in the tarball they upload to pypi.

Debian binary packages don't normally include the test suite. Some Python
library developers are treating the pypi releases in a similar way, as if
they're just for deployment. They think anybody who needs the test suite is
doing development and will clone from the github repo.

It is very useful to run the test suite when building a Debian package. It
would be better if upstream would include the test suite in their release
tarballs.

It is often just a one line change to MANIFEST.in to have the test suite
included. If the tests are in a directory called 'tests', then this is the
line that needs to added:

  recursive-include tests *.py

I've been sending github pull requests to encourage library authors to make
this change.

  https://github.com/RPi-Distro/python-gpiozero/pull/284
  https://github.com/john-kurkowski/tldextract/pull/97

You might want to take a few minutes to check any Python packages you've
worked on, see if there is a test suite in the github repo that hasn't made it
to the release tarball and send a pull request to fix it.

It is also good thing to check when making a new Debian package.
-- 
Edward.

[toc] | [next] | [standalone]


#8447

FromThomas Goirand <zigo@debian.org>
Date2016-04-21 17:00 +0200
Message-ID<rqofh-2i1-37@gated-at.bofh.it>
In reply to#8446
On 04/21/2016 04:10 PM, Edward Betts wrote:
> Recently I've come across some Python libraries that have a test suite in
> their github repo but don't include it in the tarball they upload to pypi.
> 
> Debian binary packages don't normally include the test suite.

Why? It's my view that it's a good idea to include it, if it is located
within the lib itself.

> Some Python
> library developers are treating the pypi releases in a similar way, as if
> they're just for deployment. They think anybody who needs the test suite is
> doing development and will clone from the github repo.
> 
> It is very useful to run the test suite when building a Debian package. It
> would be better if upstream would include the test suite in their release
> tarballs.
> 
> It is often just a one line change to MANIFEST.in to have the test suite
> included. If the tests are in a directory called 'tests', then this is the
> line that needs to added:
> 
>   recursive-include tests *.py

It's best that the test suite goes within the project. So if project is
called foo, then best is to get the test folder in foo/tests. This way,
you don't even need to fix the MANIFEST.in. Then you can even use it as
an autopkgtest thing.

> I've been sending github pull requests to encourage library authors to make
> this change.
> 
>   https://github.com/RPi-Distro/python-gpiozero/pull/284
>   https://github.com/john-kurkowski/tldextract/pull/97
> 
> You might want to take a few minutes to check any Python packages you've
> worked on, see if there is a test suite in the github repo that hasn't made it
> to the release tarball and send a pull request to fix it.

IMO, don't bother, and just re-generate a tarball from git, using "git
archive", which is my preferred way to use upstream sources.

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]


#8450

FromMatthias Klose <doko@debian.org>
Date2016-04-21 17:30 +0200
Message-ID<rqoIi-2PN-7@gated-at.bofh.it>
In reply to#8447
On 21.04.2016 16:52, Thomas Goirand wrote:
> On 04/21/2016 04:10 PM, Edward Betts wrote:
>> Recently I've come across some Python libraries that have a test suite in
>> their github repo but don't include it in the tarball they upload to pypi.
>>
>> Debian binary packages don't normally include the test suite.
>
> Why? It's my view that it's a good idea to include it, if it is located
> within the lib itself.

there is no benefit to include the testsuite if it cannot be run in the install 
location.  So why include that for these cases?  Write an autopkg test and see 
if it runs *and* succeeds in the installed location.  Usually testsuites tend to 
need a lot of patches for doing that.

Also for large testsuites a separate binary package would be preferred, see e.g. 
the size of python-cryptography and python-cryptography-vectors.

Matthias

[toc] | [prev] | [next] | [standalone]


#8453

FromThomas Goirand <zigo@debian.org>
Date2016-04-21 19:10 +0200
Message-ID<rqqh4-46m-5@gated-at.bofh.it>
In reply to#8450
On 04/21/2016 05:20 PM, Matthias Klose wrote:
> On 21.04.2016 16:52, Thomas Goirand wrote:
>> On 04/21/2016 04:10 PM, Edward Betts wrote:
>>> Recently I've come across some Python libraries that have a test
>>> suite in
>>> their github repo but don't include it in the tarball they upload to
>>> pypi.
>>>
>>> Debian binary packages don't normally include the test suite.
>>
>> Why? It's my view that it's a good idea to include it, if it is located
>> within the lib itself.
> 
> there is no benefit to include the testsuite if it cannot be run in the
> install location.

Probably I'm too much influenced by what I do in OpenStack, where there
is not only unit tests, but also functional tests. And it makes a lot of
sense to include functional tests, so that they can be launched once the
package is installed (and connected to other services).

> Also for large testsuites a separate binary package would be preferred,

Good point! I probably should start doing that in my packages, it would
make a lot of sense to do so.

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]


#8455

FromBarry Warsaw <barry@debian.org>
Date2016-04-21 19:40 +0200
Message-ID<rqqK7-4mT-27@gated-at.bofh.it>
In reply to#8447
On Apr 21, 2016, at 04:52 PM, Thomas Goirand wrote:

>It's best that the test suite goes within the project. So if project is
>called foo, then best is to get the test folder in foo/tests. This way,
>you don't even need to fix the MANIFEST.in.

+1 - as an upstream I always do this because I like having the tests as close
to the code its testing as possible.  It generally does no harm to include
them in the binary package so why bother, but yes in some cases you might want
to split them out.

Cheers,
-Barry

[toc] | [prev] | [next] | [standalone]


#8448

FromTristan Seligmann <mithrandi@mithrandi.net>
Date2016-04-21 17:20 +0200
Message-ID<rqoyC-2LH-11@gated-at.bofh.it>
In reply to#8446

[Multipart message — attachments visible in raw view] — view raw

On Thu, 21 Apr 2016 at 16:47 Edward Betts <edward@4angle.com> wrote:

> Debian binary packages don't normally include the test suite. Some Python
>
library developers are treating the pypi releases in a similar way, as if
> they're just for deployment. They think anybody who needs the test suite is
> doing development and will clone from the github repo.
>

With my upstream developer hat on: source packages on PyPI are meant for
end users to install via pip. They often include generated artifacts, and
don't include things that aren't intended for installation via pip (tests
being just one of these).

For distribution packaging purposes, the GitHub tags are generally
preferrable. GitHub makes archives of tagged releases available as
tarballs, so this is generally a simple tweak to debian/watch.

[toc] | [prev] | [next] | [standalone]


#8449

FromFred Drake <fred@fdrake.net>
Date2016-04-21 17:30 +0200
Message-ID<rqoIi-2PN-5@gated-at.bofh.it>
In reply to#8448
On Thu, Apr 21, 2016 at 10:54 AM, Tristan Seligmann
<mithrandi@mithrandi.net> wrote:
> With my upstream developer hat on: source packages on PyPI are meant for end
> users to install via pip. They often include generated artifacts, and don't
> include things that aren't intended for installation via pip (tests being
> just one of these).

Interesting.  In many years of working with Python (since before
either PyPI or pip), I have a slightly different perspective.

Source packages (whether on PyPI or not) with a setup.py (or whatever
comes next) are intended for both automated or developer consumption.
pip is just one possible tool; buildout is another, and others as
well.

> For distribution packaging purposes, the GitHub tags are generally
> preferrable. GitHub makes archives of tagged releases available as tarballs,
> so this is generally a simple tweak to debian/watch.

I'd generally be worried if the source package doesn't closely match a
tag in whatever VCS a project is using, but I don't think that's
essential, release processes being what they are.

This is really independent from the debate as to whether a test suite
should be included with the package, though.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

[toc] | [prev] | [next] | [standalone]


#8451

FromIan Cordasco <graffatcolmingov@gmail.com>
Date2016-04-21 18:10 +0200
Message-ID<rqpl0-3pZ-5@gated-at.bofh.it>
In reply to#8449
On Thu, Apr 21, 2016 at 10:23 AM, Fred Drake <fred@fdrake.net> wrote:
> On Thu, Apr 21, 2016 at 10:54 AM, Tristan Seligmann
> <mithrandi@mithrandi.net> wrote:
>> With my upstream developer hat on: source packages on PyPI are meant for end
>> users to install via pip. They often include generated artifacts, and don't
>> include things that aren't intended for installation via pip (tests being
>> just one of these).
>
> Interesting.  In many years of working with Python (since before
> either PyPI or pip), I have a slightly different perspective.
>
> Source packages (whether on PyPI or not) with a setup.py (or whatever
> comes next) are intended for both automated or developer consumption.
> pip is just one possible tool; buildout is another, and others as
> well.

In the long history of both Python and its packaging this is
absolutely true (all you need is an archive and a setup.py) but
Python's packaging has evolved and improved for its users through
setuptools, pip, and wheel (even though I'm sure there are many on
this list who vehemently disagree). So tar files, for the *majority*
of PyPI are there to help users with ancient, barely working, versions
of pip, buildout, etc.

>> For distribution packaging purposes, the GitHub tags are generally
>> preferrable. GitHub makes archives of tagged releases available as tarballs,
>> so this is generally a simple tweak to debian/watch.
>
> I'd generally be worried if the source package doesn't closely match a
> tag in whatever VCS a project is using, but I don't think that's
> essential, release processes being what they are.

There's also a lot of data that could be in a VCS tag that is
orthogonal to what needs to be in a package for a user or
redistributor. For example, few if any people are going to want a file
that configures the (potentially closed-source) continuous integration
service(s) that the project uses and it's inconsequential to a
redistributor if a .gitignore is in the archive too. Some of this will
just needlessly inflate the package size.

> This is really independent from the debate as to whether a test suite
> should be included with the package, though.

Agreed. It's just interesting.

[toc] | [prev] | [next] | [standalone]


#8452

FromFred Drake <fred@fdrake.net>
Date2016-04-21 18:20 +0200
Message-ID<rqpuH-3tV-33@gated-at.bofh.it>
In reply to#8451
On Thu, Apr 21, 2016 at 11:48 AM, Ian Cordasco
<graffatcolmingov@gmail.com> wrote:
> In the long history of both Python and its packaging this is
> absolutely true (all you need is an archive and a setup.py) but
> Python's packaging has evolved and improved for its users through
> setuptools, pip, and wheel (even though I'm sure there are many on
> this list who vehemently disagree). So tar files, for the *majority*
> of PyPI are there to help users with ancient, barely working, versions
> of pip, buildout, etc.

My reaction is really about the comment that source packages are in
PyPI "for pip". You definitely understand that it's broader than that,
but careful language is important to avoid confusing someone new,
leading to poor understanding across the community.

Some of us still see pip as a youngling upstart.  :-)

> There's also a lot of data that could be in a VCS tag that is
> orthogonal to what needs to be in a package for a user or
> redistributor. For example, few if any people are going to want a file
> that configures the (potentially closed-source) continuous integration
> service(s) that the project uses and it's inconsequential to a
> redistributor if a .gitignore is in the archive too.

Which is why I don't consider slavish use of "git archive" (or similar
from some other VCS) to be valuable.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

[toc] | [prev] | [next] | [standalone]


#8458 — Source distribution is for building from source (was: Test suite in github but missing from pypi tarballs)

FromBen Finney <ben+debian@benfinney.id.au>
Date2016-04-22 01:10 +0200
SubjectSource distribution is for building from source (was: Test suite in github but missing from pypi tarballs)
Message-ID<rqvTr-qs-1@gated-at.bofh.it>
In reply to#8452
Fred Drake <fred@fdrake.net> writes:

> My reaction is really about the comment that source packages are in
> PyPI "for pip". You definitely understand that it's broader than that,
> but careful language is important to avoid confusing someone new,
> leading to poor understanding across the community.
>
> Some of us still see pip as a youngling upstart.  :-)

And regardless of Pip's age, source distributions are definitely of
interest beyond that narrow use case.

Not everything in the VCS repository belongs in the Python source
distribution, and publishing the source distribution at PyPI signals
“this is the source for the purposes of building the distribution”.

The larger point is: Getting the source, building from source, and
installing the result — are all distinct steps that each need to be
considered.

The source distribution may also be a viable input for Pip, but that is
not its only widespread use. It remains the correct input for “build
from source”.

> Which is why I don't consider slavish use of "git archive" (or similar
> from some other VCS) to be valuable.

+1. The VCS is for potential developers of the code bse. The source
distribution is for people who want to build from source. Those are
always distinct use cases.

-- 
 \       “Come on Milhouse, there’s no such thing as a soul! It’s just |
  `\      something they made up to scare kids, like the Boogie Man or |
_o__)                          Michael Jackson.” —Bart, _The Simpsons_ |
Ben Finney

[toc] | [prev] | [next] | [standalone]


#8454

FromJeremy Stanley <fungi@yuggoth.org>
Date2016-04-21 19:20 +0200
Message-ID<rqqqK-4ao-3@gated-at.bofh.it>
In reply to#8449
On 2016-04-21 11:23:20 -0400 (-0400), Fred Drake wrote:
> On Thu, Apr 21, 2016 at 10:54 AM, Tristan Seligmann
[...]
> > For distribution packaging purposes, the GitHub tags are generally
> > preferrable. GitHub makes archives of tagged releases available as tarballs,
> > so this is generally a simple tweak to debian/watch.
> 
> I'd generally be worried if the source package doesn't closely match a
> tag in whatever VCS a project is using, but I don't think that's
> essential, release processes being what they are.
[...]

Agreed, as long as "closely" is interpreted in ways consistent with,
say, tarballs for C-based projects. Consider `setup.py sdist`
similar to `make dist` where the dist target of some projects may
still run additional commands to generate metadata or other files
not tracked in revision control prior to invoking tar/gzip.
-- 
Jeremy Stanley

[toc] | [prev] | [next] | [standalone]


#8457

FromFred Drake <fdrake@gmail.com>
Date2016-04-21 20:10 +0200
Message-ID<rqrd8-4YR-11@gated-at.bofh.it>
In reply to#8454

[Multipart message — attachments visible in raw view] — view raw

On Apr 21, 2016 13:16, "Jeremy Stanley" <fungi@yuggoth.org> wrote:
> Agreed, as long as "closely" is interpreted in ways consistent with,
> say, tarballs for C-based projects. Consider `setup.py sdist`
> similar to `make dist` where the dist target of some projects may
> still run additional commands to generate metadata or other files
> not tracked in revision control prior to invoking tar/gzip.

That's exactly what I mean.

   -Fred

[toc] | [prev] | [next] | [standalone]


#8456

FromBarry Warsaw <barry@debian.org>
Date2016-04-21 19:40 +0200
Message-ID<rqqK8-4mT-37@gated-at.bofh.it>
In reply to#8448
On Apr 21, 2016, at 02:54 PM, Tristan Seligmann wrote:

>With my upstream developer hat on: source packages on PyPI are meant for
>end users to install via pip. They often include generated artifacts, and
>don't include things that aren't intended for installation via pip (tests
>being just one of these).
>
>For distribution packaging purposes, the GitHub tags are generally
>preferrable. GitHub makes archives of tagged releases available as
>tarballs, so this is generally a simple tweak to debian/watch.

With my own upstream hat perched precariously on my head, I disagree.  I
generally prefer to include the test suite in the PyPI tarballs and I also
submit bug reports to upstream that don't include them.  Most have been very
accommodating, especially when you can provide a simple patch to their
MANIFEST.in.

I agree with Fred that PyPI tarballs have multiple intended consumers.

I also don't particularly like relying on GitHub generated tarballs.  Despite
popular believe, not every upstream uses GH or even git, signs their tags or
even tags their releases.  But almost *every* Python package releases tarballs
to PyPI.

There are a few very limited cases where upstreams have balked at supplying
their test suites in their PyPI tarballs.  One that I agree with is pip[*]
because it's downloaded like a jillion times a day so in that case, have a
more svelte tarball makes sense.  I think Donald and I talked about them
spinning a pip-dev tarball for distro consumers, but for now I just override
dh_auto_test in d/rules.

Cheers,
-Barry

[*] https://github.com/pypa/pip/issues/3370

[toc] | [prev] | [next] | [standalone]


#8460

FromTristan Seligmann <mithrandi@mithrandi.net>
Date2016-04-22 07:20 +0200
Message-ID<rqBFx-4Zg-7@gated-at.bofh.it>
In reply to#8456

[Multipart message — attachments visible in raw view] — view raw

Resending to the list since I accidentally replied off-list!

On Thu, 21 Apr 2016, 20:14 Tristan Seligmann, <mithrandi@mithrandi.net>
wrote:

> On Thu, 21 Apr 2016 at 19:34 Barry Warsaw <barry@debian.org> wrote:
>
>> I also don't particularly like relying on GitHub generated tarballs.
>> Despite
>> popular believe, not every upstream uses GH or even git, signs their tags
>> or
>> even tags their releases.  But almost *every* Python package releases
>> tarballs
>> to PyPI.
>
>
> There's no particular reason why every Python package has to distribute
> the Debian orig tarball via PyPI, though. *My* projects are almost all on
> GitHub these days, so I offer the git tags and github archives as an option
> to get the "full source"; in the past, I have hosted projects elsewhere,
> and released tarballs elsewhere, and there is of course nothing wrong with
> this.
>
> If you want to make your PyPI sdists do double-duty, there's usually
> nothing terribly wrong with this, but given that they must include
> generated artifacts (like .c source generated by Cython) in order to work
> well for pip, and pip is typically how people consume PyPI, I don't view
> these as the ideal "upstream source" for Debian.
>
> And of course, there are extreme examples like *cryptography*: The
> python-cryptography source package is 384K, the python-cryptography-vectors
> source package (necessary for running the cryptography tests) is 26MB!
>
> FWIW: I should note that I'm a fan of having my tests as a subpackage of
> the main package (ie. mything.tests) so they can be run by an end-user to
> verify that an installed copy of the package is actually working. I do this
> in most of the projects I'm upstream for; so the tests *are* included in
> my sdist, but other files are not (like documentation).
>

[toc] | [prev] | [next] | [standalone]


#8461

FromBrian May <bam@debian.org>
Date2016-04-22 12:10 +0200
Message-ID<rqGca-hy-11@gated-at.bofh.it>
In reply to#8448
Tristan Seligmann <mithrandi@mithrandi.net> writes:

> With my upstream developer hat on: source packages on PyPI are meant for
> end users to install via pip. They often include generated artifacts, and
> don't include things that aren't intended for installation via pip (tests
> being just one of these).
>
> For distribution packaging purposes, the GitHub tags are generally
> preferrable. GitHub makes archives of tagged releases available as
> tarballs, so this is generally a simple tweak to debian/watch.

I use to have this attitude too. However now I have reconsidered.

Far better to to retrieve source packages from PyPI.

For example, it is possible to have signed uploads to PyPI and have
uuscan automatically check the signature. This is not the case for
github.

packages that use setuptools-scm is another example - as a (over?)
simplification this requires files in the PyPI distribution, although
perhaps that is a topic best left for another discussion.
-- 
Brian May <bam@debian.org>

[toc] | [prev] | [next] | [standalone]


#8462

FromBrian May <bam@debian.org>
Date2016-04-22 12:10 +0200
Message-ID<rqGcb-hy-35@gated-at.bofh.it>
In reply to#8448
Tristan Seligmann <mithrandi@mithrandi.net> writes:

> With my upstream developer hat on: source packages on PyPI are meant for
> end users to install via pip. They often include generated artifacts, and
> don't include things that aren't intended for installation via pip (tests
> being just one of these).
>
> For distribution packaging purposes, the GitHub tags are generally
> preferrable. GitHub makes archives of tagged releases available as
> tarballs, so this is generally a simple tweak to debian/watch.

Forgot to mention: In PyPI there is an application called
"check-manifest" that will check to ensure all files in git will be
included in the PyPI release, and suggest changes to MANIFEST.in.
-- 
Brian May <bam@debian.org>

[toc] | [prev] | [next] | [standalone]


#8463

FromEdward Betts <edward@4angle.com>
Date2016-04-22 12:50 +0200
Message-ID<rqGOS-A5-23@gated-at.bofh.it>
In reply to#8446
Here is a list of packages that include a test suite on github but not in
the DPMT git.

csvkit                 https://github.com/onyxfish/csvkit
django-haystack        https://github.com/toastdriven/django-haystack
django-jinja           https://github.com/niwibe/django-jinja
django-recurrence      https://github.com/django-recurrence/django-recurrence
django-webpack-loader  https://github.com/owais/django-webpack-loader
djoser                 https://github.com/gizmag/django-fsm-log
dockerpty              https://github.com/d11wtq/dockerpty
drf-generators         https://github.com/Brobin/drf-generators
mod-wsgi               https://github.com/GrahamDumpleton/mod_wsgi
pdfrw                  https://github.com/pmaupin/pdfrw
pep8-naming            https://github.com/flintwork/pep8-naming
pkgconfig              https://github.com/matze/pkgconfig
pydot                  https://github.com/erocarrera/pydot
pylast                 http://github.com/pylast/pylast
pysrt                  https://github.com/byroot/pysrt
python-args            https://github.com/kennethreitz/args
python-astor           https://github.com/berkerpeksag/astor
python-cachecontrol    https://github.com/ionrock/cachecontrol
python-easywebdav      https://github.com/amnong/easywebdav
python-exif            https://github.com/ianare/exif-py
python-gnutls          https://github.com/AGProjects/python-gnutls
python-hpilo           https://github.com/seveas/python-hpilo
python-humanize        https://github.com/jmoiron/humanize
python-jsonpify        https://github.com/wcdolphin/flask-jsonpify
python-model-mommy     http://github.com/vandersonmota/model_mommy
python-mrjob           http://github.com/Yelp/mrjob
python-pgspecial       https://github.com/dbcli/pgspecial
python-pip             https://github.com/pypa/pip
python-pretend         https://github.com/alex/pretend
python-rply            https://github.com/alex/rply
python-scp             https://github.com/jbardin/scp.py
python-sentinels       https://github.com/vmalloc/sentinels
python-sunlight        https://github.com/sunlightlabs/python-sunlight
python-trezor          https://github.com/trezor/python-trezor
python-zxcvbn          https://github.com/dropbox/python-zxcvbn
responses              https://github.com/getsentry/responses
subliminal             https://github.com/Diaoul/subliminal
txws                   https://github.com/MostAwesomeDude/txWS
vcversioner            https://github.com/habnabit/vcversioner
whichcraft             https://github.com/pydanny/whichcraft

This is the code I ran on git.debian.org to generate the list.

#!/usr/bin/python3

from subprocess import Popen, PIPE
from urllib.request import urlopen
from urllib.error import HTTPError
import os
import re
    
re_github_url = re.compile('(https?://[^/ ]*github.com/[^/]+/[^/ ]+)')

location = '/git/python-modules/packages/'

def repo_includes_tests(repo):
    cmd = ['git', '--git-dir=' + location + repo, 'ls-tree', '-r', 'HEAD', '--name-only']
    with Popen(cmd, stdout=PIPE, stderr=PIPE) as proc:
        for line in proc.stdout:
            filename = line.decode('utf-8')[:-1]
            if not filename.endswith('.py'):
                continue
            if filename.startswith('test') or '/test' in filename:
                return True
    return False

def show_debian_file(repo, filename):
    cmd = ['git', '--git-dir=' + location + repo, 'show', 'HEAD:debian/' + filename]
    return Popen(cmd, stdout=PIPE, stderr=PIPE).stdout.read().decode('utf-8')

def github_urls(content):
    github_urls = set()
    for line in content.splitlines():
        if 'github.com' in line:
            m = re_github_url.search(line)
            if not m:
                print(line)
            github_urls.add(m.group(1))
    return github_urls

for repo in sorted(os.listdir(location)):
    if not repo.endswith('.git') or repo_includes_tests(repo):
        continue

    urls = (github_urls(show_debian_file(repo, 'copyright'))
            or github_urls(show_debian_file(repo, 'watch')))

    if len(urls) != 1:
        continue
    (url,) = urls
    try:
        page = urlopen(url).read()
    except HTTPError:
        continue
    if ' title="test' not in page.decode('utf-8'):
        continue  # no tests in github
    print('{:23s} {}'.format(repo[:-4], url))

[toc] | [prev] | [next] | [standalone]


#8467

FromEdward Betts <edward@4angle.com>
Date2016-04-23 12:40 +0200
Message-ID<rr38K-1zA-5@gated-at.bofh.it>
In reply to#8463
Here is the progress for each package.

> csvkit                 https://github.com/onyxfish/csvkit

Done, just waiting for a new pypi release. See
https://github.com/wireservice/csvkit/issues/20

> django-jinja           https://github.com/niwibe/django-jinja

Upstream not interested, claims the tests are hard to run.
https://github.com/niwinz/django-jinja/issues/135

> dockerpty              https://github.com/d11wtq/dockerpty

https://github.com/d11wtq/dockerpty/pull/21

> pdfrw                  https://github.com/pmaupin/pdfrw

https://github.com/pmaupin/pdfrw/pull/53

> pep8-naming            https://github.com/flintwork/pep8-naming

https://github.com/PyCQA/pep8-naming/pull/24

> pkgconfig              https://github.com/matze/pkgconfig

https://github.com/PyCQA/pep8-naming/pull/24

> pydot                  https://github.com/erocarrera/pydot

https://github.com/erocarrera/pydot/pull/120

> pylast                 http://github.com/pylast/pylast

https://github.com/pylast/pylast/pull/169

> pysrt                  https://github.com/byroot/pysrt

https://github.com/byroot/pysrt/issues/42
https://github.com/byroot/pysrt/pull/59

> python-args            https://github.com/kennethreitz/args

https://github.com/kennethreitz/args/pull/19

> python-astor           https://github.com/berkerpeksag/astor

https://github.com/berkerpeksag/astor/pull/52

> python-cachecontrol    https://github.com/ionrock/cachecontrol

https://github.com/ionrock/cachecontrol/pull/120

> python-easywebdav      https://github.com/amnong/easywebdav

https://github.com/amnong/easywebdav/pull/47

> python-gnutls          https://github.com/AGProjects/python-gnutls

https://github.com/AGProjects/python-gnutls/pull/1

> python-hpilo           https://github.com/seveas/python-hpilo

https://github.com/seveas/python-hpilo/pull/112

> python-humanize        https://github.com/jmoiron/humanize

https://github.com/jmoiron/humanize/issues/33
https://github.com/jmoiron/humanize/pull/34

> python-jsonpify        https://github.com/wcdolphin/flask-jsonpify

https://github.com/corydolphin/flask-jsonpify/pull/13

> txws                   https://github.com/MostAwesomeDude/txWS

https://github.com/MostAwesomeDude/txWS/pull/29

> vcversioner            https://github.com/habnabit/vcversioner

https://github.com/habnabit/vcversioner/pull/7

> whichcraft             https://github.com/pydanny/whichcraft

https://github.com/pydanny/whichcraft/pull/1

-- 
Edward.

[toc] | [prev] | [standalone]


Back to top | Article view | linux.debian.maint.python


csiph-web