Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #10280

Re: RFS: mwic 0.7.4-1

Show key headers only | View raw

On Tue, 20 Mar 2018 at 12:18:39 +0100, Gregor Riepl wrote:
> > In case I've misunderstood you, and you're referring to unit tests
> > shipped debian/tests/*, than yes, I agree. :)
> 
> As far as I understand, these tests are executed by the package builder after
> the upstream build script has finished. They're meant as a sort of integration
> test, i.e. "does this package run on Debian".
> 
> There's even a Lintian tag for them:
> https://lintian.debian.org/tags/testsuite-autopkgtest-missing.html (which, I
> think, is a bit overzealous)

I think you're slightly confusing build-time tests with as-installed
autopkgtests, but you have the right idea.

Build-time tests (dh_auto_test or similar) run the upstream "make check"
or equivalent, in the same environment as the build itself (typically an
autobuilder). For simple libraries, build-time tests are enough. However,
for more complex packages, the autobuilder environment is too artificial
or too restricted for build-time tests to give full coverage:

- can't run tests as root (system-level packages like dbus often need this
  for good coverage)
- can't contact the Internet (even if in practice you usually can, Policy
  says you must not, for good reasons)
- can't rely on a reasonable/realistic environment, like system services
  running, being in a systemd-logind login session on systems that have
  it, or even having a home directory
- can't rely on the packages being "properly installed" so that
  hard-coded paths can work, and have to rely on overrides like PYTHONPATH
  to make newly-built code visible

autopkgtest (debian/tests/) is a form of as-installed testing, which takes
the packages that were built, installs them in a relatively complete and
realistic environment (typically a lxc container or a qemu/kvm virtual
machine) and runs further tests there. Sometimes these tests just repeat
the build-time test coverage, but often they can make use of the ability
to do things that wouldn't work at build-time, like contacting Internet
services, running things as root, or relying on system services. This
often gives them better test coverage.
https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests is an example
of an upstream project that is doing similar things.

Because the autopkgtest container or VM is thrown away after running one
package's tests in it, the tests can do things that would be unacceptable
in an autobuilder environment, which again increases coverage.

In Debian, autopkgtests are run separately, by different infrastructure
(ci.debian.net), long after the package was built. The same package
will usually be tested multiple times against newer versions of its
dependencies, to look for regressions caused by a dependency change.

The normal package upload/autobuild workflow does not typically run
autopkgtests, although it could. Vectis, which is a personal package
building tool that I'm working on, builds the package in sbuild in
a virtual machine then immediately puts it through autopkgtests and
piuparts testing in separate VMs, so that test failures can be checked
(and either used as a reason not to upload, or ignored, based on the
maintainer's judgement about the severity of the failure and the urgency
of the upload - the same way the maintainer would triage Lintian errors).

    smcv

Back to linux.debian.maint.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

RFS: mwic 0.7.4-1 Georg Faerber <georg@riseup.net> - 2018-03-16 22:40 +0100
  Re: RFS: mwic 0.7.4-1 Paul Wise <pabs@debian.org> - 2018-03-17 04:30 +0100
    Re: RFS: mwic 0.7.4-1 Georg Faerber <georg@riseup.net> - 2018-03-17 11:00 +0100
      Re: RFS: mwic 0.7.4-1 Paul Wise <pabs@debian.org> - 2018-03-18 04:40 +0100
        Re: RFS: mwic 0.7.4-1 Georg Faerber <georg@riseup.net> - 2018-03-20 18:20 +0100
          Re: RFS: mwic 0.7.4-1 Georg Faerber <georg@riseup.net> - 2018-03-21 02:10 +0100
          Re: RFS: mwic 0.7.4-1 Paul Wise <pabs@debian.org> - 2018-03-21 04:40 +0100
            Re: RFS: mwic 0.7.4-1 Georg Faerber <georg@riseup.net> - 2018-03-21 11:10 +0100
              Re: RFS: mwic 0.7.4-1 Paul Wise <pabs@debian.org> - 2018-03-21 13:50 +0100
                Re: RFS: mwic 0.7.4-1 Paul Wise <pabs@debian.org> - 2018-03-25 05:10 +0200
                Re: RFS: mwic 0.7.4-1 Georg Faerber <georg@riseup.net> - 2018-03-25 05:50 +0200
      Re: RFS: mwic 0.7.4-1 Dmitry Shachnev <mitya57@debian.org> - 2018-03-19 22:50 +0100
        Re: RFS: mwic 0.7.4-1 Georg Faerber <georg@riseup.net> - 2018-03-19 23:40 +0100
          Re: RFS: mwic 0.7.4-1 Simon McVittie <smcv@debian.org> - 2018-03-20 12:50 +0100
          Re: RFS: mwic 0.7.4-1 Dmitry Shachnev <mitya57@debian.org> - 2018-03-20 18:30 +0100
            Re: RFS: mwic 0.7.4-1 Georg Faerber <georg@riseup.net> - 2018-03-20 19:10 +0100

csiph-web