Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #16539 > unrolled thread

python-sigstore / python-tuf: request for packaging help

Back to article view | Back to linux.debian.maint.python

Contents

  python-sigstore / python-tuf: request for packaging help Simon Josefsson <simon@josefsson.org> - 2024-12-05 23:40 +0100
    Re: python-sigstore / python-tuf: request for packaging help Andrey Rakhmatullin <wrar@debian.org> - 2024-12-06 08:40 +0100
      Re: python-sigstore / python-tuf: request for packaging help Simon Josefsson <simon@josefsson.org> - 2024-12-06 17:00 +0100
        Re: python-sigstore / python-tuf: request for packaging help Andrey Rakhmatullin <wrar@debian.org> - 2024-12-06 17:20 +0100

#16539 — python-sigstore / python-tuf: request for packaging help

[Multipart message — attachments visible in raw view] — view raw

Hi

I am new to python debian packaging, and I'm looking for guidance and
review my packaging.  I'm happy to team-maintain these packages if
someone can add me to the salsa group.

Right now I am working on python-sigstore and my packaging is here:

https://salsa.debian.org/jas/sigstore-python/

as you can see in the pipeline, it currently fails due to lacking tuf:

https://salsa.debian.org/jas/sigstore-python/-/jobs/6706412

It seems python-tuf has an old ITP/RFS report here:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=934151
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931178

I started from scratch on python-tuf with the latest upstream version,
instead of trying to understand the old packaging work that I couldn't
get to work.  Upstream evolved a lot.  My packaging is here:

https://salsa.debian.org/jas/python-tuf/

The self-tests tries seems to open some files which fails and I suspect
it is because srcdir != builddir reasons, or something similar, see
errors here:

https://salsa.debian.org/jas/python-tuf/-/jobs/6707693

==================================== ERRORS ====================================
______________ ERROR collecting tests/test_metadata_generation.py ______________
tests/test_metadata_generation.py:10: in <module>
    from tests.generated_data.generate_md import generate_all_files
tests/generated_data/generate_md.py:60: in <module>
    os.mkdir(OUT_DIR)
E   FileNotFoundError: [Errno 2] No such file or directory: 'generated_data/ed25519_metadata'

How is this (seamingly common) problem solved normally?  Do we copy test
data files into the build directory somehow?  Do we include them in the
package?  Do we patch hard-coded paths like this to make it work?

If anyone can review my python-tuf and python-sigstore packages, that
would be appreciated -- all nit-picks and style advice is most
recommended, as I have not worked on python packaging before.  Don't
assume I understand any of the debian/* content so feel free to question
some decision.

If you want to build python-tuf locally, it depends on a more recent
version of python3-securesystemslib and you can find it here:

https://salsa.debian.org/jas/securesystemslib/
https://salsa.debian.org/jas/securesystemslib/-/pipelines/774721
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1089125

Thanks,
/Simon

[toc] | [next] | [standalone]

#16543

[Multipart message — attachments visible in raw view] — view raw

On Thu, Dec 05, 2024 at 11:39:24PM +0100, Simon Josefsson wrote:
> The self-tests tries seems to open some files which fails and I suspect
> it is because srcdir != builddir reasons, or something similar, see
> errors here:
> 
> https://salsa.debian.org/jas/python-tuf/-/jobs/6707693
> 
> ==================================== ERRORS ====================================
> ______________ ERROR collecting tests/test_metadata_generation.py ______________
> tests/test_metadata_generation.py:10: in <module>
>     from tests.generated_data.generate_md import generate_all_files
> tests/generated_data/generate_md.py:60: in <module>
>     os.mkdir(OUT_DIR)
> E   FileNotFoundError: [Errno 2] No such file or directory: 'generated_data/ed25519_metadata'
> 
> How is this (seamingly common) problem solved normally?  Do we copy test
> data files into the build directory somehow?  Do we include them in the
> package?  Do we patch hard-coded paths like this to make it work?

It's not common and test data files are already copied because they are
under tests/. It's because of relative paths and I can reproduce this
problem by running pytest in the upstream's git checkout. See also
tox.ini:

# TODO: Consider refactoring the tests to not require the aggregation script
# being invoked from the `tests` directory.

No idea if patching the path is enough or you need PYBUILD_BEFORE_TEST
with `cd {dir}/tests` or something like that.

-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#16546

[Multipart message — attachments visible in raw view] — view raw

Andrey Rakhmatullin <wrar@debian.org> writes:

> On Thu, Dec 05, 2024 at 11:39:24PM +0100, Simon Josefsson wrote:
>> The self-tests tries seems to open some files which fails and I suspect
>> it is because srcdir != builddir reasons, or something similar, see
>> errors here:
>> 
>> https://salsa.debian.org/jas/python-tuf/-/jobs/6707693
>> 
>> ==================================== ERRORS ====================================
>> ______________ ERROR collecting tests/test_metadata_generation.py ______________
>> tests/test_metadata_generation.py:10: in <module>
>>     from tests.generated_data.generate_md import generate_all_files
>> tests/generated_data/generate_md.py:60: in <module>
>>     os.mkdir(OUT_DIR)
>> E   FileNotFoundError: [Errno 2] No such file or directory: 'generated_data/ed25519_metadata'
>> 
>> How is this (seamingly common) problem solved normally?  Do we copy test
>> data files into the build directory somehow?  Do we include them in the
>> package?  Do we patch hard-coded paths like this to make it work?
>
> It's not common and test data files are already copied because they are
> under tests/. It's because of relative paths and I can reproduce this
> problem by running pytest in the upstream's git checkout.

You are right, I reported it upstream:

https://github.com/theupdateframework/python-tuf/issues/2745

I disabled that self-check meanwhile, and got a bit further:

collected 184 items
tests/test_api.py ....................................                   [ 19%]
tests/test_examples.py FFF                                               [ 21%]
tests/test_fetcher_ng.py FFFF.FFFF...                                    [ 27%]
tests/test_metadata_eq_.py ....                                          [ 29%]
tests/test_metadata_serialization.py .........................           [ 43%]
tests/test_repository.py ........                                        [ 47%]
tests/test_trusted_metadata_set.py ..........................            [ 61%]
tests/test_updater_consistent_snapshot.py ...                            [ 63%]
tests/test_updater_delegation_graphs.py ......                           [ 66%]
tests/test_updater_fetch_target.py .....                                 [ 69%]
tests/test_updater_key_rotations.py ..                                   [ 70%]
tests/test_updater_ng.py .FF.FF.F.FFFF                                   [ 77%]
tests/test_updater_top_level_update.py ................................. [ 95%]
...                                                                      [ 97%]
tests/test_updater_validation.py ..                                      [ 98%]
tests/test_utils.py F..                                                  [100%]
...
================== 21 failed, 163 passed, 5 warnings in 5.39s ==================

See full log:

https://salsa.debian.org/jas/python-tuf/-/jobs/6710680/viewer

Several failures seems related to not being able to find some test file,
any ideas how to fix this further?

I'll see if I can figure out how to disable only the failing tests,
pending some better fix.  It seems a large portion of the self-tests now
work so I think this is in good enough shape for an upload to NEW.

/Simon

[toc] | [prev] | [next] | [standalone]

#16547

[Multipart message — attachments visible in raw view] — view raw

On Fri, Dec 06, 2024 at 04:50:07PM +0100, Simon Josefsson wrote:
> See full log:
> 
> https://salsa.debian.org/jas/python-tuf/-/jobs/6710680/viewer
> 
> Several failures seems related to not being able to find some test file,
> any ideas how to fix this further?

examples/manual_repo/basic_repo.py? You can copy the examples folder using
debian/pybuild.testfiles. But it looks like most of the failures are
network-related, you may need to disable the pybuild-set dummy proxies
(not sure how) for these to work if they use only localhost networking.

-- 
WBR, wRAR

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web