Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #17035 > unrolled thread

Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead

Back to article view | Back to linux.debian.maint.python

Contents

  Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline,  use debian/salsa-ci.yml instead Otto Kekäläinen <otto@debian.org> - 2025-08-27 22:20 +0200
    Re: Debian Python team: don't use  recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead Andrey Rakhmatullin <wrar@debian.org> - 2025-08-27 22:30 +0200
        Re: Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline,  use debian/salsa-ci.yml instead Soren Stoutner <soren@debian.org> - 2025-08-27 22:50 +0200
        Re: Debian Python team: don't use  recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead Carsten Schoenert <c.schoenert@t-online.de> - 2025-08-28 14:20 +0200
        Re: Debian Python team: don't use  recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead Andrey Rakhmatullin <wrar@debian.org> - 2025-08-28 15:30 +0200
        Re: Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline,  use debian/salsa-ci.yml instead Soren Stoutner <soren@debian.org> - 2025-08-27 23:00 +0200
    Re: Debian Python team: don't use  recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead Santiago Vila <sanvila@debian.org> - 2025-08-27 23:00 +0200
      Re: Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline,  use debian/salsa-ci.yml instead Otto Kekäläinen <otto@debian.org> - 2025-09-09 02:40 +0200
        Re: Debian Python team: don't use  recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead Santiago Vila <sanvila@debian.org> - 2025-09-09 03:20 +0200
        Re: Debian Python team: don't use  recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead Santiago Vila <sanvila@debian.org> - 2025-09-09 03:40 +0200
        Re: Debian Python team: don't use  recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead Santiago Vila <sanvila@debian.org> - 2025-09-09 15:10 +0200
            Re: Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline,  use debian/salsa-ci.yml instead Soren Stoutner <soren@debian.org> - 2025-09-09 19:20 +0200

#17035 — Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead

Hi!

tldr;
Are people on the team fine with not using
recipes/debian.yml@salsa-ci-team/pipeline as the default setting in
future and using debian/salsa-ci.yml instead?

context:
I was looking at which of the Salsa CI jobs take longest and consume
most resources and stumbled across
https://salsa.debian.org/python-team/packages/graph-tool

The build uses more memory than available on a Salsa CI runner, and it
has never, and will never build. The reason it had CI running anyway
was that in the CI/CD configuration it had
`recipes/debian.yml@salsa-ci-team/pipeline`, which in turn was added
in all Python projects in an effort to enable CI to all packages.

I suggest a better practice would be to have as the CI/CD
configuration simply `debian/salsa-ci.yml` instead, so that the
pipeline runs only on Debian branches and starts running only after a
human added it there with intention and presumably that it actually
passed initially when CI was introduced.

I did this change now on the graph-tool project and also posted
https://salsa.debian.org/python-team/packages/graph-tool/-/merge_requests/1
to have the state of Salsa CI in the project documented.
- Otto

[toc] | [next] | [standalone]

#17036 — Re: Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead

[Multipart message — attachments visible in raw view] — view raw

On Wed, Aug 27, 2025 at 01:17:20PM -0700, Otto Kekäläinen wrote:
>I suggest a better practice would be to have as the CI/CD
>configuration simply `debian/salsa-ci.yml` instead,

Yet another busywork change...

>starts running only after a
>human added it there with intention 

I remember that enabling Salsa CI by setting that GUI setting to 
"recipes/debian.yml@salsa-ci-team/pipeline" was also done by humans with 
intention. Per-package.
https://wiki.debian.org/Salsa/Doc#Salsa_CI_setup still recommends it.


-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#17037

[Multipart message — attachments visible in raw view] — view raw

On Wednesday, August 27, 2025 1:26:23 PM Mountain Standard Time Andrey 
Rakhmatullin wrote:
> On Wed, Aug 27, 2025 at 01:17:20PM -0700, Otto Kekäläinen wrote:
> >I suggest a better practice would be to have as the CI/CD
> >configuration simply `debian/salsa-ci.yml` instead,
> 
> Yet another busywork change...

It’s not just a busywork change.  If you have the CI set to `recipes/
debian.yml@salsa-ci-team/pipeline` and someone forks the package, Salsa CI 
does not automatically run on the fork because this setting is reset to the 
default.  But, if you leave this setting as the default and create a debian/
salsa-ci.yml file, then forks automatically have the same Salsa CI as the 
source repository.

-- 
Soren Stoutner
soren@debian.org

[toc] | [prev] | [next] | [standalone]

#17041 — Re: Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead

Am 27.08.25 um 22:43 schrieb Soren Stoutner:

>> Yet another busywork change...
> 
> It’s not just a busywork change.  If you have the CI set to `recipes/
> debian.yml@salsa-ci-team/pipeline` and someone forks the package, Salsa CI
> does not automatically run on the fork because this setting is reset to the
> default.  But, if you leave this setting as the default and create a debian/
> salsa-ci.yml file, then forks automatically have the same Salsa CI as the
> source repository.

That's not true, at least I can't confirm this. I moved a package into 
the DPT namespace and forked the project afterwards into my namespace 
the CI setting with the global pipeline is still there!

In general I disagree to the proposal Otto has made. The need to 
manually add a control file is the opposite what I expect, we should 
decrease the amount of work. Adding a extra files to the debian/ folder 
makes it not easier to do package maintenance. The default should be 
that some CI is triggered automatically without any needed action by a 
maintainer.

I try to use the mentioned recipe in all Python packages and avoid to 
add some extra file there I'm listed in the maintainer field.

I only add a dedicated YAML file if I'm not satisfied by the global 
recipe, e.g. mostly if I've an arch related binary and building on some 
architecture will always fail as the platform isn't supported.

-- 
Regards
Carsten

[toc] | [prev] | [next] | [standalone]

#17042 — Re: Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead

[Multipart message — attachments visible in raw view] — view raw

On Wed, Aug 27, 2025 at 01:43:37PM -0700, Soren Stoutner wrote:
>> >I suggest a better practice would be to have as the CI/CD
>> >configuration simply `debian/salsa-ci.yml` instead,
>>
>> Yet another busywork change...
>
>It’s not just a busywork change.  If you have the CI set to `recipes/
>debian.yml@salsa-ci-team/pipeline` and someone forks the package, Salsa CI
>does not automatically run on the fork because this setting is reset to the
>default.  But, if you leave this setting as the default and create a debian/
>salsa-ci.yml file, then forks automatically have the same Salsa CI as the
>source repository.

Frankly, I don't imagine somebody forking my DPT repos.


-- 
WBR, wRAR

[toc] | [prev] | [next] | [standalone]

#17039

[Multipart message — attachments visible in raw view] — view raw

On Wednesday, August 27, 2025 1:26:23 PM Mountain Standard Time Andrey 
Rakhmatullin wrote:
> https://wiki.debian.org/Salsa/Doc#Salsa_CI_setup still recommends it.

The documentation was changed a while ago, but somehow that location was 
missed (if you had followed the links it contained you would have seen it was 
no longer accurate).  I have updated it to match the Salsa CI documentation 
at:

https://salsa.debian.org/salsa-ci-team/pipeline#salsa-continuous-integration-ci--quality-assurance-for-debian-packaging

-- 
Soren Stoutner
soren@debian.org

[toc] | [prev] | [next] | [standalone]

#17038 — Re: Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead

On Wed, Aug 27, 2025 at 01:17:20PM -0700, Otto Kekäläinen wrote:

> I was looking at which of the Salsa CI jobs take longest and consume
> most resources and stumbled across
> https://salsa.debian.org/python-team/packages/graph-tool
> 
> The build uses more memory than available on a Salsa CI runner, and it
> has never, and will never build.

I don't think that's an accurate way to describe the status of graph-tool.

The problem with graph-tool is that it needs a lot of memory per CPU.

But I managed to make it work in most cases (see the commits and the
uploads I did) by making some calculations in debian/rules which
adjust the parallelism according to available memory. If those
calculations do not work in Salsa CI, we might better figure out why,
instead of saying "will never build".

Thanks.

[toc] | [prev] | [next] | [standalone]

#17051

> > I was looking at which of the Salsa CI jobs take longest and consume
> > most resources and stumbled across
> > https://salsa.debian.org/python-team/packages/graph-tool
> >
> > The build uses more memory than available on a Salsa CI runner, and it
> > has never, and will never build.
>
> I don't think that's an accurate way to describe the status of graph-tool.
>
> The problem with graph-tool is that it needs a lot of memory per CPU.
>
> But I managed to make it work in most cases (see the commits and the
> uploads I did) by making some calculations in debian/rules which
> adjust the parallelism according to available memory. If those
> calculations do not work in Salsa CI, we might better figure out why,
> instead of saying "will never build".

Thanks Santiago for the additional information. I updated the MR
description now at
https://salsa.debian.org/python-team/packages/graph-tool/-/merge_requests/1,
and I suggest we merge it now to stop CI failures, and re-enable it
later if/when the CI is fixed.

Having a regression testing system that is not passing and cannot
detect any regressions is not productive and wastes resources.

[toc] | [prev] | [next] | [standalone]

#17052 — Re: Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead

Hi.

Otto Kekäläinen wrote:
> Having a regression testing system that is not passing and cannot
> detect any regressions is not productive and wastes resources.

I agree with that.

> I updated the MR description now at
> https://salsa.debian.org/python-team/packages/graph-tool/-/merge_requests/1,
> and I suggest we merge it now to stop CI failures, and re-enable it
> later if/when the CI is fixed.

I think you did not understand my previous comment.

Currently debian/rules calculates the number of CPUs that should be
used according to available memory. If there is at least 13000 MB of RAM
available, a single CPU will be used and the build should work, provided
we allow it to run for whatever amount of time it needs.

Do Salsa CI runners have really less than 13000 MB of RAM?

(If that's really the case, then there would be a lot more packages failing).

Anyway, I don't like the wording at the top of the file, it sounds to
me as a "rant" (sorry to tell, but that's how it sounds to me).

Would you mind if I just disable the pipeline with a different (more simple)
wording as an explanation? I'm thinking about something in the line of "This
pipeline is currently disabled until someone manages to make it work".

Thanks.

[toc] | [prev] | [next] | [standalone]

#17053 — Re: Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead

Ok, I've taken a look at the build log. A while ago I inserted a "spy"
line in debian/rules so that we could see this in the build log:

===========================
MemAvailable:    7472456 kB
SwapFree:              0 kB
nproc: 2
===========================

So, yes, this will not work in Salsa CI, unless there is a lot of swap.

Simple questions:

Do all Salsa CI runners have the same amount of RAM?
Do all Salsa CI runners have zero amount of swap? (Why?)

(While we are at it: What should people do when a package needs more
than 8 GB of RAM to build? I could provide a list of such packages, as
I collect statistics during my archive rebuilds).

Thanks.

[toc] | [prev] | [next] | [standalone]

#17055 — Re: Debian Python team: don't use recipes/debian.yml@salsa-ci-team/pipeline, use debian/salsa-ci.yml instead

Hello Otto.

I've finally disabled the pipelines, not exactly in the same way you
proposed in the MR, but in another way which I believe preserves the
spirit.

I still would like to know how much memory can we expect from Salsa CI
runners, and how likely or unlikely it is that we will be able to
enable Salsa CI for graph-tool some day.

Thanks.

[toc] | [prev] | [next] | [standalone]

#17056

[Multipart message — attachments visible in raw view] — view raw

On Tuesday, September 9, 2025 6:02:14 AM Mountain Standard Time Santiago Vila 
wrote:
> Hello Otto.
> 
> I've finally disabled the pipelines, not exactly in the same way you
> proposed in the MR, but in another way which I believe preserves the
> spirit.
> 
> I still would like to know how much memory can we expect from Salsa CI
> runners, and how likely or unlikely it is that we will be able to
> enable Salsa CI for graph-tool some day.

In case it is helpful to others, I have setup a small number of custom runners 
for packages I maintain where the standard CI runners are not sufficient.

In my case, these are situations where the standard CI runners cannot finish 
at least one of the stages within 3 hours.  Using a custom runner allows me to 
set any timeout I prefer.  In some cases, because the hardware on my custom 
runner is more powerful than the standard CI runners, I no longer need to 
adjust the timeout, but in other cases an adjustment is still needed.  For 
example:

https://salsa.debian.org/python-team/packages/pyinstaller/-/blob/debian/
master/debian/salsa-ci.yml?ref_type=heads

For anyone wanting to setup a custom runner, I have updated the wiki with the 
instructions that worked for me:

https://wiki.debian.org/Salsa/Doc/CustomRunners

To round out this conversation, I am currently in the process of developing a 
custom salsa-ci.yml for qt6-webengine.  The unique problem with that package 
is that it produces over 4.5 GB of artifacts, which exceeds the maximum 
allowed by Salsa (in addition to needing to run for longer and needing more 
system resources than the standard runners).  I have not finished, but 
currently I have a modified salsa-ci.yml that performs some of the tests I am 
interested in tracking:

https://salsa.debian.org/soren/qt6-webengine/-/blob/salsa-ci/debian/salsa-ci.yml?ref_type=heads

-- 
Soren Stoutner
soren@debian.org

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web