Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #11810

Re: Streamlining the use of Salsa CI on team packages

Show key headers only | View raw

[Multipart message — attachments visible in raw view] - view raw

On 19-09-14 17 h 35, Thomas Goirand wrote:
> On 9/13/19 11:08 PM, Louis-Philippe Véronneau wrote:
>> On 19-09-13 05 h 57, Thomas Goirand wrote:
>>> On 9/5/19 7:40 AM, Louis-Philippe Véronneau wrote:
>>>> Hello folks!
>>>>
>>>> I'd like to propose we start using Salsa CI for all the team packages. I
>>>> think using a good CI for all our packages will help us find packaging
>>>> bugs and fix errors before uploads :)
>>>
>>> I would agree *IF* and only *IF* we find better runners than the one
>>> currently default in Salsa. The GCE runners are horribly slow (they are
>>> the smallest to avoid cost). As a result, some tests may just fail
>>> because of that, and it becomes just frustrating / annoying noise.
>>
>> I never experienced such timeouts, but I guess I don't work on very
>> large packages or things that take more than a few minutes to build.
> 
> The issue isn't build time. But when you have unit tests sensitive to
> timing. See for example openvswitch:
> 
> https://salsa.debian.org/openstack-team/third-party/openvswitch/pipelines/61713

Do you have similar issues running those CI tasks in a private runner?
(I'm really curious since I haven't had problems and the Salsa runners
don't seem slow compared to the private runners I run on my machines).

Maybe one solution to your problem would be to provide a fast/responsive
shared runners to the Salsa Team and tag your CI pipelines to use that
runner exclusively [1]?

[1] https://docs.gitlab.com/ee/ci/yaml/#tags

> Oh, in fact, don't ... Salsa doesn't keep artifacts / logs long enough,
> so you will see nothing. I wonder why the Salsa admins decided on saving
> them on Google infrastructure then! Or did I misunderstood the way it
> works? Hard to tell, because we get almost zero communication about this
> from the admins.
> 
>> If what you describe really is caused by the default runners not being
>> fast enough, why couldn't we ask the Salsa team for more powerful ones?
>> Have you talked to them about this?
>
> Since when Salsa admins are receptive to critics and/or suggestions? The
> last time we told them that using Google wasn't a good idea, they just
> ignored it. Do you even know how the default runners were provisioned?
> Who's paying? How much credit do we have? How much can we use them?

AFAIU, they are provisioned with terraform [2] and then configured with
ansible [3]. Generally most of the code used to run and maintain Salsa
is in their repository.

I don't know who is paying (my guess is Debian+some Google credits), but
I feel these are valid questions and I don't see why the Salsa Team
would not answer if you ask them.

I know they looked for CI runner sponsors for a while and I'm not sure
they were able to find enough to meet their requirements.

[1]
https://salsa.debian.org/salsa/salsa-terraform/blob/master/environments/prod/runner.tf
[2]
https://salsa.debian.org/salsa/salsa-ansible/tree/master/roles/gitlab-runner

>> It seems to me that spending money in QA like CI runners is very
>> profitable for the project, as it saves everyone a lot of time dealing
>> with unnecessary failure caused by lack of tests. It's not like Debian
>> is a very poor organisation...
> 
> Indeed. It'd be even better if we could have our own cloud, but nobody
> in power seem receptive to the idea.
> 
> Also, please consider what happened the last time someone added 1000+ CI
> jobs (ie: Salsa went kind of down, and the person who did that got kind
> of flamed by Salsa admins).

It's possible to push to Salsa without triggering a CI run with "git
push -o ci.skip" or by including "[ci-skip]" in the HEAD commit message.

IIUC, the problem isn't the overall amount of repositories using the CI,
but adding a 1000 ones at the same time and overloading the runners.

> At this point in time, I really don't think Salsa is ready for what you
> proposed, unless we at least:
> 
> 1/ Take a super big care when adding jobs.

I feel this is easily resolved by the "-o ci.skip" thing.

> 2/ We have our own CI runners.

I'm not 100% sure that's a good idea. The Salsa Team has pretty strict
requirements about shared runners (they require root on those machines
to make sure the .debs created by the runners can be trusted) and I'm
happy they do.

Running Gitlab CI runners takes a bit of work (I run multiple ones,
including a major shared one on 0xacab.org) and if we want people to
feel like they can download the artifacts, we need to make sure the CI
runners we use are meet the high standards.

Also, as I'm sure you are aware of, maintaining stuff takes time and
effort and if we can forgo that and let a dedicated team of volunteers
do it (the Salsa team), I think it's a win.

I really wonder how common the issues you've experienced with the Salsa
CI runners are. Has anyone here had similar problems?

I'd be fine with 95% of our package using the same default pipeline and
the last 5% using something else or disabling it and adding a few
comments in d/gitlab-ci.yml explaining why.

> As I wrote, I believe my company could provide a few of these runners
> (pending my boss approval, who's currently on holidays). However, it'd
> be nice if my company wasn't the only sponsor... Not only because of
> financial issues, but at least for redundancy.

The place I work at sadly can't provide root on VMs to other people, so
I can't really help :(

FWIW, I've opened an issue on the Salsa Support issue tracker to see
what the Salsa team thinks of this whole discussion [3]

[3]: https://salsa.debian.org/salsa/support/issues/170

-- 
  ⢀⣴⠾⠻⢶⣦⠀
  ⣾⠁⢠⠒⠀⣿⡁  Louis-Philippe Véronneau
  ⢿⡄⠘⠷⠚⠋   pollo@debian.org / veronneau.org
  ⠈⠳⣄

Back to linux.debian.maint.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Streamlining the use of Salsa CI on team packages Louis-Philippe Véronneau <pollo@debian.org> - 2019-09-05 07:50 +0200
  Re: Streamlining the use of Salsa CI on team packages Louis-Philippe Véronneau <pollo@debian.org> - 2019-09-12 17:00 +0200
  Re: Streamlining the use of Salsa CI on team packages Louis-Philippe Véronneau <pollo@debian.org> - 2019-09-13 23:10 +0200
    Re: Streamlining the use of Salsa CI on team packages Louis-Philippe Véronneau <pollo@debian.org> - 2019-09-15 04:20 +0200
      Re: Streamlining the use of Salsa CI on team packages Louis-Philippe Véronneau <pollo@debian.org> - 2019-09-16 02:20 +0200
      Re: Streamlining the use of Salsa CI on team packages Marcin Kulisz <debian@kulisz.net> - 2019-09-16 09:30 +0200
  Re: Streamlining the use of Salsa CI on team packages Louis-Philippe Véronneau <pollo@debian.org> - 2019-09-16 02:40 +0200
    Re: Streamlining the use of Salsa CI on team packages Raphael Hertzog <hertzog@debian.org> - 2019-09-17 10:10 +0200
    Re: Streamlining the use of Salsa CI on team packages Louis-Philippe Véronneau <pollo@debian.org> - 2019-10-12 01:40 +0200
    RE:Streamlining the use of Salsa CI on team packages PICCA Frederic-Emmanuel <frederic-emmanuel.picca@synchrotron-soleil.fr> - 2019-10-14 16:10 +0200
      Re: Streamlining the use of Salsa CI on team packages Peter Pentchev <roam@ringlet.net> - 2019-10-15 09:40 +0200

csiph-web