Path: csiph.com!aioe.org!bofh.it!news.nic.it!robomod From: =?UTF-8?Q?Louis-Philippe_V=c3=a9ronneau?= Newsgroups: linux.debian.maint.python Subject: Re: Streamlining the use of Salsa CI on team packages Date: Sun, 15 Sep 2019 04:20:01 +0200 Message-ID: References: X-Mailbox-Line: From debian-python-request@lists.debian.org Sun Sep 15 02:11:14 2019 Old-Return-Path: X-Amavis-Spam-Status: No, score=-11.1 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, BODY_8BITS=1.5, FOURLA=0.1, LDO_WHITELIST=-5, PGPSIGNATURE=-5, RCVD_IN_DNSWL_LOW=-0.7] autolearn=ham autolearn_force=no X-Policyd-Weight: using cached result; rate: -4.6 X-Riseup-User-ID: 6249A45B2766C8350BDA5EC9C1B3DD069DA29C2CD20A32A11EDD1E974D261882 Openpgp: preference=signencrypt Autocrypt: addr=pollo@debian.org; prefer-encrypt=mutual; keydata= mQINBFSX1NABEADH76idoFsToCSwRrCJAqoZMPFWNGdK5+lMpxfVxb3GQYzEmxexkqq2ga/N oUlEs1jKY7F513uDARm2IVsE4ypq3TqYYZtSIr+27NvWUCAleFTBcTRvlo6pdBwaF9bENRgg uoMuHdY8OkNxg1jhwiq0kWpImD1K91NokKiX/9GebCIsr3J5ETafJ+pcHc6N3v8BIC2C8QAu c+fh5Atb9qZglNdFQ9orFkvCDKvaa6ZOymwpAW27JLY1NjSj8S+bLle2XKC3mNq2jtVN0vX0 V9UrNEOHpXokuFLXQFoGZGdqyQBBG3Hj9q0+NAJxMmNbBcOroOIDLyuyVFTWngaoU7m+Tlsm enB0T38ifOV5oefO9B+X8kqi0G3OdhNR/m4NC50q5QXkhis6195f4tdlBZQtipLygAOoeenx XRnxbKNoNA2PUz5FRBzPytsNnmkW80tY0eleZ6lXmO1Kuas3jaQOtBMl2HuBO6wn0QAhFmze qfq5gbexu4iCZYlnUkqCYsiFkrjyPUo+o/bDMTGz7sSrrv9W1Rg4lTiaAiysaPZ3HeSBGTF8 feHXPkwfCZiimFU7ojildMqkq6ZWIAdoMdYi8CGYCxg+xtapXASsTeXJ7Om6R+TBaaV9WNBy WOHhOLCI57bm0dQ8HwqB/rNufYooGXyRDDLMcbLGhu64iPYN1wARAQABtDVMb3Vpcy1QaGls aXBwZSBWw6lyb25uZWF1IChkZWJpYW4pIDxwb2xsb0BkZWJpYW4ub3JnPokCVAQTAQoAPgIb IwULCQgHAwUVCgkICwUWAgMBAAIeAQIXgBYhBGd/VPH6hoGtjsC85nrqxOxqqgqXBQJc9DQN BQkKPZK3AAoJEHrqxOxqqgqXMZoP/RofvJ+bWe5xvCukbuBMrinrs541sZJgD4HtLz5/kFwH 8dH9QYIdWFcT+nwK0L3/zkc/GOgbcIZHssguGT01EBK5URJMoZla0eDfKNM+bAZoe9uO5vDu 7YHLY3a6Ky7mAkqLZRmBv5gRNUub27PSdgpxphTmQIrKwkp3E7sOF98zi4aqB+nIfPlIWRJl b2DZJ32leN94Lls+KRNFZK0LP/+XyFr7PTzWrJYfGYyeTvlBM7suUhluBFU5PuWJXtpaMcZk mgvMKjUA62KQuN86V7YfPSkyqg6dWH3rzfd10+qEE23u0TGnXRT4ynUyP0KiCwJtoqeSur/C S3LXkdrwvzaZqzYjLT45QcddCjIGpbp3bJU/DnpEHB9K7THiPx3Tz8X6QZ7211356mZT3p0f FQ81gIgYpgYggmpxvQzwLOhhW6YNHm9Qc2e+1EHLoGyqCgUR8xDa5jJrV30Z1arBWFzsJkuO Y+TMh1JgTLaNqIpsx+zLlSTM0WUWbvssnliqc9Qul4qlOrD1G1HCqqytKqaA0oE6X/pL28gC YkiNrs9DJRXEB2cBcpbsxS6EwtNfNIH+hjUZM7K7U5+EW+ai4lpTtjARYIeIdgIXJiwZSPXq Pibr1k9PkYhCCjkWn/L2RT4+SOmwgIZvAkTUII0XhLt1789LCUX2njLxcwDNCV8duQINBFSX 1NABEAC28f43K11snkvXvQsDKKyCELMeCzh4CkQ/QvalGns6ALo/SnJ+XjtaTQz1Ubx7T4Be 6Xlp7I069bv/xhV/1GrvbAE6UUJE9VF0iry2AkHr7Ug4pIqVcelMQkSYaMVUKYrGtyxOxvjn zJ4++kvzpo5cqELeywVnSdYo3ApZqDZbirEqLzehj/WyMk2kAQYdhtZDOlPmroQBRH8VvRZd u6IqOjME5T2AlG1D4YGm/tQsQl4mcK40BPGeyOECnKRD2dCfO5BJaEO91B1lYtiXqJd/e1dU Oj2vetvMmIPhYyJJHWOK7dANM/sleaygplVfCbICCM5StvBVV5k4Zu94gg+3N/DU2rOOPnMg eO2MG63qSGR2A8Id2f0/zL6rBrvOQxXRyI62qWPzwHksDxOXCId/OlUcUf5iejNouubRrnNA 8Z4J88wcdQ6sjW0FlaPp2YCeq2z2OLb/iTxgZrMYn6cOhZqDNB3pGh2l/dCxmfOm6dB5dvAL Fv1XjNEJIr6NgcZusMS2YoPDDNv1dIzf3EIEDpLR1zuTsTXhqUfinPrrJsFWLo1/lay2t+1c mobPKJgEzESKG5260Jv6pltc/jM3FSYAUgvaSO2/Rmyt7tfBnreBDPS5HUqQSdyh5/1lbkuZ +7zU+ikFdSa5lQIammrYePDxPgX5rlqSAUghaFFqHQARAQABiQI8BBgBCgAmAhsMFiEEZ39U 8fqGga2OwLzmeurE7GqqCpcFAlz0NA0FCQo9kr0ACgkQeurE7GqqCpfpEQ/9E6V2URLkfTVS mYUp5tznjTVARp5mnkjgYGJD0zJ6je/vzAwocVQmEp7vO7YjUie2mBFF+IrIZDKkYA8e9E0h 2wsYAL0Hs/V8bX0aPwdU4KpyiTVSGDDS6SavF95fkJJtTXAt11NXhhUUpYe9Ad1vhSWW4v/9 bD7XYz61EwH2WHf/cKW3GwinF/HqAG5rGMY3bCPO/FzQvwXl9mDyJ6GTw3SS/2yFTVGThQCN na8MeyFYX3oXbiCUkKBxYHVeyzZRTGLFyTP6SHUEZQ/1fzqw3CpIvEMBeIOg89HpgfSfpFyU 5SA73Ft53rXKZxqhdLf0xuT3aX1NWoM1wjpV4BJO7Xqyg5AkeLgOEF69zdaYdR69tjdI7VAB Rv8c33ue8JB9AKHrE+WnjIE4RmELC6gYn5qOOOUOlIZTlSUptHc8jVxwsby+Di9JQTyJM59E tj5C0wBCzqLZ7U4cPbRgxhgey30Lxn6MGQMEcPhQReqbbGn3VFq5PDjkvwh2Mn7vfsd2r8kp zSvqB285Q/GKjtkrt71rRGmKYqfZs7gsPn3MaI1xOmM+mqiYgUrLhX2M+bAIM+ImP5MPev8V CEs+I2oTwYxgDHvAOPVLCic0EZ4Sg0KvOwiKM3MaXlZVbpwa+U8Z91kQzkl3IvqjkDDVJNzB VNf7CZNxv+Lpm13sI2wPoWc= MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="oZmsPOKSIlNSXm9hHq0E79LLli4jPRnGC" X-Mailing-List: archive/latest/17343 List-ID: List-URL: List-Archive: https://lists.debian.org/msgid-search/81acc63f-ffe8-ff28-71e2-8219b643f289@debian.org Approved: robomod@news.nic.it Lines: 195 Organization: linux.* mail to news gateway Sender: robomod@news.nic.it X-Original-Date: Sat, 14 Sep 2019 22:10:44 -0400 X-Original-Message-ID: <81acc63f-ffe8-ff28-71e2-8219b643f289@debian.org> X-Original-References: <3429cc8e-87d4-facc-870d-1e664fe0779f@debian.org> <8b08c6e5-96f6-cffc-9c03-00d0fa9a8540@debian.org> Xref: csiph.com linux.debian.maint.python:11810 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --oZmsPOKSIlNSXm9hHq0E79LLli4jPRnGC Content-Type: multipart/mixed; boundary="SOuz7aQP0CE78gEWw0XRA8kljDZGyVcGh"; protected-headers="v1" From: =?UTF-8?Q?Louis-Philippe_V=c3=a9ronneau?= To: debian-python@lists.debian.org Message-ID: <81acc63f-ffe8-ff28-71e2-8219b643f289@debian.org> Subject: Re: Streamlining the use of Salsa CI on team packages References: <3429cc8e-87d4-facc-870d-1e664fe0779f@debian.org> <8b08c6e5-96f6-cffc-9c03-00d0fa9a8540@debian.org> In-Reply-To: <8b08c6e5-96f6-cffc-9c03-00d0fa9a8540@debian.org> --SOuz7aQP0CE78gEWw0XRA8kljDZGyVcGh Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 19-09-14 17 h 35, Thomas Goirand wrote: > On 9/13/19 11:08 PM, Louis-Philippe V=C3=A9ronneau wrote: >> On 19-09-13 05 h 57, Thomas Goirand wrote: >>> On 9/5/19 7:40 AM, Louis-Philippe V=C3=A9ronneau wrote: >>>> Hello folks! >>>> >>>> I'd like to propose we start using Salsa CI for all the team package= s. I >>>> think using a good CI for all our packages will help us find packagi= ng >>>> bugs and fix errors before uploads :) >>> >>> I would agree *IF* and only *IF* we find better runners than the one >>> currently default in Salsa. The GCE runners are horribly slow (they a= re >>> the smallest to avoid cost). As a result, some tests may just fail >>> because of that, and it becomes just frustrating / annoying noise. >> >> I never experienced such timeouts, but I guess I don't work on very >> large packages or things that take more than a few minutes to build. >=20 > The issue isn't build time. But when you have unit tests sensitive to > timing. See for example openvswitch: >=20 > https://salsa.debian.org/openstack-team/third-party/openvswitch/pipelin= es/61713 Do you have similar issues running those CI tasks in a private runner? (I'm really curious since I haven't had problems and the Salsa runners don't seem slow compared to the private runners I run on my machines). Maybe one solution to your problem would be to provide a fast/responsive shared runners to the Salsa Team and tag your CI pipelines to use that runner exclusively [1]? [1] https://docs.gitlab.com/ee/ci/yaml/#tags > Oh, in fact, don't ... Salsa doesn't keep artifacts / logs long enough,= > so you will see nothing. I wonder why the Salsa admins decided on savin= g > them on Google infrastructure then! Or did I misunderstood the way it > works? Hard to tell, because we get almost zero communication about thi= s > from the admins. >=20 >> If what you describe really is caused by the default runners not being= >> fast enough, why couldn't we ask the Salsa team for more powerful ones= ? >> Have you talked to them about this? > > Since when Salsa admins are receptive to critics and/or suggestions? Th= e > last time we told them that using Google wasn't a good idea, they just > ignored it. Do you even know how the default runners were provisioned? > Who's paying? How much credit do we have? How much can we use them? AFAIU, they are provisioned with terraform [2] and then configured with ansible [3]. Generally most of the code used to run and maintain Salsa is in their repository. I don't know who is paying (my guess is Debian+some Google credits), but I feel these are valid questions and I don't see why the Salsa Team would not answer if you ask them. I know they looked for CI runner sponsors for a while and I'm not sure they were able to find enough to meet their requirements. [1] https://salsa.debian.org/salsa/salsa-terraform/blob/master/environments/p= rod/runner.tf [2] https://salsa.debian.org/salsa/salsa-ansible/tree/master/roles/gitlab-run= ner >> It seems to me that spending money in QA like CI runners is very >> profitable for the project, as it saves everyone a lot of time dealing= >> with unnecessary failure caused by lack of tests. It's not like Debian= >> is a very poor organisation... >=20 > Indeed. It'd be even better if we could have our own cloud, but nobody > in power seem receptive to the idea. >=20 > Also, please consider what happened the last time someone added 1000+ C= I > jobs (ie: Salsa went kind of down, and the person who did that got kind= > of flamed by Salsa admins). It's possible to push to Salsa without triggering a CI run with "git push -o ci.skip" or by including "[ci-skip]" in the HEAD commit message. IIUC, the problem isn't the overall amount of repositories using the CI, but adding a 1000 ones at the same time and overloading the runners. > At this point in time, I really don't think Salsa is ready for what you= > proposed, unless we at least: >=20 > 1/ Take a super big care when adding jobs. I feel this is easily resolved by the "-o ci.skip" thing. > 2/ We have our own CI runners. I'm not 100% sure that's a good idea. The Salsa Team has pretty strict requirements about shared runners (they require root on those machines to make sure the .debs created by the runners can be trusted) and I'm happy they do. Running Gitlab CI runners takes a bit of work (I run multiple ones, including a major shared one on 0xacab.org) and if we want people to feel like they can download the artifacts, we need to make sure the CI runners we use are meet the high standards. Also, as I'm sure you are aware of, maintaining stuff takes time and effort and if we can forgo that and let a dedicated team of volunteers do it (the Salsa team), I think it's a win. I really wonder how common the issues you've experienced with the Salsa CI runners are. Has anyone here had similar problems? I'd be fine with 95% of our package using the same default pipeline and the last 5% using something else or disabling it and adding a few comments in d/gitlab-ci.yml explaining why. > As I wrote, I believe my company could provide a few of these runners > (pending my boss approval, who's currently on holidays). However, it'd > be nice if my company wasn't the only sponsor... Not only because of > financial issues, but at least for redundancy. The place I work at sadly can't provide root on VMs to other people, so I can't really help :( FWIW, I've opened an issue on the Salsa Support issue tracker to see what the Salsa team thinks of this whole discussion [3] [3]: https://salsa.debian.org/salsa/support/issues/170 --=20 =E2=A2=80=E2=A3=B4=E2=A0=BE=E2=A0=BB=E2=A2=B6=E2=A3=A6=E2=A0=80 =E2=A3=BE=E2=A0=81=E2=A2=A0=E2=A0=92=E2=A0=80=E2=A3=BF=E2=A1=81 Louis-= Philippe V=C3=A9ronneau =E2=A2=BF=E2=A1=84=E2=A0=98=E2=A0=B7=E2=A0=9A=E2=A0=8B pollo@debian.o= rg / veronneau.org =E2=A0=88=E2=A0=B3=E2=A3=84 --SOuz7aQP0CE78gEWw0XRA8kljDZGyVcGh-- --oZmsPOKSIlNSXm9hHq0E79LLli4jPRnGC Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEZ39U8fqGga2OwLzmeurE7GqqCpcFAl19naQACgkQeurE7Gqq CpdEVBAAlYRf3SEe86mOMcdOeqI2bNPfKRwXkCW5pTHA8KkQ7PvlPnbjLP50WxAx 84UQFaamo5MNKzsvdgzsZqebptNypkMfaHMbyFZW7bWPbuugX3ORZBPdiRkdGlzz VLiYcNUHH2paTdkihL60kuQMzYLhkTsDoDzHUkscUg6nuwZAVy4TkLIv39UOc1+Z NiLQC+SfEviWhyV29KNAUhgwX/N4l9aOU5BpCxfWlEPmKNLKYo6pTRPIrjYjLQxJ qVb4CFiivCrJZ/j9r808k08WXAFgn4ZLSljnGwkTtyKY/GfvWUFW+4KgKfrKu3mY kUeGWKTEj7WbF/MSuvzgkLCs3OUeD1MsFTfzWHs37e2BT3s5/g0tQJSu3PmFcHmj pK2I0tjmA1H3OROuu2NVzigyKZpRNNV5m+vEY8cNRijzxSj3jmlESOjpF12Q4fqi NDVdbQ2czSfoQSw5w9MdzXw2A3mh9kThQoCWOJcKkKzJt5hF+5aeCrkfH15damHC K0EHT+nPS5piRX2MXgBpbFmTJig97tiYk/cyMBZWAwZQbQ4UOCsvM0pa5rEM5kMK PdMyU/WeWwgnRRsCultJR8mYfT+cLXclAInA95wD6N4dKkTYB3WOJfOdvUr5A1lf CHOFQ+4wHLnjvbfGyu1Yj1dBL77yHZ+YvI1I4o9+eMl8mIW75eQ= =y0wJ -----END PGP SIGNATURE----- --oZmsPOKSIlNSXm9hHq0E79LLli4jPRnGC--