Path: csiph.com!aioe.org!bofh.it!news.nic.it!robomod From: =?UTF-8?Q?Louis-Philippe_V=c3=a9ronneau?= Newsgroups: linux.debian.maint.python Subject: Re: Streamlining the use of Salsa CI on team packages Date: Mon, 16 Sep 2019 02:20:04 +0200 Message-ID: References: X-Mailbox-Line: From debian-python-request@lists.debian.org Mon Sep 16 00:18:24 2019 Old-Return-Path: X-Amavis-Spam-Status: No, score=-11.1 tagged_above=-10000 required=5.3 tests=[BAYES_00=-2, BODY_8BITS=1.5, FOURLA=0.1, LDO_WHITELIST=-5, PGPSIGNATURE=-5, RCVD_IN_DNSWL_LOW=-0.7] autolearn=ham autolearn_force=no X-Policyd-Weight: using cached result; rate:hard: -4.6 X-Riseup-User-ID: 51EC99C2E5B26E7489D5E2F5F8451429A9E2E6B23F5CC2C7AADA1F4330A31059 Openpgp: preference=signencrypt Autocrypt: addr=pollo@debian.org; prefer-encrypt=mutual; keydata= mQINBFSX1NABEADH76idoFsToCSwRrCJAqoZMPFWNGdK5+lMpxfVxb3GQYzEmxexkqq2ga/N oUlEs1jKY7F513uDARm2IVsE4ypq3TqYYZtSIr+27NvWUCAleFTBcTRvlo6pdBwaF9bENRgg uoMuHdY8OkNxg1jhwiq0kWpImD1K91NokKiX/9GebCIsr3J5ETafJ+pcHc6N3v8BIC2C8QAu c+fh5Atb9qZglNdFQ9orFkvCDKvaa6ZOymwpAW27JLY1NjSj8S+bLle2XKC3mNq2jtVN0vX0 V9UrNEOHpXokuFLXQFoGZGdqyQBBG3Hj9q0+NAJxMmNbBcOroOIDLyuyVFTWngaoU7m+Tlsm enB0T38ifOV5oefO9B+X8kqi0G3OdhNR/m4NC50q5QXkhis6195f4tdlBZQtipLygAOoeenx XRnxbKNoNA2PUz5FRBzPytsNnmkW80tY0eleZ6lXmO1Kuas3jaQOtBMl2HuBO6wn0QAhFmze qfq5gbexu4iCZYlnUkqCYsiFkrjyPUo+o/bDMTGz7sSrrv9W1Rg4lTiaAiysaPZ3HeSBGTF8 feHXPkwfCZiimFU7ojildMqkq6ZWIAdoMdYi8CGYCxg+xtapXASsTeXJ7Om6R+TBaaV9WNBy WOHhOLCI57bm0dQ8HwqB/rNufYooGXyRDDLMcbLGhu64iPYN1wARAQABtDVMb3Vpcy1QaGls aXBwZSBWw6lyb25uZWF1IChkZWJpYW4pIDxwb2xsb0BkZWJpYW4ub3JnPokCVAQTAQoAPgIb IwULCQgHAwUVCgkICwUWAgMBAAIeAQIXgBYhBGd/VPH6hoGtjsC85nrqxOxqqgqXBQJc9DQN BQkKPZK3AAoJEHrqxOxqqgqXMZoP/RofvJ+bWe5xvCukbuBMrinrs541sZJgD4HtLz5/kFwH 8dH9QYIdWFcT+nwK0L3/zkc/GOgbcIZHssguGT01EBK5URJMoZla0eDfKNM+bAZoe9uO5vDu 7YHLY3a6Ky7mAkqLZRmBv5gRNUub27PSdgpxphTmQIrKwkp3E7sOF98zi4aqB+nIfPlIWRJl b2DZJ32leN94Lls+KRNFZK0LP/+XyFr7PTzWrJYfGYyeTvlBM7suUhluBFU5PuWJXtpaMcZk mgvMKjUA62KQuN86V7YfPSkyqg6dWH3rzfd10+qEE23u0TGnXRT4ynUyP0KiCwJtoqeSur/C S3LXkdrwvzaZqzYjLT45QcddCjIGpbp3bJU/DnpEHB9K7THiPx3Tz8X6QZ7211356mZT3p0f FQ81gIgYpgYggmpxvQzwLOhhW6YNHm9Qc2e+1EHLoGyqCgUR8xDa5jJrV30Z1arBWFzsJkuO Y+TMh1JgTLaNqIpsx+zLlSTM0WUWbvssnliqc9Qul4qlOrD1G1HCqqytKqaA0oE6X/pL28gC YkiNrs9DJRXEB2cBcpbsxS6EwtNfNIH+hjUZM7K7U5+EW+ai4lpTtjARYIeIdgIXJiwZSPXq Pibr1k9PkYhCCjkWn/L2RT4+SOmwgIZvAkTUII0XhLt1789LCUX2njLxcwDNCV8duQINBFSX 1NABEAC28f43K11snkvXvQsDKKyCELMeCzh4CkQ/QvalGns6ALo/SnJ+XjtaTQz1Ubx7T4Be 6Xlp7I069bv/xhV/1GrvbAE6UUJE9VF0iry2AkHr7Ug4pIqVcelMQkSYaMVUKYrGtyxOxvjn zJ4++kvzpo5cqELeywVnSdYo3ApZqDZbirEqLzehj/WyMk2kAQYdhtZDOlPmroQBRH8VvRZd u6IqOjME5T2AlG1D4YGm/tQsQl4mcK40BPGeyOECnKRD2dCfO5BJaEO91B1lYtiXqJd/e1dU Oj2vetvMmIPhYyJJHWOK7dANM/sleaygplVfCbICCM5StvBVV5k4Zu94gg+3N/DU2rOOPnMg eO2MG63qSGR2A8Id2f0/zL6rBrvOQxXRyI62qWPzwHksDxOXCId/OlUcUf5iejNouubRrnNA 8Z4J88wcdQ6sjW0FlaPp2YCeq2z2OLb/iTxgZrMYn6cOhZqDNB3pGh2l/dCxmfOm6dB5dvAL Fv1XjNEJIr6NgcZusMS2YoPDDNv1dIzf3EIEDpLR1zuTsTXhqUfinPrrJsFWLo1/lay2t+1c mobPKJgEzESKG5260Jv6pltc/jM3FSYAUgvaSO2/Rmyt7tfBnreBDPS5HUqQSdyh5/1lbkuZ +7zU+ikFdSa5lQIammrYePDxPgX5rlqSAUghaFFqHQARAQABiQI8BBgBCgAmAhsMFiEEZ39U 8fqGga2OwLzmeurE7GqqCpcFAlz0NA0FCQo9kr0ACgkQeurE7GqqCpfpEQ/9E6V2URLkfTVS mYUp5tznjTVARp5mnkjgYGJD0zJ6je/vzAwocVQmEp7vO7YjUie2mBFF+IrIZDKkYA8e9E0h 2wsYAL0Hs/V8bX0aPwdU4KpyiTVSGDDS6SavF95fkJJtTXAt11NXhhUUpYe9Ad1vhSWW4v/9 bD7XYz61EwH2WHf/cKW3GwinF/HqAG5rGMY3bCPO/FzQvwXl9mDyJ6GTw3SS/2yFTVGThQCN na8MeyFYX3oXbiCUkKBxYHVeyzZRTGLFyTP6SHUEZQ/1fzqw3CpIvEMBeIOg89HpgfSfpFyU 5SA73Ft53rXKZxqhdLf0xuT3aX1NWoM1wjpV4BJO7Xqyg5AkeLgOEF69zdaYdR69tjdI7VAB Rv8c33ue8JB9AKHrE+WnjIE4RmELC6gYn5qOOOUOlIZTlSUptHc8jVxwsby+Di9JQTyJM59E tj5C0wBCzqLZ7U4cPbRgxhgey30Lxn6MGQMEcPhQReqbbGn3VFq5PDjkvwh2Mn7vfsd2r8kp zSvqB285Q/GKjtkrt71rRGmKYqfZs7gsPn3MaI1xOmM+mqiYgUrLhX2M+bAIM+ImP5MPev8V CEs+I2oTwYxgDHvAOPVLCic0EZ4Sg0KvOwiKM3MaXlZVbpwa+U8Z91kQzkl3IvqjkDDVJNzB VNf7CZNxv+Lpm13sI2wPoWc= MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="aELzesorcHOnfjG0kmcl0Af8XE7NwxMzX" X-Mailing-List: archive/latest/17353 List-ID: List-URL: List-Archive: https://lists.debian.org/msgid-search/5c5dcfcf-f9ce-8db5-fd33-6bd14aaa4b48@debian.org Approved: robomod@news.nic.it Lines: 218 Organization: linux.* mail to news gateway Sender: robomod@news.nic.it X-Original-Date: Sun, 15 Sep 2019 20:17:54 -0400 X-Original-Message-ID: <5c5dcfcf-f9ce-8db5-fd33-6bd14aaa4b48@debian.org> X-Original-References: <3429cc8e-87d4-facc-870d-1e664fe0779f@debian.org> <8b08c6e5-96f6-cffc-9c03-00d0fa9a8540@debian.org> <81acc63f-ffe8-ff28-71e2-8219b643f289@debian.org> <8e52d0d6-653b-91a8-9746-a73d1a05a84a@debian.org> Xref: csiph.com linux.debian.maint.python:11818 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --aELzesorcHOnfjG0kmcl0Af8XE7NwxMzX Content-Type: multipart/mixed; boundary="eO2eEBvGwc71LGuqhXRg6CpacNpqy7Sil"; protected-headers="v1" From: =?UTF-8?Q?Louis-Philippe_V=c3=a9ronneau?= To: debian-python@lists.debian.org Message-ID: <5c5dcfcf-f9ce-8db5-fd33-6bd14aaa4b48@debian.org> Subject: Re: Streamlining the use of Salsa CI on team packages References: <3429cc8e-87d4-facc-870d-1e664fe0779f@debian.org> <8b08c6e5-96f6-cffc-9c03-00d0fa9a8540@debian.org> <81acc63f-ffe8-ff28-71e2-8219b643f289@debian.org> <8e52d0d6-653b-91a8-9746-a73d1a05a84a@debian.org> In-Reply-To: <8e52d0d6-653b-91a8-9746-a73d1a05a84a@debian.org> --eO2eEBvGwc71LGuqhXRg6CpacNpqy7Sil Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 19-09-15 18 h 01, Thomas Goirand wrote: > On 9/15/19 4:10 AM, Louis-Philippe V=C3=A9ronneau wrote: >> On 19-09-14 17 h 35, Thomas Goirand wrote: >>> On 9/13/19 11:08 PM, Louis-Philippe V=C3=A9ronneau wrote: >>>> On 19-09-13 05 h 57, Thomas Goirand wrote: >>>>> On 9/5/19 7:40 AM, Louis-Philippe V=C3=A9ronneau wrote: >>>>>> Hello folks! >>>>>> >>>>>> I'd like to propose we start using Salsa CI for all the team packa= ges. I >>>>>> think using a good CI for all our packages will help us find packa= ging >>>>>> bugs and fix errors before uploads :) >>>>> >>>>> I would agree *IF* and only *IF* we find better runners than the on= e >>>>> currently default in Salsa. The GCE runners are horribly slow (they= are >>>>> the smallest to avoid cost). As a result, some tests may just fail >>>>> because of that, and it becomes just frustrating / annoying noise. >>>> >>>> I never experienced such timeouts, but I guess I don't work on very >>>> large packages or things that take more than a few minutes to build.= >>> >>> The issue isn't build time. But when you have unit tests sensitive to= >>> timing. See for example openvswitch: >>> >>> https://salsa.debian.org/openstack-team/third-party/openvswitch/pipel= ines/61713 >> >> Do you have similar issues running those CI tasks in a private runner?= >> (I'm really curious since I haven't had problems and the Salsa runners= >> don't seem slow compared to the private runners I run on my machines).= >=20 > For this particular package, I even had issues with some buildd on some= > slow architectures like older MIPS. Just, with the Salsa default > runners, it's a complete disaster where most of the tests fails, not > just a few, because the runner is too slow. >=20 > What this shows is that we should *not* just blindly add the CI to all > of the team's package. Potentially, this will be a disaster. You may ad= d > the CI script here and there, but I am warning you: adding it to all > packages at once is a recipe for a big disaster. >=20 >> Maybe one solution to your problem would be to provide a fast/responsi= ve >> shared runners to the Salsa Team and tag your CI pipelines to use that= >> runner exclusively [1]? >> >> [1] https://docs.gitlab.com/ee/ci/yaml/#tags >=20 > Yes, that's what I've been telling to begin with. We should try > providing other runners for the team if possible. >=20 >> [1] >> https://salsa.debian.org/salsa/salsa-terraform/blob/master/environment= s/prod/runner.tf >=20 > This tells "instance_type: g1-small", which doesn't match any name at: > https://cloud.google.com/compute/vm-instance-pricing >=20 > Am I right that this is n1-standard-1, which is 1 VCPU and 3.75 GB? >=20 >> It's possible to push to Salsa without triggering a CI run with "git >> push -o ci.skip" or by including "[ci-skip]" in the HEAD commit messag= e. >> >> IIUC, the problem isn't the overall amount of repositories using the C= I, >> but adding a 1000 ones at the same time and overloading the runners. >=20 > Ah, nice, good to know. >=20 >>> 1/ Take a super big care when adding jobs. >> >> I feel this is easily resolved by the "-o ci.skip" thing. >=20 > Good! >=20 >> I'm not 100% sure that's a good idea. The Salsa Team has pretty strict= >> requirements about shared runners (they require root on those machines= >> to make sure the .debs created by the runners can be trusted) and I'm >> happy they do. >=20 > I didn't know, and this makes me question the overall way it works, and= > worries me a lot. ie we should be running on throwaway VMs, rather than= > having a VM we should be able to trust. The way you describe things, I > wonder how easy it should be to get root on these VMs by running a > crafted CI job... Well, runners aren't running the CI jobs directly: everything is ran in Docker, as that's the only available executor on the Salsa shared runners. Even then, it uses a Gitlab special sauce, so it's not even strait up Docker. >> I really wonder how common the issues you've experienced with the Sals= a >> CI runners are. Has anyone here had similar problems? >=20 > Since we're talking about the smallest type of instance possible at > google, then other people may have experience the lack of RAM for sure.= >=20 >> I'd be fine with 95% of our package using the same default pipeline an= d >> the last 5% using something else or disabling it and adding a few >> comments in d/gitlab-ci.yml explaining why. >=20 > The question is: how do you know who's the 5% that needs a better atten= tion? I don't really think a failing CI is a big deal. It's not like this will break any package anyway. The way I see it, we'd push to all the repositories and then let folks working on individual packages disable it if it's causing them trouble, as long as they document why they disabled it. Since the CI won't be ran on the first push, we won't get an avalanche of mails saying the CI failed for X Y Z reason. >> FWIW, I've opened an issue on the Salsa Support issue tracker to see >> what the Salsa team thinks of this whole discussion [3] >> >> [3]: https://salsa.debian.org/salsa/support/issues/170 >=20 > Thanks a lot for doing this, taking the time to communicate with the > Salsa people, etc. >=20 > I'm all for more CI, so feel free to ignore my remarks and go ahead, my= > intention was just bring your attention to things I've seen. If it work= s > well, then fantastic! :) The Salsa folks are asking us to wait a little before implementing such a change, as it seems they haven't yet decided what the infrastructure needs are to support a systematic use of Gitlab CI on Salsa. I feel that's a reasonable request. I was kindly pointed to this discussion on -devel by the Salsa admins: [1][2][3][4]. I think we should still try to agree on a draft modification to the policies. I'll send a recap mail of the discussions we've had to try to move forward with this. [1]: https://lists.debian.org/debian-devel/2019/09/msg00217.html [2]: https://lists.debian.org/debian-devel/2019/09/msg00220.html [3]: https://lists.debian.org/debian-devel/2019/09/msg00270.html [4]: https://lists.debian.org/debian-devel/2019/09/msg00241.html --=20 =E2=A2=80=E2=A3=B4=E2=A0=BE=E2=A0=BB=E2=A2=B6=E2=A3=A6=E2=A0=80 =E2=A3=BE=E2=A0=81=E2=A2=A0=E2=A0=92=E2=A0=80=E2=A3=BF=E2=A1=81 Louis-= Philippe V=C3=A9ronneau =E2=A2=BF=E2=A1=84=E2=A0=98=E2=A0=B7=E2=A0=9A=E2=A0=8B pollo@debian.o= rg / veronneau.org =E2=A0=88=E2=A0=B3=E2=A3=84 --eO2eEBvGwc71LGuqhXRg6CpacNpqy7Sil-- --aELzesorcHOnfjG0kmcl0Af8XE7NwxMzX Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEZ39U8fqGga2OwLzmeurE7GqqCpcFAl1+1LIACgkQeurE7Gqq Cpevig//Xl+jCLviQnT7Nr1sGk8PeqoVEvjdYZoatvAm24S93R82lKywx4OW7hVh Biw37h583W19buAN61JMd/lVBsYPYlaCBrcxZHtpUFihCSdJV1MYGwqsqCZ+LcVL dnirI2QNIKMPdGH+SxXbegoGh0RdZnkfYgnriEvkASbjBkmu2txTdmD+Uxer66jH GwZtLmNPD4tcDVYV/tPg+F245xDn6i9G4fZd6geg90DymSs0dFbtoziKjJsb/9e4 UXAXfNf0THrRi0wIZZfxhhAcjgqigvFayT6YDuTMOy4zf64QwdDM4YadvD8aQnYS OAYxgIYE9tNfA0ItWYO2K6U2i9nxrbihfhHgedERknjjRtzdmBFapSnox8aVZ1Zr UwClrasC0kSYmROej4y7uXzfTiVF5B82ZeHXwSzpp0zrKP2MOqFbbz/sVKTF7y4B Zi992D/0XNlN7A0QmoPa5kW1kq7QLY3Ox9GU3juIYs6GJBFWtpNQg0EwIIQKMwdH UoDLWENniXUTC9CaPvFb29xY8li623fIVxw8gptkD4iSOwuuMZUEp/LosbD8C/pg oWi51GzHv9pNCuG9mPisJLcwvoup6zvBudDDDFgs+480nCpoXepSwJCggqLLFVuO 0rX3XKGHehXa04z34S08nUaX/Y1NRkjVCD3gwZKPXLj/kzZYE5g= =0j9u -----END PGP SIGNATURE----- --aELzesorcHOnfjG0kmcl0Af8XE7NwxMzX--