Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.devel > #112048 > unrolled thread

Mandatory LC_ALL=C.UTF-8 during package building

Back to article view | Back to linux.debian.devel

Contents

  Mandatory LC_ALL=C.UTF-8 during package building Gioele Barabucci <gioele@svario.it> - 2024-06-06 08:20 +0200
    Re: Mandatory LC_ALL=C.UTF-8 during package building Luca Boccassi <bluca@debian.org> - 2024-06-06 11:20 +0200
    Re: Mandatory LC_ALL=C.UTF-8 during package building Simon Richter <sjr@debian.org> - 2024-06-06 11:50 +0200
      Re: Mandatory LC_ALL=C.UTF-8 during package building Johannes Schauer Marin Rodrigues <josch@debian.org> - 2024-06-06 12:10 +0200
        Re: Mandatory LC_ALL=C.UTF-8 during package building Johannes Schauer Marin Rodrigues <josch@debian.org> - 2024-06-06 13:00 +0200
          Re: Mandatory LC_ALL=C.UTF-8 during package building Simon Richter <sjr@debian.org> - 2024-06-06 13:40 +0200
          Re: Mandatory LC_ALL=C.UTF-8 during package building Simon McVittie <smcv@debian.org> - 2024-06-06 17:00 +0200
        Re: Mandatory LC_ALL=C.UTF-8 during package building Simon McVittie <smcv@debian.org> - 2024-06-06 16:40 +0200
          Re: Mandatory LC_ALL=C.UTF-8 during package building Jeremy Bícha <jeremy.bicha@canonical.com> - 2024-06-06 16:50 +0200
          Re: Mandatory LC_ALL=C.UTF-8 during package building Guillem Jover <guillem@debian.org> - 2024-06-07 14:40 +0200
            Re: Mandatory LC_ALL=C.UTF-8 during package building Holger Levsen <holger@layer-acht.org> - 2024-06-07 15:40 +0200
              Re: Mandatory LC_ALL=C.UTF-8 during package building Alexandre Detiste <alexandre.detiste@gmail.com> - 2024-06-07 15:50 +0200
                Re: Mandatory LC_ALL=C.UTF-8 during package building Simon McVittie <smcv@debian.org> - 2024-06-07 17:50 +0200
                  Re: Mandatory LC_ALL=C.UTF-8 during package building Simon McVittie <smcv@debian.org> - 2024-06-10 17:30 +0200
                    Re: Mandatory LC_ALL=C.UTF-8 during package building Simon Richter <sjr@debian.org> - 2024-06-11 11:30 +0200
                Re: Mandatory LC_ALL=C.UTF-8 during package building Guillem Jover <guillem@debian.org> - 2024-07-02 03:50 +0200
                  Re: Mandatory LC_ALL=C.UTF-8 during package building Simon McVittie <smcv@debian.org> - 2024-07-02 11:00 +0200
                    Re: Mandatory LC_ALL=C.UTF-8 during package building Guillem Jover <guillem@debian.org> - 2024-07-02 14:40 +0200
                      Re: Mandatory LC_ALL=C.UTF-8 during package building Alexandre Detiste <alexandre.detiste@gmail.com> - 2024-07-06 13:20 +0200
                        Re: Mandatory LC_ALL=C.UTF-8 during package building Guillem Jover <guillem@debian.org> - 2024-08-03 19:20 +0200
                          Re: Mandatory LC_ALL=C.UTF-8 during package building Simon McVittie <smcv@debian.org> - 2024-08-03 20:20 +0200
                          Re: Mandatory LC_ALL=C.UTF-8 during package building Chris Hofstaedtler <zeha@debian.org> - 2025-02-23 13:50 +0100
            Re: Mandatory LC_ALL=C.UTF-8 during package building Gioele Barabucci <gioele@svario.it> - 2024-06-07 16:20 +0200
            Re: Mandatory LC_ALL=C.UTF-8 during package building Simon McVittie <smcv@debian.org> - 2024-06-07 18:30 +0200
              Re: Mandatory LC_ALL=C.UTF-8 during package building Guillem Jover <guillem@debian.org> - 2024-08-03 15:30 +0200
        Re: Mandatory LC_ALL=C.UTF-8 during package building Marco d'Itri <md@Linux.IT> - 2024-06-06 18:50 +0200
        Re: Mandatory LC_ALL=C.UTF-8 during package building Andrey Rakhmatullin <wrar@debian.org> - 2024-06-06 19:00 +0200
          Re: Mandatory LC_ALL=C.UTF-8 during package building Holger Levsen <holger@layer-acht.org> - 2024-06-07 10:40 +0200
      Locale sanitizer (Was: Mandatory LC_ALL=C.UTF-8 during package  building) Daniel Gröber <dxld@darkboxed.org> - 2024-06-06 14:50 +0200
        Re: Locale sanitizer (Was: Mandatory LC_ALL=C.UTF-8 during package  building) Simon McVittie <smcv@debian.org> - 2024-06-06 15:40 +0200
    Re: Mandatory LC_ALL=C.UTF-8 during package building Adrian Bunk <bunk@debian.org> - 2024-06-18 00:40 +0200

Page 1 of 2  [1] 2  Next page →

#112048 — Mandatory LC_ALL=C.UTF-8 during package building

Hi,

setting LC_ALL=C.UTF-8 in d/rules is a common way to fix many 
reproducibility problems. It is also, in general, a more sane way to 
build packages, in comparison to using whatever locale settings happen 
to be set during a build. However, sprinkling a variant of `export 
LC_ALL=C.UTF-8` in every d/rules is error-prone and a waste of 
maintainers' time.

Would it be possible to set in stone that packages are supposed to 
always be built in an environment where LC_ALL=C.UTF-8, or, in other 
words, that builders must set LC_ALL=C.UTF-8?

In which document should this rule be stated? Policy?

Regards,

-- 
Gioele Barabucci

[toc] | [next] | [standalone]

#112052

On Thu, 6 Jun 2024 at 09:07, Gioele Barabucci <gioele@svario.it> wrote:
>
> Hi,
>
> setting LC_ALL=C.UTF-8 in d/rules is a common way to fix many
> reproducibility problems. It is also, in general, a more sane way to
> build packages, in comparison to using whatever locale settings happen
> to be set during a build. However, sprinkling a variant of `export
> LC_ALL=C.UTF-8` in every d/rules is error-prone and a waste of
> maintainers' time.
>
> Would it be possible to set in stone that packages are supposed to
> always be built in an environment where LC_ALL=C.UTF-8, or, in other
> words, that builders must set LC_ALL=C.UTF-8?
>
> In which document should this rule be stated? Policy?

This makes sense to me, seems similar enough to SOURCE_DATE_EPOCH

[toc] | [prev] | [next] | [standalone]

#112053

Hi,

> Would it be possible to set in stone that packages are supposed to
> always be built in an environment where LC_ALL=C.UTF-8, or, in other
> words, that builders must set LC_ALL=C.UTF-8?

This would be the opposite of the current rule.

Setting LC_ALL=C in debian/rules is an one-liner.

If your package is not reproducible without it, then your package is
broken. It can go in with the workaround, but the underlying problem
should be fixed at some point.

The reproducible builds checker explicitly tests different locales to
ensure reproducibility. Adding this requirement would require disabling
this check, and thus hide an entire class of bugs from detection.

   Simon

[toc] | [prev] | [next] | [standalone]

#112054

[Multipart message — attachments visible in raw view] — view raw

Hi,

Quoting Simon Richter (2024-06-06 11:32:33)
> > Would it be possible to set in stone that packages are supposed to always
> > be built in an environment where LC_ALL=C.UTF-8, or, in other words, that
> > builders must set LC_ALL=C.UTF-8?
> 
> This would be the opposite of the current rule.
> 
> Setting LC_ALL=C in debian/rules is an one-liner.
> 
> If your package is not reproducible without it, then your package is
> broken. It can go in with the workaround, but the underlying problem
> should be fixed at some point.
> 
> The reproducible builds checker explicitly tests different locales to
> ensure reproducibility. Adding this requirement would require disabling this
> check, and thus hide an entire class of bugs from detection.

this is one facet of a much bigger discussion (which we've had before). You can
argue both ways, depending on how you look at this problem.

It is the question of whether we want to:

 a) debian/rules is supposed to be runnable in a wide variety of environments.
 If your package FTBFS in a one specific environment, it is the job of d/rules
 to normalize the environment to cater for the specific needs of the package.

 b) debian/rules is supposed to be run in a well-defined environment. If your
 package FTBFS in this normalized environment, then it is the job of d/rules to
 add the specific needs of the package to d/rules.

So the question is whether you either want to have d/rules normalize
heterogeneous environments (a) or whether you want d/rules to make a normalized
environment specific to the build (b). This is of course a spectrum and I think
we currently doing much more of (a).

A question that goes in a similar direction is whether every d/rules that needs
it should have to do this:

export DPKG_EXPORT_BUILDFLAGS=y
include /usr/share/dpkg/buildflags.mk

Or whether we should switch the default and require that d/rules is run in an
environment (for example as set-up by dpkg-buildpackage) where these variables
are set?

Going back to the example of LC_ALL=C.UTF-8 and reproducibility: whether or not
this "hides" problem depends on the definition of what things are allowed to
change between two builds and what constitutes these things has changed already
in the past, for example for the build path which is not *not* changed anymore
but instead recorded in the buildinfo. The same could be argued for
LC_ALL=C.UTF-8 and the environment variables already are part of the buildinfo.

So I do not think that there is an easy answer to this question.

Thanks!

cheers, josch

[toc] | [prev] | [next] | [standalone]

#112056

[Multipart message — attachments visible in raw view] — view raw

Hi,

Quoting Hakan Bayındır (2024-06-06 12:32:27)
> On 6.06.2024 ÖS 1:08, Johannes Schauer Marin Rodrigues wrote:
> > Quoting Simon Richter (2024-06-06 11:32:33)
> >>> Would it be possible to set in stone that packages are supposed to always
> >>> be built in an environment where LC_ALL=C.UTF-8, or, in other words, that
> >>> builders must set LC_ALL=C.UTF-8?
> >>
> >> This would be the opposite of the current rule.
> >>
> >> Setting LC_ALL=C in debian/rules is an one-liner.
> >>
> >> If your package is not reproducible without it, then your package is
> >> broken. It can go in with the workaround, but the underlying problem
> >> should be fixed at some point.
> >>
> >> The reproducible builds checker explicitly tests different locales to
> >> ensure reproducibility. Adding this requirement would require disabling this
> >> check, and thus hide an entire class of bugs from detection.
> > 
> > this is one facet of a much bigger discussion (which we've had before). You can
> > argue both ways, depending on how you look at this problem.
> > 
> > It is the question of whether we want to:
> > 
> >   a) debian/rules is supposed to be runnable in a wide variety of environments.
> >   If your package FTBFS in a one specific environment, it is the job of d/rules
> >   to normalize the environment to cater for the specific needs of the package.
> > 
> >   b) debian/rules is supposed to be run in a well-defined environment. If your
> >   package FTBFS in this normalized environment, then it is the job of d/rules to
> >   add the specific needs of the package to d/rules.
> > 
> > So the question is whether you either want to have d/rules normalize
> > heterogeneous environments (a) or whether you want d/rules to make a normalized
> > environment specific to the build (b). This is of course a spectrum and I think
> > we currently doing much more of (a).
> 
> I agree with Simon here.

And, if I understand your reply correctly, you do not disagree with me either?

> C, or C.UTF-8 is not a universal locale which > works for all.

Yes. If we imagine a hypothetical switch to LC_ALL=C.UTF-8 for all source
packages by default, then there will be bugs. The question is, which bugs do we
want to fix: Bugs that happen because of a problem that occurs because we did
*not* set LC_ALL=C.UTF-8 (like reproducible builds problems) or problems that
occur because we *did* set LC_ALL=C.UTF-8 as in the example that you are
describing below.

> While C.UTF-8 solves character representation part of 
> "The Turkish Test" [0], it doesn't solve capitalization and sorting  issues.
> 
> In short, Turkish is the reason why some English text has "İ" and "ı" in 
> it, because in Turkish, they're all present (ı, i, I,  İ), and their 
> capitalization rules are different (i becomes İ and ı  becomes I; i.e. 
> no loss/gain of dot during case changes).
> 
> This  creates tons of problems with software which are not aware of the 
> issue  (Kodi completely breaks for example, and some software needs 
> forced/custom environments to run).

As I'm curious: if your software breaks depending on the LC_ALL setting, how do
you make it produce reproducible binaries? If it breaks with a certain LC_ALL,
then during the build you have to set LC_ALL (or one of its friends) to some
specific value, right?

> So, all in all, if your software is expected to run in an international 
> environment, and its build/run behavior breaks in an environment is not 
> to its liking, I also argue that the software is broken to begin with. 
> Because when this problem takes hold in a codebase, it is nigh 
> impossible to fix.
> 
> So, I think it's better to strive to evolve the software to be a better 
> international citizen rather than give all the software we build an 
> artificially sterile environment, which is iteratively harder and harder 
> to build and maintain.

Just to make sure I'm not misunderstood: I also am tending towards *not*
setting LC_ALL=C.UTF-8 (but probably not as strongly as I understood Simon's
mail) just because I like dumping my time into figuring out why my software
does something different in a very specific environment. Figuring this out
does uncover bugs that should be fixed most of the time.

At the same time though, I also get annoyed of copy-pasting d/rules snippets
from one of my packages to the next instead of making use of a few more
defaults in our package build environment.

Thanks!

cheers, josch

[toc] | [prev] | [next] | [standalone]

#112057

Hi,

On 6/6/24 19:56, Johannes Schauer Marin Rodrigues wrote:

> At the same time though, I also get annoyed of copy-pasting d/rules snippets
> from one of my packages to the next instead of making use of a few more
> defaults in our package build environment.

Same here -- I just think that such a workaround should be applied only 
when the package fails to build reproducibly, so this is definitely 
something that should not be cargo-culted in.

What we could also do (but that would be a bigger change) would be 
another flag similar to "Rules-Requires-Root" that lists aspects of the 
package that are known to affect reproducibility -- that would be 
declarative, so the reproducible-builds project can disable the test and 
get meaningful results for the remaining typical problems, and could be 
checked and handled by dpkg-buildpackage as well.

    Simon

[toc] | [prev] | [next] | [standalone]

#112068

On Thu, 06 Jun 2024 at 12:56:10 +0200, Johannes Schauer Marin Rodrigues wrote:
> If we imagine a hypothetical switch to LC_ALL=C.UTF-8 for all source
> packages by default, then there will be bugs.

Do you mean: there will be bugs that break the build of certain packages,
which previously built successfully?

Or do you mean: there will be bugs in which a package does not work as
designed at runtime for users of certain locales, and those bugs would
previously have been detected at build-time by showing up as a FTBFS or
non-reproducibility, but are now only detected by users at runtime?

I'm not convinced that either of those is going to be true, and especially
the first one, because at least some (maybe all) of our official buildds
already export LC_ALL=C.UTF-8 for builds:
https://buildd.debian.org/status/fetch.php?pkg=flatpak&arch=amd64&ver=1.14.8-1&stamp=1714492944&raw=0

(Search for "Sufficient free space" and read down a few lines further;
and this is not at all specific to Flatpak, that's just an arbitrary
example of a package that I happen to know has a recent buildd log.)

> I like dumping my time into figuring out why my software
> does something different in a very specific environment

That is of course fine, and you're welcome to do that, but the question
is in part about whether the benefit of expecting that every package
maintainer will do this exceeds its cost.

    smcv

[toc] | [prev] | [next] | [standalone]

#112065

On Thu, 06 Jun 2024 at 13:32:27 +0300, Hakan Bayındır wrote:
> C, or C.UTF-8 is not a universal locale which works
> for all.

Sure, and I don't think anyone is arguing that you or anyone else should
set the locale for your interactive terminal session, your GUI desktop
environment, or even your servers to C.UTF-8.

But, this thread is about build environments for our packages, not about
runtime environments. We have two-and-a-half possible policies:

1. Status quo, in theory:

   Packages cannot make any assumptions about build-time locales.

   The benefits are:

   - Diagnostic messages are in the maintainer's local language, and
     potentially easier to understand.

   - If a mass-QA effort wants to assess whether the program is broken by
     a particular locale, they can easily try running its build-time tests
     in that locale, **if** the tests do not already force a different
     locale. (But this comes with some serious limitations: it's likely
     to have a significant number of false-positive situations where the
     program is actually working perfectly but the **tests** make assumptions
     that are not true in all locales, and as a result many upstream
     projects set their build-time tests to force specific locales
     anyway - often C, en_US.UTF-8 or C.UTF-8 - regardless of what we
     might prefer in Debian.)

   The costs are:

   - Every program that might be run at build-time is expected to continue
     to cope with running in non-UTF-8 locales, even if we strongly deprecate
     non-UTF-8 locales for production use.

   - Diagnostic messages from the reproducible-builds infrastructure are
     in a random language chosen by the infrastructure, which the maintainer
     does not necessarily understand. (If my package fails to build in a
     Chinese locale, that's a valid bug, but if I'm expected to diagnose the
     problem by reading Chinese error messages, as a non-Chinese-speaker I
     am not going to get far.)

   - If a program that is run during build intentionally has locale-specific
     output, and its output ends up in the .deb, then the package maintainer
     must go to additional effort to force that particular program to have
     reproducible output, usually by running it in a specified locale.

2. What's being proposed in this thread:

   Each package can assume that it's built in the C.UTF-8 locale.
   If it needs a different locale during testing, it can set that itself
   (as e.g. glib2.0 does for some tests), but unless it takes explicit
   action, C.UTF-8 will be used.

   The benefit is that packages that require a UTF-8 locale during build
   or during testing (e.g. to process non-English strings in Python)
   can assume that they have one, and an equivalence class of bugs
   (packages where the content of the .deb can vary with the build-time
   locale, or where e.g. build-time tests fail if UTF-8 output is not
   possible) become non-bugs that we do not need to think about.

   The costs are that we don't get the benefits from (1.) any more.

2½. Unwelcome compromise (increasingly the status quo):

   Whenever a package is non-reproducible, fails to build or fails tests
   in certain locales (for example legacy non-UTF-8 locales like C or
   en_GB.ISO-8859-15), we add `export LC_ALL=C.UTF-8` to debian/rules and
   move on.

   This is just (2.) with extra steps, and has the same benefit and cost
   for the affected packages as (2.) plus an additional cost (someone must
   identify that the package is in this category and copy/paste the extra
   line), and the same benefit and costs for unmodified packages as (1.).

2½ seems like the same boil-the-ocean pattern as any number of
manual-work-intensive transitions: Rules-Requires-Root, debhelper compat
levels, compiler hardening flags and so on. In situations where the
desired state is a backwards-compatibility break, the benefit of having
the transition be opt-in can exceed its (considerable!) cost, but we
shouldn't let that trick us into always paying the additional cost of an
opt-in transition, even in situations where it isn't worth it.

> [Turkish dotted/dotless i]
> creates tons of problems with software which are not aware of the
> issue  (Kodi completely breaks for example, and some software needs
> forced/custom environments to run).

I agree that internationalization issues can be a serious problem **at
runtime**, and when our developers and users find such problems, they can
be reported as bugs downstream or upstream, and (hopefully!) fixed. What
I do not agree with is your suggestion that having the package build
occur in an undefined locale will solve this problem.

For example, let's imagine that we decide that perfect support for Turkish
is a release goal. Having reproducible-builds.org build packages in an
arbitrary language (in practice French is often used, I think?) doesn't
prove anything about whether they handle Turkish correctly, whatever
"correctly" might mean.

If someone wants to do a QA mass-rebuild in the tr_TR.UTF-8 locale,
that would come a little closer to having higher confidence about our
ability to run software in Turkish - but is it working *correctly*, or
are the tests making the wrong assertions, or are the code paths that
could go wrong in Turkish not even being tested? We probably won't know
any of those until a Turkish speaker investigates that specific piece
of software.

The fact that you say "Kodi completely breaks" also suggests to me that
fixing these problems is not trivial, because if it was easy, it would
have been fixed by now. And yet we ship Kodi in Debian, even knowing
that it has this bug, and it seems to work OK for most people.

Even if Kodi's problems with Turkish text are solved, **and** the
developer who solves those problems adds a build-time regression test
to avoid the bug coming back, I would expect the test to need to look
like this pseudocode:

    def test_turkish:
        old_locale = setlocale(LC_ALL, "tr_TR.UTF-8")

        if old_locale is null:
            skipTest("tr_TR.UTF-8 locale not available, try installing locales-all")

        try:
            do some stuff involving Turkish text
            assert that the right thing happens
        finally:
            setlocale(LC_ALL, old_locale)

... for which having the rest of the build happen in the tr_TR.UTF-8
locale isn't even useful!

(src:glib2.0 has several tests like this, and the packaging goes to some
lengths to make sure that the required locales are available.)

A wider point here is that artificially elevating a certain class of bugs
to be de-facto release-critical by turning them into build failures is
not necessarily always going to improve the quality of Debian: we have
no shortage of bugs to work on, and a finite amount of volunteer time
available. Any time we make a class of bugs release-critical like this,
that's taking volunteer time away from identifying and fixing different
bugs that might have a larger impact on the overall quality of the
distribution, so we should only do this if we are sure that that class
of bugs is genuinely among our highest priorities.

Stepping back from the specifics of locales, I observe that operating
systems are extremely complicated and contain an overwhelming number
of choices and code paths. Obviously most of those choices are there
because someone needs them - but some are only there for historical
reasons or as an unintended side-effect of something more beneficial. If
we can make a simplifying assumption that will take an entire equivalence
class of bugs and make them into non-bugs, without losing significant
functionality or flexibility, then it's often good to do that instead.

(For example, a while ago we replaced "it is undefined whether /usr is
mounted or not during early boot" with the simplifying assumption "if
/usr is separate then it must be mounted by the initramfs", which turned a
whole class of bugs of the form "x is in /lib but depends on y which is in
/usr/lib" into non-bugs that do not need to be fixed or even identified.)

    smcv

[toc] | [prev] | [next] | [standalone]

#112066

I believe debhelper already sets LC_ALL=C.UTF-8 for the cmake, meson,
and ninja buildsystems; therefore many but definitely not all packages
are already built with LC_ALL=C.UTF-8.

Thank you,
Jeremy Bícha

[toc] | [prev] | [next] | [standalone]

#112088

Hi!

On Thu, 2024-06-06 at 15:31:55 +0100, Simon McVittie wrote:
> On Thu, 06 Jun 2024 at 13:32:27 +0300, Hakan Bayındır wrote:
> > C, or C.UTF-8 is not a universal locale which works
> > for all.
> 
> Sure, and I don't think anyone is arguing that you or anyone else should
> set the locale for your interactive terminal session, your GUI desktop
> environment, or even your servers to C.UTF-8.
> 
> But, this thread is about build environments for our packages, not about
> runtime environments. We have two-and-a-half possible policies:

> 1. Status quo, in theory:
> 
>    Packages cannot make any assumptions about build-time locales.
> 
>    The benefits are:
> 
>    - Diagnostic messages are in the maintainer's local language, and
>      potentially easier to understand.

I think this is way more important than the relative space used to
mention it though. :) I'm a non-native speaker, who has been involved
in l10n for a long time, while at the same time I've pretty much
always run my systems with either LANG=C.UTF-8 or before that LANG=C,
LC_CTYPE=ca_ES.UTF-8 and LC_COLLATE=ca_ES.UTF-8.

And I think forcing a locale on buildds makes perfect sense, because
we want easy access to build logs. But forcing LC_ALL from the build
tools implies that no tool invoked will get translated messages at
all, and means that users (not just maintainers) might have a harder
time understanding what's going on, we make lots of l10n work rather
pointless, and if no one is running with different locales then l10n
bugs might easily creep in.

>    - If a mass-QA effort wants to assess whether the program is broken by
>      a particular locale, they can easily try running its build-time tests
>      in that locale, **if** the tests do not already force a different
>      locale. (But this comes with some serious limitations: it's likely
>      to have a significant number of false-positive situations where the
>      program is actually working perfectly but the **tests** make assumptions
>      that are not true in all locales, and as a result many upstream
>      projects set their build-time tests to force specific locales
>      anyway - often C, en_US.UTF-8 or C.UTF-8 - regardless of what we
>      might prefer in Debian.)

I consider locale sensitive misbehavior as a category of "upstream"
bugs (be that in the package upstream or the native Debian tools), that
deserve to be spotted and fixed. I can understand though the sentiment
of wanting to shrug this problem category off and wanting instead to
sweep it under the carpet, but that has accessibility consequences.

>   The costs are:

>   - […] but if I'm expected to diagnose the
>     problem by reading Chinese error messages, as a non-Chinese-speaker I
>     am not going to get far.)

Just as an aside, but while getting non-English messages makes for
harder to diagnose bugs, I've never found it a big deal to deal with
that kind of bug reports, as you can grep for (parts of) the
translated message, and then get the original English string from the
.po for example, or can translate the text back to know what it is
talking about, or ask the reported to translate it for you.

> 2½. Unwelcome compromise (increasingly the status quo):
> 
>    Whenever a package is non-reproducible, fails to build or fails tests
>    in certain locales (for example legacy non-UTF-8 locales like C or
>    en_GB.ISO-8859-15), we add `export LC_ALL=C.UTF-8` to debian/rules and
>    move on.
> 
>    This is just (2.) with extra steps, and has the same benefit and cost
>    for the affected packages as (2.) plus an additional cost (someone must
>    identify that the package is in this category and copy/paste the extra
>    line), and the same benefit and costs for unmodified packages as (1.).

I agree though, that if we end up with every debian/rules
unconditionally exporting LC_ALL, then there's not much point in not
making the build driver do it instead.


Related to this, dpkg-buildpackage 1.20.0 gained a --sanitize-env,
which for now on Debian and derivatives sets LC_COLLATE=C.UTF-8 and
umask=0022.

But _iff_ we end up with dpkg-buildpackage being declared the only
supported entry point, _and_ there is consensus that we'd want to set
some kind of locale variable from the build driver, then I guess this
could be done as a Debian vendor-specific thing, or via the
dpkg-build-api(7) interface.

Thanks,
Guillem

[toc] | [prev] | [next] | [standalone]

#112090

[Multipart message — attachments visible in raw view] — view raw

On Fri, Jun 07, 2024 at 02:32:14PM +0200, Guillem Jover wrote:
> And I think forcing a locale on buildds makes perfect sense, because
> we want easy access to build logs. But forcing LC_ALL from the build
> tools implies that no tool invoked will get translated messages at
> all, and means that users (not just maintainers) might have a harder
> time understanding what's going on, we make lots of l10n work rather
> pointless, and if no one is running with different locales then l10n
> bugs might easily creep in.

absolutly agreed & thanks for bringing up this aspect!
 
> Related to this, dpkg-buildpackage 1.20.0 gained a --sanitize-env,
> which for now on Debian and derivatives sets LC_COLLATE=C.UTF-8 and
> umask=0022.

that's great news!


-- 
cheers,
	Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

The past is over.

[toc] | [prev] | [next] | [standalone]

#112091

Maybe a compromise would be to at least mandate some UTF-8 locale.

[toc] | [prev] | [next] | [standalone]

#112094

On Fri, 07 Jun 2024 at 23:22:46 +0900, Simon Richter wrote:
> On 6/7/24 22:40, Alexandre Detiste wrote:
> > Maybe a compromise would be to at least mandate some UTF-8 locale.
> 
> Having an UTF-8 locale available would be a good thing, but allowing
> packages to rely on the active locale to be UTF-8 based reduces our testing
> scope.

I'm not sure I follow. Are you suggesting that we should build each
package *n* times (in a UTF-8 locale, in a legacy locale, in locales
known to have unique quirks like Turkish and Japanese, ...), just for
its side-effect of *potentially* passing through those locales to the
upstream test suite?

If we want to run the test suite in each of those locales, then I think
instead we should do just that: run the test suite (and only the test
suite!) in each of those locales. dh_auto_test could grow a way to do
that, if there's demand. Repeating the whole compilation seems like a
sufficiently large waste of time and resources that, in practice, we
are not going to be able to do this routinely for more than a couple
of locales.

Or, better, we should provide packages with a way to guarantee that
certain locales are available[1], and then tests that are known to be
testing locale-sensitive things should explicitly switch into the locales
of interest, to make sure that they are tested every time, not just if
the builder's locale happens to be the interesting one. For example,
glib2.0's test suite temporarily switches to a Japanese locale in order to
test its handling of formatting dates with an era (Japanese is one of the
few locales where that concept exists), and it does this even when built
by a non-Japanese-speaking developer like me. If it relied on the current
locale for its test coverage, then we would never have discovered #1060735
unless it was coincidentally built by a Japanese developer who is using
a big-endian machine, which seems quite unlikely to happen by chance!

Or, when you say "testing", do you really mean "doing the build, for
the side-effect of seeing whether it succeeds or fails"? (That's not
really the same thing as running a test suite.)

Realistically, several important tools require a UTF-8 locale and will
not work reliably otherwise. Meson either is one of these, or was in
the past, as a result of Python's Unicode behaviour; so debhelper sets
LC_ALL=C.UTF-8 when it invokes Meson, ignoring any preference that might
have been expressed by the caller of dpkg-buildpackage.

[1] Build-Depends: locales-all does this, but is rather heavy.
    debian/tests/run-with-locales in e.g. src:glib2.0 is another
    implementation, but a more centralized version of this would probably
    be better.

> Basically, we need to define the severity of locale bugs

More than that, we need to define what is a locale bug and what is a
non-bug - ideally based on what is genuinely useful, rather than on
"this is something that could theoretically work". We should try to
solve bugs, because that benefits our users and Free Software, but we
should put zero effort into solving non-bugs.

What we say is a bug, and what we say is not a bug, is a policy decision
about our scope: we support some things and we do not support others.
There's nothing magical or set-in-stone about the set of things that we
do and don't support, and it can be varied if there is something close to
consensus that it ought to be. When we're deciding what is in-scope and
what is out-of-scope, we should make that decision based on comparing the
costs and benefits of a wider/narrower scope: "this is in-scope because
I say so" or "this is in-scope because we have traditionally said it is"
are considerably weaker arguments than "this is in-scope because otherwise
we can't access this benefit".

As an analogy: we have chosen to define in Policy that /bin/sh is anything
that supports the POSIX shell language, plus a few designated extensions
like `echo -n`. A consequence of that is that "foobar fails to build when
/bin/sh is bash" is considered to be a bug (which, in an ideal world,
we would solve), because bash is a POSIX shell; but "foobar fails to
build when /bin/sh is csh" is a non-bug (which we wouldn't even leave
open as wontfix, we would just close it), because csh isn't a POSIX shell.

In a different parallel universe, we might reasonably have declared
that /bin/sh is required to be bash (like it is in e.g. Fedora), which
would result in some things that are currently bugs becoming non-bugs -
that's a narrower scope than then one that Debian-in-this-universe has,
resulting in it being easier to maintain but less flexible.

Or, conversely, in a different parallel universe, we might have said that
/bin/sh can be literally any POSIX shell, which is a wider scope than
Debian-in-this-universe: "FTBFS when /bin/sh doesn't support echo -n"
is currently a non-bug, but in that hypothetical distribution it would
be a bug, making the distribution harder to maintain but more flexible.

I am, personally, a fan of setting a scope that makes some of our more
obscure or theoretical bugs into non-bugs, because that would let us
concentrate our attention on the remaining bugs (the ones that are more
likely to indicate a genuine problem for our users).

What Giole proposed at the beginning of this thread can be rephrased as
declaring that "FTBFS when locale is not C.UTF-8" and "non-reproducible
when locale is varied" are non-bugs, and therefore they are not only
wontfix, but they should be closed altogether as being out-of-scope.
Of course, if we chose to have this be our policy, then it would be best
if dpkg-buildpackage and/or debhelper would force the C.UTF-8 locale, so
that builds with different locales simply can't happen - instead of
allowing the build to continue, but considering it to be not-a-bug for
it to fail or give different results. Fortunately, forcing a C.UTF-8
locale is very easy (set some environment variables before forking each
subprocess).

Or, Alexandre's "weaker" suggestion, to which you are replying, could
be rephrased as declaring that things like "FTBFS when locale is not
UTF-8" and "non-reproducible when one of the two builds is non-UTF-8" are
non-bugs, but "FTBFS when locale is ja_JP.UTF-8" and "non-reproducible
when the two builds are different UTF-8 locales" would still be bugs
under Alexandre's suggestion. Similarly, if we chose to have *this* be
our policy, then it would be best if dpkg-buildpackage and/or debhelper
would either detect a non-UTF-8 locale and error out early, or detect a
non-UTF-8 locale and quietly replace it with some UTF-8 locale (perhaps
C.UTF-8, or perhaps the closest equivalent UTF-8 locale, like replacing
ja_JP.EUC-JP with ja_JP.UTF-8).

I can remember several conversations in the past about potentially
dropping support for legacy non-UTF-8 locales like en_GB.ISO-8859-15
*completely* (not just de-supporting them for package builds, but
de-supporting their use on Debian under any circumstances), and
Alexandre's suggestion is a subset of that: leaving them available for
users who might still need them for whatever reason, but declaring that
they are not something we support at package-build time.

> Besides locales, there are other things that might affect outcomes, and we
> need to find some reasonable position between "packages should be
> reproducible even if built from the maintainer's user account on their
> personal machine" and "anything that is not a sterile systemd-nspawn
> container with exactly the requested Build-Depends and no Recommended
> packages causes undefined behaviour."

Yes. I think there is room for a more nuanced approach to this general
design principle: we can define some sources of variation as "possible
but not recommended", set them to a known value for official buildds,
make it as easy as possible to set them to a known value for local
test-builds, and consider FTBFS or non-reproducibility under those
variations to be a *low-severity* bug.

For instance, if a package is non-reproducible depending on whether I
happen to have libreally-obscure-dev installed, of course ideally that
should be fixed, but I would say that it's a much lower severity than
the package being non-reproducible depending on whether I have a
more commonly-required package like libglib2.0-dev which might be
difficult to remove non-disruptively.

Similarly, if a package FTBFS when built on a Tuesday, I'd say that's RC;
if it FTBFS when my locale is en_GB.UTF-8, under our current policies I'd
personally say that's annoying but non-RC (because if I'm debugging
the package, I could always grumble and work around that issue by
LC_ALL=C.UTF-8); and if it FTBFS when built on a machine where the
/nonexistent directory does, in fact, exist, then I would say that's
a non-bug.

(A concrete example of the latter: I'm pretty sure glib2.0 will fail
its test suite if /nonexistent exists, but if someone reported that as
a bug, I would be inclined to reply "/nonexistent shouldn't exist, the
clue's in the name" and close it.)

For locales and other facets of the execution environment that are
similarly easy to clear/reset/sanitize/normalize, we don't necessarily
need to be saying "if you do a build in this situation, you are doing
it wrong", because we could equally well be saying "if you do a build in
this situation, the build toolchain will automatically fix it for you" -
much more friendly towards anyone who is building packages interactively,
which seems to be the use-case that you're primarily interested in.

> Personally my preference would be as close as possible to
> [not needing a special build environment],
> because if I ever need to work on someone else's package, the chance is high
> that I will need incremental builds and a graphical debugger, and both of
> these are a major hassle in containers.

I don't think this is an either/or, but more like a spectrum: the more
your build environment diverges from what we might consider to be our
reference build environment, the more likely it is that a package will
FTBFS, fail tests, be non-reproducible or otherwise misbehave. It's
up to us, as a project, where to draw the line for "this divergence is
completely normal so the bug is RC" and, conversely, "this divergence
is so strange that it's a non-bug".

For something like the locale, it's very easy: if we decide that certain
locales are out-of-scope, then the build toolchain (dpkg-buildpackage or
similar) could just not allow the out-of-scope situations, because it's
straightforward (and doesn't require privileges) to force the build into
an in-scope situation and continue from there.

    smcv

[toc] | [prev] | [next] | [standalone]

#112122

On Sat, 08 Jun 2024 at 02:14:36 +0900, Simon Richter wrote:
> Reproducibility outside of sterile environments is however a problem for us
> as a distribution, because it affects how well people are able to contribute
> to packages they are not directly maintaining

If our package-building entry point sets up aspects of its desired
normalized (or "sterile") environment itself, surely it's equally easy
for those contributors to build every package in that way, whether they
maintain this particular package or not?

> if my package is not required to work outside
> of a very controlled environment, that is also an impediment to
> co-maintenance

I'm not sure that follows. If the only thing we require is that it works
in a controlled environment, and the controlled environment is documented
and easy to achieve, then surely every prospective co-maintainer is in
an equally good position to be contributing? That seems better, to me, than
having unwritten rules about what environment is "close enough" and what
environment doesn't actually work.

If I want to contribute to (let's say) both GNOME and KDE, but the GNOME
team expects me to be building in one controlled environment, and the KDE
team expects me to be building in a *different* controlled environment,
then sure, that would be a barrier to contribution: I'd have to do that
setup once per team, and maybe they'd be mutually incompatible. But that
isn't going to be the case if we're setting a policy for the whole distro,
which only needs to happen once?

We already do expect maintainers to be building in a specified
environment: Debian unstable (not stable, and not Ubuntu, for example).

I can see that if our policy was something like "must build in a schroot",
then that would be making us vulnerable to a lock-in where we can't
move to Podman or systemd-nspawn or Incus or whatever is the flavour of
the month because our policy says we use schroot, and then we end up
shackled to schroot's particular properties and limitations. (Indeed,
to an extent, we already have that problem by using schroot on official
buildds, and as a result being unable to gain much benefit from work
done on container technologies outside the Debian bubble.)

But that's not what was proposed by this thread: this thread is about
locales. Now that glibc has C.UTF-8 built-in and non-optional, you can
set a normalized or sterile locale regardless of whether you're building
on bare metal, in a VM, in a schroot, in Docker, or whatever, and it's
very easy to do that in a tool (or even an interactive shell) and have
it inherit down through the build? So I'm not sure I see the problem?

If you're making a wider point about use of containers etc. that is
orthogonal to setting the locale, then that would be a valid objection
to someone saying "we should standardize on building in Docker" (and I
would make a similar objection myself), but that's not this thread.

(I also do agree that it is an anti-pattern if we have a specified
environment where tests or QA will be run, and serious consequences for
failures in that environment, without it being reasonably straightforward
for contributors to repeat the testing in a sufficiently similar
controlled environment that they have a decent chance at being able to
reproduce the failure. But, again, that isn't this thread.)

> a lot of the debates we've had in the past years is who gets to
> decide what is in scope

Yes, that's always going to be the case for a community that doesn't
have an authority figure telling us "the scope is what I say it is". We
have debates when we don't all agree, and the scope of our collective
project is one of the foundations for all the other decisions we make,
so it's certainly something that we can't expect to be unanimous. (Insert
wise words from Russ Allbery about the difference between unanimity and
consensus here...)

I hope we can come close enough to a consensus that we're all generally
willing to accept it, though, even if that means sometimes accepting a
narrower or wider scope than I would personally prefer.

> > What Giole proposed at the beginning of this thread can be rephrased as
> > declaring that "FTBFS when locale is not C.UTF-8" and "non-reproducible
> > when locale is varied" are non-bugs, and therefore they are not only
> > wontfix, but they should be closed altogether as being out-of-scope.
> 
> Indeed -- however this class of bugs has already been solved because
> reproducible-builds.org have filed bugs wherever this happened, and
> maintainers have added workarounds where it was impossible to fix.

Someone (actually, quite a lot of someones) had to do that testing,
and those fixes or workarounds. Who did it benefit, and would they have
received the same benefit if we had said "building in a locale other than
C.UTF-8 is unsupported", or in some cases "building in a non-UTF-8 locale
is unsupported", and made it straightforward to build in the supported
locales?

I think there is a danger that we sink time and effort into doing work
that we are doing because our (written or unwritten) policy demands it,
even when it isn't clear that there is a real benefit from that work being
done. If that work is a fun and interesting puzzle and someone actively
wants to do it, then great!, but if it's something that a contributor
doesn't actually want to do, and is only doing because there is a rule
that demands it or a sanction that will be applied if it isn't done,
then we do need to consider whether the cost (imposing that work) is
justified by the benefit.

> Turning this workaround into boilerplate code was a mistake already, so the
> answer to the complaint about having to copy boilerplate code that should be
> moved into the framework is "do not copy boilerplate code."

If you don't want package-specific code to be responsible for forcing
a "reasonable" locale where necessary, then what layer do you want to
be responsible for it?  dpkg-buildpackage? debhelper? But then you go
on to say that you don't want those layers to set the locale either,
so I'm confused...

    smcv

[toc] | [prev] | [next] | [standalone]

#112131

Hi,

On 6/11/24 00:26, Simon McVittie wrote:

>> Reproducibility outside of sterile environments is however a problem for us
>> as a distribution, because it affects how well people are able to contribute
>> to packages they are not directly maintaining

> If our package-building entry point sets up aspects of its desired
> normalized (or "sterile") environment itself, surely it's equally easy
> for those contributors to build every package in that way, whether they
> maintain this particular package or not?

Yes, but building the package is not the hard part in making a useful 
contribution -- anything but trivial changes will need iterative 
modifications and testing, and the package building entrypoint is 
limited to "clean and build entire package" and "build package without 
cleaning first", with the latter being untested and likely broken for a 
lot of packages -- both meson and cmake utterly dislike being asked to 
configure an existing build directory as if it were new.

For my own packages, I roughly know how far I can deviate from the clean 
environment and still get meaningful test results, but for anything 
else, I will still need to deep-dive into the build system to get 
something that is capable of incremental builds.

>> if my package is not required to work outside
>> of a very controlled environment, that is also an impediment to
>> co-maintenance

> I'm not sure that follows. If the only thing we require is that it works
> in a controlled environment, and the controlled environment is documented
> and easy to achieve, then surely every prospective co-maintainer is in
> an equally good position to be contributing? That seems better, to me, than
> having unwritten rules about what environment is "close enough" and what
> environment doesn't actually work.

I will need to deviate from the clean environment, because the clean 
environment does not have vim installed. Someone else might need to 
deviate further and have a graphical environment and a lot of dbus 
services available because their preferred editor requires it.

Adding a global expectation about the environment that a package build 
can rely on *creates* an unwritten per-package rule whether it is 
permissible to deviate from this expectation during development.

I expect that pretty much no one uses the C.UTF-8 locale for their 
normal login session, so adding this as a requirement to the build 
environment creates a pretty onerous rule: "if you want to test your 
changes, you need to remember to call make/ninja with LC_ALL=C.UTF-8."

Of course we know that this rule is bullshit, because the majority of 
packages will build and test fine without it, but this knowledge is 
precisely one of the "unwritten rules" that we're trying to avoid here.

> We already do expect maintainers to be building in a specified
> environment: Debian unstable (not stable, and not Ubuntu, for example).

I develop mostly on Debian or Devuan stable, then do a pbuilder build 
right before upload to make sure it also builds in a clean unstable 
environment. The original requirement was mostly about uploading binary 
packages, which we (almost) don't do anymore.

> (I also do agree that it is an anti-pattern if we have a specified
> environment where tests or QA will be run, and serious consequences for
> failures in that environment, without it being reasonably straightforward
> for contributors to repeat the testing in a sufficiently similar
> controlled environment that they have a decent chance at being able to
> reproduce the failure. But, again, that isn't this thread.)

This is largely what I think is this thread -- narrowing the environment 
where builds, tests and QA will be run, and narrowing what will be 
considered a bug.

>> Indeed -- however this class of bugs has already been solved because
>> reproducible-builds.org have filed bugs wherever this happened, and
>> maintainers have added workarounds where it was impossible to fix.

> Someone (actually, quite a lot of someones) had to do that testing,
> and those fixes or workarounds. Who did it benefit, and would they have
> received the same benefit if we had said "building in a locale other than
> C.UTF-8 is unsupported", or in some cases "building in a non-UTF-8 locale
> is unsupported", and made it straightforward to build in the supported
> locales?

I'd say that developers who don't have English as their first language 
have directly benefited from this, and would not have benefited if it 
was not seen as a problem if a package didn't build on their machines 
without the use of a controlled environment.

I also think that we have indirectly benefited from better test coverage.

>> Turning this workaround into boilerplate code was a mistake already, so the
>> answer to the complaint about having to copy boilerplate code that should be
>> moved into the framework is "do not copy boilerplate code."

> If you don't want package-specific code to be responsible for forcing
> a "reasonable" locale where necessary, then what layer do you want to
> be responsible for it?

I want this to be package-specific, but applied only when necessary.

The original complaint was that having to copy this boilerplate code to 
fix reproducibility issues to each new package was a waste of 
maintainers' time and that it should be centralized into some framework, 
and my response to that is to stop copying unnecessary code into 
packages that don't need it.

At best, it does nothing because the package isn't broken, at worst it 
manifests additional bugs while someone is modifying the package to fix 
another problem.

If we are to move this into a framework, then this should take a 
declarative form, like "Rules-Requires-Locale: C.UTF-8", and it should 
be a goal to minimize use of this.

    Simon

[toc] | [prev] | [next] | [standalone]

#112339

[Multipart message — attachments visible in raw view] — view raw

Hi!

On Fri, 2024-06-07 at 15:40:07 +0200, Alexandre Detiste wrote:
> Maybe a compromise would be to at least mandate some UTF-8 locale.

Ah, good thinking! That would actually seem acceptable. I've prepared
the attached preliminary patch (missing better commit message, etc),
as a PoC for how this could look like. If there's consensus about
something like this, I'd be happy to merge into a future dpkg release.

Although I'm not sure though whether this would be enough to make it
possible to remove the hardcoding of LC_ALL=C.UTF-8 usage in debhelper,
which seems counter to l10n work, or perhaps to switch to a subset of
the locale settings. Niels?

Thanks,
Guillem

[toc] | [prev] | [next] | [standalone]

#112343

On Tue, 02 Jul 2024 at 03:47:29 +0200, Guillem Jover wrote:
> On Fri, 2024-06-07 at 15:40:07 +0200, Alexandre Detiste wrote:
> > Maybe a compromise would be to at least mandate some UTF-8 locale.
>
> dpkg-buildpackage: Require an UTF-8 (or ASCII) locale when
> building packages

Allowing ASCII seems counterproductive: that puts us in the code path
where various tools and runtimes (especially Python) will refuse to
process or output anything outside the 0-127 range, which I believe is
exactly the problem that debhelper aims to solve by using C.UTF-8 for
some categories of package (in particular those that build with Meson).

To get what Alexandre suggested, we'd need to allow UTF-8 but not allow
ASCII (so for example fr_FR.UTF-8 or C.UTF-8 is fine, but in particular
the C locale is not).

Or perhaps this pseudocode?

if (charset != UTF-8) {
    emit a warning
    export LC_ALL=C.UTF-8
    unset LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE (etc.)
}

    smcv

[toc] | [prev] | [next] | [standalone]

#112347

Hi!

On Tue, 2024-07-02 at 09:52:05 +0100, Simon McVittie wrote:
> On Tue, 02 Jul 2024 at 03:47:29 +0200, Guillem Jover wrote:
> > On Fri, 2024-06-07 at 15:40:07 +0200, Alexandre Detiste wrote:
> > > Maybe a compromise would be to at least mandate some UTF-8 locale.
> >
> > dpkg-buildpackage: Require an UTF-8 (or ASCII) locale when
> > building packages
> 
> Allowing ASCII seems counterproductive: that puts us in the code path
> where various tools and runtimes (especially Python) will refuse to
> process or output anything outside the 0-127 range, which I believe is
> exactly the problem that debhelper aims to solve by using C.UTF-8 for
> some categories of package (in particular those that build with Meson).
> 
> To get what Alexandre suggested, we'd need to allow UTF-8 but not allow
> ASCII (so for example fr_FR.UTF-8 or C.UTF-8 is fine, but in particular
> the C locale is not).

Err, you are right. I think I implemented this from my recollection of
the thread, trying to enforce as little as possible, and to try to let
users set "translations" to pure ASCII if desired, but that then defeats
the point brought up in the original mail, and the locale setting in
debhelper. I'll amend the PoC commit to only allow UTF-8.

(Also as long as LC_CTYPE is UTF-8 I think it should not matter whether
LC_MESSAGES is non-UTF-8 as the output codeset should still be UTF-8.)

> Or perhaps this pseudocode?
> 
> if (charset != UTF-8) {
>     emit a warning
>     export LC_ALL=C.UTF-8
>     unset LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE (etc.)
> }

As it stands, I don't think this would be good enough, because it would
introduce an implicit setting in dpkg-buildpackage while it is
currently not the only supported entry point, so packages could still
not rely on this being always set, and it still disables translated
messages.

While erroring out (even when dpkg-buildpackage is still not the only
supported entry point) would not give a full guarantee that a package
build is always done in a UTF-8 locale, it at least forces the caller
(be that a tool or a human) to change the running environment, while
not forcing untranslated messages. I guess this could be made a stronger
guarantee if debhelper switched from unconditionally setting the locale
to performing a similar check and erroring out too (instead of simply
removing the locale setting).


But from your pseudocode, now I realize the check I implemented is
probably too naive, as it should probably at least also check whether
LC_COLLATE is also UTF-8. So I'll try to think how to make it more
robust.

But, I guess I can at least unconditionally set LC_CTYPE=C.UTF-8 when
using --sanitive-env, right away though.

Thanks,
Guillem

[toc] | [prev] | [next] | [standalone]

#112370

Hi,

Thank you both.

One thing that could be fixed quite quickly is fixing the few
remaining official buildd workers that does not yet run with an UTF-8 locale.

If one is unlucky the build will mysteriously fail.

Adding  export {LC_ALL|LANG|LC_CTYPE}=C.UTF-8
in every single d/rules by fear of this seems overkill.

Greetings

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1074586
https://buildd.debian.org/status/package.php?p=xrayutilities


Le mar. 2 juil. 2024 à 14:37, Guillem Jover <guillem@debian.org> a écrit :
>
> Hi!
>
> On Tue, 2024-07-02 at 09:52:05 +0100, Simon McVittie wrote:
> > On Tue, 02 Jul 2024 at 03:47:29 +0200, Guillem Jover wrote:
> > > On Fri, 2024-06-07 at 15:40:07 +0200, Alexandre Detiste wrote:
> > > > Maybe a compromise would be to at least mandate some UTF-8 locale.
> > >
> > > dpkg-buildpackage: Require an UTF-8 (or ASCII) locale when
> > > building packages
> >
> > Allowing ASCII seems counterproductive: that puts us in the code path
> > where various tools and runtimes (especially Python) will refuse to
> > process or output anything outside the 0-127 range, which I believe is
> > exactly the problem that debhelper aims to solve by using C.UTF-8 for
> > some categories of package (in particular those that build with Meson).
> >
> > To get what Alexandre suggested, we'd need to allow UTF-8 but not allow
> > ASCII (so for example fr_FR.UTF-8 or C.UTF-8 is fine, but in particular
> > the C locale is not).
>
> But, I guess I can at least unconditionally set LC_CTYPE=C.UTF-8 when
> using --sanitive-env, right away though.
>
> Thanks,
> Guillem
>

[toc] | [prev] | [next] | [standalone]

#112823

[Multipart message — attachments visible in raw view] — view raw

Hi!

On Sat, 2024-07-06 at 13:13:48 +0200, Alexandre Detiste wrote:
> Le mar. 2 juil. 2024 à 14:37, Guillem Jover <guillem@debian.org> a écrit :
> > On Tue, 2024-07-02 at 09:52:05 +0100, Simon McVittie wrote:
> > > On Tue, 02 Jul 2024 at 03:47:29 +0200, Guillem Jover wrote:
> > > > On Fri, 2024-06-07 at 15:40:07 +0200, Alexandre Detiste wrote:
> > > > > Maybe a compromise would be to at least mandate some UTF-8 locale.
> > > >
> > > > dpkg-buildpackage: Require an UTF-8 (or ASCII) locale when
> > > > building packages
> > >
> > > Allowing ASCII seems counterproductive: that puts us in the code path
> > > where various tools and runtimes (especially Python) will refuse to
> > > process or output anything outside the 0-127 range, which I believe is
> > > exactly the problem that debhelper aims to solve by using C.UTF-8 for
> > > some categories of package (in particular those that build with Meson).
> > >
> > > To get what Alexandre suggested, we'd need to allow UTF-8 but not allow
> > > ASCII (so for example fr_FR.UTF-8 or C.UTF-8 is fine, but in particular
> > > the C locale is not).
> >
> > But, I guess I can at least unconditionally set LC_CTYPE=C.UTF-8 when
> > using --sanitive-env, right away though.

I did something like that as part of dpkg 1.22.7, with commit:

  https://git.dpkg.org/cgit/dpkg/dpkg.git/commit/?id=df60765ed4bc6640b788c796dd0c627d7714f807

Which should guarantee a UTF-8 codeset and stable sorting, while
preserving any translated output messages (and other locale settings).

> One thing that could be fixed quite quickly is fixing the few
> remaining official buildd workers that does not yet run with an UTF-8 locale.

Something I realized after adding the above change, is that sbuild has
been running dpkg-buildpackage with --sanitize-env for a while now,
which checking now I was told about at the time, but I either didn't
piece together its consequences or perhaps forgot that the sbuild
package is nowadays used in build daemons (instead of the old fork)
and then forgot. :) (BTW, not blaming josch! I think that change in
sbuild on its own makes sense, I guess I was just not expecting the
option to be used that way, and perhaps its documentation should have
somehow made it more clear. :)

I guess this is both good and "bad". It's good because now all build
daemons will have a guaranteed UTF-8 locale codeset already starting
with Debian trixie, as requested in this thread, and give us a more
uniform build environment. It's "bad" because part of the reason to
add this through a new --sanitize-env option was to make this behavior
and its guarantees opt-in, but if the official Debian builds are using
this, then it's in a way equivalent to having set this by default w/o
the option, but perhaps worse because people running local build will
not have the same environment (although it's going to be easy to
replicate by passing that option, but a bit harder when calling
debian/rules directly which we still support).

I'm not sure the current state is ideal, because we are back to
packages being able to rely on some stuff on build daemons, that are
not guaranteed by default for our supported build entry points, and if
the result of this is that we end up patching all dpkg-buildpackage
callers to pass --sanitize-env, then we could have as well simply
changed the default instead. I think a way forward could be to make
the sanitizing the default, and finally drop debian/rules as a
supported (user) build entry point, I had in mind re-proposing this
already, but the above kind of gives it more urgency, so I'll try to
do that soon.

This also means, I guess, that part of the previous freedom I thought
we had to modify the --sanitize-env behavior is kind of gone now (and
would be gone too if we move its behavior as the default one), and we
should apply similar care as if the default itself was being changed,
because it has the potential to break the archive (via build daemons).
I'm thinking that depending on the changes there, it might be better
to gate them via dpkg-build-api(7) levels. I should also document the
vendor specific behavior in some manual page, as it is currently
listed as unspecified "vendor specific".

> If one is unlucky the build will mysteriously fail.
> 
> Adding  export {LC_ALL|LANG|LC_CTYPE}=C.UTF-8
> in every single d/rules by fear of this seems overkill.

> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1074586
> https://buildd.debian.org/status/package.php?p=xrayutilities

I implemented the attached patch (also on the next/req-utf8 branch) to
force a locale with a UTF-8 codeset, which would be a no-op now when
using --sanitize-env, but I didn't merge that for now, because I'm not
sure of the potential fallout, given that other infrastructure things
might be running dpkg-buildpackage w/o passing --sanitize-env. So I
think those would need to be found and changed before deploying that
change.

  <https://git.hadrons.org/cgit/debian/dpkg/dpkg.git/log/?h=next/req-utf8>

But then, I guess whether merging that makes sense or not also depends
on how we want to proceed with the debian/rules build entry point, and
whether we are going to switch the default or transition to amend all
callers (which might still not catch private infra and similar).

Thanks,
Guillem

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.devel

csiph-web