Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #82270 > unrolled thread

Rust for Linux and host-only libraries

Back to article view | Back to linux.debian.kernel

Contents

  Rust for Linux and host-only libraries Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> - 2024-04-19 19:30 +0200
    Re: Rust for Linux and host-only libraries Fabian Grünbichler <debian@fabian.gruenbichler.email> - 2024-04-19 20:40 +0200
      Re: Rust for Linux and host-only libraries Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> - 2024-04-19 21:10 +0200
        Re: Rust for Linux and host-only libraries Fabian Grünbichler <debian@fabian.gruenbichler.email> - 2024-04-19 23:00 +0200
          Re: Rust for Linux and host-only libraries Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> - 2024-04-20 12:40 +0200

#82270 — Rust for Linux and host-only libraries

Hi Ximin, Sylvestre, Fabian, all,

In Rust for Linux, we are considering using host-only/userspace Rust
libraries (e.g. `syn`), and we were asked to check whether it would be
possible to just pick them from the distribution (since, in principle,
they don't require kernel-specific changes).

What is the policy in Debian for those? I was told about
`debcargo-conf` by Zixing Liu, and indeed I can see e.g. `syn` there
in `librust-syn-dev`, which places the sources at
`/usr/share/cargo/registry/`, which is great for what we need. Is that
correct? This would be useful for kernel developers that do not trust
third-party repositories (not even Rust's/crates.io/the upstream
repository).

By the way, is there a way to query where the sources are (i.e. that
path) in a "standard" way (e.g. like `pkg-config`)? Or, at least, can
we assume it should be stable? (e.g. for automated detection and/or
instructions we may want to provide).

Thanks!

Cheers,
Miguel

[toc] | [next] | [standalone]

#82271

On April 19, 2024 7:09:15 PM GMT+02:00, Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> wrote:
>Hi Ximin, Sylvestre, Fabian, all,
>
>In Rust for Linux, we are considering using host-only/userspace Rust
>libraries (e.g. `syn`), and we were asked to check whether it would be
>possible to just pick them from the distribution (since, in principle,
>they don't require kernel-specific changes).
>
>What is the policy in Debian for those? I was told about
>`debcargo-conf` by Zixing Liu, and indeed I can see e.g. `syn` there
>in `librust-syn-dev`, which places the sources at
>`/usr/share/cargo/registry/`, which is great for what we need. Is that
>correct? This would be useful for kernel developers that do not trust
>third-party repositories (not even Rust's/crates.io/the upstream
>repository).

Yes, all crates packaged by the Rust team (and most packaged for consumption by other packages/as build-deps by people outside of the team) ship their (patched for distro use) sources in a subdir there. The patches (except for wholesale exclusions of bundled C code and similar things, which is done via orig tarball repacking) are also shipped there, so the delta is easily reviewable. The whole directory structure can be used as (partial) drop-in for crates.io by configuring cargo accordingly (in Debian packaging this is done by a cargo wrapper shipped in /usr/share/cargo/bin/cargo, but that one is probably not suitable as-is for kernel stuff - it or rather its output might serve as an example for the needed cargo config runes though ;))

>By the way, is there a way to query where the sources are (i.e. that
>path) in a "standard" way (e.g. like `pkg-config`)? Or, at least, can
>we assume it should be stable? (e.g. for automated detection and/or
>instructions we may want to provide).

Not really - other than that directory being in the format the cargo expects for registry replacement, which is the same structure as `cargo vendor` creates. There is no index or .pc equivalent - the Cargo.toml file inside each crate has all the info and more in a standardized fashion after all :)

I don't except this to change anytime soon (basically - only if stable dynamic linking becomes feasible and we stop shipping the sources in binary packages altogether ;) or if cargo itself completely revamps how it supports vendoring). If we know people/projects outside of Debian packaging are using this, adding a comment in the tooling that currently decides those paths (and their contents/structure) so that we know we should discuss or at least give a heads up about any planned fundamental changes should be easy to do :)

Don't hesitate to reach out if you have further questions or go ahead with using the packaged crate sources! 

While there is considerable overlap w.r.t. subscribers, there is a second Debian rust list for discussions (debian-rust@lists.debian.org) - this one here gets all the automated notifications for all team maintained packages, so is rather high volume and things can get overlooked sometimes.

[toc] | [prev] | [next] | [standalone]

#82272

On Fri, Apr 19, 2024 at 8:13 PM Fabian Grünbichler
<debian@fabian.gruenbichler.email> wrote:
>
> Yes, all crates packaged by the Rust team (and most packaged for consumption by other packages/as build-deps by people outside of the team) ship their (patched for distro use) sources in a subdir there. The patches (except for wholesale exclusions of bundled C code and similar things, which is done via orig tarball repacking) are also shipped there, so the delta is easily reviewable. The whole directory structure can be used as (partial) drop-in for crates.io by configuring cargo accordingly (in Debian packaging this is done by a cargo wrapper shipped in /usr/share/cargo/bin/cargo, but that one is probably not suitable as-is for kernel stuff - it or rather its output might serve as an example for the needed cargo config runes though ;))

Thanks for the quick reply!

I see -- in principle, we only need the sources, i.e. we would not
compile the crates using Cargo. Are the patches applied to the sources
already, or are they applied by that cargo wrapper you mention at
build time? (Well, assuming users should use the patched versions).

> Not really - other than that directory being in the format the cargo expects for registry replacement, which is the same structure as `cargo vendor` creates. There is no index or .pc equivalent - the Cargo.toml file inside each crate has all the info and more in a standardized fashion after all :)
>
> I don't except this to change anytime soon (basically - only if stable dynamic linking becomes feasible and we stop shipping the sources in binary packages altogether ;) or if cargo itself completely revamps how it supports vendoring). If we know people/projects outside of Debian packaging are using this, adding a comment in the tooling that currently decides those paths (and their contents/structure) so that we know we should discuss or at least give a heads up about any planned fundamental changes should be easy to do :)

Sounds great.

I am not sure yet how we will approach it, but we would give the users
the option to customize the path, of course.

> Don't hesitate to reach out if you have further questions or go ahead with using the packaged crate sources!
>
> While there is considerable overlap w.r.t. subscribers, there is a second Debian rust list for discussions (debian-rust@lists.debian.org) - this one here gets all the automated notifications for all team maintained packages, so is rather high volume and things can get overlooked sometimes.

Thanks!

Cheers,
Miguel

[toc] | [prev] | [next] | [standalone]

#82275

On April 19, 2024 8:44:12 PM GMT+02:00, Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> wrote:
>On Fri, Apr 19, 2024 at 8:13 PM Fabian Grünbichler
><debian@fabian.gruenbichler.email> wrote:
>>
>> Yes, all crates packaged by the Rust team (and most packaged for consumption by other packages/as build-deps by people outside of the team) ship their (patched for distro use) sources in a subdir there. The patches (except for wholesale exclusions of bundled C code and similar things, which is done via orig tarball repacking) are also shipped there, so the delta is easily reviewable. The whole directory structure can be used as (partial) drop-in for crates.io by configuring cargo accordingly (in Debian packaging this is done by a cargo wrapper shipped in /usr/share/cargo/bin/cargo, but that one is probably not suitable as-is for kernel stuff - it or rather its output might serve as an example for the needed cargo config runes though ;))
>
>Thanks for the quick reply!
>
>I see -- in principle, we only need the sources, i.e. we would not
>compile the crates using Cargo. Are the patches applied to the sources
>already, or are they applied by that cargo wrapper you mention at
>build time? (Well, assuming users should use the patched versions).

No, the patches are already applied in the source files shipped by librust-xxx-dev, if there are any. The cargo wrapper takes care of other things, like instructing cargo to use the packaged sources and setting rustflags and similar options. You should be able to use the same sources with manual rustc invocations without cargo or our wrapper just fine :) 

There's a bit of peculiarity around the versioning - the librust-.. packages provide all the semver prefixes of their version as well, so you can express "I want crate X in a semver compatible version of at least 2.3" via a versioned dependency (by depending on librust-x-2-dev with a version of at least 2.3), but also the more restricted variants (e.g., X in some 2.3.n version, or a specific 2.3.4 version, by depending on librust-2.3-dev resp. librust-2.3.4-dev). Normally, the actual package providing all of those is librust-X-dev, except when we need to package more than one version of X, then there might be a librust-X-dev in version A (providing all the semver prefixes of A), and another package librust-X-B-dev, where B is the shortest semver-breaking prefix of a semver incompatible version older than A (providing all semver prefixes of that version). A might be 2.3.4 and B 1 (with B's full version being 1.2.0, for example). Most of that won't be relevant for you - except that you want to depend on e.g. librust-syn-1-dev if you want syn 1.x, even if the actual "real" package is called librust-syn-dev, as the latter might also contain syn 0.x or 2.x in other distros/releases. Hope this was not too confusing ;)

[toc] | [prev] | [next] | [standalone]

#82277

On Fri, Apr 19, 2024 at 10:34 PM Fabian Grünbichler
<debian@fabian.gruenbichler.email> wrote:
>
> No, the patches are already applied in the source files shipped by librust-xxx-dev, if there are any. The cargo wrapper takes care of other things, like instructing cargo to use the packaged sources and setting rustflags and similar options. You should be able to use the same sources with manual rustc invocations without cargo or our wrapper just fine :)

Great, that should make it straightforward then :)

> There's a bit of peculiarity around the versioning - the librust-.. packages provide all the semver prefixes of their version as well, so you can express "I want crate X in a semver compatible version of at least 2.3" via a versioned dependency (by depending on librust-x-2-dev with a version of at least 2.3), but also the more restricted variants (e.g., X in some 2.3.n version, or a specific 2.3.4 version, by depending on librust-2.3-dev resp. librust-2.3.4-dev). Normally, the actual package providing all of those is librust-X-dev, except when we need to package more than one version of X, then there might be a librust-X-dev in version A (providing all the semver prefixes of A), and another package librust-X-B-dev, where B is the shortest semver-breaking prefix of a semver incompatible version older than A (providing all semver prefixes of that version). A might be 2.3.4 and B 1 (with B's full version being 1.2.0, for example). Most of that won't be relevant for you - except that you want to depend on e.g. librust-syn-1-dev if you want syn 1.x, even if the actual "real" package is called librust-syn-dev, as the latter might also contain syn 0.x or 2.x in other distros/releases. Hope this was not too confusing ;)

That is good to know, thanks! So, ideally, we would avoid handling
versioning/packaging as much as possible, i.e. making it as
independent of distributions and so on as possible, and letting the
user install the packages themselves. It should be possible to just
pass a path to a random folder with the library, if needed.

So ideally we would suggest in the instructions what packages to
install (for major distributions), and try to be agnostic otherwise in
the actual build system. That implies that the question is trying to
automate, if possible, finding the sources, without hardcoding too
much information.

For instance, one naive way would be just testing whether any
`/usr/share/cargo/registry/{library}-{major}*` folder exists and use
the highest one if so. If people only have a single version installed
(per major) or if the dependencies' constraints are relaxed/compatible
enough, that could "just work".

Is that way too naive? Because even if it is naive, but works for most
kernel developers out there, that may be good enough. And if it fails,
one can always customize the paths as needed.

The problem, of course, is cases where silent miscompilations could
occur. For instance, Debian or another distribution could provide two
minors (of the same major version) of library A there, and then
library B depends on the lower of the two (for some reason, including
possibly a patch you may apply). And it turns out that if one picks
the highest A, things actually compile fine, but may behave badly in
some cases. Thus we could get, silently, a bad build.

I mean, such a sequence of events may be unlikely to happen in
practice, especially given the kind of library we are considering
using (so far). But still, probably we should just get the paths for
the dependencies somehow (e.g. querying Cargo, to avoid parsing
`Cargo.toml` ourselves, especially the version constraints; or perhaps
there is a better way you can think of) if we want to use the
libraries available at `/usr/share/cargo/registry/`. I checked the
Debian package, and there were two promising `.cargo-*.json` files,
but they do not give information about paths.

What do you think? Is there a proper/better/simpler way, for
distributions that use `/usr/share/cargo/registry`, to get the paths
of a library and its dependencies of a given library?

Thanks!

Cheers,
Miguel

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web