Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.security > #5920 > unrolled thread

GPG verification of apt packages

Back to article view | Back to linux.debian.security

Contents

  GPG verification of apt packages Ramin Doe <ramindough@gmail.com> - 2021-01-27 19:30 +0100
    Re: GPG verification of apt packages Jonathan Hutchins <hutchins@tarcanfel.org> - 2021-01-27 20:20 +0100
    Re: GPG verification of apt packages Noah Meyerhans <noahm@debian.org> - 2021-01-28 06:40 +0100
      Re: GPG verification of apt packages Ramin Doe <ramindough@gmail.com> - 2021-01-28 19:10 +0100
        Re: GPG verification of apt packages Clément Hermann <nodens@debian.org> - 2021-01-28 20:50 +0100
        Re: GPG verification of apt packages Noah Meyerhans <noahm@debian.org> - 2021-01-28 21:00 +0100
          Re: GPG verification of apt packages Ramin Doe <ramindough@gmail.com> - 2021-02-01 20:00 +0100
          Re: GPG verification of apt packages Ramin Doe <ramindough@gmail.com> - 2021-02-03 22:00 +0100
            Re: GPG verification of apt packages Clément Hermann <nodens@nodens.org> - 2021-02-04 10:10 +0100

#5920 — GPG verification of apt packages

[Multipart message — attachments visible in raw view] — view raw

Sorry if this has been brought up before. If there's a prior discussion
anyone can point me to, I'd really appreciate it.

I am trying to set up a Debian 10 based server to host a web app that's
visible to the public. Putting some time into finding out best practices in
this situation, I came across the Center for Internet Security (CIS)
document (CIS_Debian_Linux_10_Benchmark_v1.0.0.pdf) that on page 75,
perhaps suggest that apt is not configured correctly to verify packages. I
say "perhaps" because the wording in the PDF isn't very clear.

This lead me to search for more answers online, where I have found an
article that suggests that package metadata is verified, but that package
contents are not. (
https://blog.packagecloud.io/eng/2014/10/28/howto-gpg-sign-verify-deb-packages-apt-repositories/)
Again, it's unclear if that article applies to Debian 10, but since my
system as "no-debsig" in /etc/dpkg/dpkg.cfg, it seems like it might and
therefore my system is not verifying the contents of apt packages and is
vulnerable to some sorts of attacks through apt.

The closest to an official word on this, that I have been able to find, is
this web-page (https://wiki.debian.org/SecureApt) but it's not very
understandable to me, and it's not clear how up-to-date it is. There is
much talk of Release files, but I don't know where these files are, and so
I can't test out the mechanisms described.

I do know that if I use "apt download" to download a .deb file, break it
apart (using ar and tar), make a change, and put it back together, I can
the use "apt install ./X.deb" to install it, even though I haven't updated
any security metadata in the .deb file. Removing "no-debsig" in dpkg.cfg
doesn't affect the outcome.

And finally, it seems that even wikipedia says that package signatures
aren't being checked on most systems (
https://en.wikipedia.org/wiki/Deb_%28file_format%29#Signed_packages).

I hope someone can help me find some answers on this!

thanks,
Ramin

[toc] | [next] | [standalone]

#5921

If this were an actual problem thousands of people would be having it.

Trust the force.

-- 
Jonathan

[toc] | [prev] | [next] | [standalone]

#5922

[Multipart message — attachments visible in raw view] — view raw

On Wed, Jan 27, 2021 at 10:23:44AM -0800, Ramin Doe wrote:
>    This lead me to search for more answers online, where I have found an
>    article that suggests that package metadata is verified, but that package
>    contents are not.
>    ([1]https://blog.packagecloud.io/eng/2014/10/28/howto-gpg-sign-verify-deb-packages-apt-repositories/)

The signed metadata includes cryptographic checksums of the package
contents.  Thus, package contents can't be modified in storage on the
mirror or in transit to your system without invalidating the checksum,
and the checksums can't be updated in the repository metadata without
invalidating the signature.

>    I do know that if I use "apt download" to download a .deb file, break it
>    apart (using ar and tar), make a change, and put it back together, I can
>    the use "apt install ./X.deb" to install it, even though I haven't updated
>    any security metadata in the .deb file. Removing "no-debsig" in dpkg.cfg
>    doesn't affect the outcome.

That's correct; the validation happens during retrieval.  Once the
package is on your computer, you are free to tamper with it however you
want.

>    And finally, it seems that even wikipedia says that package signatures
>    aren't being checked on most systems
>    ([3]https://en.wikipedia.org/wiki/Deb_%28file_format%29#Signed_packages).

That's correct; package validation is done as described above.  You left
out the part of the wikipedia article that states that "packages are
verified by signing the repository metadata (i.e. Release files).  The
metadata files in turn include checksums for the repository files as a
means to verify authenticity of the files."

noah

[toc] | [prev] | [next] | [standalone]

#5923

[Multipart message — attachments visible in raw view] — view raw

On Wed, Jan 27, 2021 at 9:27 PM Noah Meyerhans <noahm@debian.org> wrote:

>
> The signed metadata includes cryptographic checksums of the package
> contents.  Thus, package contents can't be modified in storage on the
> mirror or in transit to your system without invalidating the checksum,
> and the checksums can't be updated in the repository metadata without
> invalidating the signature.
>

This all sounds pretty promising! Thank you, Noah! Do you happen to know
how to access this metadata? I'd love to be able to look at it and
understand it better.


>
> >    I do know that [I can tamper with .deb pkg on my computer prior to
> installation without triggering warnings].
>
> That's correct; the validation happens during retrieval.  Once the
> package is on your computer, you are free to tamper with it however you
> want.
>

Thanks! It's a relief that the validation occurs elsewhere.

>    And finally, it seems that even wikipedia says that package signatures
> >    aren't being checked on most systems
> >    ([3]
> https://en.wikipedia.org/wiki/Deb_%28file_format%29#Signed_packages).
>
> That's correct; package validation is done as described above.  You left
> out the part of the wikipedia article that states [...]
>

Here's the full text (minus footnote anchors):

"Debian-based distributions support GPG signature verification of signed
Debian packages, but most (if not all) have this feature disabled by
default. Instead packages are verified by signing the repository metadata
(i.e. Release files). The metadata files in turn include checksums for the
repository files as a means to verify authenticity of the files. Currently
there are two different implementations for signing individual packages.
The first is done via the debsigs / debsig-verify toolset, which is
supported by dpkg. The second is done through the dpkg-sig program which is
not supported by dpkg, so the packages have to be manually checked with the
dpkg-sig program. Both formats add new sections to the ar archive to store
the signature information, but the formats are not compatible with one
another. Neither of the modifications to the package format are listed in
the official Debian handbook or man page about the binary package format."

I'm still struggling with this paragraph. Let me see if I can break it down
sensibly.

"Debian-based distributions support GPG signature verification of signed
Debian packages, but most (if not all) have this feature disabled by
default."

OK. That's straightforward enough. There's some infrastructure for GPG
signature verification but it's disabled by default. That's not a problem
in itself, if some other form of effective verification is happening. Just
another sentence stating that in the wikipedia article would have made a
world of difference to me.

"Instead packages are verified by signing the repository metadata (i.e.
Release files). The metadata files in turn include checksums for the
repository files as a means to verify authenticity of the files."

Again, I'd love to see one of these release files, so I could see:
a) what data, exactly, is being checksummed
b) what sort of hash algorithm is involved

In my digging around so far, I found that the .deb file contains a
control.tar.xz file, which has some md5 checksum information, but it has
very patchy coverage of the files in the package. I hope that's just a
holdover from a deprecated mechanism, and is not being used nowadays.

"Currently there are two different implementations for signing individual
packages..."
I think this is referring to the GPG signature verification mechanisms that
are disabled by default. I'm happy to not try to not go down the route of
enabling GPG verification, since it seems to be poorly documented (I
haven't found a single concrete example of how to do this), so long as I
can feel that the metadata checksum method is sufficiently reliable. I
think that looking at the Release files would go a long way to relieving my
anxiety about this. Any help would be appreciated!

I do wish there was an official document giving a high-level TLDR
description of apt security, complete with caveats. As a bonus
cherry-on-top wish, it would be awesome if it furthermore made clear what
old mechanisms were deprecated and could be ignored!

[toc] | [prev] | [next] | [standalone]

#5924

Hi,

On 28/01/2021 19:08, Ramin Doe wrote:

> "Currently there are two different implementations for signing
> individual packages..."
> I think this is referring to the GPG signature verification mechanisms
> that are disabled by default. I'm happy to not try to not go down the
> route of enabling GPG verification, since it seems to be poorly
> documented (I haven't found a single concrete example of how to do
> this), so long as I can feel that the metadata checksum method is
> sufficiently reliable. I think that looking at the Release files would
> go a long way to relieving my anxiety about this. Any help would be
> appreciated!

Check any mirror ? e.g https://debian.ethz.ch/debian/dists/buster/

> I do wish there was an official document giving a high-level TLDR
> description of apt security, complete with caveats. As a bonus
> cherry-on-top wish, it would be awesome if it furthermore made clear
> what old mechanisms were deprecated and could be ignored!

The closest thing that comes to my mind would be
https://www.debian.org/doc/manuals/securing-debian-manual/deb-pack-sign.en.html.
Hope it helps!

Cheers,

-- 
nodens

[toc] | [prev] | [next] | [standalone]

#5925

[Multipart message — attachments visible in raw view] — view raw

On Thu, Jan 28, 2021 at 10:08:32AM -0800, Ramin Doe wrote:
>      The signed metadata includes cryptographic checksums of the package
>      contents.  Thus, package contents can't be modified in storage on the
>      mirror or in transit to your system without invalidating the checksum,
>      and the checksums can't be updated in the repository metadata without
>      invalidating the signature.
> 
>    This all sounds pretty promising! Thank you, Noah! Do you happen to know
>    how to access this metadata? I'd love to be able to look at it and
>    understand it better.

See the signed InRelease files in /var/lib/apt/lists

>    Again, I'd love to see one of these release files, so I could see:
>    a) what data, exactly, is being checksummed
>    b) what sort of hash algorithm is involved
>    In my digging around so far, I found that the .deb file contains a
>    control.tar.xz file, which has some md5 checksum information, but it has
>    very patchy coverage of the files in the package. I hope that's just a
>    holdover from a deprecated mechanism, and is not being used nowadays.

The MD5 sums are use by the debsums program.  Because md5 is weak and
because there are plenty of ways to bypass file-level checksum
validation in general, these are more useful at identifying corruption
or valid local modifications rather than system compromise.

You should read
https://www.debian.org/doc/manuals/securing-debian-manual/deb-pack-sign.en.html

It's a little dated, as it still mentions the use of md5 in the Release
files when we use sha256 these days, but it's good for a higher level
overview.

noah

[toc] | [prev] | [next] | [standalone]

#5926

[Multipart message — attachments visible in raw view] — view raw

>
> >    This all sounds pretty promising! Thank you, Noah! Do you happen to
> know
> >    how to access this metadata? I'd love to be able to look at it and
> >    understand it better.
>
> See the signed InRelease files in /var/lib/apt/lists
>

Ah! I see some files here that are relevant to my search. Thanks! For
instance, on my system, I see a
mirrors.linode.com_debian_dists_buster_main_binary-amd64_Packages file that
contains a single SHA256 and a single MD5sum hash for each amd64 package. I
could see how that sort of information could be used to verify, on the fly,
the packages that arrive here.


> You should read
>
> https://www.debian.org/doc/manuals/securing-debian-manual/deb-pack-sign.en.html
>
> Thanks! I will do so!

[toc] | [prev] | [next] | [standalone]

#5927

[Multipart message — attachments visible in raw view] — view raw

So after doing my reading and digging around a bit, I get a vague sense of
the pieces involved. I can see the various data files available on the
mirror site, that *could* be used to verify that all the files we receive
are what we'd expect to be getting, based on some initial data (a trusted
key) that we might have on our system. But as some have mentioned, the
documentation in Securing Debian Manual (
https://www.debian.org/doc/manuals/securing-debian-manual/) is out of date
and/or oversimplified. It's still helpful though!

So I could see how the local keys /etc/apt/trusted.gpg.d/*.gpg might be
used to verify that InRelease that is downloaded from
https://[mirrorsite]/debian/dists/buster/ is
trustworthy. And it looks like that InRelease file contains sha256
signatures that can be used to verify the contents of various files,
including Packages.gz that would be downloaded from
https://[mirrorsite]/debian/dists/buster/main/binary-amd64/).
In that Packages file, there are sha256 signatures for individual packages
that apt would install, and those could be used to verify all the files
that were downloaded by an `apt install` command before installation.

The only thing that still worries me a little bit is that I can't actually
verify that any of the described verification is actually being done, or
that I have imagined the process correctly. (Is there an existing corrupted
mirror I can use? Is it hard to set one up? If I could force a package
rejection in different ways, I'd be fully confident it was configured and
functioning correctly).

I understand, of course, that all this assumes that my system hasn't been
compromised already. But, yes, if I have a clean system, and non of the
keys or hashes are compromised, I can see how this would work. And this
would mean that I wouldn't need to do GPG verification of the packages as
described in section 7.5.2 of the Manual, and the article I first posted (
https://blog.packagecloud.io/eng/2014/10/28/howto-gpg-sign-verify-deb-packages-apt-repositories/
).

It's not entirely clear to me what the CIS guideline was expecting me to
do. It says:

    Verify GPG keys are configured correctly for your package manager:
    # apt-key list

Perhaps they want me to install apt-key, and use it to look at the gpg keys
installed on my system, and then somehow verify that they aren't
compromised? Does that sound like I'm understanding them correctly?

thanks for all the help!
Ramin


On Thu, Jan 28, 2021 at 11:56 AM Noah Meyerhans <noahm@debian.org> wrote:

> On Thu, Jan 28, 2021 at 10:08:32AM -0800, Ramin Doe wrote:
> >      The signed metadata includes cryptographic checksums of the package
> >      contents.  Thus, package contents can't be modified in storage on
> the
> >      mirror or in transit to your system without invalidating the
> checksum,
> >      and the checksums can't be updated in the repository metadata
> without
> >      invalidating the signature.
> >
> >    This all sounds pretty promising! Thank you, Noah! Do you happen to
> know
> >    how to access this metadata? I'd love to be able to look at it and
> >    understand it better.
>
> See the signed InRelease files in /var/lib/apt/lists
>
> >    Again, I'd love to see one of these release files, so I could see:
> >    a) what data, exactly, is being checksummed
> >    b) what sort of hash algorithm is involved
> >    In my digging around so far, I found that the .deb file contains a
> >    control.tar.xz file, which has some md5 checksum information, but it
> has
> >    very patchy coverage of the files in the package. I hope that's just a
> >    holdover from a deprecated mechanism, and is not being used nowadays.
>
> The MD5 sums are use by the debsums program.  Because md5 is weak and
> because there are plenty of ways to bypass file-level checksum
> validation in general, these are more useful at identifying corruption
> or valid local modifications rather than system compromise.
>
> You should read
>
> https://www.debian.org/doc/manuals/securing-debian-manual/deb-pack-sign.en.html
>
> It's a little dated, as it still mentions the use of md5 in the Release
> files when we use sha256 these days, but it's good for a higher level
> overview.
>
> noah
>
>

[toc] | [prev] | [next] | [standalone]

#5928

On 03/02/2021 21:50, Ramin Doe wrote:
> It's not entirely clear to me what the CIS guideline was expecting me to
> do. It says:
> 
>     Verify GPG keys are configured correctly for your package manager:
>     # apt-key list
> 
> Perhaps they want me to install apt-key, and use it to look at the gpg
> keys installed on my system, and then somehow verify that they aren't
> compromised? Does that sound like I'm understanding them correctly?

apt-key is considered deprecated (check the man: apt-key(8)). However,
what this command does is show you the list of keys trusted for package
installation.

I guess it's to make sure you have only legit keys there - but I didn't
read those guidelines, so I can't be completely sure :)

Cheers,

-- 
nodens

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.security

csiph-web