Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #59120 > unrolled thread

Secure boot signing infrastructure - feedback request

Back to article view | Back to linux.debian.kernel

Contents

  Secure boot signing infrastructure - feedback request Helen Koike <helen.koike@collabora.com> - 2017-10-05 20:30 +0200
    Re: Secure boot signing infrastructure - feedback request Helen Koike <helen.koike@collabora.com> - 2017-10-09 03:20 +0200
    Re: Secure boot signing infrastructure - feedback request Ben Hutchings <ben@decadent.org.uk> - 2017-10-09 15:10 +0200
      Re: Secure boot signing infrastructure - feedback request Steve McIntyre <steve@einval.com> - 2017-10-09 18:40 +0200
        Re: Secure boot signing infrastructure - feedback request Julien Cristau <jcristau@debian.org> - 2017-10-09 18:50 +0200
        Re: Secure boot signing infrastructure - feedback request Ben Hutchings <ben@decadent.org.uk> - 2017-10-09 20:10 +0200
          Re: Secure boot signing infrastructure - feedback request Ansgar Burchardt <ansgar@debian.org> - 2017-10-13 00:10 +0200
            Re: Secure boot signing infrastructure - feedback request Helen Koike <helen.koike@collabora.com> - 2017-10-13 00:20 +0200
              Re: Secure boot signing infrastructure - feedback request Steve McIntyre <steve@einval.com> - 2017-10-31 17:00 +0100
                Re: Secure boot signing infrastructure - feedback request Helen Koike <helen.koike@collabora.com> - 2017-11-15 14:50 +0100
                  Re: Secure boot signing infrastructure - feedback request Ben Hutchings <ben@decadent.org.uk> - 2017-11-23 00:20 +0100
                    Re: Secure boot signing infrastructure - feedback request Steve McIntyre <steve@einval.com> - 2017-11-23 17:30 +0100
                Re: Secure boot signing infrastructure - feedback request Ben Hutchings <ben@decadent.org.uk> - 2017-11-23 00:10 +0100
              Re: Secure boot signing infrastructure - feedback request Ben Hutchings <ben@decadent.org.uk> - 2017-11-23 00:10 +0100
                Re: Secure boot signing infrastructure - feedback request Steve McIntyre <steve@einval.com> - 2017-11-23 17:30 +0100
            Re: Secure boot signing infrastructure - feedback request Ben Hutchings <ben@decadent.org.uk> - 2017-11-22 23:10 +0100
              Re: Secure boot signing infrastructure - feedback request Tollef Fog Heen <tfheen@err.no> - 2017-11-24 21:00 +0100
          Re: Secure boot signing infrastructure - feedback request Steve McIntyre <steve@einval.com> - 2017-10-13 00:50 +0200
            Re: Secure boot signing infrastructure - feedback request Ben Hutchings <ben@decadent.org.uk> - 2017-11-22 23:20 +0100

#59120 — Secure boot signing infrastructure - feedback request

Hello all,

As you probably already know, Debian doesn't support the Secure Boot
chain yet.
To support it we need to sign Grub and the Kernel with our key, so we
are discussing the best infrastructure for this workflow.

The first approach we had was to add a by-hand script in Dak as
described here
https://wiki.debian.org/SecureBoot#First_option:_by-hand_script_in_dak
But this option wasn't well received by the ftpteam

The second approach we have is to add some debhelper scripts (e.g
dh_sign...) that will access a signing service which will sign the
binaries with Debian's key.  We would use the dh_sign... helpers when
making an extra binary package. buildd would then publish the -signed
version of the package in the archives.
Please see a more detailed explanation here
https://wiki.debian.org/SecureBoot#Second_option:_use_buildd_.2B-_debhelper_instead_of_dak
A current known issue with this approach is the NEW queue: it requires
the maintainer to also upload binaries for an architecture on first
upload, and these binaries are not rebuilt by the buildd ( see
https://wiki.debian.org/SecureBoot#Issues ).

I would like to know everyone's opinions about these approaches, if you
agree to go forward with the second approach described above and how do
we solve the NEW queue policy issue.

Thanks
Helen Koike

[toc] | [next] | [standalone]

#59144

+

On 2017-10-05 02:57 PM, Helen Koike wrote:
> Hello all,
> 
> As you probably already know, Debian doesn't support the Secure Boot
> chain yet.
> To support it we need to sign Grub and the Kernel with our key, so we
> are discussing the best infrastructure for this workflow.
> 
> The first approach we had was to add a by-hand script in Dak as
> described here
> https://wiki.debian.org/SecureBoot#First_option:_by-hand_script_in_dak
> But this option wasn't well received by the ftpteam
> 
> The second approach we have is to add some debhelper scripts (e.g
> dh_sign...) that will access a signing service which will sign the
> binaries with Debian's key.  We would use the dh_sign... helpers when
> making an extra binary package. buildd would then publish the -signed
> version of the package in the archives.
> Please see a more detailed explanation here
> https://wiki.debian.org/SecureBoot#Second_option:_use_buildd_.2B-_debhelper_instead_of_dak
> A current known issue with this approach is the NEW queue: it requires
> the maintainer to also upload binaries for an architecture on first
> upload, and these binaries are not rebuilt by the buildd ( see
> https://wiki.debian.org/SecureBoot#Issues ).
> 
> I would like to know everyone's opinions about these approaches, if you
> agree to go forward with the second approach described above and how do
> we solve the NEW queue policy issue.
> 
> Thanks
> Helen Koike
> 
> _______________________________________________
> Pkg-grub-devel mailing list
> Pkg-grub-devel@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-grub-devel
> 

[toc] | [prev] | [next] | [standalone]

#59146

[Multipart message — attachments visible in raw view] — view raw

On Thu, 2017-10-05 at 14:57 -0300, Helen Koike wrote:
> Hello all,
> 
> As you probably already know, Debian doesn't support the Secure Boot
> chain yet.
> To support it we need to sign Grub and the Kernel with our key, so we
> are discussing the best infrastructure for this workflow.
> 
> The first approach we had was to add a by-hand script in Dak as
> described here
> https://wiki.debian.org/SecureBoot#First_option:_by-hand_script_in_dak
> But this option wasn't well received by the ftpteam

I would like to know what the objection was.

> The second approach we have is to add some debhelper scripts (e.g
> dh_sign...) that will access a signing service which will sign the
> binaries with Debian's key.  We would use the dh_sign... helpers when
> making an extra binary package. buildd would then publish the -signed
> version of the package in the archives.
> Please see a more detailed explanation here
> https://wiki.debian.org/SecureBoot#Second_option:_use_buildd_.2B-_debhelper_instead_of_dak
> A current known issue with this approach is the NEW queue: it requires
> the maintainer to also upload binaries for an architecture on first
> upload, and these binaries are not rebuilt by the buildd ( see
> https://wiki.debian.org/SecureBoot#Issues ).

It also makes all these packages unreproducible, which is a policy
violation.

It also appears to mean that buildds can get anything signed on demand
with no human intervention at all, without all the checks that dak does
on uploads.  This seems to be to substantially raise the risk of
signing evil code and needing to revoke those signatures (or the
signing key).

Plus the reliability issues you pointed out on the wiki.

> I would like to know everyone's opinions about these approaches, if you
> agree to go forward with the second approach described above and how do
> we solve the NEW queue policy issue.

So far as I can see, the second approach is difficult, risky,
unreliable and leads to policy violations.

On the other side... there is an FTP team objection with no documented
explanation.

Ben.

-- 
Ben Hutchings
Humour is the best antidote to reality.

[toc] | [prev] | [next] | [standalone]

#59147

On Mon, Oct 09, 2017 at 02:01:15PM +0100, Ben Hutchings wrote:
>On Thu, 2017-10-05 at 14:57 -0300, Helen Koike wrote:
>> Hello all,
>> 
>> As you probably already know, Debian doesn't support the Secure Boot
>> chain yet.
>> To support it we need to sign Grub and the Kernel with our key, so we
>> are discussing the best infrastructure for this workflow.
>> 
>> The first approach we had was to add a by-hand script in Dak as
>> described here
>> https://wiki.debian.org/SecureBoot#First_option:_by-hand_script_in_dak
>> But this option wasn't well received by the ftpteam
>
>I would like to know what the objection was.

I'll let the ftpteam people expand on that.

>> The second approach we have is to add some debhelper scripts (e.g
>> dh_sign...) that will access a signing service which will sign the
>> binaries with Debian's key.  We would use the dh_sign... helpers when
>> making an extra binary package. buildd would then publish the -signed
>> version of the package in the archives.
>> Please see a more detailed explanation here
>> https://wiki.debian.org/SecureBoot#Second_option:_use_buildd_.2B-_debhelper_instead_of_dak
>> A current known issue with this approach is the NEW queue: it requires
>> the maintainer to also upload binaries for an architecture on first
>> upload, and these binaries are not rebuilt by the buildd ( see
>> https://wiki.debian.org/SecureBoot#Issues ).
>
>It also makes all these packages unreproducible, which is a policy
>violation.

Surely *anything* with a signature is going to be unreproducible
directly, by definition. To check for reproducibility, you'll need to
strip the signatures. Or are you claiming something else?

There's two ways to solve this, that we came up with in discussion:

 * the dh_sign helpers will be configurable on the building machine -
   they can be set up to use whatever key setup is desired. That could
   be local on the machine, or a remote signing service, or simply a
   no-sign pass-through mode. That last mode could validate the
   reproducibility.

 * include a tool to reproducibly just strip the signatures on
   binaries

>It also appears to mean that buildds can get anything signed on demand
>with no human intervention at all, without all the checks that dak does
>on uploads.  This seems to be to substantially raise the risk of
>signing evil code and needing to revoke those signatures (or the
>signing key).

We spoke about this too. Source packages uploaded and eventually built
on the buildds already go through dak and wanna-build, for one.  We
have to trust that mechanism already. What human intervention are you
wanting here? We're also looking at a whitelist of specific packages
that the buildd and signing service will allow to be signed, to reduce
the potential attack surface.

>Plus the reliability issues you pointed out on the wiki.

Right - we're trying to consider the issues as we go.

>> I would like to know everyone's opinions about these approaches, if you
>> agree to go forward with the second approach described above and how do
>> we solve the NEW queue policy issue.
>
>So far as I can see, the second approach is difficult, risky,
>unreliable and leads to policy violations.
>
>On the other side... there is an FTP team objection with no documented
>explanation.

-- 
Steve McIntyre, Cambridge, UK.                                steve@einval.com
"We're the technical experts.  We were hired so that management could
 ignore our recommendations and tell us how to do our jobs."  -- Mike Andrews

[toc] | [prev] | [next] | [standalone]

#59148

On Mon, Oct  9, 2017 at 17:38:56 +0100, Steve McIntyre wrote:

> On Mon, Oct 09, 2017 at 02:01:15PM +0100, Ben Hutchings wrote:
> >It also makes all these packages unreproducible, which is a policy
> >violation.
> 
> Surely *anything* with a signature is going to be unreproducible
> directly, by definition. To check for reproducibility, you'll need to
> strip the signatures. Or are you claiming something else?
> 
No, the previous scheme allowed reproducibility (in the
"dpkg-buildpackage from the source package results in the exact same
.deb files" sense), since the signatures were shipped as part of a
source package.  Attaching fixed signatures to fixed binaries is
reproducible.

Cheers,
Julien

[toc] | [prev] | [next] | [standalone]

#59149

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2017-10-09 at 17:38 +0100, Steve McIntyre wrote:
> On Mon, Oct 09, 2017 at 02:01:15PM +0100, Ben Hutchings wrote:
[...]
> > It also appears to mean that buildds can get anything signed on demand
> > with no human intervention at all, without all the checks that dak does
> > on uploads.  This seems to be to substantially raise the risk of
> > signing evil code and needing to revoke those signatures (or the
> > signing key).
> 
> We spoke about this too. Source packages uploaded and eventually built
> on the buildds already go through dak and wanna-build, for one. We
> have to trust that mechanism already.

It's also audited - every upload is publicly logged, which makes it
hard for an attacker to hide an evil upload.

The archive is signed, rather than individual packages, and those
signatures are valid for a limited period.  So if an evil upload is
successful, we can replace it, warn users, and then by the time it's
forgotten the signature will be invalid.

A signing service would have to implement a separate audit mechanism. 
And code signing can't be time limited so if we sign an evil upload
we'll need to revoke it somehow.

> What human intervention are you wanting here?

A human decides when processing the BYHAND queue whether this binary
upload looks reasonable.  If it's built from an NMU, I would hope this
gets extra scrutiny (perhaps including a query to the maintainer). 
(Ideally that would happen at the source upload stage for packages that
get code-signed.  I'm not sure if that's possible to do.)

Also, dak checks that every binary upload corresponds to a source
upload, so an evil upload will have to be either a source upload or a
replacement of a binary upload.  It seems like the proposed signing
service would allow an attacker that compromises a buildd to get code
signed at any time, independently of a source upload.

> We're also looking at a whitelist of specific packages
> that the buildd and signing service will allow to be signed, to reduce
> the potential attack surface.

This mechanism already exists in dak, of course.

Ben.

> > Plus the reliability issues you pointed out on the wiki.
> 
> Right - we're trying to consider the issues as we go.
> 
> > > I would like to know everyone's opinions about these approaches, if you
> > > agree to go forward with the second approach described above and how do
> > > we solve the NEW queue policy issue.
> > 
> > So far as I can see, the second approach is difficult, risky,
> > unreliable and leads to policy violations.
> > 
> > On the other side... there is an FTP team objection with no documented
> > explanation.
> 
> 
-- 
Ben Hutchings
Humour is the best antidote to reality.

[toc] | [prev] | [next] | [standalone]

#59151

Ben Hutchings writes:
> On Mon, 2017-10-09 at 17:38 +0100, Steve McIntyre wrote:
>> On Mon, Oct 09, 2017 at 02:01:15PM +0100, Ben Hutchings wrote:
> [...]
>> > It also appears to mean that buildds can get anything signed on demand
>> > with no human intervention at all, without all the checks that dak does
>> > on uploads.  This seems to be to substantially raise the risk of
>> > signing evil code and needing to revoke those signatures (or the
>> > signing key).
>> 
>> We spoke about this too. Source packages uploaded and eventually built
>> on the buildds already go through dak and wanna-build, for one. We
>> have to trust that mechanism already.
>
> It's also audited - every upload is publicly logged, which makes it
> hard for an attacker to hide an evil upload.

Does anyone ever look at the upload history of buildds and audits them?

> A signing service would have to implement a separate audit mechanism.
> And code signing can't be time limited so if we sign an evil upload
> we'll need to revoke it somehow.

Hmm, last I asked revocations weren't a concern for kernels with a valid
signature and known vulnerabilities that can be used to bypass the
secure boot foo either.

>> What human intervention are you wanting here?
>
> A human decides when processing the BYHAND queue whether this binary
> upload looks reasonable.

That's really not going to happen.  I don't want to manually process
packages.  I very strongly doubt the other ftp-masters want to either.

What should the human check anyway?  That someone signed the source
upload?  That someone (or something for buildds) signed the binary
upload?  That is already done automatically; looking at the source code
is unreasonable (and might not correspond to binaries anyway when you
consider evil uploads).

> Also, dak checks that every binary upload corresponds to a source
> upload, so an evil upload will have to be either a source upload or a
> replacement of a binary upload.  It seems like the proposed signing
> service would allow an attacker that compromises a buildd to get code
> signed at any time, independently of a source upload.

No, it doesn't.  dak only checks that a package claims to come from a
source package present in the archive; otherwise binNMUs would not work.
For an upload containing only byhand files not even that is checked.

Also we have the problem with "evil uploads" everywhere...  It doesn't
really matter if the kernel (with secure boot) is fine if userland is
not.

(And I'm not even going to start how everyone with upload permissions
can exploit buildds to upload arbitrary packages or insert bad code in
other, unrelated packages built on the same buildd.  Including the
secure boot signed packages.)

>> > On the other side... there is an FTP team objection with no documented
>> > explanation.

 - byhand files for security-master break the security-master ->
   ftp-master sync
 - The autobyhand scripts don't work for uploads that go to NEW.
 - byhand files are not that nice; I would rather not add more of them
 - byhand files aren't publish in the public archive, so harder to see
   what was actually signed.  Nothing enforces the binaries in the *.deb
   and the *.tar.gz are related after all.  Or that a *.deb is present.
 - Not clear how to publish signed binaries for the manual step in
   preparing uploads for the security archive.

I admit I don't like requiring access to a signing service on buildds
either.  It also makes it harder to trust the build process less in the
future (for example by moving it to a VM so it is restricted to do evil
stuff only the the current build, not having access to private keys and
removing access to network services).

Ansgar

[toc] | [prev] | [next] | [standalone]

#59155

On 2017-10-10 03:36 PM, Ansgar Burchardt wrote:
> Ben Hutchings writes:
>> On Mon, 2017-10-09 at 17:38 +0100, Steve McIntyre wrote:
>>> On Mon, Oct 09, 2017 at 02:01:15PM +0100, Ben Hutchings wrote:
>> [...]
>>>> It also appears to mean that buildds can get anything signed on demand
>>>> with no human intervention at all, without all the checks that dak does
>>>> on uploads.  This seems to be to substantially raise the risk of
>>>> signing evil code and needing to revoke those signatures (or the
>>>> signing key).
>>>
>>> We spoke about this too. Source packages uploaded and eventually built
>>> on the buildds already go through dak and wanna-build, for one. We
>>> have to trust that mechanism already.
>>
>> It's also audited - every upload is publicly logged, which makes it
>> hard for an attacker to hide an evil upload.
> 
> Does anyone ever look at the upload history of buildds and audits them?
> 
>> A signing service would have to implement a separate audit mechanism.
>> And code signing can't be time limited so if we sign an evil upload
>> we'll need to revoke it somehow.
> 
> Hmm, last I asked revocations weren't a concern for kernels with a valid
> signature and known vulnerabilities that can be used to bypass the
> secure boot foo either.
> 
>>> What human intervention are you wanting here?
>>
>> A human decides when processing the BYHAND queue whether this binary
>> upload looks reasonable.
> 
> That's really not going to happen.  I don't want to manually process
> packages.  I very strongly doubt the other ftp-masters want to either.
> 
> What should the human check anyway?  That someone signed the source
> upload?  That someone (or something for buildds) signed the binary
> upload?  That is already done automatically; looking at the source code
> is unreasonable (and might not correspond to binaries anyway when you
> consider evil uploads).
> 
>> Also, dak checks that every binary upload corresponds to a source
>> upload, so an evil upload will have to be either a source upload or a
>> replacement of a binary upload.  It seems like the proposed signing
>> service would allow an attacker that compromises a buildd to get code
>> signed at any time, independently of a source upload.
> 
> No, it doesn't.  dak only checks that a package claims to come from a
> source package present in the archive; otherwise binNMUs would not work.
> For an upload containing only byhand files not even that is checked.
> 
> Also we have the problem with "evil uploads" everywhere...  It doesn't
> really matter if the kernel (with secure boot) is fine if userland is
> not.
> 
> (And I'm not even going to start how everyone with upload permissions
> can exploit buildds to upload arbitrary packages or insert bad code in
> other, unrelated packages built on the same buildd.  Including the
> secure boot signed packages.)
> 
>>>> On the other side... there is an FTP team objection with no documented
>>>> explanation.
> 
>  - byhand files for security-master break the security-master ->
>    ftp-master sync
>  - The autobyhand scripts don't work for uploads that go to NEW.
>  - byhand files are not that nice; I would rather not add more of them
>  - byhand files aren't publish in the public archive, so harder to see
>    what was actually signed.  Nothing enforces the binaries in the *.deb
>    and the *.tar.gz are related after all.  Or that a *.deb is present.
>  - Not clear how to publish signed binaries for the manual step in
>    preparing uploads for the security archive.
> 
> I admit I don't like requiring access to a signing service on buildds
> either.  It also makes it harder to trust the build process less in the
> future (for example by moving it to a VM so it is restricted to do evil
> stuff only the the current build, not having access to private keys and
> removing access to network services).
> 
> Ansgar
> 
> _______________________________________________
> Pkg-grub-devel mailing list
> Pkg-grub-devel@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-grub-devel
> 

I did a summary about the current discussion here:
https://wiki.debian.org/SecureBoot#Wrap-up_of_the_discussions_so_far
Feel free to edit this wiki or let me know if I forgot something.

Questions:
* About item 2.1. (reproducible builds), if we strip the signatures from
the binaries before doing the comparison is the reproducible build
criteria acceptable? Can we just strip the signatures and if the result
is the same we consider it reproducible? Or is changing the criteria ok?

* About item 2.2. Would it be ok if we just have a easy revoke mechanism?
In Shim there is already a MokListX (a blacklist of keys or binary
hashes), but the current way to update this list is if the user has
physical access to the machine, but I was talking with the Shim upstream
maintainer and we agreed that we could implement a solution where we
could feed a blacklist to shim, and if this list is signed by the
vendor_cert (aka Debian's key in our case), and if it is a newer version
of the list, shim would update MokListX without requiring user
intervention (it would be an equivalent to KEK for UEFI)
Is this solution acceptable? If we have an easy way to revoke, then we
can easily undo an attacker's work. We can sign everything automatically
(if the package is in a whitelist) without the need for the ftp masters
to review each upload manually.

* Item 2.4 seems the strongest argument to me against the buildd
approach, but the byhand approach seems to complicated, or we need to
reformulate it, any suggestions?

LN Koike

[toc] | [prev] | [next] | [standalone]

#59302

[Multipart message — attachments visible in raw view] — view raw

Nearly three weeks later with no responses. Let's see if we get things
moving...

On Wed, Oct 11, 2017 at 09:48:46PM -0300, Helen Koike wrote:
>
>I did a summary about the current discussion here:
>https://wiki.debian.org/SecureBoot#Wrap-up_of_the_discussions_so_far
>Feel free to edit this wiki or let me know if I forgot something.

I think it's a fair summary, yes.

>Questions:
>* About item 2.1. (reproducible builds), if we strip the signatures from
>the binaries before doing the comparison is the reproducible build
>criteria acceptable? Can we just strip the signatures and if the result
>is the same we consider it reproducible? Or is changing the criteria ok?

I *personally* believe that being able to remove the signatures and
compare binaries is fine for reproducibility purposes. Arguments
otherwise would help here.

>* About item 2.2. Would it be ok if we just have a easy revoke mechanism?
>In Shim there is already a MokListX (a blacklist of keys or binary
>hashes), but the current way to update this list is if the user has
>physical access to the machine, but I was talking with the Shim upstream
>maintainer and we agreed that we could implement a solution where we
>could feed a blacklist to shim, and if this list is signed by the
>vendor_cert (aka Debian's key in our case), and if it is a newer version
>of the list, shim would update MokListX without requiring user
>intervention (it would be an equivalent to KEK for UEFI)

ACK.

>Is this solution acceptable? If we have an easy way to revoke, then we
>can easily undo an attacker's work. We can sign everything automatically
>(if the package is in a whitelist) without the need for the ftp masters
>to review each upload manually.

Right. Wanting to go the revocation route would depend on the
development of yet more new software features. But: this is not
something that any of the other SB-supporting distros seem to be
caring about so far so I don't think it's something we should have to
implement as a pre-requisite.

>* Item 2.4 seems the strongest argument to me against the buildd
>approach, but the byhand approach seems to complicated, or we need to
>reformulate it, any suggestions?

We've been discussing Secure Boot for ~5 years now, and I think it's
an important feature that we should support in Debian. Does anybody
here disagree with that?

Multiple people have spent time designing and implementing pieces of
the solution to make it work in Debian's infrastructure, but we've yet
to make any progress in actually *doing* it.

We've had a suggestion (and initial patches) to do this via
"autobyhand" scripts on ftp-master, but we've finally seen public
objections to that design from ftpmaster. Ben's hopes for that route
seem unrealistic too, in terms of manual processing and checking.

As a second option, in a discussion with Joerg (remotely) and Luke at
DebConf, we thrashed out what we thought was a reasonable alternate
solution to drive signing via the buildds instead. Ben has voiced some
concerns about the security of that path (extending the attack surface
for signed binaries) and reproducibility.

So, we're at an impasse. We've had multiple attempts to get this
going, with various subsets of this group discussing designs but not
*all* the interested parties at each point. We've seen (effective)
vetos of each of those designs, meaning we're stalled. Where do we go
from here? I appreciate that some folks might feel slighted that
they've been left out of some of the discussions. If so, sorry - that
was not intentional. :-( Can we put that behind us and work together
to make a workable solution please?

I propose a sprint to get things moving. I'm prepared to organise it,
and facilitate as much as I can. I recognise that not everybody might
be able to (or willing to) attend in person, but I don't think that
should be a barrier to constructive discussions. Thoughts?

-- 
Steve McIntyre, Cambridge, UK.                                steve@einval.com
Who needs computer imagery when you've got Brian Blessed?

[toc] | [prev] | [next] | [standalone]

#59399

Hi,


On 10/31/2017 01:58 PM, Steve McIntyre wrote:
> Nearly three weeks later with no responses. Let's see if we get things
> moving...
>
> On Wed, Oct 11, 2017 at 09:48:46PM -0300, Helen Koike wrote:
>> I did a summary about the current discussion here:
>> https://wiki.debian.org/SecureBoot#Wrap-up_of_the_discussions_so_far
>> Feel free to edit this wiki or let me know if I forgot something.
> I think it's a fair summary, yes.
>
>> Questions:
>> * About item 2.1. (reproducible builds), if we strip the signatures from
>> the binaries before doing the comparison is the reproducible build
>> criteria acceptable? Can we just strip the signatures and if the result
>> is the same we consider it reproducible? Or is changing the criteria ok?
> I *personally* believe that being able to remove the signatures and
> compare binaries is fine for reproducibility purposes. Arguments
> otherwise would help here.
>
>> * About item 2.2. Would it be ok if we just have a easy revoke mechanism?
>> In Shim there is already a MokListX (a blacklist of keys or binary
>> hashes), but the current way to update this list is if the user has
>> physical access to the machine, but I was talking with the Shim upstream
>> maintainer and we agreed that we could implement a solution where we
>> could feed a blacklist to shim, and if this list is signed by the
>> vendor_cert (aka Debian's key in our case), and if it is a newer version
>> of the list, shim would update MokListX without requiring user
>> intervention (it would be an equivalent to KEK for UEFI)
> ACK.
>
>> Is this solution acceptable? If we have an easy way to revoke, then we
>> can easily undo an attacker's work. We can sign everything automatically
>> (if the package is in a whitelist) without the need for the ftp masters
>> to review each upload manually.
> Right. Wanting to go the revocation route would depend on the
> development of yet more new software features. But: this is not
> something that any of the other SB-supporting distros seem to be
> caring about so far so I don't think it's something we should have to
> implement as a pre-requisite.
>
>> * Item 2.4 seems the strongest argument to me against the buildd
>> approach, but the byhand approach seems to complicated, or we need to
>> reformulate it, any suggestions?
> We've been discussing Secure Boot for ~5 years now, and I think it's
> an important feature that we should support in Debian. Does anybody
> here disagree with that?
>
> Multiple people have spent time designing and implementing pieces of
> the solution to make it work in Debian's infrastructure, but we've yet
> to make any progress in actually *doing* it.
>
> We've had a suggestion (and initial patches) to do this via
> "autobyhand" scripts on ftp-master, but we've finally seen public
> objections to that design from ftpmaster. Ben's hopes for that route
> seem unrealistic too, in terms of manual processing and checking.
>
> As a second option, in a discussion with Joerg (remotely) and Luke at
> DebConf, we thrashed out what we thought was a reasonable alternate
> solution to drive signing via the buildds instead. Ben has voiced some
> concerns about the security of that path (extending the attack surface
> for signed binaries) and reproducibility.
>
> So, we're at an impasse. We've had multiple attempts to get this
> going, with various subsets of this group discussing designs but not
> *all* the interested parties at each point. We've seen (effective)
> vetos of each of those designs, meaning we're stalled. Where do we go
> from here? I appreciate that some folks might feel slighted that
> they've been left out of some of the discussions. If so, sorry - that
> was not intentional. :-( Can we put that behind us and work together
> to make a workable solution please?
>
> I propose a sprint to get things moving. I'm prepared to organise it,
> and facilitate as much as I can. I recognise that not everybody might
> be able to (or willing to) attend in person, but I don't think that
> should be a barrier to constructive discussions. Thoughts?
>

I think a Sprint about Secure Boot would be great, we just need to make 
sure that at least the people who disagree with one approach or the 
other will be present.
@ftpmasters and @Ben, if you could reply if you would be willing to 
attend it would be great, so we can start organizing it and work 
together to find a good solution.

Thanks
Helen

[toc] | [prev] | [next] | [standalone]

#59470

[Multipart message — attachments visible in raw view] — view raw

On Wed, 2017-11-15 at 11:18 -0200, Helen Koike wrote:
[...]
> I think a Sprint about Secure Boot would be great, we just need to make 
> sure that at least the people who disagree with one approach or the 
> other will be present.
> @ftpmasters and @Ben, if you could reply if you would be willing to 
> attend it would be great, so we can start organizing it and work 
> together to find a good solution.

I am willing to attend and work on this if we can agree on the
direction beforehand.

Ben.

-- 
Ben Hutchings
Beware of programmers who carry screwdrivers. - Leonard Brandwein

[toc] | [prev] | [next] | [standalone]

#59480

On Wed, Nov 22, 2017 at 11:11:02PM +0000, Ben Hutchings wrote:
>On Wed, 2017-11-15 at 11:18 -0200, Helen Koike wrote:
>[...]
>> I think a Sprint about Secure Boot would be great, we just need to make 
>> sure that at least the people who disagree with one approach or the 
>> other will be present.
>> @ftpmasters and @Ben, if you could reply if you would be willing to 
>> attend it would be great, so we can start organizing it and work 
>> together to find a good solution.
>
>I am willing to attend and work on this if we can agree on the
>direction beforehand.

Cool.

-- 
Steve McIntyre, Cambridge, UK.                                steve@einval.com
Who needs computer imagery when you've got Brian Blessed?

[toc] | [prev] | [next] | [standalone]

#59468

[Multipart message — attachments visible in raw view] — view raw

On Tue, 2017-10-31 at 15:58 +0000, Steve McIntyre wrote:
[...]
> On Wed, Oct 11, 2017 at 09:48:46PM -0300, Helen Koike wrote:
[...]
> > Is this solution acceptable? If we have an easy way to revoke, then we
> > can easily undo an attacker's work. We can sign everything automatically
> > (if the package is in a whitelist) without the need for the ftp masters
> > to review each upload manually.
> 
> Right. Wanting to go the revocation route would depend on the
> development of yet more new software features. But: this is not
> something that any of the other SB-supporting distros seem to be
> caring about so far so I don't think it's something we should have to
> implement as a pre-requisite.
[...]

As I understand it, SUSE has implemented some kind of downgrade
prevention.

Ben.

-- 
Ben Hutchings
Beware of programmers who carry screwdrivers. - Leonard Brandwein

[toc] | [prev] | [next] | [standalone]

#59469

[Multipart message — attachments visible in raw view] — view raw

On Wed, 2017-10-11 at 21:48 -0300, Helen Koike wrote:
[...]
> I did a summary about the current discussion here:
> https://wiki.debian.org/SecureBoot#Wrap-up_of_the_discussions_so_far
> Feel free to edit this wiki or let me know if I forgot something.
> 
> Questions:
> * About item 2.1. (reproducible builds), if we strip the signatures from
> the binaries before doing the comparison is the reproducible build
> criteria acceptable? Can we just strip the signatures and if the result
> is the same we consider it reproducible? Or is changing the criteria ok?

The Reproducible Builds project has consistently set the standard that
results must be bit-for-bit identical.  I don't think we should expect
it to add an exception that requires parsing archives and executables
to verify that they're "mostly reproducible".

> * About item 2.2. Would it be ok if we just have a easy revoke mechanism?
> In Shim there is already a MokListX (a blacklist of keys or binary
> hashes), but the current way to update this list is if the user has
> physical access to the machine, but I was talking with the Shim upstream
> maintainer and we agreed that we could implement a solution where we
> could feed a blacklist to shim, and if this list is signed by the
> vendor_cert (aka Debian's key in our case), and if it is a newer version
> of the list, shim would update MokListX without requiring user
> intervention (it would be an equivalent to KEK for UEFI)
> Is this solution acceptable? If we have an easy way to revoke, then we
> can easily undo an attacker's work. We can sign everything automatically
> (if the package is in a whitelist) without the need for the ftp masters
> to review each upload manually.

I think we will need some sort of reliable mechanism to revoke
signatures, however we implement signing.

> * Item 2.4 seems the strongest argument to me against the buildd
> approach, but the byhand approach seems to complicated, or we need to
> reformulate it, any suggestions?

I think that it should be possible to mitigate these problems by
combining the proposed signing service with the earlier plan of doing
two source uploads:

Firstly, the signing service would not provide the signatures in plain
text but would encrypt them using a developer's GPG key (or multiple
keys).  Key selection might be done by specifying the key fingerprint
in the source package (which could be problematic for binNMUs), or in
the signing service configuration (which would effectively prevent
source NMUs).  This makes the signing service useless to an attacker
who only compromises a buildd briefly.

Secondly, the detached signatures would be uploaded as an extra package
to a separate archive suite, similar to the way debug symbol packages
are now handled.  Those extra packages could easily be ignored when
attempting to reproduce the build.  But I don't know whether dak's
logic for debug symbol suites is sufficiently general to handle this.

Ben.

-- 
Ben Hutchings
Beware of programmers who carry screwdrivers. - Leonard Brandwein

[toc] | [prev] | [next] | [standalone]

#59481

On Wed, Nov 22, 2017 at 11:07:02PM +0000, Ben Hutchings wrote:
>On Wed, 2017-10-11 at 21:48 -0300, Helen Koike wrote:
>[...]
>> I did a summary about the current discussion here:
>> https://wiki.debian.org/SecureBoot#Wrap-up_of_the_discussions_so_far
>> Feel free to edit this wiki or let me know if I forgot something.
>> 
>> Questions:
>> * About item 2.1. (reproducible builds), if we strip the signatures from
>> the binaries before doing the comparison is the reproducible build
>> criteria acceptable? Can we just strip the signatures and if the result
>> is the same we consider it reproducible? Or is changing the criteria ok?
>
>The Reproducible Builds project has consistently set the standard that
>results must be bit-for-bit identical.  I don't think we should expect
>it to add an exception that requires parsing archives and executables
>to verify that they're "mostly reproducible".

So there are always going to be things that *can't* be reproducible
then. Anything with signatures directly applied is fundamentally
incompatible with that standard. Meh, I'm not going to lose any sleep
over it.

...

>> * Item 2.4 seems the strongest argument to me against the buildd
>> approach, but the byhand approach seems to complicated, or we need to
>> reformulate it, any suggestions?
>
>I think that it should be possible to mitigate these problems by
>combining the proposed signing service with the earlier plan of doing
>two source uploads:
>
>Firstly, the signing service would not provide the signatures in plain
>text but would encrypt them using a developer's GPG key (or multiple
>keys).  Key selection might be done by specifying the key fingerprint
>in the source package (which could be problematic for binNMUs), or in
>the signing service configuration (which would effectively prevent
>source NMUs).  This makes the signing service useless to an attacker
>who only compromises a buildd briefly.

OK, that sounds like a more useful compromise.

>Secondly, the detached signatures would be uploaded as an extra package
>to a separate archive suite, similar to the way debug symbol packages
>are now handled.  Those extra packages could easily be ignored when
>attempting to reproduce the build.  But I don't know whether dak's
>logic for debug symbol suites is sufficiently general to handle this.

Pass. Ansgar?

-- 
Steve McIntyre, Cambridge, UK.                                steve@einval.com
"This dress doesn't reverse." -- Alden Spiess

[toc] | [prev] | [next] | [standalone]

#59465

[Multipart message — attachments visible in raw view] — view raw

Sorry for taking so long to respond to this.

On Tue, 2017-10-10 at 20:36 +0200, Ansgar Burchardt wrote:
> Ben Hutchings writes:
> > On Mon, 2017-10-09 at 17:38 +0100, Steve McIntyre wrote:
> > > On Mon, Oct 09, 2017 at 02:01:15PM +0100, Ben Hutchings wrote:
> > 
> > [...]
> > > > It also appears to mean that buildds can get anything signed on demand
> > > > with no human intervention at all, without all the checks that dak does
> > > > on uploads.  This seems to be to substantially raise the risk of
> > > > signing evil code and needing to revoke those signatures (or the
> > > > signing key).
> > > 
> > > We spoke about this too. Source packages uploaded and eventually built
> > > on the buildds already go through dak and wanna-build, for one. We
> > > have to trust that mechanism already.
> > 
> > It's also audited - every upload is publicly logged, which makes it
> > hard for an attacker to hide an evil upload.
> 
> Does anyone ever look at the upload history of buildds and audits them?

I don't know.  But as I understand it, nation-state attackers generally
don't like to leave traces, so public logs can be a valuable deterrent.

> > A signing service would have to implement a separate audit mechanism.
> > And code signing can't be time limited so if we sign an evil upload
> > we'll need to revoke it somehow.
> 
> Hmm, last I asked revocations weren't a concern for kernels with a valid
> signature and known vulnerabilities that can be used to bypass the
> secure boot foo either.

I suppose there's not that much difference between those in terms of
threat.  But I think that if we directly signed malicious code we would
be practically (or even legally) obliged to revoke it asap.

> > > What human intervention are you wanting here?
> > 
> > A human decides when processing the BYHAND queue whether this binary
> > upload looks reasonable.
> 
> That's really not going to happen.  I don't want to manually process
> packages.  I very strongly doubt the other ftp-masters want to either.
> 
> What should the human check anyway?  That someone signed the source
> upload?  That someone (or something for buildds) signed the binary
> upload?  That is already done automatically; looking at the source code
> is unreasonable (and might not correspond to binaries anyway when you
> consider evil uploads).

They can check whether the binary upload actually came from a buildd,
and whether there was a recent source upload or binNMU request.  It's
not that strong a check, but seems somewhat better than an on-demand
signing service.

> > Also, dak checks that every binary upload corresponds to a source
> > upload, so an evil upload will have to be either a source upload or a
> > replacement of a binary upload.  It seems like the proposed signing
> > service would allow an attacker that compromises a buildd to get code
> > signed at any time, independently of a source upload.
> 
> No, it doesn't.  dak only checks that a package claims to come from a
> source package present in the archive; otherwise binNMUs would not work.

Fair point.

> For an upload containing only byhand files not even that is checked.

I don't think anyone proposed to make uplaods that only contain byhand
files.

> Also we have the problem with "evil uploads" everywhere...  It doesn't
> really matter if the kernel (with secure boot) is fine if userland is
> not.

I know, but we have to start somewhere.  (And Matthew Garrett is
working on the userland side for Google's internal Debian derivative.)

> (And I'm not even going to start how everyone with upload permissions
> can exploit buildds to upload arbitrary packages or insert bad code in
> other, unrelated packages built on the same buildd.  Including the
> secure boot signed packages.)

This is all true, but it sounds like you are succumbing to security
nihilism: "we can't make this perfectly secure, so we shouldn't make it
more secure".

> > > > On the other side... there is an FTP team objection with no documented
> > > > explanation.
> 
>  - byhand files for security-master break the security-master ->
>    ftp-master sync
>  - The autobyhand scripts don't work for uploads that go to NEW.

Damn.

>  - byhand files are not that nice; I would rather not add more of them

I don't think any solution is going to be "nice".

>  - byhand files aren't publish in the public archive, so harder to see
>    what was actually signed.  Nothing enforces the binaries in the *.deb
>    and the *.tar.gz are related after all.  Or that a *.deb is present.

That's true, but seems fixable.

>  - Not clear how to publish signed binaries for the manual step in
>    preparing uploads for the security archive.

That seems like a relatively easy problem.

Ben.

> I admit I don't like requiring access to a signing service on buildds
> either.  It also makes it harder to trust the build process less in the
> future (for example by moving it to a VM so it is restricted to do evil
> stuff only the the current build, not having access to private keys and
> removing access to network services).

-- 
Ben Hutchings
Beware of programmers who carry screwdrivers. - Leonard Brandwein

[toc] | [prev] | [next] | [standalone]

#59491

]] Ben Hutchings 

(Sorry for the massive Cc list, it'd be nicer if this could all just go
to -efi, but I don't know if everybody is subscribed to that list, so…)

> Sorry for taking so long to respond to this.
> 
> On Tue, 2017-10-10 at 20:36 +0200, Ansgar Burchardt wrote:
> > Ben Hutchings writes:
> > > On Mon, 2017-10-09 at 17:38 +0100, Steve McIntyre wrote:
> > > > On Mon, Oct 09, 2017 at 02:01:15PM +0100, Ben Hutchings wrote:
> > > 
> > > [...]
> > > > > It also appears to mean that buildds can get anything signed on demand
> > > > > with no human intervention at all, without all the checks that dak does
> > > > > on uploads.  This seems to be to substantially raise the risk of
> > > > > signing evil code and needing to revoke those signatures (or the
> > > > > signing key).
> > > > 
> > > > We spoke about this too. Source packages uploaded and eventually built
> > > > on the buildds already go through dak and wanna-build, for one. We
> > > > have to trust that mechanism already.
> > > 
> > > It's also audited - every upload is publicly logged, which makes it
> > > hard for an attacker to hide an evil upload.
> > 
> > Does anyone ever look at the upload history of buildds and audits them?
> 
> I don't know.  But as I understand it, nation-state attackers generally
> don't like to leave traces, so public logs can be a valuable deterrent.

We can easily make the signing logs public too, possibly with some sort
of time delay mechanism if needed for handling of embargoed security.

Also, while we don't need to solve it (yet), it would be nice if the
proposed solution we go for is able to solve the case of signing all
binaries in the archive (or at least don't make it impossible to do), if
we decide to do something like Matthew talks about in
https://debconf17.debconf.org/talks/174/ .

Cheers,
-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

[toc] | [prev] | [next] | [standalone]

#59172

On Mon, Oct 09, 2017 at 07:00:14PM +0100, Ben Hutchings wrote:
>On Mon, 2017-10-09 at 17:38 +0100, Steve McIntyre wrote:
>> On Mon, Oct 09, 2017 at 02:01:15PM +0100, Ben Hutchings wrote:
>[...]
>> > It also appears to mean that buildds can get anything signed on demand
>> > with no human intervention at all, without all the checks that dak does
>> > on uploads.  This seems to be to substantially raise the risk of
>> > signing evil code and needing to revoke those signatures (or the
>> > signing key).
>> 
>> We spoke about this too. Source packages uploaded and eventually built
>> on the buildds already go through dak and wanna-build, for one. We
>> have to trust that mechanism already.
>
>It's also audited - every upload is publicly logged, which makes it
>hard for an attacker to hide an evil upload.
>
>The archive is signed, rather than individual packages, and those
>signatures are valid for a limited period.  So if an evil upload is
>successful, we can replace it, warn users, and then by the time it's
>forgotten the signature will be invalid.
>
>A signing service would have to implement a separate audit mechanism. 
>And code signing can't be time limited so if we sign an evil upload
>we'll need to revoke it somehow.

Right, thanks for explaining. I understand your concerns much better
now.

>> What human intervention are you wanting here?
>
>A human decides when processing the BYHAND queue whether this binary
>upload looks reasonable.  If it's built from an NMU, I would hope this
>gets extra scrutiny (perhaps including a query to the maintainer). 
>(Ideally that would happen at the source upload stage for packages that
>get code-signed.  I'm not sure if that's possible to do.)

The plan for the ftpmaster route (as communicated to me) was *not*
necessarily going to need actual manual processing of the BYHAND queue
for bits needing signing. Is that something we can (should?)
legitimately expect - every signed package to get extra manual
checking by ftpteam?

>Also, dak checks that every binary upload corresponds to a source
>upload, so an evil upload will have to be either a source upload or a
>replacement of a binary upload.  It seems like the proposed signing
>service would allow an attacker that compromises a buildd to get code
>signed at any time, independently of a source upload.

True, yes. The *normal* flow would also see output going through dak
and validating the source-binary relationship, but a compromised
buildd could of course sidestep that, by simply not uploading the
results at all.

On the flip side, how far are we going to get if we can't trust the
buildds? Reproducibility helps, but only if the source uploader can
already generate matching builds for all the packages on all the
architectures we want to support.

>> We're also looking at a whitelist of specific packages
>> that the buildd and signing service will allow to be signed, to reduce
>> the potential attack surface.
>
>This mechanism already exists in dak, of course.

Nod.

-- 
Steve McIntyre, Cambridge, UK.                                steve@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.

[toc] | [prev] | [next] | [standalone]

#59466

[Multipart message — attachments visible in raw view] — view raw

On Tue, 2017-10-10 at 13:19 +0100, Steve McIntyre wrote:
> On Mon, Oct 09, 2017 at 07:00:14PM +0100, Ben Hutchings wrote:
> > On Mon, 2017-10-09 at 17:38 +0100, Steve McIntyre wrote:
[...]
> > > What human intervention are you wanting here?
> > 
> > A human decides when processing the BYHAND queue whether this binary
> > upload looks reasonable.  If it's built from an NMU, I would hope this
> > gets extra scrutiny (perhaps including a query to the maintainer). 
> > (Ideally that would happen at the source upload stage for packages that
> > get code-signed.  I'm not sure if that's possible to do.)
> 
> The plan for the ftpmaster route (as communicated to me) was *not*
> necessarily going to need actual manual processing of the BYHAND queue
> for bits needing signing. Is that something we can (should?)
> legitimately expect - every signed package to get extra manual
> checking by ftpteam?

The rate of source package uploads requiring binaries to be signed
should not be very high.  Looking at the upload history of linux and
grub2, I would say no more than twice a week on average.  I would
assume (maybe wrongly) that the byhand uploads of binaries could then
be dealt with in batches.

For kernel uploads, only the kernel image should require signing with
an HSM, so the signing process should not be nearly as slow as Julien
reported.  (And if I manage to implement Merkle hashing of modules,
they won't need to be signed at all.)

[...]
> On the flip side, how far are we going to get if we can't trust the
> buildds? Reproducibility helps, but only if the source uploader can
> already generate matching builds for all the packages on all the
> architectures we want to support.
[...]

It would be desirable to reduce the amount of trust we put in any given
buildd.  I feel that allowing them to get signatures at any time is
going in the opposite direction.

Ben.

-- 
Ben Hutchings
Beware of programmers who carry screwdrivers. - Leonard Brandwein

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web