Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.kernel > #82327 > unrolled thread

Updates to linux-firmware

Back to article view | Back to linux.debian.kernel

Contents

  Updates to linux-firmware "Mark Pearson" <mpearson-lenovo@squebb.ca> - 2024-04-26 21:00 +0200
    Re: Updates to linux-firmware "Mark Pearson" <mpearson-lenovo@squebb.ca> - 2024-05-06 18:00 +0200
    Re: Updates to linux-firmware Didier 'OdyX' Raboud <odyx@debian.org> - 2024-05-07 08:30 +0200
      Re: Updates to linux-firmware Diederik de Haas <didi.debian@cknow.org> - 2024-05-07 11:30 +0200
        Re: Updates to linux-firmware "Mark Pearson" <mpearson-lenovo@squebb.ca> - 2024-05-07 16:10 +0200
          Re: Updates to linux-firmware Diederik de Haas <didi.debian@cknow.org> - 2024-05-07 17:20 +0200
            Re: Updates to linux-firmware "Mark Pearson" <mpearson-lenovo@squebb.ca> - 2024-05-07 19:50 +0200
              Re: Updates to linux-firmware Diederik de Haas <didi.debian@cknow.org> - 2024-05-07 22:10 +0200
      Re: Updates to linux-firmware "Mark Pearson" <mpearson-lenovo@squebb.ca> - 2024-05-07 15:50 +0200
    Re: Updates to linux-firmware Ben Hutchings <benh@debian.org> - 2024-07-16 15:30 +0200
      Re: Updates to linux-firmware "Mark Pearson" <mpearson-lenovo@squebb.ca> - 2024-07-16 16:40 +0200
      Re: Updates to linux-firmware Martin Steigerwald <martin@lichtvoll.de> - 2024-07-16 17:10 +0200
        Re: Updates to linux-firmware Martin Steigerwald <martin@lichtvoll.de> - 2024-07-16 17:20 +0200
        Re: Updates to linux-firmware Martin Steigerwald <martin@lichtvoll.de> - 2024-08-21 12:40 +0200

#82327 — Updates to linux-firmware

Hi Didier & Ben,

I hope all is well.
I wanted to check in with you guys, as with the new 2024 platforms coming out there are some updates to linux-firmware - for CPU, GPU, Wifi, Bluetooth, etc devices

I checked and it looked like firmware-nonfree and it hadn't been updated since last year - and I wondered if that is something I can help with? I'd like to make sure the Debian experience on this years platforms is good out-of-the-box.

Let me know if that would be helpful, and any pointers on how best to contribute and be useful

Thanks
Mark

PS - in case anyone notices, I've mothballed my old markpearson@lenovo.com address because the Lenovo outlook servers there became horribly unusable. I am using my personal domain for open source collaboration instead. 
I still have access to that address (if anybody wants to check I am who I say I am) and am still employed at Lenovo (if that makes any difference to anything)

[toc] | [next] | [standalone]

#82362

Hi all,

Thought I'd check in and see if my previous email was missed in inbox swamp.

Please let me know what I can do to help here.
Not sure if it's helpful - but I did some bugzilla tickets, as I'm guessing that's part of the process. FYI
 - 1070647 - firmware-misc-nonfree (graphics FW)
 - 1070648 - firmware-iwlwifi (wifi)
 - 1070650 - kernel (everything)

If I can be involved and/or useful - please let me know how.
Thanks!
Mark

On Fri, Apr 26, 2024, at 2:35 PM, Mark Pearson wrote:
> Hi Didier & Ben,
>
> I hope all is well.
> I wanted to check in with you guys, as with the new 2024 platforms 
> coming out there are some updates to linux-firmware - for CPU, GPU, 
> Wifi, Bluetooth, etc devices
>
> I checked and it looked like firmware-nonfree and it hadn't been 
> updated since last year - and I wondered if that is something I can 
> help with? I'd like to make sure the Debian experience on this years 
> platforms is good out-of-the-box.
>
> Let me know if that would be helpful, and any pointers on how best to 
> contribute and be useful
>
> Thanks
> Mark
>
> PS - in case anyone notices, I've mothballed my old 
> markpearson@lenovo.com address because the Lenovo outlook servers there 
> became horribly unusable. I am using my personal domain for open source 
> collaboration instead. 
> I still have access to that address (if anybody wants to check I am who 
> I say I am) and am still employed at Lenovo (if that makes any 
> difference to anything)

[toc] | [prev] | [next] | [standalone]

#82376

Hello there Mark,

thanks for your email, and for the followup ping; weeks are well-filled with 
$job, baby and other involvements.

Regarding firmware-nonfree updates, I have related my last-years' experience 
on d-devel: https://lists.debian.org/msgid-search/2390124.3VsfAaAtOV@turnagra

I insist: I am absolutely not implying that Ben is doing anything wrong; I 
just observed that he was mostly alone in a bottleneck position for that 
package.

What I did back then to try to help getting the firmware package in better 
shape was to dive in the (complex) packaging and try to produce "ready-to-
merge" merge-requests for new upstream releases, fixes, etc, then pinging Ben 
at (what I considered) reasonable intervals. I see that Diederik (CC'ed) has 
started to do the same for recent versions.

The package is complex and a mined land of legal concerns, the upstream repo 
needs repacking, and the (upstream & debian/) repositories are split; I have 
not found this a very pleasant experience, sadly. And again; no-one to blame 
here: there _is_ tooling to help, and immense work has been put towards making 
this manageable! I'm merely saying it's not a matter of running "debian/rules 
get-new-upstream" at all.

From where I sit (that is: I only modestly contributed in the past, and cannot 
commit to much these days), there are two angles to address:
A) lift the bottleneck: there needs to be more Debian Developers confident to 
comment and merge MRs, and then upload.
B) regular commitment: I feel it would be easier to get regular updates 
merged, rather than doing the huge batch shortly before release; that would 
also appease the tensions that many probably feel when they get new hardware 
but no available package.

As to how I can get involved in any of the above; I can't reasonably add this 
to my plate these days (and would rather _not do_, than commit to do and 
fail).

But I wonder if some financial sponsorship (if Lenovo and others would be open 
to something like that) could help, via Freexian for example (disclaimer: not 
for me to do it; I'm not a Freexian person). I'd be concerned that this would 
only solve B above though. It would feel like a 1-2 hours job / week.

I hope the above helps you navigate that question, and that it helps the 
situation in general.

Best,

Didier

Le vendredi, 26 avril 2024, 20.35:41 h CEST Mark Pearson a écrit :
> Hi Didier & Ben,
> 
> I hope all is well.
> I wanted to check in with you guys, as with the new 2024 platforms coming
> out there are some updates to linux-firmware - for CPU, GPU, Wifi,
> Bluetooth, etc devices
> 
> I checked and it looked like firmware-nonfree and it hadn't been updated
> since last year - and I wondered if that is something I can help with? I'd
> like to make sure the Debian experience on this years platforms is good
> out-of-the-box.
> 
> Let me know if that would be helpful, and any pointers on how best to
> contribute and be useful
> 
> Thanks
> Mark
> 
> PS - in case anyone notices, I've mothballed my old markpearson@lenovo.com
> address because the Lenovo outlook servers there became horribly unusable.
> I am using my personal domain for open source collaboration instead. I
> still have access to that address (if anybody wants to check I am who I say
> I am) and am still employed at Lenovo (if that makes any difference to
> anything)


-- 
    OdyX

[toc] | [prev] | [next] | [standalone]

#82378

[Multipart message — attachments visible in raw view] — view raw

On Tuesday, 7 May 2024 08:25:23 CEST Didier 'OdyX' Raboud wrote:
> What I did back then to try to help getting the firmware package in better
> shape was to dive in the (complex) packaging and try to produce "ready-to-
> merge" merge-requests for new upstream releases, fixes, etc, then pinging
> Ben at (what I considered) reasonable intervals. I see that Diederik
> (CC'ed) has started to do the same for recent versions.

What I did was indeed my effort to get things into better shape with the hope/
goal that it would be easier to do going forward. Which in turn hopefully 
means that it would be done more regularly which would not make it a 
monumental task ... which no-one is looking forward to do or has the capacity 
(time/motivation/etc) to do.

> The package is complex and a mined land of legal concerns,

It was a HUGE amount of work, partly because I wanted to get things into what 
I consider a better shape. Another part which made it *look* complex, were 
inconsistencies. In one of my commits I determined in which sequence the 
control fields were going to get placed and then fixed all the config files to 
conform to that sequence. I also ordered the entries alphabetically.

I haven't documented it (yet), but I followed a different procedure then 'the 
official one', which I think makes it easier and better ... lowering the barrier 
for future contributions/MRs.

It was basically the following:
1) Have the upstream repo locally and checkout the next tag/release and open
     that with gitk
2) Start a new branch in your local Debian firmware-nonfree repo
3) Go through each commit and decide what to do with it
  - Upstream merge commit: ignore it
  - Upstream working on their own infra: ignore it. Unless it has an effect on
    (future) Debian packaging, then add it to d/changelog. And use the
    secondary commit message to properly document it.
  - Updated version of firmware files: if it has a(n explicit) version number,
    update the version in the appropriate ``defines`` file.
 - New links (documented in WHENCE): Add that entry to the ``files:`` section
   of the Debian ``defines`` file.
 - New files: Add an entry to the `files:` section and add a Stanza for it with
   its 'ID'/Name, description and if available the version number
 - Check whether d/copyright covers any new files and if it doesn't expand the
   'regex' so that it does if the license is already known. Otherwise add an
   entry and the license text to d/copyright*

With updated versions or new links or new files, add an entry to d/changelog 
which is mostly or fully upstream's primary commit message, prepended by the 
Debian package name. This way there's an almost 1-on-1 match between the 
Debian and upstream's commit (message). This should also make reviews easier.

I ordered the entries by Debian package name, so that if someone only has AMD 
graphics firmware, it doesn't have to read the full changelog, only the part 
which is about amd-graphics. Upstream's git commit log sequence is rather 
random, which isn't useful or helpful to our users (IMO ofc)

*) I did improve d/copyright too, but not fully. All the license texts have 
been moved to the bottom, but I didn't fix the sequencing of all the stanzas.

> the upstream repo needs repacking, 

Upstream now also produces a deb package per tag/release, but that's 350MB in 
size ... and growing. And it includes firmware which isn't DFSG compliant and 
therefor we shouldn't ship or f.e. can only be shipped as part of the kernel, 
so we can't distribute them.

> and the (upstream & debian/) repositories are split; 

I think the split is very useful, especially for our users. Some users only 
need one or a few small firmware files. Not having to download >350MB for that 
for every upstream/Debian release/version is quite useful. Not everyone has 
unmetered GbE internet connection ... or wants to waste 300+MB for files for 
which they have no need/use.

> I have not found this a very pleasant experience, sadly. 

I didn't find it pleasant either, but I hope/expect that with my updates and 
restructuring it shouldn't be that bad either.
When things are kept up-to-date it's quite doable, but when you have to work 
through months of 'backlog' the 'fun' quickly diminishes.

> And again; no-one to blame here: there _is_ tooling to help, and immense
> work has been put towards making this manageable! I'm merely saying it's not
> a matter of running "debian/rules get-new-upstream" at all.

The tooling is mostly great.
*I* do think that the script which imports upstream's changelog should go 
though as it makes 'you' think you're quickly done. But if you want to do 
things right, you should still go through all the commits to actually update 
the versions and add the new links and new files.
If you don't, and the temptations is IMO rather high, then you'd create 
another monumental task like the one I went through.
And that REALLY did not make me happy or considered fun. At all.

As I see it, the primary problem is the lack of people actively contributing 
to the Debian kernel team's work. In general.

The amount of open *kernel* bugs is usually around 800. And that's mostly due 
to periodic mass closing of bugs. I have been triaging (kernel) bugs in the 
last say 2 years, but it was mostly me and Salvatore doing that work.
I asked for assistance but I didn't see anyone step up. And I plan to shift 
(part of) my time/focus to other things and my worry is that that means that 
bug reports get completely unanswered. Like before.
And having 800+ kernel bugs open isn't helping motivation to spend time on 
non-free stuff like firmware.

Most of the time triaging (kernel) bugs isn't that hard . It usually boils 
down to finding when the issue surfaced (last working version vs first non-
working version), possibly/ideally bisected to a specific commit and then 
finding the appropriate upstream maintainers and let the bug reporter and the 
upstream maintainer take it from there.
It's just time consuming. 

FTR: Offering to help: great.
Creating NEW bugs with vague/non-actionable descriptions basically asking for 
new upstream versions ... of which there are already several open in the BTS:
not so great. It just makes the mountain (of work) higher.

My 0.02

[toc] | [prev] | [next] | [standalone]

#82380

Hi Diederik,

On Tue, May 7, 2024, at 5:22 AM, Diederik de Haas wrote:
> On Tuesday, 7 May 2024 08:25:23 CEST Didier 'OdyX' Raboud wrote:
>> What I did back then to try to help getting the firmware package in better
>> shape was to dive in the (complex) packaging and try to produce "ready-to-
>> merge" merge-requests for new upstream releases, fixes, etc, then pinging
>> Ben at (what I considered) reasonable intervals. I see that Diederik
>> (CC'ed) has started to do the same for recent versions.
>
> What I did was indeed my effort to get things into better shape with the hope/
> goal that it would be easier to do going forward. Which in turn hopefully 
> means that it would be done more regularly which would not make it a 
> monumental task ... which no-one is looking forward to do or has the capacity 
> (time/motivation/etc) to do.
>
>> The package is complex and a mined land of legal concerns,
>
> It was a HUGE amount of work, partly because I wanted to get things into what 
> I consider a better shape. Another part which made it *look* complex, were 
> inconsistencies. In one of my commits I determined in which sequence the 
> control fields were going to get placed and then fixed all the config files to 
> conform to that sequence. I also ordered the entries alphabetically.
>
> I haven't documented it (yet), but I followed a different procedure 
> then 'the 
> official one', which I think makes it easier and better ... lowering 
> the barrier 
> for future contributions/MRs.
>
> It was basically the following:
> 1) Have the upstream repo locally and checkout the next tag/release and open
>      that with gitk
> 2) Start a new branch in your local Debian firmware-nonfree repo
> 3) Go through each commit and decide what to do with it
>   - Upstream merge commit: ignore it
>   - Upstream working on their own infra: ignore it. Unless it has an effect on
>     (future) Debian packaging, then add it to d/changelog. And use the
>     secondary commit message to properly document it.
>   - Updated version of firmware files: if it has a(n explicit) version number,
>     update the version in the appropriate ``defines`` file.
>  - New links (documented in WHENCE): Add that entry to the ``files:`` section
>    of the Debian ``defines`` file.
>  - New files: Add an entry to the `files:` section and add a Stanza for it with
>    its 'ID'/Name, description and if available the version number
>  - Check whether d/copyright covers any new files and if it doesn't expand the
>    'regex' so that it does if the license is already known. Otherwise add an
>    entry and the license text to d/copyright*
>
> With updated versions or new links or new files, add an entry to d/changelog 
> which is mostly or fully upstream's primary commit message, prepended by the 
> Debian package name. This way there's an almost 1-on-1 match between the 
> Debian and upstream's commit (message). This should also make reviews easier.
>
> I ordered the entries by Debian package name, so that if someone only has AMD 
> graphics firmware, it doesn't have to read the full changelog, only the part 
> which is about amd-graphics. Upstream's git commit log sequence is rather 
> random, which isn't useful or helpful to our users (IMO ofc)
>
> *) I did improve d/copyright too, but not fully. All the license texts have 
> been moved to the bottom, but I didn't fix the sequencing of all the stanzas.
>
>> the upstream repo needs repacking, 
>
> Upstream now also produces a deb package per tag/release, but that's 350MB in 
> size ... and growing. And it includes firmware which isn't DFSG compliant and 
> therefor we shouldn't ship or f.e. can only be shipped as part of the kernel, 
> so we can't distribute them.
>
>> and the (upstream & debian/) repositories are split; 
>
> I think the split is very useful, especially for our users. Some users only 
> need one or a few small firmware files. Not having to download >350MB for that 
> for every upstream/Debian release/version is quite useful. Not everyone has 
> unmetered GbE internet connection ... or wants to waste 300+MB for files for 
> which they have no need/use.
>
>> I have not found this a very pleasant experience, sadly. 
>
> I didn't find it pleasant either, but I hope/expect that with my updates and 
> restructuring it shouldn't be that bad either.
> When things are kept up-to-date it's quite doable, but when you have to work 
> through months of 'backlog' the 'fun' quickly diminishes.
>
>> And again; no-one to blame here: there _is_ tooling to help, and immense
>> work has been put towards making this manageable! I'm merely saying it's not
>> a matter of running "debian/rules get-new-upstream" at all.
>
> The tooling is mostly great.
> *I* do think that the script which imports upstream's changelog should go 
> though as it makes 'you' think you're quickly done. But if you want to do 
> things right, you should still go through all the commits to actually update 
> the versions and add the new links and new files.
> If you don't, and the temptations is IMO rather high, then you'd create 
> another monumental task like the one I went through.
> And that REALLY did not make me happy or considered fun. At all.
>
> As I see it, the primary problem is the lack of people actively contributing 
> to the Debian kernel team's work. In general.

Interesting read, and combined with my notes to Didier - is this something where I and some in my team could maybe help?
If you think it would be worthwhile, does it make sense to setup a training session to walk me through the steps so I can attempt a first pass, and then review where the problems are and which bits are useful/not-useful.
If the review/link updating/etc was all done - would it make it easier for a DM to look at it and go "yeah, that's solid" and release?

>
> The amount of open *kernel* bugs is usually around 800. And that's mostly due 
> to periodic mass closing of bugs. I have been triaging (kernel) bugs in the 
> last say 2 years, but it was mostly me and Salvatore doing that work.
> I asked for assistance but I didn't see anyone step up. And I plan to shift 
> (part of) my time/focus to other things and my worry is that that means that 
> bug reports get completely unanswered. Like before.
> And having 800+ kernel bugs open isn't helping motivation to spend time on 
> non-free stuff like firmware.
>
> Most of the time triaging (kernel) bugs isn't that hard . It usually boils 
> down to finding when the issue surfaced (last working version vs first non-
> working version), possibly/ideally bisected to a specific commit and then 
> finding the appropriate upstream maintainers and let the bug reporter and the 
> upstream maintainer take it from there.
> It's just time consuming. 
>
> FTR: Offering to help: great.
> Creating NEW bugs with vague/non-actionable descriptions basically asking for 
> new upstream versions ... of which there are already several open in the BTS:
> not so great. It just makes the mountain (of work) higher.
>
My apologies - I really wasn't trying to add to the problem. Not my intent at all. On all other distro's that I work with a bug is usually needed for tracking any work done (and I did reach out by email first :))

My aim with the bugs is to highlight where support is missing for new Intel Meteorlake/AMD Hawkpoint platforms for this years platforms, it wasn't supposed to be vague. I was hesitant to raise the bug, as it is so broad - but it's a valid issue. I do have users on our forum trying to run Debian and are not able to.

I can't highlight all patches needed for Meteorlake/Hawkpoint support - that would be huge. I can highlight that a 6.8 kernel is needed; and which FW packages are needed.
As an aside, it would apply for all system vendors too...if that makes any difference.

I'd really like to figure out how to do something constructive to help with new platforms and getting Debian being friendly to them. I 100% appreciate that time is precious and limited (I do suffer from that too). If feedback from the community is that this isn't of interest then let me know and I shall retreat back under my rock.

Thanks
Mark

[toc] | [prev] | [next] | [standalone]

#82381

[Multipart message — attachments visible in raw view] — view raw

On Tuesday, 7 May 2024 15:49:45 CEST Mark Pearson wrote:
> > As I see it, the primary problem is the lack of people actively
> > contributing to the Debian kernel team's work. In general.
> 
> Interesting read, and combined with my notes to Didier - is this something
> where I and some in my team could maybe help? If you think it would be
> worthwhile, does it make sense to setup a training session to walk me
> through the steps so I can attempt a first pass, and then review where the
> problems are and which bits are useful/not-useful. If the review/link
> updating/etc was all done - would it make it easier for a DM to look at it
> and go "yeah, that's solid" and release?

*I* don't see where you/your team could help now. At least not for this 
particular issue. General help with f.e. bug triaging would still be welcomed 
I presume.

At https://salsa.debian.org/kernel-team/firmware-nonfree/-/merge_requests/ you 
can find the MRs I made to make the firmware-nonfree packages all up to date.
MR 85 is 'the BIG one' and in there I also mentioned that I also have an 
update to 20240312 in my fork which I can submit as MR too.

MR 85 is also reviewed by a DD, but while they are *allowed* to make uploads 
for any package, they choose not to. And FWIW: I agree with that.

I already mentioned that MR 85 is a BIG one (but the only one which needs to 
go through NEW), which in turn means that reviewing takes more then 10 minutes 
(to put it mildly).
The update to 20230804 is also larger then average as some changes needed to 
be 'postponed' due to needing a new upstream version.

The updates after that are indicative of what I suspect/hope future updates 
will look like. Unless a kernel-team member thinks that I didn't do it 
correctly. While I did my best, that's surely a possibility.

So if you want to know what's generally involved into updating the firmware-
nonfree packages, I suggest to look into my MRs.
I have a tendency to be verbose in my commit messages, which may help.
And the procedure I described earlier should match what you'll see in the 
updates after 20230804.

> I can't highlight all patches needed for Meteorlake/Hawkpoint support - that
> would be huge. I can highlight that a 6.8 kernel is needed; and which FW
> packages are needed. 

While firmware package updates have lagged before, the kernel packages are 
normally quite up-to-date. But sometimes there are other factors which cause 
delays which normally aren't there. The time64 transition f.e. caused 'some' 
disruptions.
But (my) update-to-6.8 MR has now been merged into master, so an update to the 
6.8 kernel is in the works.

HTH,
  Diederik

[toc] | [prev] | [next] | [standalone]

#82382

Hi Diederik,

On Tue, May 7, 2024, at 11:17 AM, Diederik de Haas wrote:
> On Tuesday, 7 May 2024 15:49:45 CEST Mark Pearson wrote:
>> > As I see it, the primary problem is the lack of people actively
>> > contributing to the Debian kernel team's work. In general.
>> 
>> Interesting read, and combined with my notes to Didier - is this something
>> where I and some in my team could maybe help? If you think it would be
>> worthwhile, does it make sense to setup a training session to walk me
>> through the steps so I can attempt a first pass, and then review where the
>> problems are and which bits are useful/not-useful. If the review/link
>> updating/etc was all done - would it make it easier for a DM to look at it
>> and go "yeah, that's solid" and release?
>
> *I* don't see where you/your team could help now. At least not for this 
> particular issue. General help with f.e. bug triaging would still be welcomed 
> I presume.
>
> At https://salsa.debian.org/kernel-team/firmware-nonfree/-/merge_requests/ you 
> can find the MRs I made to make the firmware-nonfree packages all up to date.
> MR 85 is 'the BIG one' and in there I also mentioned that I also have an 
> update to 20240312 in my fork which I can submit as MR too.
>
> MR 85 is also reviewed by a DD, but while they are *allowed* to make uploads 
> for any package, they choose not to. And FWIW: I agree with that.
>
> I already mentioned that MR 85 is a BIG one (but the only one which needs to 
> go through NEW), which in turn means that reviewing takes more then 10 minutes 
> (to put it mildly).
> The update to 20230804 is also larger then average as some changes needed to 
> be 'postponed' due to needing a new upstream version.
>
> The updates after that are indicative of what I suspect/hope future updates 
> will look like. Unless a kernel-team member thinks that I didn't do it 
> correctly. While I did my best, that's surely a possibility.
>
> So if you want to know what's generally involved into updating the firmware-
> nonfree packages, I suggest to look into my MRs.
> I have a tendency to be verbose in my commit messages, which may help.
> And the procedure I described earlier should match what you'll see in the 
> updates after 20230804.
>

Cool - I have taken a very quick look and kudos - amazing job.

A few follow on questions:

 - It looks like it is stuck on someone doing the upload (guessing nobody has time to do that). Unfortunately I can't help with that bit - not a DM and nowhere near enough experience. Any recommendations on how to unblock that? It doesn't seem like testing/reviewing would help - but if you have any insight as to what would make a difference let me know.
And, thinking about the comment previously about funding - is this something where somebody needs some paid time to do the exercise? It's not something I can promise (and dealing with Lenovo procurement is a special form of hell), but if that's the blocker then I am happy to go and shake some branches and see if there's a way we can help out. I don't want to cause offence and assume that's the answer though (and FWIW, I would prefer my team to contribute directly through knowledge/skills as I think it's better in the long term).

 - I love what you've done with the monthly cadence of updates, that seems smart to me but I appreciate it's a chunk of work every month. I also noted in one of the commits you mentioned "I don't particularly care about this package, but just wanted to do something about the (big) backlog AND make it easier to prevent a backlog from appearing again in the future (by making it easier for everyone to make an update)", so I'm assuming you'd like done with it ASAP.
You noted earlier that it wasn't documented and added some useful notes. Would it be helpful to work through that process with an idiot (aka...me) to get an initial document down as a starting point?
With the last MR being a few months old now, would it be reasonable for me to take a stab at doing an update and (worst-case) learning from it? I don't want to tread on any toes, and I will almost certainly need some guidance, but it seems like there is interest in helping out from other users too (based on the comment thread). If we can figure out the upload piece, it looks like the infrastructure work you've put in would pay off.

 - Previously you noted "Upstream now also produces a deb package per tag/release" - I had a look for those but couldn't find it. Do you have a location I can check those out? Figured it would be interesting to have a look.

>> I can't highlight all patches needed for Meteorlake/Hawkpoint support - that
>> would be huge. I can highlight that a 6.8 kernel is needed; and which FW
>> packages are needed. 
>
> While firmware package updates have lagged before, the kernel packages are 
> normally quite up-to-date. But sometimes there are other factors which cause 
> delays which normally aren't there. The time64 transition f.e. caused 'some' 
> disruptions.
> But (my) update-to-6.8 MR has now been merged into master, so an update to the 
> 6.8 kernel is in the works.
>
That's awesome. If you want any sanity checks run on it, let me know - I have a number of platforms running Debian that I usually use for testing the firmware-sof package updates.

Thank you for all the insights.
Mark

[toc] | [prev] | [next] | [standalone]

#82386

[Multipart message — attachments visible in raw view] — view raw

Hi Mark,

On Tuesday, 7 May 2024 19:30:59 CEST Mark Pearson wrote:
> > I already mentioned that MR 85 is a BIG one (but the only one which needs
> > to go through NEW), which in turn means that reviewing takes more then 10
> > minutes (to put it mildly).
> > 
> > The updates after that are indicative of what I suspect/hope future
> > updates will look like. Unless a kernel-team member thinks that I didn't
> > do it correctly. While I did my best, that's surely a possibility.
> 
> Cool - I have taken a very quick look and kudos - amazing job.

Cheers :)

> A few follow on questions:
> 
>  - It looks like it is stuck on someone doing the upload (guessing nobody
> has time to do that). Unfortunately I can't help with that bit - not a DM
> and nowhere near enough experience.

As it has to go through NEW, the uploader must be a DD (not a DM).
AFAIK, it's not the upload that's the blocker (that should be rather quick), 
but the *review* of MR 85.
And those reviews are VERY important (more on that later).

> And, thinking about the comment previously about funding - is this something
> where somebody needs some paid time to do the exercise?

IIF that's an option then I'd guess they'd contact you, most likely privately.
The most likely scenario, as is the case for 99% (just a guess) of the 
software packaged for Debian, is that someone needs to find the time (and 
motivation) to do that 'work' in their FREE time.

>  - I love what you've done with the monthly cadence of updates, that seems
> smart to me but I appreciate it's a chunk of work every month.

My guess is that it'll take a couple (probably <4) of hours a month.

> I also noted in one of the commits you mentioned "I don't particularly care
> about this package, but just wanted to do something about the (big) backlog
> AND make it easier to prevent a backlog from appearing again in the future
> (by making it easier for everyone to make an update)", so I'm assuming you'd
> like done with it ASAP.

Debian is often described as a do-ocracy so I did what was in my power and 
that was making a MR which would/could fix the problem I was seeing.
It wasn't the most enjoyable thing I've done, but someone needed to do the 
'work', so why not me?

> You noted earlier that it wasn't documented and added some useful notes.
> Would it be helpful to work through that process with an idiot (aka...me) to
> get an initial document down as a starting point?

I haven't added my procedure to a MR because I wanted to have a review first.
As I've learned over time, there tends to be very good reasons why things are 
done (and described in README.source) a certain way. I didn't manage to find 
the reason why the described update procedure was as written down.

It could be that 'my' procedure does something wrong or is incomplete or not 
the best/appropriate way, which I expect will be pointed out in the review.
If the reviewer thinks my procedure is actually better, then I'll write it 
down and submit that as MR too. That shouldn't take much time.

> With the last MR being a few months old now, would it be reasonable
> for me to take a stab at doing an update and (worst-case) learning from it?

Generally the best way to learn something is by doing it ;-)

>  - Previously you noted "Upstream now also produces a deb package per
> tag/release" - I had a look for those but couldn't find it. Do you have a
> location I can check those out? Figured it would be interesting to have a
> look.

https://gitlab.com/kernel-firmware/linux-firmware/ is now the place where 
upstream changes are merged. When a release is tagged, their CI now produces a 
deb (and rpm) package.
So on https://gitlab.com/kernel-firmware/linux-firmware/-/tags you'll see a 
(hopefully green) checkmark indicating the success of the pipeline run.
https://gitlab.com/kernel-firmware/linux-firmware/-/pipelines/1247368343 would 
be the one for the 20240410 tag/release and if you click through to the 'deb-
release' job and then you can download or browse through to the artifact(s), 
which in this case is a 347 MiB deb package with all the firmware.

> >> I can't highlight all patches needed for Meteorlake/Hawkpoint support -
> >> that would be huge. I can highlight that a 6.8 kernel is needed; and
> >> which FW packages are needed.
> > 
> > But (my) update-to-6.8 MR has now been merged into master, so an update to
> > the 6.8 kernel is in the works.
> 
> That's awesome. If you want any sanity checks run on it, let me know - 

When you submit a MR you are (generally) expected to have verified it works as 
intended. While I did build a 6.8 kernel, I tend to use the 'nodoc' profile.
The review of that MR (in this case by Ben) turned out that it actually 
contained an error ... in building the documentation.

So while using the 'nodoc' is generally fine for my use case, next time I 
really should also make a build with the 'doc' profile (which is the default).
That's why those reviews are so important!

> I have a number of platforms running Debian that I usually use for testing
> the firmware-sof package updates.

What you can do is *verify* whether the Debian kernel package build from 
(current) 'master' does indeed make the devices for which you need the 6.8 
kernel, work correctly and fully.
At https://kernel-team.pages.debian.net/kernel-handbook/ you should find all 
the information/instructions on how to build your own Debian kernel.

It could be (f.e.) that you actually need one or more kernel modules enabled, 
which aren't currently enabled in the Debian kernel config. If not everything 
works properly, then you can find out which additional modules you need and 
which changes to the Debian kernel config are needed for everything to work 
properly.
That's how I got started (with Pine64's Rock64 SBC) ...

It could also turn out that there's a bug in the upstream kernel (and not in 
the Debian package of it), in which case you can immediately start to get that 
fixed upstream.

Once you have verified that your self-build Debian kernel now does work as you 
want, you can file a bug report requesting those changes ... and/or create a MR 
to get your changes into the Debian kernel (repo).

You could also build the Debian firmware packages from one of my branches/MRs 
or indeed make one yourself for the 20240410 version to verify if it does what 
you expect it will do. And take appropriate action if it does not.

So there's plenty you (and/or your team) can do to make sure that when the 
(official) Debian kernel gets released, it functions optimal for your use cases.

Good luck and have fun :)

Cheers,
  Diederik

[toc] | [prev] | [next] | [standalone]

#82379

Hi Didier

On Tue, May 7, 2024, at 2:25 AM, Didier 'OdyX' Raboud wrote:
> Hello there Mark,
>
> thanks for your email, and for the followup ping; weeks are well-filled with 
> $job, baby and other involvements.

Congrats! Exciting times (my eldest just passed her driving test last week...the time flies by far too fast...enjoy it!)

>
> Regarding firmware-nonfree updates, I have related my last-years' experience 
> on d-devel: https://lists.debian.org/msgid-search/2390124.3VsfAaAtOV@turnagra
>

Interesting read. I do skim the debian lists and I'd missed it.
Some notes further below that I think are related to the points you raise in there.

> I insist: I am absolutely not implying that Ben is doing anything wrong; I 
> just observed that he was mostly alone in a bottleneck position for that 
> package.
>
> What I did back then to try to help getting the firmware package in better 
> shape was to dive in the (complex) packaging and try to produce "ready-to-
> merge" merge-requests for new upstream releases, fixes, etc, then pinging Ben 
> at (what I considered) reasonable intervals. I see that Diederik (CC'ed) has 
> started to do the same for recent versions.
>
> The package is complex and a mined land of legal concerns, the upstream repo 
> needs repacking, and the (upstream & debian/) repositories are split; I have 
> not found this a very pleasant experience, sadly. And again; no-one to blame 
> here: there _is_ tooling to help, and immense work has been put towards making 
> this manageable! I'm merely saying it's not a matter of running "debian/rules 
> get-new-upstream" at all.
>
> From where I sit (that is: I only modestly contributed in the past, and cannot 
> commit to much these days), there are two angles to address:
> A) lift the bottleneck: there needs to be more Debian Developers confident to 
> comment and merge MRs, and then upload.
> B) regular commitment: I feel it would be easier to get regular updates 
> merged, rather than doing the huge batch shortly before release; that would 
> also appease the tensions that many probably feel when they get new hardware 
> but no available package.

So, this may be naive, but I think this is potentially a good place for myself and (hopefully) some of the Lenovo team to get involved and contribute - at least on (B) in the short term and maybe eventually on (A) in the longer term.
I really want to support the community distros that are at the core of Linux - I think it's important. It's also, honestly, what makes me happiest.

I also think that, realistically, nobody really wants to mess around with non-free FW because it's just not very interesting. But it's needed as the vendors themselves don't particularly want to get involved with packaging (I do the Debian firmware-sof-signed package for the Intel audio FW because of this).

Lenovo has a big advantage in that we have the hardware available, earlier, so are able to identify what's needed and do testing.

My biggest problem is how to get started. As noted above, I do the SOF firmware packaging so I'm not a complete noob, but I'm not particularly experienced either. I have the steps I need to follow, and I do that, test, sometimes fix minor issues and that's it. I should flag, I'm not a DD or DM (FWIW, I'd like to be one day, I've just not earned it yet :))

I'm open to ideas and suggestions, but (particularly for the linux-firmware pieces) I think we 
(Lenovo) can help here, if someone is willing to give some help to get started. I appreciate there needs to be appropriate mentoring and checking in place - but if there's something we can do to get some of the grunt work done so that MR's can be checked please let me know how to do that.

>
> As to how I can get involved in any of the above; I can't reasonably add this 
> to my plate these days (and would rather _not do_, than commit to do and 
> fail).
>
> But I wonder if some financial sponsorship (if Lenovo and others would be open 
> to something like that) could help, via Freexian for example (disclaimer: not 
> for me to do it; I'm not a Freexian person). I'd be concerned that this would 
> only solve B above though. It would feel like a 1-2 hours job / week.
>
If $'s solve it - I'm open to suggestions. I have to caveat it with the fact the Linux project here is not making money...I don't have budget to burn. I'd rather get involved directly and contribute and enable others in my team to contribute directly too, I think that has more real value. If the answer is just money, then let me know and where I should take that discussion - I'm not averse to it, but my previous experience is that money doesn't solve the problem - the issue is more having hands to do the work (I think the Debian budget reports confirm this FWIW - money is not lacking).

> I hope the above helps you navigate that question, and that it helps the 
> situation in general.
>

Thank you for all the pointers!
Mark

[toc] | [prev] | [next] | [standalone]

#83025

[Multipart message — attachments visible in raw view] — view raw

On Fri, 2024-04-26 at 14:35 -0400, Mark Pearson wrote:
> Hi Didier & Ben,
> 
> I hope all is well.
> I wanted to check in with you guys, as with the new 2024 platforms coming out there are some updates to linux-firmware - for CPU, GPU, Wifi, Bluetooth, etc devices
> 
> I checked and it looked like firmware-nonfree and it hadn't been updated since last year - and I wondered if that is something I can help with? I'd like to make sure the Debian experience on this years platforms is good out-of-the-box.
> 
> Let me know if that would be helpful, and any pointers on how best to contribute and be useful

Hi Mark,

Sorry for not responding to this earlier.

I've finally uploaded an update to (nearly) the current version.  This
incorporates some of Diederik's work, but I also made changes that
should reduce the amount of manual effort required for future updates.

Ben.

> Thanks
> Mark
> 
> PS - in case anyone notices, I've mothballed my old markpearson@lenovo.com address because the Lenovo outlook servers there became horribly unusable. I am using my personal domain for open source collaboration instead. 
> I still have access to that address (if anybody wants to check I am who I say I am) and am still employed at Lenovo (if that makes any difference to anything)

-- 
Ben Hutchings - Debian developer, member of kernel, installer and LTS
teams

[toc] | [prev] | [next] | [standalone]

#83027

Hi Ben,

On Tue, Jul 16, 2024, at 9:20 AM, Ben Hutchings wrote:
> On Fri, 2024-04-26 at 14:35 -0400, Mark Pearson wrote:
>> Hi Didier & Ben,
>> 
>> I hope all is well.
>> I wanted to check in with you guys, as with the new 2024 platforms coming out there are some updates to linux-firmware - for CPU, GPU, Wifi, Bluetooth, etc devices
>> 
>> I checked and it looked like firmware-nonfree and it hadn't been updated since last year - and I wondered if that is something I can help with? I'd like to make sure the Debian experience on this years platforms is good out-of-the-box.
>> 
>> Let me know if that would be helpful, and any pointers on how best to contribute and be useful
>
> Hi Mark,
>
> Sorry for not responding to this earlier.
>
> I've finally uploaded an update to (nearly) the current version.  This
> incorporates some of Diederik's work, but I also made changes that
> should reduce the amount of manual effort required for future updates.
>

No worries, and big thank you to both of you for all your hard work on this. Very much appreciated and I will do some testing of the latest to confirm it works. As an aside, it was useful and interesting to look into the repositories and see what was involved.

I did at one point take a stab at looking what would be involved to help with updates but backed off until the current updates have gone through.
With that done, I'm very happy to try and help with updates going forward. I know that for many people in the Debian community that dealing with non-free FW is not much fun or interest :)
I will give it a go in the future, but do feel free to ping me if you're looking for a pair of hands to do some work on this.

Hope you get to enjoy Korea :)

Mark

> Ben.
>
>> Thanks
>> Mark
>> 
>> PS - in case anyone notices, I've mothballed my old markpearson@lenovo.com address because the Lenovo outlook servers there became horribly unusable. I am using my personal domain for open source collaboration instead. 
>> I still have access to that address (if anybody wants to check I am who I say I am) and am still employed at Lenovo (if that makes any difference to anything)
>
> -- 
> Ben Hutchings - Debian developer, member of kernel, installer and LTS
> teams
>
> Attachments:
> * signature.asc

[toc] | [prev] | [next] | [standalone]

#83033

Hi Ben, hi Mark, hi all of you,

Ben Hutchings - 16.07.24, 15:20:23 CEST:
> Sorry for not responding to this earlier.
> 
> I've finally uploaded an update to (nearly) the current version.  This
> incorporates some of Diederik's work, but I also made changes that
> should reduce the amount of manual effort required for future updates.

Thank you, Ben!

I can also test it. I have been following this thread with interest, but 
did not comment so far.

I have a ThinkPad T14 AMD Gen 5 and I am in contact with Mark about some 
issues with the new platform as Lenovo readies their Linux support for it. 
It is already working quite good, but a few strange issues remain.

I currently installed linux-firmware upstream package on top of the Debian 
ones in order to for example make WLAN and Bluetooth work but I am happy 
to remove it once I see the new version of the Debian packages coming in 
and report in case I see any issues. I can report here before accidentally 
opening any bug report needlessly. But I could also open a bug report if 
need be.

Best,
-- 
Martin

[toc] | [prev] | [next] | [standalone]

#83034

Martin Steigerwald - 16.07.24, 17:08:42 CEST:
> Ben Hutchings - 16.07.24, 15:20:23 CEST:
> > Sorry for not responding to this earlier.
> > 
> > I've finally uploaded an update to (nearly) the current version.  This
> > incorporates some of Diederik's work, but I also made changes that
> > should reduce the amount of manual effort required for future updates.
> 
> Thank you, Ben!

Also of course thanks to Diederik. I appreciate it!

Thanks to everyone involved.

-- 
Martin

[toc] | [prev] | [next] | [standalone]

#83558

Hi Ben, Mark, hello.

Martin Steigerwald - 16.07.24, 17:08:42 CEST:
> I have a ThinkPad T14 AMD Gen 5 and I am in contact with Mark about some
> issues with the new platform as Lenovo readies their Linux support for
> it. It is already working quite good, but a few strange issues remain.
> 
> I currently installed linux-firmware upstream package on top of the
> Debian ones in order to for example make WLAN and Bluetooth work but I
> am happy to remove it once I see the new version of the Debian packages
> coming in and report in case I see any issues. I can report here before
> accidentally opening any bug report needlessly. But I could also open a
> bug report if need be.

I can report back that with firmware-atheros 20240709-1 both WLAN and 
Bluetooth are recognized and can be enabled on the ThinkPad T14 AMD Gen 5. 
I did not yet test using them, but I bet it will work.

Upstream firmware package is no longer installed.

Thanks.

-- 
Martin

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.kernel

csiph-web