Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1249366 > unrolled thread

[PATCH 0/14] init: deps: dependency based (parallelized) init

Back to article view | Back to linux.kernel

Contents

  [PATCH 0/14] init: deps: dependency based (parallelized) init Alexander Holler <holler@ahsoftware.de> - 2015-10-17 19:20 +0200
    [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-17 19:20 +0200
      Re: [PATCH 04/14] init: deps: order network interfaces by link order Linus Torvalds <torvalds@linux-foundation.org> - 2015-10-17 20:30 +0200
        Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-17 20:40 +0200
          Re: [PATCH 04/14] init: deps: order network interfaces by link order Linus Torvalds <torvalds@linux-foundation.org> - 2015-10-17 21:00 +0200
            Re: [PATCH 04/14] init: deps: order network interfaces by link order Linus Torvalds <torvalds@linux-foundation.org> - 2015-10-17 21:10 +0200
              Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-17 21:20 +0200
                Re: [PATCH 04/14] init: deps: order network interfaces by link order Linus Torvalds <torvalds@linux-foundation.org> - 2015-10-17 21:40 +0200
                Re: [PATCH 04/14] init: deps: order network interfaces by link order Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2015-10-17 21:40 +0200
                  Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-17 22:00 +0200
                    Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-17 23:30 +0200
                  Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-18 07:00 +0200
                    Re: [PATCH 04/14] init: deps: order network interfaces by link order Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2015-10-18 07:20 +0200
                      Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-18 07:30 +0200
                        Re: [PATCH 04/14] init: deps: order network interfaces by link order Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2015-10-18 08:00 +0200
                          Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-18 12:20 +0200
                            Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-19 13:00 +0200
                              Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-19 13:40 +0200
                                Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-22 08:50 +0200
              Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-17 23:40 +0200
            Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-17 21:10 +0200
          Re: [PATCH 04/14] init: deps: order network interfaces by link order Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2015-10-17 21:00 +0200
          Re: [PATCH 04/14] init: deps: order network interfaces by link order Linus Torvalds <torvalds@linux-foundation.org> - 2015-10-17 21:10 +0200
            Re: [PATCH 04/14] init: deps: order network interfaces by link order Alexander Holler <holler@ahsoftware.de> - 2015-10-17 21:10 +0200
    [PATCH 03/14] init: deps: dt: use (HW-specific) dependencies provided by the DT too Alexander Holler <holler@ahsoftware.de> - 2015-10-17 19:20 +0200
      Re: [PATCH 03/14] init: deps: dt: use (HW-specific) dependencies  provided by the DT too Mark Brown <broonie@kernel.org> - 2015-10-19 14:40 +0200
        Re: [PATCH 03/14] init: deps: dt: use (HW-specific) dependencies  provided by the DT too Rob Herring <robh+dt@kernel.org> - 2015-10-19 18:30 +0200
          Re: [PATCH 03/14] init: deps: dt: use (HW-specific) dependencies  provided by the DT too Alexander Holler <holler@ahsoftware.de> - 2015-10-19 19:30 +0200
        Re: [PATCH 03/14] init: deps: dt: use (HW-specific) dependencies  provided by the DT too Alexander Holler <holler@ahsoftware.de> - 2015-10-19 19:20 +0200
    [PATCH 06/14] dtc: deps: Automatically add new property 'dependencies' which contains a list of referenced phandles Alexander Holler <holler@ahsoftware.de> - 2015-10-17 19:20 +0200
    [PATCH 07/14] dtc: deps: introduce new (virtual) property no-dependencies Alexander Holler <holler@ahsoftware.de> - 2015-10-17 19:20 +0200
    [PATCH 09/14] dtc: deps: Add option to print dependency graph as dot (Graphviz) Alexander Holler <holler@ahsoftware.de> - 2015-10-17 19:20 +0200
    [PATCH 02/14] init: deps: use annotated initcalls for a dependency based (optionally parallelized) init Alexander Holler <holler@ahsoftware.de> - 2015-10-17 19:20 +0200
    [PATCH 01/14] init: deps: introduce annotated initcalls Alexander Holler <holler@ahsoftware.de> - 2015-10-17 19:20 +0200
    [PATCH 14/14] dt: dts: deps: omap: beagle: make some remote-endpoints non-dependencies Alexander Holler <holler@ahsoftware.de> - 2015-10-17 19:30 +0200
    [PATCH 13/14] dt: dts: deps: imx6q: make some remote-endpoints non-dependencies Alexander Holler <holler@ahsoftware.de> - 2015-10-17 19:30 +0200
    [PATCH 12/14] dt: dts: deps: kirkwood: dockstar: add dependency ehci -> usb power regulator Alexander Holler <holler@ahsoftware.de> - 2015-10-17 19:30 +0200
    Re: [PATCH 0/14] init: deps: dependency based (parallelized) init Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2015-10-17 19:50 +0200
      Re: [PATCH 0/14] init: deps: dependency based (parallelized) init Alexander Holler <holler@ahsoftware.de> - 2015-10-17 20:30 +0200
        Re: [PATCH 0/14] init: deps: dependency based (parallelized) init Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2015-10-17 20:40 +0200
          Re: [PATCH 0/14] init: deps: dependency based (parallelized) init Alexander Holler <holler@ahsoftware.de> - 2015-10-17 21:50 +0200
            Re: [PATCH 0/14] init: deps: dependency based (parallelized) init Greg Kroah-Hartman <gregkh@linuxfoundation.org> - 2015-10-17 22:30 +0200
              Re: [PATCH 0/14] init: deps: dependency based (parallelized) init Alexander Holler <holler@ahsoftware.de> - 2015-10-17 22:40 +0200
    Re: [PATCH 11/14] init: deps: annotate various initcalls Linus Torvalds <torvalds@linux-foundation.org> - 2015-10-17 20:50 +0200
      Re: [PATCH 11/14] init: deps: annotate various initcalls Alexander Holler <holler@ahsoftware.de> - 2015-10-17 21:10 +0200

Page 1 of 3  [1] 2 3  Next page →

#1249366 — [PATCH 0/14] init: deps: dependency based (parallelized) init

Hello,

here is the newest version of my patches to use a dependency based
initialization order. It now works without DT too.

Background:

Currently initcalls are ordered by some levels and the link order. This
means whenever a file is renamed, changes directory or a Makefile is
modified the order with which initcalls are called might change. This
might result in problems. Furthermore, the required dependencies are
often not documented, sometimes there are comments in the source or in a
commit message, but most often the knowledge why a specific initcall
belongs to a specific initcall level isn't obvious without carefully
examing he source. And initcalls are used by drivers and subsystems, and
the count of both have grown quiet a lot in the last years. So it's
rather difficult to maintain a proper link order.
Another problem is that the link order can't be modified dynamically at
boot time to include dependencies dictated by the hardware. To circumvent
this, a brute-force trial-and-error mechanism called deferred probes has
been introduced, but this approach, while beeing KISS, has its own
problems.

To solve these problems I've written patches to use a topological sort at
boot time which uses dependencies to calculate the order with which
initcalls are called.

Why? What are the benefits (assuming correct dependencies are available)?

- It offers a clear in-source documentation for dependencies between
  initcalls.
- It is robust in regard to file or directory name changes and changes in
  a Makefile.
- If enabled, the order with which drivers for interfaces are called
  (e.g. network interfaces, hard disks), can be defined independent of
  the link order. These might result in more stable interface names or
  numbers.
- If enabled, it makes the the deferred probes obsolete, which might
  result in faster boot times.
- If enabled, it is possible to call initcalls in parallel. E.g. the
  shipped kernel for Fedora 21 (4.1.7-100.fc21.x86_64) contains around
  560 initcalls. These are all called in series. Also some of them use
  asynchronous stuff by themself, most don't do.

Drawbacks:

- It requires a small amount of time to calculate the order a boot time.
  But this time is most often smaller than the time saved by using
  multiple cores to call initcalls or by not needing deferred probes.
- Dependencies are required. For everything which can be build as a
  module, looking at modules.dep might give some pointers. Looking at
  the help from menuconfig also might give some pointers. But in the
  end, the most preferable way would be if maintainers or other people
  which have a deeper knowledge about the source and functionality
  would add the dependencies.

And last but not least, these feature is totally optional. If disabled,
nothing changes.


Some words about the patches:

Patch 1 introduces annotated initcalls, patch 2 uses them and patch 3
adds hardware specific dependencies from the device tree to the existent
set of dependencies.

Patch 4 and 5 are examples about how to achieve a stable initialization
order for various type of drivers. The two patches are doing this for
network interfaces (based on the existing link order) and I2C busses
(based on the new driver IDs). I leave it for discussion which one should
be used. My suggestion would be to order the IDs like the link order and
then use the order based on IDs. That would get rid of the link order while
also beeing downwards compatible.

Patches 6-9 do contain changes for dtc to enhance the binary blob (dtb)
with type information for phandles (the dependencies used by patch 3).
I've already posted these patches a year ago.

Patch 10 contains the list of driver IDs I've annotated and is NOT meant
for merging. Patch 11 contains all the changes on drivers I've annotated
and is NOT meant for merge too. I've no idea how such patches might end
up in the kernel (in fact I've no idea what's necessary for any patch),
therefor these two patches are there to offer a quick start to evaluate
this feature.

The final 3 patches contain a few changes for the DTs of 3 ARM boards.

Because I don't want to fiddle with problems in the unstable mainline tree,
these patches are based on the latest stable kernel (4.2.3). But besides
the big patch (11) with all the driver changes, they apply without problems
to the current mainline too. And the necessary changes to use the big patch
on mainline should be trivial, as all the changes in this patch are trivial
too.

I've tested this patch series with several ARM(32) systems, x86 and x86_64
and the boot time was faster on allmost all systems. Either through
avoiding deferred probes (on single core machines) or by using the
parallel initialization. And keep in mind that, also I've already
annotated quiet a lot of initcalls, a big bunch of them still are left
and therefor are still called in series using just one core and not in
parallel.

Keep in mind that these patches are an offer, I wasn't paid for them and
I don't need them in the mainline kernel.
As long as you are able to use a polite language, you can send me any
comment, regardless if it's good or bad. But, please, keep away with
Baby-Speak or contumelious comments.

And, just in case, I'm aware that adding the necessary dependencies means
some effort and a lot of (trivial) commits and therefor having all
possible initcalls annotated would be a long term goal. But, besides that
this could be done smoothly without any need to hurry, I think it makes
sense. Otherwise, in my humble opinion, the problems to keep an overview
and ordering initcalls will just become worse.


Alexander Holler
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1249367 — [PATCH 04/14] init: deps: order network interfaces by link order

In order to provide stable interface numbers, network interface drivers
will be ordered by the link order. This is easy to accomplish by adding
dependencies.

Assuming three different ethernet-drivers, without any special code,
the dependency graph would not require any special order inbetween them
and would look like that:

    eth-driver-base
   /      |       \
eth-x   eth-y    eth-z

Now we just add dependencies. With the additional dependencies the graph
looks like:

 eth-driver-base
  |     | |
eth-x   | |
  |     | |
eth-y  -| |
  |       |
eth-z  ---|

Signed-off-by: Alexander Holler <holler@ahsoftware.de>
---
 include/linux/driver_ids.h | 11 +++++++++++
 init/dependencies.c        |  9 +++++++++
 2 files changed, 20 insertions(+)

diff --git a/include/linux/driver_ids.h b/include/linux/driver_ids.h
index 60964fe..1133a68 100644
--- a/include/linux/driver_ids.h
+++ b/include/linux/driver_ids.h
@@ -15,6 +15,17 @@
 enum {
 	drvid_unused,
 	/* To be filled */
+
+	/*
+	 * Network drivers will be ordered according to the link order
+	 * (which means not necessarily according to their appearance
+	 * here).
+	 * This provides stable interface numbers.
+	 * Therefor their IDs have to be in the following block.
+	 */
+	drvid_network_drivers_start,
+	drvid_network_drivers_end,
+
 	drvid_max
 };
 
diff --git a/init/dependencies.c b/init/dependencies.c
index b484f67..027fc4b 100644
--- a/init/dependencies.c
+++ b/init/dependencies.c
@@ -301,12 +301,21 @@ static int __init add_dependencies(void)
 static void __init build_inventory(void)
 {
 	const struct _annotated_initcall *ac;
+	unsigned id_last_network_driver = 0;
 
 	ac = __annotated_initcall_start;
 	for (; ac < __annotated_initcall_end; ++ac) {
 		include_node[ac->id] = true;
 		annotated_initcall_by_drvid[ac->id] = ac;
 		nvertices = max(nvertices, ac->id);
+		/* order network drivers by link order*/
+		if (ac->id > drvid_network_drivers_start &&
+				ac->id < drvid_network_drivers_end) {
+			if (id_last_network_driver)
+				add_initcall_dependency(ac->id,
+					id_last_network_driver);
+			id_last_network_driver = ac->id;
+		}
 	}
 }
 
-- 
2.1.0

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1249411 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

On Sat, Oct 17, 2015 at 10:14 AM, Alexander Holler <holler@ahsoftware.de> wrote:
>
> Assuming three different ethernet-drivers, without any special code,
> the dependency graph would not require any special order inbetween them
> and would look like that:

This seems *fundamentally* wrong.

This is in no way specific to network drivers (or to disk drivers, or
to anything else).

So requiring extra logic for this implies that something is seriously wrong.

If two drivers aren't ordered by dependencies, they should always be
in link order, regardless of any hacks like these. If they're not,
things are wrong.

I think your problem is that you make that dependency thing a separate
ordering, so now it matters whether a driver has a dependency or not.

If something like this is to work, it has to work *with* the normal
ordering, not outside of it and then have these kinds of broken
special cases.

The normal init orderings (ie core -> postcore -> arch -> subsys -> fs
-> rootfs -> device -> late) should just be an extra dependency, I
think.

The way that you just insert the annotated dependencies in between
levels 6 and 7 ("device" and "late") can't be right. It means - for
example - that you can't have subsystems that have dependencies.

So I really think that if we do dependencies, then the current levels
have to be added as dependencies, so that "subsys_initcall(xyz)"
basically means "xyz depends on the 'subsys' event, and 'subsys_end'
depends on xyz". Then within that, you might have another bus driver
that in turn depends on 'xyz'.

Because right now afaik we do have dependencies like that, which we
sort out by link ordering (things like 'drm depends on pci' - I'm not
actually sure that one is true, but it might be).

                     Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1249412 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

Am 17.10.2015 um 20:23 schrieb Linus Torvalds:
> On Sat, Oct 17, 2015 at 10:14 AM, Alexander Holler <holler@ahsoftware.de> wrote:
>>
>> Assuming three different ethernet-drivers, without any special code,
>> the dependency graph would not require any special order inbetween them
>> and would look like that:
>
> This seems *fundamentally* wrong.
>
> This is in no way specific to network drivers (or to disk drivers, or
> to anything else).
>
> So requiring extra logic for this implies that something is seriously wrong.
>
> If two drivers aren't ordered by dependencies, they should always be
> in link order, regardless of any hacks like these. If they're not,
> things are wrong.
>
> I think your problem is that you make that dependency thing a separate
> ordering, so now it matters whether a driver has a dependency or not.

I'm making dependencies the only ordering for annotated initcalls.

Otherwise it's impossible to call initcalls in parallel. I've seen a 
stable topological sort somewhere, but whenever you want to parallelize 
the initcalls, the stable ordering would be gone anyway. So I've decided 
not to look further at a stable topological sort.

> If something like this is to work, it has to work *with* the normal
> ordering, not outside of it and then have these kinds of broken
> special cases.
>
> The normal init orderings (ie core -> postcore -> arch -> subsys -> fs
> -> rootfs -> device -> late) should just be an extra dependency, I
> think.
>
> The way that you just insert the annotated dependencies in between
> levels 6 and 7 ("device" and "late") can't be right. It means - for
> example - that you can't have subsystems that have dependencies.

Sorry, but that's wrong.

I've choosen to place initcalls between 5 and 6 to make it easier to 
move both, subsystems as well as normal drivers to the (new) level with 
annotated initcalls. If you look at what I've already "annotated", you 
will see there are quiet a lot initcalls I've moved from below 6 to the 
new level.

> So I really think that if we do dependencies, then the current levels
> have to be added as dependencies, so that "subsys_initcall(xyz)"
> basically means "xyz depends on the 'subsys' event, and 'subsys_end'
> depends on xyz". Then within that, you might have another bus driver
> that in turn depends on 'xyz'.

It would be absolutely no problem to introduce "virtual" initcalls for 
any level, e.g. just

depencies = {
	everything_below
}

initcall foo()
{
	return 0;
}

annotated_initcall(foo, id, dependencies),

Regards,

Alexander Holler
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1249420 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

On Sat, Oct 17, 2015 at 11:37 AM, Alexander Holler <holler@ahsoftware.de> wrote:
>
> I'm making dependencies the only ordering for annotated initcalls.

Yeah, and quite frankly, that just means that I'm not going to merge it.

We do not do "flag-day" things. We've done them in the past, and it
has always been a major and unacceptable pain.

And if the dependency ordering is "outside" of the traditional
link-time ordering, then it is by definition a flag-day event. Every
time you add a dependency to even just *one* driver, it magically
changes ordering wrt all the other drivers, because now it's in a
different ordering domain.

So please reconsider. Or stop cc'ing me. Because "flag day" changes
really are not acceptable.

                       Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1249425 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

On Sat, Oct 17, 2015 at 12:01 PM, Alexander Holler <holler@ahsoftware.de> wrote:
>
> That isn't a flag day thing. It's a config option everyone can turn on and
> off whenever he wants.

That's a flag-day thing. We've done it (drm comes to mind - several times).

I'm disappointed, because I _know_ I pointed you in the direction of
stable sorting about a month ago. I really had hoped you'd have taken
that into account.

                      Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1249430 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

Am 17.10.2015 um 21:08 schrieb Linus Torvalds:
> On Sat, Oct 17, 2015 at 12:01 PM, Alexander Holler <holler@ahsoftware.de> wrote:
>>
>> That isn't a flag day thing. It's a config option everyone can turn on and
>> off whenever he wants.
>
> That's a flag-day thing. We've done it (drm comes to mind - several times).
>
> I'm disappointed, because I _know_ I pointed you in the direction of
> stable sorting about a month ago. I really had hoped you'd have taken
> that into account.

It's impossible to take it into account because I don't want to miss the 
parallelize functionality. And without that, all the stuff doesn't offer 
enough benefits to be worse the effort but just adds some time necessary 
to do the sorting. It might solve the deferred probe problems, but 
without much benefit.

Regards,

Alexander Holler

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1249434 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

On Sat, Oct 17, 2015 at 12:14 PM, Alexander Holler <holler@ahsoftware.de> wrote:
>
> It's impossible to take it into account because I don't want to miss the
> parallelize functionality. And without that, all the stuff doesn't offer
> enough benefits to be worse the effort but just adds some time necessary to
> do the sorting. It might solve the deferred probe problems, but without much
> benefit.

Quite frankly, the alleged advantage of parallelization of the init
sequence is *very* questionable.

But even if you were to want to parallelize things at that level, you'd still

 (a) want to make *that* a flag (to debug) and make it opt-in (so
you'd have another "this module can be done in parallel" marker)

 (b) end up with all the other drivers *not* being parallel, because
they were all written to not have everything happen at once.

 (c) want to avoid the whole flag-day things

so even then you'd want to sort everything using that topological
stable sort, so that you can sanely mix things, and then mark
individual drivers as "I'd like to run in parallel with the rest of
the init sequence".

The main thing that being parallel would change is likely that you'd
also have to add some kind of "I'm done" completion for each module
(or dependency point), so that things that depend on other things
(that have *started*) can explicitly wait for those other things to
actually end.

We have something like that for our "sync" points right now with the
whole asynchronous machinery. It's somewhat general (although I think
almost all users end up just doing "async_synchronize_full()", even
though we do have a cookie-based one) and I suspect the cookie-based
one could be used for individual drivers and their completions.

But I really think the whole "do initcalls in parallel" is a very bad
idea _regardless_ of anything else. We almost never actually have
initcalls that really take a lot of time, and when we do, we already
handle the big ones with the async machinery, which is actually a lot
more powerful and flexible than some "run initcalls in parallel",
exactly because you can run *parts* serially (like basic discovery),
and then run other parts asynchronously (like "spin up disk" or "wait
for power stable" or whatever - *after* you have discovered the
device).

So I personally think our async thing is much more powerful and useful
than some random "let's run initcalls in parallel".

                 Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1249435 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

On Sat, Oct 17, 2015 at 09:14:34PM +0200, Alexander Holler wrote:
> Am 17.10.2015 um 21:08 schrieb Linus Torvalds:
> >On Sat, Oct 17, 2015 at 12:01 PM, Alexander Holler <holler@ahsoftware.de> wrote:
> >>
> >>That isn't a flag day thing. It's a config option everyone can turn on and
> >>off whenever he wants.
> >
> >That's a flag-day thing. We've done it (drm comes to mind - several times).
> >
> >I'm disappointed, because I _know_ I pointed you in the direction of
> >stable sorting about a month ago. I really had hoped you'd have taken
> >that into account.
> 
> It's impossible to take it into account because I don't want to miss the
> parallelize functionality. And without that, all the stuff doesn't offer
> enough benefits to be worse the effort but just adds some time necessary to
> do the sorting. It might solve the deferred probe problems, but without much
> benefit.

Again, parallelizing does not solve anything, and causes more problems
_and_ makes things take longer.  Try it, we have done it in the past and
proven this, it's pretty easy to test :)

thanks,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1249441 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

Am 17.10.2015 um 21:36 schrieb Greg Kroah-Hartman:
> On Sat, Oct 17, 2015 at 09:14:34PM +0200, Alexander Holler wrote:
>> Am 17.10.2015 um 21:08 schrieb Linus Torvalds:
>>> On Sat, Oct 17, 2015 at 12:01 PM, Alexander Holler <holler@ahsoftware.de> wrote:
>>>>
>>>> That isn't a flag day thing. It's a config option everyone can turn on and
>>>> off whenever he wants.
>>>
>>> That's a flag-day thing. We've done it (drm comes to mind - several times).
>>>
>>> I'm disappointed, because I _know_ I pointed you in the direction of
>>> stable sorting about a month ago. I really had hoped you'd have taken
>>> that into account.
>>
>> It's impossible to take it into account because I don't want to miss the
>> parallelize functionality. And without that, all the stuff doesn't offer
>> enough benefits to be worse the effort but just adds some time necessary to
>> do the sorting. It might solve the deferred probe problems, but without much
>> benefit.
>
> Again, parallelizing does not solve anything, and causes more problems
> _and_ makes things take longer.  Try it, we have done it in the past and
> proven this, it's pretty easy to test :)

I've tested it, otherwise I wouldn't have posted the patches.

Unfortunately it's quiet a lot of work to add dependencies for everything.

So maybe I'm able to offer some better numbers in a year or such, when I 
was bored often enough to add more dependencies for initcalls.

Not to mention that small changes in the order can have quiet big 
differences in the boot time, so it's quiet hard to parallelize stuff 
(add dependencies) correctly like e.g. the pci/acpi/processor stuff. 
Especially because many reasons for the current order aren't mentioned 
in the source and are hard to see without specific knowledge about the HW.

Alexander Holler
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1249469 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

Am 17.10.2015 um 21:58 schrieb Alexander Holler:

> Unfortunately it's quiet a lot of work to add dependencies for everything.

And (just in case of), also I'm a non-native English writer, I know the 
difference between quiet and quite. But, unfortunately, that doesn't 
prevent me to type it wrong. It's like "teh" instead of "the". 
Unfortunately I'm not very good in writing English fast without errors, 
but that doesn't mean I'm dumb (also many native English writers prefer 
to assume that).

Regards,

Alexander Holler
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1250074 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

Am 17.10.2015 um 21:36 schrieb Greg Kroah-Hartman:

> Again, parallelizing does not solve anything, and causes more problems
> _and_ makes things take longer.  Try it, we have done it in the past and
> proven this, it's pretty easy to test :)

Just because I'm curious, may I ask how I would test that in the easy 
way you have in mind? I've just posted the results of my tests (the 
patch series) but I wonder what you do have in mind.

Regards,

Alexander Holler
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1250081 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

On Sun, Oct 18, 2015 at 06:59:22AM +0200, Alexander Holler wrote:
> Am 17.10.2015 um 21:36 schrieb Greg Kroah-Hartman:
> 
> >Again, parallelizing does not solve anything, and causes more problems
> >_and_ makes things take longer.  Try it, we have done it in the past and
> >proven this, it's pretty easy to test :)
> 
> Just because I'm curious, may I ask how I would test that in the easy way
> you have in mind? I've just posted the results of my tests (the patch
> series) but I wonder what you do have in mind.

Use the tool, scripts/bootgraph.pl to create a boot graph of your boot
sequence.  That should show you the drivers, or other areas, that are
causing your boot to be "slow".

Hope this helps,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1250082 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

Am 18.10.2015 um 07:14 schrieb Greg Kroah-Hartman:
> On Sun, Oct 18, 2015 at 06:59:22AM +0200, Alexander Holler wrote:
>> Am 17.10.2015 um 21:36 schrieb Greg Kroah-Hartman:
>>
>>> Again, parallelizing does not solve anything, and causes more problems
>>> _and_ makes things take longer.  Try it, we have done it in the past and
>>> proven this, it's pretty easy to test :)
>>
>> Just because I'm curious, may I ask how I would test that in the easy way
>> you have in mind? I've just posted the results of my tests (the patch
>> series) but I wonder what you do have in mind.
>
> Use the tool, scripts/bootgraph.pl to create a boot graph of your boot
> sequence.  That should show you the drivers, or other areas, that are
> causing your boot to be "slow".

So I've misunderstood you. I've read your paragraph as that it's easy to 
test parallelizing.

> Hope this helps,

Unfortunately no, but thanks anyway. ;)

Alexander Holler

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1250084 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

On Sun, Oct 18, 2015 at 07:20:34AM +0200, Alexander Holler wrote:
> Am 18.10.2015 um 07:14 schrieb Greg Kroah-Hartman:
> >On Sun, Oct 18, 2015 at 06:59:22AM +0200, Alexander Holler wrote:
> >>Am 17.10.2015 um 21:36 schrieb Greg Kroah-Hartman:
> >>
> >>>Again, parallelizing does not solve anything, and causes more problems
> >>>_and_ makes things take longer.  Try it, we have done it in the past and
> >>>proven this, it's pretty easy to test :)
> >>
> >>Just because I'm curious, may I ask how I would test that in the easy way
> >>you have in mind? I've just posted the results of my tests (the patch
> >>series) but I wonder what you do have in mind.
> >
> >Use the tool, scripts/bootgraph.pl to create a boot graph of your boot
> >sequence.  That should show you the drivers, or other areas, that are
> >causing your boot to be "slow".
> 
> So I've misunderstood you. I've read your paragraph as that it's easy to
> test parallelizing.

Ah, ok, if you want to parallelize everything, add some logic in the
driver core where the probe() callback is made to spin that off into a
new thread for every call, and when it's done, clean up the thread.
That's what I did many years ago to try this all out, if you dig in the
lkml archives there's probably a patch somewhere that you can base the
work off of to test it yourself.

hope this helps,

greg k-h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1250103 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

Am 18.10.2015 um 07:59 schrieb Greg Kroah-Hartman:
> On Sun, Oct 18, 2015 at 07:20:34AM +0200, Alexander Holler wrote:
>> Am 18.10.2015 um 07:14 schrieb Greg Kroah-Hartman:
>>> On Sun, Oct 18, 2015 at 06:59:22AM +0200, Alexander Holler wrote:
>>>> Am 17.10.2015 um 21:36 schrieb Greg Kroah-Hartman:
>>>>
>>>>> Again, parallelizing does not solve anything, and causes more problems
>>>>> _and_ makes things take longer.  Try it, we have done it in the past and
>>>>> proven this, it's pretty easy to test :)
>>>>
>>>> Just because I'm curious, may I ask how I would test that in the easy way
>>>> you have in mind? I've just posted the results of my tests (the patch
>>>> series) but I wonder what you do have in mind.
>>>
>>> Use the tool, scripts/bootgraph.pl to create a boot graph of your boot
>>> sequence.  That should show you the drivers, or other areas, that are
>>> causing your boot to be "slow".
>>
>> So I've misunderstood you. I've read your paragraph as that it's easy to
>> test parallelizing.
>
> Ah, ok, if you want to parallelize everything, add some logic in the
> driver core where the probe() callback is made to spin that off into a
> new thread for every call, and when it's done, clean up the thread.
> That's what I did many years ago to try this all out, if you dig in the
> lkml archives there's probably a patch somewhere that you can base the
> work off of to test it yourself.

Hmm, I don't think I will do that because that means to setup a new 
thread for every call. And it doesn't need much imagination (or 
experience) that this introduces quite some overhead.

But maybe it makes sense to try out what I'm doing in my patches, 
starting multiple threads once and then just giving them some work. Will 
keep that in mind, also I don't think I will post any patch in the next 
few years. ;)

Regards,

Alexander Holler
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1250564 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

Am 18.10.2015 um 12:11 schrieb Alexander Holler:
> Am 18.10.2015 um 07:59 schrieb Greg Kroah-Hartman:
>> On Sun, Oct 18, 2015 at 07:20:34AM +0200, Alexander Holler wrote:
>>> Am 18.10.2015 um 07:14 schrieb Greg Kroah-Hartman:
>>>> On Sun, Oct 18, 2015 at 06:59:22AM +0200, Alexander Holler wrote:
>>>>> Am 17.10.2015 um 21:36 schrieb Greg Kroah-Hartman:
>>>>>
>>>>>> Again, parallelizing does not solve anything, and causes more
>>>>>> problems
>>>>>> _and_ makes things take longer.  Try it, we have done it in the
>>>>>> past and
>>>>>> proven this, it's pretty easy to test :)
>>>>>
>>>>> Just because I'm curious, may I ask how I would test that in the
>>>>> easy way
>>>>> you have in mind? I've just posted the results of my tests (the patch
>>>>> series) but I wonder what you do have in mind.
>>>>
>>>> Use the tool, scripts/bootgraph.pl to create a boot graph of your boot
>>>> sequence.  That should show you the drivers, or other areas, that are
>>>> causing your boot to be "slow".
>>>
>>> So I've misunderstood you. I've read your paragraph as that it's easy to
>>> test parallelizing.
>>
>> Ah, ok, if you want to parallelize everything, add some logic in the
>> driver core where the probe() callback is made to spin that off into a
>> new thread for every call, and when it's done, clean up the thread.
>> That's what I did many years ago to try this all out, if you dig in the
>> lkml archives there's probably a patch somewhere that you can base the
>> work off of to test it yourself.
>
> Hmm, I don't think I will do that because that means to setup a new
> thread for every call. And it doesn't need much imagination (or
> experience) that this introduces quite some overhead.
>
> But maybe it makes sense to try out what I'm doing in my patches,
> starting multiple threads once and then just giving them some work. Will

After a having second thought on your simple approach to parallelize 
stuff, I have to say that it just can't work because just starting a 
thread for every probe() totally ignores possible dependencies. 
Regardless if using one thread per probe() call or if feeding probe() 
calls to just a few threads.

Maybe that's why previous attempts to parallelize stuff failed. But 
that's just an assumption as I'm unaware of these previous attempts.

Regards,

Alexander Holler
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1250579 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

Am 19.10.2015 um 12:57 schrieb Alexander Holler:
> Am 18.10.2015 um 12:11 schrieb Alexander Holler:
>> Am 18.10.2015 um 07:59 schrieb Greg Kroah-Hartman:
>>> On Sun, Oct 18, 2015 at 07:20:34AM +0200, Alexander Holler wrote:
>>>> Am 18.10.2015 um 07:14 schrieb Greg Kroah-Hartman:
>>>>> On Sun, Oct 18, 2015 at 06:59:22AM +0200, Alexander Holler wrote:
>>>>>> Am 17.10.2015 um 21:36 schrieb Greg Kroah-Hartman:
>>>>>>
>>>>>>> Again, parallelizing does not solve anything, and causes more
>>>>>>> problems
>>>>>>> _and_ makes things take longer.  Try it, we have done it in the
>>>>>>> past and
>>>>>>> proven this, it's pretty easy to test :)
>>>>>>
>>>>>> Just because I'm curious, may I ask how I would test that in the
>>>>>> easy way
>>>>>> you have in mind? I've just posted the results of my tests (the patch
>>>>>> series) but I wonder what you do have in mind.
>>>>>
>>>>> Use the tool, scripts/bootgraph.pl to create a boot graph of your boot
>>>>> sequence.  That should show you the drivers, or other areas, that are
>>>>> causing your boot to be "slow".
>>>>
>>>> So I've misunderstood you. I've read your paragraph as that it's
>>>> easy to
>>>> test parallelizing.
>>>
>>> Ah, ok, if you want to parallelize everything, add some logic in the
>>> driver core where the probe() callback is made to spin that off into a
>>> new thread for every call, and when it's done, clean up the thread.
>>> That's what I did many years ago to try this all out, if you dig in the
>>> lkml archives there's probably a patch somewhere that you can base the
>>> work off of to test it yourself.
>>
>> Hmm, I don't think I will do that because that means to setup a new
>> thread for every call. And it doesn't need much imagination (or
>> experience) that this introduces quite some overhead.
>>
>> But maybe it makes sense to try out what I'm doing in my patches,
>> starting multiple threads once and then just giving them some work. Will
>
> After a having second thought on your simple approach to parallelize
> stuff, I have to say that it just can't work because just starting a
> thread for every probe() totally ignores possible dependencies.
> Regardless if using one thread per probe() call or if feeding probe()
> calls to just a few threads.
>
> Maybe that's why previous attempts to parallelize stuff failed. But
> that's just an assumption as I'm unaware of these previous attempts.

Or to describe it more verbose, if DEBUG is turned on in 
init/dependencies.c (using my patches), it spits out a summary of groups 
with initcalls (probe() calls) which are independent from each other and 
therfore can be called in parallel. E.g. one of my systems this looks so:

[    0.288229] init: vertices: 429 edges 204 count 170
[    0.288295] init: group 0 length 66 (start 0)
[    0.288329] init: group 1 length 33 (start 66)
[    0.288364] init: group 2 length 13 (start 99)
[    0.288398] init: group 3 length 7 (start 112)
[    0.288432] init: group 4 length 9 (start 119)
[    0.288466] init: group 5 length 8 (start 128)
[    0.288500] init: group 6 length 11 (start 136)
[    0.288534] init: group 7 length 6 (start 147)
[    0.288569] init: group 8 length 4 (start 153)
[    0.288603] init: group 9 length 8 (start 157)
[    0.288637] init: group 10 length 3 (start 165)
[    0.288671] init: group 11 length 2 (start 168)
[    0.288705] init: using 4 threads to call annotated initcalls

That means the first group contains 66 initcalls which are called using 
4 threads, and, after those have finished, the second group with 33 
initcalls will be called in parallel (using the same 4 threads).

> Regards,
>
> Alexander Holler

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1253508 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

Am 19.10.2015 um 13:31 schrieb Alexander Holler:
> Am 19.10.2015 um 12:57 schrieb Alexander Holler:
>> Am 18.10.2015 um 12:11 schrieb Alexander Holler:
>>> Am 18.10.2015 um 07:59 schrieb Greg Kroah-Hartman:
>>>> On Sun, Oct 18, 2015 at 07:20:34AM +0200, Alexander Holler wrote:
>>>>> Am 18.10.2015 um 07:14 schrieb Greg Kroah-Hartman:
>>>>>> On Sun, Oct 18, 2015 at 06:59:22AM +0200, Alexander Holler wrote:
>>>>>>> Am 17.10.2015 um 21:36 schrieb Greg Kroah-Hartman:
>>>>>>>
>>>>>>>> Again, parallelizing does not solve anything, and causes more
>>>>>>>> problems
>>>>>>>> _and_ makes things take longer.  Try it, we have done it in the
>>>>>>>> past and
>>>>>>>> proven this, it's pretty easy to test :)
>>>>>>>
>>>>>>> Just because I'm curious, may I ask how I would test that in the
>>>>>>> easy way
>>>>>>> you have in mind? I've just posted the results of my tests (the
>>>>>>> patch
>>>>>>> series) but I wonder what you do have in mind.
>>>>>>
>>>>>> Use the tool, scripts/bootgraph.pl to create a boot graph of your
>>>>>> boot
>>>>>> sequence.  That should show you the drivers, or other areas, that are
>>>>>> causing your boot to be "slow".
>>>>>
>>>>> So I've misunderstood you. I've read your paragraph as that it's
>>>>> easy to
>>>>> test parallelizing.
>>>>
>>>> Ah, ok, if you want to parallelize everything, add some logic in the
>>>> driver core where the probe() callback is made to spin that off into a
>>>> new thread for every call, and when it's done, clean up the thread.
>>>> That's what I did many years ago to try this all out, if you dig in the
>>>> lkml archives there's probably a patch somewhere that you can base the
>>>> work off of to test it yourself.
>>>
>>> Hmm, I don't think I will do that because that means to setup a new
>>> thread for every call. And it doesn't need much imagination (or
>>> experience) that this introduces quite some overhead.
>>>
>>> But maybe it makes sense to try out what I'm doing in my patches,
>>> starting multiple threads once and then just giving them some work. Will
>>
>> After a having second thought on your simple approach to parallelize
>> stuff, I have to say that it just can't work because just starting a
>> thread for every probe() totally ignores possible dependencies.
>> Regardless if using one thread per probe() call or if feeding probe()
>> calls to just a few threads.
>>
>> Maybe that's why previous attempts to parallelize stuff failed. But
>> that's just an assumption as I'm unaware of these previous attempts.
>
> Or to describe it more verbose, if DEBUG is turned on in
> init/dependencies.c (using my patches), it spits out a summary of groups
> with initcalls (probe() calls) which are independent from each other and
> therfore can be called in parallel. E.g. one of my systems this looks so:
>
> [    0.288229] init: vertices: 429 edges 204 count 170
> [    0.288295] init: group 0 length 66 (start 0)
> [    0.288329] init: group 1 length 33 (start 66)
> [    0.288364] init: group 2 length 13 (start 99)
> [    0.288398] init: group 3 length 7 (start 112)
> [    0.288432] init: group 4 length 9 (start 119)
> [    0.288466] init: group 5 length 8 (start 128)
> [    0.288500] init: group 6 length 11 (start 136)
> [    0.288534] init: group 7 length 6 (start 147)
> [    0.288569] init: group 8 length 4 (start 153)
> [    0.288603] init: group 9 length 8 (start 157)
> [    0.288637] init: group 10 length 3 (start 165)
> [    0.288671] init: group 11 length 2 (start 168)
> [    0.288705] init: using 4 threads to call annotated initcalls
>
> That means the first group contains 66 initcalls which are called using
> 4 threads, and, after those have finished, the second group with 33
> initcalls will be called in parallel (using the same 4 threads).

BTW. That also means that, for the above example, the worst case would 
mean an error rate of 61% if those 170 (annotated) initcalls would be 
started in parallel while ignoring dependencies.

But that's just meant as an (hopefully) interesting number when looking 
at the above numbers a bit different.

(I've understood that the patches aren't wanted.)

>> Regards,
>>
>> Alexander Holler

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1249472 — Re: [PATCH 04/14] init: deps: order network interfaces by link order

Am 17.10.2015 um 21:08 schrieb Linus Torvalds:
> On Sat, Oct 17, 2015 at 12:01 PM, Alexander Holler <holler@ahsoftware.de> wrote:
>>
>> That isn't a flag day thing. It's a config option everyone can turn on and
>> off whenever he wants.
>
> That's a flag-day thing. We've done it (drm comes to mind - several times).
>
> I'm disappointed, because I _know_ I pointed you in the direction of
> stable sorting about a month ago. I really had hoped you'd have taken
> that into account.

Sorry, but I have to add another comment about that stable topological 
sort which can be easily found by searching the web.

As the author mentioned, he has done some search on that topic too and 
has found nothing. So, either I would have to trust his word (and his 
non-public proof) that it works or I would have to prove it myself.

Besides that he says it's based on bubble sort, which is known for its 
speed. So I would have had to write in C in order to see if the speed 
would be acceptable and would still be without any proof that it always 
works correctly.

Regards,

Alexander Holler
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

Page 1 of 3  [1] 2 3  Next page →

Back to top | Article view | linux.kernel

csiph-web