Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #12015

Re: How to map NIC to cpu socket?

Followups directed to: comp.os.linux.misc

Show all headers | View raw

On Monday 08 September 2014 10:41, Wayne conveyed the following to 
comp.os.linux.misc...

> On 9/7/2014 8:19 AM, Aragorn wrote:
>
>> On Sunday 07 September 2014 02:32, David W. Hodgins conveyed the
>> following to comp.os.linux.misc...
>> 
>>> On Sat, 06 Sep 2014 18:52:00 -0400, Wayne <nospam@all.invalid>
>>> wrote:
>>>
>>>> Info about NUMA:
>>>> <https://software.intel.com/en-us/articles/optimizing-applications-
>> for-numa?wapkw=optimizing%20software%20applications%20for%20numa>
>>>
>>> Thanks for that link. I was not aware of numa, and don't have it on
>>> any of the systems I have access to, hence my earlier comment about
>>> the nic being accessed over the pci bus, by any cpu.
>>>
>>> The op's question does make sense now, assuming the system in use
>>> does support numa.
>> 
>> I own a NUMA machine - not in use at the moment - based upon a Tyan
>> motherboard and second generation AMD Opteron processors, and it has
>> two NICs, one on each node [*].
>> 
>> As I understand it, the kernel will handle the NUMA layout based upon
>> the information it receives from the BIOS.  Linux has already had
>> excellent NUMA support for a very long time.  If I'm not mistaken,
>> then that was already implemented somewhere early on during the
>> lifetime of the 2.6 kernel generation.
> 
> What I don't get is that NICs don't pass data directly to
> applications.

In a monolithic kernel like Linux, that is not possible.  The whole 
networking stack runs as part of the kernel.  

Something like the GNU Mach/HURD on the other hand might be able to do 
just that, because that's a microkernel-based system and thus it runs 
its entire networking stack in userspace.  However, I don't have any 
experience with GNU/HURD, and I'm also not a kernel hacker, so don't 
take my word for it.

> So unless you have a kernel (or part of one) running on every NUMA
> node, the packet headers from all but one NIC will still need
> to be accessed by the remote node, where the kernel is.  So remote
> access can't be avoided, is that right?

The kernel will normally try as much as possible to run userspace 
processes on the same node as where the proces' data is stored in 
memory.  However, each CPU - by which I mean "each core", because a 
single node may contain more than one core - has four privilege rings, 
and thus of course also a kernel ring.

Now, provided that the BIOS is set up to initialize the APICs in a 
discrete way - which is the best choice for NUMA systems with NICS 
connected to different nodes - the kernel will have to execute some of 
its code on either node.  By consequence, you will always have a bit of 
remote access going on, given that the kernel itself runs in a single 
address space and will thus most likely be contained within the RAM of a 
single NUMA node.

Now, say that you start up a web browser.  This will typically connect 
to the WAN-side of the machine.  However, the kernel does not know about 
that.  What the kernel does know is what load each of the nodes (and 
their respective cores) are bearing at the moment, and how much memory 
load is available in the RAM of either node.  It will thus try to 
balance the load by putting the application in the most suitable node, 
and having its code execute on the processor cores of that node.  Now, 
if that node requires access to the networking stack, then the kernel 
will check on what interface is connected to which node and will thus 
run the pertinent code where it is needed.

Remote access is inevitable, and this is one of the main reasons why 
such remote access is possible in NUMA systems.  The Linux NUMA 
scheduling is pretty good, so I wouldn't worry about it. ;-)

-- 
= Aragorn =

         http://www.linuxcounter.net - registrant #223157

Back to comp.os.linux.misc | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

How to map NIC to cpu socket? chrippa08@gmail.com - 2014-09-05 06:01 -0700
  Re: How to map NIC to cpu socket? Joe Beanfish <joebeanfish@nospam.duh> - 2014-09-05 13:09 +0000
    Re: How to map NIC to cpu socket? Robert Heller <heller@deepsoft.com> - 2014-09-05 08:51 -0500
      Re: How to map NIC to cpu socket? chrippa08@gmail.com - 2014-09-05 07:49 -0700
        Re: How to map NIC to cpu socket? Robert Heller <heller@deepsoft.com> - 2014-09-05 11:15 -0500
        Re: How to map NIC to cpu socket? "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2014-09-05 12:15 -0400
        Re: How to map NIC to cpu socket? William Unruh <unruh@invalid.ca> - 2014-09-05 20:48 +0000
          Re: How to map NIC to cpu socket? chrippa08@gmail.com - 2014-09-05 23:34 -0700
            Re: How to map NIC to cpu socket? Wayne <nospam@all.invalid> - 2014-09-06 18:52 -0400
              Re: How to map NIC to cpu socket? "David W. Hodgins" <dwhodgins@nomail.afraid.org> - 2014-09-06 20:32 -0400
                Re: How to map NIC to cpu socket? Aragorn <thorongil@telenet.be.invalid> - 2014-09-07 14:19 +0200
                Re: How to map NIC to cpu socket? Wayne <nospam@all.invalid> - 2014-09-08 04:41 -0400
                Re: How to map NIC to cpu socket? Aragorn <thorongil@telenet.be.invalid> - 2014-09-08 14:07 +0200
              Re: How to map NIC to cpu socket? chrippa08@gmail.com - 2014-09-08 03:05 -0700

csiph-web