Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1738947 > unrolled thread

[PATCH v2 00/16] Thunderbolt networking

Back to article view | Back to linux.kernel

Contents

  [PATCH v2 00/16] Thunderbolt networking Mika Westerberg <mika.westerberg@linux.intel.com> - 2017-09-25 13:20 +0200
    Re: [PATCH v2 00/16] Thunderbolt networking Andy Shevchenko <andriy.shevchenko@linux.intel.com> - 2017-09-26 19:40 +0200
    Re: [PATCH v2 06/16] thunderbolt: Add support for XDomain  discovery protocol David Miller <davem@davemloft.net> - 2017-09-27 06:40 +0200
    Re: [PATCH v2 16/16] net: Add support for networking over  Thunderbolt cable David Miller <davem@davemloft.net> - 2017-09-27 06:50 +0200
      Re: [PATCH v2 16/16] net: Add support for networking over  Thunderbolt cable Mika Westerberg <mika.westerberg@linux.intel.com> - 2017-09-27 15:50 +0200
        Re: [PATCH v2 16/16] net: Add support for networking over  Thunderbolt cable David Miller <davem@davemloft.net> - 2017-09-27 18:30 +0200
          Re: [PATCH v2 16/16] net: Add support for networking over  Thunderbolt cable Mika Westerberg <mika.westerberg@linux.intel.com> - 2017-09-27 19:30 +0200
            Re: [PATCH v2 16/16] net: Add support for networking over  Thunderbolt cable David Miller <davem@davemloft.net> - 2017-09-27 20:30 +0200

#1738947 — [PATCH v2 00/16] Thunderbolt networking

Hi all,

In addition of tunneling PCIe, Display Port and USB traffic, Thunderbolt
allows connecting two hosts (domains) over a Thunderbolt cable. It is
possible to tunnel arbitrary data packets over such connection using
high-speed DMA rings available in the Thunderbolt host controller.

In order to discover Thunderbolt services the other host supports, there is
a software protocol running on top of the automatically configured control
channel (ring 0). This protocol is called XDomain discovery protocol and it
uses XDomain properties to describe the host (domain) and the services it
supports.

Once both sides have agreed what services are supported they can enable
high-speed DMA rings to transfer data over the cable.

This series adds support for the XDomain protocol so that we expose each
remote connection as Thunderbolt XDomain device and each service as
Thunderbolt service device. On top of that we create an API that allows
writing drivers for these services and finally we provide an example
Thunderbolt service driver that creates virtual ethernet inferface that
allows tunneling networking packets over Thunderbolt cable. The API could
be used for creating other Thunderbolt services, such as tunneling SCSI for
example.

The XDomain protocol and networking support is also available in macOS and
Windows so this makes it possible to connect Linux to macOS and Windows as
well.

The patches are based on previous Thunderbolt networking patch series by
Amir Levy and Michael Jamet, that can be found here:

  https://lwn.net/Articles/705998/

The main difference to that patch series is that we have the XDomain
protocol running in the kernel now so there is no need for a separate
userspace daemon.

Note this does not affect the existing functionality, so security levels
and NVM firmware upgrade continue to work as before (with the small
exception that now sysfs also shows the XDomain connections and services in
addition to normal Thunderbolt devices). It is also possible to connect up
to 5 Thunderbolt devices and then another host, and the network driver
works exactly the same.

This is second version of the patch series. The previous version (v1) can
be found here:

  https://lwn.net/Articles/734019/

Changes from the v1:

  * Add include/linux/thunderbolt.h to MAINTAINERS
  * Correct Linux version and date of new sysfs entries in
    Documentation/ABI/testing/sysfs-bus-thunderbolt
  * Move network driver from drivers/thunderbolt/net.c to
    drivers/net/thunderbolt.c and update it to follow coding style in
    drivers/net/*.
  * Add MAINTAINERS entry for the network driver
  * Minor cleanups

In case someone wants to try this out, patch [16/16] adds documentation how
the networking driver can be used. In short, if you connect Linux to a
macOS or Windows, everything is done automatically (as those systems have
the networking service enabled by default). For Linux to Linux connection
one host needs to load the networking driver first (so that the other side
can locate the networking service and load the corresponding driver).

Amir Levy (1):
  net: Add support for networking over Thunderbolt cable

Mika Westerberg (15):
  byteorder: Move {cpu_to_be32,be32_to_cpu}_array() from Thunderbolt to core
  thunderbolt: Add support for XDomain properties
  thunderbolt: Move enum tb_cfg_pkg_type to thunderbolt.h
  thunderbolt: Move thunderbolt domain structure to thunderbolt.h
  thunderbolt: Move tb_switch_phy_port_from_link() to thunderbolt.h
  thunderbolt: Add support for XDomain discovery protocol
  thunderbolt: Configure interrupt throttling for all interrupts
  thunderbolt: Add support for frame mode
  thunderbolt: Export ring handling functions to modules
  thunderbolt: Move ring descriptor flags to thunderbolt.h
  thunderbolt: Use spinlock in ring serialization
  thunderbolt: Use spinlock in NHI serialization
  thunderbolt: Add polling mode for rings
  thunderbolt: Add function to retrieve DMA device for the ring
  thunderbolt: Allocate ring HopID automatically if requested

 Documentation/ABI/testing/sysfs-bus-thunderbolt |   48 +
 Documentation/admin-guide/thunderbolt.rst       |   24 +
 MAINTAINERS                                     |    7 +
 drivers/net/Kconfig                             |   12 +
 drivers/net/Makefile                            |    3 +
 drivers/net/thunderbolt.c                       | 1379 ++++++++++++++++++++
 drivers/thunderbolt/Makefile                    |    2 +-
 drivers/thunderbolt/ctl.c                       |   46 +-
 drivers/thunderbolt/ctl.h                       |    3 +-
 drivers/thunderbolt/domain.c                    |  197 ++-
 drivers/thunderbolt/icm.c                       |  218 +++-
 drivers/thunderbolt/nhi.c                       |  409 ++++--
 drivers/thunderbolt/nhi.h                       |  141 +-
 drivers/thunderbolt/nhi_regs.h                  |   11 +-
 drivers/thunderbolt/property.c                  |  670 ++++++++++
 drivers/thunderbolt/switch.c                    |    7 +-
 drivers/thunderbolt/tb.h                        |   88 +-
 drivers/thunderbolt/tb_msgs.h                   |  140 +-
 drivers/thunderbolt/xdomain.c                   | 1576 +++++++++++++++++++++++
 include/linux/byteorder/generic.h               |   16 +
 include/linux/mod_devicetable.h                 |   26 +
 include/linux/thunderbolt.h                     |  598 +++++++++
 scripts/mod/devicetable-offsets.c               |    7 +
 scripts/mod/file2alias.c                        |   25 +
 24 files changed, 5304 insertions(+), 349 deletions(-)
 create mode 100644 drivers/net/thunderbolt.c
 create mode 100644 drivers/thunderbolt/property.c
 create mode 100644 drivers/thunderbolt/xdomain.c
 create mode 100644 include/linux/thunderbolt.h

-- 
2.14.1

[toc] | [next] | [standalone]

#1740059

On Mon, 2017-09-25 at 14:07 +0300, Mika Westerberg wrote:
> Hi all,
> 
> In addition of tunneling PCIe, Display Port and USB traffic,
> Thunderbolt
> allows connecting two hosts (domains) over a Thunderbolt cable. It is
> possible to tunnel arbitrary data packets over such connection using
> high-speed DMA rings available in the Thunderbolt host controller.
> 
> In order to discover Thunderbolt services the other host supports,
> there is
> a software protocol running on top of the automatically configured
> control
> channel (ring 0). This protocol is called XDomain discovery protocol
> and it
> uses XDomain properties to describe the host (domain) and the services
> it
> supports.
> 
> Once both sides have agreed what services are supported they can
> enable
> high-speed DMA rings to transfer data over the cable.
> 
> This series adds support for the XDomain protocol so that we expose
> each
> remote connection as Thunderbolt XDomain device and each service as
> Thunderbolt service device. On top of that we create an API that
> allows
> writing drivers for these services and finally we provide an example
> Thunderbolt service driver that creates virtual ethernet inferface
> that
> allows tunneling networking packets over Thunderbolt cable. The API
> could
> be used for creating other Thunderbolt services, such as tunneling
> SCSI for
> example.
> 
> The XDomain protocol and networking support is also available in macOS
> and
> Windows so this makes it possible to connect Linux to macOS and
> Windows as
> well.
> 
> The patches are based on previous Thunderbolt networking patch series
> by
> Amir Levy and Michael Jamet, that can be found here:
> 
>   https://lwn.net/Articles/705998/
> 
> The main difference to that patch series is that we have the XDomain
> protocol running in the kernel now so there is no need for a separate
> userspace daemon.
> 
> Note this does not affect the existing functionality, so security
> levels
> and NVM firmware upgrade continue to work as before (with the small
> exception that now sysfs also shows the XDomain connections and
> services in
> addition to normal Thunderbolt devices). It is also possible to
> connect up
> to 5 Thunderbolt devices and then another host, and the network driver
> works exactly the same.
> 
> This is second version of the patch series. The previous version (v1)
> can
> be found here:
> 
>   https://lwn.net/Articles/734019/
> 
> Changes from the v1:
> 
>   * Add include/linux/thunderbolt.h to MAINTAINERS
>   * Correct Linux version and date of new sysfs entries in
>     Documentation/ABI/testing/sysfs-bus-thunderbolt
>   * Move network driver from drivers/thunderbolt/net.c to
>     drivers/net/thunderbolt.c and update it to follow coding style in
>     drivers/net/*.
>   * Add MAINTAINERS entry for the network driver
>   * Minor cleanups
> 
> In case someone wants to try this out, patch [16/16] adds
> documentation how
> the networking driver can be used. In short, if you connect Linux to a
> macOS or Windows, everything is done automatically (as those systems
> have
> the networking service enabled by default). For Linux to Linux
> connection
> one host needs to load the networking driver first (so that the other
> side
> can locate the networking service and load the corresponding driver).


Looks awesome!

Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>

(I told privately to Mika about some minors and we agreed with him that
they will be considered only if there will be more comments on the
series)

> 
> Amir Levy (1):
>   net: Add support for networking over Thunderbolt cable
> 
> Mika Westerberg (15):
>   byteorder: Move {cpu_to_be32,be32_to_cpu}_array() from Thunderbolt
> to core
>   thunderbolt: Add support for XDomain properties
>   thunderbolt: Move enum tb_cfg_pkg_type to thunderbolt.h
>   thunderbolt: Move thunderbolt domain structure to thunderbolt.h
>   thunderbolt: Move tb_switch_phy_port_from_link() to thunderbolt.h
>   thunderbolt: Add support for XDomain discovery protocol
>   thunderbolt: Configure interrupt throttling for all interrupts
>   thunderbolt: Add support for frame mode
>   thunderbolt: Export ring handling functions to modules
>   thunderbolt: Move ring descriptor flags to thunderbolt.h
>   thunderbolt: Use spinlock in ring serialization
>   thunderbolt: Use spinlock in NHI serialization
>   thunderbolt: Add polling mode for rings
>   thunderbolt: Add function to retrieve DMA device for the ring
>   thunderbolt: Allocate ring HopID automatically if requested
> 
>  Documentation/ABI/testing/sysfs-bus-thunderbolt |   48 +
>  Documentation/admin-guide/thunderbolt.rst       |   24 +
>  MAINTAINERS                                     |    7 +
>  drivers/net/Kconfig                             |   12 +
>  drivers/net/Makefile                            |    3 +
>  drivers/net/thunderbolt.c                       | 1379
> ++++++++++++++++++++
>  drivers/thunderbolt/Makefile                    |    2 +-
>  drivers/thunderbolt/ctl.c                       |   46 +-
>  drivers/thunderbolt/ctl.h                       |    3 +-
>  drivers/thunderbolt/domain.c                    |  197 ++-
>  drivers/thunderbolt/icm.c                       |  218 +++-
>  drivers/thunderbolt/nhi.c                       |  409 ++++--
>  drivers/thunderbolt/nhi.h                       |  141 +-
>  drivers/thunderbolt/nhi_regs.h                  |   11 +-
>  drivers/thunderbolt/property.c                  |  670 ++++++++++
>  drivers/thunderbolt/switch.c                    |    7 +-
>  drivers/thunderbolt/tb.h                        |   88 +-
>  drivers/thunderbolt/tb_msgs.h                   |  140 +-
>  drivers/thunderbolt/xdomain.c                   | 1576
> +++++++++++++++++++++++
>  include/linux/byteorder/generic.h               |   16 +
>  include/linux/mod_devicetable.h                 |   26 +
>  include/linux/thunderbolt.h                     |  598 +++++++++
>  scripts/mod/devicetable-offsets.c               |    7 +
>  scripts/mod/file2alias.c                        |   25 +
>  24 files changed, 5304 insertions(+), 349 deletions(-)
>  create mode 100644 drivers/net/thunderbolt.c
>  create mode 100644 drivers/thunderbolt/property.c
>  create mode 100644 drivers/thunderbolt/xdomain.c
>  create mode 100644 include/linux/thunderbolt.h
> 

-- 
Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Intel Finland Oy

[toc] | [prev] | [next] | [standalone]

#1740383 — Re: [PATCH v2 06/16] thunderbolt: Add support for XDomain discovery protocol

From: Mika Westerberg <mika.westerberg@linux.intel.com>
Date: Mon, 25 Sep 2017 14:07:28 +0300

> +struct icm_fr_event_xdomain_connected {
> +	struct icm_pkg_header hdr;
> +	u16 reserved;
> +	u16 link_info;
> +	uuid_t remote_uuid;
> +	uuid_t local_uuid;
> +	u32 local_route_hi;
> +	u32 local_route_lo;
> +	u32 remote_route_hi;
> +	u32 remote_route_lo;
> +} __packed;
> +
> +struct icm_fr_event_xdomain_disconnected {
> +	struct icm_pkg_header hdr;
> +	u16 reserved;
> +	u16 link_info;
> +	uuid_t remote_uuid;
> +} __packed;
> +
>  struct icm_fr_pkg_add_device_key {
>  	struct icm_pkg_header hdr;
>  	uuid_t ep_uuid;

Again, __packed should be avoided unless absolutely necessary.

Thank you.

[toc] | [prev] | [next] | [standalone]

#1740386 — Re: [PATCH v2 16/16] net: Add support for networking over Thunderbolt cable

From: Mika Westerberg <mika.westerberg@linux.intel.com>
Date: Mon, 25 Sep 2017 14:07:38 +0300

> +struct thunderbolt_ip_header {
> +	u32 route_hi;
> +	u32 route_lo;
> +	u32 length_sn;
> +	uuid_t uuid;
> +	uuid_t initiator_uuid;
> +	uuid_t target_uuid;
> +	u32 type;
> +	u32 command_id;
> +} __packed;

Again, the __packed attribute should not be necessary and needs to be
removed.

> +static void tbnet_pull_tail(struct sk_buff *skb)
> +{
> +	skb_frag_t *frag = &skb_shinfo(skb)->frags[0];
> +	unsigned int pull_len;
> +	void *hdr;
> +
> +	hdr = skb_frag_address(frag);
> +	pull_len = eth_get_headlen(hdr, TBNET_RX_HDR_SIZE);
> +
> +	/* Align pull length to size of long to optimize memcpy performance */
> +	skb_copy_to_linear_data(skb, hdr, ALIGN(pull_len, sizeof(long)));

You do not need to copy here, instead you can build SKB's where the
skb->data points directly at the head of your first frag page memory.

See build_skb().

> +		skb = net->skb;
> +		if (!skb) {
> +			skb = netdev_alloc_skb_ip_align(net->dev,
> +							TBNET_RX_HDR_SIZE);
> +			net->skb = skb;
> +		}
> +		if (!skb)
> +			break;
> +
> +		/* Single small buffer we can copy directly to the
> +		 * header part of the skb.
> +		 */
> +		if (hdr->frame_count == 1 && frame_size <= TBNET_RX_HDR_SIZE) {

Here you would use build_skb() instead of netdev_alloc_skb*() for the first
frag, and keep the existing code tacking on subsequent frags using
skb_add_Rx_frag().

> +	ret = register_netdev(dev);
> +	if (ret) {
> +		free_netdev(dev);
> +		return ret;
> +	}
> +
> +	net->handler.uuid = &tbnet_svc_uuid;
> +	net->handler.callback = tbnet_handle_packet,
> +	net->handler.data = net;
> +	tb_register_protocol_handler(&net->handler);
> +
> +	tb_service_set_drvdata(svc, net);

There could be races here.

At the exact moment you call register_netdev(), your device can be
brought UP, packets transmitted, etc.  You entire set of driver code
paths can be executed.

The rest of those initializations after register_netdev() probably
are needed by the rest of the driver to function properly, so may
need to happen before register_netdev() publishes the device to the
entire world.

[toc] | [prev] | [next] | [standalone]

#1740739 — Re: [PATCH v2 16/16] net: Add support for networking over Thunderbolt cable

On Tue, Sep 26, 2017 at 09:47:21PM -0700, David Miller wrote:
> From: Mika Westerberg <mika.westerberg@linux.intel.com>
> Date: Mon, 25 Sep 2017 14:07:38 +0300
> 
> > +struct thunderbolt_ip_header {
> > +	u32 route_hi;
> > +	u32 route_lo;
> > +	u32 length_sn;
> > +	uuid_t uuid;
> > +	uuid_t initiator_uuid;
> > +	uuid_t target_uuid;
> > +	u32 type;
> > +	u32 command_id;
> > +} __packed;
> 
> Again, the __packed attribute should not be necessary and needs to be
> removed.

OK, will do.

> > +static void tbnet_pull_tail(struct sk_buff *skb)
> > +{
> > +	skb_frag_t *frag = &skb_shinfo(skb)->frags[0];
> > +	unsigned int pull_len;
> > +	void *hdr;
> > +
> > +	hdr = skb_frag_address(frag);
> > +	pull_len = eth_get_headlen(hdr, TBNET_RX_HDR_SIZE);
> > +
> > +	/* Align pull length to size of long to optimize memcpy performance */
> > +	skb_copy_to_linear_data(skb, hdr, ALIGN(pull_len, sizeof(long)));
> 
> You do not need to copy here, instead you can build SKB's where the
> skb->data points directly at the head of your first frag page memory.
> 
> See build_skb().
> 
> > +		skb = net->skb;
> > +		if (!skb) {
> > +			skb = netdev_alloc_skb_ip_align(net->dev,
> > +							TBNET_RX_HDR_SIZE);
> > +			net->skb = skb;
> > +		}
> > +		if (!skb)
> > +			break;
> > +
> > +		/* Single small buffer we can copy directly to the
> > +		 * header part of the skb.
> > +		 */
> > +		if (hdr->frame_count == 1 && frame_size <= TBNET_RX_HDR_SIZE) {
> 
> Here you would use build_skb() instead of netdev_alloc_skb*() for the first
> frag, and keep the existing code tacking on subsequent frags using
> skb_add_Rx_frag().

I'm reading kernel-doc of build_skb() (or rather __build_skb()) and it
says caller needs to reserve head room of NET_SKB_PAD and then make sure
there is space for SKB_DATA_ALIGN(skb_shared_info).

Now, in case of ThunderboltIP frames, they look like this:

  +---------+
  | hdr     | 12 bytes
  +---------+
  | data    | 4096 - 12 = 4084 bytes
  |         |
  |         |
  +---------+

A packet can consist of multiple frames where each have the 12-byte
header and 4084 bytes of TSO/LRO payload except the last one which can
be smaller than 4084.

Using build_skb() then would require to allocate larger buffer, that
includes NET_SKB_PAD + SKB_DATA_ALIGN(skb_shared_info) and that exceeds
page size. Is this something supported by build_skb()? It was not clear
to me based on the code and other users of build_skb() but I may be
missing something.

> > +	ret = register_netdev(dev);
> > +	if (ret) {
> > +		free_netdev(dev);
> > +		return ret;
> > +	}
> > +
> > +	net->handler.uuid = &tbnet_svc_uuid;
> > +	net->handler.callback = tbnet_handle_packet,
> > +	net->handler.data = net;
> > +	tb_register_protocol_handler(&net->handler);
> > +
> > +	tb_service_set_drvdata(svc, net);
> 
> There could be races here.
> 
> At the exact moment you call register_netdev(), your device can be
> brought UP, packets transmitted, etc.  You entire set of driver code
> paths can be executed.
> 
> The rest of those initializations after register_netdev() probably
> are needed by the rest of the driver to function properly, so may
> need to happen before register_netdev() publishes the device to the
> entire world.

You're right. I'll change the ordering so that register_netdev() happens
last.

Thanks!

[toc] | [prev] | [next] | [standalone]

#1740873 — Re: [PATCH v2 16/16] net: Add support for networking over Thunderbolt cable

From: Mika Westerberg <mika.westerberg@linux.intel.com>
Date: Wed, 27 Sep 2017 16:42:38 +0300

> Using build_skb() then would require to allocate larger buffer, that
> includes NET_SKB_PAD + SKB_DATA_ALIGN(skb_shared_info) and that exceeds
> page size. Is this something supported by build_skb()? It was not clear
> to me based on the code and other users of build_skb() but I may be
> missing something.

You need NET_SKB_PAD before and SKB_DATA_ALIGN(skb_shared_info) afterwards.
An order 1 page, if that's what you need, should work just fine.

[toc] | [prev] | [next] | [standalone]

#1740904 — Re: [PATCH v2 16/16] net: Add support for networking over Thunderbolt cable

On Wed, Sep 27, 2017 at 09:27:09AM -0700, David Miller wrote:
> From: Mika Westerberg <mika.westerberg@linux.intel.com>
> Date: Wed, 27 Sep 2017 16:42:38 +0300
> 
> > Using build_skb() then would require to allocate larger buffer, that
> > includes NET_SKB_PAD + SKB_DATA_ALIGN(skb_shared_info) and that exceeds
> > page size. Is this something supported by build_skb()? It was not clear
> > to me based on the code and other users of build_skb() but I may be
> > missing something.
> 
> You need NET_SKB_PAD before and SKB_DATA_ALIGN(skb_shared_info) afterwards.
> An order 1 page, if that's what you need, should work just fine.

I mean in order to fit a single ThunderboltIP frame, I would need to
allocate NET_SKB_PAD+4096+SKB_DATA_ALIGN(skb_shared_info) size buffer.
Is that still fine for build_skb()? Also can I use that with
skb_add_rx_frag() which seem to take single page?

ThunderboltIP protocol basically takes advantage of TSO/LRO but it
actually does not do any segmentation. Instead it just splits the 64kB
large package into smaller 4k frames (which each include 12 byte header)
and pushes those over the Thunderbolt medium. The receiver side then
does the opposite.

Thanks and sorry for dummy questions. I'm just not too familiar with
the networking subsystem (yet).

[toc] | [prev] | [next] | [standalone]

#1740940 — Re: [PATCH v2 16/16] net: Add support for networking over Thunderbolt cable

From: Mika Westerberg <mika.westerberg@linux.intel.com>
Date: Wed, 27 Sep 2017 20:27:02 +0300

> On Wed, Sep 27, 2017 at 09:27:09AM -0700, David Miller wrote:
>> From: Mika Westerberg <mika.westerberg@linux.intel.com>
>> Date: Wed, 27 Sep 2017 16:42:38 +0300
>> 
>> > Using build_skb() then would require to allocate larger buffer, that
>> > includes NET_SKB_PAD + SKB_DATA_ALIGN(skb_shared_info) and that exceeds
>> > page size. Is this something supported by build_skb()? It was not clear
>> > to me based on the code and other users of build_skb() but I may be
>> > missing something.
>> 
>> You need NET_SKB_PAD before and SKB_DATA_ALIGN(skb_shared_info) afterwards.
>> An order 1 page, if that's what you need, should work just fine.
> 
> I mean in order to fit a single ThunderboltIP frame, I would need to
> allocate NET_SKB_PAD+4096+SKB_DATA_ALIGN(skb_shared_info) size buffer.

Which would be an order 1 page or 8192 bytes.

> Is that still fine for build_skb()? Also can I use that with
> skb_add_rx_frag() which seem to take single page?

Again, an order 1 page should work fine.

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web