Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #199610 > unrolled thread

Re: Network issue

Started byReco <recoverym4n@gmail.com>
First post2018-08-29 21:10 +0200
Last post2018-08-30 13:00 +0200
Articles 10 — 5 participants

Back to article view | Back to linux.debian.user

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Network issue Reco <recoverym4n@gmail.com> - 2018-08-29 21:10 +0200
    Re: Network issue Fekete Tamás <fektom@gmail.com> - 2018-08-29 21:30 +0200
      Re: Network issue Fekete Tamás <fektom@gmail.com> - 2018-08-29 21:50 +0200
      Re: Network issue Lee <ler762@gmail.com> - 2018-08-30 00:40 +0200
    Re: Network issue Nicholas Geovanis <nickgeovanis@gmail.com> - 2018-08-29 22:40 +0200
      Re: Network issue Reco <recoverym4n@gmail.com> - 2018-08-30 06:30 +0200
    Re: Network issue Michael Stone <mstone@debian.org> - 2018-08-30 00:50 +0200
      Re: Network issue Reco <recoverym4n@gmail.com> - 2018-08-30 06:40 +0200
        Re: Network issue Michael Stone <mstone@debian.org> - 2018-08-30 12:40 +0200
          Re: Network issue Reco <recoverym4n@gmail.com> - 2018-08-30 13:00 +0200

#199610 — Re: Network issue

FromReco <recoverym4n@gmail.com>
Date2018-08-29 21:10 +0200
SubjectRe: Network issue
Message-ID<wse0O-3ch-11@gated-at.bofh.it>
	Hi.

Please do not top post. This is a mailing list, not a corporate e-mail
spamfest.

On Wed, Aug 29, 2018 at 01:18:52PM -0500, Nicholas Geovanis wrote:
> Hi, I've never used OVH. How certain are you that the e1000 network driver
> is the correct one?
> Under VMWare/ESX the network driver choice can be crucial, for example.

1) You cannot run Xen in VSphere/ESXi.

2) No sane public provider will use VSphere/ESXi for hosting, the costs
can dim a budget of a small country.

3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday
reality. SR-IOV is a way to go.


Now, to the issue at hand.

> On Wed, Aug 29, 2018 at 6:30 AM Kevin DAGNEAUX <kevin.dagneaux@fiitelcom.fr>
> wrote:

Please note that you seem to have "link down" first:

> > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down

and 'malfunctioning' netfilter rules next.

> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0

Observe 'RST' flag at each and every 'DROPPED' message.
I find it highly unlikely that whatever PHP load test you did it
involved sending mass amounts of TCP Reset packets.

It seems that the following scenario has much higher probability:

1) You fired your load generator application.

2) Your hosting provided immediately got a signal of your typical DOS
attack (not to be confused with DDOS) coming from that seems to be a
typical mobile phone.

3) DDOS protection kicked in:

a) Isolating your server from the network to stop DOS.

b) Sending forged TCP RST to your server to break existing connections
*and* termninate unneeded Apache workers (or whatever you have there).

c) Banning the initiator of DOS (i.e. you on mobile network) temporarily.

4) Real network outage of your server was 2 seconds (time between "link
down" and "link up").

Reco

> 
> 
> > Hi,
> >
> > I've a server in OVH datacenter, on this server i've 7 VMs, on 1 of them
> > in run Apache.
> > To debug a slow upload (who was ~2Mo/s instead 12Mo/s) i've installed an
> > HTML5/PHP speed test application.
> > When i use this app, i've no problem in general, but, when a make a speed
> > test from a source who have more bandwith than the server (the server is
> > limited at 100Mb/s by OVH and i make the test from a 4G+ network where i've
> > ~150Mb/s of bandwith), in this case, the DOM0 lost his network connection
> > (like the ethernet cable is unplugged) until i reboot the server.
> >
> > When i check the syslog of DOM0, i see that iptables drop incomming packet
> > on port 80 instead of routing them to the VM.
> >
> > This is my iptables script i use on DOM0 :
> >
> > #!/bin/bash
> >
> > IPT="/sbin/iptables"
> >
> >
> > ###########################################################################################
> > # Filter
> >
> > ## Remise par defaut des regles
> > $IPT -t filter -P INPUT   ACCEPT
> > $IPT -t filter -P FORWARD ACCEPT
> > $IPT -t filter -P OUTPUT  ACCEPT
> >
> > ## On purge les tables
> > $IPT -t filter -F
> >
> > ## On autorise lo
> > $IPT -t filter -A INPUT -i lo -j ACCEPT
> >
> > ## On ouvre les ports nécéssaires au DOM0
> > $IPT -t filter -A INPUT -m tcp -p tcp --dport 22      -j
> > ACCEPT                                         ## SSH
> > $IPT -t filter -A INPUT -m udp -p udp --dport 53      -j
> > ACCEPT                                         ## DNS
> > $IPT -t filter -A INPUT -m icmp -p icmp --icmp-type 8 -j
> > ACCEPT                                         ## Ping
> > $IPT -t filter -A INPUT -s 10.0.0.0/24 -j ACCEPT
> >
> > ## On accepte si la connexion est déjà établie
> > $IPT -t filter -A INPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
> >
> > ## On log ce qui n'a pas été matché par les règles précédente
> > $IPT -A INPUT -p tcp -j LOG --log-prefix "DROPED packets "
> >
> > ## On bloque tout le reste
> > $IPT -t filter -P INPUT DROP
> >
> >
> > ############################################################################################
> > # Nat
> >
> > ## Remise par defaut des regles
> > $IPT -t nat -P PREROUTING  ACCEPT
> > $IPT -t nat -P POSTROUTING ACCEPT
> > $IPT -t nat -P INPUT       ACCEPT
> > $IPT -t nat -P OUTPUT      ACCEPT
> >
> > ## On purge
> > $IPT -t nat -F
> >
> > ### Routage des ports entrants pour la VM "mails"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22030 -j DNAT --to
> > 10.0.0.30:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 25    -j DNAT --to
> > 10.0.0.30:25                       ## SMTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 587   -j DNAT --to
> > 10.0.0.30:587                      ## SMTP SUBMISSION
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 465   -j DNAT --to
> > 10.0.0.30:465                      ## SMTP SSL
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 143   -j DNAT --to
> > 10.0.0.30:143                      ## IMAP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 993   -j DNAT --to
> > 10.0.0.30:993                      ## IMAP SSL
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 9930  -j DNAT --to
> > 10.0.0.30:9930                     ## IMAP SSL
> >
> > ### Routage des ports entrants pour la VM "sql"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22020 -j DNAT --to
> > 10.0.0.20:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 3306  -j DNAT --to
> > 10.0.0.20:3306                     ## MariaDB
> >
> > ### Routage des ports entrants pour la VM "files"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22010 -j DNAT --to
> > 10.0.0.10:22                       ## SSH
> >
> > ### Routage des ports entrant pour la VM "web"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22040 -j DNAT --to
> > 10.0.0.40:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 80    -j DNAT --to
> > 10.0.0.40:80                       ## HTTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 443   -j DNAT --to
> > 10.0.0.40:443                      ## HTTPS
> >
> > ### Routage des ports entrants pour la VM "monitor"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22050 -j DNAT --to
> > 10.0.0.50:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 850 -j DNAT --to
> > 10.0.0.50:80                 ## HTTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 855 -j DNAT --to
> > 10.0.0.50:443                        ## HTTPS
> >
> > ### Routage des ports entrants pour la VM "comm"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22060 -j DNAT --to
> > 10.0.0.60:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5222  -j DNAT --to
> > 10.0.0.60:5222                     ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5269  -j DNAT --to
> > 10.0.0.60:5269                     ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5280  -j DNAT --to
> > 10.0.0.60:5280                     ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5281  -j DNAT --to
> > 10.0.0.60:5281                     ## Jabber
> >
> > ### Routage des ports entrants pour la VM "secure"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22070 -j DNAT --to
> > 10.0.0.70:22                       ## SSH
> >
> > ### Routage des ports entrants pour la VM "net"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22080 -j DNAT --to
> > 10.0.0.80:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 8388  -j DNAT --to
> > 10.0.0.80:8388                     ## shadowsocks
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p udp --dport 8388  -j DNAT --to
> > 10.0.0.80:8388                     ## shadowsocks
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 11094 -j DNAT --to
> > 10.0.0.80:1194                     ## OpenVPN
> >
> > ### Autorise les VMs a accéder a internet
> > $IPT -t nat -A POSTROUTING -s 10.0.0.0/24 -j  MASQUERADE
> >
> > And this is what i have in syslogs :
> >
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2226 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2227 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2228 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> >
> > *** I've several hundred of similare line where only the ID change, then :
> > ***
> >
> > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down
> > Aug 28 15:50:32 ovh-1 systemd-networkd[20998]: enp1s0: Lost carrier
> > Aug 28 15:50:34 ovh-1 systemd-networkd[20998]: enp1s0: Gained carrier
> > Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps Full
> > Duplex, Flow Control: None
> > Aug 28 15:50:34 ovh-1 kernel: e1000e 0000:01:00.0 enp1s0: 10/100 speed:
> > disabling TSO
> >
> > The log show that the network is UP but the server is offline.
> > In the VM's logs i don't have abnormal things.
> >
> > OVH have changed the motherboard and the ethernet cable but the problem
> > still persist, the syslog show that it's probably a software problem.
> >
> > Do you have idea of what happen or a way to help me to debug this problem?
> >
> > Thank you in advance.
> >
> > Kevin
> > --
> >
> > DAGNEAUX Kevin
> > Service informatique
> > 03 29 36 88 85
> > kevin.dagneaux@fiitelcom.fr
> >

[toc] | [next] | [standalone]

#199614

FromFekete Tamás <fektom@gmail.com>
Date2018-08-29 21:30 +0200
Message-ID<wsek9-3ia-5@gated-at.bofh.it>
In reply to#199610

[Multipart message — attachments visible in raw view] — view raw

Dear all,

the explanation of Reco seems quite good for me.
If you use load generator on your mobile device it could be the reason.

What happens if you use a normal download for a large file? Do you get the
same result?

If yes, I advise to look at your hardware configuration like flow control.
This line doesn't seem good for me (mind the text in bold):

"Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps Full
Duplex, *Flow Control: None"*

See this link:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-flow_control_for_e1000_network_interface_cards

Let's see what happens if you reconfigure your network device.

- Tamas Fekete

2018-08-29 20:58 GMT+02:00 Reco <recoverym4n@gmail.com>:

>         Hi.
>
> Please do not top post. This is a mailing list, not a corporate e-mail
> spamfest.
>
> On Wed, Aug 29, 2018 at 01:18:52PM -0500, Nicholas Geovanis wrote:
> > Hi, I've never used OVH. How certain are you that the e1000 network
> driver
> > is the correct one?
> > Under VMWare/ESX the network driver choice can be crucial, for example.
>
> 1) You cannot run Xen in VSphere/ESXi.
>
> 2) No sane public provider will use VSphere/ESXi for hosting, the costs
> can dim a budget of a small country.
>
> 3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday
> reality. SR-IOV is a way to go.
>
>
> Now, to the issue at hand.
>
> > On Wed, Aug 29, 2018 at 6:30 AM Kevin DAGNEAUX <
> kevin.dagneaux@fiitelcom.fr>
> > wrote:
>
> Please note that you seem to have "link down" first:
>
> > > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down
>
> and 'malfunctioning' netfilter rules next.
>
> > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
> > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
>
> Observe 'RST' flag at each and every 'DROPPED' message.
> I find it highly unlikely that whatever PHP load test you did it
> involved sending mass amounts of TCP Reset packets.
>
> It seems that the following scenario has much higher probability:
>
> 1) You fired your load generator application.
>
> 2) Your hosting provided immediately got a signal of your typical DOS
> attack (not to be confused with DDOS) coming from that seems to be a
> typical mobile phone.
>
> 3) DDOS protection kicked in:
>
> a) Isolating your server from the network to stop DOS.
>
> b) Sending forged TCP RST to your server to break existing connections
> *and* termninate unneeded Apache workers (or whatever you have there).
>
> c) Banning the initiator of DOS (i.e. you on mobile network) temporarily.
>
> 4) Real network outage of your server was 2 seconds (time between "link
> down" and "link up").
>
> Reco
>
> >
> >
> > > Hi,
> > >
> > > I've a server in OVH datacenter, on this server i've 7 VMs, on 1 of
> them
> > > in run Apache.
> > > To debug a slow upload (who was ~2Mo/s instead 12Mo/s) i've installed
> an
> > > HTML5/PHP speed test application.
> > > When i use this app, i've no problem in general, but, when a make a
> speed
> > > test from a source who have more bandwith than the server (the server
> is
> > > limited at 100Mb/s by OVH and i make the test from a 4G+ network where
> i've
> > > ~150Mb/s of bandwith), in this case, the DOM0 lost his network
> connection
> > > (like the ethernet cable is unplugged) until i reboot the server.
> > >
> > > When i check the syslog of DOM0, i see that iptables drop incomming
> packet
> > > on port 80 instead of routing them to the VM.
> > >
> > > This is my iptables script i use on DOM0 :
> > >
> > > #!/bin/bash
> > >
> > > IPT="/sbin/iptables"
> > >
> > >
> > > ############################################################
> ###############################
> > > # Filter
> > >
> > > ## Remise par defaut des regles
> > > $IPT -t filter -P INPUT   ACCEPT
> > > $IPT -t filter -P FORWARD ACCEPT
> > > $IPT -t filter -P OUTPUT  ACCEPT
> > >
> > > ## On purge les tables
> > > $IPT -t filter -F
> > >
> > > ## On autorise lo
> > > $IPT -t filter -A INPUT -i lo -j ACCEPT
> > >
> > > ## On ouvre les ports nécéssaires au DOM0
> > > $IPT -t filter -A INPUT -m tcp -p tcp --dport 22      -j
> > > ACCEPT                                         ## SSH
> > > $IPT -t filter -A INPUT -m udp -p udp --dport 53      -j
> > > ACCEPT                                         ## DNS
> > > $IPT -t filter -A INPUT -m icmp -p icmp --icmp-type 8 -j
> > > ACCEPT                                         ## Ping
> > > $IPT -t filter -A INPUT -s 10.0.0.0/24 -j ACCEPT
> > >
> > > ## On accepte si la connexion est déjà établie
> > > $IPT -t filter -A INPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
> > >
> > > ## On log ce qui n'a pas été matché par les règles précédente
> > > $IPT -A INPUT -p tcp -j LOG --log-prefix "DROPED packets "
> > >
> > > ## On bloque tout le reste
> > > $IPT -t filter -P INPUT DROP
> > >
> > >
> > > ############################################################
> ################################
> > > # Nat
> > >
> > > ## Remise par defaut des regles
> > > $IPT -t nat -P PREROUTING  ACCEPT
> > > $IPT -t nat -P POSTROUTING ACCEPT
> > > $IPT -t nat -P INPUT       ACCEPT
> > > $IPT -t nat -P OUTPUT      ACCEPT
> > >
> > > ## On purge
> > > $IPT -t nat -F
> > >
> > > ### Routage des ports entrants pour la VM "mails"
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22030 -j DNAT
> --to
> > > 10.0.0.30:22                       ## SSH
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 25    -j DNAT
> --to
> > > 10.0.0.30:25                       ## SMTP
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 587   -j DNAT
> --to
> > > 10.0.0.30:587                      ## SMTP SUBMISSION
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 465   -j DNAT
> --to
> > > 10.0.0.30:465                      ## SMTP SSL
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 143   -j DNAT
> --to
> > > 10.0.0.30:143                      ## IMAP
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 993   -j DNAT
> --to
> > > 10.0.0.30:993                      ## IMAP SSL
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 9930  -j DNAT
> --to
> > > 10.0.0.30:9930                     ## IMAP SSL
> > >
> > > ### Routage des ports entrants pour la VM "sql"
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22020 -j DNAT
> --to
> > > 10.0.0.20:22                       ## SSH
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 3306  -j DNAT
> --to
> > > 10.0.0.20:3306                     ## MariaDB
> > >
> > > ### Routage des ports entrants pour la VM "files"
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22010 -j DNAT
> --to
> > > 10.0.0.10:22                       ## SSH
> > >
> > > ### Routage des ports entrant pour la VM "web"
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22040 -j DNAT
> --to
> > > 10.0.0.40:22                       ## SSH
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 80    -j DNAT
> --to
> > > 10.0.0.40:80                       ## HTTP
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 443   -j DNAT
> --to
> > > 10.0.0.40:443                      ## HTTPS
> > >
> > > ### Routage des ports entrants pour la VM "monitor"
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22050 -j DNAT
> --to
> > > 10.0.0.50:22                       ## SSH
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 850 -j DNAT
> --to
> > > 10.0.0.50:80                 ## HTTP
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 855 -j DNAT
> --to
> > > 10.0.0.50:443                        ## HTTPS
> > >
> > > ### Routage des ports entrants pour la VM "comm"
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22060 -j DNAT
> --to
> > > 10.0.0.60:22                       ## SSH
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5222  -j DNAT
> --to
> > > 10.0.0.60:5222                     ## Jabber
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5269  -j DNAT
> --to
> > > 10.0.0.60:5269                     ## Jabber
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5280  -j DNAT
> --to
> > > 10.0.0.60:5280                     ## Jabber
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5281  -j DNAT
> --to
> > > 10.0.0.60:5281                     ## Jabber
> > >
> > > ### Routage des ports entrants pour la VM "secure"
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22070 -j DNAT
> --to
> > > 10.0.0.70:22                       ## SSH
> > >
> > > ### Routage des ports entrants pour la VM "net"
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22080 -j DNAT
> --to
> > > 10.0.0.80:22                       ## SSH
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 8388  -j DNAT
> --to
> > > 10.0.0.80:8388                     ## shadowsocks
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p udp --dport 8388  -j DNAT
> --to
> > > 10.0.0.80:8388                     ## shadowsocks
> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 11094 -j DNAT
> --to
> > > 10.0.0.80:1194                     ## OpenVPN
> > >
> > > ### Autorise les VMs a accéder a internet
> > > $IPT -t nat -A POSTROUTING -s 10.0.0.0/24 -j  MASQUERADE
> > >
> > > And this is what i have in syslogs :
> > >
> > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2226 DF PROTO=TCP
> > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2227 DF PROTO=TCP
> > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2228 DF PROTO=TCP
> > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
> > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > >
> > > *** I've several hundred of similare line where only the ID change,
> then :
> > > ***
> > >
> > > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down
> > > Aug 28 15:50:32 ovh-1 systemd-networkd[20998]: enp1s0: Lost carrier
> > > Aug 28 15:50:34 ovh-1 systemd-networkd[20998]: enp1s0: Gained carrier
> > > Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps
> Full
> > > Duplex, Flow Control: None
> > > Aug 28 15:50:34 ovh-1 kernel: e1000e 0000:01:00.0 enp1s0: 10/100 speed:
> > > disabling TSO
> > >
> > > The log show that the network is UP but the server is offline.
> > > In the VM's logs i don't have abnormal things.
> > >
> > > OVH have changed the motherboard and the ethernet cable but the problem
> > > still persist, the syslog show that it's probably a software problem.
> > >
> > > Do you have idea of what happen or a way to help me to debug this
> problem?
> > >
> > > Thank you in advance.
> > >
> > > Kevin
> > > --
> > >
> > > DAGNEAUX Kevin
> > > Service informatique
> > > 03 29 36 88 85
> > > kevin.dagneaux@fiitelcom.fr
> > >
>
>
>
>

[toc] | [prev] | [next] | [standalone]

#199616

FromFekete Tamás <fektom@gmail.com>
Date2018-08-29 21:50 +0200
Message-ID<wseDv-3oD-5@gated-at.bofh.it>
In reply to#199614

[Multipart message — attachments visible in raw view] — view raw

Dear Kevin,

and just one more thing.
I am beginner with iptables, but your POSTROUTING doesn't seem good as well.

You made a POSTROUTING in the case of VM-s what you manage (they can see
the internet), but there is no source change on IP packets which goes in
the direction of your VM-s. Your current rule works only in two direction,
if any source from any direction is in the 10.0.0.0/24 LAN.

I think you should add there a second postrouting rule.
And for POSTROUTING you should use -i and -o flags instead of -s, because I
am not sure if iptables understands which IP address should be the source
when it makes the masquerade.

But first of all, I think you should check the hardware flow control
setting.

- Tamas Fekete


2018-08-29 21:24 GMT+02:00 Fekete Tamás <fektom@gmail.com>:

> Dear all,
>
> the explanation of Reco seems quite good for me.
> If you use load generator on your mobile device it could be the reason.
>
> What happens if you use a normal download for a large file? Do you get the
> same result?
>
> If yes, I advise to look at your hardware configuration like flow control.
> This line doesn't seem good for me (mind the text in bold):
>
> "Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps
> Full Duplex, *Flow Control: None"*
>
> See this link:
> https://access.redhat.com/documentation/en-us/red_hat_
> enterprise_linux/5/html/tuning_and_optimizing_red_hat_
> enterprise_linux_for_oracle_9i_and_10g_databases/sect-
> oracle_9i_and_10g_tuning_guide-adjusting_network_
> settings-flow_control_for_e1000_network_interface_cards
>
> Let's see what happens if you reconfigure your network device.
>
> - Tamas Fekete
>
> 2018-08-29 20:58 GMT+02:00 Reco <recoverym4n@gmail.com>:
>
>>         Hi.
>>
>> Please do not top post. This is a mailing list, not a corporate e-mail
>> spamfest.
>>
>> On Wed, Aug 29, 2018 at 01:18:52PM -0500, Nicholas Geovanis wrote:
>> > Hi, I've never used OVH. How certain are you that the e1000 network
>> driver
>> > is the correct one?
>> > Under VMWare/ESX the network driver choice can be crucial, for example.
>>
>> 1) You cannot run Xen in VSphere/ESXi.
>>
>> 2) No sane public provider will use VSphere/ESXi for hosting, the costs
>> can dim a budget of a small country.
>>
>> 3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday
>> reality. SR-IOV is a way to go.
>>
>>
>> Now, to the issue at hand.
>>
>> > On Wed, Aug 29, 2018 at 6:30 AM Kevin DAGNEAUX <
>> kevin.dagneaux@fiitelcom.fr>
>> > wrote:
>>
>> Please note that you seem to have "link down" first:
>>
>> > > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down
>>
>> and 'malfunctioning' netfilter rules next.
>>
>> > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
>> > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
>> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
>> > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
>>
>> Observe 'RST' flag at each and every 'DROPPED' message.
>> I find it highly unlikely that whatever PHP load test you did it
>> involved sending mass amounts of TCP Reset packets.
>>
>> It seems that the following scenario has much higher probability:
>>
>> 1) You fired your load generator application.
>>
>> 2) Your hosting provided immediately got a signal of your typical DOS
>> attack (not to be confused with DDOS) coming from that seems to be a
>> typical mobile phone.
>>
>> 3) DDOS protection kicked in:
>>
>> a) Isolating your server from the network to stop DOS.
>>
>> b) Sending forged TCP RST to your server to break existing connections
>> *and* termninate unneeded Apache workers (or whatever you have there).
>>
>> c) Banning the initiator of DOS (i.e. you on mobile network) temporarily.
>>
>> 4) Real network outage of your server was 2 seconds (time between "link
>> down" and "link up").
>>
>> Reco
>>
>> >
>> >
>> > > Hi,
>> > >
>> > > I've a server in OVH datacenter, on this server i've 7 VMs, on 1 of
>> them
>> > > in run Apache.
>> > > To debug a slow upload (who was ~2Mo/s instead 12Mo/s) i've installed
>> an
>> > > HTML5/PHP speed test application.
>> > > When i use this app, i've no problem in general, but, when a make a
>> speed
>> > > test from a source who have more bandwith than the server (the server
>> is
>> > > limited at 100Mb/s by OVH and i make the test from a 4G+ network
>> where i've
>> > > ~150Mb/s of bandwith), in this case, the DOM0 lost his network
>> connection
>> > > (like the ethernet cable is unplugged) until i reboot the server.
>> > >
>> > > When i check the syslog of DOM0, i see that iptables drop incomming
>> packet
>> > > on port 80 instead of routing them to the VM.
>> > >
>> > > This is my iptables script i use on DOM0 :
>> > >
>> > > #!/bin/bash
>> > >
>> > > IPT="/sbin/iptables"
>> > >
>> > >
>> > > ############################################################
>> ###############################
>> > > # Filter
>> > >
>> > > ## Remise par defaut des regles
>> > > $IPT -t filter -P INPUT   ACCEPT
>> > > $IPT -t filter -P FORWARD ACCEPT
>> > > $IPT -t filter -P OUTPUT  ACCEPT
>> > >
>> > > ## On purge les tables
>> > > $IPT -t filter -F
>> > >
>> > > ## On autorise lo
>> > > $IPT -t filter -A INPUT -i lo -j ACCEPT
>> > >
>> > > ## On ouvre les ports nécéssaires au DOM0
>> > > $IPT -t filter -A INPUT -m tcp -p tcp --dport 22      -j
>> > > ACCEPT                                         ## SSH
>> > > $IPT -t filter -A INPUT -m udp -p udp --dport 53      -j
>> > > ACCEPT                                         ## DNS
>> > > $IPT -t filter -A INPUT -m icmp -p icmp --icmp-type 8 -j
>> > > ACCEPT                                         ## Ping
>> > > $IPT -t filter -A INPUT -s 10.0.0.0/24 -j ACCEPT
>> > >
>> > > ## On accepte si la connexion est déjà établie
>> > > $IPT -t filter -A INPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
>> > >
>> > > ## On log ce qui n'a pas été matché par les règles précédente
>> > > $IPT -A INPUT -p tcp -j LOG --log-prefix "DROPED packets "
>> > >
>> > > ## On bloque tout le reste
>> > > $IPT -t filter -P INPUT DROP
>> > >
>> > >
>> > > ############################################################
>> ################################
>> > > # Nat
>> > >
>> > > ## Remise par defaut des regles
>> > > $IPT -t nat -P PREROUTING  ACCEPT
>> > > $IPT -t nat -P POSTROUTING ACCEPT
>> > > $IPT -t nat -P INPUT       ACCEPT
>> > > $IPT -t nat -P OUTPUT      ACCEPT
>> > >
>> > > ## On purge
>> > > $IPT -t nat -F
>> > >
>> > > ### Routage des ports entrants pour la VM "mails"
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22030 -j DNAT
>> --to
>> > > 10.0.0.30:22                       ## SSH
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 25    -j DNAT
>> --to
>> > > 10.0.0.30:25                       ## SMTP
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 587   -j DNAT
>> --to
>> > > 10.0.0.30:587                      ## SMTP SUBMISSION
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 465   -j DNAT
>> --to
>> > > 10.0.0.30:465                      ## SMTP SSL
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 143   -j DNAT
>> --to
>> > > 10.0.0.30:143                      ## IMAP
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 993   -j DNAT
>> --to
>> > > 10.0.0.30:993                      ## IMAP SSL
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 9930  -j DNAT
>> --to
>> > > 10.0.0.30:9930                     ## IMAP SSL
>> > >
>> > > ### Routage des ports entrants pour la VM "sql"
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22020 -j DNAT
>> --to
>> > > 10.0.0.20:22                       ## SSH
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 3306  -j DNAT
>> --to
>> > > 10.0.0.20:3306                     ## MariaDB
>> > >
>> > > ### Routage des ports entrants pour la VM "files"
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22010 -j DNAT
>> --to
>> > > 10.0.0.10:22                       ## SSH
>> > >
>> > > ### Routage des ports entrant pour la VM "web"
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22040 -j DNAT
>> --to
>> > > 10.0.0.40:22                       ## SSH
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 80    -j DNAT
>> --to
>> > > 10.0.0.40:80                       ## HTTP
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 443   -j DNAT
>> --to
>> > > 10.0.0.40:443                      ## HTTPS
>> > >
>> > > ### Routage des ports entrants pour la VM "monitor"
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22050 -j DNAT
>> --to
>> > > 10.0.0.50:22                       ## SSH
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 850 -j DNAT
>> --to
>> > > 10.0.0.50:80                 ## HTTP
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 855 -j DNAT
>> --to
>> > > 10.0.0.50:443                        ## HTTPS
>> > >
>> > > ### Routage des ports entrants pour la VM "comm"
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22060 -j DNAT
>> --to
>> > > 10.0.0.60:22                       ## SSH
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5222  -j DNAT
>> --to
>> > > 10.0.0.60:5222                     ## Jabber
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5269  -j DNAT
>> --to
>> > > 10.0.0.60:5269                     ## Jabber
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5280  -j DNAT
>> --to
>> > > 10.0.0.60:5280                     ## Jabber
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5281  -j DNAT
>> --to
>> > > 10.0.0.60:5281                     ## Jabber
>> > >
>> > > ### Routage des ports entrants pour la VM "secure"
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22070 -j DNAT
>> --to
>> > > 10.0.0.70:22                       ## SSH
>> > >
>> > > ### Routage des ports entrants pour la VM "net"
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22080 -j DNAT
>> --to
>> > > 10.0.0.80:22                       ## SSH
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 8388  -j DNAT
>> --to
>> > > 10.0.0.80:8388                     ## shadowsocks
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p udp --dport 8388  -j DNAT
>> --to
>> > > 10.0.0.80:8388                     ## shadowsocks
>> > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 11094 -j DNAT
>> --to
>> > > 10.0.0.80:1194                     ## OpenVPN
>> > >
>> > > ### Autorise les VMs a accéder a internet
>> > > $IPT -t nat -A POSTROUTING -s 10.0.0.0/24 -j  MASQUERADE
>> > >
>> > > And this is what i have in syslogs :
>> > >
>> > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
>> > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
>> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2226 DF PROTO=TCP
>> > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
>> > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
>> > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
>> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2227 DF PROTO=TCP
>> > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
>> > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
>> > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
>> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2228 DF PROTO=TCP
>> > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
>> > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
>> > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
>> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
>> > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
>> > >
>> > > *** I've several hundred of similare line where only the ID change,
>> then :
>> > > ***
>> > >
>> > > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down
>> > > Aug 28 15:50:32 ovh-1 systemd-networkd[20998]: enp1s0: Lost carrier
>> > > Aug 28 15:50:34 ovh-1 systemd-networkd[20998]: enp1s0: Gained carrier
>> > > Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps
>> Full
>> > > Duplex, Flow Control: None
>> > > Aug 28 15:50:34 ovh-1 kernel: e1000e 0000:01:00.0 enp1s0: 10/100
>> speed:
>> > > disabling TSO
>> > >
>> > > The log show that the network is UP but the server is offline.
>> > > In the VM's logs i don't have abnormal things.
>> > >
>> > > OVH have changed the motherboard and the ethernet cable but the
>> problem
>> > > still persist, the syslog show that it's probably a software problem.
>> > >
>> > > Do you have idea of what happen or a way to help me to debug this
>> problem?
>> > >
>> > > Thank you in advance.
>> > >
>> > > Kevin
>> > > --
>> > >
>> > > DAGNEAUX Kevin
>> > > Service informatique
>> > > 03 29 36 88 85
>> > > kevin.dagneaux@fiitelcom.fr
>> > >
>>
>>
>>
>>
>

[toc] | [prev] | [next] | [standalone]

#199620

FromLee <ler762@gmail.com>
Date2018-08-30 00:40 +0200
Message-ID<wshi1-4Za-1@gated-at.bofh.it>
In reply to#199614
> ... I advise to look at your hardware configuration like flow control.
> This line doesn't seem good for me (mind the text in bold):
>
> "Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps Full
> Duplex, *Flow Control: None"*

In general, enabling flow control is a bad idea - more so if qos is enabled.

I saved this snippet because it's a good explanation of why not to
enable flow control (& the snark about the former wife still makes me
grin :)

>From seifert@netcom.com Tue Mar 23 12:58:14 1999
Newsgroups: comp.dcom.lans.ethernet
Subject: Re: 802.3x Flow Control -- Who has it?

802.3x reminds me of my former marriage--at the time, they both seemed like
good ideas. However, in retrospect, neither made a lot of sense. =8^)

Here's an excerpt on the subject (of .3x, not my former marriage), from my
upcoming book, "Make the Switch! A Comprehensive Guide to LAN Switching
Technology":

--------begin excerpt--------
It is important to remember that the PAUSE function was designed for a very
specific use at a very specific time‹to prevent buffer overflow for
memory-constrained switches with an input-queued architecture. At the time
the protocol was devised (1995-96), many low-cost switches used an
input-queued approach, and switch memory was relatively expensive. Reducing
the cost of a switch usually meant reducing its memory capacity; by using
PAUSE-style flow control, such a switch could provide a good
price/performance mix and yet avoid frame loss under peak overload
conditions. The PAUSE protocol is less useful today with lower-cost memory
and the popularity of output-queued switches.

In addition, TCP uses frame loss within the network as its indication of
congestion. That is, TCP¹s end-to-end flow control mechanism detects
underlying frame loss, and uses this information to throttle its flow. If
switches prevent frame loss under congestion conditions, TCP will never
recognize the congestion and will continue sourcing data into the network;
in fact, TCP will see a lack of frame loss over time as an indication that
it can increase the offered load, further exacerbating the congestion
problem. If the switches dropped an occasional frame, TCP would throttle
back the offered load and alleviate much of the network congestion with no
need for PAUSE. This is the concept behind the Random Early Discard (RED)
approach to flow control. Implementing link-layer flow control on switches
can actually interfere with the end-to-end flow control.
  ---------

Regards,
Lee

[toc] | [prev] | [next] | [standalone]

#199618

FromNicholas Geovanis <nickgeovanis@gmail.com>
Date2018-08-29 22:40 +0200
Message-ID<wsfpT-3T8-15@gated-at.bofh.it>
In reply to#199610
On Wed, Aug 29, 2018 at 2:00 PM Reco <recoverym4n@gmail.com> wrote:
> On Wed, Aug 29, 2018 at 01:18:52PM -0500, Nicholas Geovanis wrote:
> > Hi, I've never used OVH. How certain are you that the e1000 network driver
> > is the correct one?
> > Under VMWare/ESX the network driver choice can be crucial, for example.
>
> 1) You cannot run Xen in VSphere/ESXi.

I didn't suggest that one could. Only that network driver choice is
crucial to network
performance in a virtual setting. Hence I wrote "For example".

> 2) No sane public provider will use VSphere/ESXi for hosting, the costs
> can dim a budget of a small country.

See above. I'm a customer, I'm familiar with the costs.

> 3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday
> reality. SR-IOV is a way to go.

You simply can't get good high-level throughput using e1000 under ESX.
But that wasn't the
point. Instead it was that driver selection and config can be crucial
in a virtual env.

[toc] | [prev] | [next] | [standalone]

#199625

FromReco <recoverym4n@gmail.com>
Date2018-08-30 06:30 +0200
Message-ID<wsmKJ-8kD-1@gated-at.bofh.it>
In reply to#199618
	Hi.

On Wed, Aug 29, 2018 at 03:37:52PM -0500, Nicholas Geovanis wrote:
> On Wed, Aug 29, 2018 at 2:00 PM Reco <recoverym4n@gmail.com> wrote:
> > On Wed, Aug 29, 2018 at 01:18:52PM -0500, Nicholas Geovanis wrote:
> > > Hi, I've never used OVH. How certain are you that the e1000 network driver
> > > is the correct one?
> > > Under VMWare/ESX the network driver choice can be crucial, for example.
> >
> > 1) You cannot run Xen in VSphere/ESXi.
> 
> I didn't suggest that one could. Only that network driver choice is
> crucial to network performance in a virtual setting. Hence I wrote
> "For example".

And OP is running Xen. So this means real hardware.

Reco

[toc] | [prev] | [next] | [standalone]

#199621

FromMichael Stone <mstone@debian.org>
Date2018-08-30 00:50 +0200
Message-ID<wshrH-52h-1@gated-at.bofh.it>
In reply to#199610
On Wed, Aug 29, 2018 at 09:58:47PM +0300, Reco wrote:
>3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday
>reality. SR-IOV is a way to go.

everyday? really? when did you last try this?

[toc] | [prev] | [next] | [standalone]

#199626

FromReco <recoverym4n@gmail.com>
Date2018-08-30 06:40 +0200
Message-ID<wsmUp-8ob-5@gated-at.bofh.it>
In reply to#199621
	Hi.

On Wed, Aug 29, 2018 at 06:43:00PM -0400, Michael Stone wrote:
> On Wed, Aug 29, 2018 at 09:58:47PM +0300, Reco wrote:
> > 3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday
> > reality. SR-IOV is a way to go.
> 
> everyday? really? when did you last try this?

Saw the problem with my own eyes yesterday.

Relatively low amount of free RAM (like 64M out of 16G) and a oom killer
stack trace containing tcp_sendmsg + some vmxnet3 NICs.

Those RHEL kernels are outright broken then it comes to memory
management.

Reco

[toc] | [prev] | [next] | [standalone]

#199630

FromMichael Stone <mstone@debian.org>
Date2018-08-30 12:40 +0200
Message-ID<wsswO-3hV-5@gated-at.bofh.it>
In reply to#199626
On Thu, Aug 30, 2018 at 07:32:24AM +0300, Reco wrote:
>On Wed, Aug 29, 2018 at 06:43:00PM -0400, Michael Stone wrote:
>> On Wed, Aug 29, 2018 at 09:58:47PM +0300, Reco wrote:
>> > 3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday
>> > reality. SR-IOV is a way to go.
>>
>> everyday? really? when did you last try this?
>
>Saw the problem with my own eyes yesterday.
>
>Relatively low amount of free RAM (like 64M out of 16G) and a oom killer
>stack trace containing tcp_sendmsg + some vmxnet3 NICs.
>
>Those RHEL kernels are outright broken then it comes to memory
>management.

Well, this is a debian list...

[toc] | [prev] | [next] | [standalone]

#199631

FromReco <recoverym4n@gmail.com>
Date2018-08-30 13:00 +0200
Message-ID<wssQ9-3oj-5@gated-at.bofh.it>
In reply to#199630
	Hi.

On Thu, Aug 30, 2018 at 06:38:50AM -0400, Michael Stone wrote:
> On Thu, Aug 30, 2018 at 07:32:24AM +0300, Reco wrote:
> > On Wed, Aug 29, 2018 at 06:43:00PM -0400, Michael Stone wrote:
> > > On Wed, Aug 29, 2018 at 09:58:47PM +0300, Reco wrote:
> > > > 3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday
> > > > reality. SR-IOV is a way to go.
> > > 
> > > everyday? really? when did you last try this?
> > 
> > Saw the problem with my own eyes yesterday.
> > 
> > Relatively low amount of free RAM (like 64M out of 16G) and a oom killer
> > stack trace containing tcp_sendmsg + some vmxnet3 NICs.
> > 
> > Those RHEL kernels are outright broken then it comes to memory
> > management.
> 
> Well, this is a debian list...

So? The kernel is a kernel, even if it's a RedHat fork.

Besides, I did not bring VMWare into this thread. In fact, VMWare has
nothing to do with OPs problem.

Reco

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web