Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.networking > #2592 > unrolled thread

iproute2 fwmark doesn't work (or iptables --set-mark doesn't)

Back to article view | Back to comp.os.linux.networking

Contents

  iproute2 fwmark doesn't work (or iptables --set-mark doesn't) gof@somewhere.invalid (Adam Wysocki) - 2013-12-03 13:45 +0000
    Re: iproute2 fwmark doesn't work (or iptables --set-mark doesn't) Tauno Voipio <tauno.voipio@notused.fi.invalid> - 2013-12-03 16:46 +0200
      Re: iproute2 fwmark doesn't work (or iptables --set-mark doesn't) gof@somewhere.invalid (Adam Wysocki) - 2013-12-03 18:07 +0000
      Re: iproute2 fwmark doesn't work (or iptables --set-mark doesn't) Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> - 2013-12-03 23:58 +0100
    Re: iproute2 fwmark doesn't work (or iptables --set-mark doesn't) Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> - 2013-12-04 00:13 +0100
      Re: iproute2 fwmark doesn't work (or iptables --set-mark doesn't) gof@somewhere.invalid (Adam Wysocki) - 2013-12-07 12:36 +0000
        Re: iproute2 fwmark doesn't work (or iptables --set-mark doesn't) Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> - 2013-12-07 16:14 +0100
          Re: iproute2 fwmark doesn't work (or iptables --set-mark doesn't) Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> - 2013-12-07 16:21 +0100
    Re: iproute2 fwmark doesn't work (or iptables --set-mark doesn't) logsnaath@gmail.com - 2014-08-03 13:06 -0700
      Re: iproute2 fwmark doesn't work (or iptables --set-mark doesn't) Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> - 2014-08-03 23:22 +0200

#2592 — iproute2 fwmark doesn't work (or iptables --set-mark doesn't)

Hi,

I just subscribed this group. Hello everyone!

I have a problem with setting up routing. In short, I want to mark certain 
packets with iptables and then catch these marks with iproute2. However 
Linux seems to ignore these marks.

I have a 3.11.4 kernel with CONFIG_IP_ADVANCED_ROUTER and CONFIG_IP_MULTIPLE_TABLES 
set. Here's my setup:

- eth0: LAN
  - IP 192.168.0.1, netmask /24
  - It connects to the LAN

- eth1: WAN
  - IP 192.168.1.180, netmask /24, gateway 192.168.1.1
  - It connects to the Internet

- tap0: VPN (created with OpenVPN)
  - Local IP 172.24.25.4, remote IP (and gateway) 172.24.25.2, netmask /24

I want to pass certain traffic through 192.168.1.1 (eth1) and certain through 
172.24.25.2 (tap0) - local traffic as well as routed traffic. Rules will be 
somewhat complicated (for locally-generated traffic and routed traffic), now 
just let's assume that I want to mark all packets to go through 192.168.1.1.

I have added two new routing tables to /etc/iproute2/rt_tables:

#v+
2 vpn
3 wan
#v-

I've created two of them, so the main table will not have a default gateway. 
This way I can avoid errors - if I make a mistake and traffic goes through 
the main table, it won't get out.

Tables are then populated:

#v+
ip route flush table main
ip route add 192.168.0.0/24 dev eth0 table main
ip route add 192.168.1.0/24 dev eth1 table main
ip route add 172.24.25.0/24 dev tap0 table main

ip route flush table wan
ip route add 192.168.0.0/24 dev eth0 table wan
ip route add 192.168.1.0/24 dev eth1 table wan
ip route add 172.24.25.0/24 dev tap0 table wan
ip route add default via 192.168.1.1 dev eth1 table wan

ip route flush table vpn
ip route add 192.168.0.0/24 dev eth0 table vpn
ip route add 192.168.1.0/24 dev eth1 table vpn
ip route add 172.24.25.0/24 dev tap0 table vpn
ip route add default via 172.24.25.2 dev tap0 table vpn
#v-

And now rules. I want packets marked with 2 going through WAN and packets 
marked with 3 going through VPN.

#v+
ip rule flush

# restore default rules (deleted by flush)
ip rule add from all lookup main prio 32766
ip rule add from all lookup default prio 32767

ip rule add fwmark 2 table wan
ip rule add fwmark 3 table vpn
#v-

Now the cache is reloaded...

#v+
ip route flush table cache
#v-

And iptables flushed:

#v+
iptables -t filter -F
iptables -t nat -F
iptables -t mangle -F
#v-

And now the tricky part. I can't make iptables mark packets (or can't 
make iproute2 honor fwmark). I've tried different combinations of:

#v+
iptables -t mangle -A PREROUTING -j MARK --set-mark 2
iptables -t mangle -A POSTROUTING -j MARK --set-mark 2
iptables -t mangle -A OUTPUT -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A POSTROUTING -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
iptables -t mangle -A POSTROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -m connmark --mark 2 -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j CONNMARK --set-mark 2
iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
#v-

And nothing. All commands succeeed, but traffic still gets through the 
main table. Regarding to this:

http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.netfilter.html

it should work. What am I missing?

Kernel doesn't have CONFIG_IP_ROUTE_FWMARK (it's completely absent), but 
I've read that in new kernels it is enabled by default and not found in 
config.

Thanks for your time.

Cheers.

AW

[toc] | [next] | [standalone]

#2594

On 3.12.13 15:45, Adam Wysocki wrote:
> Hi,
>
> I just subscribed this group. Hello everyone!
>
> I have a problem with setting up routing. In short, I want to mark certain
> packets with iptables and then catch these marks with iproute2. However
> Linux seems to ignore these marks.
>
> I have a 3.11.4 kernel with CONFIG_IP_ADVANCED_ROUTER and CONFIG_IP_MULTIPLE_TABLES
> set. Here's my setup:
>
> - eth0: LAN
>    - IP 192.168.0.1, netmask /24
>    - It connects to the LAN
>
> - eth1: WAN
>    - IP 192.168.1.180, netmask /24, gateway 192.168.1.1
>    - It connects to the Internet
>
> - tap0: VPN (created with OpenVPN)
>    - Local IP 172.24.25.4, remote IP (and gateway) 172.24.25.2, netmask /24
>
> I want to pass certain traffic through 192.168.1.1 (eth1) and certain through
> 172.24.25.2 (tap0) - local traffic as well as routed traffic. Rules will be
> somewhat complicated (for locally-generated traffic and routed traffic), now
> just let's assume that I want to mark all packets to go through 192.168.1.1.
>
> I have added two new routing tables to /etc/iproute2/rt_tables:
>
> #v+
> 2 vpn
> 3 wan
> #v-
>
> I've created two of them, so the main table will not have a default gateway.
> This way I can avoid errors - if I make a mistake and traffic goes through
> the main table, it won't get out.
>
> Tables are then populated:
>
> #v+
> ip route flush table main
> ip route add 192.168.0.0/24 dev eth0 table main
> ip route add 192.168.1.0/24 dev eth1 table main
> ip route add 172.24.25.0/24 dev tap0 table main
>
> ip route flush table wan
> ip route add 192.168.0.0/24 dev eth0 table wan
> ip route add 192.168.1.0/24 dev eth1 table wan
> ip route add 172.24.25.0/24 dev tap0 table wan
> ip route add default via 192.168.1.1 dev eth1 table wan
>
> ip route flush table vpn
> ip route add 192.168.0.0/24 dev eth0 table vpn
> ip route add 192.168.1.0/24 dev eth1 table vpn
> ip route add 172.24.25.0/24 dev tap0 table vpn
> ip route add default via 172.24.25.2 dev tap0 table vpn
> #v-
>
> And now rules. I want packets marked with 2 going through WAN and packets
> marked with 3 going through VPN.
>
> #v+
> ip rule flush
>
> # restore default rules (deleted by flush)
> ip rule add from all lookup main prio 32766
> ip rule add from all lookup default prio 32767
>
> ip rule add fwmark 2 table wan
> ip rule add fwmark 3 table vpn
> #v-
>
> Now the cache is reloaded...
>
> #v+
> ip route flush table cache
> #v-
>
> And iptables flushed:
>
> #v+
> iptables -t filter -F
> iptables -t nat -F
> iptables -t mangle -F
> #v-
>
> And now the tricky part. I can't make iptables mark packets (or can't
> make iproute2 honor fwmark). I've tried different combinations of:
>
> #v+
> iptables -t mangle -A PREROUTING -j MARK --set-mark 2
> iptables -t mangle -A POSTROUTING -j MARK --set-mark 2
> iptables -t mangle -A OUTPUT -j MARK --set-mark 2
> iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
> iptables -t mangle -A POSTROUTING -j CONNMARK --restore-mark
> iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
> iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A POSTROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A OUTPUT -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 2
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j CONNMARK --set-mark 2
> iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
> #v-
>
> And nothing. All commands succeeed, but traffic still gets through the
> main table. Regarding to this:
>
> http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.netfilter.html
>
> it should work. What am I missing?
>
> Kernel doesn't have CONFIG_IP_ROUTE_FWMARK (it's completely absent), but
> I've read that in new kernels it is enabled by default and not found in
> config.
>
> Thanks for your time.
>
> Cheers.
>
> AW


Iptables and routing apply to layer 3 (network).
Your tunnel interface (tap0) is a layer 2 (data link) device.

If you're tunneling at data link layer, the firewall does
not do anything at it.

I'd start by changing the VPN to use tun0 instead.

-- 

Tauno Voipio

[toc] | [prev] | [next] | [standalone]

#2596

Tauno Voipio <tauno.voipio@notused.fi.invalid> wrote:

> Iptables and routing apply to layer 3 (network).
> Your tunnel interface (tap0) is a layer 2 (data link) device.
> 
> If you're tunneling at data link layer, the firewall does
> not do anything at it.
> 
> I'd start by changing the VPN to use tun0 instead.

Hi,

Thanks for the reply, but I don't think it's the case here. Why:

1. For now I'm dealing only with two tables - "main" and "wan".
I'll use "vpn" table when I'll be able to correctly mark packets
to pass through "wan" table instead of "main" table. For now we
can assume that VPN doesn't exist, it will come into play later.

2. tap0 is on layer 2, but it's only an interface, I want to
redirect layer 3 traffic to this tunnel in the same way it
would be done with other, physical datalink interfaces (e.g.
Ethernet), basing on their assigned IP addresses.

Once again thanks for the reply.

Cheers.

AW

[toc] | [prev] | [next] | [standalone]

#2599

Tauno Voipio a écrit :
> 
> Iptables and routing apply to layer 3 (network).
> Your tunnel interface (tap0) is a layer 2 (data link) device.

So what ? eth0 and eth1 are layer 2 devices too.

> If you're tunneling at data link layer, the firewall does
> not do anything at it.
> 
> I'd start by changing the VPN to use tun0 instead.

Total nonsense. Iptables and ip do not care about the type of interface.
As you wrote, they work at the layer 3.

[toc] | [prev] | [next] | [standalone]

#2600

Adam Wysocki a écrit :
> 
> And now the tricky part. I can't make iptables mark packets (or can't 
> make iproute2 honor fwmark). I've tried different combinations of:

Which combinations exactly ? You cannot have all those rules together,
it just does not make sense.

> #v+
> iptables -t mangle -A PREROUTING -j MARK --set-mark 2

Fine, that should mark all incoming packets.

> iptables -t mangle -A POSTROUTING -j MARK --set-mark 2

Useless : POSTROUTING is after the routing decision, so it will have no
effet.

> iptables -t mangle -A OUTPUT -j MARK --set-mark 2

Fine, that should mark all locally generated packets.
I'd start with just the above two rules in PREROUTING and OUTPUT, append
a couple of LOG rules to print the packet mark in the kernel log and see
what happens.

> iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
> iptables -t mangle -A POSTROUTING -j CONNMARK --restore-mark
> iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
> iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A POSTROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A OUTPUT -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 2
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j CONNMARK --set-mark 2
> iptables -t mangle -A PREROUTING -m connmark --mark 2 -j CONNMARK --restore-mark
> iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark

There are a lot of rules with --restore-mark (copy the connection mark
to the packet mark), but only one rule setting the connection mark.

> And nothing. All commands succeeed,

Did you check the actual results by printing the routing tables, routing
rules and iptables ruleset ?

> but traffic still gets through the main table.

How do you know ?

[toc] | [prev] | [next] | [standalone]

#2626

Pascal Hambourg <boite-a-spam@plouf.fr.eu.org> wrote:

Hi,

Sorry for the late reply, but I finally found some time to fiddle with it...

>> And now the tricky part. I can't make iptables mark packets (or can't 
>> make iproute2 honor fwmark). I've tried different combinations of:
> 
> Which combinations exactly ? You cannot have all those rules together,
> it just does not make sense.

Ok, I left only these rules that you said.

> I'd start with just the above two rules in PREROUTING and OUTPUT, append
> a couple of LOG rules to print the packet mark in the kernel log and see
> what happens.

It seems that marking is done properly, but iproute2 does not direct 
matched packets to proper tables.

iptables -t mangle -A PREROUTING -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -j LOG --log-prefix "prerouting "

iptables -t mangle -A OUTPUT -j MARK --set-mark 2
iptables -t mangle -A OUTPUT -j LOG --log-prefix "output "

It gave log lines with MARK=0x02:

Dec  7 13:27:32 somewhere kernel: [71053.449380] output IN= OUT=eth1 SRC=192.168.1.180 DST=89.72.158.60 LEN=107 TOS=0x00 PREC=0x00 TTL=64 ID=3020 DF PROTO=TCP SPT=43104 DPT=443 WINDOW=28723 RES=0x00 ACK PSH URGP=0 MARK=0x2
Dec  7 13:27:35 somewhere kernel: [71056.520793] prerouting IN=eth1 OUT= MAC=00:50:04:30:e1:90:74:ea:3a:bd:5d:10:08:00 SRC=89.72.158.60 DST=192.168.1.180 LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=59395 DF PROTO=TCP SPT=443 DPT=44304 WINDOW=12600 RES=0x00 ACK URGP=0 MARK=0x2
Dec  7 13:27:38 somewhere kernel: [71059.616705] output IN= OUT=eth0 SRC=192.168.0.1 DST=192.168.0.3 LEN=116 TOS=0x10 PREC=0x00 TTL=64 ID=14433 DF PROTO=TCP SPT=22 DPT=33868 WINDOW=330 RES=0x00 ACK PSH URGP=0 MARK=0x2
Dec  7 13:27:38 somewhere kernel: [71059.617382] prerouting IN=eth0 OUT= MAC=00:0a:5e:60:ce:b0:00:19:7e:19:1e:10:08:00 SRC=192.168.0.3 DST=192.168.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=58487 DF PROTO=TCP SPT=33868 DPT=22 WINDOW=1001 RES=0x00 ACK URGP=0 MARK=0x2

>> And nothing. All commands succeeed,
> 
> Did you check the actual results by printing the routing tables, routing
> rules and iptables ruleset ?

Yes:

[gof@raid ~]$ ip r s
172.24.25.0/24 dev tap0  scope link
192.168.0.0/24 dev eth0  scope link
192.168.1.0/24 dev eth1  scope link

[gof@raid ~]$ ip r s t vpn
default via 172.24.25.2 dev tap0
172.24.25.0/24 dev tap0  scope link
192.168.0.0/24 dev eth0  scope link
192.168.1.0/24 dev eth1  scope link

[gof@raid ~]$ ip r s t wan
default via 192.168.1.1 dev eth1
172.24.25.0/24 dev tap0  scope link
192.168.0.0/24 dev eth0  scope link
192.168.1.0/24 dev eth1  scope link

[gof@raid ~]$ ip ru s
0:      from all lookup local
32764:  from all fwmark 0x3 lookup vpn
32765:  from all fwmark 0x2 lookup wan
32766:  from all lookup main
32767:  from all lookup default

[gof@raid ~]$ sudo iptables -t mangle -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0           MARK set 0x2
LOG        all  --  0.0.0.0/0            0.0.0.0/0           LOG flags 0 level 4 prefix `prerouting '

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
MARK       all  --  0.0.0.0/0            0.0.0.0/0           MARK set 0x2
LOG        all  --  0.0.0.0/0            0.0.0.0/0           LOG flags 0 level 4 prefix `output '

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

>> but traffic still gets through the main table.
> 
> How do you know ?

Main table doesn't have default route, so:

[gof@raid ~]$ ping 1.2.3.4
connect: Network is unreachable

[gof@raid ~]$ telnet 1.2.3.4 1234
Trying 1.2.3.4...
telnet: Unable to connect to remote host: Network is unreachable

AW

[toc] | [prev] | [next] | [standalone]

#2629

Adam Wysocki a écrit :
> 
>>> but traffic still gets through the main table.
>> How do you know ?
> 
> Main table doesn't have default route, so:
> 
> [gof@raid ~]$ ping 1.2.3.4
> connect: Network is unreachable
> 
> [gof@raid ~]$ telnet 1.2.3.4 1234
> Trying 1.2.3.4...
> telnet: Unable to connect to remote host: Network is unreachable

That's the result of the first routing decision which happens while the
packet is being built, before it is passed to iptables' OUTPUT chains
and marked. Thus the main routing table must have a default route
(possibly with a nonexistent gateway so that packets won't actually be
sent out using it) to avoid this error. However routing of incoming
packets should have worked.

[toc] | [prev] | [next] | [standalone]

#2630

Pascal Hambourg a écrit :
> 
> That's the result of the first routing decision which happens while the
> packet is being built, before it is passed to iptables' OUTPUT chains
> and marked.

One more side note about this : you can see in the OUTPUT chain log that
locally generated packets have an initial source address and output
interface as a result of this routing decision. The rerouting after the
OUTPUT chain will update the output interface, but not the source
address. So you may need to update it with SNAT or MASQUERADE.

[toc] | [prev] | [next] | [standalone]

#3053

I am facing similar issue.

Does locally generated packets reach the OUTPUT chain after routing decision?
Should there be any entries in main table always
Any good solution.

=============
 ip route flush cache
 ip route flush table main

 iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
 iptables -t mangle -A OUTPUT -m owner --uid-owner squid -j MARK --set-mark 0x5
 iptables -t mangle -A OUTPUT -j LOG --log-prefix "IPtab: "

 ip route add 192.168.1.0/24 dev wlan0 src 192.168.1.103 table logu
 ip route add default via 192.168.1.1 dev wlan0 table logu
 ip rule add fwmark 0x5 table logu

 ip route flush cache

========================
As squid user

$ ping 74.125.236.39
connect: Network is unreachable

Adding an entry like this in main table works
$ route add -net 0.0.0.0/0 dev wlan0

Regards
Logu

[toc] | [prev] | [next] | [standalone]

#3054

logsnaath@gmail.com a écrit :
> I am facing similar issue.

Similar to what ? That thread is so old that my news server does not
have the messages any more.

> Does locally generated packets reach the OUTPUT chain after routing decision?

Yes. There is always a routing decision before the OUTPUT chains. There
may be a rerouting decision after the OUTPUT chains.

> Should there be any entries in main table always

Not necessarily, but the routing rules and routes must be able to route
the original packets before it is mangled in the OUTPUT chains,
otherwise it is discarded immediately.

Anyway, what would be the purpose of flushing the main routing table ?

[toc] | [prev] | [standalone]

Back to top | Article view | comp.os.linux.networking

csiph-web