Linux + OpenVPN; bridge interface not responding to ARP

By Posted on

Linux + OpenVPN; bridge interface not responding to ARP – A server stack is the collection of software that forms the operational infrastructure on a given machine. In a computing context, a stack is an ordered pile. A server stack is one type of solution stack — an ordered selection of software that makes it possible to complete a particular task. Like in this post about Linux + OpenVPN; bridge interface not responding to ARP was one problem in server stack that need for a solution. Below are some tips in manage your linux server when you find problem about linux, openvpn, bridge, tap, .

I am trying to set up an OpenVPN connection with a Layer2 ethernet bridge to bridge two LANs. ARP responses do not seem to being generated correctly…

Please note I want to create a full Layer2 bridge, so that all traffic, including broadcast packets, can be relayed. I will use ebtables and other methods to block DHCP, etc.

Netork Topology

We have two LANs (we’ll call them LAN1 and LAN2) at different physical locations. Each LAN is a standard residential network on a consumer-grade internet connection.

------------     ---------     ----------    ------------------------
* Internet * --> * Modem * --> * Router * -> * Switches and Clients *
------------     ---------     ----------    ------------------------

On each network, we’ve built a Linux server, designed to act as a backup fileserver and VPN bridge. The servers, called Thing1 (on LAN1) and Thing2 (on LAN2) are client devices.

LAN1 uses 192.168.110.0/24

  • Router1 (ASUS RT-AC68U) as DHCP server and internet gateway at 192.168.110.254
  • Various switches and client PCs, phones, TVs, etc.
  • Thing1 (OpenVPN server) connected at 192.168.110.250

LAN2 uses 192.168.111.0/24

  • Router2 (ASUS RT-AC66U) as DHCP server and internet gateway at 192.168.111.254
  • Various switches and client PCs, phones, TVs, etc.
  • Thing2 (OpenVPN server) connected at 192.168.111.250

Configuration

Each of the OpenVPN machines (Thing1 and Thing2) has OpenVPN installed and running. The connection has been created successfully.

On each server, we create our tap and bridge using a bridge-start script. I’ve trimmed it here…

bridge-start

echo 1 > /proc/sys/net/ipv4/ip_forward
br="br0"
tap="tap0"
eth="enp2s0"

#obtain the Hardware Mac address of the physical ethernet interface
eth_hw_mac=`ifconfig $eth | grep 'HWaddr' | cut -d' ' -f9`

for t in $tap; do
    ip tuntap add dev $t mode tap
done

brctl addbr $br
brctl addif $br $eth

for t in $tap; do
    brctl addif $br $t
done

for t in $tap; do
    ifconfig $t 0.0.0.0 promisc up
done

ifconfig $eth 0.0.0.0 promisc up
ifconfig $br hw ether $eth_hw_mac

dhclient $br

# Load ebtables rules to block DHCP traffic across the bridge.
ebtables -A INPUT -i tap0 -p ipv4 --ip-proto udp --ip-dport 67:68 -j DROP
ebtables -A INPUT -i tap0 -p ipv4 --ip-proto udp --ip-sport 67:68 -j DROP
ebtables -A FORWARD -o tap0 -p ipv4 --ip-proto udp --ip-dport 67:68 -j DROP
ebtables -A FORWARD -o tap0 -p ipv4 --ip-proto udp --ip-sport 67:68 -j DROP

The only difference is that on Thing2, the tap interface is named tap1, rather than tap0.

Relevant ifconfig on Thing1 (192.168.110.250)

br0       Link encap:Ethernet  HWaddr b8:ae:ed:fc:4a:4f
          inet addr:192.168.110.250  Bcast:192.168.110.255  Mask:255.255.255.0
          inet6 addr: fe80::baae:edff:fefc:4a4f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:560719 errors:0 dropped:0 overruns:0 frame:0
          TX packets:271873 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:120753983 (120.7 MB)  TX bytes:72273308 (72.2 MB)

enp2s0    Link encap:Ethernet  HWaddr b8:ae:ed:fc:4a:4f
          inet6 addr: fe80::baae:edff:fefc:4a4f/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:525244 errors:0 dropped:68 overruns:0 frame:0
          TX packets:548223 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:142488031 (142.4 MB)  TX bytes:129824161 (129.8 MB)

tap0      Link encap:Ethernet  HWaddr c6:5d:10:17:5c:b9
          inet6 addr: fe80::c45d:10ff:fe17:5cb9/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:240521 errors:0 dropped:0 overruns:0 frame:0
          TX packets:238686 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:49398722 (49.3 MB)  TX bytes:47097224 (47.0 MB)

Relevant ifconfig on Thing2 (192.168.111.250)

br0       Link encap:Ethernet  HWaddr f4:4d:30:08:f1:e5
          inet addr:192.168.111.250  Bcast:192.168.111.255  Mask:255.255.255.0
          inet6 addr: fe80::f64d:30ff:fe08:f1e5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1049729 errors:0 dropped:0 overruns:0 frame:0
          TX packets:757240 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5142387727 (5.1 GB)  TX bytes:302576425 (302.5 MB)

enp2s0    Link encap:Ethernet  HWaddr f4:4d:30:08:f1:e5
          inet6 addr: fe80::f64d:30ff:fe08:f1e5/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:7254547 errors:0 dropped:389 overruns:0 frame:0
          TX packets:3661240 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6092825279 (6.0 GB)  TX bytes:1214754910 (1.2 GB)

tap1      Link encap:Ethernet  HWaddr 46:6f:ee:61:de:b2
          inet addr:192.168.110.1  Bcast:192.168.110.255  Mask:255.255.255.0
          inet6 addr: fe80::446f:eeff:fe61:deb2/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:233278 errors:0 dropped:0 overruns:0 frame:0
          TX packets:294344 errors:0 dropped:46711 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:46740438 (46.7 MB)  TX bytes:129353655 (129.3 MB)

Relevant lines from OpenVPN server.conf on Thing1 (192.168.110.250)

port 1194
proto udp
dev tap0
topology subnet
ifconfig-pool-persist ipp.txt
server-bridge 192.168.110.250 255.255.255.0 192.168.110.1 192.168.110.10
client-to-client
keepalive 10 120
persist-key
persist-tun

Route table on Thing1

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         router.asus.com 0.0.0.0         UG    0      0        0 br0
192.168.110.0   *               255.255.255.0   U     0      0        0 br0
192.168.111.0   192.168.111.250 255.255.255.0   UG    0      0        0 tap0
192.168.111.250 *               255.255.255.255 UH    0      0        0 tap0

Route table on Thing2

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         router.asus.com 0.0.0.0         UG    0      0        0 br0
192.168.110.0   192.168.110.250 255.255.255.0   UG    0      0        0 tap1
192.168.110.250 *               255.255.255.255 UH    0      0        0 tap1
192.168.111.0   *               255.255.255.0   U     0      0        0 br0

Problem

I attempt to ping .111.250 from .110.250, but the ARP resolution never happens…

ping 192.168.111.250

PING 192.168.111.250 (192.168.111.250) 56(84) bytes of data.
From 192.168.110.250 icmp_seq=1 Destination Host Unreachable
From 192.168.110.250 icmp_seq=2 Destination Host Unreachable
From 192.168.110.250 icmp_seq=3 Destination Host Unreachable

Doing a tcpdump on .110.250’s tap while the ping is happening…

tcpdump -i tap0 -en | grep "ARP"

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:43:08.283935 c6:5d:10:17:5c:b9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.250 tell 192.168.110.250, length 28
15:43:08.300129 46:6f:ee:61:de:b2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.110.250 tell 192.168.110.1, length 28
15:43:08.300181 b8:ae:ed:fc:4a:4f > 46:6f:ee:61:de:b2, ethertype ARP (0x0806), length 42: Reply 192.168.110.250 is-at b8:ae:ed:fc:4a:4f, length 28
15:43:09.300441 46:6f:ee:61:de:b2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.110.250 tell 192.168.110.1, length 28
15:43:09.300493 b8:ae:ed:fc:4a:4f > 46:6f:ee:61:de:b2, ethertype ARP (0x0806), length 42: Reply 192.168.110.250 is-at b8:ae:ed:fc:4a:4f, length 28
15:43:10.302120 46:6f:ee:61:de:b2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.110.250 tell 192.168.110.1, length 28
15:43:10.302170 b8:ae:ed:fc:4a:4f > 46:6f:ee:61:de:b2, ethertype ARP (0x0806), length 42: Reply 192.168.110.250 is-at b8:ae:ed:fc:4a:4f, length 28
15:43:11.300760 46:6f:ee:61:de:b2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.110.250 tell 192.168.110.1, length 28
15:43:11.300810 b8:ae:ed:fc:4a:4f > 46:6f:ee:61:de:b2, ethertype ARP (0x0806), length 42: Reply 192.168.110.250 is-at b8:ae:ed:fc:4a:4f, length 28
15:43:12.300762 46:6f:ee:61:de:b2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.110.250 tell 192.168.110.1, length 28
15:43:12.300815 b8:ae:ed:fc:4a:4f > 46:6f:ee:61:de:b2, ethertype ARP (0x0806), length 42: Reply 192.168.110.250 is-at b8:ae:ed:fc:4a:4f, length 28

This sequence repeats for each ping attempt.

Meanwhile, on .111.250’s tap1 (which has IP 192.168.110.1)…

tcpdump -i tap1 -en | grep "ARP"

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap1, link-type EN10MB (Ethernet), capture size 262144 bytes
15:49:47.315913 c6:5d:10:17:5c:b9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.250 tell 192.168.110.250, length 28
15:49:47.813360 46:6f:ee:61:de:b2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.110.250 tell 192.168.110.1, length 28
15:49:47.830572 b8:ae:ed:fc:4a:4f > 46:6f:ee:61:de:b2, ethertype ARP (0x0806), length 42: Reply 192.168.110.250 is-at b8:ae:ed:fc:4a:4f, length 28
15:49:48.314726 c6:5d:10:17:5c:b9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.250 tell 192.168.110.250, length 28
15:49:48.812622 46:6f:ee:61:de:b2 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.110.250 tell 192.168.110.1, length 28
15:49:48.832273 b8:ae:ed:fc:4a:4f > 46:6f:ee:61:de:b2, ethertype ARP (0x0806), length 42: Reply 192.168.110.250 is-at b8:ae:ed:fc:4a:4f, length 28

If we look at .111.250’s br0 (which actually has 192.168.111.250)…

tcpdump -i br0 -en | grep "ARP"

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:53:49.969025 c6:5d:10:17:5c:b9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.250 tell 192.168.110.250, length 28
15:53:50.341753 b8:ae:ed:fc:4a:4f > 46:6f:ee:61:de:b2, ethertype ARP (0x0806), length 42: Reply 192.168.110.250 is-at b8:ae:ed:fc:4a:4f, length 28
15:53:50.968877 c6:5d:10:17:5c:b9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.250 tell 192.168.110.250, length 28
15:53:51.341336 b8:ae:ed:fc:4a:4f > 46:6f:ee:61:de:b2, ethertype ARP (0x0806), length 42: Reply 192.168.110.250 is-at b8:ae:ed:fc:4a:4f, length 28

But at the physical ethernet adapter…

tcpdump -i enp2s0 -en | grep "ARP"

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:56:13.344685 c6:5d:10:17:5c:b9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.250 tell 192.168.110.250, length 28
15:56:14.344167 c6:5d:10:17:5c:b9 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.250 tell 192.168.110.250, length 28

As far as I understand, Thing2’s br0, which has address 192.168.111.250 should be sending a reply something like…

f4:4d:30:08:f1:e5 > c6:5d:10:17:5c:b9, ethertype ARP (0x0806), length 42: Reply 192.168.111.250 is-at f4:4d:30:08:f1:e5, length 28

I don’t know that this will fix everything that’s broken, or if I’m missing something very obvious. Feel free to make any suggestions which you think would move me in the right direction.

The problem is that you are trying to bridge two different IP subnets, which does not work. One IP subnet consists of a single broadcast domain, and now you are trying to set up two different IP subnets in a single broadcast domain (bridge over the VPN).

What happens during the ping is that your 192.168.111.250 sends ARP request to 192.168.110.250, which will not answer it because the request comes from an IP address outside of its subnet 192.168.110.0/24.

Another issue in your configuration is that you have defined an IP address both to your br0 device, which is the bridge device, and the tapX device, which is part of the bridge.

You should only assign an IP address to the bridge device.

You need to redesign your networking concept so that you use a single IP subnet over your bridge if you really want a full L2 networking solution.

Related posts:

Leave a Reply

Your email address will not be published. Required fields are marked *