KVM: Need multiple Virtual Machines to share a VLAN’ed interface – A server stack is the collection of software that forms the operational infrastructure on a given machine. In a computing context, a stack is an ordered pile. A server stack is one type of solution stack — an ordered selection of software that makes it possible to complete a particular task. Like in this post about KVM: Need multiple Virtual Machines to share a VLAN’ed interface was one problem in server stack that need for a solution. Below are some tips in manage your linux server when you find problem about linux, centos, virtual-machines, vlan, qemu.
I’m not sure how to configure my Linux server which has a single VLAN’ed interface to support a number of Virtual machines which I want to bridge to my network (so that all their services appear to come from a number of real machines on my network). Do I simply create a single bridged network interface
br0, tied to my existing VLAN’ed device, and have all the VMs use that interface? Or do I need to set up
br2, etc. for each VM? Can I even tie those bridge interfaces to a VLAN’ed interface? I suppose I need to say
VLAN="yes" in the bridge’s ifcfg file?
True, I could simply try it and see, but at this point I am so unsure about how it all comes together that I’d like to perform a sanity check before I get a sanity wreck 🙂 .
I have set up a CentOS7 host:
# uname -a Linux cha028 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core)
It has a single interface which has a VLAN. So hardware interface em1 has:
# cat ifcfg-em1 NAME="em1" DEVICE="em1" ONBOOT="yes" TYPE="Ethernet" BOOTPROTO="none" HWADDR=14:fe:b5:d6:07:cc NM_CONTROLLED=no
…and I have a ifcfg-em1.144 that has all the IP addressing in it and such. I have installed a plurality of VMs on the box. Now I need to make the network work.
How do I do it? Simply create ifcfg-br0 as per https://www.banym.de/linux/centos/setup-bridge-device-on-centos (with its own unique IP address), and simply add a line to my ifcfg-em1.144 file which says
And, having done so, can I attach all my VMs to the br0 device?
Thank you for your indulgence, and your help.
Here is my
ifcfg-em1.144 file. Note that
/etc/sysconfig/network contains the GATEWAY line:
VLAN="yes" DEVICE="em1.144" PHYSDEV="em1" TYPE="Ethernet" BOOTPROTO="static" DEFROUTE="yes" ONBOOT="yes" IPADDR="10.144.101.28" PREFIX="24" NM_CONTROLLED=no
If you bridge to the vlan interface (as opposed to the hardware interface), the bridged packets from your VMs will be tagged as they are bridged out onto the wire. If you bridge to the hardware interface, all traffic will be bridged regardless of tag and you will have to move your vlan interface to be off the bridge (base interface vlan is now meaningless). Any number of VMs can share a bridge.
Typically, you would handle the vlans at the host so the guests don’t have to manually configure vlans. A number of different permutations of this sounds like they would work for you.
Here is what I did to get the bridged network set up on my Host machine over a VLAN:
Physical Device – the thing that is real. The only thing it knows is that it’s an Ethernet device and it has a MAC address. Oh, and get that damn Network Manager out of its face:
(file == /etc/sysconfig/network-scripts/ifcfg-em1)
DEVICE="em1" TYPE=Ethernet HWADDR=14:fe:b5:d6:07:cc BOOTPROTO=none ONBOOT="yes" NM_CONTROLLED=no
VLAN device – My vlan here is numbered 144. YMMV. You tell CentOS/Redhat’s network this simply by suffixing the device name with a dot followed by a number; the
ifupscript looks for that and extracts the VLAN ID from the string. And you tell it it is a VLAN using the
(file == /etc/sysconfig/network-scripts/ifcfg-em1.144)
DEVICE=em1.144 TYPE=Ethernet BOOTPROTO=none ONBOOT=yes VLAN=yes BRIDGE=br0 NM_CONTROLLED=no
Bridged device – the Bridge is the Keeper of the IP ™. This holds the layer 3 (IP address) information… that is, DNS, netmask, etc. For me, I like to put the default gateway in /etc/sysconfig/network. And I like to keep that ZEROCONF (169.254.x.y) IP addressing out of my routing table. So we have:
(file == /etc/sysconfig/network-scripts/br0)
DEVICE="br0" TYPE="Bridge" BOOTPROTO="static" DEFROUTE="yes" ONBOOT="yes" IPADDR="10.144.1.12" NETMASK=255.255.255.0 DNS1="10.144.1.101" DOMAIN="example.com" NM_CONTROLLED=no DELAY=0
(file == /etc/sysconfig/network):
Note that after network is started it may take 30-60 seconds for your switch to actually accept the packets (ie, Cisco may be configured to do this). So if you see “Destination host unreachable” when you ping your gateway, wait a minute. It may just take a while to come up.
Once the host is set up and working (ie, you can ping local gateway, then other hosts on the network, then other hosts in your infrastructure… all by IP mind you… then you can ping/traceroute/telnet-into-port-22 by hostname), then you are ready to create networking on your VMs.
This is trivially easy. You just need to ensure that they are using br0 as their network device. If you use virt-install, like I did, you can do this:
virt-install --name monitor0 --memory 2048 --disk /dev/vm_group0/thin_lv_monitor0 --cdrom ./CentOS-7-x86_64-Everything-1511.iso --network bridge=br0
If you are using qemu and you build the virtual host without networking, or with NAT, you can correct it. Just bring the host down. Then edit
/etc/libvirt/qemu/<vm name>.xml. Look for a section called
interface. Change it to follow this example:
<interface type='bridge'> <mac address='52:54:00:22:29:cc'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
…That is, you will probably need to change the first
interface line and the
source line only (that’s what I did, after having built the machines without specifying networking [it sets up NAT by default]).
Save the file, then restart libvirtd using your favorite method:
systemctl restart libvirtd
Restart the VM using
virsh start <vm name>. Go into your VM using its console, and edit the network as appropriate. Remember, your VM is now a bona fide member of your network so you are going to give it an IP address in the same subnet as the bridged interface
I have set up two VMs sharing the VLAN’ed
br0 so I am a happy camper!