Build a network emulator using Libvirt and KVM

In this post, I demonstrate how to create a network emulation scenario using Libvirt, the Qemu/KVM hypervisor, and Linux bridges to create and manage interconnected virtual machines on a host system. As I do so, I will share what I have learned about network virtualization on a Linux system.

Libvirt provides a command-line interface that hides the low-level virtualization and networking details, enabling one to easily create and manage virtual networking scenarios. It is already used as a basis for some existing network emulators, and other applications and tools. It is available in almost every Linux distribution.

The network emulation scenario

As you work through the examples in this post, you will create a very simple network topology which is intended to demonstrate the use of Libvirt and other virtualization tools to build a network emulator and is not intended to emulate a real-world network. However, once you understand its operation, you may use Libvirt to create large, complex network topologies intended to emulate real-world network scenarios.

The example I created for this post consists of three virtual machines serving as routers connected to each other in a ring topology. On each side of this emulated network, you will create virtual machines acting as a user and as a server, so you can test the emulated network’s operations. Each node in the virtual network is also connected to an “out of band” management network so you can configure and manage it. See the network diagram below.

Conventions used in this post

I log into, and run commands on, the host system and on multiple virtual machines while I work through the examples in this post. To make it clear which machine I am currently using, I show the bash prompts in most of the command line listings I show below.

My host system is named T420 so when I am logged into it, the prompt is:

brian@T420:~$ 

When I am logged into the virtual machines, each will show its unique hostname in the bash prompt. For example, if I am logged into router r01, the prompt is:

sim@r01:~$ 

Please carefully note which node you are supposed to be working with for each command or output shown in the examples, below.

Prepare the host system

The host system could be your personal computer, such as a laptop, or it could be a cloud instance that supports nested virtualization. In this example, the host system is running Ubuntu 18.04. To prepare your system to support network emulation using Libvirt, do the following:

• Install virtualization and guest tools software
• Add your userid to the correct groups
• Fix the Linux kernel file permissions
• Create a directory structure in which you will store your disk images and other files
  • And, add the directory to the libvirt group
• Enable Libvirt’s NSS plugin so you can use SSH to connect to Libvirt VMs using their hostnames

Install virtualization software

Verify your computer, or cloud instance, can support accelerated virtualization. Enter the following commands in your computer’s terminal:

brian@T420:~$ grep -cw vmx /proc/cpuinfo

It should return a value equal to the number of CPU threads available on the computer or cloud instance. If it returns 0, something is wrong. In my case, I am using an old Lenovo Thinkpad T420 laptop computer with a dual-core intel i5 processor which supports hyper-threading so I see the value 4 when I run the above command because the processor has two physical CPU cores which each support two “virtual CPU” threads.

Install the virtualization software:

brian@T420:~$ sudo apt-get update
brian@T420:~$ sudo apt install qemu-kvm libvirt-clients \
          libvirt-daemon-system virt-manager bridge-utils \
          libguestfs-tools libnss-libvirt

Add your userid to groups

In addition to the libvirt group, which is configured by the installer, you should also add your username to the other Libvirt groups and to the kvm group.

brian@T420:~$ sudo adduser `id -un` libvirt-qemu
brian@T420:~$ sudo adduser `id -un` kvm
brian@T420:~$ sudo adduser `id -un` libvirt-dnsmasq

Logout and log back in to activate the group ownership changes for your user, or restart your system:

brian@T420:~$ logout

After logging back in, check that the libvirtd systemd service is installed and running:

brian@T420:~$ systemctl status libvirtd

You should see that the libvirtd service is in the active (running) state.

Also verify your userid is part of the libvirt and libvirt-qemu groups using the groups command.

brian@T420:~$ groups
brian adm cdrom sudo dip plugdev lpadmin sambashare kvm libvirt libvirt-dnsmasq libvirt-qemu

Fix the Linux kernel file permissions

Many of the virtualization tools used in this example require read access to the host’s Linux kernel file but, in Ubuntu 18.04, the Canonical developers decided to make the Linux kernel readable only by the root user or by users running sudo. Canonical says they did this to improve security but others strongly disagree with them. The libguestfs tools install guide refutes Canonical on this point ((To see what the libguestfs development team really thinks, search for “completely stupid” in the libguestfs frequently-asked-questions web page)) so I set the Linux kernel file to be readable by all users on my host system, instead of running virtualization tools with root privileges. I suggest you do the same.

brian@T420:~$ sudo chmod 0644 /boot/vmlinuz*

Libvirt configuration files

There are two configuration files that impact how Libvirt functions with KVM. we are concerned that the permissions are set correctly. Usually, all the default settings are OK.

Check the file, /etc/libvirt/libvirtd.conf and verify that the libvirt group is defined with read and write permissions:

brian@T420:~$ nano /etc/libvirt/libvirtd.conf

Scroll down through the file and look for the two lines below. Ensure that the group is set to libvirt and the permissions is set to 0770 or 0777.

unix_sock_group = "libvirt"
unix_sock_rw_perms = "0770"

Check the file /etc/libvirt/qemu.conf and verify it is set to all default values. That is, everything should be commented out.

Create a directory for emulation scenarios

You need a directory to store your disk images, XML files, and scripts. Create a directory in your home folder. I chose to name mine simulator. I place it outside my $HOME directory because the default permissions on $HOME are typically too restrictive to allow access.

brian@T420:~$ sudo mkdir /simulator

Configure the directory’s permissions so that it is owned by your userid and the libvirt group. Set the permissions so that groups may write to the directory and other users may list files in the directory

brian@T420:~$ sudo chown -R brian:libvirt /simulator
brian@T420:~$ chmod g+w /simulator
brian@T420:~$ chmod o+x /simulator

I used Linux Access Control Lists so that disk images created by Libvirt, which are owned by the root user and group, are also owned by the libvirt group and tools like virt-sysprep can write to them. Linux ACLs lets you create a second level of permissions so you do not have to keep changing ownership of Libvirt disk files to the libvirt group.

brian@T420:~$ sudo setfacl -m g:libvirt:rw /simulator
brian@T420:~$ sudo setfacl -dm g:libvirt:rw /simulator

For your information, you can check the ACL setting on a file or directory with the getfacl command. For example:

brian@T420:~$ getfacl /simulator

Create a sub-directory for your network emulation scenario. I called mine sim01. Set the same owners and permissions as the parent directory:

brian@T420:~$ cd /simulator
brian@T420:~$ mkdir sim01

Enable Libvirt’s NSS plugin

When you start a virtual machine (VM) for the first time, the Libvirt default network will assign it an IP network configuration using DHCP. However, you cannot address the virtual machine by its hostname because the DHCP server on a Libvirt NAT network does not share any information with the host system. There are multiple ways to solve this issue ((See the following links for information about configuring the default Libvirt network to assign static IP addresses, and maintaining the /etc/hosts file in sync with virtual machines’ static IP addresses, and automating DHCP and Libvirt domains.)) and I suggest you use the Libvirt NSS module, which plugs the Libvirt network information into the information sources consulted by your system’s Name Service Switch.

You already installed the libnss-libvirt package so you just need to edit the host system’s NSS configuration file so it will consult the Libvirt NSS modules, libvirt and libvirt_guest to get the IP address of any running virtual machine attached to a Libvirt-managed NAT network.

Edit the file /etc/nsswitch.conf:

brian@T420:~$ sudo nano /etc/nsswitch.conf

In the file’s hosts: line, add in the libvirt and libvirt_guest modules, in the order in which you want the system to consult its available name services. The file should look like the listing below when you are completed:

passwd:         compat systemd
group:          compat systemd
shadow:         compat
gshadow:        files

hosts:          files libvirt libvirt_guest mdns4_minimal [NOTFOUND=return] dns$
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

When your system is looking up a hostname, the above configuration will cause the system to first consult the available name services in the following order: the /etc/hosts file, the static IP addresses configured in the Libvirt network’s XML file, the Libvirt-managed VMs’ names listed in the DHCP server, and the dnsmasq service. It chooses the first match it finds.

Save the file.

Now, you can use SSH to login to a virtual machine without knowing its IP address. For example, to login to the sim account on a virtual machine named server, run the following command:

brian@T420:~$ ssh sim@server
sim@server:~$ 

Plan your network emulation topology

Create a network plan that you can use to guide your configurations and to help you troubleshoot problems. Draw a detailed diagram of the virtual network you want to create. This will help you visualize how all the bridges, nodes, and ports connect to each other.

In the network diagram I created below, you should see that there are five VM-to-VM connections in the high-level network diagram. You will implement each connection using a Linux bridge ((Create a new bridge for each “wire” that will connect nodes together, not counting management connections, which all go to the same management bridge.)) connected to virtual interfaces on the two virtual machines at either end of the connection.

From the diagram, you see which ports on each virtual machine are connected to which bridge. Convert the information in the drawing into a table that will help you map ports to bridges when you are building your virtual network. On each line in the table, add the MAC addresses and IP addresses you chose for each port.

For this example, I created the following network planning table.

VM name	VM port	MAC address	IP address	Bridge name
user	1	assigned by Libvirt	DHCP	virbr0
user	2	02:00:aa:0a:01:02	10.10.100.1/24	br_user_r1
r01	1	assigned by Libvirt	DHCP	virbr0
r01	2	02:00:aa:01:0a:02	10.10.100.2/24	br_user_r1
r01	3	02:00:aa:01:02:03	10.10.12.1/24	br_r1_r2
r01	4	02:00:aa:01:03:04	10.10.13.1/24	br_r1_r3
r02	1	assigned by Libvirt	DHCP	virbr0
r02	2	02:00:aa:02:03:02	10.10.23.1/24	br_r2_r3
r02	3	02:00:aa:02:01:03	10.10.12.2/24	br_r1_r2
r03	1	assigned by Libvirt	DHCP	virbr0
r03	2	02:00:aa:03:01:02	10.10.13.2/24	br_r1_r3
r03	3	02:00:aa:03:02:03	10.10.23.2/24	br_r2_r3
r03	4	02:00:aa:03:0b:04	10.10.200.2/24	br_r3_serv
server	1	assigned by Libvirt	DHCP	virbr0
server	2	02:00:aa:0b:03:02	10.10.200.1/24	br_r3_serv

I chose arbitrary MAC addresses, using a convention that helps me remember which MAC address I assigned to which node and port. I also chose arbitrary IP addresses from private IP address space, again following a convention that helps me quickly determine which node and port are associated with each address.

As you can imagine, this can get very complex when you add more nodes and links. I suggest you plan carefully, build your virtual network a little bit at a time, and test connectivity on new links as you build them.

Create the base VMs for the PC nodes and router nodes

Create a base, or template, virtual machine for each type of node you will deploy in your network emulation scenario. This enables you to quickly create new network nodes by cloning new virtual machines from one of your base VMs. Your new virtual machines will come with all the default software and configurations you staged on the base VMs.

In this network emulation scenario, you have two node types: a PC and a router. To create base VMs for each of them, install Ubuntu Server 18.04 on a new VM and configure it as a base PC VM. Clone the base PC VM to create a second VM, which you will configure and use as the base router VM.

Create the PC base virtual machine

Get a Linux disk image from the appropriate distribution and use it to build the base PC VM. I used the Ubuntu Server distribution to build the network nodes. Find an Ubuntu mirror closest to you.

Enable serial console access by inserting extra arguments into the virtual machine’s boot process so you can use its command-line-interface installer in your terminal. You must use the virt-install command’s location option instead of the cdrom option because the cdrom option does not allow us to insert extra arguments into the VM at boot time. ((See the LOCATION section in the virt-install man pages for more information.)) The location option does not support an ISO file as the install media; it requires access to a repository directory. Find a mirror that offers the Ubuntu repository directory.

brian@T420:~$ virt-install \
  --name pc-base \
  --virt-type=kvm --hvm --ram 1024 \
  --disk path=/simulator/sim01/pc-base.qcow2,size=4 \
  --vcpus 1 --os-type linux --os-variant ubuntu18.04 \
  --network bridge=virbr0 \
  --graphics none \
  --location 'http://mirror.math.princeton.edu/pub/ubuntu/dists/bionic/main/installer-amd64/' \
  --extra-args='console=ttyS0'

I configured the system with userid sim, and password sim. When asked to select software, I chose OpenSSH Server and Basic Ubuntu Server options.

Fix the serial interface

When the installation process ends, the new virtual machine will reset. Your local terminal will show a blank screen because the extra arguments you passed to the virtual machine’s boot configuration during system installation were not permanently saved in the VM’s boot configuration. You cannot access the new VM’s serial interface.

To fix this, stop the virtual machine and use the guestmount utility to configure a serial interface on the its disk.

Press Ctrl-] to get back to your host system’s terminal.

Shutdown the guest VM, as follows:

brian@T420:~$ virsh shutdown pc-base

Mount guest’s disk and enable a serial port (by manually enabling a getty service) using the following commands:

brian@T420:~$ sudo mkdir /mnt/pc
brian@T420:~$ sudo guestmount --domain pc-base \
    --inspector /mnt/pc
brian@T420:~$ sudo ln -s \
    /mnt/pc/lib/systemd/system/[email protected] \
    /mnt/pc/etc/systemd/system/getty.target.wants/[email protected]
brian@T420:~$ sudo umount /mnt/pc

Start virtual machine again:

brian@T420:~$ virsh start pc-base 

After the virtual machine starts, test that the console works. Try to access the pc-base VM from your host system using the virsh console command:

brian@T420:~$ virsh console pc-base

You should see a login prompt. Login to the virtual machine and configure it.

Configure the pc-base VM

The first virtual machine will be used as the template from which you will clone other “host” VMs in your network emulation scenario. Stage it with the basic configurations that you want any other machine cloned from it to have. For example, configure some basic network tools:

sim@pc-base:~$ sudo apt update
sim@pc-base:~$ sudo apt install -y traceroute tcpdump nmap

Stop the pc-base VM

So that you can clone it, shut down the virtual machine you created and configured. Exit its console with the CTRL-] key combination and stop it:

sim@pc-base:~$ 
CTRL-]
brian@T420:~$ virsh shutdown pc-base

If you want to see how Libvirt defines the virtual machine in an XML file, run the dumpxml command. The dumpxml command may also be used in scripts when you want to check some of its attributes — which I will demonstrate later.

brian@T420:~$ virsh dumpxml pc-base

If you need to tweak the virtual machine’s Libvirt settings in the future, you can edit its Libvirt XML file with the command:

brian@T420:~$ virsh edit pc-base

Fix permissions on VM disk image files

Use libguestfs-tools to manipulate the VM’s disk images. First, though, fix the file permissions of the disk images you created. Libvirt creates the disk images so that they can only be accessed by the root user and group. Fix them so that users in the libvirt group may also access them.

Remember, you previously added your userid to the libvirt group and configured Linux ACLs so all files created in the /simulator/ directory and its subdirectories were also owned by the libvirt group. However, you still need to fix the file permissions so users that are members of other groups can access the disk files.

brian@T420:~$ sudo chmod g+rw /simulator/sim01/*

The libguestfs developers recommend that you do not use root privileges when you run libguestfs tools, like virt-sysprep, on your VM disk images. That means you will need to run the chmod command, above, every time you clone a new VM ((I am looking for a permanent fix for the virt-clone disk file ownership and permissions issue. If you have one, please post it in the comments, below. Thanks!)).

Minimizing disk storage size

I am keeping things simple in this example, so I created an independent disk image for each virtual machine. This uses a lot of disk space. If you are creating many virtual machines, you may wish to investigate using copy-on-write images while using the a QCOW2 image as backing file, or master disk image. This will greatly reduce the storage space consumed by disk images.

In this case, one way to improve disk usage is to sparsify the VM disk image. The virt-sparsify tool converts free space inside the VM’s disk image back to free space on the host’s filesystem.

brian@T420:~$ virt-sparsify pc-base.qcow2 pc-base-sparse.qcow2
brian@T420:~$ mv pc-base.qcow2 pc-base-fixed.qcow2
brian@T420:~$ mv pc-base-sparse.qcow2 pc-base.qcow2
brian@T420:~$ sudo chmod g+rw /simulator/sim01/*

You will see that the virtual machine’s disk image, which used to (apparently) take up 4.1 GB on the host system’s filesystem, now consumes only 1.9 GB. ((Note that, if you run the du -sh pc-base-old.fixed command, you will see that the original VM disk image really only consumed 2.8 GB on the host filesystem. Still, this is a 30% reduction in disk usage.))

Clone the pc-base VM to create the router-base VM

Three of the nodes in this network emulation scenario are routers so you need to create a base router VM from which you can clone other router VMs. Start by cloning the pc-base VM to create the router-base VM. Log in to the router-base VM and configure it to operate as a router.

brian@T420:~$ virt-clone --original pc-base \
    --name router-base \
    --file /simulator/sim01/router-base.qcow2
brian@T420:~$ sudo chmod g+rw /simulator/sim01/*

Individualize the cloned router-base VM

When you clone a VM, you copy all configurations from the source VM to the new cloned VM. This creates problems because every node on a network is expected to have a unique hostname and machine-id. ((The machine-id identifies the VM as a unique node for DHCP network interface configuration. If you see strange DHCP behaviour on your management network, verify that all your VMs have a unique machine-id.))

One way to individualize each cloned VM is to start each one, log into it, change the required settings, and shut it down. However, a better way is to directly manipulate the cloned VM’s disk image using libguestfs tools like virt-sysprep.

Use the virt-sysprep command to configure the router-base VM with a unique hostname and machine ID, and to clear the DHCP client state:

brian@T420:~$ virt-sysprep --domain router-base \
    --enable customize,dhcp-client-state,machine-id \
    --hostname 'router-base'

Configure the router-base VM

Start the router-base VM and connect to it via SSH or via its console. Remember that the VMs’ userid and password are both set to “sim”:

brian@T420:~$ virsh start router-base
brian@T420:~$ ssh sim@router-base
sim@router-base:~$ 

Install the Free Range Routing protocol suite. This involves many steps, as shown below. The following instructions show how to build FRR on Ubuntu 18.04.

Get the latest version of FRR from the FRR Releases page at: https://github.com/FRRouting/frr/releases. Be sure to check for the latest release. The examples below may no be the latest release.

sim@router-base:~$ mkdir ~/software
sim@router-base:~$ cd ~/software
sim@router-base:~$ wget https://github.com/FRRouting/frr/releases/download/frr-6.0.2/frr_6.0.2-0.ubuntu18.04.1_amd64.deb
sim@router-base:~$ wget https://github.com/FRRouting/frr/releases/download/frr-6.0.2/frr-pythontools_6.0.2-0.ubuntu18.04.1_all.deb
sim@router-base:~$ wget https://github.com/FRRouting/frr/releases/download/frr-6.0.2/frr-doc_6.0.2-0.ubuntu18.04.1_all.deb

Install the FRR packages from the downloaded packages:

sim@router-base:~$ sudo apt -y install ./frr_6.0.2-0.ubuntu18.04.1_amd64.deb
sim@router-base:~$ sudo apt -y install ./frr-doc_6.0.2-0.ubuntu18.04.1_all.deb
sim@router-base:~$ sudo apt -y install ./frr-pythontools_6.0.2-0.ubuntu18.04.1_all.deb

Get the latest Libyang packages at: https://ci1.netdef.org/browse/LIBYANG-YANGRELEASE/latestSuccessful/artifact, and install them:

sim@router-base:~$ wget https://ci1.netdef.org/artifact/LIBYANG-YANGRELEASE/shared/build-1/Ubuntu-18.04-x86_64-Packages/libyang-dev_0.16.46_amd64.deb
sim@router-base:~$ wget https://ci1.netdef.org/artifact/LIBYANG-YANGRELEASE/shared/build-1/Ubuntu-18.04-x86_64-Packages/libyang_0.16.46_amd64.deb
sim@router-base:~$ sudo apt -y install ./libyang_0.16.46_amd64.deb
sim@router-base:~$ sudo apt -y install ./libyang-dev_0.16.46_amd64.deb

To enable IPv4 & IPv6 forwarding, edit the /etc/sysctl.conf file

sim@router-base:~$ sudo nano /etc/sysctl.conf

Uncomment the following lines (ignore the other settings):

net.ipv4.ip_forward=1

net.ipv6.conf.all.forwarding=1

Save the file.

To enable MPLS on the router, edit the /etc/modules-load.d/modules.conf file:

sim@router-base:~$ sudo nano /etc/modules-load.d/modules.conf

Add the following lines to /etc/modules-load.d/modules.conf:

# Load MPLS Kernel Modules
mpls_router
mpls_iptunnel

Save the file. Run sysctl -p to apply the new config to the running system.

sim@router-base:~$ sudo sysctl -p

To enable the protocol daemons, edit the /etc/frr/daemons file:

sim@router-base:~$ sudo nano /etc/frr/daemons

Change the each daemon’s value from “no” to “yes” if you want it to start when the VM starts. For example, I suggest you start OSPF on every router to create a simple, single-area IGP domain:

bgpd=no
ospfd=yes
ospf6d=no
ripd=no
ripngd=no
isisd=no
pimd=no
ldpd=no
nhrpd=no
eigrpd=no
babeld=no
sharpd=no
pbrd=no
bfdd=no

Save the file.

You can’t enable MPLS forwarding on interfaces because, at this point, there are no interfaces available. you will be able to enable MPLS forwarding on each router instance after you add interfaces to them — after you plug them into the network emulation scenario.

Stop the router-base VM

Like the pc-base VM earlier, you need to shut down the router-base VM so you can clone router instances from it. Exit the VM’s console with the CTRL-] key combination and stop the VM.

sim@router-base:~$ logout
brian@T420:~$ virsh shutdown router-base

Create the VMs for the network emulation scenario

In the previous section, you created the pc-base or router-base VMs that will serve as “golden master” images. Clone these virtual machines to create the virtual machines that run in your network emulation scenarios.

Clone the pc-base VM to create the user and server VMs

In this example, you are creating a network of five nodes, connected together on the same virtual network. Two of those nodes emulate a user and a server on the network. Create those VMs from clones of the pc-base VM:

brian@T420:~$ virt-clone --original pc-base \
    --name user \
    --file /simulator/sim01/user.qcow2
brian@T420:~$ sudo chmod g+rw /simulator/sim01/*
brian@T420:~$ virt-sysprep --domain user \
    --enable customize,dhcp-client-state,machine-id \
    --hostname 'user'

brian@T420:~$ virt-clone --original pc-base \
    --name server \
    --file /simulator/sim01/server.qcow2
brian@T420:~$ sudo chmod g+rw /simulator/sim01/*
brian@T420:~$ virt-sysprep --domain server \
    --enable customize,dhcp-client-state,machine-id \
    --hostname 'server'

Clone the router-base VM to create router instances

In this example, you are creating a network of three routers, connected together. Create the router VMs by cloning the router-base VM, and individualizing each instance:

brian@T420:~$ virt-clone --original router-base \
    --name r01 \
    --file /simulator/sim01/r01.qcow2
brian@T420:~$ sudo chmod g+rw /simulator/sim01/*
brian@T420:~$ virt-sysprep --domain r01 \
    --enable customize,dhcp-client-state,machine-id \
    --hostname 'r01'

brian@T420:~$ virt-clone --original router-base \
    --name r02 \
    --file /simulator/sim01/r02.qcow2
brian@T420:~$ sudo chmod g+rw /simulator/sim01/*
brian@T420:~$ virt-sysprep --domain r02 \
    --enable customize,dhcp-client-state,machine-id \
    --hostname 'r02'

brian@T420:~$ virt-clone --original router-base \
    --name r03 \
    --file /simulator/sim01/r03.qcow2
brian@T420:~$ sudo chmod g+rw /simulator/sim01/*
brian@T420:~$ virt-sysprep --domain r03 \
    --enable customize,dhcp-client-state,machine-id \
    --hostname 'r03'

Check Libvirt definitions

Verify the virtual machines are defined in Libvirt:

brian@T420:~$ virsh list --all

You should see the following output:

 Id    Name                           State
----------------------------------------------------
 -     pc-base                        shut off
 -     r01                            shut off
 -     r02                            shut off
 -     r03                            shut off
 -     router-base                    shut off
 -     server                         shut off
 -     user                           shut off

Start the VMs in the network emulation scenario

Run the VM’s you created for the planned network emulation scenario so you can build network links one-by-one on live VMs and test them as you go along. Run the following commands to start the VMs you recently created.

brian@T420:~$ virsh start r01
brian@T420:~$ virsh start r02
brian@T420:~$ virsh start r03
brian@T420:~$ virsh start user
brian@T420:~$ virsh start server

Verify that all the VMs are running:

brian@T420:~$ virsh list
 Id    Name                           State
----------------------------------------------------
 3     r01                            running
 4     r02                            running
 5     r03                            running
 6     user                           running
 7     server                         running

Create the first Libvirt network

So you can understand how Libvirt creates and manages networks and connections to virtual machines, I will walk through a detailed example in which you will create your first network that connects the user VM to the r01 VM.

Refer to the network diagram, above. See the network bridge and the VM interfaces that that must be added up to create the network link between the user VM to the r01 VM.

From the original network planning table, I pulled out the specific connections and configurations you need to create for the first Libvirt network, and listed them below:

VM name	VM port	MAC address	IP address	Bridge name
user	2	02:00:aa:0a:01:02	10.10.100.1/24	br_user_r1
r01	2	02:00:aa:01:0a:02	10.10.100.2/24	br_user_r1

To create a Libvirt-managed network, first create an XML file that defines the network attributes so you can import that file into Libvirt. Create and edit the file /tmp/r1user.xml:

brian@T420:~$ nano /tmp/r1user.xml

Follow the guidelines in the Libvirt XML format documentation. Configure only the minimum information needed to define the bridge and the network. Libvirt will fill in all other details with default values.

Enter the following XML code into the file:

<network>
  <name>net_user_r1</name>
  <bridge name="br_user_r1" stp='off' macTableManager="libvirt"/> 
  <mtu size="9216"/>
</network>

I gave the network and the bridge it creates the name net_user_r1, which helps you understand its place in the planned network topology. I wanted to create a point-to-point “virtual wire” and did not want the network bridge to interact with the devices connected to it so I turned off Spanning Tree Protocol, set Libvirt to manage the MAC forwarding table, and set the MTU size to the jumbo-frame size. ((This is good enough for most scenarios. If you need fully transparent flooding across the bridge (hub emulation), you need to use a more complex setup with Open vSwitch or macvtap with Libvirt. If you want to continue using Linux bridges, you may enable LLDP frame forwarding by tweaking the system settings and enable LACP and STP frame forwarding by patching and re-compiling the Linux kernel.))

https://www.linux.com/learn/intro-to-linux/2018/4/how-compile-linux-kernel-0

Save the file. Import the network XML file into Libvirt with the following command. It will cause Libvirt to define the network described in the XML file, which in this case is the network named net_user_r1.

brian@T420:~$ virsh net-define /tmp/r1user.xml

Now, the network net_user_r1 is managed by Libvirt. Verify this with the following command:

brian@T420:~$ virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes
 net_user_r1          inactive   no            yes

The network is not started so the bridge does not exist, yet. Start the network to create the bridge:

brian@T420:~$ virsh net-start net_user_r1

Now, the network is started and you can verify the bridge exists with the brctl command:

brian@T420:~$ brctl show br_user_r1
bridge name  bridge id          STP enabled  interfaces
br_user_r1   8000.525400e1f59a  no           br_user_r1-nic

Connect the user and r01 VMs to the new network

Connect the VMs r01 and user to the new bridge br_user_r1 by using Libvirt to create new interfaces on each virtual machine and connect those interfaces to the bridge. Libvirt makes this easy.

Get information about the existing interfaces on r01 and user.

brian@T420:~$ virsh domiflist user
Interface  Type     Source    Model    MAC
-------------------------------------------------
vnet3      bridge   virbr0    virtio   52:54:00:9b:4e:16

brian@T420:~$ virsh domiflist r01
Interface  Type     Source    Model    MAC
-------------------------------------------------
vnet0      bridge   virbr0    virtio   52:54:00:be:5c:3a

You see that PC user‘s and router r01‘s management interfaces, vnet3 and vnet0, are attached to the management bridge virbr0

Add a new interface on user connected to the new network net_user_r1 which, practically, connects it to the bridge br_user_r1. Use the MAC interface from the value I listed in the network planning table, above.

brian@T420:~$ virsh attach-interface \
          --domain user \
          --type network \
          --source net_user_r1 \
          --model virtio \
          --mac 02:00:aa:0a:01:02 \
          --config --live

Do the same on the other side of the “wire”. Create a new interface on VM r01 and connect it to the same network net_user_r1.

brian@T420:~$ virsh attach-interface \
          --domain r01 \
          --type network \
          --source net_user_r1 \
          --model virtio \
          --mac 02:00:aa:01:0a:02 \
          --config --live

Check the bridges again. See that the new interfaces, named vnet5 and vnet6, have been added to the bridge br_user_r1:

brian@T420:~$ brctl show br_user_r1
bridge name  bridge id          STP enabled  interfaces
br_user_r1   8000.525400e1f59a  no           br_user_r1-nic
                                             vnet5
                                             vnet6

Also, check the VM interfaces. See interface the vnet5 on VM user and interface vnet6 on VM r01:

brian@T420:~$ virsh domiflist user
Interface  Type     Source      Model    MAC
-------------------------------------------------------
vnet3      bridge   virbr0      virtio   52:54:00:9b:4e:16
vnet5      network  net_user_r1 virtio   02:00:aa:0a:01:02

brian@T420:~$ virsh domiflist r01
Interface  Type     Source      Model    MAC
-----------------------------------------------------
vnet0      bridge   virbr0      virtio   52:54:00:be:5c:3a
vnet6      network  net_user_r1 virtio   02:00:aa:01:0a:02

Test the connection

Test the new connection between VMs user and r01. Log into each of the virtual machines and look for the new interface. Configure the new interfaces with IP addresses and test that you can ping from one node to the other.

Log into VM user:

brian@T420:~$ virsh console user
sim@user:~$ sudo su
sim@user:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:9b:4e:16 brd ff:ff:ff:ff:ff:ff
3: ens6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:00:aa:0a:01:02 brd ff:ff:ff:ff:ff:ff

A new interface, ens6, is the second interface (not including the loopback interface) on the VM. Interface ens6 is “port 2”, as shown in the network diagram and planning table. Configure this interface with the commands:

sim@user:~# ip addr add 10.10.100.1/24 dev ens6
sim@user:~# ip link set dev ens6 up
sim@user:~# 
Ctrl-[
brian@T420:~$

Configure the corresponding interface on r01:

brian@T420:~$ ssh sim@r01
sim@r01:~$ sudo su
sim@r01:~# ip addr add 10.10.100.2/24 dev ens6
sim@r01:~# ip link set dev ens6 up

Test the connection to the VM user with the commands:

sim@r01:~# ping -c 1 10.10.100.1
PING 10.10.100.1 (10.10.100.1) 56(84) bytes of data.
64 bytes from 10.10.100.1: icmp_seq=1 ttl=64 time=1.82 ms

--- 10.10.100.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.820/1.820/1.820/0.000 ms
sim@r01:~# exit
sim@r01:~$ exit
brian@T420:~$

You have demonstrated that the link you created between VM user and VM r01 is working.

Create the remaining networks

Connect the remaining VMs to each other by creating an XML file that defines each network according to your network plan. After you save the XML file, run the virsh net-define command to import it into Libvirt, run the virsh net-start command for each network.

You have already set up five virtual machines and one network named net_user_r1 between two of those nodes. The following commands will create the rest of the virtual networks, according to the plan:

brian@T420:~$

cat >> /tmp/r1r2.xml << EOF
<network>
  <name>net_r1_r2</name>
  <bridge name="br_r1_r2" stp='off' macTableManager="libvirt"/> 
  <mtu size="9216"/> 
</network>
EOF

cat >> /tmp/r2r3.xml << EOF
<network>
  <name>net_r2_r3</name>
  <bridge name="br_r2_r3" stp='off' macTableManager="libvirt"/> 
  <mtu size="9216"/> 
</network>
EOF

cat >> /tmp/r1r3.xml << EOF
<network>
  <name>net_r1_r3</name>
  <bridge name="br_r1_r3" stp='off' macTableManager="libvirt"/> 
  <mtu size="9216"/> 
</network>
EOF

cat >> /tmp/r3serv.xml << EOF
<network>
  <name>net_r3_serv</name>
  <bridge name="br_r3_serv" stp='off' macTableManager="libvirt"/> 
  <mtu size="9216"/> 
</network>
EOF

virsh net-define /tmp/r1r2.xml
virsh net-define /tmp/r2r3.xml
virsh net-define /tmp/r1r3.xml
virsh net-define /tmp/r3serv.xml

virsh net-start net_r1_r2
virsh net-start net_r2_r3
virsh net-start net_r1_r3
virsh net-start net_r3_serv

Connect the VMs to the networks

Use the virsh attach-interface command to create interfaces on the VMs and connect them to each network according to the plan. Use the MAC address information you listed in your plan. Test each connection after you create it, to ensure you are following the plan.

Be careful about the order in which you run the attach-interface commands. Interfaces are created on virtual machines in the order in which you attach them. For example, if you want “port 4” on r03 connected to bridge br_r3_serv, ensure that you make that connection the third time you use the attach-interface command on r03 (since “port 1” already exists).

Let’s walk through the process, interface-by-interface.

You previously connected user and r01 to bridge br_user_r1. So you know that port ens6 (which is “port 2” in the diagram) on user and port ens6 (also “port 2”) on r01 have been assigned by Libvirt. So when you run the following command, you know it will attach “port 3” (knows as ens7 on the router) on r01 to bridge br_r1_r2.

Connect “port 3” (ens7) on r01 to bridge net_r1_r2.

virsh attach-interface --domain r01 \
    --type network --source net_r1_r2 \
    --model virtio --mac 02:00:aa:01:02:03 \
    --config --live

Connect “port 4” (ens8) on r01 to bridge net_r1_r3.

virsh attach-interface --domain r01 \
--type network --source net_r1_r3 \
--model virtio --mac 02:00:aa:01:03:04 \
--config --live

Connect “port 2” (ens6) on r02 to bridge net_r2_r3.

virsh attach-interface --domain r02 \
--type network --source net_r2_r3 \
--model virtio --mac 02:00:aa:02:03:02 \
--config --live

Connect “port 3” (ens7) on r02 to bridge net_r1_r2.

virsh attach-interface --domain r02 \
--type network --source net_r1_r2 \
--model virtio --mac 02:00:aa:02:01:03 \
--config --live

Connect “port 2” (ens6) on r03 to bridge net_r1_r3.

virsh attach-interface --domain r03 \
--type network --source net_r1_r3 \
--model virtio --mac 02:00:aa:03:01:02 \
--config --live

Connect “port 3” (ens7) on r03 to bridge net_r2_r3.

virsh attach-interface --domain r03 \
--type network --source net_r2_r3 \
--model virtio --mac 02:00:aa:03:02:03 \
--config --live

Connect “port 4” (ens8) on r03 to bridge net_r3_serv.

virsh attach-interface --domain r03 \
--type network --source net_r3_serv \
--model virtio --mac 02:00:aa:03:0b:04 \
--config --live

Connect “port 2” (ens6) on server to bridge net_r3_serv.

virsh attach-interface --domain server \
--type network --source net_r3_serv \
--model virtio --mac 02:00:aa:0b:03:02 \
--config --live

Configure each network node

At this point, a researcher would install product software and configure networking on each virtual machine in the virtual network and begin testing the network emulation scenario’s behaviour.

The following commands, run on each network node, will create a minimal network configuration that uses OSPF routing protocol to distribute node reachability information and enable on each node to ping any other node in the network.

Run the following commands:

VM user:

Log into VM user:

brian@T420:~$ ssh sim@user
sim@user:~$ sudo su
sim@user:~#

Copy and paste the following text into the user VM’s terminal:

bash <<EOF2
rm /etc/netplan/01-netcfg.yaml
cat >> /etc/netplan/01-netcfg.yaml << EOF
network:
  version: 2
  renderer: networkd
  ethernets:
    ens2:
      dhcp4: yes
    ens6:
      addresses:
        - 10.10.100.1/24
      #gateway4:    
      routes:
        - to: 10.10.0.0/16
          via: 10.10.100.2
          metric: 100
EOF
chmod 644 /etc/netplan/01-netcfg.yaml
netplan apply
EOF2

Exit the VM:

sim@user:~# exit
sim@user:~$ exit
brian@T420:~$

VM r01:

Log into VM r01.

brian@T420:~$ ssh sim@r01
sim@r01:~$ sudo su
sim@r01:~#

Copy and paste the following text into the r01 VM’s terminal:

bash <<EOF2
cat >> /etc/frr/frr.conf << EOF
frr version 6.0.2
frr defaults traditional
hostname r01
service integrated-vtysh-config
!
interface ens6
 ip address 10.10.100.2/24
!
interface ens7
 ip address 10.10.12.1/24
!
interface ens8
 ip address 10.10.13.1/24
!
router ospf
 ospf router-id 1.1.1.1
 redistribute connected
 passive-interface ens6
 network 10.10.12.0/24 area 0
 network 10.10.13.0/24 area 0
 network 10.10.100.0/24 area 0
!
line vty
!
EOF
systemctl reload frr
EOF2

Exit the VM:

sim@r01:~# exit
sim@r01:~$ exit
brian@T420:~$

VM r02:

Log into VM r02.

brian@T420:~$ ssh sim@r02
sim@r02:~$ sudo su
sim@r02:~# 

Copy and paste the following text into the r02 VM’s terminal:

bash <<EOF2
cat >> /etc/frr/frr.conf << EOF
frr version 6.0.2
frr defaults traditional
hostname r02
service integrated-vtysh-config
!
interface ens6
 ip address 10.10.23.1/24
!
interface ens7
 ip address 10.10.12.2/24
!
router ospf
 ospf router-id 2.2.2.2
 redistribute connected
 network 10.10.23.0/24 area 0
 network 10.10.12.0/24 area 0
!
line vty
!
EOF
systemctl reload frr
EOF2

Exit the VM:

sim@r02:~# exit
sim@r02:~$ exit
brian@T420:~$

VM r03:

Log into VM r03.

brian@T420:~$ ssh sim@r03
sim@r03:~$ sudo su 
sim@r03:~# 

Copy and paste the following text into the r03 VM’s terminal:

bash <<EOF2
cat >> /etc/frr/frr.conf << EOF
frr version 6.0.2
frr defaults traditional
hostname r03
service integrated-vtysh-config
!
interface ens6
 ip address 10.10.13.2/24
!
interface ens7
 ip address 10.10.23.2/24
!
interface ens8
 ip address 10.10.200.2/24
!
router ospf
 ospf router-id 3.3.3.3
 redistribute connected
 passive-interface ens8
 network 10.10.13.0/24 area 0
 network 10.10.23.0/24 area 0
 network 10.10.200.0/24 area 0
!
line vty
!
EOF
systemctl reload frr
EOF2

Exit the VM:

sim@r03:~# exit
sim@r03:~$ exit
brian@T420:~$

VM server

Log into VM server.

brian@T420:~$ ssh sim@server
sim@server:~$ sudo su
sim@server:~#

Copy and paste the following text into the server VM’s terminal:

bash <<EOF2
rm /etc/netplan/01-netcfg.yaml
cat >> /etc/netplan/01-netcfg.yaml << EOF
network:
  version: 2
  renderer: networkd
  ethernets:
    ens2:
      dhcp4: yes
    ens6:
      addresses:
        - 10.10.200.1/24
      #gateway4:    
      routes:
        - to: 10.10.0.0/16
          via: 10.10.200.2
          metric: 100
EOF
chmod 644 /etc/netplan/01-netcfg.yaml
netplan apply
EOF2

Exit the VM:

sim@server:~# exit
sim@server:~$ exit
brian@T420:~$

The above commands complete the minimum configuration, which is also stored in each VM’s configuration files so the network emulation scenario will start in this state whenever the VM’s and networks are started.

Network Emulation scripts

You may simplify the setup and tear-down of your network emulation scenarios by creating a few simple scripts. Libvirt handles all the complexity of creating the networks and virtual machines so you need create just two simple scripts: startlab.sh and stoplab.sh.

startlab.sh

Create a new file named startlab.sh by pasting the following text into your terminal:

brian@T420:~$

cat >> startlab.sh << EOF
#!/usr/bin/env bash
set -e
set -x
virsh net-start net_r1_r2
virsh net-start net_r1_r3
virsh net-start net_r2_r3
virsh net-start net_r3_serv
virsh net-start net_user_r1
virsh start R01
virsh start R02
virsh start R03
virsh start server
virsh start user
EOF

stoplab.sh

Create a new file named stoplab.sh by pasting the following text into your terminal:

brian@T420:~$

cat >> stoplab.sh << EOF
#!/usr/bin/env bash
set -e
set -x
virsh destroy R01
virsh destroy R02
virsh destroy R03
virsh destroy server
virsh destroy user
virsh net-destroy net_r1_r2
virsh net-destroy net_r1_r3
virsh net-destroy net_r2_r3
virsh net-destroy net_r3_serv
virsh net-destroy net_user_r1
EOF

Set the files so they are executable:

brian@T420:~$ chmod +x *.sh

In the future, when you want to start a lab, navigate the the lab’s directory and run the startlab.sh script. Stop the lab by running the stoplab.sh script.

Conclusion

I showed you that you can use Libvirt and libguestfs tools to create a simple network emulation scenario consisting of virtual machines and virtual networks.

I performed many of the operations that created the network nodes and the virtual networks by running commands on the host system, but I configured each node by logging into it and making local changes. According to the libguestfs documentation, it should be possible to configure every guest VM in the network emulation scenario using libguestfs tools running on the host system. This would eliminate the need to log in to each VM to configure it, and would enable scripts to build and configure a complex network scenario.

I think that manually building complex network emulation scenarios with Libvirt’s command-line-interface is difficult and the risk of making a configuration error at some point is high. However, one could write a program that use the Libvirt API to create complex network emulation scenarios based on data read in from some sort of topology file, like a dot file.

Appendix A: Why not use Libvirt storage pools?

Libvirt users may store virtual machine disk images in a storage pool managed by Libvirt. Users can define a directory as the storage pool and store disk images in that directory, or they can use a wide variety of different file system technologies or remote repositories.

Storage pools may be useful in more complex Libvirt use-cases, and they are a building block for applications build on top of Libvirt. However, network emulation scenarios is relatively simple (as a virtualization technology) and some of the other virtualization tools I plan to use, such as virt-clone and other libguestfs tools, do not support Libvirt storage pools. For this reason, I did not define a Libvirt storage pool and used the –file or –disk options in Libvirt to point to disk images in a directory.

Appendix B: Other distributions for guests

I used Ubuntu Server to create the guest VMs in this network emulation scenario. You may wish to use different Linux distributions to create the VMs. Be aware, however, that some Linux distributions — especially network appliances, and distributions tuned to be as small as possible or to offer a high degree of security — may not be compatible with the some of tools I used in this post, such as virt-sysprep.