Tags:
create new tag
, view all tags

Running DRAGON Software under User-Mode Linux (UML)

Overview

DRAGON's GMPLS Control Plane software is intended to be run on multiple Unix PCs across a network. The software uses raw IPv4 sockets to communicate to neighboring network elements via GRE/IPIP/IPsec tunnels (for OSPF adjacencies and RSVP signaling), so multiple instances cannot be run on a single operating system. Not much can be done with a single instance of the DRAGON code running on a single machine, so using user-mode linux was investigated as a possible solution to this problem.

It is now possible to run DRAGON's GMPLS Control Plane software in a user-mode linux environment, allowing an entire GMPLS network to be simulated on a single Linux PC. This page is intended to contain all of the information a user would need to setup such an environment. Please note that only the GMPLS control plane is currently emulated, we do not yet simulate a data plane. More information about this can be found in the section on future work.

For background information on User-Mode Linux, we recommend visiting the User-mode Linux Kernel Home Page or by reading Jeff Dike's book on User-Mode Linux. There is also a whitepaper here on Advanced Network Simulation under User-Mode Linux which is exploring many similar concepts.

Virtual Network Experiments (VNE)

An existing open-source package called Virtual Network Experiments (VNE) has been slightly modified to be able to run the DRAGON code in the UML environment, in such a way that an XML file can be used as to express the desired topology of the simulated GMPLS network. A paper describing VNE can be found at http://www.os3.nl/~gjv/uml/anp-jvdh-gjv.pdf or refer to the mirrored copy. Example XML files are included below to allow the user to quickly bring up simulated GMPLS networks of varying levels of complexity. The VNE software was originally developed by the Universiteit of Amsterdam's Systems and Networking Engineering research group. A webpage for VNE has been setup at http://ndl.uva.netherlight.nl/trac/vne/

VNE overview graphic

Requirements

  • any x86-based PC that is capable of running Linux
    • a minimum of 512MB of RAM in the UML host machine will be required to run the basic examples
    • 1GB of RAM (or more) is needed for larger, more complex simulations
    • each UML instance requires about 32MB of physical memory
  • any Linux distribution
    • our testing was done with the Debian 4.0 ("etch"), Debian 3.1 ("sarge"), Ubuntu 6.10 ("edgy eft") and Redhat Fedora Core 3 distributions
    • the latest debian ("unstable") should also be fine
    • any system running a Linux 2.4 or 2.6 kernel should work

The remaining prerequisites are available for download from this page:

  • Virtual Network Experiments (VNE) software package
  • Virtual Distributed Ethernet (VDE) software package
  • uml-utilities software package
  • pre-compiled UML kernel binary
  • custom UML root filesystem based on Debian 4.0 ("etch") distribution
    • includes DRAGON's GMPLS Control Plane Software and supporting utilities

Step-by-Step Installation

  • install Linux
    • see notes above under requirements regarding different distributions
  • install python
    • in Debian/Ubuntu, run apt-get install python
    • alternatively, see the python homepage
  • install uml-utilities package
  • install xterm
    • in Debian/Ubuntu, run apt-get install xterm
    • for more information, refer to the wikipedia page
    • NOTE: in Debian, X11 forwarding will not work if the xauth application is missing
      • run apt-get install xorg to make sure xauth is properly installed
  • install GNU screen
    • in Debian/Ubuntu, run apt-get install screen
    • alternatively, see the screen homepage
  • install Virtual Distributed Ethernet (vde) package
    • this package includes the vde_switch application, which is used for establishing network connectivity between UML instances
    • in Debian/Ubuntu, run apt-get install vde2
    • alternatively, see the VDE homepage and download the latest version of vde2
  • install the pre-compiled UML kernel binary, or optionally compile your own UML kernel
  • install custom UML root filesystem
    • download Debian-4.0-x86-root_fs-2008Dec10.bz2 — Debian 4.0 x86 root filesystem + dragon-sw (144 MBytes)
    • install this file into a directory such /a/uml/debian and run bzip2 -d to decompress
  • install VNE
    • download VNE-snapshot-2009Jan12.tar.bz2: Virtual Network Experiments (VNE) snapshot as of 2009-Jan-12 (this version is patched to work with vde_switch)
    • decompress this file into a directory such as /home/dragon
    • install by running: cd /home/dragon/VNE-snapshot-2009Jan12/src; python setup.py install

Configuration

  • This example assume the following structure:
    • everything is running as user dragon
    • location of root filesystem: /a/uml/debian/Debian-4.0-x86-root_fs
    • location of UML kernel binary: /a/uml/debian/uml_linux

  • Create the file ~/.vnerc (in your home directory) that contains the following, changing any paths as needed for your system. The defaults are contained in a text file called VNEDefaults.cfg within the VNE distribution.
[DEFAULT]
kernel     = /a/uml/debian/uml_linux
filesystem = /a/uml/debian/Debian-4.0-x86-root_fs
fstype     = cow
xmllint    = /usr/bin/xmllint
screen     = /usr/bin/screen
xterm      = /usr/X11R6/bin/xterm
uml_switch = /usr/bin/vde_switch
memorysize = 64
logdir     = /home/dragon/uml
removecow  = True
tap_device = tap0

  • Use which to find the location of the binaries on your system, e.g. which screen
    • default location for screen binary is /usr/bin on Debian/Ubuntu
      • Fedora/Redhat systems seem to use /usr/local/sbin/screen
    • default location for xterm binary is /usr/X11R6/bin on Debian/Ubuntu
      • Fedora/Redhat systems install a symlink from /usr/X11R6/bin/xterm to /usr/bin/xterm
    • default location for vde_switch binary is /usr/bin/vde_switch on Debian/Ubuntu
      • vde_switch will be located at /usr/local/bin/vde_switch if you build/install vde2 from source (by default)

  • If your system has a lot of memory, you can change memorysize to a higher number. This will allow you to perform development work, e.g. re-compiling the DRAGON software. We recommend setting memorysize to 256 or 512 for development work.

  • Set UML kernel binary to be executable: chmod +x /a/uml/debian/uml_linux-2.6.27.8

  • Setup a symbolic link to UML kernel binary:
    • run a command such as ln -s /a/uml/debian/uml_linux-2.6.27.8 /a/uml/debian/uml_linux
    • optionally, point .vnerc directly to the UML kernel

  • Setup a symbolic link to Debian 4.0 root filesystem:
    • run a command such as ln -s /a/uml/debian/Debian-4.0-x86-root_fs-2008Dec10 /a/uml/debian/Debian-4.0-x86-root_fs
    • optionally, point .vnerc directly to the root filesystem

  • mkdir ~/uml
    • This is where the copy-on-write (COW) files will be placed, one for each UML instance
    • COW files contain changes/diffs to the read-only root filesystem (so they allow for a full read-write root partition in UML)
    • NOTE: This directory must exist in advance before running VNE! (TODO: fix this at some point)

Usage

  • ssh to UML host with X11 forwarding turned on
    • we highly recommend using compression, as it greatly reduces the bandwidth usage for the X11 forwarding
    • ssh -C -X user@hostname
    • make sure that you have X11Forwarding yes in the server's sshd_config file
  • Run VNE with some XML topology file as input
    • There are explanations and examples of how to provision an LSP inside the comments of each XML file: (located inside the VNE-dragon-snapshot distribution)
      • bridged_triangle.xml —
      • config-4domains.xml —
      • config-narb-inter-uni-csa.xml —
      • config-narb-intra-p2p-csa.xml —
      • config-narb-intra-uni-csa.xml —
      • config-stress.xml —
      • config.xml —
      • example.xml —
      • test_config.xml —
      • testconfig.xml —
    • Assuming you've used the suggested paths, it should be possible to do something like this:
dragon@hydra:~% cd ~/VNE-snapshot-2008Dec10/configs
dragon@hydra:~/VNE-snapshot-2008Dec10/configs% VNE new-dragon-syntax.xml
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Welcome to the text interface of Virtual Network Experiments!
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Important commands: help (or ?), shell (or !) and quit
Type 'help' or press <Tab> twice for a complete list of commands.

> 

  • Try running the config command to dump the configuration loaded from the XML file:
> config
------------------------------------------------------------------------------
                Configuration for es1 (Host)
------------------------------------------------------------------------------
es1: /a/uml/debian/uml_linux mem=128m con1=pts umid=es1 ubda=/home/dragon/uml/es1,/a/uml/debian/Debian-4.0-x86-root_fs eth0=daemon,,unix,/tmp/tmpmA16Q4/ctl
kernel: /a/uml/debian/uml_linux
filesystem: /a/uml/debian/Debian-4.0-x86-root_fs (cow)
memorysize: 64
dragonconfig: [...]

  • Or list ints to list the network interfaces:
> list ints
Interface eth0 on host es1: eth0 (ipv4Address=10.0.0.2, ipv4Netmask=255.0.0.0)
Interface eth0 on host es2: eth0 (ipv4Address=10.0.0.5, ipv4Netmask=255.0.0.0)
Interface eth0 on host switch1: eth0 (macAddress=00:00:AA:BB:00:01, part of bridge br0)
Interface eth1 on host switch1: eth1 (macAddress=00:00:AA:BB:00:02, part of bridge br0)
Interface eth2 on host switch1: eth2 (macAddress=00:00:AA:BB:00:03, part of bridge br0)
Interface eth3 on host switch1: eth3 (macAddress=00:00:AA:BB:00:04, part of bridge br0)
Interface eth0 on host vlsr1: eth0 (ipv4Address=10.0.0.3, ipv4Netmask=255.0.0.0)
Interface eth0 on host vlsr2: eth0 (ipv4Address=10.0.0.4, ipv4Netmask=255.0.0.0)

  • To really get things going, run the following commands and you should get control of the UML instances:
> start all
> xterm all

  • Once the xterms are loaded, click on one, press enter and login as root.
    • By default, there is no password set.

  • In VNE's interactive CLI:
    • config — prints the configuration details to the console
    • start all or start host — start all (or one) of the UML instances
    • xterm all or xterm host — launch xterms for user interaction with all (or one) of the UML instances
    • stop all or stop host — stop all (or one) of the UML instances
    • quit — exit VNE

  • You don't have to use xterms to interact with the instances — you can also start another SSH session to the UML host and run screen -r vlsr1 to connect to the instance named vlsr1.

Screenshots

  • UML screenshot (using the config.xml example from the VNE package):
    UML screenshot (using the config.xml example from the VNE package)

#Movies

Movies

  • Demo movie showing user running VNE and "pinging" over circuits using a simulated data plane:
    VNE Circuit Demo Movie w/ simulated data plane
    • Data plane is simulated using vconfig/brctl on Linux switching nodes to simulate hardware Layer 2 switches

Presentations

Advanced Topics

hostfs

hostfs is an alternative to using COW sparse files. Instead of using an image file for the root filesystem and COW sparse files for the diffs to that read-only filesystem, hostfs allows you to simply mount a directory from the UML host machine. This provides a convenient mechanism to get files in and out of UML instances, share data between instances, etc. It is even possible to make the root partition use hostfs, effectively using a directory on the UML host machine as the root filesystem for the UML instance. When using hostfs with the root partition, root has to be mounted read-only.

If you create a directory on the UML host such as /a/uml/hostfs, you can now access this directory from within the UML instance by running this command inside the instance: mount none /mnt -t hostfs -o /a/uml/hostfs

dumping GMPLS-RSVP-TE/GMPLS-OSPF-TE packets with tshark

tshark is installed in the root filesystem, the CLI version of Ethereal/Wireshark. If you are interested in dumping the RSVP and OSPF packets, run commands such as these on different GRE interfaces to see what is happening with these protocols:

vlsr1:~# tshark -nV -i gre1 ip proto 46
vlsr1:~# tshark -nV -i gre1 ip proto 89

Notes on compiling your own UML kernel

The details about how to compile a custom UML kernel can be found on the User-Mode Linux Home Page: Building from source page.

Below is a copy of the .config file used to build our custom UML kernel:

UML Kernel Debugging

kernel with debugging option turned on can be debugged in gdb, provide example of how to attach gdb to a running UML instance. The pre-compiled UML kernel provided on this page already includes debugging symbols.

Modifying the root filesystem

We use the following script to make changes to the root filesystem. It is important to be sure that no UML instances are running while making modifications, and the copy-on-write (COW) files should also be deleted when making changes to the base filesystem. They will be recreated automatically the next time that you run VNE.

% cat mount_fs_local.sh 
#!/bin/sh
echo make sure there are no linux processes below using the filesystem:
ps auxwww|grep linux
echo removing any existing copy-on-write COW files from ~/uml..
rm -vf ~/uml/*
mount -v /a/uml/debian/Debian-4.0-x86-root_fs /mnt -o loop

Known Bugs

  • There were problems with hardcoded paths in to certain executables in older versions of VNE, but this has been taken care of with the ~/.vnerc file

  • The quit command in the VNE text interface may not properly halt the UML instances
    • reason: VNEDevice.py's deactivate() function is never called...?
    • run stop all to correctly halt all UML instances before running quit

  • Sometimes UML instances can freeze or become unresponsive or not exit cleanly...or VNE may crash for some reason, leaving stale instances around.
    • This is usually sufficient to clean things up:
      • killall uml_linux; sleep 1; killall -9 uml_linux=; rm ~/uml/* (this will remove all of your COW files!)
    • Instances may "hang" in tcsetattr():

  • fsck wants to run after a certain number of bootups, causing problems/delays..
    • UPDATE: This is already fixed in the latest version of the root filesystem included on this page!
    • You may need to delete all of the COW files with rm ~/uml/*
    • /etc/fstab can be modified to prevent this behavior
    • you may also be able to use tune2fs -c 0 -i 0 [root_fs_file] to turn off these checks
    • You may need to "boot up" a single UML instance without a COW file if there is an error on the root filesystem that fsck needs to fix
      • e.g. run /a/uml/debian/uml_linux mem=32m con1=pts umid=test ubda=/a/uml/debian/Debian-4.0-x86-root_fs manually from a shell prompt, login as root, then halt the machine to exit cleanly
      • make sure that the root filesystem /a/uml/debian/Debian-4.0-x86-root_fs is writeable by the user you run this command as (or else fsck will not be able to write changes and may get stuck in a continuous rebooting cycle)

  • /tmp fills up with a lot of directories
    • these are from the UML networking (vde_switch), each directory contains a special FIFO file
    • e.g. /usr/local/bin/vde_switch -unix /tmp/tmprlixlq -hub -tap tap0
    • when VNE/UML is not running, these files can safely be removed, rm -rf /tmp/tmp* (be careful)

  • tshark does not seem to work correctly if only 32MB of RAM is given to the UML instance
    • give the UML instance 128MB and it seems to work OK

  • locale is not being set properly in the current root filesystem
    • consider adding /etc/default/locale file with LANG=C (??)

  • NARB/RCE CLI crashes when pressing backspace key
    • this is because the terminal settings are not being set correctly in UML for some reason
    • e.g. run stty -a to see rows 0; columns 0; in the output
    • then run tset -s to reset the terminal (to Linux)
    • now re-run stty -a to see rows 24; columns 80; in the output
    • if you start the DRAGON NARB/RCE in this environment, it will not crash when you press backspace

Future Work

  • add some checking to increase chances of success
    • does the directory ~/uml/ exist? if not, create it..
    • check if various files exist: screen, xterm, UML kernel, root filesystem, throw an error if they do not

  • data plane simulation
    • currently, only control plane is implemented
    • implementing data plane would allow a user to "ping" across a provisioned LSP, for example
    • use brctl/vconfig in the UML instance to setup the VLAN cross-connect
    • how should VLSR interact with this "software switch" ?
      • possible implementations include:
        • new SwitchCtrl class on VLSR that runs the appropriate brctl/vconfig commands
        • using NET-SNMP's snmpd with the pass_persist option to create an RFC2674-compliant switch
      • Update: This 'software switch' functionality is now supported in the VLSR but has not been tested yet within the UML environment.
    • Update: This actually works now, but the DRAGON software must be recompiled within the UML environment first by running cd /home/dragon/snapshot.current/dragon-sw; ./do_build.sh vlsr-linux (press enter at the prompts) and then ./do_install.sh. When a path is provisioned, the VLSR will run brctl and vconfig to create the forwarding path
      • for a demonstration, see the simulated data plane example under the Movies section of this page
      • To do: config examples need to be committed to the repository and included on this wiki page

  • vde_switch: commit patch to VNE repository to make vde_switch/uml_switch work more seamlessly together

  • make quit command in VNE text interface work correctly (shut down UML instances cleanly)

Related Work

Questions? Comments? Problems?

Please let us know if you have any questions, comments or problems, so that we can make this page better. Thanks!

-- ChrisTracy - 12 Feb 2007

Topic attachments
I Attachment Action Size Date Who Comment
PDFpdf DFN_UML.pdf manage 1230.0 K 2007-06-15 - 02:40 UnknownUser paper on Advanced Network Simulation under User-Mode Linux, a good reference for this topic
PNGpng UML_screenshot_basic_2007_02_13.png manage 206.4 K 2007-02-13 - 15:31 UnknownUser UML screenshot (using the config.xml example from the VNE package)
Unknown file formatbz2 VNE-snapshot-2009Jan12.tar.bz2 manage 12691.2 K 2009-01-12 - 20:53 UnknownUser Virtual Network Experiments (VNE) snapshot as of 2009-Jan-12
PNGpng VNE_overview.png manage 44.6 K 2007-02-13 - 16:41 UnknownUser VNE overview graphic
PDFpdf anp-jvdh-gjv.pdf manage 269.5 K 2007-06-15 - 05:24 UnknownUser Virtual environments for networking experiments
Unknown file formatbz2 uml_linux-2.6.27.8.bz2 manage 15048.1 K 2008-12-10 - 20:03 UnknownUser linux 2.6.27.8 pre-compiled UML kernel (with debugging symbols)
Unknown file formatconfig uml_linux-2.6.27.8.config manage 21.2 K 2008-12-10 - 20:05 UnknownUser .config file used for building uml_linux-2.6.27.8
Unknown file formatbz2 uml_utilities_20070815.tar.bz2 manage 49.0 K 2008-03-27 - 19:48 UnknownUser locally cached copy of uml_utilities source
Unknown file formatbz2 vde2-2.2.2.tar.bz2 manage 461.0 K 2008-12-10 - 20:07 UnknownUser locally cached copy of Virtual Distributed Ethernet (vde) package, includes vde_switch
Topic revision: r22 - 2009-01-12 - ChrisTracy
 
This site is powered by the TWiki collaboration platform Powered by Perl Copyright 1999-2021.
The information contained in these pages is the property of the Mid-Atlantic Crossroads (MAX).
If you have questions or comments, please contact WebBottomBar">MAX Administration