如何用使用DPDK优化VirtIO和OVS网络，针对这个问题，这篇文章详细介绍了相对应的分析和解答，希望可以帮助更多想解决这个问题的小伙伴找到更简单易行的方法。
让客户满意是我们工作的目标，不断超越客户的期望值来自于我们对这个行业的热爱。我们立志把好的技术通过有效、简单的方式提供给客户，将通过不懈努力成为客户在信息化领域值得信任、有价值的长期合作伙伴，公司提供的服务项目有：国际域名空间、虚拟主机 、营销软件、网站建设、单县网站维护 、网站推广。
准备测试环境 一共有2个节点，配置基本相同。节点A用于运行虚拟机，节点B用于测试性能。
查看系统信息 $ cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core) $ uname -a
Linux osdev-gpu 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux $ cat /proc/cpuinfo | tail -n26
processor : 71
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
stepping : 4
microcode : 0x200002c
cpu MHz : 1499.941
cache size : 25344 KB
physical id : 1
siblings : 36
core id : 27
cpu cores : 18
apicid : 119
initial apicid : 119
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req
bogomips : 4604.72
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management: $ free -h
total used free shared buff/cache available
Mem: 754G 5.7G 748G 13M 425M 746G
Swap: 0B 0B 0B # 节点A：
$ lspci | grep Ethernet
1a:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
5f:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
5f:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
$ lspci | awk '$0~/Ethernet/{printf($1 " ");gsub(":","\\:");cmd="ls /sys/bus/pci/devices/0000\\:" $1 "/driver/module/drivers/"; system(cmd)}'
1a:00.0 pci:i40e
1a:00.1 pci:i40e
1a:00.2 pci:i40e
1a:00.3 pci:i40e
5f:00.0 pci:i40e
5f:00.1 pci:i40e
# 节点B：
$ lspci | grep Ethernet
1a:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
1a:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09)
5e:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
5e:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
60:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
60:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
$ lspci | awk '$0~/Ethernet/{printf($1 " ");gsub(":","\\:");cmd="ls /sys/bus/pci/devices/0000\\:" $1 "/driver/module/drivers/"; system(cmd)}'
1a:00.0 pci:i40e
1a:00.1 pci:i40e
1a:00.2 pci:i40e
1a:00.3 pci:i40e
5e:00.0 pci:i40e
5e:00.1 pci:i40e
60:00.0 pci:i40e
60:00.1 pci:i40e 开启IOMMU 如果要使用VFIO驱动，则必须开启IOMMU支持。进入主板的BIOS，打开处理器的Intel VT-d支持。如果需要使用单根虚拟化，则需要同时开启PCI的SR-IOV支持。
$ vi /etc/default/grub
GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt crashkernel=auto biosdevname=0 rhgb quiet" $ grub2-mkconfig -o /boot/grub2/grub.cfg
$ reboot $ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=UUID=1645c2f2-2308-436e-86e6-91ebdc76477e ro intel_iommu=on iommu=pt crashkernel=auto biosdevname=0 rhgb quiet $ dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 000000006ca320d8 002A0 (v01 ALASKA A M I 00000001 INTL 20091013)
[ 0.000000] DMAR: IOMMU enabled
[ 0.348112] DMAR: Host address width 46
[ 0.348114] DMAR: DRHD base: 0x000000d37fc000 flags: 0x0
[ 0.348123] DMAR: dmar0: reg_base_addr d37fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.348124] DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0
[ 0.348128] DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.348129] DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0
[ 0.348133] DMAR: dmar2: reg_base_addr ee7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.348134] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[ 0.348138] DMAR: dmar3: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.348139] DMAR: DRHD base: 0x000000aaffc000 flags: 0x0
[ 0.348156] DMAR: dmar4: reg_base_addr aaffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[ 0.348157] DMAR: DRHD base: 0x000000b87fc000 flags: 0x0
[ 0.348162] DMAR: dmar5: reg_base_addr b87fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
... $ ls /sys/kernel/iommu_groups/
0 11 14 17 2 22 25 28 30 33 36 39 41 44 47 5 52 55 58 60 63 66 69 71 74 77 8 82 85
1 12 15 18 20 23 26 29 31 34 37 4 42 45 48 50 53 56 59 61 64 67 7 72 75 78 80 83 86
10 13 16 19 21 24 27 3 32 35 38 40 43 46 49 51 54 57 6 62 65 68 70 73 76 79 81 84 9 安装Qemu-KVM $ yum install -y epel-release
$ yum install -y centos-release-qemu-ev $ yum install -y qemu-kvm-ev
$ /usr/libexec/qemu-kvm --version
QEMU emulator version 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.14.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
$ qemu-img --version
qemu-img version 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.14.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers 准备虚拟机镜像 下载Cloud镜像 $ yum install -y libvirt libguestfs-tools libguestfs-xfs genisoimage
$ systemctl enable libvirtd && systemctl start libvirtd && systemctl status libvirtd $ export OVS_ROOT=/opt/ovs
$ mkdir -pv $OVS_ROOT/images
$ wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 --directory-prefix $OVS_ROOT/images $ qemu-img info $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2
image: /opt/ovs/images/CentOS-7-x86_64-GenericCloud.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 832M
cluster_size: 65536
Format specific information:
compat: 0.10
refcount bits: 16
$ virt-filesystems --long --parts --blkdevs -h -a $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2
Name Type MBR Size Parent
/dev/sda1 partition 83 8.0G /dev/sda
/dev/sda device - 8.0G -
$ virt-df -h $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2
文件系统 大小 已用空间 可用空间 使用百分比%
CentOS-7-x86_64-GenericCloud.qcow2:/dev/sda1
8.0G 795M 7.2G 10% 扩展镜像分区 $ qemu-img create -f qcow2 $OVS_ROOT/images/CentOS-7-x86_64.qcow2 20G
Formatting '/opt/ovs/images/CentOS-7-x86_64.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 $ virt-resize $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2 $OVS_ROOT/images/CentOS-7-x86_64.qcow2 --expand /dev/sda1
[ 0.0] Examining /opt/ovs/images/CentOS-7-x86_64-GenericCloud.qcow2
25% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒══════════════════════════════════════════⟧ --:--
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ --:--
**********
Summary of changes:
/dev/sda1: This partition will be resized from 8.0G to 20.0G. The
filesystem xfs on /dev/sda1 will be expanded using the 'xfs_growfs' method.
**********
[ 17.3] Setting up initial partition table on /opt/ovs/images/CentOS-7-x86_64.qcow2
[ 17.5] Copying /dev/sda1
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 26.8] Expanding /dev/sda1 using the 'xfs_growfs' method
Resize operation completed with no errors. Before deleting the old disk,
carefully check that the resized disk boots and works correctly. $ qemu-img info $OVS_ROOT/images/CentOS-7-x86_64.qcow2
image: /opt/ovs/images/CentOS-7-x86_64.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 834M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
$ virt-filesystems --long --parts --blkdevs -h -a $OVS_ROOT/images/CentOS-7-x86_64.qcow2
Name Type MBR Size Parent
/dev/sda1 partition 83 20G /dev/sda
/dev/sda device - 20G -
$ virt-df -h $OVS_ROOT/images/CentOS-7-x86_64.qcow2
文件系统 大小 已用空间 可用空间 使用百分比%
CentOS-7-x86_64.qcow2:/dev/sda1 20G 795M 19G 4% 创建元数据镜像 $ vi $OVS_ROOT/images/meta-data
instance-id: centos7-ovs;
local-hostname: centos7-ovs; $ vi $OVS_ROOT/images/user-data
#cloud-config
user: root
password: 123456
chpasswd: { expire: False }
ssh_pwauth: True
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6ijVswX40X2ercmKDZD8mVRD6GFkhdeBM/OjvGd8mSnLDdOjhZw83jhMm/rptNEDlpBW+IjxOZsDO3wm+iVcGn5LblJ3qXtdGEIHlsttFAkLsF8B3jnLBeSTRee8JXZBic5KdfYffq9JC3WrgGJl+OQz6mNW7rqquBFI98QCVlsZEvsJzw5LEm1/ej3Ka2pSkTEei+sB4PBPolnH9cUahq5T8Wwgtlw6JutNob1e5OgFWvPThTRWAtCqLaFWenedagKEA4jPseuF7dq/Eb7nEqL2jYNsWKyR1JpuUdejxAfw434guitORyvLCbj022Sgn5bwEYPIw3EVykcc6XxoZ root@osdev-gpu
final_message: "SYSTEM READY TO LOG IN" $ genisoimage -output $OVS_ROOT/images/centos7-init.iso -volid cidata -joliet -rock $OVS_ROOT/images/user-data $OVS_ROOT/images/meta-data 不使用DPDK的测试 编译安装OVS $ yum -y install gcc autoconf automake libtool kernel kernel-devel $ export OVS_ROOT=/opt/ovs && cd $OVS_ROOT
$ git clone https://github.com/openvswitch/ovs.git
$ cd ovs && git checkout -b v2.9.0/origin v2.9.0 && git checkout -b v2.9.0/devel $ ./boot.sh
$ mkdir -pv $OVS_ROOT/build-nodpdk $OVS_ROOT/target-nodpdk && cd $OVS_ROOT/build-nodpdk
$ ../ovs/configure --enable-shared --with-linux=/lib/modules/$(uname -r)/build --prefix=$OVS_ROOT/target-nodpdk CFLAGS="-g -Ofast"
$ make -j16 'CFLAGS=-g -Ofast -march=native' && make install
$ mkdir -pv $OVS_ROOT/target-nodpdk/modules && cp -vf $OVS_ROOT/build-nodpdk/datapath/linux/*ko $OVS_ROOT/target-nodpdk/modules/ 启动停止OVS $ export OVS_ROOT=/opt/ovs && export OVS_DIR=$OVS_ROOT/target-nodpdk && export PATH=$PATH:$OVS_DIR/share/openvswitch/scripts && cd $OVS_DIR $ ovs-ctl start $ ovs-ctl stop $ mkdir -pv $OVS_DIR/var/run/openvswitch $OVS_DIR/etc/openvswitch
# 创建ovsdb-server数据库：
$ $OVS_DIR/bin/ovsdb-tool create $OVS_DIR/etc/openvswitch/conf.db $OVS_DIR/share/openvswitch/vswitch.ovsschema
# 启动ovsdb-server数据库服务：
$ $OVS_DIR/sbin/ovsdb-server --remote=punix:$OVS_DIR/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach
# 初始化数据库（只在第一次运行时需要执行）：
$ $OVS_DIR/bin/ovs-vsctl --no-wait init
# 查看依赖的内核模块：
$ cat /usr/lib/modules/`uname -r`/modules.dep | awk '$1~"^extra/openvswitch"{for(i=2;i<=NF;i++) {print $i}}' | xargs echo
kernel/net/ipv6/netfilter/nf_nat_ipv6.ko.xz kernel/net/ipv4/netfilter/nf_nat_ipv4.ko.xz kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko.xz kernel/net/netfilter/nf_nat.ko.xz kernel/net/netfilter/nf_conntrack.ko.xz kernel/net/ipv4/udp_tunnel.ko.xz kernel/lib/libcrc32c.ko.xz
# 加载openvswitch.ko依赖的内核模块：
$ cat /usr/lib/modules/`uname -r`/modules.dep | awk '$1~"^extra/openvswitch"{for(i=2;i<=NF;i++) {mod=gensub(/\.ko\.xz/,"",1,gensub(/.*\//,"",1,$i));cmd="modprobe " mod;system(cmd)}}'
# 加载openvswitch.ko内核模块：
$ insmod $OVS_DIR/modules/openvswitch.ko
# 启动ovs-vswitchd守护进程：
$ $OVS_DIR/sbin/ovs-vswitchd --pidfile --detach --log-file $ pkill -9 ovs
$ rm -rfv $OVS_DIR/var/run/openvswitch $OVS_DIR/etc/openvswitch/ $OVS_DIR/etc/openvswitch/conf.db $ vi $OVS_ROOT/ovs-env-nodpdk.sh
export OVS_ROOT=/opt/ovs
export OVS_DIR=$OVS_ROOT/target-nodpdk
export PATH=$OVS_DIR/bin/:$OVS_DIR/share/openvswitch/scripts:$PATH $ vi $OVS_ROOT/start-ovs.sh
#!/bin/bash
$OVS_DIR/sbin/ovsdb-server --remote=punix:$OVS_DIR/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach
cat /usr/lib/modules/`uname -r`/modules.dep | awk '$1~"^extra/openvswitch"{for(i=2;i<=NF;i++) {mod=gensub(/\.ko\.xz/,"",1,gensub(/.*\//,"",1,$i));cmd="modprobe " mod;system(cmd)}}'
insmod $OVS_DIR/modules/openvswitch.ko
$OVS_DIR/sbin/ovs-vswitchd --pidfile --detach --log-file
$ chmod a+x $OVS_ROOT/start-ovs.sh $ vi $OVS_ROOT/stop-ovs.sh
#!/bin/bash
pkill -9 ovs
# rm -rfv $OVS_DIR/var/run/openvswitch $OVS_DIR/etc/openvswitch/ $OVS_DIR/etc/openvswitch/conf.db
$ chmod a+x $OVS_ROOT/stop-ovs.sh 创建网络环境 设置查看环境 $ export OVS_ROOT=/opt/ovs && export OVS_DIR=$OVS_ROOT/target-nodpdk && export PATH=$OVS_DIR/bin/:$OVS_DIR/share/openvswitch/scripts:$PATH
# Or
$ . /opt/ovs/ovs-env-nodpdk.sh $ echo 1 > /proc/sys/net/ipv4/ip_forward # 节点A：
$ ip a | grep -A2 "enp.*: "
2: enp26s0f0: mtu 1500 qdisc noop portid 6c92bf74beac state DOWN qlen 1000
link/ether 6c:92:bf:74:be:ac brd ff:ff:ff:ff:ff:ff
3: enp26s0f1: mtu 1500 qdisc noop portid 6c92bf74bead state DOWN qlen 1000
link/ether 6c:92:bf:74:be:ad brd ff:ff:ff:ff:ff:ff
4: enp26s0f2: mtu 1500 qdisc mq portid 6c92bf74beae state UP qlen 1000
link/ether 6c:92:bf:74:be:ae brd ff:ff:ff:ff:ff:ff
inet 172.29.101.166/24 brd 172.29.101.255 scope global enp26s0f2
--
5: enp26s0f3: mtu 1500 qdisc noop portid 6c92bf74beaf state DOWN qlen 1000
link/ether 6c:92:bf:74:be:af brd ff:ff:ff:ff:ff:ff
6: enp95s0f0: mtu 1500 qdisc mq portid 6c92bf5cdc5e state UP qlen 1000
link/ether 6c:92:bf:5c:dc:5e brd ff:ff:ff:ff:ff:ff
inet 172.29.113.28/24 brd 172.29.113.255 scope global enp95s0f0
--
7: enp95s0f1: mtu 1500 qdisc noop portid 6c92bf5cdc5f state DOWN qlen 1000
link/ether 6c:92:bf:5c:dc:5f brd ff:ff:ff:ff:ff:ff
8: docker0: mtu 1500 qdisc noqueue state UP
# 节点B：
$ ip a | grep -A2 "enp.*: "
2: enp26s0f0: mtu 1500 qdisc mq portid 6c92bf74bfd0 state DOWN qlen 1000
link/ether 6c:92:bf:74:bf:d0 brd ff:ff:ff:ff:ff:ff
3: enp26s0f1: mtu 1500 qdisc mq portid 6c92bf74bfd1 state DOWN qlen 1000
link/ether 6c:92:bf:74:bf:d1 brd ff:ff:ff:ff:ff:ff
4: enp26s0f2: mtu 1500 qdisc mq portid 6c92bf74bfd2 state UP qlen 1000
link/ether 6c:92:bf:74:bf:d2 brd ff:ff:ff:ff:ff:ff
5: enp26s0f3: mtu 1500 qdisc mq portid 6c92bf74bfd3 state UP qlen 1000
link/ether 6c:92:bf:74:bf:d3 brd ff:ff:ff:ff:ff:ff
inet 172.29.101.171/24 brd 172.29.101.255 scope global enp26s0f3
--
6: enp94s0f0: mtu 1500 qdisc mq portid 6c92bf5bc5cd state UP qlen 1000
link/ether 6c:92:bf:5b:c5:cd brd ff:ff:ff:ff:ff:ff
inet 172.29.122.43/24 brd 172.29.122.255 scope global enp94s0f0
--
7: enp94s0f1: mtu 1500 qdisc mq portid 6c92bf5bc5ce state UP qlen 1000
link/ether 6c:92:bf:5b:c5:ce brd ff:ff:ff:ff:ff:ff
8: enp96s0f0: mtu 1500 qdisc mq portid 6c92bf5bc59d state UP qlen 1000
link/ether 6c:92:bf:5b:c5:9d brd ff:ff:ff:ff:ff:ff
9: enp96s0f1: mtu 1500 qdisc mq portid 6c92bf5bc59e state UP qlen 1000
link/ether 6c:92:bf:5b:c5:9e brd ff:ff:ff:ff:ff:ff
10: virbr0: mtu 1500 qdisc noqueue state DOWN qlen 1000 创建NAT网络 在节点A上创建的NAT网络网段为192.168.2.0/24，网关为192.168.2.99。
$ ovs-vsctl add-br br-ovs
$ ip link set dev br-ovs up
$ ip addr add 192.168.2.99/24 dev br-ovs
$ ip addr show br-ovs
21: br-ovs: mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether a2:e8:a6:bb:55:46 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.99/24 scope global br-ovs
valid_lft forever preferred_lft forever
inet6 fe80::a0e8:a6ff:febb:5546/64 scope link
valid_lft forever preferred_lft forever $ iptables -t nat -A POSTROUTING -s 192.168.2.0/24 ! -d 192.168.2.0/24 -j MASQUERADE $ yum install -y DNSmasq
$ dnsmasq --strict-order --except-interface=lo --interface=br-ovs --listen-address=192.168.2.99 --bind-interfaces --dhcp-range=192.168.2.128,192.168.2.192 --conf-file="" --pid-file=/var/run/br-ovs-dhcp.pid --dhcp-leasefile=/var/run/br-ovs-dhcp.leases --dhcp-no-override $ vi $OVS_ROOT/start-nat.sh
#!/bin/bash
#ovs-vsctl del-br br-ovs || return 0
#ovs-vsctl add-br br-ovs
ip link set dev br-ovs up
ip addr add 192.168.2.99/24 dev br-ovs
iptables -t nat -A POSTROUTING -s 192.168.2.0/24 ! -d 192.168.2.0/24 -j MASQUERADE
dnsmasq --strict-order --except-interface=lo --interface=br-ovs --listen-address=192.168.2.99 --bind-interfaces --dhcp-range=192.168.2.128,192.168.2.192 --conf-file="" --pid-file=/var/run/br-ovs-dhcp.pid --dhcp-leasefile=/var/run/br-ovs-dhcp.leases --dhcp-no-override
$ chmod a+x $OVS_ROOT/start-nat.sh 测试NAT网络 $ ip netns add ovs-ns $ ip link add ovs-veth-br type veth peer name ovs-veth-in $ ovs-vsctl add-port br-ovs ovs-veth-br
$ ip link set dev ovs-veth-br up $ ip link set ovs-veth-in netns ovs-ns
$ ip netns exec ovs-ns ip link set dev ovs-veth-in up
$ ip netns exec ovs-ns ip addr add 192.168.2.95/24 dev ovs-veth-in
$ ip netns exec ovs-ns ifconfig
ovs-veth-in: flags=4163 mtu 1500
inet 192.168.2.95 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::25:e8ff:fe78:9b31 prefixlen 64 scopeid 0x20 $ ip netns exec ovs-ns route add default gw 192.168.2.99
$ ip netns exec ovs-ns route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.2.99 0.0.0.0 UG 0 0 0 ovs-veth-in
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 ovs-veth-in $ ip netns exec ovs-ns ping -c 4 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=43 time=49.5 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=43 time=49.0 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=43 time=49.0 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=43 time=49.2 ms
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 49.022/49.197/49.504/0.193 ms
$ ip netns exec ovs-ns ping -c 4 www.163.com
PING 163.xdwscache.ourglb0.com (58.221.28.167) 56(84) bytes of data.
64 bytes from 58.221.28.167 (58.221.28.167): icmp_seq=1 ttl=47 time=15.1 ms
64 bytes from 58.221.28.167 (58.221.28.167): icmp_seq=2 ttl=47 time=14.5 ms
64 bytes from 58.221.28.167 (58.221.28.167): icmp_seq=3 ttl=47 time=14.6 ms
64 bytes from 58.221.28.167 (58.221.28.167): icmp_seq=4 ttl=47 time=14.8 ms
--- 163.xdwscache.ourglb0.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3009ms
rtt min/avg/max/mdev = 14.561/14.810/15.183/0.252 ms $ ip netns exec ovs-ns dhclient
$ tcpdump -n -i br-ovs
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-ovs, link-type EN10MB (Ethernet), capture size 262144 bytes
19:31:30.619593 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 02:25:e8:78:9b:31, length 300
19:31:30.619817 IP 192.168.2.99.bootps > 192.168.2.158.bootpc: BOOTP/DHCP, Reply, length 300
19:31:30.680620 ARP, Request who-has 192.168.2.158 (Broadcast) tell 0.0.0.0, length 28
19:31:31.681806 ARP, Request who-has 192.168.2.158 (Broadcast) tell 0.0.0.0, length 28
19:31:35.628085 ARP, Request who-has 192.168.2.158 tell 192.168.2.99, length 28
19:31:35.628350 ARP, Reply 192.168.2.158 is-at 02:25:e8:78:9b:31, length 28
$ ip netns exec ovs-ns ip addr
1: lo: mtu 65536 qdisc noop state DOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
24: ovs-veth-in@if25: mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 02:25:e8:78:9b:31 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.2.95/24 scope global ovs-veth-in
valid_lft forever preferred_lft forever
inet 192.168.2.158/24 brd 192.168.2.255 scope global secondary dynamic ovs-veth-in
valid_lft 3377sec preferred_lft 3377sec
inet6 fe80::25:e8ff:fe78:9b31/64 scope link
valid_lft forever preferred_lft forever 创建桥接网络 两个节点添加的物理网卡在一个交换机的相同VLAN上。
$ ovs-vsctl add-port br-ovs enp26s0f2
$ ovs-vsctl show
249d4a72-bf90-4250-aafd-23e12b2a0868
Bridge br-ovs
Port "tap-test1"
Interface "tap-test1"
Port br-ovs
Interface br-ovs
type: internal
Port ovs-veth-br
Interface ovs-veth-br
Port "enp26s0f2"
Interface "enp26s0f2"
Port "tap-test2"
Interface "tap-test2" $ ovs-vsctl add-br br-ovs
$ ovs-vsctl set bridge br-ovs stp_enable=false
$ ovs-vsctl add-port br-ovs enp26s0f2
$ ovs-vsctl show
c50241cc-7046-4673-9f4c-c9a33ea3bb28
Bridge br-ovs
Port "enp26s0f2"
Interface "enp26s0f2"
Port br-ovs
Interface br-ovs
type: internal
$ ip link set dev br-ovs up
$ ip addr add 192.168.2.98/24 dev br-ovs # 节点A：
$ ping -c 4 192.168.2.98
PING 192.168.2.98 (192.168.2.98) 56(84) bytes of data.
64 bytes from 192.168.2.98: icmp_seq=1 ttl=64 time=0.545 ms
64 bytes from 192.168.2.98: icmp_seq=2 ttl=64 time=0.089 ms
64 bytes from 192.168.2.98: icmp_seq=3 ttl=64 time=0.081 ms
64 bytes from 192.168.2.98: icmp_seq=4 ttl=64 time=0.082 ms
--- 192.168.2.98 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rtt min/avg/max/mdev = 0.081/0.199/0.545/0.199 ms
# 节点B：
$ ping -c 4 192.168.2.99
PING 192.168.2.99 (192.168.2.99) 56(84) bytes of data.
64 bytes from 192.168.2.99: icmp_seq=1 ttl=64 time=0.113 ms
64 bytes from 192.168.2.99: icmp_seq=2 ttl=64 time=0.092 ms
64 bytes from 192.168.2.99: icmp_seq=3 ttl=64 time=0.085 ms
64 bytes from 192.168.2.99: icmp_seq=4 ttl=64 time=0.081 ms
--- 192.168.2.99 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.081/0.092/0.113/0.017 ms 创建网卡脚本 $ vi $OVS_ROOT/ovs-ifup
#!/bin/sh
set -ue
switch='br-ovs'
/usr/sbin/ip link set dev $1 up
$OVS_DIR/bin/ovs-vsctl add-port ${switch} $1 $ vi $OVS_ROOT/ovs-ifdown
#!/bin/sh
set -ue
switch='br-ovs'
/usr/sbin/ip link set dev $1 down
$OVS_DIR/bin/ovs-vsctl del-port ${switch} $1 $ chmod a+x $OVS_ROOT/ovs-ifup && chmod a+x $OVS_ROOT/ovs-ifup 正式开始测试 $ . /opt/ovs/ovs-env-nodpdk.sh $ qemu-img create -f qcow2 -b $OVS_ROOT/images/CentOS-7-x86_64.qcow2 -o backing_fmt=qcow2 $OVS_ROOT/images/CentOS-7-x86_64_Snapshot1.qcow2 20G
$ qemu-img create -f qcow2 -b $OVS_ROOT/images/CentOS-7-x86_64.qcow2 -o backing_fmt=qcow2 $OVS_ROOT/images/CentOS-7-x86_64_Snapshot2.qcow2 20G $ /usr/libexec/qemu-kvm -smp 2 -m 1024 -serial stdio -cdrom $OVS_ROOT/images/centos7-init.iso -hda $OVS_ROOT/images/CentOS-7-x86_64_Snapshot1.qcow2 -net nic,model=virtio,macaddr=fa:16:3e:4d:58:6f -net tap,ifname=tap-test1,script=$OVS_ROOT/ovs-ifup,downscript=$OVS_ROOT/ovs-ifdown
$ ip addr show eth0
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:4d:58:6f brd ff:ff:ff:ff:ff:ff
inet 192.168.2.139/24 brd 192.168.2.255 scope global dynamic eth0
valid_lft 3509sec preferred_lft 3509sec
inet6 fe80::f816:3eff:fe4d:586f/64 scope link
valid_lft forever preferred_lft forever $ /usr/libexec/qemu-kvm -smp 2 -m 1024 -serial stdio -cdrom $OVS_ROOT/images/centos7-init.iso -hda $OVS_ROOT/images/CentOS-7-x86_64_Snapshot2.qcow2 -net nic,model=virtio,macaddr=fa:16:3e:4d:58:7f -net tap,ifname=tap-test2,script=$OVS_ROOT/ovs-ifup,downscript=$OVS_ROOT/ovs-ifdown
$ ip addr show eth0
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:4d:58:7f brd ff:ff:ff:ff:ff:ff
inet 192.168.2.155/24 brd 192.168.2.255 scope global dynamic eth0
valid_lft 2634sec preferred_lft 2634sec
inet6 fe80::f816:3eff:fe4d:587f/64 scope link
valid_lft forever preferred_lft forever $ yum install -y vim pciutils iperf3
$ lspci | grep Virtio
00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
$ ethtool -i eth0
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:00:03.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no Guest1 $ iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.155, port 41600
[ 5] local 192.168.2.139 port 5201 connected to 192.168.2.155 port 41602
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 401 MBytes 3.36 Gbits/sec
[ 5] 1.00-2.00 sec 415 MBytes 3.48 Gbits/sec
[ 5] 2.00-3.00 sec 419 MBytes 3.52 Gbits/sec
[ 5] 3.00-4.00 sec 414 MBytes 3.47 Gbits/sec
[ 5] 4.00-5.00 sec 320 MBytes 2.69 Gbits/sec
[ 5] 5.00-6.00 sec 404 MBytes 3.39 Gbits/sec
[ 5] 6.00-7.00 sec 408 MBytes 3.43 Gbits/sec
[ 5] 7.00-8.00 sec 386 MBytes 3.24 Gbits/sec
[ 5] 8.00-9.00 sec 373 MBytes 3.13 Gbits/sec
[ 5] 9.00-10.00 sec 375 MBytes 3.15 Gbits/sec
[ 5] 10.00-10.04 sec 15.3 MBytes 3.21 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.04 sec 3.84 GBytes 3.28 Gbits/sec receiver
Guest2 $ iperf3 -c 192.168.2.139
Connecting to host 192.168.2.139, port 5201
[ 4] local 192.168.2.155 port 41602 connected to 192.168.2.139 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 419 MBytes 3.51 Gbits/sec 18 1.30 MBytes
[ 4] 1.00-2.00 sec 415 MBytes 3.48 Gbits/sec 1 1.09 MBytes
[ 4] 2.00-3.00 sec 420 MBytes 3.52 Gbits/sec 0 1.35 MBytes
[ 4] 3.00-4.00 sec 410 MBytes 3.44 Gbits/sec 1 1.14 MBytes
[ 4] 4.00-5.00 sec 324 MBytes 2.72 Gbits/sec 0 1.33 MBytes
[ 4] 5.00-6.00 sec 404 MBytes 3.39 Gbits/sec 1 1.12 MBytes
[ 4] 6.00-7.00 sec 409 MBytes 3.43 Gbits/sec 0 1.36 MBytes
[ 4] 7.00-8.00 sec 385 MBytes 3.23 Gbits/sec 2 1.12 MBytes
[ 4] 8.00-9.00 sec 372 MBytes 3.12 Gbits/sec 0 1.35 MBytes
[ 4] 9.00-10.00 sec 376 MBytes 3.16 Gbits/sec 4 1.09 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 3.84 GBytes 3.30 Gbits/sec 27 sender
[ 4] 0.00-10.00 sec 3.84 GBytes 3.30 Gbits/sec receiver
iperf Done. Guest1 $ iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.98, port 42720
[ 5] local 192.168.2.139 port 5201 connected to 192.168.2.98 port 42722
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 108 MBytes 907 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 939 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec
[ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec
[ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec
[ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec
[ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec
[ 5] 10.00-10.04 sec 4.19 MBytes 942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.04 sec 1.10 GBytes 938 Mbits/sec receiver
Node B $ iperf3 -c 192.168.2.139
Connecting to host 192.168.2.139, port 5201
[ 4] local 192.168.2.98 port 42722 connected to 192.168.2.139 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 114 MBytes 958 Mbits/sec 2 370 KBytes
[ 4] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 0 373 KBytes
[ 4] 2.00-3.00 sec 112 MBytes 940 Mbits/sec 0 379 KBytes
[ 4] 3.00-4.00 sec 113 MBytes 944 Mbits/sec 0 379 KBytes
[ 4] 4.00-5.00 sec 112 MBytes 937 Mbits/sec 0 382 KBytes
[ 4] 5.00-6.00 sec 113 MBytes 946 Mbits/sec 0 383 KBytes
[ 4] 6.00-7.00 sec 112 MBytes 937 Mbits/sec 0 385 KBytes
[ 4] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 386 KBytes
[ 4] 8.00-9.00 sec 112 MBytes 940 Mbits/sec 0 427 KBytes
[ 4] 9.00-10.00 sec 113 MBytes 952 Mbits/sec 0 553 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.10 GBytes 944 Mbits/sec &n
http://www.bluegullmedia.com/article/ggedpg.html
