Disclaimer: This article may contain information about third-party products. Such information is for reference only. Alibaba Cloud does not make any guarantee, express or implied, with respect to the performance and reliability of third-party products, as well as potential impacts of operations on the products.
Overview
This topic describes how to use Pktgen to test the network performance of Elastic Compute Service (ECS) instances.
Background information
Take note of the following items:
- Before you perform risky operations such as modifying instance configurations or data, we recommend that you check the disaster recovery and fault tolerance capabilities of the instances to ensure data security.
- You can modify the configurations and data of instances including ECS and ApsaraDB RDS instances. We recommend that before you modify instance configurations or data, you create snapshots or enable RDS log backup.
- If you have authorized or submitted sensitive information such as the logon account and password in the Alibaba Cloud Management Console, we recommend that you modify these information in a timely manner.
The following steps describe how to use Pktgen to test the network performance of ECS instances:
Step 1: Prepare the environment
Prepare two ECS instances, one for the generator and the other for the receiver. Prepare two network interface controllers (NICs), one for logon and the other for testing. Make sure that the two NICs can communicate with each other. Then, run the ifconfig
command and record the IP addresses and media access control (MAC) addresses of the two NICs, which are required when you configure traffic.
Step 2: Install software
Perform the following operations on both the generator and the receiver:
- Log on to a Linux instance. For more information about how to log on to the Linux instance, see Connect to a Linux instance by using a username and password.
- Run the following command to create the
pktgen
directory:mkdir pktgen
- Run the following command to go to the
pktgen
directory:cd pktgen
- Run the following command to download the installation packages of Pktgen, Lua, and Data Plane Development Kit (DPDK):
wget https://git.dpdk.org/apps/pktgen-dpdk/snapshot/pktgen-dpdk-pktgen-3.6.5.zip https://www.lua.org/ftp/lua-5.4.3.tar.gz https://fast.dpdk.org/rel/dpdk-18.11.11.tar.xz
Note: The versions of Pktgen, Lua, and DPDK must be compatible with each other. Otherwise, compilation errors may occur. In this topic, the versions in the preceding command are used. Before you use the installation packages of other versions, make sure that the versions are compatible with each other.
- Run the following commands in sequence to decompress the installation packages of Pktgen, Lua, and DPDK:
tar -Jxvf dpdk-18.11.11.tar.xz
tar -zxvf lua-5.4.3.tar.gz
unzip pktgen-dpdk-pktgen-3.6.5.zip - If some dependencies are missing during the compilation process, run the following commands to install the corresponding installation packages:
- CentOS 7
yum -y install kernel-devel.x86_64
yum -y install numactl-devel.x86_64
yum -y install elfutils-libelf-devel
yum -y install libpcap-devel - CentOS 8
Note: By default, libpcap-devel is not installed on the CentOS 8 operating system. Run the following commands to install libpcap-devel:
dnf config-manager --add-repo /etc/yum.repos.d/CentOS-PowerTools.repo
dnf config-manager --set-enabled PowerTools
dnf install epel-release
dnf install libpcap-devel
- CentOS 7
- Run the following commands in sequence to compile and install Pktgen:
cd dpdk-stable-18.11.11
make config T=x86_64-native-linuxapp-gcc
make all -j32 - Run the following commands in sequence to compile and install Lua:
cd lua-5.4.3
make
make install - Run the following commands in sequence to compile and install DPDK:
cd pktgen-dpdk-pktgen-3.6.5/
export RTE_SDK=/root/pktgen/dpdk-stable-18.11.11/
export RTE_TARGET=build
make -j32
Step 3: Bind drivers to VFIO
Perform the following operations on both the generator and the receiver:
- View the
/proc/cmdline
file and check whether the line starting with BOOT_IMAGE containsthe intel_iommu=on
parameter. If the line does not contain the intel_iommu=on parameter, edit the /etc/default/grub file, add the intel_iommu=on parameter to the line starting with GRUB_CMDLINE_LINUX, and then restart the ECS instances. - Run the following commands in sequence to install the Virtual Function I/O (VFIO) and VFIO-PCI drivers:
modprobe vfio
modprobe vfio-pci - Run the following command to configure noniommu_mode:
echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
- Run the following command to check and record the bus-info value of the NIC:
ethtool -i eth1
Note: In this topic, the eth1 NIC is used. In actual operations, replace eth1 with your NIC.
A command output similar to the following one is returned:driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:00:08.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no - Run the following command to unbind the virtio driver from eth1.
echo "[$Bus-info]" > /sys/bus/pci/drivers/virtio-pci/unbind
Note: [$Bus-info] is the bus-info value of the NIC obtained in the previous step.
- Run the following command to bind bus-info to the VFIO-PCI driver:
/root/pktgen/dpdk-stable-18.11.11/usertools/dpdk-devbind.py -b vfio-pci [$Bus-info]
Step 4: Configure Hugepages
Perform the following operations on both the generator and the receiver. Check the number of your nodes and run the corresponding commands:
- Dual nodes
echo 8000 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
echo 8000 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages - Single node
echo 8000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
Step 5: Start Pktgen
Perform the following operations on both the generator and the receiver:
- Run the following command in the
codepktgen-dpdk-pktgen-3.6.5code
directory to start Pktgen:
./app/build/pktgen -c 0x1ffff -n 4 --socket-mem 16000 -w [$Bus-info] -- -P -m "[1-16:1-16].0"
Note:
A command output similar to the following one indicates that Pktgen is started:- 0x1ffff: the mask that indicates the CPUs used by threads. In this example, the lowest set bit of the mask is bit0, which indicates that CPU0 is used to control shell and CPU1 to CPU16 are used to generate and receive packets.
- "[1-16:1-16].0" indicates the CPU number. For more information, see the following descriptions. The CPU number cannot exceed the CPU range within which to generate and receive packets in the mask.
- 1-16 (before the colon): CPU number. One of 16 CPUs is used to receive packets.
- 1-16 (after the colon): CPU number. One of 16 CPUs is used to generate packets
- .0 indicates that the port number is 0.
- You can modify the CPU range to improve the generation performance. Note that the more CPUs required for generating packets, the more Hugepages are needed. The Pktgen process may stop due to insufficient Hugepages.
Copyright (c) <2010-2019>, Intel Corporation. All rights reserved. Powered by DPDK
EAL: Detected 64 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:00:08.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 1af4:1000 net_virtio
EAL: using IOMMU type 8 (No-IOMMU)
Lua 5.4.3 Copyright (C) 1994-2021 Lua.org, PUC-Rio
*** Copyright (c) <2010-2019>, Intel Corporation. All rights reserved.
*** Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<
Initialize Port 0 -- TxQ 16, RxQ 16, Src MAC 00:16:3e:24:9b:a4
Port 0: Link Up - speed 10000 Mbps - full-duplex <Enable promiscuous mode>
RX/TX processing lcore: 1 rx: 1 tx: 1
RX/TX processing lcore: 2 rx: 1 tx: 1
RX/TX processing lcore: 3 rx: 1 tx: 1
RX/TX processing lcore: 4 rx: 1 tx: 1
- Ports 0-0 of 1 <Main Page> Copyright (c) <2010-2019>, Intel Corporation
Flags:Port : P-----R--------:0
Link State : <UP-10000-FD> ----TotalRate----
Pkts/s Max/Rx : 1/0 1/0
Max/Tx : 0/0 0/0
MBits/s Rx/Tx : 0/0 0/0
Broadcast : 0
Multicast : 0
Sizes 64 : 0
65-127 : 0
128-255 : 0
256-511 : 0
512-1023 : 0
1024-1518 : 0
Runts/Jumbos : 2/0
ARP/ICMP Pkts : 2/0
Errors Rx/Tx : 0/0
Total Rx Pkts : 2
Tx Pkts : 0
Rx MBs : 0
Tx MBs : 0
:
Pattern Type : abcd...
Tx Count/% Rate : Forever /100%
Pkt Size/Tx Burst : 64 / 64
Port Src/Dest : 1234 / 5678
Pkt Type:VLAN ID : IPv4 / UDP:0001
802.1p CoS/DSCP/IPP : 0/ 0/ 0
VxLAN Flg/Grp/vid : 0000/ 0/ 0
IP Destination : 192.168.X.X
Source : 192.168.X.X/24
MAC Destination : 00:00:00:00:00:00
Source : 00:XX:XX:24:9b:a4
PCI Vendor/Addr : 1af4:1000/00:08.0
-- Pktgen Ver: 3.6.5 (DPDK 18.11.11) Powered by DPDK (pid:4873) -------------
Pktgen:/> - Run the following commands in sequence in Pktgen to configure traffic for the generator:
set 0 proto udp
range 0 src mac start [$MAC]
range 0 src mac min [$MAC]
range 0 src mac inc 00:00:00:00:00:00
range 0 dst mac start ee:ff:ff:ff:ff:ff
range 0 dst mac min ee:ff:ff:ff:ff:ff
range 0 dst mac inc 00:00:00:00:00:00
range 0 src port start 10000
range 0 src port min 10000
range 0 src port max 10000
range 0 src port inc 0
range 0 dst port start 2000
range 0 dst port min 2000
range 0 dst port max 2255
range 0 dst port inc 1
range 0 proto udp
range 0 src ip start [$Local_IP]
range 0 src ip min [$Local_IP]
range 0 src ip inc 0.0.0.0
range 0 size start 64
range 0 size min 64
range 0 size max 64
range 0 size inc 0
range 0 dst ip start [$Receive_IP]
range 0 dst ip min [$Receive_IP]
range 0 dst ip inc 0.0.0.0
enable all rangeNote:
- [$MAC] is the MAC address of the generator.
- [$Local_IP] is the IP address of the generator.
- [$Receive_IP] is the IP address of the receiver.
- Run the following command in Pktgen to start the sending process in the generator:
start 0
View the traffic data:- Generator
-iPorts 0-0 of 1 <Main Page> Copyright (c) <2010-2019>, Intel Corporation
Flags:Port : P-----R--------:0 1/0
Link State : <UP-10000-FD> ----TotalRate----
Pkts/s Max/Rx : 1/0 1/0
Max/Tx : 14906304/14904704 14906304/14904704
MBits/s Rx/Tx : 0/10015 0/10015
Broadcast : 0
Multicast : 0
Sizes 64 : 0
65-127 : 0
128-255 : 0
256-511 : 0
512-1023 : 0
1024-1518 : 0
Runts/Jumbos : 16/0
ARP/ICMP Pkts : 16/0
Errors Rx/Tx : 0/0
Total Rx Pkts : 16
Tx Pkts : 1655962864 - Receiver
- Ports 0-0 of 1 <Main Page> Copyright (c) <2010-2019>, Intel Corporation
Flags:Port : P--------------:0
Link State : <UP-10000-FD> ----TotalRate----
Pkts/s Max/Rx : 14911585/14910795 14911585/14910795
Max/Tx : 0/0 0/0
MBits/s Rx/Tx : 10020/0 10020/0
Broadcast : 0
Multicast : 0
Sizes 64 : 1862740038
65-127 : 0
128-255 : 0
256-511 : 0
512-1023 : 0
1024-1518 : 0
Runts/Jumbos : 6/0
ARP/ICMP Pkts : 6/0
Errors Rx/Tx : 0/0
Total Rx Pkts : 2298743466
Tx Pkts : 0
Rx MBs : 1544755
Tx MBs : 0
:
Pattern Type : abcd...
Tx Count/% Rate : Forever /100%
Pkt Size/Tx Burst : 64 / 64
Port Src/Dest : 1234 / 5678
Pkt Type:VLAN ID : IPv4 / TCP:0001
802.1p CoS/DSCP/IPP : 0/ 0/ 0
VxLAN Flg/Grp/vid : 0000/ 0/ 0
IP Destination : 192.168.X.1
Source : 192.168.X.X/24
MAC Destination : 00:00:00:00:00:00
Source : 00:XX:XX:24:2d:55
PCI Vendor/Addr : 1af4:1000/00:08.0
-- Pktgen Ver: 3.6.5 (DPDK 18.11.11) Powered by DPDK (pid:4072) -------------
- Generator
- Run the following command in the generator to stop the sending process:
stop 0
Application scope
- ECS